karafka 2.5.8 → 2.5.10.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9a0dd67b39af3fbad16ce4c96f061309d97e21ae95bcace2bab1e7879823a622
-  data.tar.gz: 0061a806b411a0526826327be18946fb39fe90c324a82e45d329ef9a45e3168a
+  metadata.gz: 953c11ca73eda62bac76aaca94946b50b880a54e50756b0665da68155c9ad68d
+  data.tar.gz: a5d200ac66bc102d6dac680aaa03df1cc897113dc2af52f4684f3f7128608340
 SHA512:
-  metadata.gz: 7cae09aabcea7cf6eb692bc5747dc51327601eebd680314a62c064c626f79db17a7a6c3f0e49a6a95859eeb4d6e30a51b7a6cb3b7acd1c69400734981e926db6
-  data.tar.gz: 1c1c031bd16d1da91fa7fc3a2627e157db698b51883c91197c6cf5311d19abd8d9be75742db447d588e73c1a30dca196f991a4ec9ee49f31090fcb2863f45b47
+  metadata.gz: 96d8087bda95065a1b56f5a525b9dd53c9ec71d01d9dd51f56403f7a14fd5ee406f2c40a366e1bb607495ebea8ce76ddd379df08e8e76594cdfae5905ff9c3d0
+  data.tar.gz: 393f0264db679e7b71eeb2fe6c419090d6ff46a43354789fcef2e1e7e6baec3c839583982129669d02a6eae1d245396f48c0dcfebe475946ca2aa97dadce90ed

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Karafka Framework Changelog
 
+## 2.5.10 (Unreleased)
+- **[Feature]** Add `Processing::WorkersPool` with dynamic thread pool scaling via `#scale`, `nil` sentinel-based worker exit for downscaling, and `worker.scaling.up`/`worker.scaling.down` instrumentation events.
+- [Enhancement] Add per-partition generation tracking to `AssignmentsTracker` to distinguish first-time assignments from reassignments.
+- [Enhancement] Make liveness listeners fiber-safe.
+- [Fix] Include orphaned node detection in Pro swarm liveness listener.
+- [Fix] Report liveness on `connection.listener.before_fetch_loop` so nodes send an initial healthy report before the first consumption.
+- [Fix] Fire liveness events during `wait_pinging` so nodes continue reporting health status during shutdown with active LRJ jobs.
+- [Maintenance] Use namespaced topic naming format in all integration specs for consistent traceability.
+- [Fix] Fix `DataCollector::SPEC_HASH` producing non-deterministic hashes for pristine and poro specs by passing the original spec path via `KARAFKA_SPEC_PATH` env var.
+- [Maintenance] Add `bin/tests_topics_hashes` script for looking up spec files by their topic name hash prefix.
+- [Change] Require `karafka-rdkafka` `>=` `0.26.1` to support upcoming features relying on low-level Rdkafka APIs.
+
+## 2.5.9 (2026-03-30)
+- [Enhancement] Validate that `statistics.interval.ms` is not zero when dynamic multiplexing is enabled (Pro).
+- [Fix] Fix swarm liveness reporting to also use `on_connection_listener_fetch_loop` so nodes stay alive when `statistics.interval.ms` is disabled.
+- [Fix] Fix `ConfigProxy#producer` when called as a reader resetting the producer to nil instead of returning its value (#3076).
+- [Fix] Fix `Replication#build_generate_command` using `--reassignment-json-file` with `--generate` instead of `--topics-to-move-json-file` and `--broker-list` ([#3087](https://github.com/karafka/karafka/issues/3087)).
+
 ## 2.5.8 (2026-03-23)
 - **[Feature]** Add `Karafka::Admin::Recovery` for coordinator-bypass offset reading and consumer group migration when the Kafka group coordinator is in a FAILED state (Pro).
 

data/LICENSE-COMM

@@ -10,7 +10,7 @@ In order to use the Software under this Agreement, you must either: (a) receive
 
 1. License Grant
 
-1.1 General Use. This Agreement grants you a non-exclusive, non-transferable, limited license to the use rights for the Software, without the right to grant sublicenses, subject to the terms and conditions in this Agreement. The Software is licensed, not sold.
+1.1 General Use. This Agreement grants you a non-exclusive, non-transferable, limited license to the use rights for the Software, without the right to grant sublicenses, subject to the terms and conditions in this Agreement. The Software is licensed, not sold. No patent rights are granted under this Agreement. This license grant covers only the copyright rights necessary to use the Software as expressly permitted herein.
 
 1.1.1 Entity Definition. Where you have purchased or otherwise acquired the Software as or for an entity, "you" and "your" refer exclusively to the single legal entity (such as a corporation, limited liability company, partnership, or other legally recognized organization) that is the named licensee. This license does not extend to any parent company, subsidiary, affiliate, related entity, or other member of a corporate group, regardless of ownership structure or control relationships. Each separate legal entity requires its own license.
 
@@ -30,7 +30,9 @@ In order to use the Software under this Agreement, you must either: (a) receive
 
 3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE AS PART OF A PRODUCT OR SERVICE THAT PROVIDES SIMILAR FUNCTIONALITY TO THE SOFTWARE ITSELF.
 
-3.3 ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING PROHIBITION. WITHOUT LIMITING THE GENERALITY OF SECTION 3.1, THE SOFTWARE AND ITS SOURCE CODE MAY NOT BE USED IN ANY MANNER TO TRAIN, DEVELOP, IMPROVE, OR VALIDATE ANY ARTIFICIAL INTELLIGENCE OR MACHINE LEARNING SYSTEM. THIS PROHIBITION APPLIES REGARDLESS OF WHETHER THE SOFTWARE IS ACCESSED DIRECTLY, THROUGH AUTOMATED MEANS, OR VIA THIRD-PARTY DATASETS OR SERVICES. ANY SUCH USE CONSTITUTES A MATERIAL BREACH OF THIS AGREEMENT AND MAY RESULT IN IMMEDIATE TERMINATION OF YOUR LICENSE AND PURSUIT OF ALL AVAILABLE LEGAL REMEDIES. For clarity, this prohibition applies to the training and development of AI systems, and does not restrict licensed users from utilizing AI-powered development tools in the ordinary course of developing software that integrates with the Software.
+3.3 ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING PROHIBITION. WITHOUT LIMITING THE GENERALITY OF SECTION 3.1, THE SOFTWARE AND ITS SOURCE CODE MAY NOT BE USED IN ANY MANNER TO TRAIN, DEVELOP, IMPROVE, OR VALIDATE ANY ARTIFICIAL INTELLIGENCE OR MACHINE LEARNING SYSTEM. THIS PROHIBITION APPLIES REGARDLESS OF WHETHER THE SOFTWARE IS ACCESSED DIRECTLY, THROUGH AUTOMATED MEANS, OR VIA THIRD-PARTY DATASETS OR SERVICES. ANY SUCH USE CONSTITUTES A MATERIAL BREACH OF THIS AGREEMENT AND MAY RESULT IN IMMEDIATE TERMINATION OF YOUR LICENSE AND PURSUIT OF ALL AVAILABLE LEGAL REMEDIES. For clarity, this prohibition applies to the training and development of AI systems, and does not restrict licensed users from utilizing AI-powered development tools in the ordinary course of developing software that integrates with the Software. Additionally, runtime retrieval of the Software's source code by AI assistants, coding agents, and retrieval-augmented generation (RAG) systems is permitted solely for the purpose of providing contextual help to Karafka users, provided that such retrieval does not result in the storage, reproduction, or use of the source code for training, fine-tuning, or improving any AI or machine learning system.
+
+3.4 Source-Available Access. The Software's source code is made available for reading, referencing, and linking for personal or commercial use. This permits viewing the source code for informational purposes, referencing it in discussions or documentation, and linking to the official repository. Such access does not constitute a license to use, copy, reproduce, modify, or distribute the Software or any portion thereof. All restrictions in Sections 3.1, 3.2, and 3.3 remain fully applicable regardless of how the source code was accessed. Receipt, viewing, or possession of the Software's source code does not convey or imply any license or right beyond those expressly granted in this Agreement.
 
 The Open Source version of the Software ("LGPL Version") is licensed under the terms of the GNU Lesser General Public License version 3.0 ("LGPL") and not under this EULA.
 

data/certs/expired.txt

@@ -1,2 +1,85 @@
 # Expired/revoked license checksums (SHA256 of license token)
 # One checksum per line, lines starting with # are comments
+
+01603767339fd25cf874e6fe29315fb27b6f6bbdf2e769c3b88e6efcf4096d37
+024e2c943693c4ea9eea852eb02d32acc9cb34df21aea90af5ab39dcf3b1d8cf
+0371303ba704d8f256b40932aa7840a1adb1cdc6fd032ac902794320d0d35ff1
+051dc2d8176df8d3bde1595e01af786a3f58ccab4cc48743d624e68125d80bd6
+06e3d91c9881ebf9686f3492e53400adc2c6a5fd4a409f6775cd36094b24f181
+0ae8daf74b8ccd6c4828ed707edf0d05df3e7d328e91ddc973e160ae8b120787
+0b4420e202d9c6075bb108ac4bd5917e35e89aa7d1741afaab93ca07839d06fc
+11936f8e0637565a309157d1fd4e84a3c44e5a3f9f226435e7d38bc43e452db6
+122ecfd0493596239995b9e7425046737b4893e28e33f31815eb4727c3f607d9
+17c14c38e10d0d2bd4a9c847a2c23fd0726459337af1d9815b6e34d1fd9af8be
+227f48f0b22974c43f120de28d5ca33a77e58d33064f9a73748501ba700f1474
+24d59b10d982e5642dfda3ec54cd56dda9022927c89584bbe12d31f3df28e17d
+2634761cc75479548003f941346f0a51a6c535415696f386b40ae28f88f63efa
+284967207a93e9c989d05ad6e5bb0dbe3e0a8a871c3fba48324558f127166bda
+2e2b4ccd902e43feb418cc8b3aed5874ec194ae8495ded864aff917538d91f7a
+39aeae710122bf14729a05f24f5bd702d2a1127e5c7fb9bd90de0e20890575dc
+3ada575caaf0b8bdeca0686a93074de9ed830751e6f46251849475f70e84e408
+4367e57ca36a4b447f23745d76611514df61b56611225be4c8f9a90560bbace9
+47380207ca3041b14c7d4c81a69f820852069ecab4d80a068d39133f61a6e641
+4d49b497ece81604358afc8e9adc7d1163995c4c08446bcd5c0bbfe3d7ebc1e7
+4d97c95c46e51105705964aecbfa5219120295e7d0a172f5c098d12c3a5cbf7e
+60171be4b327d48b2445b00f1fca3d2a3554d06c69f63f3e97742b29030a0a69
+630907de3c6bf258bfc2c51d768d51ca446ebd49fb29b29a0b24d363d46d1a6e
+68db4a9700f6cc3494f8eac1e0df2471f11ef7312e7b58dae839b193e226fae5
+6a06fb85cee01a27a5e819e464b0b5fd84051d234540d11774bc17e90c91abd4
+6d79bcac25c0026dad0f1871ccae79fc07d696df2e0f080ce1d1038454545a05
+6e2ed71a0925d59815c49ea1e72dfd0ba28bdbb57f93d81fa34a31c50db53830
+711a3cced55062707b64f6d652e107d0f1318fd9f3762c732fad23d04884cdd2
+71c2d2ab89a11010c41c9938fbc2565297364b4e8378e78ddf58b22ea2c43ca2
+72b19e6e274b1f36799eea65b07f4849db44c82315647266608ebdb6fde0cad6
+731c0d75d9b47cca49d9d95fc8d291fda452383f920e70849373959892443329
+77b6974b96701d82a0670c79830543e345bf0a00d9e5b5a1bbf8443439d75987
+77e461a9a4534870cfb7cca350d56ed0a504643bb39eeb750b338179d05069a9
+7c06718c4314e28561f337b3ecc66fb012c2510f1ed5fcb82dbab5ba009a8f7d
+7ca5a6183ed9c97e4da2c7030f809bfb32da49319d106377bd0e6d7d52734141
+7ce4cade640a4f34e2909360c4c0ceb8a647b7bfead7a57880d8ce2bf6eea26c
+7d3a2cb0ab08f2f25324124973370da30210dab04abaeeed97192576238a03e0
+7dc790fd8bf18e663c2e4276f4362b0c14795945b508807ef03a92ece266cff8
+81e69732cc6e9c5266fffea08fd1c69fd12c1d841b979723a77533c5efef244e
+839f4a449e0d68acccc0f6a2d0209d2d137375d740ae6d0244dde17e63131fbb
+85d26e4fedf364e20ea33c1b0102d34cea19156243fe781ddf597f6677cf2de3
+8609d26341a5c4e908af8b63bcc6895d99d8cc13f0403dc7b36a2920bb8e048a
+8d3e09a94bd00710a49b186be832f74cc3340ec3d87e780e10159e5e8d048b81
+8e933669352c272935c310d7ce798b3ea2687611378b978e412491e7fddfce4d
+8f3190524285a5c0a0d1e24bdc2a5c417416c2cb92b9da26f0e492aaa4600bd7
+9221a49b42e05d39e1939cf9353805abea2042457c2247859e3274976ac2fbc6
+96c8dde1a7f938899f3f5bd764eff295f090e1c40635e7efd25d69ef55881db2
+9b0b791259e86740d4a2cab736802614938b7043479ac8770c570b1b37f38fb7
+a0db6c5e7c8714a2d33adff3b69d2d688d5a34e6d5ba871d1cbdfcba3edf1e89
+a7ce882d45a00ad82e653b60d14f1a1ebefdc9033d6cc60ed0a2ed507949a9a2
+ac2941c112f397543fde83cc1b0dcf70ec34d77e149d3e90ad07a0f1bc809933
+adea773fadaaaeadf0f20643e84ae6c40bcb55fb73ecee5894a41836453e4abb
+b1411e92ad939b59c898e78520c8ec82547950493a45004b747f12820aa7d390
+b3984622fbc6ba91a2180283a6ada7e1bbe01fb705dbaae37c831e9c893b8e00
+b5ed1ff168e86300735e59dc2786be6d535fc9e0f3c6c8a76bc4354cbbed4ae0
+b9e38e9c1b34db847a4b82ae35382b06e61a5872aa8c24f988d7d5246ded4648
+bdc4f155153a59c25c7c52932f77973487d6336e99eb0beeb90edf4a4120a94a
+bf6c5c58b02fa5e7c7dc845213ad943c5b6613f12ac6012be00b1484bb85aa04
+c011c3d168a159c135ef35bf55121edaf1a4cc94f663a1f461595776b0313160
+c311cb6cfb38a8b2e83b3080c1f87444a52f3f645be8337559a123aaacf1344a
+c87222a94f98c3a2191dda9254bd6ed031fd8ca190f75b40c67a0fcd4b56f795
+ca53af70cd7e67abbb9dbf0a8da6788a9b2192aa3a5a1dd8720aaa68a2511722
+d02da0cbd2c1d83432a2cee7984dd311249692373554002d1d543dd5007f6a58
+d13295e597120b88bf4c4f36de63d0aa3e5a68e3578c09a2e026bd1f6aaaf04f
+d862dd776d5b42a73a63574dc00187c587403101cd211ed348fc422f6460844e
+d9f25b56eae6a62b9383c068c2a7c22ef6b051f1cf4137164ba3dd1baf0bac61
+db1f2c50f3dc6956c498b06f67f82412a76f0b138265a55713076b2a4a35725f
+ded7405c91e9bb1994c58c728f83b705fdc5e816ce0cb3fdf16887127fff27f0
+dfe08da10ae5e74e9bbf648261860f3b94201a11ab4269cd4db7c73017452dde
+dff7372b5ee3136af892561507602423bbae42d12d3a65277ae2af1add666723
+e1a040c0c9a6c5e4603d9aebbc4201075238b47446348ab635b53704b3095a63
+e3b94496b65bb626e723637926e729a33be673fe0355d964306bde563ff6fbf7
+e577d74987f1d31c10fc4ec67dd74084486514f844668d7d01bd4cad2d8f09c4
+e7a91a5ce0e68b888026bbddb57c92c7d5ad61e9416c8188721eee0a8ba85bc1
+eda2ab2c99f769d646dd6fff564af9ec6f32e4462390e3b3246bf953cd7a9c70
+f031aa98b2375c6573f6f775f6000dc81e07b1034694742a900a9115eb124900
+f1ade1306fd64c2e32b15e505d87c0a9ec7a8392f5bea014fc4e70acd17f9935
+f397ccd272b6ee7b33595aa93a6e40b725af265b681efca05bb26f9a7766d6c2
+f54723b14f7cfc4e3c88b946cab881b907dd5ac8372b4600213241ab82778979
+f9cc7b7d9c7d81ba477a70749d166edbc2b44f1714096fce10118bb18fd9696c
+fb417a807d7595a4aff44cc79712e9bc887181e64eb9a98fe9f200b585856ae7
+fd5009f463fe0c38cfe66677ff9b8e5031cbc4ce354883c99c392109ae8ae3b8

data/config/locales/errors.yml

@@ -25,6 +25,9 @@ en:
           Decrease max_wait_time or increase node_report_timeout
         kafka_format: needs to be a filled hash
         key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
+        managed_key_not_supported: >
+          this kafka setting is managed internally by Karafka and should not be set directly.
+          Setting it manually may cause misbehaviours and other unexpected issues
         pause.timeout_max_timeout_vs_pause_max_timeout: pause.timeout must be less or equal to pause.max_timeout
         shutdown_timeout_vs_max_wait_time: shutdown_timeout must be more than max_wait_time
         worker_thread_priority_format: must be between -3 and 3

data/config/locales/pro_errors.yml

@@ -2,6 +2,7 @@ en:
   validations:
     routing:
       swarm_nodes_not_used: 'At least one of the nodes has no assignments'
+      multiplexing_statistics_required_for_dynamic_multiplexing: 'Dynamic multiplexing requires statistics to be enabled (statistics.interval.ms must be greater than 0)'
 
       topic:
         virtual_partitions.partitioner_respond_to_call: needs to be defined and needs to respond to `#call`

data/karafka.gemspec

@@ -21,8 +21,8 @@ Gem::Specification.new do |spec|
     without having to focus on things that are not your business domain.
   DESC
 
-  spec.add_dependency "karafka-core", ">= 2.5.6", "< 2.6.0"
-  spec.add_dependency "karafka-rdkafka", ">= 0.24.0"
+  spec.add_dependency "karafka-core", ">= 2.5.13", "< 2.6.0"
+  spec.add_dependency "karafka-rdkafka", ">= 0.26.1"
   spec.add_dependency "waterdrop", ">= 2.8.14", "< 3.0.0"
   spec.add_dependency "zeitwerk", "~> 2.3"
 

data/lib/karafka/admin/replication.rb

@@ -55,6 +55,7 @@ module Karafka
         :target_replication_factor,
         :partitions_assignment,
         :reassignment_json,
+        :topics_to_move_json,
         :execution_commands,
         :steps
       )
@@ -108,6 +109,7 @@ module Karafka
         @cluster_info = cluster_info
 
         generate_reassignment_json
+        generate_topics_to_move_json
         generate_execution_commands
         generate_steps
 
@@ -256,6 +258,15 @@ module Karafka
         file_path
       end
 
+      # Export the topics-to-move JSON to a file for use with kafka-reassign-partitions.sh
+      # --generate. This optional step lets you ask Kafka to propose its own reassignment plan
+      # so you can compare it against the plan Karafka generated before executing.
+      # @param file_path [String] path where to save the JSON file
+      def export_topics_to_move_file(file_path)
+        File.write(file_path, @topics_to_move_json)
+        file_path
+      end
+
       # @return [String] human-readable summary of the plan
       def summary
         broker_count = @cluster_info[:brokers].size
@@ -428,6 +439,19 @@ module Karafka
         @reassignment_json = JSON.pretty_generate(reassignment_data)
       end
 
+      # Generates the topics-to-move JSON required by kafka-reassign-partitions.sh --generate.
+      # This file is only needed when you want Kafka to propose its own reassignment plan for
+      # comparison against the plan Karafka already computed. It is not required for execution.
+      # @return [void]
+      def generate_topics_to_move_json
+        topics_data = {
+          version: 1,
+          topics: [{ topic: @topic }]
+        }
+
+        @topics_to_move_json = JSON.pretty_generate(topics_data)
+      end
+
       # Generates command templates for executing the reassignment plan
       # Builds generate, execute, and verify command templates with placeholders
       # @return [void]
@@ -439,11 +463,16 @@ module Karafka
         }
       end
 
-      # Builds the kafka-reassign-partitions.sh command for generating reassignment plan
-      # @return [String] command template with placeholder for broker addresses
+      # Builds the kafka-reassign-partitions.sh --generate command that asks Kafka to propose
+      # its own reassignment plan. This is optional — you can compare Kafka's suggestion against
+      # the plan Karafka already computed, or skip this step and go straight to --execute.
+      # @return [String] command template with placeholders for broker addresses and IDs
       def build_generate_command
+        broker_ids = @cluster_info[:brokers].map { |b| b[:node_id] }.sort.join(",")
+
         "kafka-reassign-partitions.sh --bootstrap-server <KAFKA_BROKERS> " \
-          "--reassignment-json-file reassignment.json --generate"
+          "--topics-to-move-json-file topics-to-move.json " \
+          "--broker-list #{broker_ids} --generate"
       end
 
       # Builds the kafka-reassign-partitions.sh command for executing reassignment
@@ -466,7 +495,10 @@ module Karafka
       def generate_steps
         @steps = [
           "1. Export the reassignment JSON using: plan.export_to_file('reassignment.json')",
-          "2. Validate the plan (optional): #{@execution_commands[:generate]}",
+          "2. (Optional) Compare against Kafka's own proposal:",
+          "   a. Export topics-to-move JSON: plan.export_topics_to_move_file('topics-to-move.json')",
+          "   b. Ask Kafka to generate its plan: #{@execution_commands[:generate]}",
+          "   c. Compare Kafka's output with the Karafka-generated reassignment.json above",
           "3. Execute the reassignment: #{@execution_commands[:execute]}",
           "4. Monitor progress: #{@execution_commands[:verify]}",
           "5. Verify completion by checking topic metadata",

data/lib/karafka/connection/client.rb

@@ -401,10 +401,14 @@ module Karafka
       #   - OAUTHBEARER token refresh callbacks
       #
       # @param timeout [Integer] number of milliseconds to wait on events or 0 not to wait.
+      # @param safe [Boolean] when true, rescues Rdkafka::RdkafkaError so callers in
+      #   shutdown/quiet paths do not trigger a full listener reset. When shutting down, errors
+      #   at this layer are not relevant enough. We want to log them but we should not propagate
+      #   them any further.
       #
       # @note It is non-blocking when timeout 0 and will not wait if queue empty. It costs up to
       #   2ms when no callbacks are triggered.
-      def events_poll(timeout = 0)
+      def events_poll(timeout = 0, safe: false)
         kafka.events_poll(timeout)
 
         # Emit event for monitoring - happens once per tick_interval (default 5s)
@@ -414,6 +418,8 @@ module Karafka
           caller: self,
           subscription_group: @subscription_group
         )
+      rescue Rdkafka::RdkafkaError
+        safe ? nil : raise
       end
 
       # Returns pointer to the consumer group metadata. It is used only in the context of

data/lib/karafka/connection/listener.rb

@@ -50,7 +50,7 @@ module Karafka
         @executors = Processing::ExecutorsBuffer.new(@client, subscription_group)
         @partitioner = partitioner_class.new(subscription_group)
         @scheduler = scheduler
-        @events_poller = Helpers::IntervalRunner.new { @client.events_poll }
+        @events_poller = Helpers::IntervalRunner.new { |**opts| @client.events_poll(**opts) }
         # We keep one buffer for messages to preserve memory and not allocate extra objects
         # We can do this that way because we always first schedule jobs using messages before we
         # fetch another batch.
@@ -499,6 +499,7 @@ module Karafka
         until wait_until.call
           @client.ping
           @scheduler.on_manage
+          @events_poller.call(safe: true)
 
           after_ping.call
           sleep(0.2)

data/lib/karafka/helpers/interval_runner.rb

@@ -25,14 +25,14 @@ module Karafka
       end
 
       # Runs the requested code if it was not executed previously recently
-      def call
+      def call(...)
         now = monotonic_now
 
         return if now - @last_called_at < @interval
 
         @last_called_at = now
 
-        @block.call
+        @block.call(...)
       end
 
       # Runs the requested code bypassing any time frequencies

data/lib/karafka/instrumentation/assignments_tracker.rb

@@ -14,10 +14,33 @@ module Karafka
     class AssignmentsTracker
       include Singleton
 
+      class << self
+        # @return [Hash{Karafka::Routing::Topic => Array<Integer>}]
+        # @see #current
+        def current
+          instance.current
+        end
+
+        # @return [Hash{Karafka::Routing::Topic => Hash{Integer => Integer}}]
+        # @see #generations
+        def generations
+          instance.generations
+        end
+
+        # @param topic [Karafka::Routing::Topic]
+        # @param partition [Integer]
+        # @return [Integer]
+        # @see #generation
+        def generation(topic, partition)
+          instance.generation(topic, partition)
+        end
+      end
+
       # Initializes the assignments tracker with empty assignments
       def initialize
         @mutex = Mutex.new
         @assignments = Hash.new { |hash, key| hash[key] = [] }
+        @generations = Hash.new { |h, k| h[k] = {} }
       end
 
       # Returns all the active/current assignments of this given process
@@ -44,10 +67,41 @@ module Karafka
         assignments.freeze
       end
 
-      # Clears all the assignments
+      # Returns the generation counts for all partitions that have ever been assigned
+      #
+      # @return [Hash{Karafka::Routing::Topic => Hash{Integer => Integer}}] topic to partition
+      #   generation mapping. Generation starts at 1 on first assignment and increments on each
+      #   reassignment. Revoked partitions remain in the hash with their last generation value.
+      #
+      # @note Returns a frozen deep copy to prevent external mutation
+      def generations
+        result = {}
+
+        @mutex.synchronize do
+          @generations.each do |topic, partitions|
+            result[topic] = partitions.dup.freeze
+          end
+        end
+
+        result.freeze
+      end
+
+      # Returns the generation count for a specific topic-partition
+      #
+      # @param topic [Karafka::Routing::Topic]
+      # @param partition [Integer]
+      # @return [Integer] generation count (0 if never assigned, 1+ otherwise)
+      def generation(topic, partition)
+        @mutex.synchronize do
+          @generations.dig(topic, partition) || 0
+        end
+      end
+
+      # Clears all the assignments and generations
       def clear
         @mutex.synchronize do
           @assignments.clear
+          @generations.clear
         end
       end
 
@@ -125,7 +179,16 @@ module Karafka
           event[:tpl].to_h.each do |topic, partitions|
             topic = sg.topics.find(topic)
 
-            @assignments[topic] += partitions.map(&:partition)
+            partition_ids = []
+
+            partitions.each do |partition|
+              partition_id = partition.partition
+              partition_ids << partition_id
+              @generations[topic][partition_id] ||= 0
+              @generations[topic][partition_id] += 1
+            end
+
+            @assignments[topic] += partition_ids
             @assignments[topic].sort!
           end
         end

data/lib/karafka/instrumentation/logger_listener.rb

@@ -93,6 +93,25 @@ module Karafka
         MSG
       end
 
+      # Logs info about the workers pool scaling up
+      #
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
+      def on_worker_scaling_up(event)
+        from = event[:from]
+        to = event[:to]
+        info "Workers pool scaled up from #{from} to #{to} workers"
+      end
+
+      # Logs info about the workers pool scaling down.
+      # The actual size change happens asynchronously as workers pick up nil sentinels and exit.
+      #
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
+      def on_worker_scaling_down(event)
+        from = event[:from]
+        to = event[:to]
+        info "Workers pool scaling down from #{from} to #{to} workers"
+      end
+
       # Prints info about a consumer pause occurrence. Irrelevant if user or system initiated.
       #
       # @param event [Karafka::Core::Monitoring::Event] event details including payload

data/lib/karafka/instrumentation/notifications.rb

@@ -108,6 +108,8 @@ module Karafka
         worker.process
         worker.processed
         worker.completed
+        worker.scaling.up
+        worker.scaling.down
 
         error.occurred
       ].freeze

data/lib/karafka/instrumentation/vendors/appsignal/metrics_listener.rb

@@ -328,10 +328,8 @@ module Karafka
 
           # Sends minute based probing metrics
           def minute_probe
-            concurrency = Karafka::App.config.concurrency
-
             count("processes_count", 1, {})
-            count("threads_count", concurrency, {})
+            count("threads_count", Karafka::Server.workers.size, {})
           end
         end
       end

data/lib/karafka/instrumentation/vendors/datadog/metrics_listener.rb

@@ -164,7 +164,7 @@ module Karafka
             jq_stats = event[:jobs_queue].statistics
 
             tags = default_tags
-            gauge("worker.total_threads", Karafka::App.config.concurrency, tags: tags)
+            gauge("worker.total_threads", Karafka::Server.workers.size, tags: tags)
             histogram("worker.processing", jq_stats[:busy], tags: tags)
             histogram("worker.enqueued_jobs", jq_stats[:enqueued], tags: tags)
           end

data/lib/karafka/instrumentation/vendors/kubernetes/liveness_listener.rb

@@ -158,36 +158,39 @@ module Karafka
             @mutex.synchronize(&)
           end
 
-          # @return [Integer] object id of the current thread
-          def thread_id
-            Thread.current.object_id
+          # @return [Integer] object id of the current fiber
+          # @note We use fiber object id instead of thread object id to ensure fiber-safety.
+          #   Multiple fibers can run on the same thread, and using thread id would cause them
+          #   to overwrite each other's timestamps.
+          def fiber_id
+            Fiber.current.object_id
           end
 
-          # Update the polling tick time for current thread
+          # Update the polling tick time for current fiber
           def mark_polling_tick
             synchronize do
-              @pollings[thread_id] = monotonic_now
+              @pollings[fiber_id] = monotonic_now
             end
           end
 
-          # Clear current thread polling time tracker
+          # Clear current fiber polling time tracker
           def clear_polling_tick
             synchronize do
-              @pollings.delete(thread_id)
+              @pollings.delete(fiber_id)
             end
           end
 
           # Update the processing tick time
           def mark_consumption_tick
             synchronize do
-              @consumptions[thread_id] = monotonic_now
+              @consumptions[fiber_id] = monotonic_now
             end
           end
 
-          # Clear current thread consumption time tracker
+          # Clear current fiber consumption time tracker
           def clear_consumption_tick
             synchronize do
-              @consumptions.delete(thread_id)
+              @consumptions.delete(fiber_id)
             end
           end
 

data/lib/karafka/pro/active_job/consumer.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io
 

data/lib/karafka/pro/active_job/dispatcher.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io
 

data/lib/karafka/pro/active_job/job_options_contract.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io
 

data/lib/karafka/pro/admin/recovery/errors.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io