karafka 2.5.10.rc1 → 2.6.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 953c11ca73eda62bac76aaca94946b50b880a54e50756b0665da68155c9ad68d
-  data.tar.gz: a5d200ac66bc102d6dac680aaa03df1cc897113dc2af52f4684f3f7128608340
+  metadata.gz: 04dfc2086393c6d3d1b8c78eca503c425842453163b014dcba68a5c4231fe090
+  data.tar.gz: 5f62160e469b6eb2383234f630654b46fe704e9dedd896e863771c945ab8e30e
 SHA512:
-  metadata.gz: 96d8087bda95065a1b56f5a525b9dd53c9ec71d01d9dd51f56403f7a14fd5ee406f2c40a366e1bb607495ebea8ce76ddd379df08e8e76594cdfae5905ff9c3d0
-  data.tar.gz: 393f0264db679e7b71eeb2fe6c419090d6ff46a43354789fcef2e1e7e6baec3c839583982129669d02a6eae1d245396f48c0dcfebe475946ca2aa97dadce90ed
+  metadata.gz: 3f1bb0b95294bb925b71ff4cd91d0603b538b4f7f9a222a4789aa9aeef89f7f80b8b26ebf2623f1c44585e1de1d87d4be6dbb55748f61de9b1dd90cab68059d7
+  data.tar.gz: 1c79e3b072d6b53c0062f0ade986f357f677cae6d476297389732c3c130fd89e84687526729ba2590763d15d5f0279e08e3ca9fceec386401b2d9d5ec3ef2096

data/CHANGELOG.md

@@ -1,6 +1,24 @@
 # Karafka Framework Changelog
 
-## 2.5.10 (Unreleased)
+## 2.6.0 (Unreleased)
+- [Maintenance] Replace manual per-version `Warning[:performance]` / `Warning[:deprecated]` flag setup in specs with a dynamic `Warning.categories`-based approach that enables all non-experimental warning categories automatically, keeping spec warning configuration in sync with karafka-rdkafka.
+- [Enhancement] Replace per-partition `query_watermark_offsets` consumer calls in `Admin::Topics#read_watermark_offsets` with two batch `list_offsets` admin calls (`:earliest` and `:latest`), reducing N sequential consumer roundtrips to 2 admin roundtrips regardless of how many topics or partitions are queried. Replace per-partition `query_watermark_offsets` fallback in `Admin::ConsumerGroups#seek` (when a time-based offset resolves beyond the last message) with a single batch `list_offsets` call covering all affected partitions. Replace per-partition `query_watermark_offsets` loop in `Pro::Iterator::TplBuilder#resolve_partitions_with_negative_offsets` with one batched `offsets_for_times(epoch)` call for consumer metadata warm-up plus two batch admin `list_offsets` calls for LWM/HWM resolution, reducing N sequential calls to 3 total (Pro).
+- **[Feature]** Add `Karafka::Admin.read_partition_offsets` (and `Admin::Topics#read_partition_offsets`) to expose the `rd_kafka_ListOffsets` admin API. Allows querying partition offsets by spec (`:earliest`, `:latest`, `:max_timestamp`, or a timestamp in ms) without a consumer group. Accepts an optional `isolation_level:` keyword — pass `Karafka::Admin::IsolationLevels::READ_COMMITTED` to get the Last Stable Offset (LSO) instead of the high-watermark, giving accurate lag figures on transactionally-produced topics.
+- **[Feature]** Add `Karafka::Admin::IsolationLevels` module with `READ_UNCOMMITTED` and `READ_COMMITTED` constants. These wrap the underlying `Rdkafka::Bindings` integer constants so callers do not need to reference rdkafka internals directly.
+- **[Feature]** Introduce standalone `Karafka::Declaratives` subsystem (`Declaratives::Topic`, `Declaratives::Repository`, `Declaratives::Builder`) decoupled from routing. Topic declarations can now be defined via `Karafka::App.declaratives.draw` independently of routing blocks. The existing `routing#config(partitions:, ...)` DSL continues to work as a backwards-compatible bridge that populates the shared declaratives repository.
+- [Enhancement] CLI topic commands (`create`, `delete`, `reset`, `repartition`, `align`, `migrate`, `plan`) now read from `Karafka::App.declaratives.topics` instead of walking the routing tree. Topics declared via the standalone `declaratives.draw` DSL are now managed by all CLI commands without requiring a routing entry. The routing bridge sets `bootstrap_servers` on declarations for cluster filtering.
+- [Enhancement] Move consumer-group-specific `Processing` components (`Coordinator`, `CoordinatorsBuffer`, `Executor`, `ExecutorsBuffer`, `Partitioner`, `ExpansionsSelector`, `InlineInsights`) and Pro equivalents (including `SubscriptionGroupsCoordinator`, `Collapser`, `AdaptiveIterator`, `PeriodicJob`, `ParallelSegments`, `VirtualPartitions`, `Filters`, `OffsetMetadata`, `Piping`) under the `ConsumerGroups` namespace (internal). The `config.processing.coordinator_class`, `executor_class`, `partitioner_class`, and `expansions_selector` defaults are updated accordingly; user-provided classes are unaffected. Generic processing components (`JobsQueue`, `Worker`, `WorkersPool`, `Schedulers::*`, `Jobs::Base`) stay in their original locations.
+- [Enhancement] Move processing strategies (`Karafka::Processing::Strategies::*` and `Karafka::Pro::Processing::Strategies::*`) under the `ConsumerGroups` namespace (internal). Prepares the processing layer for parallel `ShareGroups::Strategies` once KIP-932 lands.
+- [Enhancement] Move `Karafka::Processing::StrategySelector` and `Karafka::Pro::Processing::StrategySelector` under the `ConsumerGroups` namespace (internal). The `config.processing.strategy_selector` default is updated accordingly; user-provided selectors are unaffected.
+- [Enhancement] Move `Karafka::Connection::RebalanceManager` to `Karafka::Connection::ConsumerGroups::RebalanceManager` (internal). Prepares the connection layer for a parallel `ShareGroups` namespace once KIP-932 lands.
+- [Enhancement] Move the rebalance librdkafka callback to `Karafka::Instrumentation::Callbacks::ConsumerGroups::Rebalance` (internal). Published event names (`rebalance.*`) and payload shape are unchanged.
+- [Enhancement] Move `Karafka::Instrumentation::Callbacks::Error` and `Karafka::Instrumentation::Callbacks::Statistics` to `Karafka::Instrumentation::Callbacks::ConsumerGroups::Error` and `ConsumerGroups::Statistics` (internal). Completes the callbacks layer namespacing — all consumer-group-specific librdkafka callbacks now live under `Callbacks::ConsumerGroups`. Published event names and payload shape are unchanged.
+- [Enhancement] Namespace consumer-group-specific processing job classes under `Karafka::Processing::ConsumerGroups::Jobs` (OSS: `Consume`, `Eofed`, `Revoked`, `Shutdown`, `Idle`) and `Karafka::Pro::Processing::ConsumerGroups::Jobs` (Pro: `ConsumeNonBlocking`, `EofedNonBlocking`, `Periodic`, `PeriodicNonBlocking`, `RevokedNonBlocking`). Only `Jobs::Base` stays under `Karafka::Processing::Jobs` as the generic base class. Internal move only — no public API changes; prepares the processing layer for parallel `ShareGroups::Jobs` (KIP-932).
+- [Enhancement] Move `Karafka::Processing::JobsBuilder` to `Karafka::Processing::ConsumerGroups::JobsBuilder` and `Karafka::Pro::Processing::JobsBuilder` to `Karafka::Pro::Processing::ConsumerGroups::JobsBuilder` (internal). The jobs builder only creates consumer-group-specific jobs; prepares the processing layer for a parallel `ShareGroups::JobsBuilder` once KIP-932 lands.
+- [Enhancement] Nest consumer-group-specific processing config settings (`coordinator_class`, `errors_tracker_class`, `partitioner_class`, `strategy_selector`, `expansions_selector`, `executor_class`, `jobs_builder`) under `config.internal.processing.consumer_groups`. Shared settings (`jobs_queue_class`, `scheduler_class`, `worker_job_call_wrapper`) remain at the `config.internal.processing` level. Prepares the config structure for a parallel `config.internal.processing.share_groups` sub-namespace once KIP-932 lands.
+- [Enhancement] Move OSS consumer-group-specific routing features (`ActiveJob`, `DeadLetterQueue`, `Eofed`, `InlineInsights`, `ManualOffsetManagement`) from `Routing::Features` to `Routing::Features::ConsumerGroups` (internal). Agnostic features (`Declaratives`, `Deserializers`) remain at the `Routing::Features` level. Prepares the routing layer for a parallel `Routing::Features::ShareGroups` namespace once KIP-932 lands.
+- [Enhancement] Move Pro consumer-group-specific routing features (`ActiveJob`, `AdaptiveIterator`, `DeadLetterQueue`, `Delaying`, `DirectAssignments`, `InlineInsights`, `LongRunningJob`, `Multiplexing`, `NonBlockingJob`, `OffsetMetadata`, `ParallelSegments`, `Patterns`, `PeriodicJob`, `RecurringTasks`, `ScheduledMessages`, `Swarm`, `VirtualPartitions`) from `Pro::Routing::Features` to `Pro::Routing::Features::ConsumerGroups` (internal). Agnostic features (`Expiring`, `Filtering`, `Pausing`, `Throttling`) remain at the `Pro::Routing::Features` level.
+- [Enhancement] Introduce polymorphic `Routing::Topic#group` and `Routing::SubscriptionGroup#group` accessors (with `#consumer_group` kept as a backwards-compatible alias), expose `group_id` alongside `consumer_group_id` in `Routing::Topic#to_h`, emit parallel `group:` / `group_id:` keys alongside `consumer_group:` / `consumer_group_id:` in `rebalance.*` payloads, and emit parallel `group_id:` alongside `consumer_group_id:` in `error.occurred` and `statistics.emitted` payloads (these callbacks only carry the group id, not the group object). Prepares routing for additional group types (e.g. KIP-932 share groups); legacy keys remain and will be retired in Karafka 3.0 once share groups land.
 - **[Feature]** Add `Processing::WorkersPool` with dynamic thread pool scaling via `#scale`, `nil` sentinel-based worker exit for downscaling, and `worker.scaling.up`/`worker.scaling.down` instrumentation events.
 - [Enhancement] Add per-partition generation tracking to `AssignmentsTracker` to distinguish first-time assignments from reassignments.
 - [Enhancement] Make liveness listeners fiber-safe.
@@ -9,8 +27,12 @@
 - [Fix] Fire liveness events during `wait_pinging` so nodes continue reporting health status during shutdown with active LRJ jobs.
 - [Maintenance] Use namespaced topic naming format in all integration specs for consistent traceability.
 - [Fix] Fix `DataCollector::SPEC_HASH` producing non-deterministic hashes for pristine and poro specs by passing the original spec path via `KARAFKA_SPEC_PATH` env var.
+- [Fix] Fix swarm supervisor leaking reader pipe file descriptors on node restarts by closing the old reader before creating a new pipe in `Node#start`.
 - [Maintenance] Add `bin/tests_topics_hashes` script for looking up spec files by their topic name hash prefix.
-- [Change] Require `karafka-rdkafka` `>=` `0.26.1` to support upcoming features relying on low-level Rdkafka APIs.
+- [Change] Require `karafka-rdkafka` `>=` `0.27.1` to pick up the fix for `poll_batch` and `poll_batch_nb` raising `RdkafkaError` with only the integer error code, which discarded topic/partition/offset context from `e.details` and caused `partition_eof` handling to call `@buffer.eof(nil, nil)`, resulting in a `TopicNotFoundError` crash.
+- [Fix] Raise `InvalidLicenseTokenError` when a manually-defined `Karafka::License` module is missing required methods (`#token` or `#version`) instead of silently skipping pro component loading, which previously caused confusing `NameError` exceptions later in the boot process.
+- [Enhancement] Bump swarm supervisor `SHUTDOWN_GRACE_PERIOD` from 1s to 15s to give forked nodes enough time to finish post-`shutdown_timeout` cleanup (at_exit handlers, librdkafka finalization, connection pool close) before the supervisor forcefully terminates them, especially on CI where `sleep` granularity and `waitpid` cost stretch each supervision loop iteration.
+- [Maintenance] Fix flaky `pro/admin/recovery/coordinator_for_spec.rb` by warming up the `__consumer_offsets` internal topic with a produce + `seek_consumer_group` and retrying the first `Recovery.coordinator_for` call with exponential backoff on `MetadataError`.
 
 ## 2.5.9 (2026-03-30)
 - [Enhancement] Validate that `statistics.interval.ms` is not zero when dynamic multiplexing is enabled (Pro).

data/config/locales/errors.yml

@@ -34,16 +34,16 @@ en:
 
         oauth.token_provider_listener_format: 'must be false or respond to #on_oauthbearer_token_refresh'
 
-        internal.processing.jobs_builder_format: cannot be nil
         internal.processing.jobs_queue_class_format: cannot be nil
         internal.processing.scheduler_class_format: cannot be nil
-        internal.processing.coordinator_class_format: cannot be nil
-        internal.processing.partitioner_class_format: cannot be nil
-        internal.processing.strategy_selector_format: cannot be nil
-        internal.processing.expansions_selector_format: cannot be nil
-        internal.processing.executor_class_format: cannot be nil
         internal.processing.worker_job_call_wrapper_format: 'needs to be false or respond to #wrap'
-        internal.processing.errors_tracker_class_format: 'needs to be nil or a class'
+        internal.processing.consumer_groups.jobs_builder_format: cannot be nil
+        internal.processing.consumer_groups.coordinator_class_format: cannot be nil
+        internal.processing.consumer_groups.errors_tracker_class_format: 'needs to be nil or a class'
+        internal.processing.consumer_groups.partitioner_class_format: cannot be nil
+        internal.processing.consumer_groups.strategy_selector_format: cannot be nil
+        internal.processing.consumer_groups.expansions_selector_format: cannot be nil
+        internal.processing.consumer_groups.executor_class_format: cannot be nil
 
         internal.active_job.dispatcher_format: cannot be nil
         internal.active_job.job_options_contract_format: cannot be nil
@@ -176,6 +176,13 @@ en:
           all topic names within a single consumer group must be unique considering namespacing styles
           disable this validation by setting config.strict_topics_namespacing to false
 
+    declaratives:
+      topic:
+        declaratives.partitions_format: needs to be more or equal to 1
+        declaratives.active_format: needs to be either true or false
+        declaratives.replication_factor_format: needs to be more or equal to 1
+        declaratives.details_format: needs to be a hash with only symbol keys
+
     job_options:
       missing: needs to be present
       dispatch_method_format: needs to be either :produce_async or :produce_sync

data/karafka.gemspec

@@ -21,9 +21,9 @@ Gem::Specification.new do |spec|
     without having to focus on things that are not your business domain.
   DESC
 
-  spec.add_dependency "karafka-core", ">= 2.5.13", "< 2.6.0"
-  spec.add_dependency "karafka-rdkafka", ">= 0.26.1"
-  spec.add_dependency "waterdrop", ">= 2.8.14", "< 3.0.0"
+  spec.add_dependency "karafka-core", ">= 2.6.0", "< 2.7.0"
+  spec.add_dependency "karafka-rdkafka", ">= 0.27.2"
+  spec.add_dependency "waterdrop", ">= 2.10.1", "< 3.0.0"
   spec.add_dependency "zeitwerk", "~> 2.3"
 
   spec.required_ruby_version = ">= 3.2.0"

data/lib/active_job/queue_adapters/karafka_adapter.rb

@@ -28,8 +28,7 @@ module ActiveJob
       Object
     end
 
-    # Karafka adapter for enqueuing jobs
-    # This is here for ease of integration with ActiveJob.
+    # Karafka adapter for enqueuing jobs This is here for ease of integration with ActiveJob.
     class KarafkaAdapter < base
       include ::Karafka::Helpers::ConfigImporter.new(
         dispatcher: %i[internal active_job dispatcher]

data/lib/karafka/active_job/job_extensions.rb

@@ -19,8 +19,7 @@ module Karafka
       def karafka_options(new_options = {})
         return _karafka_options if new_options.empty?
 
-        # Make sure, that karafka options that someone wants to use are valid before assigning
-        # them
+        # Make sure, that karafka options that someone wants to use are valid before assigning them
         App.config.internal.active_job.job_options_contract.validate!(
           new_options,
           scope: %w[active_job]

data/lib/karafka/admin/configs/resource.rb

@@ -76,8 +76,7 @@ module Karafka
 
         private
 
-        # Recognizes whether the type is provided and remaps it to a symbol representation if
-        # needed
+        # Recognizes whether the type is provided and remaps it to a symbol representation if needed
         #
         # @param type [Symbol, Integer]
         # @return [Symbol]

data/lib/karafka/admin/consumer_groups.rb

@@ -18,65 +18,64 @@ module Karafka
       private_constant :LONG_TIME_AGO, :DAY_IN_SECONDS
 
       class << self
-        # @param consumer_group_id [String] consumer group for which we want to move offsets
+        # @param group_id [String] group for which we want to move offsets
         # @param topics_with_partitions_and_offsets [Hash] hash with topics and settings
         # @see #seek
-        def seek(consumer_group_id, topics_with_partitions_and_offsets)
-          new.seek(consumer_group_id, topics_with_partitions_and_offsets)
+        def seek(group_id, topics_with_partitions_and_offsets)
+          new.seek(group_id, topics_with_partitions_and_offsets)
         end
 
-        # @param previous_name [String] old consumer group name
-        # @param new_name [String] new consumer group name
+        # @param previous_name [String] old group name
+        # @param new_name [String] new group name
         # @param topics [Array<String>] topics for which we want to copy offsets
         # @see #copy
         def copy(previous_name, new_name, topics)
           new.copy(previous_name, new_name, topics)
         end
 
-        # @param previous_name [String] old consumer group name
-        # @param new_name [String] new consumer group name
+        # @param previous_name [String] old group name
+        # @param new_name [String] new group name
         # @param topics [Array<String>] topics for which we want to migrate offsets
-        # @param delete_previous [Boolean] should we delete previous consumer group after rename
+        # @param delete_previous [Boolean] should we delete previous group after rename
         # @see #rename
         def rename(previous_name, new_name, topics, delete_previous: true)
           new.rename(previous_name, new_name, topics, delete_previous: delete_previous)
         end
 
-        # @param consumer_group_id [String] consumer group name
+        # @param group_id [String] group name
         # @see #delete
-        def delete(consumer_group_id)
-          new.delete(consumer_group_id)
+        def delete(group_id)
+          new.delete(group_id)
         end
 
-        # @param consumer_group_id [String] consumer group id to trigger rebalance for
+        # @param group_id [String] group id to trigger rebalance for
         # @see #trigger_rebalance
-        def trigger_rebalance(consumer_group_id)
-          new.trigger_rebalance(consumer_group_id)
+        def trigger_rebalance(group_id)
+          new.trigger_rebalance(group_id)
         end
 
-        # @param consumer_groups_with_topics [Hash{String => Array<String>}] hash with consumer
-        #   groups names with array of topics
+        # @param groups_with_topics [Hash{String => Array<String>}] hash with group names with
+        #   array of topics
         # @param active_topics_only [Boolean] if set to false, will select also inactive topics
         # @see #read_lags_with_offsets
-        def read_lags_with_offsets(consumer_groups_with_topics = {}, active_topics_only: true)
+        def read_lags_with_offsets(groups_with_topics = {}, active_topics_only: true)
           new.read_lags_with_offsets(
-            consumer_groups_with_topics,
+            groups_with_topics,
             active_topics_only: active_topics_only
           )
         end
       end
 
-      # Moves the offset on a given consumer group and provided topic to the requested location
+      # Moves the offset on a given group and provided topic to the requested location
       #
-      # @param consumer_group_id [String] id of the consumer group for which we want to move the
-      #   existing offset
+      # @param group_id [String] id of the group for which we want to move the existing offset
       # @param topics_with_partitions_and_offsets [Hash] Hash with list of topics and settings to
       #   where to move given consumer. It allows us to move particular partitions or whole
       #   topics if we want to reset all partitions to for example a point in time.
       #
       # @return [void]
       #
-      # @note This method should **not** be executed on a running consumer group as it creates a
+      # @note This method should **not** be executed on a running group as it creates a
       #   "fake" consumer and uses it to move offsets.
       #
       # @example Move a single topic partition nr 1 offset to 100
@@ -99,7 +98,7 @@ module Karafka
       #
       # @example Move offset of a single partition to latest
       #   Karafka::Admin::ConsumerGroups.seek('group-id', { 'topic' => { 1 => 'latest' } })
-      def seek(consumer_group_id, topics_with_partitions_and_offsets)
+      def seek(group_id, topics_with_partitions_and_offsets)
         tpl_base = {}
 
         # Normalize the data so we always have all partitions and topics in the same format
@@ -161,28 +160,41 @@ module Karafka
           end
         end
 
-        settings = { "group.id": consumer_group_id }
+        settings = { "group.id": group_id }
 
         with_consumer(settings) do |consumer|
           # If we have any time based stuff to resolve, we need to do it prior to commits
           unless time_tpl.empty?
             real_offsets = consumer.offsets_for_times(time_tpl)
 
-            real_offsets.to_h.each do |name, results|
+            # Collect all partitions where the timestamp is beyond the last message
+            # (negative offset returned) so we can fetch their HWMs in a single batch
+            # admin call instead of one per-partition consumer call
+            real_offsets_by_topic = real_offsets.to_h
+            hwm_specs = {}
+            real_offsets_by_topic.each do |name, results|
               results.each do |result|
                 raise(Errors::InvalidTimeBasedOffsetError) unless result
+                next unless result.offset.negative?
 
-                partition = result.partition
+                (hwm_specs[name] ||= []) << { partition: result.partition, offset: :latest }
+              end
+            end
 
-                # Negative offset means we're beyond last message and we need to query for the
-                # high watermark offset to get the most recent offset and move there
-                if result.offset.negative?
-                  _, offset = consumer.query_watermark_offsets(name, result.partition)
-                else
-                  # If we get an offset, it means there existed a message close to this time
-                  # location
-                  offset = result.offset
+            hwms = {}
+            unless hwm_specs.empty?
+              with_admin do |admin|
+                admin.list_offsets(hwm_specs).wait(max_wait_timeout_ms: max_wait_time_ms).offsets.each do |r|
+                  (hwms[r[:topic]] ||= {})[r[:partition]] = r[:offset]
                 end
+              end
+            end
+
+            real_offsets_by_topic.each do |name, results|
+              results.each do |result|
+                partition = result.partition
+                # If offset is negative the timestamp was beyond the last message; use HWM
+                offset = result.offset.negative? ? hwms.dig(name, partition) : result.offset
 
                 # Since now we have proper offsets, we can add this to the final tpl for commit
                 tpl.to_h[name] ||= []
@@ -198,18 +210,18 @@ module Karafka
         end
       end
 
-      # Takes consumer group and its topics and copies all the offsets to a new named group
+      # Takes group and its topics and copies all the offsets to a new named group
       #
-      # @param previous_name [String] old consumer group name
-      # @param new_name [String] new consumer group name
+      # @param previous_name [String] old group name
+      # @param new_name [String] new group name
       # @param topics [Array<String>] topics for which we want to migrate offsets during rename
       #
       # @return [Boolean] true if anything was migrated, otherwise false
       #
-      # @note This method should **not** be executed on a running consumer group as it creates a
+      # @note This method should **not** be executed on a running group as it creates a
       #   "fake" consumer and uses it to move offsets.
       #
-      # @note If new consumer group exists, old offsets will be added to it.
+      # @note If new group exists, old offsets will be added to it.
       def copy(previous_name, new_name, topics)
         remap = Hash.new { |h, k| h[k] = {} }
 
@@ -236,24 +248,24 @@ module Karafka
         true
       end
 
-      # Takes consumer group and its topics and migrates all the offsets to a new named group
+      # Takes group and its topics and migrates all the offsets to a new named group
       #
-      # @param previous_name [String] old consumer group name
-      # @param new_name [String] new consumer group name
+      # @param previous_name [String] old group name
+      # @param new_name [String] new group name
       # @param topics [Array<String>] topics for which we want to migrate offsets during rename
-      # @param delete_previous [Boolean] should we delete previous consumer group after rename.
+      # @param delete_previous [Boolean] should we delete previous group after rename.
       #   Defaults to true.
       #
       # @return [Boolean] true if rename (and optionally removal) was ok or false if there was
       #   nothing really to rename
       #
-      # @note This method should **not** be executed on a running consumer group as it creates a
+      # @note This method should **not** be executed on a running group as it creates a
       #   "fake" consumer and uses it to move offsets.
       #
       # @note After migration unless `delete_previous` is set to `false`, old group will be
       #   removed.
       #
-      # @note If new consumer group exists, old offsets will be added to it.
+      # @note If new group exists, old offsets will be added to it.
       def rename(previous_name, new_name, topics, delete_previous: true)
         copy_result = copy(previous_name, new_name, topics)
 
@@ -265,35 +277,35 @@ module Karafka
         true
       end
 
-      # Removes given consumer group (if exists)
+      # Removes given group (if exists)
       #
-      # @param consumer_group_id [String] consumer group name
+      # @param group_id [String] group name
       #
       # @return [void]
       #
-      # @note This method should not be used on a running consumer group as it will not yield any
+      # @note This method should not be used on a running group as it will not yield any
       #   results.
-      def delete(consumer_group_id)
+      def delete(group_id)
         with_admin do |admin|
-          handler = admin.delete_group(consumer_group_id)
+          handler = admin.delete_group(group_id)
           handler.wait(max_wait_timeout_ms: max_wait_time_ms)
         end
       end
 
-      # Triggers a rebalance for the specified consumer group by briefly joining and leaving
+      # Triggers a rebalance for the specified group by briefly joining and leaving
       #
-      # @param consumer_group_id [String] consumer group id to trigger rebalance for
+      # @param group_id [String] group id to trigger rebalance for
       #
       # @return [void]
       #
-      # @raise [Karafka::Errors::InvalidConfigurationError] when consumer group is not found in
+      # @raise [Karafka::Errors::InvalidConfigurationError] when group is not found in
       #   routing or has no topics
       #
-      # @note This method creates a temporary "fake" consumer that joins the consumer group,
+      # @note This method creates a temporary "fake" consumer that joins the group,
       #   triggering a rebalance when it joins and another when it leaves. This should only be
       #   used for operational/testing purposes as it causes two rebalances.
       #
-      # @note The consumer group does not need to be running for this to work, but if it is,
+      # @note The group does not need to be running for this to work, but if it is,
       #   it will experience rebalances.
       #
       # @note The behavior follows the configured rebalance protocol. For cooperative sticky
@@ -302,38 +314,38 @@ module Karafka
       #   stopping all consumers.
       #
       # @note Topics are always detected from the routing configuration. The consumer settings
-      #   (kafka config) are taken from the first topic in the consumer group to ensure
+      #   (kafka config) are taken from the first topic in the group to ensure
       #   consistency with the actual consumer configuration.
       #
-      # @example Trigger rebalance for a consumer group
+      # @example Trigger rebalance for a group
       #   Karafka::Admin::ConsumerGroups.trigger_rebalance('my-group')
-      def trigger_rebalance(consumer_group_id)
-        consumer_group = Karafka::App.routes.find { |cg| cg.id == consumer_group_id }
+      def trigger_rebalance(group_id)
+        group = Karafka::App.routes.find { |g| g.id == group_id }
 
-        unless consumer_group
+        unless group
           raise(
             Errors::InvalidConfigurationError,
-            "Consumer group '#{consumer_group_id}' not found in routing"
+            "Group '#{group_id}' not found in routing"
           )
         end
 
-        topics = consumer_group.topics.map(&:name)
+        topics = group.topics.map(&:name)
 
         if topics.empty?
           raise(
             Errors::InvalidConfigurationError,
-            "Consumer group '#{consumer_group_id}' has no topics"
+            "Group '#{group_id}' has no topics"
           )
         end
 
         # Get the first topic to extract kafka settings
-        first_topic = consumer_group.topics.first
+        first_topic = group.topics.first
 
-        # Build consumer settings using the consumer group's kafka config from first topic
+        # Build consumer settings using the group's kafka config from first topic
         # This ensures we use the same settings as the actual consumers
         # Following the same pattern as in Karafka::Connection::Client#build_kafka
         consumer_settings = Setup::AttributesMap.consumer(first_topic.kafka.dup)
-        consumer_settings[:"group.id"] = consumer_group.id
+        consumer_settings[:"group.id"] = group.id
         consumer_settings[:"enable.auto.offset.store"] = false
         consumer_settings[:"auto.offset.reset"] ||= first_topic.initial_offset
 
@@ -350,26 +362,25 @@ module Karafka
         end
       end
 
-      # Reads lags and offsets for given topics in the context of consumer groups defined in the
-      #   routing
+      # Reads lags and offsets for given topics in the context of groups defined in the routing
       #
-      # @param consumer_groups_with_topics [Hash{String => Array<String>}] hash with consumer
-      #   groups names with array of topics to query per consumer group inside
+      # @param groups_with_topics [Hash{String => Array<String>}] hash with group names with
+      #   array of topics to query per group inside
       # @param active_topics_only [Boolean] if set to false, when we use routing topics, will
       #   select also topics that are marked as inactive in routing
       #
       # @return [Hash{String => Hash{Integer => Hash{Integer => Object}}}] hash where the top
-      #   level keys are the consumer groups and values are hashes with topics and inside
+      #   level keys are the groups and values are hashes with topics and inside
       #   partitions with lags and offsets
       #
       # @note For topics that do not exist, topic details will be set to an empty hash
       #
-      # @note For topics that exist but were never consumed by a given CG we set `-1` as lag and
-      #   the offset on each of the partitions that were not consumed.
+      # @note For topics that exist but were never consumed by a given group we set `-1` as lag
+      #   and the offset on each of the partitions that were not consumed.
       #
       # @note This lag reporting is for committed lags and is "Kafka-centric", meaning that this
       #   represents lags from Kafka perspective and not the consumer. They may differ.
-      def read_lags_with_offsets(consumer_groups_with_topics = {}, active_topics_only: true)
+      def read_lags_with_offsets(groups_with_topics = {}, active_topics_only: true)
         # We first fetch all the topics with partitions count that exist in the cluster so we
         # do not query for topics that do not exist and so we can get partitions count for all
         # the topics we may need. The non-existent and not consumed will be filled at the end
@@ -377,82 +388,82 @@ module Karafka
           [topic[:topic_name], topic[:partition_count]]
         end.freeze
 
-        # If no expected CGs, we use all from routing that have active topics
-        if consumer_groups_with_topics.empty?
-          consumer_groups_with_topics = Karafka::App.routes.to_h do |cg|
-            cg_topics = cg.topics.select do |cg_topic|
-              active_topics_only ? cg_topic.active? : true
+        # If no expected groups, we use all from routing that have active topics
+        if groups_with_topics.empty?
+          groups_with_topics = Karafka::App.routes.to_h do |group|
+            group_topics = group.topics.select do |group_topic|
+              active_topics_only ? group_topic.active? : true
             end
 
-            [cg.id, cg_topics.map(&:name)]
+            [group.id, group_topics.map(&:name)]
           end
         end
 
         # We make a copy because we will remove once with non-existing topics
-        # We keep original requested consumer groups with topics for later backfilling
-        cgs_with_topics = consumer_groups_with_topics.dup
-        cgs_with_topics.transform_values!(&:dup)
+        # We keep original requested groups with topics for later backfilling
+        queryable_groups = groups_with_topics.dup
+        queryable_groups.transform_values!(&:dup)
 
         # We can query only topics that do exist, this is why we are cleaning those that do not
         # exist
-        cgs_with_topics.each_value do |requested_topics|
+        queryable_groups.each_value do |requested_topics|
           requested_topics.delete_if { |topic| !existing_topics.include?(topic) }
         end
 
         groups_lags = Hash.new { |h, k| h[k] = {} }
         groups_offs = Hash.new { |h, k| h[k] = {} }
 
-        cgs_with_topics.each do |cg, topics|
+        queryable_groups.each do |group, topics|
           # Do not add to tpl topics that do not exist
           next if topics.empty?
 
           tpl = Rdkafka::Consumer::TopicPartitionList.new
 
-          with_consumer("group.id": cg) do |consumer|
+          with_consumer("group.id": group) do |consumer|
             topics.each { |topic| tpl.add_topic(topic, existing_topics[topic]) }
 
             commit_offsets = consumer.committed(tpl)
 
             commit_offsets.to_h.each do |topic, partitions|
-              groups_offs[cg][topic] = {}
+              groups_offs[group][topic] = {}
 
               partitions.each do |partition|
                 # -1 when no offset is stored
-                groups_offs[cg][topic][partition.partition] = partition.offset || -1
+                groups_offs[group][topic][partition.partition] = partition.offset || -1
               end
             end
 
             consumer.lag(commit_offsets).each do |topic, partitions_lags|
-              groups_lags[cg][topic] = partitions_lags
+              groups_lags[group][topic] = partitions_lags
             end
           end
         end
 
-        consumer_groups_with_topics.each do |cg, topics|
-          groups_lags[cg]
+        groups_with_topics.each do |group, topics|
+          groups_lags[group]
 
           topics.each do |topic|
-            groups_lags[cg][topic] ||= {}
+            groups_lags[group][topic] ||= {}
 
             next unless existing_topics.key?(topic)
 
-            # We backfill because there is a case where our consumer group would consume for
+            # We backfill because there is a case where our group would consume for
             # example only one partition out of 20, rest needs to get -1
             existing_topics[topic].times do |partition_id|
-              groups_lags[cg][topic][partition_id] ||= -1
+              groups_lags[group][topic][partition_id] ||= -1
             end
           end
         end
 
         merged = Hash.new { |h, k| h[k] = {} }
 
-        groups_lags.each do |cg, topics|
+        groups_lags.each do |group, topics|
           topics.each do |topic, partitions|
-            merged[cg][topic] = {}
+            merged[group][topic] = {}
 
             partitions.each do |partition, lag|
-              merged[cg][topic][partition] = {
-                offset: groups_offs.fetch(cg).fetch(topic).fetch(partition),
+              merged[group][topic][partition] = {
+                offset: groups_offs.fetch(group).fetch(topic).fetch(partition),
                 lag: lag
               }
             end

data/lib/karafka/admin/isolation_levels.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module Karafka
+  class Admin
+    # Isolation level constants for Kafka operations.
+    #
+    # Controls whether a query returns the high-watermark (includes messages from uncommitted
+    # or in-flight transactions) or the Last Stable Offset (excludes them).
+    #
+    # For non-transactional topics both values give the same result.
+    module IsolationLevels
+      # Default. Returns the high-watermark offset, which includes messages written by
+      # producers that have not yet committed (or aborted) their transaction.
+      READ_UNCOMMITTED = Rdkafka::Bindings::RD_KAFKA_ISOLATION_LEVEL_READ_UNCOMMITTED
+
+      # Returns the Last Stable Offset (LSO): the highest offset that a READ_COMMITTED
+      # consumer would actually see. Use this when computing lag on topics produced with
+      # transactional producers, otherwise the high-watermark overstates lag.
+      READ_COMMITTED = Rdkafka::Bindings::RD_KAFKA_ISOLATION_LEVEL_READ_COMMITTED
+    end
+  end
+end