karafka 2.6.0.beta1 → 2.6.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 04dfc2086393c6d3d1b8c78eca503c425842453163b014dcba68a5c4231fe090
-  data.tar.gz: 5f62160e469b6eb2383234f630654b46fe704e9dedd896e863771c945ab8e30e
+  metadata.gz: bc797708a61eca1cc0309dce91d05ab1dd623f3c7f6abfd74ac7af69c98f5254
+  data.tar.gz: 9e0e3bace7cc9f5393dfe788035ddc6e89cd64c8afd7ed5a2400d841cce95637
 SHA512:
-  metadata.gz: 3f1bb0b95294bb925b71ff4cd91d0603b538b4f7f9a222a4789aa9aeef89f7f80b8b26ebf2623f1c44585e1de1d87d4be6dbb55748f61de9b1dd90cab68059d7
-  data.tar.gz: 1c79e3b072d6b53c0062f0ade986f357f677cae6d476297389732c3c130fd89e84687526729ba2590763d15d5f0279e08e3ca9fceec386401b2d9d5ec3ef2096
+  metadata.gz: 512bed7912b22916225f0428cf1ef949e534765eaca459606155e451566e847c3c0993a83978f230f72ac202e2d6e797415df4086ed00b37bfdbcdb94f5f4e06
+  data.tar.gz: f0152dec64b5ba30ebae473db2236848b6a3298473be0dec7956330e44e973593311a70e616eb68cc8705fdbcdf55a53eff1d54a230658a0307d7ba6b00cb413

data/CHANGELOG.md

@@ -1,38 +1,75 @@
 # Karafka Framework Changelog
 
 ## 2.6.0 (Unreleased)
-- [Maintenance] Replace manual per-version `Warning[:performance]` / `Warning[:deprecated]` flag setup in specs with a dynamic `Warning.categories`-based approach that enables all non-experimental warning categories automatically, keeping spec warning configuration in sync with karafka-rdkafka.
-- [Enhancement] Replace per-partition `query_watermark_offsets` consumer calls in `Admin::Topics#read_watermark_offsets` with two batch `list_offsets` admin calls (`:earliest` and `:latest`), reducing N sequential consumer roundtrips to 2 admin roundtrips regardless of how many topics or partitions are queried. Replace per-partition `query_watermark_offsets` fallback in `Admin::ConsumerGroups#seek` (when a time-based offset resolves beyond the last message) with a single batch `list_offsets` call covering all affected partitions. Replace per-partition `query_watermark_offsets` loop in `Pro::Iterator::TplBuilder#resolve_partitions_with_negative_offsets` with one batched `offsets_for_times(epoch)` call for consumer metadata warm-up plus two batch admin `list_offsets` calls for LWM/HWM resolution, reducing N sequential calls to 3 total (Pro).
-- **[Feature]** Add `Karafka::Admin.read_partition_offsets` (and `Admin::Topics#read_partition_offsets`) to expose the `rd_kafka_ListOffsets` admin API. Allows querying partition offsets by spec (`:earliest`, `:latest`, `:max_timestamp`, or a timestamp in ms) without a consumer group. Accepts an optional `isolation_level:` keyword — pass `Karafka::Admin::IsolationLevels::READ_COMMITTED` to get the Last Stable Offset (LSO) instead of the high-watermark, giving accurate lag figures on transactionally-produced topics.
-- **[Feature]** Add `Karafka::Admin::IsolationLevels` module with `READ_UNCOMMITTED` and `READ_COMMITTED` constants. These wrap the underlying `Rdkafka::Bindings` integer constants so callers do not need to reference rdkafka internals directly.
-- **[Feature]** Introduce standalone `Karafka::Declaratives` subsystem (`Declaratives::Topic`, `Declaratives::Repository`, `Declaratives::Builder`) decoupled from routing. Topic declarations can now be defined via `Karafka::App.declaratives.draw` independently of routing blocks. The existing `routing#config(partitions:, ...)` DSL continues to work as a backwards-compatible bridge that populates the shared declaratives repository.
-- [Enhancement] CLI topic commands (`create`, `delete`, `reset`, `repartition`, `align`, `migrate`, `plan`) now read from `Karafka::App.declaratives.topics` instead of walking the routing tree. Topics declared via the standalone `declaratives.draw` DSL are now managed by all CLI commands without requiring a routing entry. The routing bridge sets `bootstrap_servers` on declarations for cluster filtering.
-- [Enhancement] Move consumer-group-specific `Processing` components (`Coordinator`, `CoordinatorsBuffer`, `Executor`, `ExecutorsBuffer`, `Partitioner`, `ExpansionsSelector`, `InlineInsights`) and Pro equivalents (including `SubscriptionGroupsCoordinator`, `Collapser`, `AdaptiveIterator`, `PeriodicJob`, `ParallelSegments`, `VirtualPartitions`, `Filters`, `OffsetMetadata`, `Piping`) under the `ConsumerGroups` namespace (internal). The `config.processing.coordinator_class`, `executor_class`, `partitioner_class`, and `expansions_selector` defaults are updated accordingly; user-provided classes are unaffected. Generic processing components (`JobsQueue`, `Worker`, `WorkersPool`, `Schedulers::*`, `Jobs::Base`) stay in their original locations.
-- [Enhancement] Move processing strategies (`Karafka::Processing::Strategies::*` and `Karafka::Pro::Processing::Strategies::*`) under the `ConsumerGroups` namespace (internal). Prepares the processing layer for parallel `ShareGroups::Strategies` once KIP-932 lands.
-- [Enhancement] Move `Karafka::Processing::StrategySelector` and `Karafka::Pro::Processing::StrategySelector` under the `ConsumerGroups` namespace (internal). The `config.processing.strategy_selector` default is updated accordingly; user-provided selectors are unaffected.
-- [Enhancement] Move `Karafka::Connection::RebalanceManager` to `Karafka::Connection::ConsumerGroups::RebalanceManager` (internal). Prepares the connection layer for a parallel `ShareGroups` namespace once KIP-932 lands.
-- [Enhancement] Move the rebalance librdkafka callback to `Karafka::Instrumentation::Callbacks::ConsumerGroups::Rebalance` (internal). Published event names (`rebalance.*`) and payload shape are unchanged.
-- [Enhancement] Move `Karafka::Instrumentation::Callbacks::Error` and `Karafka::Instrumentation::Callbacks::Statistics` to `Karafka::Instrumentation::Callbacks::ConsumerGroups::Error` and `ConsumerGroups::Statistics` (internal). Completes the callbacks layer namespacing — all consumer-group-specific librdkafka callbacks now live under `Callbacks::ConsumerGroups`. Published event names and payload shape are unchanged.
-- [Enhancement] Namespace consumer-group-specific processing job classes under `Karafka::Processing::ConsumerGroups::Jobs` (OSS: `Consume`, `Eofed`, `Revoked`, `Shutdown`, `Idle`) and `Karafka::Pro::Processing::ConsumerGroups::Jobs` (Pro: `ConsumeNonBlocking`, `EofedNonBlocking`, `Periodic`, `PeriodicNonBlocking`, `RevokedNonBlocking`). Only `Jobs::Base` stays under `Karafka::Processing::Jobs` as the generic base class. Internal move only — no public API changes; prepares the processing layer for parallel `ShareGroups::Jobs` (KIP-932).
-- [Enhancement] Move `Karafka::Processing::JobsBuilder` to `Karafka::Processing::ConsumerGroups::JobsBuilder` and `Karafka::Pro::Processing::JobsBuilder` to `Karafka::Pro::Processing::ConsumerGroups::JobsBuilder` (internal). The jobs builder only creates consumer-group-specific jobs; prepares the processing layer for a parallel `ShareGroups::JobsBuilder` once KIP-932 lands.
-- [Enhancement] Nest consumer-group-specific processing config settings (`coordinator_class`, `errors_tracker_class`, `partitioner_class`, `strategy_selector`, `expansions_selector`, `executor_class`, `jobs_builder`) under `config.internal.processing.consumer_groups`. Shared settings (`jobs_queue_class`, `scheduler_class`, `worker_job_call_wrapper`) remain at the `config.internal.processing` level. Prepares the config structure for a parallel `config.internal.processing.share_groups` sub-namespace once KIP-932 lands.
-- [Enhancement] Move OSS consumer-group-specific routing features (`ActiveJob`, `DeadLetterQueue`, `Eofed`, `InlineInsights`, `ManualOffsetManagement`) from `Routing::Features` to `Routing::Features::ConsumerGroups` (internal). Agnostic features (`Declaratives`, `Deserializers`) remain at the `Routing::Features` level. Prepares the routing layer for a parallel `Routing::Features::ShareGroups` namespace once KIP-932 lands.
-- [Enhancement] Move Pro consumer-group-specific routing features (`ActiveJob`, `AdaptiveIterator`, `DeadLetterQueue`, `Delaying`, `DirectAssignments`, `InlineInsights`, `LongRunningJob`, `Multiplexing`, `NonBlockingJob`, `OffsetMetadata`, `ParallelSegments`, `Patterns`, `PeriodicJob`, `RecurringTasks`, `ScheduledMessages`, `Swarm`, `VirtualPartitions`) from `Pro::Routing::Features` to `Pro::Routing::Features::ConsumerGroups` (internal). Agnostic features (`Expiring`, `Filtering`, `Pausing`, `Throttling`) remain at the `Pro::Routing::Features` level.
-- [Enhancement] Introduce polymorphic `Routing::Topic#group` and `Routing::SubscriptionGroup#group` accessors (with `#consumer_group` kept as a backwards-compatible alias), expose `group_id` alongside `consumer_group_id` in `Routing::Topic#to_h`, emit parallel `group:` / `group_id:` keys alongside `consumer_group:` / `consumer_group_id:` in `rebalance.*` payloads, and emit parallel `group_id:` alongside `consumer_group_id:` in `error.occurred` and `statistics.emitted` payloads (these callbacks only carry the group id, not the group object). Prepares routing for additional group types (e.g. KIP-932 share groups); legacy keys remain and will be retired in Karafka 3.0 once share groups land.
+- **[Feature]** Add `Karafka::Admin.read_partition_offsets` (and `Admin::Topics#read_partition_offsets`) exposing the `rd_kafka_ListOffsets` admin API to query partition offsets by spec (`:earliest`, `:latest`, `:max_timestamp`, or a timestamp in ms) without a consumer group. Accepts an optional `isolation_level:` keyword (pass `READ_COMMITTED` for the Last Stable Offset instead of the high-watermark, giving accurate lag on transactional topics).
+- **[Feature]** Add `Karafka::Connection::Client#read_partition_offsets` (and `Connection::Proxy#read_partition_offsets`) exposing the batched `ListOffsets` query on a consumer's own connection, forwarding its `isolation.level`. Note that `:latest` resolves to the high watermark regardless of isolation level, so it includes uncommitted messages on topics with in-flight transactions.
+- **[Feature]** Allow `Karafka::Admin` (and subsystems like `Admin::ConsumerGroups`, `Admin::Topics`) to operate on an external client via `Karafka::Admin.new(external_client: client)`. Reuses the caller's client (no per-call bootstrap); its lifecycle stays with its owner.
+- **[Feature]** Add `Karafka::Admin::IsolationLevels` module with `READ_UNCOMMITTED` and `READ_COMMITTED` constants wrapping the underlying rdkafka integer constants.
+- **[Feature]** Introduce standalone `Karafka::Declaratives` subsystem (`Declaratives::Topic`, `Declaratives::Repository`, `Declaratives::Builder`) decoupled from routing. Topics can be declared via `Karafka::App.declaratives.draw`; the existing `routing#config(...)` DSL keeps working as a backwards-compatible bridge.
 - **[Feature]** Add `Processing::WorkersPool` with dynamic thread pool scaling via `#scale`, `nil` sentinel-based worker exit for downscaling, and `worker.scaling.up`/`worker.scaling.down` instrumentation events.
+- [Enhancement] Refresh watermark offsets and lags of long-paused partitions in `statistics.emitted` (Pro). Opt-in via `config.internal.statistics.consumer_groups.lag_compensation.interval` (ms, default `0` - disabled); partitions qualify after being paused for at least `lag_compensation.pause_age` (ms, min `5_000`, default `30_000`). Refreshed values are kept on resume until live statistics catch up, then dropped in bulk on the next rebalance.
+- [Enhancement] Replace per-partition `query_watermark_offsets` calls with batched `list_offsets` in `Admin::Topics#read_watermark_offsets`, `Admin::ConsumerGroups#seek`, and `Pro::Iterator::TplBuilder#resolve_partitions_with_negative_offsets`, cutting N sequential roundtrips to a small constant and avoiding a dedicated admin client in the iterator path (Pro).
+- [Enhancement] Add `stability_ttl` to `Kubernetes::LivenessListener` and `Pro::Swarm::LivenessListener` to detect consumers frozen in a single non-`"steady"` librdkafka `cgrp.join_state`. On breach, reports `stability_ttl_exceeded: true` (HTTP 500) / status code `4`; clears once the group returns to `"steady"`. Requires `statistics.interval.ms`; defaults to twice the max `max.poll.interval.ms` across active subscription groups.
+- [Enhancement] Contain non-`StandardError` errors (`ScriptError`, `SystemStackError`, etc.) raised in the consumption flow through the normal retry/pause/DLQ flow instead of skipping the batch. Process-critical errors (`SystemExit`, `SignalException`, `NoMemoryError`) are recorded, keep the partition paused, and trigger a graceful shutdown via the auto-subscribed `Instrumentation::CriticalErrorsListener` instead of being retried or dispatched to the DLQ. Configurable via `config.internal.processing.critical_errors`.
+- [Enhancement] Introduce polymorphic `Routing::Topic#group` and `Routing::SubscriptionGroup#group` accessors (`#consumer_group` kept as a backwards-compatible alias), expose `group_id` in `Routing::Topic#to_h`, and emit parallel `group`/`group_id` keys alongside the legacy `consumer_group`/`consumer_group_id` keys in `rebalance.*`, `error.occurred`, and `statistics.emitted` payloads. Prepares routing for KIP-932 share groups; legacy keys remain until Karafka 3.0.
+- [Enhancement] Add `Pro::Processing::ConsumerGroups::Filters::Actions` as the single source of truth for filter post-execution actions (`Actions.skip`/`Actions.pause`/`Actions.seek` value methods, the `ALL` list, and `#skip?`/`#pause?`/`#seek?` predicates mixed into `Filters::Base` and `Coordinators::FiltersApplier`). Returned values remain the same plain symbols, so existing filters keep working (Pro).
+- [Enhancement] Make the statistics decorator used by `Instrumentation::Callbacks::ConsumerGroups::Statistics` configurable via `config.internal.statistics.consumer_groups.decorator_class` instead of hardcoding `Karafka::Core::Monitoring::StatisticsDecorator`.
+- [Enhancement] Declare recurring tasks and scheduled messages internal topics via the standalone `Karafka::App.declaratives.draw` DSL instead of the routing `config(...)` bridge; the resulting declarations are unchanged (Pro).
+- [Enhancement] CLI topic commands (`create`, `delete`, `reset`, `repartition`, `align`, `migrate`, `plan`) now read from `Karafka::App.declaratives.topics` instead of walking the routing tree, so topics declared via `declaratives.draw` are managed without a routing entry.
 - [Enhancement] Add per-partition generation tracking to `AssignmentsTracker` to distinguish first-time assignments from reassignments.
 - [Enhancement] Make liveness listeners fiber-safe.
-- [Fix] Include orphaned node detection in Pro swarm liveness listener.
-- [Fix] Report liveness on `connection.listener.before_fetch_loop` so nodes send an initial healthy report before the first consumption.
-- [Fix] Fire liveness events during `wait_pinging` so nodes continue reporting health status during shutdown with active LRJ jobs.
+- [Enhancement] Bump swarm supervisor `SHUTDOWN_GRACE_PERIOD` from 1s to 15s to give forked nodes enough time to finish post-`shutdown_timeout` cleanup before forceful termination.
+- [Enhancement] Move consumer-group-specific `Processing` components (`Coordinator`, `CoordinatorsBuffer`, `Executor`, `ExecutorsBuffer`, `Partitioner`, `ExpansionsSelector`, `InlineInsights`) and their Pro equivalents under the `ConsumerGroups` namespace (internal). Related config defaults are updated; user-provided classes are unaffected.
+- [Enhancement] Move processing strategies (`Processing::Strategies::*` and `Pro::Processing::Strategies::*`) under the `ConsumerGroups` namespace (internal), preparing for parallel `ShareGroups::Strategies` (KIP-932).
+- [Enhancement] Move `Processing::StrategySelector` (OSS and Pro) under the `ConsumerGroups` namespace (internal); the `config.processing.strategy_selector` default is updated, user selectors unaffected.
+- [Enhancement] Move `Karafka::Connection::RebalanceManager` to `Connection::ConsumerGroups::RebalanceManager` (internal).
+- [Enhancement] Move the rebalance librdkafka callback to `Instrumentation::Callbacks::ConsumerGroups::Rebalance` (internal); event names and payloads unchanged.
+- [Enhancement] Move `Instrumentation::Callbacks::Error` and `Callbacks::Statistics` under `Callbacks::ConsumerGroups` (internal), completing the callbacks-layer namespacing; event names and payloads unchanged.
+- [Enhancement] Namespace consumer-group-specific processing job classes under `Processing::ConsumerGroups::Jobs` (OSS) and `Pro::Processing::ConsumerGroups::Jobs` (Pro); only `Jobs::Base` stays generic. Internal move, no public API changes.
+- [Enhancement] Move `Processing::JobsBuilder` (OSS and Pro) under the `ConsumerGroups` namespace (internal).
+- [Enhancement] Nest consumer-group-specific processing config settings (`coordinator_class`, `errors_tracker_class`, `partitioner_class`, `strategy_selector`, `expansions_selector`, `executor_class`, `jobs_builder`) under `config.internal.processing.consumer_groups`; shared settings stay at `config.internal.processing`.
+- [Enhancement] Move OSS consumer-group-specific routing features (`ActiveJob`, `DeadLetterQueue`, `Eofed`, `InlineInsights`, `ManualOffsetManagement`) under `Routing::Features::ConsumerGroups` (internal); agnostic features stay put.
+- [Enhancement] Move Pro consumer-group-specific routing features under `Pro::Routing::Features::ConsumerGroups` (internal); agnostic features (`Expiring`, `Filtering`, `Pausing`, `Throttling`) stay put.
+- [Change] Require `karafka-rdkafka` `>= 0.28.0`, which exposes `Rdkafka::Consumer#list_offsets` and rebuilds `#lag` on top of it, so lag reads issue a single batched end-offsets query instead of one watermark roundtrip per partition.
+- [Change] Require `karafka-rdkafka` `>= 0.27.1` for the `poll_batch`/`poll_batch_nb` fix that discarded topic/partition/offset context from `e.details`, causing `TopicNotFoundError` crashes on `partition_eof`.
+- [Change] Remove the public `custom_kafka` reader from `Karafka::Admin` (internal collaboration detail, unused by any public API; custom kafka settings are provided at construction via `Karafka::Admin.new(kafka: ...)`).
+- [Refactor] Remove the unused `@mutex` from `Pro::Connection::Manager` (dead since #1851 replaced lock-based `@changes` eviction with key preloading; the lock-free threading model is now documented) (Pro).
+- [Maintenance] Split declarative topic configuration from routing in the integration suite: add a `draw_topics` spec helper wrapping `Karafka::App.declaratives.draw`, source spec topic creation from the declaratives repository, and migrate general integration specs to declare partitions/config via `draw_topics`.
+- [Maintenance] Replace manual per-version `Warning[:performance]`/`Warning[:deprecated]` spec setup with a dynamic `Warning.categories`-based approach that enables all non-experimental warning categories automatically.
 - [Maintenance] Use namespaced topic naming format in all integration specs for consistent traceability.
-- [Fix] Fix `DataCollector::SPEC_HASH` producing non-deterministic hashes for pristine and poro specs by passing the original spec path via `KARAFKA_SPEC_PATH` env var.
-- [Fix] Fix swarm supervisor leaking reader pipe file descriptors on node restarts by closing the old reader before creating a new pipe in `Node#start`.
 - [Maintenance] Add `bin/tests_topics_hashes` script for looking up spec files by their topic name hash prefix.
-- [Change] Require `karafka-rdkafka` `>=` `0.27.1` to pick up the fix for `poll_batch` and `poll_batch_nb` raising `RdkafkaError` with only the integer error code, which discarded topic/partition/offset context from `e.details` and caused `partition_eof` handling to call `@buffer.eof(nil, nil)`, resulting in a `TopicNotFoundError` crash.
-- [Fix] Raise `InvalidLicenseTokenError` when a manually-defined `Karafka::License` module is missing required methods (`#token` or `#version`) instead of silently skipping pro component loading, which previously caused confusing `NameError` exceptions later in the boot process.
-- [Enhancement] Bump swarm supervisor `SHUTDOWN_GRACE_PERIOD` from 1s to 15s to give forked nodes enough time to finish post-`shutdown_timeout` cleanup (at_exit handlers, librdkafka finalization, connection pool close) before the supervisor forcefully terminates them, especially on CI where `sleep` granularity and `waitpid` cost stretch each supervision loop iteration.
-- [Maintenance] Fix flaky `pro/admin/recovery/coordinator_for_spec.rb` by warming up the `__consumer_offsets` internal topic with a produce + `seek_consumer_group` and retrying the first `Recovery.coordinator_for` call with exponential backoff on `MetadataError`.
+- [Maintenance] Fix flaky `pro/admin/recovery/coordinator_for_spec.rb` by warming up the `__consumer_offsets` topic and retrying the first `Recovery.coordinator_for` call with exponential backoff on `MetadataError`.
+- [Fix] Compare recurring task schedule versions with `Gem::Version` instead of as strings, so multi-segment upgrades like `1.9.0` → `1.10.0` are no longer misread as downgrades and rejected with `IncompatibleScheduleError` (Pro).
+- [Fix] Fix dynamic multiplexing `scale_delay` being applied globally instead of per subscription-group family (`Pro::Connection::Manager#stable?` memoized the first family's value for all); it is now read per family on every check (Pro).
+- [Fix] Make `Routing::Topics#<<` copy-on-write so a topic discovered at runtime by a pattern subscription can no longer be observed mid-append by a concurrent reader (e.g. `Router#find_by` from a `Pro::Iterator` or `Admin` call on another thread).
+- [Fix] Bound `Instrumentation::AssignmentsTracker`'s `@generations` map (`GENERATIONS_TOPICS_LIMIT`, LRU eviction) so it no longer grows for the whole process lifetime under regex pattern subscriptions; static topologies are unaffected.
+- [Fix] Derive the filtering pause timeout only from filters that actually want to pause, so a coexisting `:seek` filter reporting a `0` timeout no longer collapses a `:pause` to `0` and busy-spins (Pro).
+- [Fix] Clamp scheduled message CreateTime to the wall clock in the loaded heuristic, so a single future-dated message can no longer be taken as proof of catch-up and cause double dispatch of not-yet-loaded cancellations (Pro).
+- [Fix] Reset `Pro::Iterator#each` EOF/stop tracking in an `ensure`, so breaking out of the yielded block no longer leaves `done?` immediately true and silently drops a fresh backlog on reuse (Pro).
+- [Fix] Fix the `parallel_segments` CLI commands (`distribute`, `collapse`, `reset`) silently skipping groups when more than one was requested via `--groups` (the matched group was deleted from the array being iterated); all requested groups are now processed (Pro).
+- [Fix] Evict a partition's samples from `Pro::Instrumentation::PerformanceTracker` on revoke and scope samples by subscription group id, fixing unbounded growth and cross-group interference. **Breaking (internal API):** `processing_time_p95` now takes the subscription group id first (`processing_time_p95(group_id, topic, partition)`) (Pro).
+- [Fix] Fall back to the current offset metadata (instead of raising `KeyError`) in Virtual Partitions `VirtualOffsetManager#markable` under the `:exact` strategy when the lowest-offset virtual-partition group is left unmarked (Pro).
+- [Fix] Leave tombstone records (key with a `nil` payload) untouched on produce and consume instead of crashing in `cipher.encrypt`/`decrypt`, so they remain valid tombstones for log compaction (Pro).
+- [Fix] Recompute the `Pro::Processing::JobsQueue#clear` async-locking fast-path flag from the remaining groups' locks instead of forcing it off globally, so another subscription group's active `lock_async` locks are no longer ignored (Pro).
+- [Fix] Reset the per-partition retry attempt counter on revocation, so a message reclaimed after a rebalance is no longer treated as having exhausted its retries and dispatched to the DLQ early.
+- [Fix] Reset ActiveJob `CurrentAttributes` in an `ensure`, so a failed job's attributes no longer leak into the next job on the same worker.
+- [Fix] Return `false` from `#mark_as_consumed`, `#mark_as_consumed!`, and `#commit_offsets!` when the operation fails because the partition was lost (previously could return `true`).
+- [Fix] Reset `seek_offset` only after a `#seek` succeeds, so a raising seek (e.g. an unresolvable time-based offset) no longer pauses without seeking back and skips the rest of the batch.
+- [Fix] Skip DLQ dispatch when the partition was revoked mid-job in the ActiveJob + DLQ + long-running job strategies (matching the non-AJ LRJ strategies), so the new owner handles it instead of producing a duplicate (Pro).
+- [Fix] Advance the in-memory `seek_offset` only after a transactional DLQ dispatch succeeds, so an aborted dispatch transaction with `mark_after_dispatch: false` no longer skips the broken message (Pro).
+- [Fix] Report ownership truthfully from Virtual Partitions `#mark_as_consumed`/`#mark_as_consumed!` when there is no real offset to materialize yet (previously inverted) (Pro).
+- [Fix] Stop reverting recurring task commands consumed during the boot replay phase, which previously persisted the reverted state and lost the command permanently (Pro).
+- [Fix] Always precede Virtual Partitions DLQ `:dispatch`/`:skip` decisions with a collapsed retry, so they run on the linear deterministic flow instead of an arbitrary virtual-partition subset that could lose messages or dispatch the wrong one (Pro).
+- [Fix] Compare scheduled messages schema versions as `Gem::Version` instead of strings, fixing mis-evaluation once a version segment reaches two digits (Pro).
+- [Fix] Fix `Karafka::Admin.read_topic` with a `Time` start offset beyond the last message returning the newest messages instead of none.
+- [Fix] Prevent silent worker thread death draining the workers pool: worker continuation is now an explicit framework decision (only a closed queue or downscale sentinel stops a worker), so a user `Consumer#wrap` returning a falsy value no longer kills the thread.
+- [Fix] Track `Pro::Iterator` EOF and explicit stops per topic-partition instead of globally, so a partition receiving new data mid-iteration no longer exhausts the EOF budget and terminates `#each` while others still have backlog; `#each` also resets `#stop` on finish so the iterator can be reused (Pro).
+- [Fix] Include orphaned node detection in the Pro swarm liveness listener.
+- [Fix] Report liveness on `connection.listener.before_fetch_loop` so nodes send an initial healthy report before the first consumption.
+- [Fix] Fire liveness events during `wait_pinging` so nodes keep reporting health during shutdown with active LRJ jobs.
+- [Fix] Fix `DataCollector::SPEC_HASH` producing non-deterministic hashes for pristine and poro specs by passing the original spec path via the `KARAFKA_SPEC_PATH` env var.
+- [Fix] Fix the swarm supervisor leaking reader pipe file descriptors on node restarts by closing the old reader before creating a new pipe in `Node#start`.
+- [Fix] Raise `InvalidLicenseTokenError` when a manually-defined `Karafka::License` module is missing `#token` or `#version` instead of silently skipping Pro loading and raising a confusing `NameError` later in boot.
 
 ## 2.5.9 (2026-03-30)
 - [Enhancement] Validate that `statistics.interval.ms` is not zero when dynamic multiplexing is enabled (Pro).

data/certs/expired.txt

@@ -19,6 +19,7 @@
 39aeae710122bf14729a05f24f5bd702d2a1127e5c7fb9bd90de0e20890575dc
 3ada575caaf0b8bdeca0686a93074de9ed830751e6f46251849475f70e84e408
 4367e57ca36a4b447f23745d76611514df61b56611225be4c8f9a90560bbace9
+4611be91c816b50c54e7e4ec422e02363bc5450f9107a35d7e915d949ccf7bdb
 47380207ca3041b14c7d4c81a69f820852069ecab4d80a068d39133f61a6e641
 4d49b497ece81604358afc8e9adc7d1163995c4c08446bcd5c0bbfe3d7ebc1e7
 4d97c95c46e51105705964aecbfa5219120295e7d0a172f5c098d12c3a5cbf7e
@@ -62,6 +63,7 @@ bf6c5c58b02fa5e7c7dc845213ad943c5b6613f12ac6012be00b1484bb85aa04
 c011c3d168a159c135ef35bf55121edaf1a4cc94f663a1f461595776b0313160
 c311cb6cfb38a8b2e83b3080c1f87444a52f3f645be8337559a123aaacf1344a
 c87222a94f98c3a2191dda9254bd6ed031fd8ca190f75b40c67a0fcd4b56f795
+c9a5cc3f4c0fc7cb8784978d1f5185340ab585baf3b5d94300a29b212310871b
 ca53af70cd7e67abbb9dbf0a8da6788a9b2192aa3a5a1dd8720aaa68a2511722
 d02da0cbd2c1d83432a2cee7984dd311249692373554002d1d543dd5007f6a58
 d13295e597120b88bf4c4f36de63d0aa3e5a68e3578c09a2e026bd1f6aaaf04f

data/config/locales/errors.yml

@@ -37,6 +37,7 @@ en:
         internal.processing.jobs_queue_class_format: cannot be nil
         internal.processing.scheduler_class_format: cannot be nil
         internal.processing.worker_job_call_wrapper_format: 'needs to be false or respond to #wrap'
+        internal.processing.critical_errors_format: 'needs to be an array of Exception subclasses'
         internal.processing.consumer_groups.jobs_builder_format: cannot be nil
         internal.processing.consumer_groups.coordinator_class_format: cannot be nil
         internal.processing.consumer_groups.errors_tracker_class_format: 'needs to be nil or a class'
@@ -45,6 +46,10 @@ en:
         internal.processing.consumer_groups.expansions_selector_format: cannot be nil
         internal.processing.consumer_groups.executor_class_format: cannot be nil
 
+        internal.statistics.consumer_groups.decorator_class_format: cannot be nil
+        internal.statistics.consumer_groups.lag_compensation.interval_format: needs to be an Integer greater or equal to 0
+        internal.statistics.consumer_groups.lag_compensation.pause_age_format: needs to be an Integer greater or equal to 5000
+
         internal.active_job.dispatcher_format: cannot be nil
         internal.active_job.job_options_contract_format: cannot be nil
         internal.active_job.consumer_class_format: cannot be nil

data/karafka.gemspec

@@ -22,7 +22,7 @@ Gem::Specification.new do |spec|
   DESC
 
   spec.add_dependency "karafka-core", ">= 2.6.0", "< 2.7.0"
-  spec.add_dependency "karafka-rdkafka", ">= 0.27.2"
+  spec.add_dependency "karafka-rdkafka", ">= 0.28.0"
   spec.add_dependency "waterdrop", ">= 2.10.1", "< 3.0.0"
   spec.add_dependency "zeitwerk", "~> 2.3"
 

data/lib/active_job/queue_adapters/karafka_adapter.rb

@@ -20,7 +20,7 @@ module ActiveJob
     # - Other Rails versions: Inherit from AbstractAdapter (normal behavior)
     # - No Rails: Inherit from Object (standalone ActiveJob usage)
     #
-    # @see https://github.com/sidekiq/sidekiq/issues/6746 Similar issue in Sidekiq
+    # See https://github.com/sidekiq/sidekiq/issues/6746 Similar issue in Sidekiq
     base = if defined?(Rails::VERSION)
       ((Rails::VERSION::MAJOR == 7 && Rails::VERSION::MINOR < 2) ? Object : AbstractAdapter)
     else

data/lib/karafka/active_job/current_attributes/loading.rb

@@ -26,7 +26,7 @@ module Karafka
             end
 
             yield(job)
-
+          ensure
             resetable.each(&:reset)
           end
         end

data/lib/karafka/active_job/current_attributes.rb

@@ -5,11 +5,13 @@ require_relative "current_attributes/job_wrapper"
 require_relative "current_attributes/loading"
 require_relative "current_attributes/persistence"
 
-# This code is based on Sidekiqs approach to persisting current attributes
-# @see https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/middleware/current_attributes.rb
 module Karafka
   module ActiveJob
     # Module that allows to persist current attributes on Karafka jobs
+    #
+    # This code is based on Sidekiqs approach to persisting current attributes
+    #
+    # @see https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/middleware/current_attributes.rb
     module CurrentAttributes
       # Allows for persistence of given current attributes via AJ + Karafka
       #

data/lib/karafka/active_job/dispatcher.rb

@@ -8,8 +8,7 @@ module Karafka
         deserializer: %i[internal active_job deserializer]
       )
 
-      # Defaults for dispatching
-      # The can be updated by using `#karafka_options` on the job
+      # Defaults for dispatching. They can be updated by using `#karafka_options` on the job
       DEFAULTS = {
         dispatch_method: :produce_async,
         dispatch_many_method: :produce_many_async

data/lib/karafka/admin/consumer_groups.rb

@@ -110,7 +110,7 @@ module Karafka
           if partitions_with_offsets.is_a?(Hash)
             tpl_base[topic] = partitions_with_offsets
           else
-            topic_info = Topics.new(kafka: @custom_kafka).info(topic)
+            topic_info = topics_admin.info(topic)
             topic_info[:partition_count].times do |partition|
               tpl_base[topic][partition] = partitions_with_offsets
             end
@@ -358,7 +358,9 @@ module Karafka
           sleep(0.1)
 
           # Unsubscribe - this will trigger the second rebalance when the consumer closes
-          # The ensure block in with_consumer will handle the unsubscribe and close
+          # The ensure block in with_consumer will handle the unsubscribe and close for the
+          # dedicated consumer created here. This operation subscribes and must never run on
+          # an external client as nothing would restore its subscription
         end
       end
 
@@ -380,6 +382,10 @@ module Karafka
       #
       # @note This lag reporting is for committed lags and is "Kafka-centric", meaning that this
       #   represents lags from Kafka perspective and not the consumer. They may differ.
+      #
+      # @note When this instance operates on an external client, every queried group runs
+      #   through that single consumer identity, so query it only about the external client's
+      #   own group.
       def read_lags_with_offsets(groups_with_topics = {}, active_topics_only: true)
         # We first fetch all the topics with partitions count that exist in the cluster so we
         # do not query for topics that do not exist and so we can get partitions count for all

data/lib/karafka/admin/replication.rb

@@ -84,6 +84,8 @@ module Karafka
       # 2. As a plan result object when called with topic and plan parameters
       #
       # @param kafka [Hash] custom kafka configuration for admin operations (optional)
+      # @param external_client [Object, nil] external client for admin operations (optional)
+      #   @see Karafka::Admin#initialize
       # @param topic [String] topic name (for plan result)
       # @param current_replication_factor [Integer] current replication factor (for plan result)
       # @param target_replication_factor [Integer] target replication factor (for plan result)
@@ -91,13 +93,14 @@ module Karafka
       # @param cluster_info [Hash] broker information (for plan result)
       def initialize(
         kafka: nil,
+        external_client: nil,
         topic: nil,
         current_replication_factor: nil,
         target_replication_factor: nil,
         partitions_assignment: nil,
         cluster_info: nil
       )
-        super(kafka: kafka || {})
+        super(kafka: kafka || {}, external_client: external_client)
 
         # If topic is provided, this is a plan result object
         return unless topic
@@ -293,7 +296,7 @@ module Karafka
       # @param topic [String] name of the topic
       # @return [Hash] topic information with partitions metadata
       def fetch_topic_info(topic)
-        Topics.new(kafka: @custom_kafka).info(topic)
+        topics_admin.info(topic)
       end
 
       # Fetches cluster broker information from Kafka metadata
@@ -464,7 +467,7 @@ module Karafka
       end
 
       # Builds the kafka-reassign-partitions.sh --generate command that asks Kafka to propose
-      # its own reassignment plan. This is optional — you can compare Kafka's suggestion against
+      # its own reassignment plan. This is optional - you can compare Kafka's suggestion against
       # the plan Karafka already computed, or skip this step and go straight to --execute.
       # @return [String] command template with placeholders for broker addresses and IDs
       def build_generate_command

data/lib/karafka/admin/topics.rb

@@ -68,7 +68,8 @@ module Karafka
       # @param count [Integer] how many messages we want to get at most
       # @param start_offset [Integer, Time] offset from which we should start. If -1 is provided
       #   (default) we will start from the latest offset. If time is provided, the appropriate
-      #   offset will be resolved. If negative beyond -1 is provided, we move backwards more.
+      #   offset will be resolved. A time beyond the last message yields no messages. If
+      #   negative beyond -1 is provided, we move backwards more.
       # @param settings [Hash] kafka extra settings (optional)
       #
       # @return [Array<Karafka::Messages::Message>] array with messages
@@ -76,6 +77,7 @@ module Karafka
         messages = []
         tpl = Rdkafka::Consumer::TopicPartitionList.new
         low_offset, high_offset = nil
+        time_based = start_offset.is_a?(Time)
 
         with_consumer(settings) do |consumer|
           # Convert the time offset (if needed)
@@ -83,9 +85,20 @@ module Karafka
 
           low_offset, high_offset = consumer.query_watermark_offsets(name, partition)
 
-          # Select offset dynamically if -1 or less and move backwards with the negative
-          # offset, allowing to start from N messages back from high-watermark
-          start_offset = high_offset - count - start_offset.abs + 1 if start_offset.negative?
+          if start_offset.negative?
+            start_offset = if time_based
+              # A negative result of time resolution means there is no message at or after the
+              # requested time. It must not fall into the numeric relative form below (which
+              # would return the newest messages, all older than requested): starting at the
+              # high watermark yields the correct empty result
+              high_offset
+            else
+              # Select offset dynamically if -1 or less and move backwards with the negative
+              # offset, allowing to start from N messages back from high-watermark
+              high_offset - count - start_offset.abs + 1
+            end
+          end
+
           start_offset = low_offset if start_offset.negative?
 
           # Build the requested range - since first element is on the start offset we need to
@@ -264,8 +277,8 @@ module Karafka
       #   `READ_COMMITTED` consumer will never see those messages, so lag calculated from the
       #   high-watermark is overstated on transactionally-produced topics. Passing
       #   `isolation_level: Karafka::Admin::IsolationLevels::READ_COMMITTED` here
-      #   returns the Last Stable Offset (LSO) — the highest offset a `READ_COMMITTED` consumer
-      #   would actually reach — giving accurate lag figures.
+      #   returns the Last Stable Offset (LSO) - the highest offset a `READ_COMMITTED` consumer
+      #   would actually reach - giving accurate lag figures.
       #
       # - `:max_timestamp` spec: returns the offset of the message with the highest timestamp
       #   in the partition. Not available via watermarks.
@@ -297,12 +310,19 @@ module Karafka
       #
       # @raise [Rdkafka::RdkafkaError] on per-partition errors or connection issues
       #
+      # @note The specs must be passed as an explicit hash (in curly braces). Ruby parses a
+      #   brace-less trailing hash as keyword arguments, and since this method takes an
+      #   `isolation_level:` keyword, such a call raises an `ArgumentError` about a missing
+      #   positional argument.
+      #
       # @example Query earliest offset for partition 0 and latest for partition 1
       #   Karafka::Admin::Topics.read_partition_offsets(
-      #     'events' => [
-      #       { partition: 0, offset: :earliest },
-      #       { partition: 1, offset: :latest }
-      #     ]
+      #     {
+      #       'events' => [
+      #         { partition: 0, offset: :earliest },
+      #         { partition: 1, offset: :latest }
+      #       ]
+      #     }
       #   )
       #   # => [
       #   #   { topic: 'events', partition: 0, offset: 0, timestamp: -1, leader_epoch: nil },
@@ -311,13 +331,13 @@ module Karafka
       #
       # @example Get LSO (Last Stable Offset) for accurate lag on transactional topics
       #   Karafka::Admin::Topics.read_partition_offsets(
-      #     'events' => [{ partition: 0, offset: :latest }],
+      #     { 'events' => [{ partition: 0, offset: :latest }] },
       #     isolation_level: Karafka::Admin::IsolationLevels::READ_COMMITTED
       #   )
       #
       # @example Find offset at a specific point in time
       #   Karafka::Admin::Topics.read_partition_offsets(
-      #     'events' => [{ partition: 0, offset: 1_700_000_000_000 }]
+      #     { 'events' => [{ partition: 0, offset: 1_700_000_000_000 }] }
       #   )
       def read_partition_offsets(topic_partition_offsets, isolation_level: nil)
         with_admin do |admin|