karafka 2.6.0 → 2.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 00213ecae5aa07409c67a645f32e89beeab05399cb8e323a86bbcd8af538dd03
-  data.tar.gz: b790e3752b012e70095fa97510617f833107ca74282f721cb7afe378e4285e49
+  metadata.gz: a2d4a5f8788806e379defbbbd60e3ba605396b2cfed7fd78211ea5f14724c1b4
+  data.tar.gz: d9a183f86bb901603eec8dcaa1b98861fa48ded21ef9b8d95e47d3e100bd9f85
 SHA512:
-  metadata.gz: b3f03f3ba3d3bcdb616f3dc666e719224947a8de3a4a72ed1c4162da76f903bc99365a51bffe5e43a18894c2426756f75825e3b09e6174c47ad385c5aabb26e6
-  data.tar.gz: bf1ce2ca9cf4444349c2bf81c9b4c896f70a29b02a0415ef149d1f000705caf324b61513ddf348ee5dfb1bb763ce22de09227bde7ab2f76c79d21ed58992d05b
+  metadata.gz: 7f418e4dfac658f95fb12d1190e26d903e31d5ea28e38234c3bcb9eb90aee242e11af8f344544d51f3dfa0fe560aa9328a4517c01152c41773c625c99e3118c1
+  data.tar.gz: f15f728a174c61cb9d1ab56280f69d98dcee9d4f2e515b5c085bc6b69f2e87b6c8ad4c521f8cb0cf844252eb1d5dec813c4efda2954c7d3b7304c8973fb44cf1

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Karafka Framework Changelog
 
+## 2.6.1 (2026-08-24)
+- **[Feature]** [Pro] Add an opt-in envelope encryption mode for Messages At Rest encryption (`config.encryption.mode = :envelope`, requires the openssl gem `>= 3.0`). It removes the RSA payload size ceiling of the default `:direct` mode (~245 bytes for 2048-bit keys) by encrypting each message with a one-time RSA-wrapped AES-256-GCM key. Both formats are recognized automatically on decryption, so data already at rest stays readable with no migration. Deployment: upgrade all consuming processes before enabling - older versions cannot decrypt envelope payloads, while upgraded consumers read both formats and keep producing `:direct` until switched, so the rollout can be staged safely. The default will switch to `:envelope` in a future release with prior notice; the `:direct` format stays decryptable indefinitely as data at rest never expires. Cipher key material is now pre-parsed (warmed) during setup - this introduces no new boot failure modes, as the configuration contract has always parsed and validated all configured keys at boot.
+- [Fix] [Pro] Add a missing comma in the filtering strategy's unsupported-action guard so it raises `Karafka::Errors::UnsupportedCaseError` instead of a `NoMethodError`.
+- [Fix] [Pro] Evict scheduled messages from the daily buffer per confirmed flush chunk instead of only after the whole flush. A broker error partway through a multi-chunk flush left the already-produced chunks in the buffer, re-dispatching them on the next tick (a duplicate window for non-transactional producers). Chunk size is now rounded up to even before shifting off the buffer, so a message's target and its own tombstone (always buffered as an adjacent pair) can no longer land in different chunks - with an odd or `1` `flush_batch_size`, the target's chunk could confirm and evict the key before the tombstone chunk ran, leaving the schedule non-tombstoned in Kafka (and re-dispatched after a restart/reload) if that later chunk then failed.
+- [Fix] Prevent the `Karafka::Server#stop` specs from leaking a sub-second `shutdown_timeout` into the global config. The `#stop` examples mutate `Karafka::App.config.shutdown_timeout` (down to `500` ms, below the default `max_wait_time` of `1_000` ms) but never restored it, so with a random run order the invalid value bled into unrelated specs and made them fail contract validation with `shutdown_timeout must be more than max_wait_time`. The original value is now captured and restored (and the memoized class-level value cleared) in the example teardown.
+- [Fix] [Pro] Reject a `schedule_target_epoch` set implausibly far in the future (e.g. a milliseconds value passed as seconds) at publish time instead of silently dropping it on the consumer.
+- [Fix] [Pro] Raise `Karafka::Errors::UnsupportedCaseError` instead of a `NameError` (wrong constant namespace) when a custom DLQ strategy returns an unsupported flow, in both the default and virtual-partitions DLQ strategies.
+- [Enhancement] Support named `Karafka::Constraints` registered in two phases: `:load` (require time) and `:config` (during setup, after contract validation), so feature-specific environment requirements are verified in one place.
+- [Fix] Message `key` and `headers` deserialization results were not cached when the deserializer returned `nil` or `false` (e.g. keyless messages), re-running the deserializer on every access. Falsy results are now cached like truthy ones, matching the payload behavior and the documented contract.
+- [Fix] [Pro] Reset the `Pro::Iterator` stored-offsets latch between runs. `#mark_as_consumed` set an internal `@stored_offsets` flag that was never cleared, so once a reused iterator marked anything, the teardown guard fired a spurious blocking sync `commit_offsets` on the end of every subsequent `#each` - each running on a brand-new consumer with no stored offsets. The flag is now reset alongside the other per-run state in the `#each` `ensure` block, so it reflects only the current run's marking activity (benign no-op commit removed; no correctness impact).
+- [Fix] [Pro] Deduplicate runtime pattern discovery under multiplexing. Multiplexed subscription groups share a single consumer group but each runs its own listener thread and independently discovers the same topic; `ConsumerGroup#topic=` appended unconditionally, so the shared consumer group accumulated up to `multiplex_factor` duplicate `Topic` objects per discovered topic (polluting the routing tree and slowing linear topic scans, though the per-subscription-group consume path stayed correct). The detector now registers each discovered topic in the shared group only once (under its existing mutex) while still giving every subscription group its own `Topic` instance, mirroring how the multiplexing subscription groups builder dups topics per group at boot.
+- [Fix] Fix sub-second `shutdown_timeout` values collapsing to a zero-iteration supervision loop. The graceful-shutdown wait computed its iteration count with integer division (`shutdown_timeout / 1_000`), so any configured `shutdown_timeout` below 1000ms (valid config, as long as it exceeds `max_wait_time`) floored to zero iterations, skipping the grace period entirely and forcing an immediate forceful shutdown (killing in-flight jobs before their offset commits). The count is now computed with float math and `ceil`, guaranteeing at least one supervision check for any positive timeout. The swarm supervisor shared the same formula (masked by the added grace period) and was fixed the same way.
+- [Fix] Remove the backwards-compatible forwarding shim that kept the pre-nesting `config.internal.processing.*` paths (`coordinator_class`, `errors_tracker_class`, `partitioner_class`, `strategy_selector`, `expansions_selector`, `executor_class`, `jobs_builder`) resolving after the 2.6.0 move to `config.internal.processing.consumer_groups.*`. The shim was shipped by accident; the nesting was announced as having no backward compatibility. Use the nested `config.internal.processing.consumer_groups.*` paths instead. Requires `karafka-testing >= 2.6.2`, which reads these settings from their nested location.
+- [Fix] Stop `PausesManager#@pauses` from growing unbounded across rebalances. `CoordinatorsBuffer#revoke` reset the pause tracker's retry attempt count on revocation but never removed it, and nothing else pruned the hash outside of critical-error recovery, so a `Pause` (and its `Mutex`) was retained forever for every distinct topic-partition ever assigned - harmless for static routing (`Topic` objects are stable and get reused on reassignment) but unbounded under regex pattern subscriptions with ephemeral, per-discovery topic names. `#revoke` now removes a tracker outright when it is not currently paused (added `PausesManager#delete`), since it carries no state worth preserving; a tracker that is paused is still reset-and-kept, unchanged, because its remaining backoff may need to be honoured again if the same partition is reclaimed (routine under eager rebalancing, where every previously owned partition is revoked and reassigned even when nothing has actually changed for it).
+- [Fix] Allow-list the benign auto-create `TOPIC_ALREADY_EXISTS` broker warning in `unexpected_patterns_loop_spec.rb`, the same pre-existing broker-side race already allow-listed for other pattern-matched-topic specs, unrelated to this release's other fixes.
+- [Fix] [Pro] Fix unbounded memory growth from `Pro::Processing::JobsQueue`'s per-group semaphore accumulating unconsumed `#tick` signals under non-blocking/LRJ and async-locking workloads.
+- [Fix] [Pro] Stop `Pro::Processing::JobsQueue#unlock` from decrementing `@statistics[:waiting]` before confirming the job was actually tracked. It unconditionally decremented first and only afterwards checked whether the job was still present in `@in_waiting`, so a job unlocked after its subscription group had already been reset via `#clear` (which decrements `waiting` itself as part of recovery) both over-decremented the counter a second time and raised `JobsQueueSynchronizationError` - the counter corruption was silent while the (still legitimate) synchronization error masked it. `#unlock` now only decrements when the job is found, mirroring `#unlock_async`'s existing check-then-act pattern; both raise consistently when the group has already been cleared.
+- [Fix] `Swarm::Node#signal` (used by `#stop`/`#terminate`/`#quiet`) could send a signal to a reaped node's stale `@pid`, which the OS may have already reassigned to an unrelated process. It now returns `false` without signaling if the node is already known to be dead.
+
 ## 2.6.0 (2026-08-05)
 - **[Breaking]** Remove the flat global pause configuration accessors (`config.pause_timeout`, `config.pause_max_timeout`, `config.pause_with_exponential_backoff`) deprecated in 2.5.2. Use the nested `config.pause.*` namespace (`config.pause.timeout`, `config.pause.max_timeout`, `config.pause.with_exponential_backoff`) instead.
 - **[Breaking]** Nest the per-topic pause configuration under `topic.pause`. The flat `topic.pause_timeout`, `topic.pause_max_timeout`, and `topic.pause_with_exponential_backoff` readers are removed in favor of `topic.pause.timeout`, `topic.pause.max_timeout`, and `topic.pause.with_exponential_backoff`; `topic.to_h` now emits a nested `pause:` hash instead of the flat keys. The Pro Granular Backoffs override DSL is unchanged (`pause(timeout:, max_timeout:, with_exponential_backoff:)`), and `topic.pausing`/`topic.pausing?` are now `topic.pause`/`topic.pause?`. In OSS `topic.pause` reflects the global `config.pause.*` settings; per-topic overriding remains a Pro feature.

data/README.md

@@ -14,7 +14,7 @@ Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework
 - Has [ActiveJob backend](https://karafka.io/docs/Active-Job) support (including [ordered jobs](https://karafka.io/docs/Pro-Enhanced-Active-Job#ordered-jobs))
 - Has a seamless [Dead Letter Queue](https://karafka.io/docs/Dead-Letter-Queue/) functionality built-in
 - Supports in-development [code reloading](https://karafka.io/docs/Auto-reload-of-code-changes-in-development)
-- Is powered by [librdkafka](https://github.com/edenhill/librdkafka) (the Apache Kafka C/C++ client library)
+- Is powered by [librdkafka](https://github.com/confluentinc/librdkafka) (the Apache Kafka C/C++ client library)
 - Has an out-of the box [AppSignal](https://karafka.io/docs/Monitoring-and-Logging/#appsignal-metrics-and-error-tracking) and [StatsD/DataDog](https://karafka.io/docs/Monitoring-and-Logging/#datadog-and-statsd-integration) monitoring with dashboard templates.
 
 ```ruby

data/config/locales/pro_errors.yml

@@ -115,6 +115,7 @@ en:
     setup:
       config:
         encryption.active_format: 'needs to be either true or false'
+        encryption.mode_format: 'needs to be either :envelope or :direct'
         encryption.public_key_invalid: 'is not a valid public RSA key'
         encryption.public_key_needs_to_be_public: 'is a private RSA key not a public one'
         encryption.private_keys_format: 'needs to be a hash of version and private key value'
@@ -156,5 +157,6 @@ en:
       key_missing: must be present and should be unique within the partition
       key_format: needs to be a non-empty string unique within the partition
       headers_schedule_target_epoch_in_the_past: 'scheduling cannot happen in the past'
+      headers_schedule_target_epoch_too_far_in_future: 'scheduling is too far in the future (schedule_target_epoch must be Unix time in seconds)'
       headers_format: are not correct
       not_a_scheduled_messages_topic: 'the envelope topic is not a scheduled messages topic'

data/lib/karafka/connection/pauses_manager.rb

@@ -25,12 +25,19 @@ module Karafka
         )
       end
 
-      # Resets the attempt count of a given topic partition pause tracker.
+      # Resets the attempt count of a given topic partition pause tracker, or removes it entirely
+      # if it is not currently paused.
       #
       # Used on revocation so that a later reclaim of the same partition starts counting retry
       # attempts from zero instead of carrying the stale count across the rebalance. We reset
-      # rather than remove the tracker because the pause itself may still need to be resumed after
-      # the reclaim (the partition can be re-paused via the retained paused offsets on rebalance).
+      # rather than remove a tracker that is currently paused because the pause itself may still
+      # need to be resumed after the reclaim (the partition can be re-paused via the retained
+      # paused offsets on rebalance) - this matters under eager rebalancing, where every
+      # previously owned partition is revoked and then reassigned even when nothing has actually
+      # changed for it. A tracker that is not paused has no state worth preserving, so we remove
+      # it instead via `#delete` - this is what actually bounds `@pauses`, since otherwise entries
+      # accumulate forever for topics whose routing `Topic` object is never reused across
+      # reassignment (e.g. regex pattern subscriptions with ephemeral, per-discovery topic names).
       #
       # A coordinator and its pause tracker are created together in
       # `CoordinatorsBuffer#find_or_create`, and `CoordinatorsBuffer#revoke` only calls us once it
@@ -39,7 +46,23 @@ module Karafka
       # @param topic [::Karafka::Routing::Topic] topic
       # @param partition [Integer] partition number
       def revoke(topic, partition)
-        @pauses[topic][partition].reset
+        pause = @pauses[topic][partition]
+
+        return delete(topic, partition) unless pause.paused?
+
+        pause.reset
+      end
+
+      # Removes the pause tracker of a given topic partition, dropping the topic entry entirely
+      # once it no longer tracks any partitions.
+      #
+      # @param topic [::Karafka::Routing::Topic] topic
+      # @param partition [Integer] partition number
+      def delete(topic, partition)
+        partitions = @pauses[topic]
+        partitions.delete(partition)
+
+        @pauses.delete(topic) if partitions.empty?
       end
 
       # Resumes processing of partitions for which pause time has ended.

data/lib/karafka/constraints.rb

@@ -1,32 +1,55 @@
 # frozen_string_literal: true
 
 module Karafka
-  # Module used to check some constraints that cannot be easily defined by Bundler
-  # At the moment we use it to ensure, that if Karafka is used, it operates with the expected
-  # web ui version and that older versions of Web UI that would not be compatible with the API
-  # changes in karafka are not used.
+  # Module used to check optional requirements (constraints) that cannot be easily defined by
+  # Bundler. Constraints are registered under one of two phases and verified centrally, so all
+  # environment requirements live and surface in one place:
   #
-  # We can make Web UI require certain karafka version range, but at the moment we do not have a
-  # strict 1:1 release pattern matching those two.
+  # - `:load` - verified when karafka itself is required, for requirements independent of the
+  #   configuration (like ecosystem gems version compatibility)
+  # - `:config` - verified during `Karafka::App.setup` right after the configuration is
+  #   validated, for requirements that depend on what features are actually enabled (features
+  #   register those themselves, keeping their specifics out of this generic module)
   module Constraints
+    # Phases in which constraints can be verified
+    PHASES = %i[load config].freeze
+
+    private_constant :PHASES
+
     class << self
-      # Verifies that optional requirements are met.
-      def verify!
-        # Skip verification if web is not used at all
-        return unless require_version("karafka/web")
-
-        # All good if version higher than 1.0.0.rc1 because we expect 1.0.0.rc2 or higher
-        return if version(Karafka::Web::VERSION) >= version("1.0.0.rc2")
-
-        # If older web-ui used, we cannot allow it
-        raise(
-          Errors::DependencyConstraintsError,
-          "karafka-web < 1.0.0.rc2 is not compatible with this karafka version"
-        )
+      # Registers a constraint for verification. Registrations are keyed by name, so
+      # re-registration (e.g. when setup runs multiple times in tests) overwrites a previous
+      # one instead of accumulating duplicates.
+      #
+      # @param name [Symbol] unique constraint name
+      # @param phase [Symbol] `:load` or `:config`
+      # @param block [Proc] verification receiving the config node (`nil` in the `:load`
+      #   phase) and expected to raise `Karafka::Errors::DependencyConstraintsError` when the
+      #   requirement is not met
+      def register(name, phase:, &block)
+        raise(Errors::UnsupportedCaseError, phase) unless PHASES.include?(phase)
+
+        constraints[phase][name] = block
+      end
+
+      # Verifies all the constraints registered for a given phase
+      #
+      # @param phase [Symbol] `:load` or `:config`
+      # @param config [Karafka::Core::Configurable::Node, nil] config node for the `:config`
+      #   phase, `nil` in the `:load` phase
+      def verify!(phase = :load, config = nil)
+        raise(Errors::UnsupportedCaseError, phase) unless PHASES.include?(phase)
+
+        constraints[phase].each_value { |constraint| constraint.call(config) }
       end
 
       private
 
+      # @return [Hash{Symbol => Hash{Symbol => Proc}}] registered constraints per phase
+      def constraints
+        @constraints ||= PHASES.to_h { |phase| [phase, {}] }
+      end
+
       # Requires given version file from a gem location
       # @param version_location [String]
       # @return [Boolean] true if it was required or false if not reachable
@@ -45,5 +68,22 @@ module Karafka
         Gem::Version.new(string)
       end
     end
+
+    # If Karafka is used with the Web UI, it needs to be in a version compatible with the API
+    # changes in this karafka version. We can make Web UI require a certain karafka version
+    # range, but at the moment we do not have a strict 1:1 release pattern matching those two.
+    register(:karafka_web_version, phase: :load) do |_config|
+      # Skip verification if web is not used at all
+      next unless require_version("karafka/web")
+
+      # All good if version higher than 1.0.0.rc1 because we expect 1.0.0.rc2 or higher
+      next if version(Karafka::Web::VERSION) >= version("1.0.0.rc2")
+
+      # If older web-ui used, we cannot allow it
+      raise(
+        Errors::DependencyConstraintsError,
+        "karafka-web < 1.0.0.rc2 is not compatible with this karafka version"
+      )
+    end
   end
 end

data/lib/karafka/messages/metadata.rb

@@ -16,18 +16,27 @@ module Karafka
       keyword_init: true
     ) do
       # @return [Object] deserialized key. By default in the raw string format.
+      # @note The result is cached after the first deserialization, including `nil` and other
+      #   falsy results (e.g. keyless messages), hence the explicit flag instead of a
+      #   truthiness check. As with `Message#payload`, the flag is set only after a successful
+      #   deserialization, so an error is not cached and the next access retries.
       def key
-        return @key if @key
+        return @key if @key_deserialized
 
         @key = deserializers.key.call(self)
+        @key_deserialized = true
+        @key
       end
 
       # @return [Object] deserialized headers. By default its a hash with keys and payload being
       #   strings
+      # @note Caching works as in {#key}, including falsy results
       def headers
-        return @headers if @headers
+        return @headers if @headers_deserialized
 
         @headers = deserializers.headers.call(self)
+        @headers_deserialized = true
+        @headers
       end
     end
   end

data/lib/karafka/pro/encryption/cipher.rb

@@ -32,47 +32,77 @@ module Karafka
   module Pro
     module Encryption
       # Cipher for encrypting and decrypting data
+      #
+      # A facade over the cipher implementations in {Ciphers}:
+      #
+      # - {Ciphers::Direct} (default) - legacy scheme with the payload RSA-encrypted directly,
+      #   limited to payloads smaller than the RSA key capacity
+      # - {Ciphers::Envelope} - hybrid scheme with a one-time AES-256-GCM key per payload,
+      #   supporting payloads of any size
+      #
+      # Encryption follows the `encryption.mode` setting. Decryption is mode-independent: the
+      # format is recognized per message (a valid direct RSA ciphertext is always exactly the
+      # key modulus size, while an envelope is always at least 29 bytes longer), so consumers
+      # decrypt both formats transparently regardless of the configured mode, making staged
+      # producer-side rollout of the envelope mode safe.
+      #
+      # Format detection is deliberately payload-based rather than header-based, even though
+      # the encryption middleware already writes message headers that could carry a format
+      # marker. A header marker would remove the truncation blind spot documented on
+      # {Ciphers::Direct#owns?}, but the payload would stop being self-describing: it must
+      # remain decryptable also when it leaves Kafka through channels that do not preserve
+      # headers (mirroring and replication tools, dumps, storage sinks) and when handled by
+      # custom parsers or ciphers that only receive the payload through the stable
+      # `#decrypt(version, content)` contract. We accept the blind spot as the cheaper cost.
       class Cipher
         include Helpers::ConfigImporter.new(
           encryption: %i[encryption]
         )
 
-        # Initializes the cipher with empty private keys cache
-        def initialize
-          @private_pems = {}
-        end
-
-        # Encrypts given string content with the public key
+        # Encrypts given string content according to the configured `encryption.mode`
         # @param content [String]
         # @return [String]
         def encrypt(content)
-          public_pem.public_encrypt(content)
+          (encryption.mode == :envelope) ? envelope.encrypt(content) : direct.encrypt(content)
         end
 
-        # Decrypts provided content using `version` key
+        # Decrypts provided content using `version` key with the cipher implementation that
+        # recognizes the content format, independently of the configured mode
         # @param version [String] encryption version
         # @param content [String] encrypted content
         # @return [String] decrypted content
         def decrypt(version, content)
-          private_pem(version).private_decrypt(content)
+          if direct.owns?(version, content)
+            direct.decrypt(version, content)
+          else
+            envelope.decrypt(version, content)
+          end
         end
 
-        private
-
-        # @return [::OpenSSL::PKey::RSA] rsa public key
-        def public_pem
-          @public_pem ||= OpenSSL::PKey::RSA.new(encryption.public_key)
+        # Eagerly builds the underlying ciphers and parses the key material of the given
+        # config. Invoked during the single-threaded setup phase so that runtime encryption
+        # and decryption only read already-built, effectively frozen state and the lazy
+        # initialization below never races across worker threads.
+        #
+        # @param root_config [Karafka::Core::Configurable::Node] config whose key material to
+        #   warm. During setup this is the same app config the ciphers read at runtime.
+        def warmup(root_config)
+          direct.warmup(root_config.encryption)
+          envelope.warmup(root_config.encryption)
         end
 
-        # @param version [String] version for which we want to get the rsa key
-        # @return [::OpenSSL::PKey::RSA] rsa private key
-        def private_pem(version)
-          return @private_pems[version] if @private_pems.key?(version)
+        private
 
-          key_string = encryption.private_keys[version]
-          key_string || raise(Errors::PrivateKeyNotFoundError, version)
+        # Lazily built so this facade can be instantiated as a config default while the cipher
+        # implementation files may not be loaded yet
+        # @return [Ciphers::Direct]
+        def direct
+          @direct ||= Ciphers::Direct.new
+        end
 
-          @private_pems[version] = OpenSSL::PKey::RSA.new(key_string)
+        # @return [Ciphers::Envelope]
+        def envelope
+          @envelope ||= Ciphers::Envelope.new
         end
       end
     end

data/lib/karafka/pro/encryption/ciphers/base.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+# Karafka Pro - Source Available Commercial Software
+# Copyright (c) 2017-present Maciej Mensfeld. All rights reserved.
+#
+# This software is NOT open source. It is source-available commercial software
+# requiring a paid license for use. It is NOT covered by LGPL.
+#
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
+# PROHIBITED:
+# - Use without a valid commercial license
+# - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
+# - Use as training data for AI/ML models or inclusion in datasets
+# - Scraping, crawling, or automated collection for any purpose
+#
+# PERMITTED:
+# - Reading, referencing, and linking for personal or commercial use
+# - Runtime retrieval by AI assistants, coding agents, and RAG systems
+#   for the purpose of providing contextual help to Karafka users
+#
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
+# License: https://karafka.io/docs/Pro-License-Comm/
+# Contact: contact@karafka.io
+
+module Karafka
+  module Pro
+    module Encryption
+      # Namespace for the cipher implementations behind the encryption modes
+      module Ciphers
+        # Base for the cipher implementations, providing shared access to the configured RSA
+        # key material with per-version private key resolution
+        class Base
+          include Helpers::ConfigImporter.new(
+            encryption: %i[encryption]
+          )
+
+          # Initializes the cipher with an empty private keys cache
+          #
+          # @note Each cipher instance holds its own tiny cache of parsed pem objects. With two
+          #   cipher implementations composed by {Encryption::Cipher} this means the material
+          #   is parsed at most twice per version, which we accept over introducing a shared
+          #   keyring concept
+          #
+          # @note The caches are populated via {#warmup} during the single-threaded setup
+          #   phase, so under normal operations runtime access is read-only. Should a key
+          #   version appear only at runtime, the lazy population is idempotent and benign
+          #   under MRI (worst case the same pem is parsed twice)
+          def initialize
+            @private_pems = {}
+          end
+
+          # Eagerly parses the given encryption config key material into the instance caches
+          #
+          # @param encryption_config [Karafka::Core::Configurable::Node] encryption config
+          #   node. During setup it is the same node the lazy readers resolve at runtime,
+          #   passed explicitly so this method does not silently couple to the global state
+          def warmup(encryption_config)
+            @public_pem ||= OpenSSL::PKey::RSA.new(encryption_config.public_key)
+
+            encryption_config.private_keys.each do |version, key|
+              @private_pems[version] ||= OpenSSL::PKey::RSA.new(key)
+            end
+          end
+
+          private
+
+          # @return [::OpenSSL::PKey::RSA] rsa public key
+          def public_pem
+            @public_pem ||= OpenSSL::PKey::RSA.new(encryption.public_key)
+          end
+
+          # @param version [String] version for which we want to get the rsa key
+          # @return [::OpenSSL::PKey::RSA] rsa private key
+          def private_pem(version)
+            return @private_pems[version] if @private_pems.key?(version)
+
+            key_string = encryption.private_keys[version]
+            key_string || raise(Errors::PrivateKeyNotFoundError, version)
+
+            @private_pems[version] = OpenSSL::PKey::RSA.new(key_string)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/pro/encryption/ciphers/direct.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+# Karafka Pro - Source Available Commercial Software
+# Copyright (c) 2017-present Maciej Mensfeld. All rights reserved.
+#
+# This software is NOT open source. It is source-available commercial software
+# requiring a paid license for use. It is NOT covered by LGPL.
+#
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
+# PROHIBITED:
+# - Use without a valid commercial license
+# - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
+# - Use as training data for AI/ML models or inclusion in datasets
+# - Scraping, crawling, or automated collection for any purpose
+#
+# PERMITTED:
+# - Reading, referencing, and linking for personal or commercial use
+# - Runtime retrieval by AI assistants, coding agents, and RAG systems
+#   for the purpose of providing contextual help to Karafka users
+#
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
+# License: https://karafka.io/docs/Pro-License-Comm/
+# Contact: contact@karafka.io
+
+module Karafka
+  module Pro
+    module Encryption
+      module Ciphers
+        # Legacy cipher where the payload is RSA-encrypted directly with PKCS1 v1.5 padding.
+        #
+        # RSA can only encrypt data smaller than the key size minus padding (e.g. ~245 bytes
+        # for a 2048-bit key, ~501 bytes for a 4096-bit key), so it is unsuitable for larger
+        # payloads and remains available only for backwards compatibility with data already
+        # encrypted at rest and with fleets not yet fully upgraded.
+        class Direct < Base
+          # Encrypts given content with the public key
+          # @param content [String]
+          # @return [String] RSA ciphertext, always exactly the key modulus size
+          def encrypt(content)
+            public_pem.public_encrypt(content)
+          end
+
+          # Decrypts provided content using the `version` private key
+          # @param version [String] encryption version
+          # @param content [String] encrypted content
+          # @return [String] decrypted content
+          def decrypt(version, content)
+            private_pem(version).private_decrypt(content)
+          end
+
+          # @param version [String] encryption version
+          # @param content [String] encrypted content
+          # @return [Boolean] true if the content matches this cipher's format. A valid direct
+          #   RSA ciphertext is always exactly the key modulus size
+          #
+          # @note One inherent blind spot: an envelope truncated to exactly the modulus size is
+          #   indistinguishable from a direct ciphertext. It surfaces as an RSA padding error
+          #   or - when the PKCS1 v1.5 padding coincidentally validates - as garbage output,
+          #   never as the envelope diagnostics. The legacy direct format carries no marker
+          #   that could disambiguate this.
+          def owns?(version, content)
+            content.bytesize == private_pem(version).n.num_bytes
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/pro/encryption/ciphers/envelope.rb

@@ -0,0 +1,164 @@
+# frozen_string_literal: true
+
+# Karafka Pro - Source Available Commercial Software
+# Copyright (c) 2017-present Maciej Mensfeld. All rights reserved.
+#
+# This software is NOT open source. It is source-available commercial software
+# requiring a paid license for use. It is NOT covered by LGPL.
+#
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
+# PROHIBITED:
+# - Use without a valid commercial license
+# - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
+# - Use as training data for AI/ML models or inclusion in datasets
+# - Scraping, crawling, or automated collection for any purpose
+#
+# PERMITTED:
+# - Reading, referencing, and linking for personal or commercial use
+# - Runtime retrieval by AI assistants, coding agents, and RAG systems
+#   for the purpose of providing contextual help to Karafka users
+#
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
+# License: https://karafka.io/docs/Pro-License-Comm/
+# Contact: contact@karafka.io
+
+module Karafka
+  module Pro
+    module Encryption
+      module Ciphers
+        # Hybrid cipher where each payload is encrypted with a one-time AES-256-GCM key and
+        # only that key is RSA-wrapped (OAEP padding). Handles payloads of any size.
+        #
+        # The GCM auth tag covers the whole envelope (header included), so corruption and
+        # truncation are detected reliably. Note this is corruption detection, not
+        # authenticity: the RSA public key is distributed to all producers, so any of its
+        # holders can construct a valid envelope.
+        class Envelope < Base
+          # Envelope binary format:
+          #
+          #   [1B format version][RSA-wrapped AES key (modulus size)][12B iv][16B tag][ciphertext]
+          #
+          # The wrapped key needs no size prefix as OAEP output is always exactly the modulus
+          # size of the wrapping key. The version byte allows introducing new envelope layouts
+          # (different AEAD, compression, etc.) without falling back to length arithmetic.
+          VERSION = "\x01".b.freeze
+
+          # Number of bytes of the leading format version marker
+          VERSION_BYTES = 1
+
+          # AES cipher used for the envelope payload encryption
+          AES = "aes-256-gcm"
+
+          # Number of bytes of the AES key wrapped inside the envelope. Derived from the
+          # cipher so a future AES change cannot silently break the invariant
+          KEY_BYTES = OpenSSL::Cipher.new(AES).key_len
+
+          # Number of bytes of the AES-GCM initialization vector, derived like the key size
+          IV_BYTES = OpenSSL::Cipher.new(AES).iv_len
+
+          # Number of bytes of the AES-GCM authentication tag. This is a choice (GCM supports
+          # shorter tags), not a cipher-derived property - 16 is the full, recommended size
+          TAG_BYTES = 16
+
+          # OAEP options for the AES key wrapping. SHA-256 for both MGF1 and the label hash -
+          # OAEP does not lean on collision resistance, but the SHA-1 default draws flags from
+          # scanners and compliance checklists
+          OAEP_OPTIONS = {
+            rsa_padding_mode: "oaep",
+            rsa_oaep_md: "sha256",
+            rsa_mgf1_md: "sha256"
+          }.freeze
+
+          private_constant :VERSION, :VERSION_BYTES, :AES, :KEY_BYTES, :IV_BYTES, :TAG_BYTES,
+            :OAEP_OPTIONS
+
+          # Encrypts content with a one-time AES-256-GCM key and RSA-wraps that key using OAEP
+          # padding. Unlike PKCS1 v1.5, OAEP unwrapping with a non-matching key fails reliably
+          # instead of occasionally yielding garbage. The GCM tag additionally authenticates
+          # the whole envelope header, so any bit flip in the version byte, wrapped key or iv
+          # is detected, not only ciphertext corruption.
+          #
+          # @param content [String] content to encrypt
+          # @return [String] binary envelope (see {VERSION} for the format)
+          def encrypt(content)
+            aes = OpenSSL::Cipher.new(AES).encrypt
+            aes_key = aes.random_key
+            iv = aes.random_iv
+
+            wrapped_key = public_pem.encrypt(aes_key, OAEP_OPTIONS)
+
+            header = VERSION + wrapped_key + iv
+            aes.auth_data = header
+
+            # `Cipher#update` rejects empty input on openssl gem < 3.1 (`data must not be
+            # empty`), so empty payloads go straight to `#final`
+            ciphertext = content.empty? ? aes.final : aes.update(content) + aes.final
+
+            header + aes.auth_tag(TAG_BYTES) + ciphertext
+          end
+
+          # Decrypts an envelope produced by {#encrypt}
+          # @param version [String] encryption version
+          # @param content [String] binary envelope
+          # @return [String] decrypted content
+          # @note All failure paths stay within the `OpenSSL::PKey` error family: the explicit
+          #   guards raise `RSAError`, while an OAEP unwrap failure (e.g. non-matching private
+          #   key) surfaces from the EVP API as its parent `PKeyError`
+          def decrypt(version, content)
+            content = content.b
+
+            if content.bytesize < VERSION_BYTES
+              raise(OpenSSL::PKey::RSAError, "corrupted or truncated envelope")
+            end
+
+            # Version goes next so future layouts of different sizes are reported as
+            # unsupported to older consumers instead of as corrupted
+            if content[0, VERSION_BYTES] != VERSION
+              raise(OpenSSL::PKey::RSAError, "unsupported envelope version")
+            end
+
+            pem = private_pem(version)
+            wrapped_size = pem.n.num_bytes
+            header_size = VERSION_BYTES + wrapped_size + IV_BYTES
+            min_size = header_size + TAG_BYTES
+
+            if content.bytesize < min_size
+              raise(OpenSSL::PKey::RSAError, "corrupted or truncated envelope")
+            end
+
+            wrapped_key = content[VERSION_BYTES, wrapped_size]
+            iv = content[VERSION_BYTES + wrapped_size, IV_BYTES]
+            tag = content[header_size, TAG_BYTES]
+            ciphertext = content[min_size..]
+
+            aes_key = pem.decrypt(wrapped_key, OAEP_OPTIONS)
+
+            # OAEP unwrapping of a foreign envelope fails reliably, but anyone holding the
+            # public key can wrap a string of arbitrary length. Without this guard such input
+            # would surface as an ArgumentError from the AES key assignment, escaping the
+            # OpenSSL error family this method otherwise normalizes to
+            unless aes_key.bytesize == KEY_BYTES
+              raise(OpenSSL::PKey::RSAError, "invalid envelope key size")
+            end
+
+            aes = OpenSSL::Cipher.new(AES).decrypt
+            aes.key = aes_key
+            aes.iv = iv
+            aes.auth_tag = tag
+            aes.auth_data = content[0, header_size]
+
+            # Same empty-input guard as on the encryption side; `#final` still runs and thus
+            # still verifies the auth tag for empty payloads
+            ciphertext.empty? ? aes.final : aes.update(ciphertext) + aes.final
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/pro/encryption/contracts/config.rb

@@ -43,6 +43,7 @@ module Karafka
 
           nested(:encryption) do
             required(:active) { |val| [true, false].include?(val) }
+            required(:mode) { |val| %i[envelope direct].include?(val) }
             required(:version) { |val| val.is_a?(String) && !val.empty? }
             required(:public_key) { |val| val.is_a?(String) }
             required(:fingerprinter) { |val| val == false || val.respond_to?(:hexdigest) }

data/lib/karafka/pro/encryption/setup/config.rb

@@ -54,6 +54,28 @@ module Karafka
           # This allows us to support key rotation
           setting(:private_keys, default: {})
 
+          # Encryption mode used when producing messages:
+          #
+          # - `:direct` (default) - payload is RSA-encrypted directly, which limits it to the
+          #   RSA key capacity (key size minus padding, ~245 bytes for 2048-bit keys). Default
+          #   for backwards compatibility with already running deployments. The default will
+          #   switch to `:envelope` in a future release (with prior notice); decryption of the
+          #   `:direct` format is never planned for removal, as data at rest never expires.
+          # - `:envelope` - payload is encrypted with a one-time AES-256-GCM key and only that
+          #   key is RSA-wrapped, so payloads of any size are supported and the GCM auth tag
+          #   detects corruption and truncation. Note this is not authenticity: the public key
+          #   is distributed to all producers, so any of its holders can build a valid envelope
+          #
+          # Decryption always supports both formats regardless of this setting. Since processes
+          # older than the version that introduced this setting cannot decrypt envelope
+          # payloads, when enabling `:envelope` upgrade all consuming processes first and only
+          # then switch producers to the envelope mode.
+          #
+          # The envelope openssl gem requirement (>= 3.0) is verified during setup. Flipping
+          # this setting to `:envelope` at runtime on an unsupported openssl bypasses that
+          # friendly boot error and fails on first use instead.
+          setting(:mode, default: :direct)
+
           # Cipher used to encrypt and decrypt data
           setting(:cipher, default: Encryption::Cipher.new)
 

data/lib/karafka/pro/encryption.rb

@@ -43,6 +43,12 @@ module Karafka
           config.instance_eval do
             setting(:encryption, default: Setup::Config.config)
           end
+
+          # Registered as a config-phase constraint (verified centrally during setup together
+          # with all other environment requirements) instead of being checked ad hoc here
+          Karafka::Constraints.register(:pro_encryption_envelope_openssl, phase: :config) do |cfg|
+            verify_envelope_requirements!(cfg)
+          end
         end
 
         # @param config [Karafka::Core::Configurable::Node] root node config
@@ -60,6 +66,13 @@ module Karafka
 
           # Encryption for WaterDrop
           config.producer.middleware.append(Messages::Middleware.new)
+
+          # Warm the cipher internals (sub-ciphers, parsed key material) in this
+          # single-threaded phase so runtime encryption and decryption across worker threads
+          # only read already-built state. Custom ciphers can opt in by exposing
+          # #warmup(config)
+          cipher = config.encryption.cipher
+          cipher.warmup(config) if cipher.respond_to?(:warmup)
         end
 
         # This feature does not need any changes post-fork
@@ -69,6 +82,27 @@ module Karafka
         def post_fork(_config, _pre_fork_producer)
           true
         end
+
+        private
+
+        # The envelope mode relies on the EVP PKey API (`PKey#encrypt`/`#decrypt` with an
+        # options hash), available since the openssl gem 3.0. All supported Rubies bundle a
+        # sufficient version as a default gem, but it can be pinned lower in a Gemfile, hence
+        # this runtime constraint instead of a gemspec dependency that everyone would carry
+        # for a single opt-in feature.
+        #
+        # @param config [Karafka::Core::Configurable::Node] root node config
+        def verify_envelope_requirements!(config)
+          return unless config.encryption.active
+          return unless config.encryption.mode == :envelope
+          return if Gem::Version.new(OpenSSL::VERSION) >= Gem::Version.new("3.0.0")
+
+          raise(
+            Karafka::Errors::DependencyConstraintsError,
+            "encryption.mode = :envelope requires the openssl gem >= 3.0, " \
+            "#{OpenSSL::VERSION} detected"
+          )
+        end
       end
     end
   end

data/lib/karafka/pro/iterator.rb

@@ -128,6 +128,11 @@ module Karafka
         # and the whole iteration becomes a silent no-op.
         @stopped_partitions = Set.new
         @stopped = false
+        # Reset the stored-offsets latch so it reflects only the current run's marking activity.
+        # Otherwise a single `mark_as_consumed` would keep firing a spurious blocking sync commit on
+        # the teardown of every subsequent `#each`, each of which runs on a brand-new consumer that
+        # has no stored offsets.
+        @stored_offsets = false
       end
 
       # Stops the partition we're currently yielded into

data/lib/karafka/pro/processing/consumer_groups/strategies/dlq/default.rb

@@ -262,7 +262,7 @@ module Karafka
                   # Use custom topic if it was returned from the strategy
                   @_dispatch_to_dlq_topic = target_topic || topic.dead_letter_queue.topic
                 else
-                  raise Karafka::UnsupportedCaseError, flow
+                  raise Karafka::Errors::UnsupportedCaseError, flow
                 end
 
                 yield

data/lib/karafka/pro/processing/consumer_groups/strategies/dlq/vp.rb

@@ -97,7 +97,7 @@ module Karafka
                   # Use custom topic if it was returned from the strategy
                   @_dispatch_to_dlq_topic = target_topic || topic.dead_letter_queue.topic
                 else
-                  raise Karafka::UnsupportedCaseError, flow
+                  raise Karafka::Errors::UnsupportedCaseError, flow
                 end
 
                 yield

data/lib/karafka/pro/processing/consumer_groups/strategies/ftr/default.rb

@@ -130,7 +130,7 @@ module Karafka
                     pause(throttle_message.offset, throttle_timeout, false)
                   end
                 else
-                  raise Karafka::Errors::UnsupportedCaseError filter.action
+                  raise Karafka::Errors::UnsupportedCaseError, filter.action
                 end
               end
             end

data/lib/karafka/pro/processing/jobs_queue.rb

@@ -90,12 +90,14 @@ module Karafka
         # @param job [Jobs::Base] job that locked the queue
         def unlock(job)
           @mutex.synchronize do
-            @statistics[:waiting] -= 1
+            if @in_waiting[job.group_id].delete(job)
+              @statistics[:waiting] -= 1
 
-            return if @in_waiting[job.group_id].delete(job)
+              return
+            end
 
             # This should never happen. It means there was a job being unlocked that was never
-            # locked in the first place
+            # locked in the first place (or its group was already reset via `#clear`)
             raise(Errors::JobsQueueSynchronizationError, job.group_id)
           end
         end
@@ -185,6 +187,14 @@ module Karafka
         #   Thanks to the fact that we use the minimum lock time as a timeout, we do not have to
         #   wait a whole ticking period to unlock async locks.
         def wait(group_id)
+          # Drain stale `#tick` signals up front: for non-blocking/LRJ jobs and released async
+          # locks (both Pro-only) `#wait?` can be false from the start, so the loop below - the
+          # only other consumer - never runs and signals would otherwise accumulate unbounded.
+          # Safe without locking as `#wait` runs only on the single listener thread that owns
+          # (and has already registered) this group.
+          semaphore = @semaphores.fetch(group_id)
+          semaphore.pop(timeout: 0) until semaphore.empty?
+
           return super unless @async_locking
 
           # We do not generalize this flow because this one is more expensive as it has to allocate

data/lib/karafka/pro/routing/features/consumer_groups/patterns/detector.rb

@@ -79,11 +79,34 @@ module Karafka
               def install(pattern, discovered_topic, sg_topics)
                 group = pattern.topic.group
 
-                # Build new topic and register within the owning group
-                topic = group.public_send(:topic=, discovered_topic, &pattern.config)
-                topic.patterns(active: true, type: :discovered)
+                # Under multiplexing several subscription groups share the same consumer group but
+                # each runs its own listener thread and independently discovers the same topic.
+                # ConsumerGroup#topic= appends unconditionally, so without this guard the shared
+                # group would accumulate up to `multiplex_factor` duplicate Topic objects per
+                # discovered topic, polluting the routing tree. We run under the process-wide
+                # MUTEX (see #expand), so the find + register below is atomic across subscription
+                # groups: the discovered topic is registered in the shared group only once.
+                topic = group.topics.detect { |existing| existing.name == discovered_topic }
 
-                # Assign the appropriate subscription group to this topic
+                if topic
+                  # A sibling multiplexed subscription group already registered this topic in the
+                  # shared consumer group. Each subscription group must still hold its OWN Topic
+                  # instance in its topics array: the Topic doubles as a per-subscription-group
+                  # key (Instrumentation::AssignmentsTracker keys assignments on the instance and
+                  # reads topic.subscription_group.id), so reusing one instance across
+                  # subscription groups would collide their assignments. We inject a
+                  # per-subscription-group copy, mirroring how the multiplexing subscription
+                  # groups builder dups topics per group at boot.
+                  topic = topic.dup
+                else
+                  # Build new topic and register within the owning group
+                  topic = group.public_send(:topic=, discovered_topic, &pattern.config)
+                  topic.patterns(active: true, type: :discovered)
+                end
+
+                # Assign the appropriate subscription group to this topic. Each subscription group
+                # owns its own Topic instance (freshly built or dup'd above), so this never leaks
+                # across multiplexed subscription groups.
                 topic.subscription_group = pattern.topic.subscription_group
 
                 # Inject into subscription group topics array always, so everything is reflected

data/lib/karafka/pro/scheduled_messages/consumer.rb

@@ -121,22 +121,19 @@ module Karafka
           # that all tombstone events are loaded not to duplicate dispatches
           return unless @state.loaded?
 
-          keys = []
-
-          # We first collect all the data for dispatch and then dispatch and **only** after
-          # dispatch that is sync is successful we remove those messages from the daily buffer
-          # and update the max epoch. Since only the dispatch itself is volatile and can crash
-          # with timeouts, etc, we need to be sure it wen through prior to deleting those messages
-          # from the daily buffer. That way we ensure the at least once delivery and in case of
-          # a transactional producer, exactly once delivery.
+          # We first collect all the data for dispatch and then dispatch it in sync batches. Each
+          # key is removed from the daily buffer only once the batch carrying it is confirmed
+          # delivered - not after the whole flush - so a broker error partway through does not
+          # leave already-produced batches in the buffer to be re-dispatched on the next tick
+          # (a duplicate-dispatch window for non-transactional producers). This keeps the
+          # at-least-once guarantee, and exactly-once when a transactional producer is used.
           @daily_buffer.for_dispatch do |message|
-            keys << message.key
             @dispatcher << message
           end
 
-          @dispatcher.flush
-
-          keys.each { |key| @daily_buffer.delete(key) }
+          @dispatcher.flush do |dispatched_keys|
+            dispatched_keys.each { |key| @daily_buffer.delete(key) }
+          end
 
           @states_reporter.call
         end

data/lib/karafka/pro/scheduled_messages/contracts/message.rb

@@ -58,6 +58,15 @@ module Karafka
             schedule_source_type
           ].freeze
 
+          # Upper bound (in seconds) for how far ahead a schedule may target: 100 years.
+          # schedule_target_epoch is Unix time in seconds; anything beyond this is almost certainly
+          # a unit mistake (for example milliseconds passed as seconds, ~1000x too large). We reject
+          # it here so it fails loudly at publish time instead of being silently dropped by the
+          # consumer.
+          MAX_FUTURE_INTERVAL = 100 * 365 * 24 * 60 * 60
+
+          private_constant :MAX_FUTURE_INTERVAL
+
           required(:key) { |val| val.is_a?(String) && val.size.positive? }
 
           # Ensure that schedule has all correct keys and that others have other related data
@@ -84,9 +93,15 @@ module Karafka
 
             # We allow for small lag as those will be dispatched but we should prevent dispatching
             # in the past in general as often it is a source of errors
-            next if epoch_time >= Time.now.to_i - 10
+            if epoch_time < Time.now.to_i - 10
+              next [[[:headers], :schedule_target_epoch_in_the_past]]
+            end
+
+            # Reject epochs implausibly far ahead (e.g. milliseconds passed as seconds), which would
+            # otherwise validate here and then be silently dropped by the consumer, never scheduled
+            next if epoch_time <= Time.now.to_i + MAX_FUTURE_INTERVAL
 
-            [[[:headers], :schedule_target_epoch_in_the_past]]
+            [[[:headers], :schedule_target_epoch_too_far_in_future]]
           end
 
           # Makes sure, that the target envelope topic we dispatch to is a scheduled messages topic

data/lib/karafka/pro/scheduled_messages/dispatcher.rb

@@ -46,6 +46,10 @@ module Karafka
           @topic = topic
           @partition = partition
           @buffer = []
+          # Source (daily buffer) key aligned 1:1 with each `@buffer` entry, so `#flush` can report
+          # which keys were confirmed delivered per chunk and the consumer can evict them
+          # incrementally instead of only after the whole flush succeeds
+          @keys = []
           @serializer = Serializer.new
         end
 
@@ -78,12 +82,21 @@ module Karafka
           extract(target, message.headers, :key)
           extract(target, message.headers, :partition_key)
 
+          # `message.key` is pushed to `@keys` once per `@buffer` entry (here and below), not once
+          # per message: `@keys` must stay 1:1 aligned with `@buffer` (see `#flush`) so that
+          # shifting a chunk off both arrays together always yields the correct keys for that
+          # chunk. The target and its tombstone are two separate `@buffer` entries for the same
+          # schedule, so the same key is intentionally paired with each. `DailyBuffer#delete` is a
+          # plain `Hash#delete`, so a key evicted twice within the same yielded chunk (once per
+          # entry) is a harmless no-op the second time.
           @buffer << target
+          @keys << message.key
 
           # Tombstone message so this schedule is no longer in use and gets removed from Kafka by
           # Kafka itself during compacting. It will not cancel it because already dispatched but
           # will cause it not to be sent again and will be marked as dispatched.
           @buffer << Proxy.tombstone(message: message)
+          @keys << message.key
         end
 
         # Builds and dispatches the state report message with schedules details
@@ -105,14 +118,42 @@ module Karafka
         # Sends all messages to Kafka in a sync way.
         # We use sync with batches to prevent overloading.
         # When transactional producer in use, this will be wrapped in a transaction automatically.
+        #
+        # @yieldparam [Array<String>] keys of the chunk that was just confirmed delivered. Yielded
+        #   after each chunk's sync produce returns, so the caller can evict those keys from the
+        #   daily buffer incrementally. If a later chunk raises, the chunks already produced have
+        #   still been reported, so a non-transactional producer will not re-dispatch them.
+        # @raise [ArgumentError] when called without a block, since a caller that cannot observe
+        #   per-chunk confirmations cannot evict incrementally and would reintroduce the
+        #   whole-flush-or-nothing duplicate-dispatch window this method exists to close
         def flush
+          raise ArgumentError, "#flush requires a block to report per-chunk confirmations" unless block_given?
+
           until @buffer.empty?
-            config.producer.produce_many_sync(
-              # We can remove this prior to the dispatch because we only evict messages from the
-              # daily buffer once dispatch is successful
-              @buffer.shift(config.flush_batch_size)
-            )
+            batch_size = config.flush_batch_size
+
+            # A message's target and its tombstone are always buffered as an adjacent pair (see
+            # `#<<`). Rounding the chunk size up to even guarantees a chunk boundary can never
+            # fall between them - with an odd (or 1) `flush_batch_size`, a chunk could otherwise
+            # confirm and yield a key whose target was produced but whose tombstone was not, and a
+            # later chunk failing would then leave that schedule non-tombstoned in Kafka, to be
+            # re-dispatched after a restart/reload.
+            batch_size += 1 if batch_size.odd?
+
+            messages = @buffer.shift(batch_size)
+            keys = @keys.shift(batch_size)
+
+            config.producer.produce_many_sync(messages)
+
+            yield(keys)
           end
+        ensure
+          # Whether flush finished normally (buffer already empty here, so this is a no-op) or
+          # raised partway through, drop anything left. Those messages are still in the daily
+          # buffer (their keys were never yielded, so never evicted) and will be re-buffered on
+          # the next tick, so stale leftovers here must not be dispatched a second time.
+          @buffer.clear
+          @keys.clear
         end
 
         private

data/lib/karafka/processing/consumer_groups/coordinators_buffer.rb

@@ -48,10 +48,11 @@ module Karafka
         def revoke(topic_name, partition)
           return unless @coordinators[topic_name].key?(partition)
 
-          # Reset the partition's pause tracker attempt count. The tracker lives in the pauses
-          # manager keyed by topic-partition and would otherwise be reused as-is if we reclaim this
-          # partition, carrying a stale retry attempt count across the rebalance (which, with DLQ,
-          # would send the next failure straight to the dead letter queue, skipping the retries)
+          # Reset (or, if not currently paused, remove) the partition's pause tracker. This
+          # prevents a stale retry attempt count from being reused as-is if we reclaim this
+          # partition (which, with DLQ, would send the next failure straight to the dead letter
+          # queue, skipping the retries) while also keeping `PausesManager#@pauses` from growing
+          # unbounded. See `PausesManager#revoke` for the full reasoning
           @pauses_manager.revoke(@topics.find(topic_name), partition)
 
           # The fact that we delete here does not change the fact that the executor still holds the

data/lib/karafka/server.rb

@@ -115,8 +115,11 @@ module Karafka
 
         # We check from time to time (for the timeout period) if all the threads finished
         # their work and if so, we can just return and normal shutdown process will take place
-        # We divide it by 1000 because we use time in ms.
-        ((timeout / 1_000) * (1 / supervision_sleep)).to_i.times do
+        # We use float math and divide by 1000 (ms -> s) so that sub-second `shutdown_timeout`
+        # values do not collapse to zero iterations (integer division would floor them to 0,
+        # skipping the grace period entirely and forcing an immediate forceful shutdown). We
+        # `ceil` to guarantee at least one supervision check for any positive timeout.
+        (timeout / (supervision_sleep * 1_000)).ceil.times do
           all_listeners_stopped = listeners.all?(&:stopped?)
           all_workers_stopped = workers.stopped?
 

data/lib/karafka/setup/config.rb

@@ -430,12 +430,12 @@ module Karafka
             scope: %w[config]
           )
 
-          configure_components(proxy)
+          # Verify config-dependent constraints (registered by features for their optional
+          # environment requirements) once the config shape is guaranteed by the contract but
+          # before any components spin up
+          Karafka::Constraints.verify!(:config, config)
 
-          # Install backwards-compatible forwarding so that gems (e.g. karafka-testing) that
-          # still access config.internal.processing.strategy_selector (etc.) keep working after
-          # the move to config.internal.processing.consumer_groups.*
-          install_processing_cg_forwarders(config)
+          configure_components(proxy)
 
           # Refreshes the references that are cached that might have been changed by the config
           Karafka.refresh!
@@ -460,38 +460,6 @@ module Karafka
 
         private
 
-        # Installs forwarding reader methods on the processing config node so that the old
-        # (pre-nesting) paths like `config.internal.processing.strategy_selector` still resolve
-        # by delegating to `config.internal.processing.consumer_groups.strategy_selector`.
-        # This keeps external gems (e.g. karafka-testing) working until they migrate.
-        #
-        # @param config [Karafka::Core::Configurable::Node] root config node
-        def install_processing_cg_forwarders(config)
-          processing = config.internal.processing
-          cg_node = processing.consumer_groups
-
-          %i[
-            jobs_builder
-            coordinator_class
-            errors_tracker_class
-            partitioner_class
-            strategy_selector
-            expansions_selector
-            executor_class
-          ].each do |setting_name|
-            writer = :"#{setting_name}="
-
-            # Remove previous definitions (if setup runs more than once) to avoid
-            # "method redefined" warnings that spec_helper promotes to errors
-            sc = processing.singleton_class
-            sc.remove_method(setting_name) if processing.respond_to?(setting_name)
-            sc.remove_method(writer) if processing.respond_to?(writer)
-
-            processing.define_singleton_method(setting_name) { cg_node.public_send(setting_name) }
-            processing.define_singleton_method(writer) { |val| cg_node.public_send(writer, val) }
-          end
-        end
-
         # Sets up all the components that are based on the user configuration
         # @param config_proxy [ConfigProxy] the configuration proxy containing deferred setup
         #   blocks

data/lib/karafka/swarm/node.rb

@@ -190,6 +190,8 @@ module Karafka
       # @param signal [String]
       # @return [Boolean] true if signal was sent, false if process doesn't exist
       def signal(signal)
+        @mutex.synchronize { return false if @alive == false }
+
         ::Process.kill(signal, @pid)
         true
       rescue Errno::ESRCH

data/lib/karafka/swarm/supervisor.rb

@@ -143,8 +143,10 @@ module Karafka
 
         # We check from time to time (for the timeout period) if all the threads finished
         # their work and if so, we can just return and normal shutdown process will take place
-        # We divide it by 1000 because we use time in ms.
-        ((total_shutdown_timeout / 1_000) * (1 / supervision_sleep)).to_i.times do
+        # We use float math and divide by 1000 (ms -> s) so that sub-second timeout values do not
+        # collapse to zero iterations via integer division. We `ceil` to guarantee at least one
+        # supervision check for any positive timeout.
+        (total_shutdown_timeout / (supervision_sleep * 1_000)).ceil.times do
           if manager.stopped?
             manager.cleanup
             return

data/lib/karafka/version.rb

@@ -2,5 +2,5 @@
 
 module Karafka
   # Current Karafka version
-  VERSION = "2.6.0"
+  VERSION = "2.6.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: karafka
 version: !ruby/object:Gem::Version
-  version: 2.6.0
+  version: 2.6.1
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -234,6 +234,9 @@ files:
 - lib/karafka/pro/contracts/base.rb
 - lib/karafka/pro/encryption.rb
 - lib/karafka/pro/encryption/cipher.rb
+- lib/karafka/pro/encryption/ciphers/base.rb
+- lib/karafka/pro/encryption/ciphers/direct.rb
+- lib/karafka/pro/encryption/ciphers/envelope.rb
 - lib/karafka/pro/encryption/contracts/config.rb
 - lib/karafka/pro/encryption/errors.rb
 - lib/karafka/pro/encryption/messages/middleware.rb