karafka 2.4.13 → 2.4.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4556d1cccdb4ff51132f141f71a3c81d05dfdf71839241566bed12aa40ffcb83
-  data.tar.gz: 92af5f2db3c7614efdff4f4e0e92e012befd2b3f61bd89eb6b509cfb82a0569c
+  metadata.gz: 5b49de31147b2a64d8927c91e25e6b607fdf1a8a7280b7cc3ba8c3663cf96b6f
+  data.tar.gz: 4a3d9e439e21b79b9ac6a2c8bf4e76219f970be1f6bb7316feb5c7f2f792271e
 SHA512:
-  metadata.gz: a1a28ea5cfc19cb7b3c2861fdc755bdb5e1c0e718b8b0244d87675e0d1cbae4cc568f8af28984a2845287e291207d4532e6c92cf87327ef17fb87f8263cd0d57
-  data.tar.gz: cc77cceea79fd8cbbe5013221fb4427daf9209aed781fff0d684ea1128a9b3984d69cb42dd14bd88c0d01968820e4f8e8ce3e1450d505614f7d12434cc46aedf
+  metadata.gz: 527a1c169ddc5a0f978f69dc3dbae6d30288997d2cfc95104ad6343d7a468c67afbde4c55051c1e7902d2c161fc1e615f8d703b9dfba123f734b9fbd7ccec5d1
+  data.tar.gz: a895657a8dece0c9bdb65afeeb00bc8df2e03e1957d345bc71ceab9d6cce74e5d0b4afcc251e26dfe230778a05b8742e40505a779695325844c6be630307ad82

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,51 +1,43 @@
 ---
 name: Bug Report
-about: Report an issue with Karafka you've discovered.
+about: Report an issue within the Karafka ecosystem you've discovered.
 ---
 
-*Be clear, concise and precise in your description of the problem.
-Open an issue with a descriptive title and a summary in grammatically correct,
-complete sentences.*
+To make this process smoother for everyone involved, please read the following information before filling out the template.
 
-*Use the template below when reporting bugs. Please, make sure that
-you're running the latest stable Karafka and that the problem you're reporting
-hasn't been reported (and potentially fixed) already.*
+Scope of the OSS Support
+===========
 
-*Before filing the ticket you should replace all text above the horizontal
-rule with your own words.*
+We do not provide OSS support for outdated versions of Karafka and its components.
 
---------
+Please ensure that you are using a version that is still actively supported. We cannot assist with any no longer maintained versions unless you support us with our Pro offering (https://karafka.io/docs/Pro-Support/).
 
-## Expected behavior
+We acknowledge that understanding the specifics of your application and its configuration can be essential for resolving certain issues. However, due to the extensive time and resources such analysis can require, this may fall beyond our Open Source Support scope.
 
-Describe here how you expected Karafka to behave in this particular situation.
+If Karafka or its components are critical to your infrastructure, we encourage you to consider our Pro Offering.
 
-## Actual behavior
+By backing us up, you can gain direct assistance and ensure your use case receives the dedicated attention it deserves.
 
-Describe here what actually happened.
 
-## Steps to reproduce the problem
+Important Links to Read
+===========
 
-This is extremely important! Providing us with a reliable way to reproduce
-a problem will expedite its solution.
+Please take a moment to review the following resources before submitting your report:
 
-## Your setup details
+- Issue Reporting Guide: https://karafka.io/docs/Support/#issue-reporting-guide
+- Support Policy: https://karafka.io/docs/Support/
+- Versions, Lifecycle, and EOL: https://karafka.io/docs/Versions-Lifecycle-and-EOL/
 
-Please provide kafka version and the output of `karafka info` or `bundle exec karafka info` if using Bundler.
 
-Here's an example:
+Bug Report Details
+===========
 
-```
-$ [bundle exec] karafka info
-Karafka version: 2.2.10 + Pro
-Ruby version: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
-Rdkafka version: 0.13.8
-Consumer groups count: 2
-Subscription groups count: 2
-Workers count: 2
-Application client id: example_app
-Boot file: /app/karafka.rb
-Environment: development
-License: Commercial
-License entity: karafka-ci
-```
+Please provide all the details per our Issue Reporting Guide: https://karafka.io/docs/Support/#issue-reporting-guide
+
+Failing to provide the required details may result in the issue being closed. Please include all necessary information to help us understand and resolve your issue effectively.
+
+
+Additional Context
+===========
+
+Add any other context about the problem here.

data/.github/workflows/ci.yml

@@ -89,6 +89,13 @@ jobs:
         run: |
           docker compose up -d || (sleep 5 && docker compose up -d)
 
+      # Newer versions of ActiveSupport and Rails do not work with Ruby 3.1 anymore.
+      # While we use newer by default we do want to resolve older and test, thus we remove
+      # Gemfile.lock and let it resolve to the most compatible version possible
+      - name: Remove Gemfile.lock if Ruby 3.1
+        if: matrix.ruby == '3.1'
+        run: rm -f Gemfile.lock
+
       - name: Set up Ruby
         uses: ruby/setup-ruby@v1
         with:

data/.ruby-version

@@ -1 +1 @@
-3.3.5
+3.3.6

data/CHANGELOG.md

@@ -1,7 +1,19 @@
 # Karafka Framework Changelog
 
+## 2.4.15 (2024-12-04)
+- [Fix] Assignment tracker current state fetch during a rebalance loop can cause an error on multi CG setup.
+- [Fix] Prevent double post-transaction offset dispatch to Kafka.
+
+## 2.4.14 (2024-11-25)
+- [Enhancement] Improve low-level critical error reporting.
+- [Enhancement] Expand Kubernetes Liveness state reporting with critical errors detection.
+- [Enhancement] Save several string allocations and one array allocation on each job execution when using Datadog instrumentation.
+- [Enhancement] Support `eofed` jobs in the AppSignal instrumentation.
+- [Enhancement] Allow running bootfile-less Rails setup Karafka CLI commands where stuff is configured in the initializers.
+- [Fix] `Instrumentation::Vendors::Datadog::LoggerListener` treats eof jobs as consume jobs.
+
 ## 2.4.13 (2024-10-11)
-- [Enhancement] Make declarative topics return different exit codes on migrable/non-migrable states (0 - no changes, 2 - changes).
+- [Enhancement] Make declarative topics return different exit codes on migrable/non-migrable states (0 - no changes, 2 - changes) when used with `--detailed-exitcode` flag.
 - [Enhancement] Introduce `config.strict_declarative_topics` that should force declaratives on all non-pattern based topics and DLQ topics
 - [Enhancement] Report ignored repartitioning to lower number of partitions in declarative topics.
 - [Enhancement] Promote the `LivenessListener#healty?` to a public API.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    karafka (2.4.13)
+    karafka (2.4.15)
       base64 (~> 0.2)
       karafka-core (>= 2.4.4, < 2.5.0)
       karafka-rdkafka (>= 0.17.2)
@@ -11,11 +11,12 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    activejob (7.2.1)
-      activesupport (= 7.2.1)
+    activejob (8.0.0)
+      activesupport (= 8.0.0)
       globalid (>= 0.3.6)
-    activesupport (7.2.1)
+    activesupport (8.0.0)
       base64
+      benchmark (>= 0.3)
       bigdecimal
       concurrent-ruby (~> 1.0, >= 1.3.1)
       connection_pool (>= 2.2.5)
@@ -25,7 +26,9 @@ GEM
       minitest (>= 5.1)
       securerandom (>= 0.3)
       tzinfo (~> 2.0, >= 2.0.5)
+      uri (>= 0.13.1)
     base64 (0.2.0)
+    benchmark (0.3.0)
     bigdecimal (3.1.8)
     byebug (11.1.3)
     concurrent-ruby (1.3.4)
@@ -44,10 +47,10 @@ GEM
       raabro (~> 1.4)
     globalid (1.2.1)
       activesupport (>= 6.1)
-    i18n (1.14.5)
+    i18n (1.14.6)
       concurrent-ruby (~> 1.0)
-    karafka-core (2.4.4)
-      karafka-rdkafka (>= 0.15.0, < 0.18.0)
+    karafka-core (2.4.5)
+      karafka-rdkafka (>= 0.17.6, < 0.18.0)
     karafka-rdkafka (0.17.6)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
@@ -55,18 +58,18 @@ GEM
     karafka-testing (2.4.6)
       karafka (>= 2.4.0, < 2.5.0)
       waterdrop (>= 2.7.0)
-    karafka-web (0.10.3)
+    karafka-web (0.10.4)
       erubi (~> 1.4)
       karafka (>= 2.4.10, < 2.5.0)
       karafka-core (>= 2.4.0, < 2.5.0)
       roda (~> 3.68, >= 3.69)
       tilt (~> 2.0)
     logger (1.6.1)
-    mini_portile2 (2.8.7)
+    mini_portile2 (2.8.8)
     minitest (5.25.1)
-    ostruct (0.6.0)
+    ostruct (0.6.1)
     raabro (1.4.0)
-    rack (3.1.7)
+    rack (3.1.8)
     rake (13.2.1)
     roda (3.84.0)
       rack
@@ -93,6 +96,7 @@ GEM
     tilt (2.4.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
+    uri (1.0.0)
     waterdrop (2.8.0)
       karafka-core (>= 2.4.3, < 3.0.0)
       karafka-rdkafka (>= 0.17.5)

data/bin/integrations

@@ -243,9 +243,13 @@ ARGV.each do |filter|
   end
 end
 
-# Remove Rails 7.2 specs from Ruby 3.0 because it requires 3.1
+# Remove Rails 7.2 specs from Ruby < 3.1 because it requires 3.1
+# Remove Rails 8.0 specs from Ruby < 3.2 because it requires 3.2
 specs.delete_if do |spec|
-  RUBY_VERSION < '3.1' && spec.include?('rails72')
+  next true if RUBY_VERSION < '3.1' && spec.include?('rails72')
+  next true if RUBY_VERSION < '3.2' && spec.include?('rails8')
+
+  false
 end
 
 raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?

data/docker-compose.yml

@@ -3,7 +3,7 @@ version: '2'
 services:
   kafka:
     container_name: kafka
-    image: confluentinc/cp-kafka:7.7.1
+    image: confluentinc/cp-kafka:7.8.0
 
     ports:
       - 9092:9092

data/lib/karafka/cli/base.rb

@@ -41,22 +41,38 @@ module Karafka
       class << self
         # Loads proper environment with what is needed to run the CLI
         def load
+          rails_env_rb = File.join(Dir.pwd, 'config/environment.rb')
+          is_rails = Kernel.const_defined?(:Rails) && File.exist?(rails_env_rb)
+
+          # If the boot file is disabled and this is a Rails app, we assume that user moved the
+          # karafka app configuration to initializers or other Rails loading related place.
+          # It is not recommended but some users tend to do this. In such cases we just try to load
+          # the Rails stuff hoping that it will also load Karafka stuff
+          if Karafka.boot_file.to_s == 'false' && is_rails
+            require rails_env_rb
+
+            return
+          end
+
           # If there is a boot file, we need to require it as we expect it to contain
           # Karafka app setup, routes, etc
           if File.exist?(::Karafka.boot_file)
-            rails_env_rb = File.join(Dir.pwd, 'config/environment.rb')
-
             # Load Rails environment file that starts Rails, so we can reference consumers and
             # other things from `karafka.rb` file. This will work only for Rails, for non-rails
             # a manual setup is needed
-            require rails_env_rb if Kernel.const_defined?(:Rails) && File.exist?(rails_env_rb)
-
+            require rails_env_rb if is_rails
             require Karafka.boot_file.to_s
+
+            return
+          end
+
           # However when it is unavailable, we still want to be able to run help command
           # and install command as they don't require configured app itself to run
-          elsif %w[-h install].none? { |cmd| cmd == ARGV[0] }
-            raise ::Karafka::Errors::MissingBootFileError, ::Karafka.boot_file
-          end
+          return if %w[-h install].any? { |cmd| cmd == ARGV[0] }
+
+          # All other commands except help and install do require an existing boot file if it was
+          # declared
+          raise ::Karafka::Errors::MissingBootFileError, ::Karafka.boot_file
         end
 
         # Allows to set options for Thor cli

data/lib/karafka/connection/client.rb

@@ -41,6 +41,9 @@ module Karafka
         :topic_authorization_failed, # 29
         :group_authorization_failed, # 30
         :cluster_authorization_failed, # 31
+        :illegal_generation,
+        # this will not recover as fencing is permanent
+        :fenced, # -144
         # This can happen for many reasons, including issues with static membership being fenced
         :fatal # -150
       ].freeze

data/lib/karafka/instrumentation/assignments_tracker.rb

@@ -31,8 +31,13 @@ module Karafka
       def current
         assignments = {}
 
-        @assignments.each do |topic, partitions|
-          assignments[topic] = partitions.dup.freeze
+        # Since the `@assignments` state can change during a rebalance, if we would iterate over
+        # it exactly during state change, we would end up with the following error:
+        #   RuntimeError: can't add a new key into hash during iteration
+        @mutex.synchronize do
+          @assignments.each do |topic, partitions|
+            assignments[topic] = partitions.dup.freeze
+          end
         end
 
         assignments.freeze

data/lib/karafka/instrumentation/vendors/appsignal/metrics_listener.rb

@@ -48,6 +48,7 @@ module Karafka
             consumer.revoked.error
             consumer.shutdown.error
             consumer.tick.error
+            consumer.eofed.error
           ].freeze
 
           private_constant :USER_CONSUMER_ERROR_TYPES
@@ -107,7 +108,8 @@ module Karafka
           [
             %i[revoke revoked revoked],
             %i[shutting_down shutdown shutdown],
-            %i[tick ticked tick]
+            %i[tick ticked tick],
+            %i[eof eofed eofed]
           ].each do |before, after, name|
             class_eval <<~RUBY, __FILE__, __LINE__ + 1
               # Keeps track of user code execution

data/lib/karafka/instrumentation/vendors/datadog/logger_listener.rb

@@ -35,6 +35,7 @@ module Karafka
           def initialize(&block)
             configure
             setup(&block) if block
+            @job_types_cache = {}
           end
 
           # @param block [Proc] configuration block
@@ -51,7 +52,7 @@ module Karafka
             push_tags
 
             job = event[:job]
-            job_type = job.class.to_s.split('::').last
+            job_type = fetch_job_type(job.class)
             consumer = job.executor.topic.consumer
             topic = job.executor.topic.name
 
@@ -68,8 +69,16 @@ module Karafka
                        'revoked'
                      when 'Idle'
                        'idle'
-                     else
+                     when 'Eofed'
+                       'eofed'
+                     when 'EofedNonBlocking'
+                       'eofed'
+                     when 'ConsumeNonBlocking'
                        'consume'
+                     when 'Consume'
+                       'consume'
+                     else
+                       raise Errors::UnsupportedCaseError, job_type
                      end
 
             current_span.resource = "#{consumer}##{action}"
@@ -121,6 +130,8 @@ module Karafka
               error "Consumer on shutdown failed due to an error: #{error}"
             when 'consumer.tick.error'
               error "Consumer tick failed due to an error: #{error}"
+            when 'consumer.eofed.error'
+              error "Consumer eofed failed due to an error: #{error}"
             when 'worker.process.error'
               fatal "Worker processing failed due to an error: #{error}"
             when 'connection.listener.fetch_loop.error'
@@ -169,6 +180,18 @@ module Karafka
 
             Karafka.logger.pop_tags
           end
+
+          private
+
+          # Takes the job class and extracts the job type.
+          # @param job_class [Class] job class
+          # @return [String]
+          # @note It does not have to be thread-safe despite running in multiple threads because
+          #   the assignment race condition is irrelevant here since the same value will be
+          #   assigned.
+          def fetch_job_type(job_class)
+            @job_types_cache[job_class] ||= job_class.to_s.split('::').last
+          end
         end
       end
     end

data/lib/karafka/instrumentation/vendors/datadog/metrics_listener.rb

@@ -82,10 +82,11 @@ module Karafka
             statistics = event[:statistics]
             consumer_group_id = event[:consumer_group_id]
 
-            base_tags = default_tags + ["consumer_group:#{consumer_group_id}"]
+            tags = ["consumer_group:#{consumer_group_id}"]
+            tags.concat(default_tags)
 
             rd_kafka_metrics.each do |metric|
-              report_metric(metric, statistics, base_tags)
+              report_metric(metric, statistics, tags)
             end
           end
 
@@ -93,13 +94,14 @@ module Karafka
           #
           # @param event [Karafka::Core::Monitoring::Event]
           def on_error_occurred(event)
-            extra_tags = ["type:#{event[:type]}"]
+            tags = ["type:#{event[:type]}"]
+            tags.concat(default_tags)
 
             if event.payload[:caller].respond_to?(:messages)
-              extra_tags += consumer_tags(event.payload[:caller])
+              tags.concat(consumer_tags(event.payload[:caller]))
             end
 
-            count('error_occurred', 1, tags: default_tags + extra_tags)
+            count('error_occurred', 1, tags: tags)
           end
 
           # Reports how many messages we've polled and how much time did we spend on it
@@ -111,10 +113,11 @@ module Karafka
 
             consumer_group_id = event[:subscription_group].consumer_group.id
 
-            extra_tags = ["consumer_group:#{consumer_group_id}"]
+            tags = ["consumer_group:#{consumer_group_id}"]
+            tags.concat(default_tags)
 
-            histogram('listener.polling.time_taken', time_taken, tags: default_tags + extra_tags)
-            histogram('listener.polling.messages', messages_count, tags: default_tags + extra_tags)
+            histogram('listener.polling.time_taken', time_taken, tags: tags)
+            histogram('listener.polling.messages', messages_count, tags: tags)
           end
 
           # Here we report majority of things related to processing as we have access to the
@@ -125,7 +128,8 @@ module Karafka
             messages = consumer.messages
             metadata = messages.metadata
 
-            tags = default_tags + consumer_tags(consumer)
+            tags = consumer_tags(consumer)
+            tags.concat(default_tags)
 
             count('consumer.messages', messages.count, tags: tags)
             count('consumer.batches', 1, tags: tags)
@@ -146,7 +150,8 @@ module Karafka
               #
               # @param event [Karafka::Core::Monitoring::Event]
               def on_consumer_#{after}(event)
-                tags = default_tags + consumer_tags(event.payload[:caller])
+                tags = consumer_tags(event.payload[:caller])
+                tags.concat(default_tags)
 
                 count('consumer.#{name}', 1, tags: tags)
               end
@@ -158,9 +163,10 @@ module Karafka
           def on_worker_process(event)
             jq_stats = event[:jobs_queue].statistics
 
-            gauge('worker.total_threads', Karafka::App.config.concurrency, tags: default_tags)
-            histogram('worker.processing', jq_stats[:busy], tags: default_tags)
-            histogram('worker.enqueued_jobs', jq_stats[:enqueued], tags: default_tags)
+            tags = default_tags
+            gauge('worker.total_threads', Karafka::App.config.concurrency, tags: tags)
+            histogram('worker.processing', jq_stats[:busy], tags: tags)
+            histogram('worker.enqueued_jobs', jq_stats[:enqueued], tags: tags)
           end
 
           # We report this metric before and after processing for higher accuracy
@@ -240,11 +246,14 @@ module Karafka
                 # node ids
                 next if broker_statistics['nodeid'] == -1
 
+                tags = ["broker:#{broker_statistics['nodename']}"]
+                tags.concat(base_tags)
+
                 public_send(
                   metric.type,
                   metric.name,
                   broker_statistics.dig(*metric.key_location),
-                  tags: base_tags + ["broker:#{broker_statistics['nodename']}"]
+                  tags: tags
                 )
               end
             when :topics
@@ -259,14 +268,14 @@ module Karafka
                   next if partition_statistics['fetch_state'] == 'stopped'
                   next if partition_statistics['fetch_state'] == 'none'
 
+                  tags = ["topic:#{topic_name}", "partition:#{partition_name}"]
+                  tags.concat(base_tags)
+
                   public_send(
                     metric.type,
                     metric.name,
                     partition_statistics.dig(*metric.key_location),
-                    tags: base_tags + [
-                      "topic:#{topic_name}",
-                      "partition:#{partition_name}"
-                    ]
+                    tags: tags
                   )
                 end
               end

data/lib/karafka/instrumentation/vendors/kubernetes/liveness_listener.rb

@@ -26,6 +26,19 @@ module Karafka
         #
         # @note Please use `Kubernetes::SwarmLivenessListener` when operating in the swarm mode
         class LivenessListener < BaseListener
+          # When any of those occurs, it means something went wrong in a way that cannot be
+          # recovered. In such cases we should report that the consumer process is not healthy.
+          # - `fenced` - This instance has been fenced by a newer instance and will not do any
+          #   processing at all never. Fencing most of the time means the instance.group.id has
+          #   been reused without properly terminating the previous consumer process first
+          # - `fatal` - any fatal error that halts the processing forever
+          UNRECOVERABLE_RDKAFKA_ERRORS = [
+            :fenced, # -144
+            :fatal # -150
+          ].freeze
+
+          private_constant :UNRECOVERABLE_RDKAFKA_ERRORS
+
           # @param hostname [String, nil] hostname or nil to bind on all
           # @param port [Integer] TCP port on which we want to run our HTTP status server
           # @param consuming_ttl [Integer] time in ms after which we consider consumption hanging.
@@ -40,6 +53,11 @@ module Karafka
             consuming_ttl: 5 * 60 * 1_000,
             polling_ttl: 5 * 60 * 1_000
           )
+            # If this is set to true, it indicates unrecoverable error like fencing
+            # While fencing can be partial (for one of the SGs), we still should consider this
+            # as an undesired state for the whole process because it halts processing in a
+            # non-recoverable manner forever
+            @unrecoverable = false
             @polling_ttl = polling_ttl
             @consuming_ttl = consuming_ttl
             @mutex = Mutex.new
@@ -86,10 +104,19 @@ module Karafka
             RUBY
           end
 
-          # @param _event [Karafka::Core::Monitoring::Event]
-          def on_error_occurred(_event)
+          # @param event [Karafka::Core::Monitoring::Event]
+          def on_error_occurred(event)
             clear_consumption_tick
             clear_polling_tick
+
+            error = event[:error]
+
+            # We are only interested in the rdkafka errors
+            return unless error.is_a?(Rdkafka::RdkafkaError)
+            # We mark as unrecoverable only on certain errors that will not be fixed by retrying
+            return unless UNRECOVERABLE_RDKAFKA_ERRORS.include?(error.code)
+
+            @unrecoverable = true
           end
 
           # Deregister the polling tracker for given listener
@@ -117,6 +144,7 @@ module Karafka
           def healthy?
             time = monotonic_now
 
+            return false if @unrecoverable
             return false if @pollings.values.any? { |tick| (time - tick) > @polling_ttl }
             return false if @consumptions.values.any? { |tick| (time - tick) > @consuming_ttl }
 

data/lib/karafka/pro/processing/strategies/default.rb

@@ -63,6 +63,16 @@ module Karafka
               # Ignore earlier offsets than the one we already committed
               return true if coordinator.seek_offset > message.offset
               return false if revoked?
+
+              # If we have already marked this successfully in a transaction that was running
+              # we should not mark it again with the client offset delegation but instead we should
+              # just align the in-memory state
+              if @_in_transaction_marked
+                coordinator.seek_offset = message.offset + 1
+
+                return true
+              end
+
               return revoked? unless client.mark_as_consumed(message, offset_metadata)
 
               coordinator.seek_offset = message.offset + 1
@@ -90,6 +100,12 @@ module Karafka
               return true if coordinator.seek_offset > message.offset
               return false if revoked?
 
+              if @_in_transaction_marked
+                coordinator.seek_offset = message.offset + 1
+
+                return true
+              end
+
               return revoked? unless client.mark_as_consumed!(message, offset_metadata)
 
               coordinator.seek_offset = message.offset + 1
@@ -132,6 +148,7 @@ module Karafka
             transaction_started = true
             @_transaction_marked = []
             @_in_transaction = true
+            @_in_transaction_marked = false
 
             producer.transaction(&block)
 
@@ -143,6 +160,11 @@ module Karafka
             # @note We never need to use the blocking `#mark_as_consumed!` here because the offset
             #   anyhow was already stored during the transaction
             #
+            # @note Since the offset could have been already stored in Kafka (could have because
+            #   you can have transactions without marking), we use the `@_in_transaction_marked`
+            #   state to decide if we need to dispatch the offset via client at all
+            #   (if post transaction, then we do not have to)
+            #
             # @note In theory we could only keep reference to the most recent marking and reject
             #   others. We however do not do it for two reasons:
             #   - User may have non standard flow relying on some alternative order and we want to
@@ -158,6 +180,7 @@ module Karafka
             if transaction_started
               @_transaction_marked.clear
               @_in_transaction = false
+              @_in_transaction_marked = false
             end
           end
 
@@ -178,6 +201,7 @@ module Karafka
               offset_metadata
             )
 
+            @_in_transaction_marked = true
             @_transaction_marked ||= []
             @_transaction_marked << [message, offset_metadata, async]
           end

data/lib/karafka/routing/builder.rb

@@ -67,6 +67,20 @@ module Karafka
         end
       end
 
+      # Clear out the drawn routes.
+      alias array_clear clear
+      private :array_clear
+
+      # Clear routes and draw them again with the given block. Helpful for testing purposes.
+      # @param block [Proc] block we will evaluate within the builder context
+      def redraw(&block)
+        @mutex.synchronize do
+          @draws.clear
+          array_clear
+        end
+        draw(&block)
+      end
+
       # @return [Array<Karafka::Routing::ConsumerGroup>] only active consumer groups that
       #   we want to use. Since Karafka supports multi-process setup, we need to be able
       #   to pick only those consumer groups that should be active in our given process context
@@ -79,7 +93,7 @@ module Karafka
         @mutex.synchronize do
           @defaults = EMPTY_DEFAULTS
           @draws.clear
-          super
+          array_clear
         end
       end
 

data/lib/karafka/version.rb

@@ -3,5 +3,5 @@
 # Main module namespace
 module Karafka
   # Current Karafka version
-  VERSION = '2.4.13'
+  VERSION = '2.4.15'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: karafka
 version: !ruby/object:Gem::Version
-  version: 2.4.13
+  version: 2.4.15
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
   ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
   -----END CERTIFICATE-----
-date: 2024-10-11 00:00:00.000000000 Z
+date: 2024-12-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: base64
@@ -620,7 +620,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.16
+rubygems_version: 3.5.22
 signing_key: 
 specification_version: 4
 summary: Karafka is Ruby and Rails efficient Kafka processing framework.