karafka 2.2.8 → 2.2.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e440e6a6a067d23095d6c0b60f4cd534080e2fd1cc3e528f1052a6368b60cab0
-  data.tar.gz: d5d1c98a0ac28677b860e74ae0c59b13bba68c07bd84964fb0ca7ac473454fcc
+  metadata.gz: 460029e2dbc352c56f107e18bbff28f5914b5b56a0e42be9670e13f0f3592fc9
+  data.tar.gz: 68c25bacd1995a8e9c97a915e1e1b2b4435437c97a5e31e2824499c9ad531057
 SHA512:
-  metadata.gz: f5ea25e23361caba43073fb36c7cf91dacd105ed4a9595929c0e48b608e07701dbcf0be32f8982ae14809a1511f5ddc0cd2ae13fe5263f9ea9d8b2aed538fb16
-  data.tar.gz: 175c25bc43361facb9a7664fbe89fc48a7e7426be8ded047d5a1c549b1db67a0172ee3954972b07adeecf92c3dc4dc2c61e979d9905fa40967133d1872c6a46b
+  metadata.gz: adc877c3be3cb8885b0244168014387f0353bed89575b6931f5e1ca062d01ba6a0e5a0566da4dd282f5117c7daf27ff2116f27025ec4213972c1ca52e92980de
+  data.tar.gz: a62ff96ef96ea85d98288bad3c8f6cf0bfafc1ee6de1cf928fe803ad80435b0fa137aac9cf5eff4b232e854b3c4df76c3c16efaa4a117d6b86e442975cfc9a30

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -37,14 +37,15 @@ Here's an example:
 
 ```
 $ [bundle exec] karafka info
-Karafka version: 1.3.0
-Ruby version: 2.6.3
-Ruby-kafka version: 0.7.9
-Application client id: karafka-local
-Backend: inline
-Batch fetching: true
-Batch consuming: true
-Boot file: /app/karafka/karafka.rb
+Karafka version: 2.2.10 + Pro
+Ruby version: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
+Rdkafka version: 0.13.8
+Consumer groups count: 2
+Subscription groups count: 2
+Workers count: 2
+Application client id: example_app
+Boot file: /app/karafka.rb
 Environment: development
-Kafka seed brokers: ["kafka://kafka:9092"]
+License: Commercial
+License entity: karafka-ci
 ```

data/.github/workflows/ci.yml

@@ -1,6 +1,8 @@
 name: ci
 
-concurrency: ci-${{ github.ref }}
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
 
 on:
   pull_request:

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Karafka framework changelog
 
+## 2.2.10 (2023-11-02)
+- [Improvement] Allow for running `#pause` without specifying the offset (provide offset or `:consecutive`). This allows for pausing on the consecutive message (last received + 1), so after resume we will get last message received + 1 effectively not using `#seek` and not purging `librdafka` buffer preserving on networking. Please be mindful that this uses notion of last message passed from **librdkafka**, and not the last one available in the consumer (`messages.last`). While for regular cases they will be the same, when using things like DLQ, LRJs, VPs or Filtering API, those may not be the same.
+- [Improvement] **Drastically** improve network efficiency of operating with LRJ by using the `:consecutive` offset as default strategy for running LRJs without moving the offset in place and purging the data.
+- [Improvement] Do not "seek in place". When pausing and/or seeking to the same location as the current position, do nothing not to purge buffers and not to move to the same place where we are.
+- [Fix] Pattern regexps should not be part of declaratives even when configured.
+
+### Upgrade Notes
+
+In the latest Karafka release, there are no breaking changes. However, please note the updates to #pause and #seek. If you spot any issues, please report them immediately. Your feedback is crucial.
+
+## 2.2.9 (2023-10-24)
+- [Improvement] Allow using negative offset references in `Karafka::Admin#read_topic`.
+- [Change] Make sure that WaterDrop `2.6.10` or higher is used with this release to support transactions fully and the Web-UI.
+
 ## 2.2.8 (2023-10-20)
 - **[Feature]** Introduce Appsignal integration for errors and metrics tracking.
 - [Improvement] Expose `#synchronize` for VPs to allow for locks when cross-VP consumers work is needed.
@@ -7,7 +21,7 @@
 - [Refactor] Reorganize how rebalance events are propagated from `librdkafka` to Karafka. Replace `connection.client.rebalance_callback` with `rebalance.partitions_assigned` and `rebalance.partitions_revoked`. Introduce two extra events: `rebalance.partitions_assign` and `rebalance.partitions_revoke` to handle pre-rebalance future work.
 - [Refactor] Remove `thor` as a CLI layer and rely on Ruby `OptParser`
 
-### Upgrade notes
+### Upgrade Notes
 
 1. Unless you were using `connection.client.rebalance_callback` which was considered private, nothing.
 2. None of the CLI commands should change but `thor` has been removed so please report if you find any bugs.
@@ -49,7 +63,7 @@
 - [Enhancement] Make sure that consumer group used by `Karafka::Admin` obeys the `ConsumerMapper` setup.
 - [Fix] Fix a case where subscription group would not accept a symbol name.
 
-### Upgrade notes
+### Upgrade Notes
 
 As always, please make sure you have upgraded to the most recent version of `2.1` before upgrading to `2.2`.
 
@@ -180,7 +194,7 @@ If you want to maintain the `2.1` behavior, that is `karafka_admin` admin group,
 - [Change] Removed `Karafka::Pro::BaseConsumer` in favor of `Karafka::BaseConsumer`. (#1345)
 - [Fix] Fix for `max_messages` and `max_wait_time` not having reference in errors.yml (#1443)
 
-### Upgrade notes
+### Upgrade Notes
 
 1. Upgrade to Karafka `2.0.41` prior to upgrading to `2.1.0`.
 2. Replace `Karafka::Pro::BaseConsumer` references to `Karafka::BaseConsumer`.
@@ -252,7 +266,7 @@ If you want to maintain the `2.1` behavior, that is `karafka_admin` admin group,
 - [Improvement] Attach an `embedded` tag to Karafka processes started using the embedded API.
 - [Change] Renamed `Datadog::Listener` to `Datadog::MetricsListener` for consistency. (#1124)
 
-### Upgrade notes
+### Upgrade Notes
 
 1. Replace `Datadog::Listener` references to `Datadog::MetricsListener`.
 
@@ -266,7 +280,7 @@ If you want to maintain the `2.1` behavior, that is `karafka_admin` admin group,
 - [Improvement] Introduce a `strict_topics_namespacing` config option to enable/disable the strict topics naming validations. This can be useful when working with pre-existing topics which we cannot or do not want to rename.
 - [Fix] Karafka monitor is prematurely cached (#1314)
 
-### Upgrade notes
+### Upgrade Notes
 
 Since `#tags` were introduced on consumers, the `#tags` method is now part of the consumers API.
 
@@ -328,7 +342,7 @@ end
 - [Improvement] Include `original_consumer_group` in the DLQ dispatched messages in Pro.
 - [Improvement] Use Karafka `client_id` as kafka `client.id` value by default
 
-### Upgrade notes
+### Upgrade Notes
 
 If you want to continue to use `karafka` as default for kafka `client.id`, assign it manually:
 
@@ -381,7 +395,7 @@ class KarafkaApp < Karafka::App
 - [Fix] Do not trigger code reloading when `consumer_persistence` is enabled.
 - [Fix] Shutdown producer after all the consumer components are down and the status is stopped. This will ensure, that any instrumentation related Kafka messaging can still operate.
 
-### Upgrade notes
+### Upgrade Notes
 
 If you want to disable `librdkafka` statistics because you do not use them at all, update the `kafka` `statistics.interval.ms` setting and set it to `0`:
 
@@ -443,7 +457,7 @@ end
 - [Fix] Few typos around DLQ and Pro DLQ Dispatch original metadata naming.
 - [Fix] Narrow the components lookup to the appropriate scope (#1114)
 
-### Upgrade notes
+### Upgrade Notes
 
 1. Replace `original-*` references from DLQ dispatched metadata with `original_*`
 
@@ -486,7 +500,7 @@ end
 - [Specs] Split specs into regular and pro to simplify how resources are loaded
 - [Specs] Add specs to ensure, that all the Pro components have a proper per-file license (#1099)
 
-### Upgrade notes
+### Upgrade Notes
 
 1. Remove the `manual_offset_management` setting from the main config if you use it:
 

data/Gemfile.lock

@@ -1,9 +1,9 @@
 PATH
   remote: .
   specs:
-    karafka (2.2.8)
+    karafka (2.2.10)
       karafka-core (>= 2.2.2, < 2.3.0)
-      waterdrop (>= 2.6.6, < 3.0.0)
+      waterdrop (>= 2.6.10, < 3.0.0)
       zeitwerk (~> 2.3)
 
 GEM
@@ -39,24 +39,24 @@ GEM
       activesupport (>= 6.1)
     i18n (1.14.1)
       concurrent-ruby (~> 1.0)
-    karafka-core (2.2.3)
+    karafka-core (2.2.5)
       concurrent-ruby (>= 1.1)
-      karafka-rdkafka (>= 0.13.6, < 0.14.0)
-    karafka-rdkafka (0.13.6)
+      karafka-rdkafka (>= 0.13.8, < 0.15.0)
+    karafka-rdkafka (0.13.8)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
       rake (> 12)
-    karafka-web (0.7.7)
+    karafka-web (0.7.10)
       erubi (~> 1.4)
-      karafka (>= 2.2.8.beta1, < 3.0.0)
-      karafka-core (>= 2.2.2, < 3.0.0)
+      karafka (>= 2.2.9, < 3.0.0)
+      karafka-core (>= 2.2.4, < 3.0.0)
       roda (~> 3.68, >= 3.69)
       tilt (~> 2.0)
-    mini_portile2 (2.8.4)
+    mini_portile2 (2.8.5)
     minitest (5.20.0)
     mutex_m (0.1.2)
     rack (3.0.8)
-    rake (13.0.6)
+    rake (13.1.0)
     roda (3.73.0)
       rack
     rspec (3.12.0)
@@ -82,8 +82,8 @@ GEM
     tilt (2.3.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    waterdrop (2.6.7)
-      karafka-core (>= 2.1.1, < 3.0.0)
+    waterdrop (2.6.11)
+      karafka-core (>= 2.2.3, < 3.0.0)
       zeitwerk (~> 2.3)
     zeitwerk (2.6.12)
 

data/karafka.gemspec

@@ -22,7 +22,7 @@ Gem::Specification.new do |spec|
   DESC
 
   spec.add_dependency 'karafka-core', '>= 2.2.2', '< 2.3.0'
-  spec.add_dependency 'waterdrop', '>= 2.6.6', '< 3.0.0'
+  spec.add_dependency 'waterdrop', '>= 2.6.10', '< 3.0.0'
   spec.add_dependency 'zeitwerk', '~> 2.3'
 
   if $PROGRAM_NAME.end_with?('gem')

data/lib/karafka/admin.rb

@@ -18,7 +18,7 @@ module Karafka
       # @param count [Integer] how many messages we want to get at most
       # @param start_offset [Integer, Time] offset from which we should start. If -1 is provided
       #   (default) we will start from the latest offset. If time is provided, the appropriate
-      #   offset will be resolved.
+      #   offset will be resolved. If negative beyond -1 is provided, we move backwards more.
       # @param settings [Hash] kafka extra settings (optional)
       #
       # @return [Array<Karafka::Messages::Message>] array with messages
@@ -33,8 +33,10 @@ module Karafka
 
           low_offset, high_offset = consumer.query_watermark_offsets(name, partition)
 
-          # Select offset dynamically if -1 or less
-          start_offset = high_offset - count if start_offset.negative?
+          # Select offset dynamically if -1 or less and move backwards with the negative
+          # offset, allowing to start from N messages back from high-watermark
+          start_offset = high_offset - count - start_offset.abs + 1 if start_offset.negative?
+          start_offset = low_offset if start_offset.negative?
 
           # Build the requested range - since first element is on the start offset we need to
           # subtract one from requested count to end up with expected number of elements

data/lib/karafka/base_consumer.rb

@@ -171,23 +171,28 @@ module Karafka
       @used
     end
 
-    # Pauses processing on a given offset for the current topic partition
+    # Pauses processing on a given offset or consecutive offset for the current topic partition
     #
     # After given partition is resumed, it will continue processing from the given offset
-    # @param offset [Integer] offset from which we want to restart the processing
+    # @param offset [Integer, Symbol] offset from which we want to restart the processing or
+    #  `:consecutive` if we want to pause and continue without changing the consecutive offset
+    #  (cursor position)
     # @param timeout [Integer, nil] how long in milliseconds do we want to pause or nil to use the
     #   default exponential pausing strategy defined for retries
     # @param manual_pause [Boolean] Flag to differentiate between user pause and system/strategy
     #   based pause. While they both pause in exactly the same way, the strategy application
     #   may need to differentiate between them.
+    #
+    # @note It is **critical** to understand how pause with `:consecutive` offset operates. While
+    #   it provides benefit of not purging librdkafka buffer, in case of usage of filters, retries
+    #   or other advanced options the consecutive offset may not be the one you want to pause on.
+    #   Test it well to ensure, that this behaviour is expected by you.
     def pause(offset, timeout = nil, manual_pause = true)
       timeout ? coordinator.pause_tracker.pause(timeout) : coordinator.pause_tracker.pause
 
-      client.pause(
-        topic.name,
-        partition,
-        offset
-      )
+      offset = nil if offset == :consecutive
+
+      client.pause(topic.name, partition, offset)
 
       # Indicate, that user took a manual action of pausing
       coordinator.manual_pause if manual_pause

data/lib/karafka/connection/client.rb

@@ -160,16 +160,17 @@ module Karafka
       #
       # @param topic [String] topic name
       # @param partition [Integer] partition
-      # @param offset [Integer] offset of the message on which we want to pause (this message will
-      #   be reprocessed after getting back to processing)
+      # @param offset [Integer, nil] offset of the message on which we want to pause (this message
+      #   will be reprocessed after getting back to processing) or nil if we want to pause and
+      #   resume from the consecutive offset (+1 from the last message passed to us by librdkafka)
       # @note This will pause indefinitely and requires manual `#resume`
-      def pause(topic, partition, offset)
+      # @note When `#internal_seek` is not involved (when offset is `nil`) we will not purge the
+      #   librdkafka buffers and continue from the last cursor offset
+      def pause(topic, partition, offset = nil)
         @mutex.synchronize do
           # Do not pause if the client got closed, would not change anything
           return if @closed
 
-          pause_msg = Messages::Seek.new(topic, partition, offset)
-
           internal_commit_offsets(async: true)
 
           # Here we do not use our cached tpls because we should not try to pause something we do
@@ -190,6 +191,14 @@ module Karafka
           @paused_tpls[topic][partition] = tpl
 
           @kafka.pause(tpl)
+
+          # If offset is not provided, will pause where it finished.
+          # This makes librdkafka not purge buffers and can provide significant network savings
+          # when we just want to pause before further processing without changing the offsets
+          return unless offset
+
+          pause_msg = Messages::Seek.new(topic, partition, offset)
+
           internal_seek(pause_msg)
         end
       end
@@ -354,6 +363,9 @@ module Karafka
       #
       # @param message [Messages::Message, Messages::Seek] message to which we want to seek to.
       #   It can have the time based offset.
+      #
+      # @note Will not invoke seeking if the desired seek would lead us to the current position.
+      #   This prevents us from flushing librdkafka buffer when it is not needed.
       def internal_seek(message)
         # If the seek message offset is in a time format, we need to find the closest "real"
         # offset matching before we seek
@@ -378,6 +390,14 @@ module Karafka
           message.offset = detected_partition&.offset || raise(Errors::InvalidTimeBasedOffsetError)
         end
 
+        # Never seek if we would get the same location as we would get without seeking
+        # This prevents us from the expensive buffer purges that can lead to increased network
+        # traffic and can cost a lot of money
+        #
+        # This code adds around 0.01 ms per seek but saves from many user unexpected behaviours in
+        # seeking and pausing
+        return if message.offset == topic_partition_position(message.topic, message.partition)
+
         @kafka.seek(message)
       end
 
@@ -432,6 +452,18 @@ module Karafka
         Rdkafka::Consumer::TopicPartitionList.new({ topic => [rdkafka_partition] })
       end
 
+      # @param topic [String]
+      # @param partition [Integer]
+      # @return [Integer] current position within topic partition or `-1` if it could not be
+      #   established. It may be `-1` in case we lost the assignment or we did not yet fetch data
+      #   for this topic partition
+      def topic_partition_position(topic, partition)
+        rd_partition = ::Rdkafka::Consumer::Partition.new(partition, nil, 0)
+        tpl = ::Rdkafka::Consumer::TopicPartitionList.new(topic => [rd_partition])
+
+        @kafka.position(tpl).to_h.fetch(topic).first.offset || -1
+      end
+
       # Performs a single poll operation and handles retries and error
       #
       # @param timeout [Integer] timeout for a single poll

data/lib/karafka/instrumentation/logger_listener.rb

@@ -71,6 +71,8 @@ module Karafka
       # Prints info about a consumer pause occurrence. Irrelevant if user or system initiated.
       #
       # @param event [Karafka::Core::Monitoring::Event] event details including payload
+      # @note There may be no offset provided in case user wants to pause on the consecutive offset
+      #   position. This can be beneficial when not wanting to purge the buffers.
       def on_client_pause(event)
         topic = event[:topic]
         partition = event[:partition]
@@ -78,7 +80,9 @@ module Karafka
         client = event[:caller]
 
         info <<~MSG.tr("\n", ' ').strip!
-          [#{client.id}] Pausing on topic #{topic}/#{partition} on offset #{offset}
+          [#{client.id}]
+          Pausing on topic #{topic}/#{partition}
+          on #{offset ? "offset #{offset}" : 'the consecutive offset'}
         MSG
       end
 

data/lib/karafka/pro/processing/strategies/aj/lrj_mom_vp.rb

@@ -37,7 +37,7 @@ module Karafka
               super
 
               coordinator.on_enqueued do
-                pause(coordinator.seek_offset, Strategies::Lrj::Default::MAX_PAUSE_TIME, false)
+                pause(:consecutive, Strategies::Lrj::Default::MAX_PAUSE_TIME, false)
               end
             end
 

data/lib/karafka/pro/processing/strategies/lrj/default.rb

@@ -35,12 +35,10 @@ module Karafka
               # This ensures that when running LRJ with VP, things operate as expected run only
               # once for all the virtual partitions collectively
               coordinator.on_enqueued do
-                # Pause at the first message in a batch. That way in case of a crash, we will not
-                # loose any messages.
-                #
-                # For VP it applies the same way and since VP cannot be used with MOM we should not
-                # have any edge cases here.
-                pause(coordinator.seek_offset, MAX_PAUSE_TIME, false)
+                # Pause and continue with another batch in case of a regular resume.
+                # In case of an error, the `#retry_after_pause` will move the offset to the first
+                # message out of this batch.
+                pause(:consecutive, MAX_PAUSE_TIME, false)
               end
             end
 

data/lib/karafka/pro/processing/strategies/lrj/mom.rb

@@ -35,12 +35,7 @@ module Karafka
               # This ensures that when running LRJ with VP, things operate as expected run only
               # once for all the virtual partitions collectively
               coordinator.on_enqueued do
-                # Pause at the first message in a batch. That way in case of a crash, we will not
-                # loose any messages.
-                #
-                # For VP it applies the same way and since VP cannot be used with MOM we should not
-                # have any edge cases here.
-                pause(coordinator.seek_offset, Strategies::Lrj::Default::MAX_PAUSE_TIME, false)
+                pause(:consecutive, Strategies::Lrj::Default::MAX_PAUSE_TIME, false)
               end
             end
 

data/lib/karafka/pro/routing/features/patterns/consumer_group.rb

@@ -50,6 +50,10 @@ module Karafka
               virtual_topic = public_send(:topic=, pattern.name, &block)
               # Indicate the nature of this topic (matcher)
               virtual_topic.patterns(active: true, type: :matcher, pattern: pattern)
+              # Pattern subscriptions should never be part of declarative topics definitions
+              # Since they are subscribed by regular expressions, we do not know the target
+              # topics names so we cannot manage them via declaratives
+              virtual_topic.config(active: false)
               pattern.topic = virtual_topic
               @patterns << pattern
             end

data/lib/karafka/version.rb

@@ -3,5 +3,5 @@
 # Main module namespace
 module Karafka
   # Current Karafka version
-  VERSION = '2.2.8'
+  VERSION = '2.2.10'
 end

data/renovate.json

@@ -1,6 +1,9 @@
 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
   "extends": [
-    "config:base"
+      "config:base"
+  ],
+  "ignorePaths": [
+      "spec/integrations"
   ]
 }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: karafka
 version: !ruby/object:Gem::Version
-  version: 2.2.8
+  version: 2.2.10
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   AnG1dJU+yL2BK7vaVytLTstJME5mepSZ46qqIJXMuWob/YPDmVaBF39TDSG9e34s
   msG3BiCqgOgHAnL23+CN3Rt8MsuRfEtoTKpJVcCfoEoNHOkc
   -----END CERTIFICATE-----
-date: 2023-10-20 00:00:00.000000000 Z
+date: 2023-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: karafka-core
@@ -63,7 +63,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.6.6
+        version: 2.6.10
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -73,7 +73,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.6.6
+        version: 2.6.10
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0