waterdrop 2.6.0 → 2.6.1.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ed73a2332f0161e71e385fd2250c96bc43383eead9f2945b2580226f855f2643
-  data.tar.gz: 20705696d8534e5b3e10e84a4ae85f05a01025007879b0f59cff7d26f61d28e3
+  metadata.gz: 5e637029ce9d9113e2d64cadcec57ac980602d489f2207507194082065604b16
+  data.tar.gz: 21d0a5a643b3e48603fe8e8adb13dbe827a0b9b50871675e57aa62b71abe7573
 SHA512:
-  metadata.gz: 92533a6e46992a10b2c7d4f3c6cece7f905697cb3c7c2576ec3f0364400b14c71695f1202d8c4b896168753c1d38b888e996632e6894a7cdbdc9f92e940540a6
-  data.tar.gz: '0945428add01cbf32e84c15458db0e39cbc1caabf8e7af9db4221b8e6149162a26946206632858154c202e702c2564a0857d9d420cce3711916f64ccfa1eab87'
+  metadata.gz: e34c6c5225022f191dd80da877126b5c37075b3356b1e99e4e6b79adb0c54b8939e6b43f2efded8f68a0b68424b3cdbdcaa8abe19a781456faa4f653ae9063ea
+  data.tar.gz: 5060d94c549d5d0ac4b2e09f8ab4888643008ea732943e76f75dfedd386f08ada8a60ad278f85dcfa1172bf813c993a38b2feb71ee75ea2ce1a72bf3021edbcb

data/CHANGELOG.md

@@ -1,12 +1,18 @@
 # WaterDrop changelog
 
+### 2.6.1 (Unreleased)
+- [Refactor] Remove no longer needed patches.
+- [Fix] Fork detection on a short lived processes seems to fail. Clear the used parent process client reference not to close it in the finalizer (#356).
+- [Change] Require `karafka-rdkafka` `>= 0.13.0.beta2`.
+- [Change] Require 'karafka-core' `>= 2.1.0`
+
 ### 2.6.0 (2023-06-11)
 - [Improvement] Introduce `client_class` setting for ability to replace underlying client with anything specific to a given env (dev, test, etc).
 - [Improvement] Introduce `Clients::Buffered` useful for writing specs that do not have to talk with Kafka (id-ilych)
 - [Improvement] Make `#produce` method private to avoid confusion and make sure it is not used directly (it is not part of the official API).
 - [Change] Change `wait_on_queue_full` from `false` to `true` as a default.
 - [Change] Rename `wait_on_queue_full_timeout` to `wait_backoff_on_queue_full` to match what it actually does.
-- [Enhancement] Introduce `wait_timeout_on_queue_full` with proper meaning. That is, this represents time after which despite backoff the error will be raised. This should allow to raise an error in case the backoff attempts were insufficient. This prevents from a case, where upon never deliverable messages we would end up with an invite loop.
+- [Enhancement] Introduce `wait_timeout_on_queue_full` with proper meaning. That is, this represents time after which despite backoff the error will be raised. This should allow to raise an error in case the backoff attempts were insufficient. This prevents from a case, where upon never deliverable messages we would end up with an infinite loop.
 - [Fix] Provide `type` for queue full errors that references the appropriate public API method correctly.
 
 ### Upgrade notes

data/Gemfile.lock

@@ -1,14 +1,14 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.6.0)
-      karafka-core (>= 2.0.13, < 3.0.0)
+    waterdrop (2.6.1.beta1)
+      karafka-core (>= 2.1.0.beta1, < 3.0.0)
       zeitwerk (~> 2.3)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (7.0.4.3)
+    activesupport (7.0.5)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
@@ -20,25 +20,25 @@ GEM
     factory_bot (6.2.1)
       activesupport (>= 5.0.0)
     ffi (1.15.5)
-    i18n (1.12.0)
+    i18n (1.14.1)
       concurrent-ruby (~> 1.0)
-    karafka-core (2.0.13)
+    karafka-core (2.1.0.beta1)
       concurrent-ruby (>= 1.1)
-      karafka-rdkafka (>= 0.12.3)
-    karafka-rdkafka (0.12.3)
+      karafka-rdkafka (>= 0.13.0.beta2, < 0.14.0)
+    karafka-rdkafka (0.13.0.beta2)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
       rake (> 12)
-    mini_portile2 (2.8.1)
+    mini_portile2 (2.8.2)
     minitest (5.18.0)
     rake (13.0.6)
     rspec (3.12.0)
       rspec-core (~> 3.12.0)
       rspec-expectations (~> 3.12.0)
       rspec-mocks (~> 3.12.0)
-    rspec-core (3.12.1)
+    rspec-core (3.12.2)
       rspec-support (~> 3.12.0)
-    rspec-expectations (3.12.2)
+    rspec-expectations (3.12.3)
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.12.0)
     rspec-mocks (3.12.5)
@@ -53,10 +53,9 @@ GEM
     simplecov_json_formatter (0.1.4)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    zeitwerk (2.6.7)
+    zeitwerk (2.6.8)
 
 PLATFORMS
-  arm64-darwin-21
   x86_64-linux
 
 DEPENDENCIES

data/README.md

@@ -333,6 +333,12 @@ producer.close
 
 See the `WaterDrop::Instrumentation::Notifications::EVENTS` for the list of all the supported events.
 
+### Karafka Web-UI
+
+Karafka [Web UI](https://karafka.io/docs/Web-UI-Getting-Started/) is a user interface for the Karafka framework. The Web UI provides a convenient way for monitor all producers related errors out of the box.
+
+![Example producer errors in Karafka Web-UI](https://raw.githubusercontent.com/karafka/misc/master/printscreens/web-ui/errors-producer.png)
+
 ### Usage statistics
 
 WaterDrop is configured to emit internal `librdkafka` metrics every five seconds. You can change this by setting the `kafka` `statistics.interval.ms` configuration property to a value greater of equal `0`. Emitted statistics are available after subscribing to the `statistics.emitted` publisher event. If set to `0`, metrics will not be published.

data/lib/waterdrop/producer.rb

@@ -76,13 +76,18 @@ module WaterDrop
       @connecting_mutex.synchronize do
         return @client if @client && @pid == Process.pid
 
-        # We should raise an error when trying to use a producer from a fork, that is already
-        # connected to Kafka. We allow forking producers only before they are used
-        raise Errors::ProducerUsedInParentProcess, Process.pid if @status.connected?
-
         # We undefine all the finalizers, in case it was a fork, so the finalizers from the parent
         # process don't leak
         ObjectSpace.undefine_finalizer(id)
+
+        # We should raise an error when trying to use a producer with client from a fork. Always.
+        if @client
+          # We need to reset the client, otherwise there might be attempt to close the parent
+          # client
+          @client = nil
+          raise Errors::ProducerUsedInParentProcess, Process.pid
+        end
+
         # Finalizer tracking is needed for handling shutdowns gracefully.
         # I don't expect everyone to remember about closing all the producers all the time, thus
         # this approach is better. Although it is still worth keeping in mind, that this will
@@ -138,7 +143,7 @@ module WaterDrop
           # It is safe to run it several times but not exactly the same moment
           # We also mark it as closed only if it was connected, if not, it would trigger a new
           # connection that anyhow would be immediately closed
-          client.close(@config.max_wait_timeout) if @client
+          client.close if @client
 
           # Remove callbacks runners that were registered
           ::Karafka::Core::Instrumentation.statistics_callbacks.delete(@id)

data/lib/waterdrop/version.rb

@@ -3,5 +3,5 @@
 # WaterDrop library
 module WaterDrop
   # Current WaterDrop version
-  VERSION = '2.6.0'
+  VERSION = '2.6.1.beta1'
 end

data/waterdrop.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |spec|
   spec.description   = spec.summary
   spec.license       = 'MIT'
 
-  spec.add_dependency 'karafka-core', '>= 2.0.13', '< 3.0.0'
+  spec.add_dependency 'karafka-core', '>= 2.1.0.beta1', '< 3.0.0'
   spec.add_dependency 'zeitwerk', '~> 2.3'
 
   if $PROGRAM_NAME.end_with?('gem')

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.6.0
+  version: 2.6.1.beta1
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   Qf04B9ceLUaC4fPVEz10FyobjaFoY4i32xRto3XnrzeAgfEe4swLq8bQsR3w/EF3
   MGU0FeSV2Yj7Xc2x/7BzLK8xQn5l7Yy75iPF+KP3vVmDHnNl
   -----END CERTIFICATE-----
-date: 2023-06-11 00:00:00.000000000 Z
+date: 2023-06-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: karafka-core
@@ -43,7 +43,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.0.13
+        version: 2.1.0.beta1
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -53,7 +53,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.0.13
+        version: 2.1.0.beta1
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -112,9 +112,6 @@ files:
 - lib/waterdrop/instrumentation/vendors/datadog/dashboard.json
 - lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb
 - lib/waterdrop/middleware.rb
-- lib/waterdrop/patches/rdkafka/client.rb
-- lib/waterdrop/patches/rdkafka/metadata.rb
-- lib/waterdrop/patches/rdkafka/producer.rb
 - lib/waterdrop/producer.rb
 - lib/waterdrop/producer/async.rb
 - lib/waterdrop/producer/buffer.rb
@@ -147,9 +144,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubygems_version: 3.4.10
 signing_key: 

metadata.gz.sig

@@ -1,2 +1 @@
-��Mܩ;s�}դ���.]'�z�5U�jO+N4�A�YNV��̦�Y�o�u@��A�ם��������i=h��W�0�w�T���lU����C���r�<Y��
���]2�w8:���FS�atcPw!�MK�:�o��>ɯ}z��̯籌�G�ġ�5)�]%^��Kw�<�j��f(��l����uoz�n�`H�	qY�#?	�&�6�@�r+oK�������A{�ǖ_N���3ar���Y�����x��л�
-ٹ��@��l�.K��Z�2��'�#�1V)�J8�B;�˲�XsIx���~�{�])����$�-v���_�gԖd}���MM>�5lF\��b�/�F�T\�V��#�
+Z9}O��Tg+��zvQ�7�w����jh%/4�A�Ѱc�)�0A��)��u~�.]�!S�

data/lib/waterdrop/patches/rdkafka/client.rb

@@ -1,34 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  module Patches
-    module Rdkafka
-      # Patches for the producer client
-      module Client
-        # @param _object_id [nil] rdkafka API compatibility argument
-        # @param timeout_ms [Integer] final flush timeout in ms
-        def close(_object_id = nil, timeout_ms = 5_000)
-          return unless @native
-
-          # Indicate to polling thread that we're closing
-          @polling_thread[:closing] = true
-          # Wait for the polling thread to finish up
-          @polling_thread.join
-
-          ::Rdkafka::Bindings.rd_kafka_flush(@native, timeout_ms)
-          ::Rdkafka::Bindings.rd_kafka_destroy(@native)
-
-          @native = nil
-        end
-      end
-    end
-  end
-end
-
-::Rdkafka::Bindings.attach_function(
-  :rd_kafka_flush,
-  %i[pointer int],
-  :void
-)
-
-Rdkafka::Producer::Client.prepend WaterDrop::Patches::Rdkafka::Client

data/lib/waterdrop/patches/rdkafka/metadata.rb

@@ -1,45 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  # Patches to external components
-  module Patches
-    # Rdkafka related patches
-    module Rdkafka
-      # Rdkafka::Metadata patches
-      module Metadata
-        # Errors upon which we retry the metadata fetch
-        RETRIED_ERRORS = %i[
-          timed_out
-          leader_not_available
-        ].freeze
-
-        private_constant :RETRIED_ERRORS
-
-        # We overwrite this method because there were reports of metadata operation timing out
-        # when Kafka was under stress. While the messages dispatch will be retried, metadata
-        # fetch happens prior to that, effectively crashing the process. Metadata fetch was not
-        # being retried at all.
-        #
-        # @param args [Array<Object>] all the metadata original arguments
-        def initialize(*args)
-          attempt ||= 0
-          attempt += 1
-
-          super(*args)
-        rescue ::Rdkafka::RdkafkaError => e
-          raise unless RETRIED_ERRORS.include?(e.code)
-          raise if attempt > 10
-
-          backoff_factor = 2**attempt
-          timeout = backoff_factor * 0.1
-
-          sleep(timeout)
-
-          retry
-        end
-      end
-    end
-  end
-end
-
-::Rdkafka::Metadata.prepend ::WaterDrop::Patches::Rdkafka::Metadata

data/lib/waterdrop/patches/rdkafka/producer.rb

@@ -1,86 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  # Patches to external components
-  module Patches
-    # Rdkafka related patches
-    module Rdkafka
-      # Rdkafka::Producer patches
-      module Producer
-        include ::Karafka::Core::Helpers::Time
-
-        # Cache partitions count for 30 seconds
-        PARTITIONS_COUNT_TTL = 30_000
-
-        private_constant :PARTITIONS_COUNT_TTL
-
-        # @param args [Object] arguments accepted by the original rdkafka producer
-        def initialize(*args)
-          super
-
-          @_partitions_count_cache = Concurrent::Hash.new do |cache, topic|
-            topic_metadata = ::Rdkafka::Metadata.new(inner_kafka, topic).topics&.first
-
-            cache[topic] = [
-              monotonic_now,
-              topic_metadata ? topic_metadata[:partition_count] : nil
-            ]
-          end
-        end
-
-        # Adds a method that allows us to get the native kafka producer name
-        #
-        # In between rdkafka versions, there are internal changes that force us to add some extra
-        # magic to support all the versions.
-        #
-        # @return [String] producer instance name
-        def name
-          @_name ||= ::Rdkafka::Bindings.rd_kafka_name(inner_kafka)
-        end
-
-        # This patch makes sure we cache the partition count for a given topic for given time
-        # This prevents us in case someone uses `partition_key` from querying for the count with
-        # each message. Instead we query once every 30 seconds at most
-        #
-        # @param topic [String] topic name
-        # @return [Integer] partition count for a given topic
-        def partition_count(topic)
-          closed_producer_check(__method__)
-
-          @_partitions_count_cache.delete_if do |_, cached|
-            monotonic_now - cached.first > PARTITIONS_COUNT_TTL
-          end
-
-          @_partitions_count_cache[topic].last
-        end
-
-        # @return [FFI::Pointer] pointer to the raw librdkafka
-        def inner_kafka
-          unless @_inner_kafka
-            version = ::Gem::Version.new(::Rdkafka::VERSION)
-
-            if version < ::Gem::Version.new('0.12.0')
-              @_inner_kafka = @native_kafka
-            elsif version < ::Gem::Version.new('0.13.0.beta.1')
-              @_inner_kafka = @client.native
-            else
-              @_inner_kafka = @native_kafka.inner
-            end
-          end
-
-          @_inner_kafka
-        end
-
-        # Closes our librdkafka instance with the flush patch
-        # @param timeout_ms [Integer] flush timeout
-        def close(timeout_ms = 5_000)
-          ObjectSpace.undefine_finalizer(self)
-
-          @client.close(nil, timeout_ms)
-        end
-      end
-    end
-  end
-end
-
-::Rdkafka::Producer.prepend ::WaterDrop::Patches::Rdkafka::Producer