waterdrop 2.7.0.rc1 → 2.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e715326d8ff0715491d71ec9f42c60eb049258a9b5eaf739a0b088cf515427dd
-  data.tar.gz: ed94552c972a77b5509907f59910d9ad37f033b718666d5520335bbb7f5d545f
+  metadata.gz: 5308262b20199b02906783387f294a58beb01fa8850db3db19bb7be39395121a
+  data.tar.gz: d35c18c4b7352c20c8eeb623f54581476f108cb656912a571ef067cc796e884c
 SHA512:
-  metadata.gz: 460c2e3f8989059293835af6f5a96c5c3af68a4f47608d98d22a217cfd15179edd8a834e8c621ff10ae894c5713878c833f23a6af47d83cfc215454326aabc37
-  data.tar.gz: 67d497d37e6be9593dbe14bfb0d4df7d8c80c18f176dda66987a372acaffa68f172ae415d9797b713af0c67f6127113bd49a6ebb7a778eec4f8b69b864bceea7
+  metadata.gz: ac6693e44080e4edf9b201a5e735b283bb7fa81d36ae10bf0d7501faa00e5f099917144966beb7febc4b95c50ed78feb7c659e59753a78e4495111e3d00af322
+  data.tar.gz: 100439b79cc59bd668f40e4fed8086c49f13bfedebb68409981d8c70a39c692eb2b4f453d9a45c367f2612578b2c6ac8303bd56acdbbd27d3a02d5aab803d57a

data/.ruby-version

@@ -1 +1 @@
-3.3.0
+3.3.1

data/CHANGELOG.md

@@ -1,6 +1,6 @@
 # WaterDrop changelog
 
-## 2.7.0 (Unreleased)
+## 2.7.0 (2024-04-26)
 
 This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
 
@@ -9,13 +9,34 @@ This release contains **BREAKING** changes. Make sure to read and apply upgrade
 - **[Breaking]** Change default timeouts so final delivery `message.timeout.ms` is less that `max_wait_time` so we do not end up with not final verdict.
 - **[Breaking]** Update all the time related configuration settings to be in `ms` and not mixed.
 - **[Breaking]** Remove no longer needed `wait_timeout` configuration option.
+- **[Breaking]** Do **not** validate or morph (via middleware) messages added to the buffer prior to `flush_sync` or `flush_async`.
 - [Enhancement] Provide `WaterDrop::Producer#transaction?` that returns only when producer has an active transaction running.
 - [Enhancement] Introduce `instrument_on_wait_queue_full` flag (defaults to `true`) to be able to configure whether non critical (retryable) queue full errors should be instrumented in the error pipeline. Useful when building high-performance pipes with WaterDrop queue retry backoff as a throttler.
+- [Enhancement] Protect critical `rdkafka` thread executable code sections.
+- [Enhancement] Treat the queue size as a gauge rather than a cumulative stat (isturdy).
+- [Fix] Fix a case where purge on non-initialized client would crash.
+- [Fix] Middlewares run twice when using buffered produce.
+- [Fix] Validations run twice when using buffered produce.
 
 ### Upgrade Notes
 
 **PLEASE MAKE SURE TO READ AND APPLY THEM!**
 
+#### `wait_timeout` Configuration No Longer Needed
+
+The `wait_timeout` WaterDrop configuration option is no longer needed. You can safely remove it.
+
+```ruby
+producer = WaterDrop::Producer.new
+
+producer.setup do |config|
+  # Other config...
+
+  # Remove this, no longer needed
+  config.wait_timeout = 30
+end
+```
+
 #### Time Settings Format Alignment
 
 **All** time-related values are now configured in milliseconds instead of some being in seconds and some in milliseconds.
@@ -23,7 +44,6 @@ This release contains **BREAKING** changes. Make sure to read and apply upgrade
 The values that were changed from seconds to milliseconds are:
 
 - `max_wait_timeout`
-- `wait_timeout`
 - `wait_backoff_on_queue_full`
 - `wait_timeout_on_queue_full`
 - `wait_backoff_on_transaction_command, default`
@@ -37,10 +57,10 @@ producer.setup do |config|
   config.deliver = true
 
   # Replace this:
-  config.wait_timeout = 30
+  config.max_wait_timeout = 30
 
   # With
-  config.wait_timeout = 30_000
+  config.max_wait_timeout = 30_000
   # ...
 end
 ```
@@ -82,6 +102,32 @@ Below, you can find a table with what has changed, the new defaults, and the cur
 
 This alignment ensures that when using sync operations or invoking `#wait`, any exception you get should give you a conclusive and final delivery verdict.
 
+#### Buffering No Longer Early Validates Messages
+
+As of version `2.7.0`, WaterDrop has changed how message buffering works. Previously, messages underwent validation and middleware processing when they were buffered. Now, these steps are deferred until just before dispatching the messages. The buffer functions strictly as a thread-safe storage area without performing any validations or middleware operations until the messages are ready to be sent.
+
+This adjustment was made primarily to ensure that middleware runs and validations are applied when most relevant—shortly before message dispatch. This approach addresses potential issues with buffers that might hold messages for extended periods:
+
+- **Temporal Relevance**: Validating and processing messages near their dispatch time helps ensure that actions such as partition assignments reflect the current system state. This is crucial in dynamic environments where system states are subject to rapid changes.
+
+- **Stale State Management**: By delaying validations and middleware to the dispatch phase, the system minimizes the risk of acting on outdated information, which could lead to incorrect processing or partitioning decisions.
+
+```ruby
+# Prior to 2.7.0 this would raise an error
+producer.buffer(topic: nil, payload: '')
+# => WaterDrop::Errors::MessageInvalidError
+
+# After 2.7.0 buffer will not, but flush_async will
+producer.buffer(topic: nil, payload: '')
+# => all good here
+producer.flush_async(topic: nil, payload: '')
+# => WaterDrop::Errors::MessageInvalidError
+```
+
+#### Middleware Execution Prior to Flush When Buffering
+
+The timing of middleware execution has been adjusted. Middleware, which was previously run when messages were added to the buffer, will now only execute immediately before the messages are flushed from the buffer and dispatched. This change is similar to the validation-related changes.
+
 ## 2.6.14 (2024-02-06)
 - [Enhancement] Instrument `producer.connected` and `producer.closing` lifecycle events.
 

data/Gemfile.lock

@@ -1,8 +1,8 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.7.0.rc1)
-      karafka-core (>= 2.4.0.rc1, < 3.0.0)
+    waterdrop (2.7.0)
+      karafka-core (>= 2.4.0, < 3.0.0)
       zeitwerk (~> 2.3)
 
 GEM
@@ -31,16 +31,16 @@ GEM
     ffi (1.16.3)
     i18n (1.14.4)
       concurrent-ruby (~> 1.0)
-    karafka-core (2.4.0.rc1)
-      karafka-rdkafka (>= 0.15.0.rc1, < 0.16.0)
-    karafka-rdkafka (0.15.0.rc1)
+    karafka-core (2.4.0)
+      karafka-rdkafka (>= 0.15.0, < 0.16.0)
+    karafka-rdkafka (0.15.0)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
       rake (> 12)
-    mini_portile2 (2.8.5)
+    mini_portile2 (2.8.6)
     minitest (5.22.3)
     mutex_m (0.2.0)
-    rake (13.1.0)
+    rake (13.2.1)
     rspec (3.13.0)
       rspec-core (~> 3.13.0)
       rspec-expectations (~> 3.13.0)
@@ -76,4 +76,4 @@ DEPENDENCIES
   waterdrop!
 
 BUNDLED WITH
-   2.5.7
+   2.5.9

data/docker-compose.yml

@@ -3,7 +3,7 @@ version: '2'
 services:
   kafka:
     container_name: kafka
-    image: confluentinc/cp-kafka:7.6.0
+    image: confluentinc/cp-kafka:7.6.1
 
     ports:
       - 9092:9092

data/lib/waterdrop/instrumentation/callbacks/delivery.rb

@@ -44,6 +44,17 @@ module WaterDrop
           else
             instrument_error(delivery_report)
           end
+        # This runs from the rdkafka thread, thus we want to safe-guard it and prevent absolute
+        # crashes even if the instrumentation code fails. If it would bubble-up, it could crash
+        # the rdkafka background thread
+        rescue StandardError => e
+          @monitor.instrument(
+            'error.occurred',
+            caller: self,
+            error: e,
+            producer_id: @producer_id,
+            type: 'callbacks.delivery.error'
+          )
         end
 
         private

data/lib/waterdrop/instrumentation/callbacks/error.rb

@@ -32,6 +32,17 @@ module WaterDrop
             producer_id: @producer_id,
             type: 'librdkafka.error'
           )
+        # This runs from the rdkafka thread, thus we want to safe-guard it and prevent absolute
+        # crashes even if the instrumentation code fails. If it would bubble-up, it could crash
+        # the rdkafka background thread
+        rescue StandardError => e
+          @monitor.instrument(
+            'error.occurred',
+            caller: self,
+            error: e,
+            producer_id: @producer_id,
+            type: 'callbacks.error.error'
+          )
         end
       end
     end

data/lib/waterdrop/instrumentation/callbacks/oauthbearer_token_refresh.rb

@@ -31,6 +31,17 @@ module WaterDrop
             bearer: @bearer,
             caller: self
           )
+        # This runs from the rdkafka thread, thus we want to safe-guard it and prevent absolute
+        # crashes even if the instrumentation code fails. If it would bubble-up, it could crash
+        # the rdkafka background thread
+        rescue StandardError => e
+          @monitor.instrument(
+            'error.occurred',
+            caller: self,
+            error: e,
+            producer_id: @producer_id,
+            type: 'callbacks.oauthbearer_token_refresh.error'
+          )
         end
       end
     end

data/lib/waterdrop/instrumentation/callbacks/statistics.rb

@@ -34,6 +34,17 @@ module WaterDrop
             producer_id: @producer_id,
             statistics: @statistics_decorator.call(statistics)
           )
+        # This runs from the rdkafka thread, thus we want to safe-guard it and prevent absolute
+        # crashes even if the instrumentation code fails. If it would bubble-up, it could crash
+        # the rdkafka background thread
+        rescue StandardError => e
+          @monitor.instrument(
+            'error.occurred',
+            caller: self,
+            error: e,
+            producer_id: @producer_id,
+            type: 'callbacks.statistics.error'
+          )
         end
       end
     end

data/lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb

@@ -36,7 +36,7 @@ module WaterDrop
           setting :rd_kafka_metrics, default: [
             # Client metrics
             RdKafkaMetric.new(:count, :root, 'calls', 'tx_d'),
-            RdKafkaMetric.new(:histogram, :root, 'queue.size', 'msg_cnt_d'),
+            RdKafkaMetric.new(:histogram, :root, 'queue.size', 'msg_cnt'),
 
             # Broker metrics
             RdKafkaMetric.new(:count, :brokers, 'deliver.attempts', 'txretries_d'),

data/lib/waterdrop/producer/buffer.rb

@@ -12,9 +12,6 @@ module WaterDrop
       def buffer(message)
         ensure_active!
 
-        message = middleware.run(message)
-        validate_message!(message)
-
         @monitor.instrument(
           'message.buffered',
           producer_id: id,
@@ -32,9 +29,6 @@ module WaterDrop
       def buffer_many(messages)
         ensure_active!
 
-        messages = middleware.run_many(messages)
-        messages.each { |message| validate_message!(message) }
-
         @monitor.instrument(
           'messages.buffered',
           producer_id: id,

data/lib/waterdrop/producer.rb

@@ -133,7 +133,12 @@ module WaterDrop
           @messages = []
         end
 
-        @client.purge
+        # We should not purge if there is no client initialized
+        # It may not be initialized if we created a new producer that never connected to kafka,
+        # we used buffer and purged. In cases like this client won't exist
+        @connecting_mutex.synchronize do
+          @client&.purge
+        end
       end
     end
 

data/lib/waterdrop/version.rb

@@ -3,5 +3,5 @@
 # WaterDrop library
 module WaterDrop
   # Current WaterDrop version
-  VERSION = '2.7.0.rc1'
+  VERSION = '2.7.0'
 end

data/waterdrop.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |spec|
   spec.description   = spec.summary
   spec.license       = 'MIT'
 
-  spec.add_dependency 'karafka-core', '>= 2.4.0.rc1', '< 3.0.0'
+  spec.add_dependency 'karafka-core', '>= 2.4.0', '< 3.0.0'
   spec.add_dependency 'zeitwerk', '~> 2.3'
 
   spec.required_ruby_version = '>= 3.0.0'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.7.0.rc1
+  version: 2.7.0
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   AnG1dJU+yL2BK7vaVytLTstJME5mepSZ46qqIJXMuWob/YPDmVaBF39TDSG9e34s
   msG3BiCqgOgHAnL23+CN3Rt8MsuRfEtoTKpJVcCfoEoNHOkc
   -----END CERTIFICATE-----
-date: 2024-04-10 00:00:00.000000000 Z
+date: 2024-04-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: karafka-core
@@ -43,7 +43,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.4.0.rc1
+        version: 2.4.0
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -53,7 +53,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.4.0.rc1
+        version: 2.4.0
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -152,7 +152,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.3
+rubygems_version: 3.5.9
 signing_key: 
 specification_version: 4
 summary: Kafka messaging made easy!