waterdrop 2.3.2 → 2.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 154b5f8a103fcfaa47c5d410583d1aa2c0b689b52cd284614b06e7f7a9460179
-  data.tar.gz: c01ae313dbcf2830267438d3dd19f269a0a7bf29cad824af95f4313f6f04368e
+  metadata.gz: 70b84fa09ce807d6d72679917d6c9b98a4dd15b9eb50f0ae9c556acc6f084632
+  data.tar.gz: 8ccffe9e5331865dc93ab4c8b4b8ace2e0c3297f164c378b7919a14e307edf70
 SHA512:
-  metadata.gz: 33e5ab153d45e265dd6cdb2b3ab9bb63bda7c8c4399097f1c5c130d4c493de635ee7fad957c83b97d6e5fdd6c4c049a0419ca5e0e063f8163478182cde6b92ed
-  data.tar.gz: 3858e9ac8d2f860f888a5445ec8c59d98d35dce199816058da850bfaaf80308ccd40cff2eb5e872eb3ba111259c4c4d07d0a0219e0a641da0ad0c456d184030a
+  metadata.gz: 66f25845bada1a7de0f6bd100b7966f246604701f0394891bc959817f222b739a73fc674b49d288ebf3674c85e9872db37069771f347fad98eeade561373610c
+  data.tar.gz: 308eafaec616bf936517086de4bcee0efa03ed555ad91a061588b781d9214198fa67258a0d7044eae523fbba4da57fa447ddafdcf3169a234cdc4b0785151c6e

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # WaterDrop changelog
 
+## 2.4.1 (2022-08-01)
+- Replace local statistics decorator with the one extracted to `karafka-core`.
+
+## 2.4.0 (2022-07-28)
+- Small refactor of the DataDog/Statsd listener to align for future extraction to `karafka-common`.
+- Replace `dry-monitor` with home-brew notification layer (API compatible) and allow for usage with `ActiveSupport::Notifications`.
+- Remove all the common code into `karafka-core` and add it as a dependency.
+
+## 2.3.3 (2022-07-18)
+- Replace `dry-validation` with home-brew validation layer and drop direct dependency on `dry-validation`.
+- Remove indirect dependency on dry-configurable from DataDog listener (no changes required).
+
 ## 2.3.2 (2022-07-17)
 - Replace `dry-configurable` with home-brew config and drop direct dependency on `dry-configurable`.
 

data/Gemfile.lock

@@ -1,17 +1,15 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.3.2)
-      concurrent-ruby (>= 1.1)
-      dry-monitor (~> 0.5)
-      dry-validation (~> 1.7)
+    waterdrop (2.4.1)
+      karafka-core (>= 2.0.2, < 3.0.0)
       rdkafka (>= 0.10)
       zeitwerk (~> 2.3)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (7.0.3)
+    activesupport (7.0.3.1)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
@@ -20,52 +18,15 @@ GEM
     concurrent-ruby (1.1.10)
     diff-lcs (1.5.0)
     docile (1.4.0)
-    dry-configurable (0.15.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 0.6)
-    dry-container (0.10.0)
-      concurrent-ruby (~> 1.0)
-    dry-core (0.8.0)
-      concurrent-ruby (~> 1.0)
-    dry-events (0.3.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 0.5, >= 0.5)
-    dry-inflector (0.3.0)
-    dry-initializer (3.1.1)
-    dry-logic (1.2.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 0.5, >= 0.5)
-    dry-monitor (0.6.1)
-      dry-configurable (~> 0.13, >= 0.13.0)
-      dry-core (~> 0.5, >= 0.5)
-      dry-events (~> 0.2)
-      zeitwerk (~> 2.5)
-    dry-schema (1.9.3)
-      concurrent-ruby (~> 1.0)
-      dry-configurable (~> 0.13, >= 0.13.0)
-      dry-core (~> 0.5, >= 0.5)
-      dry-initializer (~> 3.0)
-      dry-logic (~> 1.0)
-      dry-types (~> 1.5)
-    dry-types (1.5.1)
-      concurrent-ruby (~> 1.0)
-      dry-container (~> 0.3)
-      dry-core (~> 0.5, >= 0.5)
-      dry-inflector (~> 0.1, >= 0.1.2)
-      dry-logic (~> 1.0, >= 1.0.2)
-    dry-validation (1.8.1)
-      concurrent-ruby (~> 1.0)
-      dry-container (~> 0.7, >= 0.7.1)
-      dry-core (~> 0.5, >= 0.5)
-      dry-initializer (~> 3.0)
-      dry-schema (~> 1.8, >= 1.8.0)
     factory_bot (6.2.1)
       activesupport (>= 5.0.0)
     ffi (1.15.5)
-    i18n (1.10.0)
+    i18n (1.12.0)
       concurrent-ruby (~> 1.0)
+    karafka-core (2.0.2)
+      concurrent-ruby (>= 1.1)
     mini_portile2 (2.8.0)
-    minitest (5.16.0)
+    minitest (5.16.2)
     rake (13.0.6)
     rdkafka (0.12.0)
       ffi (~> 1.15)
@@ -90,12 +51,11 @@ GEM
       simplecov_json_formatter (~> 0.1)
     simplecov-html (0.12.3)
     simplecov_json_formatter (0.1.4)
-    tzinfo (2.0.4)
+    tzinfo (2.0.5)
       concurrent-ruby (~> 1.0)
     zeitwerk (2.6.0)
 
 PLATFORMS
-  arm64-darwin-21
   x86_64-linux
 
 DEPENDENCIES

data/README.md

@@ -2,7 +2,7 @@
 
 **Note**: Documentation presented here refers to WaterDrop `2.x`.
 
-WaterDrop `2.x` does **not** work with Karafka `1.*` and aims to either work as a standalone producer outside of Karafka `1.*` ecosystem or as a part of soon to be released Karafka `2.0.*`.
+WaterDrop `2.x` works with Karafka `2.*` and aims to either work as a standalone producer or as a part of the Karafka `2.*`.
 
 Please refer to [this](https://github.com/karafka/waterdrop/tree/1.4) branch and its documentation for details about WaterDrop `1.*` usage.
 
@@ -10,17 +10,17 @@ Please refer to [this](https://github.com/karafka/waterdrop/tree/1.4) branch and
 [![Gem Version](https://badge.fury.io/rb/waterdrop.svg)](http://badge.fury.io/rb/waterdrop)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
-Gem used to send messages to Kafka in an easy way with an extra validation layer. It is a part of the [Karafka](https://github.com/karafka/karafka) ecosystem.
+A gem to send messages to Kafka easily with an extra validation layer. It is a part of the [Karafka](https://github.com/karafka/karafka) ecosystem.
 
 It:
 
- - Is thread safe
+ - Is thread-safe
  - Supports sync producing
  - Supports async producing
  - Supports buffering
  - Supports producing messages to multiple clusters
  - Supports multiple delivery policies
- - Works with Kafka 1.0+ and Ruby 2.6+
+ - Works with Kafka `1.0+` and Ruby `2.7+`
 
 ## Table of contents
 
@@ -30,6 +30,8 @@ It:
   * [Kafka configuration options](#kafka-configuration-options)
 - [Usage](#usage)
   * [Basic usage](#basic-usage)
+  * [Using WaterDrop across the application and with Ruby on Rails](#using-waterdrop-across-the-application-and-with-ruby-on-rails)
+  * [Using WaterDrop with a connection-pool](#using-waterdrop-with-a-connection-pool)
   * [Buffering](#buffering)
       + [Using WaterDrop to buffer messages based on the application logic](#using-waterdrop-to-buffer-messages-based-on-the-application-logic)
       + [Using WaterDrop with rdkafka buffers to achieve periodic auto-flushing](#using-waterdrop-with-rdkafka-buffers-to-achieve-periodic-auto-flushing)
@@ -42,6 +44,8 @@ It:
 
 ## Installation
 
+**Note**: If you want to both produce and consume messages, please use [Karafka](https://github.com/karafka/karafka/). It integrates WaterDrop automatically.
+
 Add this to your Gemfile:
 
 ```ruby
@@ -56,7 +60,7 @@ bundle install
 
 ## Setup
 
-WaterDrop is a complex tool, that contains multiple configuration options. To keep everything organized, all the configuration options were divided into two groups:
+WaterDrop is a complex tool that contains multiple configuration options. To keep everything organized, all the configuration options were divided into two groups:
 
 - WaterDrop options - options directly related to WaterDrop and its components
 - Kafka driver options - options related to `rdkafka`
@@ -103,8 +107,6 @@ You can create producers with different `kafka` settings. Documentation of the a
 
 ## Usage
 
-Please refer to the [documentation](https://www.rubydoc.info/gems/waterdrop) in case you're interested in the more advanced API.
-
 ### Basic usage
 
 To send Kafka messages, just create a producer and use it:
@@ -157,6 +159,41 @@ Here are all the things you can provide in the message hash:
 
 Keep in mind, that message you want to send should be either binary or stringified (to_s, to_json, etc).
 
+### Using WaterDrop across the application and with Ruby on Rails
+
+If you plan to both produce and consume messages using Kafka, you should install and use [Karafka](https://github.com/karafka/karafka). It integrates automatically with Ruby on Rails applications and auto-configures WaterDrop producer to make it accessible via `Karafka#producer` method.
+
+If you want to only produce messages from within your application, since WaterDrop is thread-safe you can create a single instance in an initializer like so:
+
+```ruby
+KAFKA_PRODUCER = WaterDrop::Producer.new
+
+KAFKA_PRODUCER.setup do |config|
+  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
+end
+
+# And just dispatch messages
+KAFKA_PRODUCER.produce_sync(topic: 'my-topic', payload: 'my message')
+```
+
+### Using WaterDrop with a connection-pool
+
+While WaterDrop is thread-safe, there is no problem in using it with a connection pool inside high-intensity applications. The only thing worth keeping in mind, is that WaterDrop instances should be shutdown before the application is closed.
+
+```ruby
+KAFKA_PRODUCERS_CP = ConnectionPool.new do
+  WaterDrop::Producer.new do |config|
+    config.kafka = { 'bootstrap.servers': 'localhost:9092' }
+  end
+end
+
+KAFKA_PRODUCERS_CP.with do |producer|
+  producer.produce_async(topic: 'my-topic', payload: 'my message')
+end
+
+KAFKA_PRODUCERS_CP.shutdown { |producer| producer.close }
+```
+
 ### Buffering
 
 WaterDrop producers support buffering messages in their internal buffers and on the `rdkafka` level via `queue.buffering.*` set of settings.
@@ -316,6 +353,8 @@ producer.monitor.subscribe(listener)
 
 You can also find [here](https://github.com/karafka/waterdrop/blob/master/lib/waterdrop/instrumentation/vendors/datadog/dashboard.json) a ready to import DataDog dashboard configuration file that you can use to monitor all of your producers.
 
+![Example WaterDrop DD dashboard](https://raw.githubusercontent.com/karafka/misc/master/printscreens/waterdrop_dd_dashboard_example.png)
+
 ### Error notifications
 
 WaterDrop allows you to listen to all errors that occur while producing messages and in its internal background threads. Things like reconnecting to Kafka upon network errors and others unrelated to publishing messages are all available under `error.occurred` notification key. You can subscribe to this event to ensure your setup is healthy and without any problems that would otherwise go unnoticed as long as messages are delivered.

data/config/errors.yml

@@ -1,7 +1,30 @@
 en:
-  dry_validation:
-    errors:
-      invalid_key_type: all keys need to be of type String
-      invalid_value_type: all values need to be of type String
-      max_payload_size: is more than `max_payload_size` config value
+  validations:
+    config:
+      missing: must be present
+      logger_format: must be present
+      deliver_format: must be boolean
+      id_format: must be a non-empty string
+      max_payload_size_format: must be an integer that is equal or bigger than 1
+      wait_timeout_format: must be a numeric that is bigger than 0
+      max_wait_timeout_format: must be an integer that is equal or bigger than 0
+      kafka_format: must be a hash with symbol based keys
       kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
+
+    message:
+      missing: must be present
+      partition_format: must be an integer greater or equal to -1
+      topic_format: 'does not match the topic allowed format'
+      partition_key_format: must be a non-empty string
+      timestamp_format: must be either time or integer
+      payload_format: must be string
+      headers_format: must be a hash
+      key_format: must be a non-empty string
+      payload_max_size: is more than `max_payload_size` config value
+      headers_invalid_key_type: all headers keys need to be of type String
+      headers_invalid_value_type: all headers values need to be of type String
+
+    test:
+      missing: must be present
+      nested.id_format: 'is invalid'
+      nested.id2_format: 'is invalid'

data/lib/waterdrop/config.rb

@@ -5,7 +5,7 @@
 module WaterDrop
   # Configuration object for setting up all options required by WaterDrop
   class Config
-    include Configurable
+    include ::Karafka::Core::Configurable
 
     # Defaults for kafka settings, that will be overwritten only if not present already
     KAFKA_DEFAULTS = {
@@ -77,7 +77,7 @@ module WaterDrop
     # Propagates the kafka setting defaults unless they are already present
     # This makes it easier to set some values that users usually don't change but still allows them
     # to overwrite the whole hash if they want to
-    # @param config [Dry::Configurable::Config] dry config of this producer
+    # @param config [WaterDrop::Configurable::Node] config of this producer
     def merge_kafka_defaults!(config)
       KAFKA_DEFAULTS.each do |key, value|
         next if config.kafka.key?(key)

data/lib/waterdrop/contracts/config.rb

@@ -3,27 +3,37 @@
 module WaterDrop
   module Contracts
     # Contract with validation rules for WaterDrop configuration details
-    class Config < Base
-      params do
-        required(:id).filled(:str?)
-        required(:logger).filled
-        required(:deliver).filled(:bool?)
-        required(:max_payload_size).filled(:int?, gteq?: 1)
-        required(:max_wait_timeout).filled(:number?, gteq?: 0)
-        required(:wait_timeout).filled(:number?, gt?: 0)
-        required(:kafka).filled(:hash?)
+    class Config < ::Karafka::Core::Contractable::Contract
+      configure do |config|
+        config.error_messages = YAML.safe_load(
+          File.read(
+            File.join(WaterDrop.gem_root, 'config', 'errors.yml')
+          )
+        ).fetch('en').fetch('validations').fetch('config')
       end
 
+      required(:id) { |val| val.is_a?(String) && !val.empty? }
+      required(:logger) { |val| !val.nil? }
+      required(:deliver) { |val| [true, false].include?(val) }
+      required(:max_payload_size) { |val| val.is_a?(Integer) && val >= 1 }
+      required(:max_wait_timeout) { |val| val.is_a?(Numeric) && val >= 0 }
+      required(:wait_timeout) { |val| val.is_a?(Numeric) && val.positive? }
+      required(:kafka) { |val| val.is_a?(Hash) && !val.empty? }
+
       # rdkafka allows both symbols and strings as keys for config but then casts them to strings
       # This can be confusing, so we expect all keys to be symbolized
-      rule(:kafka) do
-        next unless value.is_a?(Hash)
+      virtual do |config, errors|
+        next true unless errors.empty?
+
+        errors = []
 
-        value.each_key do |key|
-          next if key.is_a?(Symbol)
+        config
+          .fetch(:kafka)
+          .keys
+          .reject { |key| key.is_a?(Symbol) }
+          .each { |key| errors << [[:kafka, key], :kafka_key_must_be_a_symbol] }
 
-          key(:"kafka.#{key}").failure(:kafka_key_must_be_a_symbol)
-        end
+        errors
       end
     end
   end

data/lib/waterdrop/contracts/message.rb

@@ -4,36 +4,56 @@ module WaterDrop
   module Contracts
     # Contract with validation rules for validating that all the message options that
     # we provide to producer ale valid and usable
-    class Message < Base
+    class Message < ::Karafka::Core::Contractable::Contract
+      configure do |config|
+        config.error_messages = YAML.safe_load(
+          File.read(
+            File.join(WaterDrop.gem_root, 'config', 'errors.yml')
+          )
+        ).fetch('en').fetch('validations').fetch('message')
+      end
+
       # Regex to check that topic has a valid format
       TOPIC_REGEXP = /\A(\w|-|\.)+\z/
 
-      # Checks, that the given value is a string
-      STRING_ASSERTION = ->(value) { value.is_a?(String) }.to_proc
+      private_constant :TOPIC_REGEXP
 
-      private_constant :TOPIC_REGEXP, :STRING_ASSERTION
+      attr_reader :max_payload_size
 
-      option :max_payload_size
-
-      params do
-        required(:topic).filled(:str?, format?: TOPIC_REGEXP)
-        required(:payload).filled(:str?)
-        optional(:key).maybe(:str?, :filled?)
-        optional(:partition).filled(:int?, gteq?: -1)
-        optional(:partition_key).maybe(:str?, :filled?)
-        optional(:timestamp).maybe { time? | int? }
-        optional(:headers).maybe(:hash?)
+      # @param max_payload_size [Integer] max payload size
+      def initialize(max_payload_size:)
+        super()
+        @max_payload_size = max_payload_size
       end
 
-      rule(:headers) do
-        next unless value.is_a?(Hash)
+      required(:topic) { |val| val.is_a?(String) && TOPIC_REGEXP.match?(val) }
+      required(:payload) { |val| val.is_a?(String) }
+      optional(:key) { |val| val.nil? || (val.is_a?(String) && !val.empty?) }
+      optional(:partition) { |val| val.is_a?(Integer) && val >= -1 }
+      optional(:partition_key) { |val| val.nil? || (val.is_a?(String) && !val.empty?) }
+      optional(:timestamp) { |val| val.nil? || (val.is_a?(Time) || val.is_a?(Integer)) }
+      optional(:headers) { |val| val.nil? || val.is_a?(Hash) }
+
+      virtual do |config, errors|
+        next true unless errors.empty?
+        next true unless config.key?(:headers)
+        next true if config[:headers].nil?
 
-        key.failure(:invalid_key_type) unless value.keys.all?(&STRING_ASSERTION)
-        key.failure(:invalid_value_type) unless value.values.all?(&STRING_ASSERTION)
+        errors = []
+
+        config.fetch(:headers).each do |key, value|
+          errors << [%i[headers], :invalid_key_type] unless key.is_a?(String)
+          errors << [%i[headers], :invalid_value_type] unless value.is_a?(String)
+        end
+
+        errors
       end
 
-      rule(:payload) do
-        key.failure(:max_payload_size) if value.bytesize > max_payload_size
+      virtual do |config, errors, validator|
+        next true unless errors.empty?
+        next true if config[:payload].bytesize <= validator.max_payload_size
+
+        [[%i[payload], :max_size]]
       end
     end
   end

data/lib/waterdrop/instrumentation/callbacks/statistics.rb

@@ -17,7 +17,7 @@ module WaterDrop
           @producer_id = producer_id
           @client_name = client_name
           @monitor = monitor
-          @statistics_decorator = StatisticsDecorator.new
+          @statistics_decorator = ::Karafka::Core::Monitoring::StatisticsDecorator.new
         end
 
         # Emits decorated statistics to the monitor

data/lib/waterdrop/instrumentation/monitor.rb

@@ -2,41 +2,18 @@
 
 module WaterDrop
   module Instrumentation
-    # Monitor is used to hookup external monitoring services to monitor how WaterDrop works
-    # Since it is a pub-sub based on dry-monitor, you can use as many subscribers/loggers at the
-    # same time, which means that you might have for example file logging and NewRelic at the same
-    # time
-    # @note This class acts as a singleton because we are only permitted to have single monitor
-    #   per running process (just as logger)
-    class Monitor < Dry::Monitor::Notifications
-      # List of events that we support in the system and to which a monitor client can hook up
-      # @note The non-error once support timestamp benchmarking
-      EVENTS = %w[
-        producer.closed
-
-        message.produced_async
-        message.produced_sync
-        message.acknowledged
-        message.buffered
-
-        messages.produced_async
-        messages.produced_sync
-        messages.buffered
-
-        buffer.flushed_async
-        buffer.flushed_sync
-
-        statistics.emitted
-
-        error.occurred
-      ].freeze
-
-      private_constant :EVENTS
-
-      # @return [WaterDrop::Instrumentation::Monitor] monitor instance for system instrumentation
-      def initialize
-        super(:waterdrop)
-        EVENTS.each(&method(:register_event))
+    # WaterDrop instrumentation monitor that we use to publish events
+    # By default uses our internal notifications bus but can be used with
+    # `ActiveSupport::Notifications` as well
+    class Monitor < ::Karafka::Core::Monitoring::Monitor
+      # @param notifications_bus [Object] either our internal notifications bus or
+      #   `ActiveSupport::Notifications`
+      # @param namespace [String, nil] namespace for events or nil if no namespace
+      def initialize(
+        notifications_bus = WaterDrop::Instrumentation::Notifications.new,
+        namespace = nil
+      )
+        super(notifications_bus, namespace)
       end
     end
   end

data/lib/waterdrop/instrumentation/notifications.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module WaterDrop
+  module Instrumentation
+    # Instrumented is used to hookup external monitoring services to monitor how WaterDrop works
+    class Notifications < ::Karafka::Core::Monitoring::Notifications
+      # List of events that we support in the system and to which a monitor client can hook up
+      # @note The non-error once support timestamp benchmarking
+      EVENTS = %w[
+        producer.closed
+
+        message.produced_async
+        message.produced_sync
+        message.acknowledged
+        message.buffered
+
+        messages.produced_async
+        messages.produced_sync
+        messages.buffered
+
+        buffer.flushed_async
+        buffer.flushed_sync
+
+        statistics.emitted
+
+        error.occurred
+      ].freeze
+
+      private_constant :EVENTS
+
+      # @return [WaterDrop::Instrumentation::Monitor] monitor instance for system instrumentation
+      def initialize
+        super
+        EVENTS.each { |event| register_event(event) }
+      end
+    end
+  end
+end

data/lib/waterdrop/instrumentation/vendors/datadog/listener.rb

@@ -11,26 +11,29 @@ module WaterDrop
         #
         # @note You need to setup the `dogstatsd-ruby` client and assign it
         class Listener
-          include Dry::Configurable
+          include ::Karafka::Core::Configurable
+          extend Forwardable
+
+          def_delegators :config, :client, :rd_kafka_metrics, :namespace, :default_tags
 
           # Value object for storing a single rdkafka metric publishing details
           RdKafkaMetric = Struct.new(:type, :scope, :name, :key_location)
 
           # Namespace under which the DD metrics should be published
-          setting :namespace, default: 'waterdrop', reader: true
+          setting :namespace, default: 'waterdrop'
 
           # Datadog client that we should use to publish the metrics
-          setting :client, reader: true
+          setting :client
 
           # Default tags we want to publish (for example hostname)
           # Format as followed (example for hostname): `["host:#{Socket.gethostname}"]`
-          setting :default_tags, default: [], reader: true
+          setting :default_tags, default: []
 
           # All the rdkafka metrics we want to publish
           #
           # By default we publish quite a lot so this can be tuned
           # Note, that the once with `_d` come from WaterDrop, not rdkafka or Kafka
-          setting :rd_kafka_metrics, reader: true, default: [
+          setting :rd_kafka_metrics, default: [
             # Client metrics
             RdKafkaMetric.new(:count, :root, 'calls', 'tx_d'),
             RdKafkaMetric.new(:histogram, :root, 'queue.size', 'msg_cnt_d'),
@@ -47,8 +50,11 @@ module WaterDrop
             RdKafkaMetric.new(:gauge, :brokers, 'network.latency.p99', %w[rtt p99])
           ].freeze
 
+          configure
+
           # @param block [Proc] configuration block
           def initialize(&block)
+            configure
             setup(&block) if block
           end
 
@@ -60,7 +66,7 @@ module WaterDrop
 
           # Hooks up to WaterDrop instrumentation for emitted statistics
           #
-          # @param event [Dry::Events::Event]
+          # @param event [WaterDrop::Monitor::Event]
           def on_statistics_emitted(event)
             statistics = event[:statistics]
 
@@ -71,22 +77,15 @@ module WaterDrop
 
           # Increases the errors count by 1
           #
-          # @param _event [Dry::Events::Event]
+          # @param _event [WaterDrop::Monitor::Event]
           def on_error_occurred(_event)
-            client.count(
-              namespaced_metric('error_occurred'),
-              1,
-              tags: default_tags
-            )
+            count('error_occurred', 1, tags: default_tags)
           end
 
           # Increases acknowledged messages counter
-          # @param _event [Dry::Events::Event]
+          # @param _event [WaterDrop::Monitor::Event]
           def on_message_acknowledged(_event)
-            client.increment(
-              namespaced_metric('acknowledged'),
-              tags: default_tags
-            )
+            increment('acknowledged', tags: default_tags)
           end
 
           %i[
@@ -94,12 +93,12 @@ module WaterDrop
             produced_async
           ].each do |event_scope|
             class_eval <<~METHODS, __FILE__, __LINE__ + 1
-              # @param event [Dry::Events::Event]
+              # @param event [WaterDrop::Monitor::Event]
               def on_message_#{event_scope}(event)
                 report_message(event[:message][:topic], :#{event_scope})
               end
 
-              # @param event [Dry::Events::Event]
+              # @param event [WaterDrop::Monitor::Event]
               def on_messages_#{event_scope}(event)
                 event[:messages].each do |message|
                   report_message(message[:topic], :#{event_scope})
@@ -114,10 +113,10 @@ module WaterDrop
             messages_buffered
           ].each do |event_scope|
             class_eval <<~METHODS, __FILE__, __LINE__ + 1
-              # @param event [Dry::Events::Event]
+              # @param event [WaterDrop::Monitor::Event]
               def on_#{event_scope}(event)
-                client.histogram(
-                  namespaced_metric('buffer.size'),
+                histogram(
+                  'buffer.size',
                   event[:buffer].size,
                   tags: default_tags
                 )
@@ -132,7 +131,7 @@ module WaterDrop
             flushed_async
           ].each do |event_scope|
             class_eval <<~METHODS, __FILE__, __LINE__ + 1
-              # @param event [Dry::Events::Event]
+              # @param event [WaterDrop::Monitor::Event]
               def on_buffer_#{event_scope}(event)
                 event[:messages].each do |message|
                   report_message(message[:topic], :#{event_scope})
@@ -143,14 +142,28 @@ module WaterDrop
 
           private
 
+          %i[
+            count
+            gauge
+            histogram
+            increment
+            decrement
+          ].each do |metric_type|
+            class_eval <<~METHODS, __FILE__, __LINE__ + 1
+              def #{metric_type}(key, *args)
+                client.#{metric_type}(
+                  namespaced_metric(key),
+                  *args
+                )
+              end
+            METHODS
+          end
+
           # Report that a message has been produced to a topic.
           # @param topic [String] Kafka topic
           # @param method_name [Symbol] method from which this message operation comes
           def report_message(topic, method_name)
-            client.increment(
-              namespaced_metric(method_name),
-              tags: default_tags + ["topic:#{topic}"]
-            )
+            increment(method_name, tags: default_tags + ["topic:#{topic}"])
           end
 
           # Wraps metric name in listener's namespace
@@ -166,9 +179,9 @@ module WaterDrop
           def report_metric(metric, statistics)
             case metric.scope
             when :root
-              client.public_send(
+              public_send(
                 metric.type,
-                namespaced_metric(metric.name),
+                metric.name,
                 statistics.fetch(*metric.key_location),
                 tags: default_tags
               )
@@ -179,9 +192,9 @@ module WaterDrop
                 # node ids
                 next if broker_statistics['nodeid'] == -1
 
-                client.public_send(
+                public_send(
                   metric.type,
-                  namespaced_metric(metric.name),
+                  metric.name,
                   broker_statistics.dig(*metric.key_location),
                   tags: default_tags + ["broker:#{broker_statistics['nodename']}"]
                 )

data/lib/waterdrop/version.rb

@@ -3,5 +3,5 @@
 # WaterDrop library
 module WaterDrop
   # Current WaterDrop version
-  VERSION = '2.3.2'
+  VERSION = '2.4.1'
 end

data/lib/waterdrop.rb

@@ -3,9 +3,8 @@
 # External components
 # delegate should be removed because we don't need it, we just add it because of ruby-kafka
 %w[
-  concurrent/array
-  dry/monitor/notifications
-  dry-validation
+  karafka-core
+  forwardable
   rdkafka
   json
   zeitwerk

data/waterdrop.gemspec

@@ -16,9 +16,7 @@ Gem::Specification.new do |spec|
   spec.description   = spec.summary
   spec.license       = 'MIT'
 
-  spec.add_dependency 'concurrent-ruby', '>= 1.1'
-  spec.add_dependency 'dry-monitor', '~> 0.5'
-  spec.add_dependency 'dry-validation', '~> 1.7'
+  spec.add_dependency 'karafka-core', '>= 2.0.2', '< 3.0.0'
   spec.add_dependency 'rdkafka', '>= 0.10'
   spec.add_dependency 'zeitwerk', '~> 2.3'
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.3.2
+  version: 2.4.1
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -34,50 +34,28 @@ cert_chain:
   R2P11bWoCtr70BsccVrN8jEhzwXngMyI2gVt750Y+dbTu1KgRqZKp/ECe7ZzPzXj
   pIy9vHxTANKYVyI4qj8OrFdEM5BQNu8oQpL0iQ==
   -----END CERTIFICATE-----
-date: 2022-07-17 00:00:00.000000000 Z
+date: 2022-08-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: concurrent-ruby
+  name: karafka-core
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '1.1'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
+        version: 2.0.2
+    - - "<"
       - !ruby/object:Gem::Version
-        version: '1.1'
-- !ruby/object:Gem::Dependency
-  name: dry-monitor
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.5'
+        version: 3.0.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.5'
-- !ruby/object:Gem::Dependency
-  name: dry-validation
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.7'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
+        version: 2.0.2
+    - - "<"
       - !ruby/object:Gem::Version
-        version: '1.7'
+        version: 3.0.0
 - !ruby/object:Gem::Dependency
   name: rdkafka
   requirement: !ruby/object:Gem::Requirement
@@ -130,11 +108,7 @@ files:
 - docker-compose.yml
 - lib/waterdrop.rb
 - lib/waterdrop/config.rb
-- lib/waterdrop/configurable.rb
-- lib/waterdrop/configurable/leaf.rb
-- lib/waterdrop/configurable/node.rb
 - lib/waterdrop/contracts.rb
-- lib/waterdrop/contracts/base.rb
 - lib/waterdrop/contracts/config.rb
 - lib/waterdrop/contracts/message.rb
 - lib/waterdrop/errors.rb
@@ -142,10 +116,10 @@ files:
 - lib/waterdrop/instrumentation/callbacks/delivery.rb
 - lib/waterdrop/instrumentation/callbacks/error.rb
 - lib/waterdrop/instrumentation/callbacks/statistics.rb
-- lib/waterdrop/instrumentation/callbacks/statistics_decorator.rb
 - lib/waterdrop/instrumentation/callbacks_manager.rb
 - lib/waterdrop/instrumentation/logger_listener.rb
 - lib/waterdrop/instrumentation/monitor.rb
+- lib/waterdrop/instrumentation/notifications.rb
 - lib/waterdrop/instrumentation/vendors/datadog/dashboard.json
 - lib/waterdrop/instrumentation/vendors/datadog/listener.rb
 - lib/waterdrop/patches/rdkafka/bindings.rb

metadata.gz.sig

@@ -1 +1,4 @@
--���m�E�ΕQQiPq�ǶKP k�]�N|�<�b^@�{�@�i�[��B��7މ_˘d�v;G�8%��Htic�D�U+
+A��ˣ�v=�o�*>�.C{Ys�bA,��aB��?#
Ϻ��8\����wC�Ƈ��]%�S��Om�ȏY�u�v�Rm�l�a/����L��ӥ�J�����&����Ŷ{��a�&&Be�>b����}�}H�iݞL��h�����IE��
+%�Z#�i��ޭ��P��%���A��%D"��۩u0lJ)��Yͦ��
+}"�"�9�� B�.��?��)�)yLY3{q��7�x��-�E��fE�xmg�c��b�7Ԗ&��kt�Dx�7f����AY�X�\R����#L�_�
+�*8/I���ȖskMo��?�_���A��\�V��&*>���ؼ�ң����_=�ry*�:�D�CD��\�������[0���I	�6

data/lib/waterdrop/configurable/leaf.rb

@@ -1,8 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  module Configurable
-    # Single end config value representation
-    Leaf = Struct.new(:name, :default, :constructor)
-  end
-end

data/lib/waterdrop/configurable/node.rb

@@ -1,100 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  module Configurable
-    # Single non-leaf node
-    # This is a core component for the configurable settings
-    #
-    # The idea here is simple: we collect settings (leafs) and children (nodes) information and we
-    # only compile/initialize the values prior to user running the `#configure` API. This API needs
-    # to run prior to using the result stuff even if there is nothing to configure
-    class Node
-      attr_reader :name, :nestings
-
-      # We need to be able to redefine children for deep copy
-      attr_accessor :children
-
-      # @param name [Symbol] node name
-      # @param nestings [Proc] block for nested settings
-      def initialize(name, nestings = ->(_) {})
-        @name = name
-        @children = []
-        @nestings = nestings
-        instance_eval(&nestings)
-      end
-
-      # Allows for a single leaf or nested node definition
-      #
-      # @param name [Symbol] setting or nested node name
-      # @param default [Object] default value
-      # @param constructor [#call, nil] callable or nil
-      # @param block [Proc] block for nested settings
-      def setting(name, default: nil, constructor: nil, &block)
-        @children << if block
-                       Node.new(name, block)
-                     else
-                       Leaf.new(name, default, constructor)
-                     end
-      end
-
-      # Allows for the configuration and setup of the settings
-      #
-      # Compile settings, allow for overrides via yielding
-      # @return [Node] returns self after configuration
-      def configure
-        compile
-        yield(self) if block_given?
-        self
-      end
-
-      # @return [Hash] frozen config hash representation
-      def to_h
-        config = {}
-
-        @children.each do |value|
-          config[value.name] = if value.is_a?(Leaf)
-                                 public_send(value.name)
-                               else
-                                 value.to_h
-                               end
-        end
-
-        config.freeze
-      end
-
-      # Deep copies all the children nodes to allow us for templates building on a class level and
-      # non-side-effect usage on an instance/inherited.
-      # @return [Node] duplicated node
-      def deep_dup
-        dupped = Node.new(name, nestings)
-
-        dupped.children += children.map do |value|
-          value.is_a?(Leaf) ? value.dup : value.deep_dup
-        end
-
-        dupped
-      end
-
-      # Converts the settings definitions into end children
-      # @note It runs once, after things are compiled, they will not be recompiled again
-      def compile
-        @children.each do |value|
-          # Do not redefine something that was already set during compilation
-          # This will allow us to reconfigure things and skip override with defaults
-          next if respond_to?(value.name)
-
-          singleton_class.attr_accessor value.name
-
-          initialized = if value.is_a?(Leaf)
-                          value.constructor ? value.constructor.call(value.default) : value.default
-                        else
-                          value.compile
-                          value
-                        end
-
-          public_send("#{value.name}=", initialized)
-        end
-      end
-    end
-  end
-end

data/lib/waterdrop/configurable.rb

@@ -1,71 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  # A simple dry-configuration API compatible module for defining settings with defaults and a
-  # constructor.
-  module Configurable
-    # A simple settings layer that works similar to dry-configurable
-    # It allows us to define settings on a class and per instance level with templating on a class
-    # level. It handles inheritance and allows for nested settings.
-    #
-    # @note The core settings template needs to be defined on a class level
-    class << self
-      # Sets up all the class methods and inits the core root node.
-      # Useful when only per class settings are needed as does not include instance methods
-      # @param base [Class] class that we extend
-      def extended(base)
-        base.extend ClassMethods
-      end
-
-      # Sets up all the class and instance methods and inits the core root node
-      #
-      # @param base [Class] class to which we want to add configuration
-      #
-      # Needs to be used when per instance configuration is needed
-      def included(base)
-        base.include InstanceMethods
-        base.extend self
-      end
-    end
-
-    # Instance related methods
-    module InstanceMethods
-      # @return [Node] config root node
-      def config
-        @config ||= self.class.config.deep_dup
-      end
-
-      # Allows for a per instance configuration (if needed)
-      # @param block [Proc] block for configuration
-      def configure(&block)
-        config.configure(&block)
-      end
-    end
-
-    # Class related methods
-    module ClassMethods
-      # @return [Node] root node for the settings
-      def config
-        return @config if @config
-
-        # This will handle inheritance
-        @config = if superclass.respond_to?(:config)
-                    superclass.config.deep_dup
-                  else
-                    Node.new(:root)
-                  end
-      end
-
-      # Allows for a per class configuration (if needed)
-      # @param block [Proc] block for configuration
-      def configure(&block)
-        config.configure(&block)
-      end
-
-      # Pipes the settings setup to the config root node
-      def setting(...)
-        config.setting(...)
-      end
-    end
-  end
-end

data/lib/waterdrop/contracts/base.rb

@@ -1,23 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  module Contracts
-    # Base for all the contracts in WaterDrop
-    class Base < Dry::Validation::Contract
-      config.messages.load_paths << File.join(WaterDrop.gem_root, 'config', 'errors.yml')
-
-      # @param data [Hash] data for validation
-      # @param error_class [Class] error class that should be used when validation fails
-      # @return [Boolean] true
-      # @raise [StandardError] any error provided in the error_class that inherits from the
-      #   standard error
-      def validate!(data, error_class)
-        result = call(data)
-
-        return true if result.success?
-
-        raise error_class, result.errors.to_h
-      end
-    end
-  end
-end

data/lib/waterdrop/instrumentation/callbacks/statistics_decorator.rb

@@ -1,77 +0,0 @@
-# frozen_string_literal: true
-
-module WaterDrop
-  module Instrumentation
-    module Callbacks
-      # Many of the librdkafka statistics are absolute values instead of a gauge.
-      # This means, that for example number of messages sent is an absolute growing value
-      # instead of being a value of messages sent from the last statistics report.
-      # This decorator calculates the diff against previously emited stats, so we get also
-      # the diff together with the original values
-      class StatisticsDecorator
-        def initialize
-          @previous = {}.freeze
-        end
-
-        # @param emited_stats [Hash] original emited statistics
-        # @return [Hash] emited statistics extended with the diff data
-        # @note We modify the emited statistics, instead of creating new. Since we don't expose
-        #   any API to get raw data, users can just assume that the result of this decoration is
-        #   the proper raw stats that they can use
-        def call(emited_stats)
-          diff(
-            @previous,
-            emited_stats
-          )
-
-          @previous = emited_stats
-
-          emited_stats.freeze
-        end
-
-        private
-
-        # Calculates the diff of the provided values and modifies in place the emited statistics
-        #
-        # @param previous [Object] previous value from the given scope in which
-        #   we are
-        # @param current [Object] current scope from emitted statistics
-        # @return [Object] the diff if the values were numerics or the current scope
-        def diff(previous, current)
-          if current.is_a?(Hash)
-            # @note We cannot use #each_key as we modify the content of the current scope
-            #   in place (in case it's a hash)
-            current.keys.each do |key|
-              append(
-                current,
-                key,
-                diff((previous || {})[key], (current || {})[key])
-              )
-            end
-          end
-
-          # Diff can be computed only for numerics
-          return current unless current.is_a?(Numeric)
-          # If there was no previous value, delta is always zero
-          return 0 unless previous
-          # Should never happen but just in case, a type changed in between stats
-          return current unless previous.is_a?(Numeric)
-
-          current - previous
-        end
-
-        # Appends the result of the diff to a given key as long as the result is numeric
-        #
-        # @param current [Hash] current scope
-        # @param key [Symbol] key based on which we were diffing
-        # @param result [Object] diff result
-        def append(current, key, result)
-          return unless result.is_a?(Numeric)
-          return if current.frozen?
-
-          current["#{key}_d"] = result
-        end
-      end
-    end
-  end
-end