waterdrop 2.0.0 → 2.4.0

data/README.md

@@ -1,33 +1,52 @@
 # WaterDrop
 
-**Note**: Documentation presented here refers to WaterDrop `2.0.0.pre1`.
+**Note**: Documentation presented here refers to WaterDrop `2.x`.
 
-WaterDrop `2.0` does **not** work with Karafka `1.*` and aims to either work as a standalone producer outside of Karafka `1.*` ecosystem or as a part of not yet released Karafka `2.0.*`.
+WaterDrop `2.x` works with Karafka `2.*` and aims to either work as a standalone producer or as a part of the Karafka `2.*`.
 
-Please refer to [this](https://github.com/karafka/waterdrop/tree/1.4) branch and it's documentation for details about WaterDrop `1.*` usage.
+Please refer to [this](https://github.com/karafka/waterdrop/tree/1.4) branch and its documentation for details about WaterDrop `1.*` usage.
 
 [![Build Status](https://github.com/karafka/waterdrop/workflows/ci/badge.svg)](https://github.com/karafka/waterdrop/actions?query=workflow%3Aci)
-[![Join the chat at https://gitter.im/karafka/karafka](https://badges.gitter.im/karafka/karafka.svg)](https://gitter.im/karafka/karafka?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![Gem Version](https://badge.fury.io/rb/waterdrop.svg)](http://badge.fury.io/rb/waterdrop)
+[![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
-Gem used to send messages to Kafka in an easy way with an extra validation layer. It is a part of the [Karafka](https://github.com/karafka/karafka) ecosystem.
+A gem to send messages to Kafka easily with an extra validation layer. It is a part of the [Karafka](https://github.com/karafka/karafka) ecosystem.
 
 It:
 
- - Is thread safe
+ - Is thread-safe
  - Supports sync producing
  - Supports async producing
  - Supports buffering
  - Supports producing messages to multiple clusters
  - Supports multiple delivery policies
- - Works with Kafka 1.0+ and Ruby 2.5+
+ - Works with Kafka `1.0+` and Ruby `2.7+`
+
+## Table of contents
+
+- [Installation](#installation)
+- [Setup](#setup)
+  * [WaterDrop configuration options](#waterdrop-configuration-options)
+  * [Kafka configuration options](#kafka-configuration-options)
+- [Usage](#usage)
+  * [Basic usage](#basic-usage)
+  * [Using WaterDrop across the application and with Ruby on Rails](#using-waterdrop-across-the-application-and-with-ruby-on-rails)
+  * [Using WaterDrop with a connection-pool](#using-waterdrop-with-a-connection-pool)
+  * [Buffering](#buffering)
+      + [Using WaterDrop to buffer messages based on the application logic](#using-waterdrop-to-buffer-messages-based-on-the-application-logic)
+      + [Using WaterDrop with rdkafka buffers to achieve periodic auto-flushing](#using-waterdrop-with-rdkafka-buffers-to-achieve-periodic-auto-flushing)
+- [Instrumentation](#instrumentation)
+  * [Usage statistics](#usage-statistics)
+  * [Error notifications](#error-notifications)
+  * [Datadog and StatsD integration](#datadog-and-statsd-integration)
+  * [Forking and potential memory problems](#forking-and-potential-memory-problems)
+- [Note on contributions](#note-on-contributions)
 
 ## Installation
 
-```ruby
-gem install waterdrop
-```
+**Note**: If you want to both produce and consume messages, please use [Karafka](https://github.com/karafka/karafka/). It integrates WaterDrop automatically.
 
-or add this to your Gemfile:
+Add this to your Gemfile:
 
 ```ruby
 gem 'waterdrop'
@@ -41,10 +60,10 @@ bundle install
 
 ## Setup
 
-WaterDrop is a complex tool, that contains multiple configuration options. To keep everything organized, all the configuration options were divided into two groups:
+WaterDrop is a complex tool that contains multiple configuration options. To keep everything organized, all the configuration options were divided into two groups:
 
-- WaterDrop options - options directly related to Karafka framework and it's components
-- Kafka driver options - options related to `Kafka`
+- WaterDrop options - options directly related to WaterDrop and its components
+- Kafka driver options - options related to `rdkafka`
 
 To apply all those configuration options, you need to create a producer instance and use the ```#setup``` method:
 
@@ -88,8 +107,6 @@ You can create producers with different `kafka` settings. Documentation of the a
 
 ## Usage
 
-Please refer to the [documentation](https://www.rubydoc.info/github/karafka/waterdrop) in case you're interested in the more advanced API.
-
 ### Basic usage
 
 To send Kafka messages, just create a producer and use it:
@@ -130,20 +147,60 @@ Each message that you want to publish, will have its value checked.
 
 Here are all the things you can provide in the message hash:
 
-| Option      | Required | Value type    | Description                                           |
-|-------------|----------|---------------|-------------------------------------------------------|
-| `topic`     | true     | String        | The Kafka topic that should be written to             |
-| `payload`   | true     | String        | Data you want to send to Kafka                        |
-| `key`       | false    | String        | The key that should be set in the Kafka message       |
-| `partition` | false    | Integer       | A specific partition number that should be written to |
-| `timestamp` | false    | Time, Integer | The timestamp that should be set on the message       |
-| `headers`   | false    | Hash          | Headers for the message                               |
+| Option          | Required | Value type    | Description                                              |
+|-----------------|----------|---------------|----------------------------------------------------------|
+| `topic`         | true     | String        | The Kafka topic that should be written to                |
+| `payload`       | true     | String        | Data you want to send to Kafka                           |
+| `key`           | false    | String        | The key that should be set in the Kafka message          |
+| `partition`     | false    | Integer       | A specific partition number that should be written to    |
+| `partition_key` | false    | String        | Key to indicate the destination partition of the message |
+| `timestamp`     | false    | Time, Integer | The timestamp that should be set on the message          |
+| `headers`       | false    | Hash          | Headers for the message                                  |
 
 Keep in mind, that message you want to send should be either binary or stringified (to_s, to_json, etc).
 
+### Using WaterDrop across the application and with Ruby on Rails
+
+If you plan to both produce and consume messages using Kafka, you should install and use [Karafka](https://github.com/karafka/karafka). It integrates automatically with Ruby on Rails applications and auto-configures WaterDrop producer to make it accessible via `Karafka#producer` method.
+
+If you want to only produce messages from within your application, since WaterDrop is thread-safe you can create a single instance in an initializer like so:
+
+```ruby
+KAFKA_PRODUCER = WaterDrop::Producer.new
+
+KAFKA_PRODUCER.setup do |config|
+  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
+end
+
+# And just dispatch messages
+KAFKA_PRODUCER.produce_sync(topic: 'my-topic', payload: 'my message')
+```
+
+### Using WaterDrop with a connection-pool
+
+While WaterDrop is thread-safe, there is no problem in using it with a connection pool inside high-intensity applications. The only thing worth keeping in mind, is that WaterDrop instances should be shutdown before the application is closed.
+
+```ruby
+KAFKA_PRODUCERS_CP = ConnectionPool.new do
+  WaterDrop::Producer.new do |config|
+    config.kafka = { 'bootstrap.servers': 'localhost:9092' }
+  end
+end
+
+KAFKA_PRODUCERS_CP.with do |producer|
+  producer.produce_async(topic: 'my-topic', payload: 'my message')
+end
+
+KAFKA_PRODUCERS_CP.shutdown { |producer| producer.close }
+```
+
 ### Buffering
 
-WaterDrop producers support buffering of messages, which means that you can easily implement periodic flushing for long running processes as well as buffer several messages to be flushed the same moment:
+WaterDrop producers support buffering messages in their internal buffers and on the `rdkafka` level via `queue.buffering.*` set of settings.
+
+This means that depending on your use case, you can achieve both granular buffering and flushing control when needed with context awareness and periodic and size-based flushing functionalities.
+
+#### Using WaterDrop to buffer messages based on the application logic
 
 ```ruby
 producer = WaterDrop::Producer.new
@@ -152,16 +209,41 @@ producer.setup do |config|
   config.kafka = { 'bootstrap.servers': 'localhost:9092' }
 end
 
-time = Time.now - 10
+# Simulating some events states of a transaction - notice, that the messages will be flushed to
+# kafka only upon arrival of the `finished` state.
+%w[
+  started
+  processed
+  finished
+].each do |state|
+  producer.buffer(topic: 'events', payload: state)
+
+  puts "The messages buffer size #{producer.messages.size}"
+  producer.flush_sync if state == 'finished'
+  puts "The messages buffer size #{producer.messages.size}"
+end
+
+producer.close
+```
+
+#### Using WaterDrop with rdkafka buffers to achieve periodic auto-flushing
+
+```ruby
+producer = WaterDrop::Producer.new
 
-while time < Time.now
-  time += 1
-  producer.buffer(topic: 'times', payload: Time.now.to_s)
+producer.setup do |config|
+  config.kafka = {
+    'bootstrap.servers': 'localhost:9092',
+    # Accumulate messages for at most 10 seconds
+    'queue.buffering.max.ms': 10_000
+  }
 end
 
-puts "The messages buffer size #{producer.messages.size}"
-producer.flush_sync
-puts "The messages buffer size #{producer.message.size}"
+# WaterDrop will flush messages minimum once every 10 seconds
+30.times do |i|
+  producer.produce_async(topic: 'events', payload: i.to_s)
+  sleep(1)
+end
 
 producer.close
 ```
@@ -240,27 +322,79 @@ producer.close
 
 Note: The metrics returned may not be completely consistent between brokers, toppars and totals, due to the internal asynchronous nature of librdkafka. E.g., the top level tx total may be less than the sum of the broker tx values which it represents.
 
-### Forking and potential memory problems
+### Datadog and StatsD integration
 
-If you work with forked processes, make sure you **don't** use the producer before the fork. You can easily configure the producer and then fork and use it.
+WaterDrop comes with (optional) full Datadog and StatsD integration that you can use. To use it:
 
-To tackle this [obstacle](https://github.com/appsignal/rdkafka-ruby/issues/15) related to rdkafka, WaterDrop adds finalizer to each of the producers to close the rdkafka client before the Ruby process is shutdown. Due to the [nature of the finalizers](https://www.mikeperham.com/2010/02/24/the-trouble-with-ruby-finalizers/), this implementation prevents producers from being GCed (except upon VM shutdown) and can cause memory leaks if you don't use persistent/long-lived producers in a long-running process or if you don't use the `#close` method of a producer when it is no longer needed. Creating a producer instance for each message is anyhow a rather bad idea, so we recommend not to.
+```ruby
+# require datadog/statsd and the listener as it is not loaded by default
+require 'datadog/statsd'
+require 'waterdrop/instrumentation/vendors/datadog/listener'
+
+# initialize your producer with statistics.interval.ms enabled so the metrics are published
+producer = WaterDrop::Producer.new do |config|
+  config.deliver = true
+  config.kafka = {
+    'bootstrap.servers': 'localhost:9092',
+    'statistics.interval.ms': 1_000
+  }
+end
 
-## References
+# initialize the listener with statsd client
+listener = ::WaterDrop::Instrumentation::Vendors::Datadog::Listener.new do |config|
+  config.client = Datadog::Statsd.new('localhost', 8125)
+  # Publish host as a tag alongside the rest of tags
+  config.default_tags = ["host:#{Socket.gethostname}"]
+end
 
-* [WaterDrop code documentation](https://www.rubydoc.info/github/karafka/waterdrop)
-* [Karafka framework](https://github.com/karafka/karafka)
-* [WaterDrop Actions CI](https://github.com/karafka/waterdrop/actions?query=workflow%3Ac)
-* [WaterDrop Coditsu](https://app.coditsu.io/karafka/repositories/waterdrop)
+# Subscribe with your listener to your producer and you should be ready to go!
+producer.monitor.subscribe(listener)
+```
 
-## Note on contributions
+You can also find [here](https://github.com/karafka/waterdrop/blob/master/lib/waterdrop/instrumentation/vendors/datadog/dashboard.json) a ready to import DataDog dashboard configuration file that you can use to monitor all of your producers.
+
+![Example WaterDrop DD dashboard](https://raw.githubusercontent.com/karafka/misc/master/printscreens/waterdrop_dd_dashboard_example.png)
+
+### Error notifications
+
+WaterDrop allows you to listen to all errors that occur while producing messages and in its internal background threads. Things like reconnecting to Kafka upon network errors and others unrelated to publishing messages are all available under `error.occurred` notification key. You can subscribe to this event to ensure your setup is healthy and without any problems that would otherwise go unnoticed as long as messages are delivered.
 
-First, thank you for considering contributing to WaterDrop! It's people like you that make the open source community such a great community!
+```ruby
+producer = WaterDrop::Producer.new do |config|
+  # Note invalid connection port...
+  config.kafka = { 'bootstrap.servers': 'localhost:9090' }
+end
+
+producer.monitor.subscribe('error.occurred') do |event|
+  error = event[:error]
+
+  p "WaterDrop error occurred: #{error}"
+end
+
+# Run this code without Kafka cluster
+loop do
+  producer.produce_async(topic: 'events', payload: 'data')
+
+  sleep(1)
+end
 
-Each pull request must pass all the RSpec specs and meet our quality requirements.
+# After you stop your Kafka cluster, you will see a lot of those:
+#
+# WaterDrop error occurred: Local: Broker transport failure (transport)
+#
+# WaterDrop error occurred: Local: Broker transport failure (transport)
+```
+
+### Forking and potential memory problems
+
+If you work with forked processes, make sure you **don't** use the producer before the fork. You can easily configure the producer and then fork and use it.
+
+To tackle this [obstacle](https://github.com/appsignal/rdkafka-ruby/issues/15) related to rdkafka, WaterDrop adds finalizer to each of the producers to close the rdkafka client before the Ruby process is shutdown. Due to the [nature of the finalizers](https://www.mikeperham.com/2010/02/24/the-trouble-with-ruby-finalizers/), this implementation prevents producers from being GCed (except upon VM shutdown) and can cause memory leaks if you don't use persistent/long-lived producers in a long-running process or if you don't use the `#close` method of a producer when it is no longer needed. Creating a producer instance for each message is anyhow a rather bad idea, so we recommend not to.
+
+## Note on contributions
 
-To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
+First, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community!
 
-Coditsu will automatically check your work against our quality standards. You can find your commit check results on the [builds page](https://app.coditsu.io/karafka/repositories/waterdrop/builds/commit_builds) of WaterDrop repository.
+Each pull request must pass all the RSpec specs, integration tests and meet our quality requirements.
 
-[![coditsu](https://coditsu.io/assets/quality_bar.svg)](https://app.coditsu.io/karafka/repositories/waterdrop/builds/commit_builds)
+Fork it, update and wait for the Github Actions results.

data/certs/mensfeld.pem

@@ -1,25 +1,25 @@
 -----BEGIN CERTIFICATE-----
 MIIEODCCAqCgAwIBAgIBATANBgkqhkiG9w0BAQsFADAjMSEwHwYDVQQDDBhtYWNp
-ZWovREM9bWVuc2ZlbGQvREM9cGwwHhcNMjAwODExMDkxNTM3WhcNMjEwODExMDkx
-NTM3WjAjMSEwHwYDVQQDDBhtYWNpZWovREM9bWVuc2ZlbGQvREM9cGwwggGiMA0G
-CSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDCpXsCgmINb6lHBXXBdyrgsBPSxC4/
-2H+weJ6L9CruTiv2+2/ZkQGtnLcDgrD14rdLIHK7t0o3EKYlDT5GhD/XUVhI15JE
-N7IqnPUgexe1fbZArwQ51afxz2AmPQN2BkB2oeQHXxnSWUGMhvcEZpfbxCCJH26w
-hS0Ccsma8yxA6hSlGVhFVDuCr7c2L1di6cK2CtIDpfDaWqnVNJEwBYHIxrCoWK5g
-sIGekVt/admS9gRhIMaIBg+Mshth5/DEyWO2QjteTodItlxfTctrfmiAl8X8T5JP
-VXeLp5SSOJ5JXE80nShMJp3RFnGw5fqjX/ffjtISYh78/By4xF3a25HdWH9+qO2Z
-tx0wSGc9/4gqNM0APQnjN/4YXrGZ4IeSjtE+OrrX07l0TiyikzSLFOkZCAp8oBJi
-Fhlosz8xQDJf7mhNxOaZziqASzp/hJTU/tuDKl5+ql2icnMv5iV/i6SlmvU29QNg
-LCV71pUv0pWzN+OZbHZKWepGhEQ3cG9MwvkCAwEAAaN3MHUwCQYDVR0TBAIwADAL
-BgNVHQ8EBAMCBLAwHQYDVR0OBBYEFImGed2AXS070ohfRidiCEhXEUN+MB0GA1Ud
+ZWovREM9bWVuc2ZlbGQvREM9cGwwHhcNMjEwODExMTQxNTEzWhcNMjIwODExMTQx
+NTEzWjAjMSEwHwYDVQQDDBhtYWNpZWovREM9bWVuc2ZlbGQvREM9cGwwggGiMA0G
+CSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDV2jKH4Ti87GM6nyT6D+ESzTI0MZDj
+ak2/TEwnxvijMJyCCPKT/qIkbW4/f0VHM4rhPr1nW73sb5SZBVFCLlJcOSKOBdUY
+TMY+SIXN2EtUaZuhAOe8LxtxjHTgRHvHcqUQMBENXTISNzCo32LnUxweu66ia4Pd
+1mNRhzOqNv9YiBZvtBf7IMQ+sYdOCjboq2dlsWmJiwiDpY9lQBTnWORnT3mQxU5x
+vPSwnLB854cHdCS8fQo4DjeJBRZHhEbcE5sqhEMB3RZA3EtFVEXOxlNxVTS3tncI
+qyNXiWDaxcipaens4ObSY1C2HTV7OWb7OMqSCIybeYTSfkaSdqmcl4S6zxXkjH1J
+tnjayAVzD+QVXGijsPLE2PFnJAh9iDET2cMsjabO1f6l1OQNyAtqpcyQcgfnyW0z
+g7tGxTYD+6wJHffM9d9txOUw6djkF6bDxyqB8lo4Z3IObCx18AZjI9XPS9QG7w6q
+LCWuMG2lkCcRgASqaVk9fEf9yMc2xxz5o3kCAwEAAaN3MHUwCQYDVR0TBAIwADAL
+BgNVHQ8EBAMCBLAwHQYDVR0OBBYEFBqUFCKCOe5IuueUVqOB991jyCLLMB0GA1Ud
 EQQWMBSBEm1hY2llakBtZW5zZmVsZC5wbDAdBgNVHRIEFjAUgRJtYWNpZWpAbWVu
-c2ZlbGQucGwwDQYJKoZIhvcNAQELBQADggGBAKiHpwoENVrMi94V1zD4o8/6G3AU
-gWz4udkPYHTZLUy3dLznc/sNjdkJFWT3E6NKYq7c60EpJ0m0vAEg5+F5pmNOsvD3
-2pXLj9kisEeYhR516HwXAvtngboUcb75skqvBCU++4Pu7BRAPjO1/ihLSBexbwSS
-fF+J5OWNuyHHCQp+kGPLtXJe2yUYyvSWDj3I2//Vk0VhNOIlaCS1+5/P3ZJThOtm
-zJUBI7h3HgovwRpcnmk2mXTmU4Zx/bCzX8EA6VY0khEvnmiq7S6eBF0H9qH8KyQ6
-EkVLpvmUDFcf/uNaBQdazEMB5jYtwoA8gQlANETNGPi51KlkukhKgaIEDMkBDJOx
-65N7DzmkcyY0/GwjIVIxmRhcrCt1YeCUElmfFx0iida1/YRm6sB2AXqScc1+ECRi
-2DND//YJUikn1zwbz1kT70XmHd97B4Eytpln7K+M1u2g1pHVEPW4owD/ammXNpUy
-nt70FcDD4yxJQ+0YNiHd0N8IcVBM1TMIVctMNQ==
+c2ZlbGQucGwwDQYJKoZIhvcNAQELBQADggGBADD0/UuTTFgW+CGk2U0RDw2RBOca
+W2LTF/G7AOzuzD0Tc4voc7WXyrgKwJREv8rgBimLnNlgmFJLmtUCh2U/MgxvcilH
+yshYcbseNvjkrtYnLRlWZR4SSB6Zei5AlyGVQLPkvdsBpNegcG6w075YEwzX/38a
+8V9B/Yri2OGELBz8ykl7BsXUgNoUPA/4pHF6YRLz+VirOaUIQ4JfY7xGj6fSOWWz
+/rQ/d77r6o1mfJYM/3BRVg73a3b7DmRnE5qjwmSaSQ7u802pJnLesmArch0xGCT/
+fMmRli1Qb+6qOTl9mzD6UDMAyFR4t6MStLm0mIEqM0nBO5nUdUWbC7l9qXEf8XBE
+2DP28p3EqSuS+lKbAWKcqv7t0iRhhmaod+Yn9mcrLN1sa3q3KSQ9BCyxezCD4Mk2
+R2P11bWoCtr70BsccVrN8jEhzwXngMyI2gVt750Y+dbTu1KgRqZKp/ECe7ZzPzXj
+pIy9vHxTANKYVyI4qj8OrFdEM5BQNu8oQpL0iQ==
 -----END CERTIFICATE-----

data/config/errors.yml

@@ -1,6 +1,30 @@
 en:
-  dry_validation:
-    errors:
-      invalid_key_type: all keys need to be of type String
-      invalid_value_type: all values need to be of type String
-      max_payload_size: is more than `max_payload_size` config value
+  validations:
+    config:
+      missing: must be present
+      logger_format: must be present
+      deliver_format: must be boolean
+      id_format: must be a non-empty string
+      max_payload_size_format: must be an integer that is equal or bigger than 1
+      wait_timeout_format: must be a numeric that is bigger than 0
+      max_wait_timeout_format: must be an integer that is equal or bigger than 0
+      kafka_format: must be a hash with symbol based keys
+      kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
+
+    message:
+      missing: must be present
+      partition_format: must be an integer greater or equal to -1
+      topic_format: 'does not match the topic allowed format'
+      partition_key_format: must be a non-empty string
+      timestamp_format: must be either time or integer
+      payload_format: must be string
+      headers_format: must be a hash
+      key_format: must be a non-empty string
+      payload_max_size: is more than `max_payload_size` config value
+      headers_invalid_key_type: all headers keys need to be of type String
+      headers_invalid_value_type: all headers values need to be of type String
+
+    test:
+      missing: must be present
+      nested.id_format: 'is invalid'
+      nested.id2_format: 'is invalid'

data/docker-compose.yml

@@ -5,7 +5,7 @@ services:
     ports:
       - "2181:2181"
   kafka:
-    image: wurstmeister/kafka:1.0.1
+    image: wurstmeister/kafka:2.12-2.5.0
     ports:
       - "9092:9092"
     environment:
@@ -13,5 +13,6 @@ services:
       KAFKA_ADVERTISED_PORT: 9092
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
+      KAFKA_CREATE_TOPICS: 'example_topic:1:1'
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock

data/lib/{water_drop → waterdrop}/config.rb

@@ -5,35 +5,55 @@
 module WaterDrop
   # Configuration object for setting up all options required by WaterDrop
   class Config
-    include Dry::Configurable
+    include ::Karafka::Core::Configurable
+
+    # Defaults for kafka settings, that will be overwritten only if not present already
+    KAFKA_DEFAULTS = {
+      'client.id': 'waterdrop'
+    }.freeze
+
+    private_constant :KAFKA_DEFAULTS
 
     # WaterDrop options
     #
     # option [String] id of the producer. This can be helpful when building producer specific
-    #   instrumentation or loggers. It is not the kafka producer id
-    setting(:id, false) { |id| id || SecureRandom.uuid }
+    #   instrumentation or loggers. It is not the kafka client id. It is an id that should be
+    #   unique for each of the producers
+    setting(
+      :id,
+      default: false,
+      constructor: ->(id) { id || SecureRandom.uuid }
+    )
     # option [Instance] logger that we want to use
     # @note Due to how rdkafka works, this setting is global for all the producers
-    setting(:logger, false) { |logger| logger || Logger.new($stdout, level: Logger::WARN) }
+    setting(
+      :logger,
+      default: false,
+      constructor: ->(logger) { logger || Logger.new($stdout, level: Logger::WARN) }
+    )
     # option [Instance] monitor that we want to use. See instrumentation part of the README for
     #   more details
-    setting(:monitor, false) { |monitor| monitor || WaterDrop::Instrumentation::Monitor.new }
+    setting(
+      :monitor,
+      default: false,
+      constructor: ->(monitor) { monitor || WaterDrop::Instrumentation::Monitor.new }
+    )
     # option [Integer] max payload size allowed for delivery to Kafka
-    setting :max_payload_size, 1_000_012
+    setting :max_payload_size, default: 1_000_012
     # option [Integer] Wait that long for the delivery report or raise an error if this takes
     #   longer than the timeout.
-    setting :max_wait_timeout, 5
+    setting :max_wait_timeout, default: 5
     # option [Numeric] how long should we wait between re-checks on the availability of the
     #   delivery report. In a really robust systems, this describes the min-delivery time
     #   for a single sync message when produced in isolation
-    setting :wait_timeout, 0.005 # 5 milliseconds
+    setting :wait_timeout, default: 0.005 # 5 milliseconds
     # option [Boolean] should we send messages. Setting this to false can be really useful when
     #   testing and or developing because when set to false, won't actually ping Kafka but will
     #   run all the validations, etc
-    setting :deliver, true
+    setting :deliver, default: true
     # rdkafka options
     # @see https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
-    setting :kafka, {}
+    setting :kafka, default: {}
 
     # Configuration method
     # @yield Runs a block of code providing a config singleton instance to it
@@ -41,21 +61,29 @@ module WaterDrop
     def setup
       configure do |config|
         yield(config)
-        validate!(config.to_h)
+
+        merge_kafka_defaults!(config)
+
+        Contracts::Config.new.validate!(config.to_h, Errors::ConfigurationInvalidError)
+
+        ::Rdkafka::Config.logger = config.logger
       end
+
+      self
     end
 
     private
 
-    # Validates the configuration and if anything is wrong, will raise an exception
-    # @param config_hash [Hash] config hash with setup details
-    # @raise [WaterDrop::Errors::ConfigurationInvalidError] raised when something is wrong with
-    #   the configuration
-    def validate!(config_hash)
-      result = Contracts::Config.new.call(config_hash)
-      return true if result.success?
+    # Propagates the kafka setting defaults unless they are already present
+    # This makes it easier to set some values that users usually don't change but still allows them
+    # to overwrite the whole hash if they want to
+    # @param config [WaterDrop::Configurable::Node] config of this producer
+    def merge_kafka_defaults!(config)
+      KAFKA_DEFAULTS.each do |key, value|
+        next if config.kafka.key?(key)
 
-      raise Errors::ConfigurationInvalidError, result.errors.to_h
+        config.kafka[key] = value
+      end
     end
   end
 end

data/lib/waterdrop/contracts/config.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module WaterDrop
+  module Contracts
+    # Contract with validation rules for WaterDrop configuration details
+    class Config < ::Karafka::Core::Contractable::Contract
+      configure do |config|
+        config.error_messages = YAML.safe_load(
+          File.read(
+            File.join(WaterDrop.gem_root, 'config', 'errors.yml')
+          )
+        ).fetch('en').fetch('validations').fetch('config')
+      end
+
+      required(:id) { |val| val.is_a?(String) && !val.empty? }
+      required(:logger) { |val| !val.nil? }
+      required(:deliver) { |val| [true, false].include?(val) }
+      required(:max_payload_size) { |val| val.is_a?(Integer) && val >= 1 }
+      required(:max_wait_timeout) { |val| val.is_a?(Numeric) && val >= 0 }
+      required(:wait_timeout) { |val| val.is_a?(Numeric) && val.positive? }
+      required(:kafka) { |val| val.is_a?(Hash) && !val.empty? }
+
+      # rdkafka allows both symbols and strings as keys for config but then casts them to strings
+      # This can be confusing, so we expect all keys to be symbolized
+      virtual do |config, errors|
+        next true unless errors.empty?
+
+        errors = []
+
+        config
+          .fetch(:kafka)
+          .keys
+          .reject { |key| key.is_a?(Symbol) }
+          .each { |key| errors << [[:kafka, key], :kafka_key_must_be_a_symbol] }
+
+        errors
+      end
+    end
+  end
+end

data/lib/waterdrop/contracts/message.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module WaterDrop
+  module Contracts
+    # Contract with validation rules for validating that all the message options that
+    # we provide to producer ale valid and usable
+    class Message < ::Karafka::Core::Contractable::Contract
+      configure do |config|
+        config.error_messages = YAML.safe_load(
+          File.read(
+            File.join(WaterDrop.gem_root, 'config', 'errors.yml')
+          )
+        ).fetch('en').fetch('validations').fetch('message')
+      end
+
+      # Regex to check that topic has a valid format
+      TOPIC_REGEXP = /\A(\w|-|\.)+\z/
+
+      private_constant :TOPIC_REGEXP
+
+      attr_reader :max_payload_size
+
+      # @param max_payload_size [Integer] max payload size
+      def initialize(max_payload_size:)
+        super()
+        @max_payload_size = max_payload_size
+      end
+
+      required(:topic) { |val| val.is_a?(String) && TOPIC_REGEXP.match?(val) }
+      required(:payload) { |val| val.is_a?(String) }
+      optional(:key) { |val| val.nil? || (val.is_a?(String) && !val.empty?) }
+      optional(:partition) { |val| val.is_a?(Integer) && val >= -1 }
+      optional(:partition_key) { |val| val.nil? || (val.is_a?(String) && !val.empty?) }
+      optional(:timestamp) { |val| val.nil? || (val.is_a?(Time) || val.is_a?(Integer)) }
+      optional(:headers) { |val| val.nil? || val.is_a?(Hash) }
+
+      virtual do |config, errors|
+        next true unless errors.empty?
+        next true unless config.key?(:headers)
+        next true if config[:headers].nil?
+
+        errors = []
+
+        config.fetch(:headers).each do |key, value|
+          errors << [%i[headers], :invalid_key_type] unless key.is_a?(String)
+          errors << [%i[headers], :invalid_value_type] unless value.is_a?(String)
+        end
+
+        errors
+      end
+
+      virtual do |config, errors, validator|
+        next true unless errors.empty?
+        next true if config[:payload].bytesize <= validator.max_payload_size
+
+        [[%i[payload], :max_size]]
+      end
+    end
+  end
+end

data/lib/waterdrop/instrumentation/callbacks/delivery.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module WaterDrop
+  module Instrumentation
+    module Callbacks
+      # Creates a callable that we want to run upon each message delivery or failure
+      #
+      # @note We don't have to provide client_name here as this callback is per client instance
+      class Delivery
+        # @param producer_id [String] id of the current producer
+        # @param monitor [WaterDrop::Instrumentation::Monitor] monitor we are using
+        def initialize(producer_id, monitor)
+          @producer_id = producer_id
+          @monitor = monitor
+        end
+
+        # Emits delivery details to the monitor
+        # @param delivery_report [Rdkafka::Producer::DeliveryReport] delivery report
+        def call(delivery_report)
+          @monitor.instrument(
+            'message.acknowledged',
+            producer_id: @producer_id,
+            offset: delivery_report.offset,
+            partition: delivery_report.partition
+          )
+        end
+      end
+    end
+  end
+end