racecar 1.3.0 → 2.0.0.alpha1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f648450d1e12410d465bd1b72ee8e31d677e59be8002a6dd4122fe724a9ab59c
-  data.tar.gz: 598e77c323f7e09e2571a00bb84ec050ee0465a0b91c6a833658a23a0f26e204
+  metadata.gz: 1004df9eda77cd501025c24124d3bd01702bcd272571cddf4db2dd4da2ae9140
+  data.tar.gz: 539fea6b9d8a836d85af104e3441d3481c58d9861843d18ec70be2e2dd842af4
 SHA512:
-  metadata.gz: f0bfc386dd30b01328bb42435180f9118180eb78fb63a267ad6379150ff53df6da0524bef8e42cd53d25cad301c1e763a7c55328f43425efc1076eaebeddea14
-  data.tar.gz: fe73ab4baefbe01efca16b4f5b16b7379280a962a1829d857f850d56a0ef9da0b9fb5a2e6a367c308cf784abe3920bac0f5fd134242888067cb6079a6b812b39
+  metadata.gz: f86885745b07c476fd5bc104ebe1293ad1fd7c25e28089f7c7b53badfb5ee51ace00e8259ea1d1c668e9ea27289d6d11cdeaf9b208fe64457a4bde039271ff60
+  data.tar.gz: 20cedc0b4278fdf0c391057660fa2835071f005a43c8e133e01818e2ad314ebad8b1e9c5246ddda7d4a405019d9223840ecb67f6635a30884cd5836111a30838

data/CHANGELOG.md

@@ -1,28 +1,30 @@
 # Changelog
 
-## Unreleased
-
-## racecar v1.3.0
-
-* Support for `partitioner` configuration. (#268)
-
-## racecar v1.2.1
-
-* Require king_konf v1.0 or higher.
-
-## racecar v1.2.0
-
-* Support for `ssl_client_cert_key_password` (#173).
-* Support for `OAUTHBEARER` authentication (`sasl_oauth_token_provider`) (#178).
-
-## racecar v1.1.0
-
-* Require ruby-kafka v1.0 or higher.
-* Add error handling for required libraries (#149).
-
-## racecar v1.0.1
-
-* Add `--without-rails` option to boot consumer without Rails (#139).
+## racecar v2.0.0
+
+* Replace `ruby-kafka` with `rdkafka-ruby`
+* Removed config option `sasl_over_ssl`
+* [Racecar::Consumer] Do not pause consuming partitions on exception
+* [Racecar::Consumer] `topic`, `payload` and `key` are mandadory to method `produce`
+* [Racecar::Consumer] `process_batch` retrieves an array of messages instead of batch object
+* [Racecar::Consumer] Remove `offset_retention_time`
+* [Racecar::Consumer] Allow providing `additional_config` for subscriptions
+* [Racecar::Consumer] Provide access to `producer` and `consumer`
+* [Racecar::Consumer] Enforce delivering messages with method `deliver!`
+* [Racecar::Consumer] instead of raising when a partition EOF is reached, the result can be queried through `consumer.last_poll_read_partition_eof?`
+* [Racecar::Config] Remove `offset_retention_time`, `connect_timeout` and `offset_commit_threshold`
+* [Racecar::Config] Pass config to `rdkafka-ruby` via `producer` and `consumer`
+* [Racecar::Config] Replace `max_fetch_queue_size` with `min_message_queue_size`
+* [Racecar::Config] Add `synchronous_commits` to control blocking of `consumer.commit` (default `false`)
+* [Racecar::Config] Add `security_protocol` to control protocol between client and broker
+* [Racecar::Config] SSL configuration via `ssl_ca_location`, `ssl_crl_location`, `ssl_keystore_location` and `ssl_keystore_password`
+* [Racecar::Config] SASL configuration via `sasl_mechanism`, `sasl_kerberos_service_name`, `sasl_kerberos_principal`, `sasl_kerberos_kinit_cmd`, `sasl_kerberos_keytab`, `sasl_kerberos_min_time_before_relogin`, `sasl_username` and `sasl_password`
+* [Instrumentation] `produce_message.racecar` sent whenever a produced message is queued. Payload includes `topic`, `key`, `value` and `create_time`.
+* [Instrumentation] `acknowledged_message.racecar` send whenever a produced message was successfully received by Kafka. Payload includes `offset` and `partition`, but no message details.
+* [Instrumentation] `rdkafka-ruby` does not yet provide instrumentation [rdkafka-ruby#54](https://github.com/appsignal/rdkafka-ruby/issues/54)
+* [Instrumentation] if processors define a `statistics_callback`, it will be called once every second for every subscription or producer connection. The first argument will be a Hash, for contents see [librdkafka STATISTICS.md](https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md)
+* Add current directory to `$LOAD_PATH` only when `--require` option is used (#117).
+* Remove manual heartbeat support, see [Long-running message processing section in README](README.md#long-running-message-processing)
 
 ## racecar v1.0.0
 

data/README.md

@@ -1,8 +1,10 @@
+**IMPORTANT:** The `master` branch is unstable, working towards a v2 release that breaks a lot of stuff. Use the `v1-stable` branch if you want to suggest changes.
+
 # Racecar
 
 Racecar is a friendly and easy-to-approach Kafka consumer framework. It allows you to write small applications that process messages stored in Kafka topics while optionally integrating with your Rails models.
 
-The framework is based on [ruby-kafka](https://github.com/zendesk/ruby-kafka), which, when used directly, can be a challenge: it's a flexible library with lots of knobs and options. Most users don't need that level of flexibility, though. Racecar provides a simple and intuitive way to build and configure Kafka consumers.
+The framework is based on [rdkafka-ruby](https://github.com/appsignal/rdkafka-ruby), which, when used directly, can be a challenge: it's a flexible library with lots of knobs and options. Most users don't need that level of flexibility, though. Racecar provides a simple and intuitive way to build and configure Kafka consumers.
 
 **NOTE:** Racecar requires Kafka 0.10 or higher.
 
@@ -121,23 +123,23 @@ Note that once the consumer has started, it will commit the offsets it has proce
 
 #### Processing messages in batches
 
-If you want to process whole _batches_ of messages at a time, simply rename your `#process` method to `#process_batch`. The method will now be called with a "batch" object rather than a message:
+If you want to process whole _batches_ of messages at a time, simply rename your `#process` method to `#process_batch`. The method will now be called with an array of message objects:
 
 ```ruby
 class ArchiveEventsConsumer < Racecar::Consumer
   subscribes_to "events"
 
-  def process_batch(batch)
+  def process_batch(messages)
     file_name = [
-      batch.topic, # the topic this batch of messages came from.
-      batch.partition, # the partition this batch of messages came from.
-      batch.first_offset, # offset of the first message in the batch.
-      batch.last_offset, # offset of the last message in the batch.
+      messages.first.topic, # the topic this batch of messages came from.
+      messages.first.partition, # the partition this batch of messages came from.
+      messages.first.offset, # offset of the first message in the batch.
+      messages.last.offset, # offset of the last message in the batch.
     ].join("-")
 
     File.open(file_name, "w") do |file|
       # the messages in the batch.
-      batch.messages.each do |message|
+      messages.each do |message|
         file << message.value
       end
     end
@@ -155,23 +157,13 @@ Any headers set on the message will be available when consuming the message:
 message.headers #=> { "Header-A" => 42, ... }
 ```
 
-#### Heartbeats
+#### Long-running message processing
 
-In order to avoid your consumer being kicked out of its group during long-running message processing operations, it may be a good idea to periodically send so-called _heartbeats_ back to Kafka. This is done automatically for you after each message has been processed, but if the processing of a _single_ message takes a long time you may run into group stability issues.
+In order to avoid your consumer being kicked out of its group during long-running message processing operations, you'll need to let Kafka regularly know that the consumer is still healthy. There's two mechanisms in place to ensure that:
 
-If possible, intersperse `heartbeat` calls in between long-running operations in your consumer, e.g.
-
-```ruby
-def process(message)
-  long_running_op_one(message)
-
-  # Signals back to Kafka that we're still alive!
-  heartbeat
-
-  long_running_op_two(message)
-end
-```
+*Heartbeats:* They are automatically sent in the background and ensure the broker can still talk to the consumer. This will detect network splits, ungraceful shutdowns, etc.
 
+*Message Fetch Interval:* Kafka expects the consumer to query for new messages within this time limit. This will detect situations with slow IO or the consumer being stuck in an infinite loop without making actual progress. This limit applies to a whole batch if you do batch processing. Use `max_poll_interval` to increase the default 5 minute timeout, or reduce batching with `fetch_messages`.
 
 #### Tearing down resources when stopping
 
@@ -222,11 +214,13 @@ class GeoCodingConsumer < Racecar::Consumer
 
     # The `produce` method enqueues a message to be delivered after #process
     # returns. It won't actually deliver the message.
-    produce(JSON.dump(pageview), topic: "pageviews-with-country")
+    produce(payload: JSON.dump(pageview), topic: "pageviews-with-country", key: pageview["id"])
   end
 end
 ```
 
+The `deliver!` method can be used to block until the broker received all queued published messages (according to the publisher ack settings). This will automatically being called in the shutdown procedure of a consumer.
+
 You can set message headers by passing a `headers:` option with a Hash of headers.
 
 ### Configuration
@@ -263,8 +257,6 @@ end
 The consumers will checkpoint their positions from time to time in order to be able to recover from failures. This is called _committing offsets_, since it's done by tracking the offset reached in each partition being processed, and committing those offset numbers to the Kafka offset storage API. If you can tolerate more double-processing after a failure, you can increase the interval between commits in order to better performance. You can also do the opposite if you prefer less chance of double-processing.
 
 * `offset_commit_interval` – How often to save the consumer's position in Kafka. Default is every 10 seconds.
-* `offset_commit_threshold` – How many messages to process before forcing a checkpoint. Default is 0, which means there's no limit. Setting this to e.g. 100 makes the consumer stop every 100 messages to checkpoint its position.
-* `offset_retention_time` - How long committed offsets will be retained. Defaults to the broker setting.
 
 #### Timeouts & intervals
 
@@ -272,8 +264,9 @@ All timeouts are defined in number of seconds.
 
 * `session_timeout` – The idle timeout after which a consumer is kicked out of the group. Consumers must send heartbeats with at least this frequency.
 * `heartbeat_interval` – How often to send a heartbeat message to Kafka.
-* `pause_timeout` – How long to pause a partition for if the consumer raises an exception while processing a message. Default is to pause for 10 seconds. Set this to zero in order to disable automatic pausing of partitions.
-* `connect_timeout` – How long to wait when trying to connect to a Kafka broker. Default is 10 seconds.
+* `max_poll_interval` – The maximum time between two message fetches before the consumer is kicked out of the group. Put differently, your (batch) processing must finish earlier than this.
+* `pause_timeout` – How long to pause a partition for if the consumer raises an exception while processing a message. Default is to pause for 10 seconds. Set this to `0` in order to disable automatic pausing of partitions or to `-1` to pause indefinitely.
+* `pause_with_exponential_backoff` – Set to `true` if you want to double the `pause_timeout` on each consecutive failure of a particular partition.
 * `socket_timeout` – How long to wait when trying to communicate with a Kafka broker. Default is 30 seconds.
 * `max_wait_time` – How long to allow the Kafka brokers to wait before returning messages. A higher number means larger batches, at the cost of higher latency. Default is 1 second.
 
@@ -283,55 +276,38 @@ Kafka is _really_ good at throwing data at consumers, so you may want to tune th
 
 Racecar uses ruby-kafka under the hood, which fetches messages from the Kafka brokers in a background thread. This thread pushes fetch responses, possible containing messages from many partitions, into a queue that is read by the processing thread (AKA your code). The main way to control the fetcher thread is to control the size of those responses and the size of the queue.
 
-* `max_bytes` — The maximum size of message sets returned from a single fetch request.
-* `max_fetch_queue_size` — The maximum number of fetch responses to keep in the queue. Once reached, the fetcher will back off until the queue gets back down under to limit.
+* `max_bytes` — Maximum amount of data the broker shall return for a Fetch request.
+* `min_message_queue_size` — The minimum number of messages in the local consumer queue.
 
-The memory usage limit is roughly estimated as `max_bytes * max_fetch_queue_size`, plus whatever your application uses.
+The memory usage limit is roughly estimated as `max_bytes * min_message_queue_size`, plus whatever your application uses.
 
 #### SSL encryption, authentication & authorization
 
-* `ssl_ca_cert` – A valid SSL certificate authority, as a string.
-* `ssl_ca_cert_file_path` - The path to a valid SSL certificate authority file.
-* `ssl_client_cert` – A valid SSL client certificate, as a string.
-* `ssl_client_cert_key` – A valid SSL client certificate key, as a string.
-* `ssl_client_cert_key_password` – The password for the client cert key, as a string (optional).
+* `security_protocol` – Protocol used to communicate with brokers (`:ssl`)
+* `ssl_ca_location` – File or directory path to CA certificate(s) for verifying the broker's key
+* `ssl_crl_location` – Path to CRL for verifying broker's certificate validity
+* `ssl_keystore_location` – Path to client's keystore (PKCS#12) used for authentication
+* `ssl_keystore_password` – Client's keystore (PKCS#12) password
 
 #### SASL encryption, authentication & authorization
 
-Racecar has support for using SASL to authenticate clients using either the GSSAPI or PLAIN mechanism.
+Racecar has support for using SASL to authenticate clients using either the GSSAPI or PLAIN mechanism either via plaintext or SSL connection.
 
-If using GSSAPI:
+* `security_protocol` – Protocol used to communicate with brokers (`:sasl_plaintext` `:sasl_ssl`)
+* `sasl_mechanism` – SASL mechanism to use for authentication (`GSSAPI` `PLAIN` `SCRAM-SHA-256` `SCRAM-SHA-512`)
 
-* `sasl_gssapi_principal` – The GSSAPI principal.
-* `sasl_gssapi_keytab` – Optional GSSAPI keytab.
-
-If using PLAIN:
-
-* `sasl_plain_authzid` – The authorization identity to use.
-* `sasl_plain_username` – The username used to authenticate.
-* `sasl_plain_password` – The password used to authenticate.
-
-If using SCRAM:
-
-* `sasl_scram_username` – The username used to authenticate.
-* `sasl_scram_password` – The password used to authenticate.
-* `sasl_scram_mechanism` – The SCRAM mechanism to use, either `sha256` or `sha512`.
-
-If using OAUTHBEARER:
-
-* `sasl_oauth_token_provider`- In order to authenticate using OAUTHBEARER, you must set the client with an instance of a class that implements a token method (the interface is described in Kafka::Sasl::OAuth) which returns an ID/Access token.
-This mechanism is supported in kafka >= 2.0.0 as of KIP-255.
-
-See more at [here](https://github.com/zendesk/ruby-kafka/tree/master#oauthbearer).
-
-NOTE: `sasl_oauth_token_provider` only works using the `config/racecar.rb` configuration file.
+* `sasl_kerberos_principal` – This client's Kerberos principal name
+* `sasl_kerberos_kinit_cmd` – Full kerberos kinit command string, `%{config.prop.name}` is replaced by corresponding config object value, `%{broker.name}` returns the broker's hostname
+* `sasl_kerberos_keytab` – Path to Kerberos keytab file. Uses system default if not set
+* `sasl_kerberos_min_time_before_relogin` – Minimum time in milliseconds between key refresh attempts
+* `sasl_username` – SASL username for use with the PLAIN and SASL-SCRAM-.. mechanism
+* `sasl_password` – SASL password for use with the PLAIN and SASL-SCRAM-.. mechanism
 
 #### Producing messages
 
 These settings are related to consumers that _produce messages to Kafka_.
 
 * `producer_compression_codec` – If defined, Racecar will compress messages before writing them to Kafka. The codec needs to be one of `gzip`, `lz4`, or `snappy`, either as a Symbol or a String.
-* `partitioner` - The algorithm used to calculate message destination partitions. `crc32` is the default, `murmur2` is also supported. Can be passed as symbol or a string.
 
 #### Datadog monitoring
 
@@ -343,9 +319,6 @@ Racecar supports configuring ruby-kafka's [Datadog](https://www.datadoghq.com/)
 * `datadog_namespace` – The namespace to use for Datadog metrics.
 * `datadog_tags` – Tags that should always be set on Datadog metrics.
 
-#### Consumers Without Rails ####
-
-By default, if Rails is detected, it will be automatically started when the consumer is started. There are cases where you might not want or need Rails. You can pass the `--without-rails` option when starting the consumer and Rails won't be started.
 
 ### Testing consumers
 

data/lib/ensure_hash_compact.rb

@@ -0,0 +1,10 @@
+# only needed when ruby < 2.4 and not using active support
+
+unless {}.respond_to? :compact
+  # https://github.com/rails/rails/blob/fc5dd0b85189811062c85520fd70de8389b55aeb/activesupport/lib/active_support/core_ext/hash/compact.rb
+  class Hash
+    def compact
+      select { |_, value| !value.nil? }
+    end
+  end
+end

data/lib/racecar/cli.rb

@@ -23,7 +23,7 @@ module Racecar
     def run
       $stderr.puts "=> Starting Racecar consumer #{consumer_name}..."
 
-      RailsConfigFileLoader.load! unless config.without_rails?
+      RailsConfigFileLoader.load!
 
       if File.exist?("config/racecar.rb")
         require "./config/racecar"
@@ -102,12 +102,7 @@ module Racecar
         opts.on("-r", "--require STRING", "Require a library before starting the consumer") do |lib|
           $LOAD_PATH.unshift(Dir.pwd) unless load_path_modified
           load_path_modified = true
-          begin
-            require lib
-          rescue => e
-            $stderr.puts "=> #{lib} failed to load: #{e.message}"
-            exit
-          end
+          require lib
         end
 
         opts.on("-l", "--log STRING", "Log to the specified file") do |logfile|

data/lib/racecar/config.rb

@@ -13,17 +13,23 @@ module Racecar
     desc "How frequently to commit offset positions"
     float :offset_commit_interval, default: 10
 
-    desc "How many messages to process before forcing a checkpoint"
-    integer :offset_commit_threshold, default: 0
-
     desc "How often to send a heartbeat message to Kafka"
     float :heartbeat_interval, default: 10
 
-    desc "How long committed offsets will be retained."
-    integer :offset_retention_time
+    desc "The minimum number of messages in the local consumer queue"
+    integer :min_message_queue_size, default: 2000
+
+    desc "Kafka consumer configuration options, separated with '=' -- https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md"
+    list :consumer, default: []
+
+    desc "Kafka producer configuration options, separated with '=' -- https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md"
+    list :producer, default: []
 
-    desc "The maximum number of fetch responses to keep queued before processing"
-    integer :max_fetch_queue_size, default: 10
+    desc "The maxium number of messages that get consumed within one batch"
+    integer :fetch_messages, default: 1000
+
+    desc "Automatically store offset of last message provided to application"
+    boolean :synchronous_commits, default: false
 
     desc "How long to pause a partition for if the consumer raises an exception while processing a message -- set to -1 to pause indefinitely"
     float :pause_timeout, default: 10
@@ -37,8 +43,8 @@ module Racecar
     desc "The idle timeout after which a consumer is kicked out of the group"
     float :session_timeout, default: 30
 
-    desc "How long to wait when trying to connect to a Kafka broker"
-    float :connect_timeout, default: 10
+    desc "The maximum time between two message fetches before the consumer is kicked out of the group (in seconds)"
+    integer :max_poll_interval, default: 5*60
 
     desc "How long to wait when trying to communicate with a Kafka broker"
     float :socket_timeout, default: 30
@@ -46,7 +52,7 @@ module Racecar
     desc "How long to allow the Kafka brokers to wait before returning messages"
     float :max_wait_time, default: 1
 
-    desc "The maximum size of message sets returned from a single fetch"
+    desc "Maximum amount of data the broker shall return for a Fetch request"
     integer :max_bytes, default: 10485760
 
     desc "A prefix used when generating consumer group names"
@@ -61,47 +67,44 @@ module Racecar
     desc "The log level for the Racecar logs"
     string :log_level, default: "info"
 
-    desc "A valid SSL certificate authority"
-    string :ssl_ca_cert
-
-    desc "The path to a valid SSL certificate authority file"
-    string :ssl_ca_cert_file_path
+    desc "Protocol used to communicate with brokers"
+    symbol :security_protocol, allowed_values: %i{plaintext ssl sasl_plaintext sasl_ssl}
 
-    desc "A valid SSL client certificate"
-    string :ssl_client_cert
+    desc "File or directory path to CA certificate(s) for verifying the broker's key"
+    string :ssl_ca_location
 
-    desc "A valid SSL client certificate key"
-    string :ssl_client_cert_key
+    desc "Path to CRL for verifying broker's certificate validity"
+    string :ssl_crl_location
 
-    desc "The password for the SSL client certificate key"
-    string :ssl_client_cert_key_password
+    desc "Path to client's keystore (PKCS#12) used for authentication"
+    string :ssl_keystore_location
 
-    desc "Support for using the CA certs installed on your system by default for SSL. More info, see: https://github.com/zendesk/ruby-kafka/pull/521"
-    boolean :ssl_ca_certs_from_system, default: false
+    desc "Client's keystore (PKCS#12) password"
+    string :ssl_keystore_password
 
-    desc "The GSSAPI principal"
-    string :sasl_gssapi_principal
+    desc "SASL mechanism to use for authentication"
+    string :sasl_mechanism, allowed_values: %w{GSSAPI PLAIN SCRAM-SHA-256 SCRAM-SHA-512}
 
-    desc "Optional GSSAPI keytab"
-    string :sasl_gssapi_keytab
+    desc "Kerberos principal name that Kafka runs as, not including /hostname@REALM"
+    string :sasl_kerberos_service_name
 
-    desc "The authorization identity to use"
-    string :sasl_plain_authzid
+    desc "This client's Kerberos principal name"
+    string :sasl_kerberos_principal
 
-    desc "The username used to authenticate"
-    string :sasl_plain_username
+    desc "Full kerberos kinit command string, %{config.prop.name} is replaced by corresponding config object value, %{broker.name} returns the broker's hostname"
+    string :sasl_kerberos_kinit_cmd
 
-    desc "The password used to authenticate"
-    string :sasl_plain_password
+    desc "Path to Kerberos keytab file. Uses system default if not set"
+    string :sasl_kerberos_keytab
 
-    desc "The username used to authenticate"
-    string :sasl_scram_username
+    desc "Minimum time in milliseconds between key refresh attempts"
+    integer :sasl_kerberos_min_time_before_relogin
 
-    desc "The password used to authenticate"
-    string :sasl_scram_password
+    desc "SASL username for use with the PLAIN and SASL-SCRAM-.. mechanism"
+    string :sasl_username
 
-    desc "The SCRAM mechanism to use, either `sha256` or `sha512`"
-    string :sasl_scram_mechanism, allowed_values: ["sha256", "sha512"]
+    desc "SASL password for use with the PLAIN and SASL-SCRAM-.. mechanism"
+    string :sasl_password
 
     desc "Whether to use SASL over SSL."
     boolean :sasl_over_ssl, default: true
@@ -113,10 +116,7 @@ module Racecar
     boolean :daemonize, default: false
 
     desc "The codec used to compress messages with"
-    symbol :producer_compression_codec
-
-    desc "The partitioner used to calculate message destination partition"
-    symbol :partitioner, default: :crc32, allowed_values: [:crc32, :murmur2]
+    symbol :producer_compression_codec, allowed_values: %i{none lz4 snappy gzip}
 
     desc "Enable Datadog metrics"
     boolean :datadog_enabled, default: false
@@ -136,17 +136,11 @@ module Racecar
     desc "Whether to check the server certificate is valid for the hostname"
     boolean :ssl_verify_hostname, default: true
 
-    desc "Whether to boot Rails when starting the consumer"
-    boolean :without_rails, default: false
-
     # The error handler must be set directly on the object.
     attr_reader :error_handler
 
     attr_accessor :subscriptions, :logger
 
-    # The OAUTHBEARER token provider class.
-    attr_accessor :sasl_oauth_token_provider
-
     def initialize(env: ENV)
       super(env: env)
       @error_handler = proc {}
@@ -170,17 +164,9 @@ module Racecar
         raise ConfigError, "`socket_timeout` must be longer than `max_wait_time`"
       end
 
-      if connect_timeout <= max_wait_time
-        raise ConfigError, "`connect_timeout` must be longer than `max_wait_time`"
-      end
-
       if max_pause_timeout && !pause_with_exponential_backoff?
         raise ConfigError, "`max_pause_timeout` only makes sense when `pause_with_exponential_backoff` is enabled"
       end
-
-      if ssl_client_cert_key_password && !ssl_client_cert_key
-        raise ConfigError, "`ssl_client_cert_key_password` must be used in conjunction with `ssl_client_cert_key`"
-      end
     end
 
     def load_consumer_class(consumer_class)
@@ -196,12 +182,47 @@ module Racecar
 
       self.subscriptions = consumer_class.subscriptions
       self.max_wait_time = consumer_class.max_wait_time || self.max_wait_time
-      self.offset_retention_time = consumer_class.offset_retention_time || self.offset_retention_time
       self.pidfile ||= "#{group_id}.pid"
     end
 
     def on_error(&handler)
       @error_handler = handler
     end
+
+    def rdkafka_consumer
+      consumer_config = consumer.map do |param|
+        param.split("=", 2).map(&:strip)
+      end.to_h
+      consumer_config.merge!(rdkafka_security_config)
+      consumer_config
+    end
+
+    def rdkafka_producer
+      producer_config = producer.map do |param|
+        param.split("=", 2).map(&:strip)
+      end.to_h
+      producer_config.merge!(rdkafka_security_config)
+      producer_config
+    end
+
+    private
+
+    def rdkafka_security_config
+      {
+        "security.protocol" => security_protocol,
+        "ssl.ca.location" => ssl_ca_location,
+        "ssl.crl.location" => ssl_crl_location,
+        "ssl.keystore.location" => ssl_keystore_location,
+        "ssl.keystore.password" => ssl_keystore_password,
+        "sasl.mechanism" => sasl_mechanism,
+        "sasl.kerberos.service.name" => sasl_kerberos_service_name,
+        "sasl.kerberos.principal" => sasl_kerberos_principal,
+        "sasl.kerberos.kinit.cmd" => sasl_kerberos_kinit_cmd,
+        "sasl.kerberos.keytab" => sasl_kerberos_keytab,
+        "sasl.kerberos.min.time.before.relogin" => sasl_kerberos_min_time_before_relogin,
+        "sasl.username" => sasl_username,
+        "sasl.password" => sasl_password,
+      }.compact
+    end
   end
 end

data/lib/racecar/consumer.rb

@@ -1,11 +1,11 @@
 module Racecar
   class Consumer
-    Subscription = Struct.new(:topic, :start_from_beginning, :max_bytes_per_partition)
+    Subscription = Struct.new(:topic, :start_from_beginning, :max_bytes_per_partition, :additional_config)
 
     class << self
       attr_accessor :max_wait_time
       attr_accessor :group_id
-      attr_accessor :offset_retention_time
+      attr_accessor :producer, :consumer
 
       def subscriptions
         @subscriptions ||= []
@@ -20,29 +20,50 @@ module Racecar
       #   of each partition.
       # @param max_bytes_per_partition [Integer] the maximum number of bytes to fetch from
       #   each partition at a time.
+      # @param additional_config [Hash] Configuration properties for consumer.
+      #   See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
       # @return [nil]
-      def subscribes_to(*topics, start_from_beginning: true, max_bytes_per_partition: 1048576)
+      def subscribes_to(*topics, start_from_beginning: true, max_bytes_per_partition: 1048576, additional_config: {})
         topics.each do |topic|
-          subscriptions << Subscription.new(topic, start_from_beginning, max_bytes_per_partition)
+          subscriptions << Subscription.new(topic, start_from_beginning, max_bytes_per_partition, additional_config)
         end
       end
     end
 
-    def configure(consumer:, producer:)
-      @_consumer = consumer
-      @_producer = producer
+    def configure(producer:, consumer:, instrumenter:)
+      @producer = producer
+      @consumer = consumer
+      @instrumenter = instrumenter
     end
 
     def teardown; end
 
+    # Delivers messages that got produced.
+    def deliver!
+      @delivery_handles ||= []
+      @delivery_handles.each(&:wait)
+      @delivery_handles.clear
+    end
+
     protected
 
-    def heartbeat
-      @_consumer.trigger_heartbeat
+    # https://github.com/appsignal/rdkafka-ruby#producing-messages
+    def produce(topic:, payload:, key:, headers: nil)
+      @delivery_handles ||= []
+
+      extra_info = {
+        value:       payload,
+        key:         key,
+        topic:       topic,
+        create_time: Time.now,
+      }
+      @instrumenter.instrument("produce_message.racecar", extra_info) do
+        @delivery_handles << @producer.produce(topic: topic, payload: payload, key: key, headers: headers)
+      end
     end
 
-    def produce(value, **options)
-      @_producer.produce(value, **options)
+    def heartbeat
+      warn "DEPRECATION WARNING: Manual heartbeats are not supported and not needed with librdkafka."
     end
   end
 end