racecar 2.1.0 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 90dddff7c52ef8440ab154be364aacd71f687e7121eaa3456e6b580c3717c294
-  data.tar.gz: '092eb0699d0e60537840cd0114497003ee32e18d611a5b386d430fdfeabd7e03'
+  metadata.gz: 6380c598cedfca4662aa4c0edf6c55b8a0641d972dad715e3b1e961ac06e4d4f
+  data.tar.gz: 19a30f8515b82f9cbfa783319a76059688ef6096a4dd5e5a5673d6d0c101878c
 SHA512:
-  metadata.gz: cd4d053961a7f228aeda34bb01e758f9caa6356cf4fb7a3e773c14b9976c19ab30587ce23a010e31b115f30d85999bb38c120e0313b99b43cc744b71f273fb3c
-  data.tar.gz: d0d1865ef2c7adcf94e1bf9c551d7bcdc02a223ee50fbfe4e7293b89629ca7853c55c5cfe144ce879c8aa28c3b8939dcd87fc01c54d441369cfe3c4a186643f3
+  metadata.gz: 66c6013d77d63a121673e7d4e49fb87b3b1520a3e80a9b564a116b284dbd262ead6fd5b570afa4fa55d77822ac6a6ab31a86917b08630aad2fc9af2f7c08d171
+  data.tar.gz: ce9c89a671478a806dce62979bb695a2673cd598acfb3d2547194a9367a12c96d5f34d38c32f79a3ae809f9eb79303e5330aec4f5e0dc2a23f6f6ffc1e286817

data/CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## Unreleased
 
+## racecar v2.1.1
+
+* [Bugfix] Close RdKafka consumer in ConsumerSet#reset_current_consumer to prevent memory leak (#196)
+* [Bugfix] `poll`/`batch_poll` would not retry in edge cases and raise immediately. They still honor the `max_wait_time` setting, but might return no messages instead and only retry on their next call. ([#177](https://github.com/zendesk/racecar/pull/177))
+
 ## racecar v2.1.0
 
 * Bump rdkafka to 0.8.0 (#191)

data/README.md

@@ -50,9 +50,7 @@ This will add a config file in `config/racecar.yml`.
 
 ## Usage
 
-Racecar is built for simplicity of development and operation. If you need more flexibility, it's quite straightforward to build your own Kafka consumer executables using [ruby-kafka](https://github.com/zendesk/ruby-kafka#consuming-messages-from-kafka) directly.
-
-First, a short introduction to the Kafka consumer concept as well as some basic background on Kafka.
+Racecar is built for simplicity of development and operation. First, a short introduction to the Kafka consumer concept as well as some basic background on Kafka.
 
 Kafka stores messages in so-called _partitions_ which are grouped into _topics_. Within a partition, each message gets a unique offset.
 
@@ -226,7 +224,7 @@ You can set message headers by passing a `headers:` option with a Hash of header
 
 Racecar provides a flexible way to configure your consumer in a way that feels at home in a Rails application. If you haven't already, run `bundle exec rails generate racecar:install` in order to generate a config file. You'll get a separate section for each Rails environment, with the common configuration values in a shared `common` section.
 
-**Note:** many of these configuration keys correspond directly to similarly named concepts in [ruby-kafka](https://github.com/zendesk/ruby-kafka); for more details on low-level operations, read that project's documentation.
+**Note:** many of these configuration keys correspond directly to similarly named concepts in [rdkafka-ruby](https://github.com/appsignal/rdkafka-ruby); for more details on low-level operations, read that project's documentation.
 
 It's also possible to configure Racecar using environment variables. For any given configuration key, there should be a corresponding environment variable with the prefix `RACECAR_`, in upper case. For instance, in order to configure the client id, set `RACECAR_CLIENT_ID=some-id` in the process in which the Racecar consumer is launched. You can set `brokers` by passing a comma-separated list, e.g. `RACECAR_BROKERS=kafka1:9092,kafka2:9092,kafka3:9092`.
 
@@ -273,7 +271,7 @@ All timeouts are defined in number of seconds.
 
 Kafka is _really_ good at throwing data at consumers, so you may want to tune these variables in order to avoid ballooning your process' memory or saturating your network capacity.
 
-Racecar uses ruby-kafka under the hood, which fetches messages from the Kafka brokers in a background thread. This thread pushes fetch responses, possible containing messages from many partitions, into a queue that is read by the processing thread (AKA your code). The main way to control the fetcher thread is to control the size of those responses and the size of the queue.
+Racecar uses [rdkafka-ruby](https://github.com/appsignal/rdkafka-ruby) under the hood, which fetches messages from the Kafka brokers in a background thread. This thread pushes fetch responses, possible containing messages from many partitions, into a queue that is read by the processing thread (AKA your code). The main way to control the fetcher thread is to control the size of those responses and the size of the queue.
 
 * `max_bytes` — Maximum amount of data the broker shall return for a Fetch request.
 * `min_message_queue_size` — The minimum number of messages in the local consumer queue.
@@ -313,7 +311,7 @@ These settings are related to consumers that _produce messages to Kafka_.
 
 #### Datadog monitoring
 
-Racecar supports configuring ruby-kafka's [Datadog](https://www.datadoghq.com/) monitoring integration. If you're running a normal Datadog agent on your host, you just need to set `datadog_enabled` to `true`, as the rest of the settings come with sane defaults.
+Racecar supports [Datadog](https://www.datadoghq.com/) monitoring integration. If you're running a normal Datadog agent on your host, you just need to set `datadog_enabled` to `true`, as the rest of the settings come with sane defaults.
 
 * `datadog_enabled` – Whether Datadog monitoring is enabled (defaults to `false`).
 * `datadog_host` – The host running the Datadog agent.
@@ -492,6 +490,8 @@ In order to safely upgrade from Racecar v1 to v2, you need to completely shut do
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rspec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
+The integration tests run against a Kafka instance that is not automatically started from within `rspec`. You can set one up using the provided `docker-compose.yml` by running `docker-compose up`.
+
 
 ## Contributing
 
@@ -500,9 +500,9 @@ Bug reports and pull requests are welcome on [GitHub](https://github.com/zendesk
 
 ## Support and Discussion
 
-If you've discovered a bug, please file a [Github issue](https://github.com/zendesk/racecar/issues/new), and make sure to include all the relevant information, including the version of Racecar, ruby-kafka, and Kafka that you're using.
+If you've discovered a bug, please file a [Github issue](https://github.com/zendesk/racecar/issues/new), and make sure to include all the relevant information, including the version of Racecar, rdkafka-ruby, and Kafka that you're using.
 
-If you have other questions, or would like to discuss best practises, how to contribute to the project, or any other ruby-kafka related topic, [join our Slack team](https://ruby-kafka-slack.herokuapp.com/)!
+If you have other questions, or would like to discuss best practises, or how to contribute to the project, [join our Slack team](https://ruby-kafka-slack.herokuapp.com/)!
 
 
 ## Copyright and license

data/lib/racecar/consumer_set.rb

@@ -12,49 +12,39 @@ module Racecar
       @consumers = []
       @consumer_id_iterator = (0...@config.subscriptions.size).cycle
 
+      @previous_retries = 0
+
       @last_poll_read_nil_message = false
     end
 
-    def poll(timeout_ms)
-      maybe_select_next_consumer
-      started_at ||= Time.now
-      try ||= 0
-      remain ||= timeout_ms
-
-      msg = remain <= 0 ? nil : current.poll(remain)
-    rescue Rdkafka::RdkafkaError => e
-      wait_before_retry_ms = 100 * (2**try) # 100ms, 200ms, 400ms, …
-      try += 1
-      raise if try >= MAX_POLL_TRIES || remain <= wait_before_retry_ms
-
-      @logger.error "(try #{try}): Error for topic subscription #{current_subscription}: #{e}"
-
-      case e.code
-      when :max_poll_exceeded, :transport # -147, -195
-        reset_current_consumer
-      end
-
-      remain = remaining_time_ms(timeout_ms, started_at)
-      raise if remain <= wait_before_retry_ms
-
-      sleep wait_before_retry_ms/1000.0
-      retry
-    ensure
-      @last_poll_read_nil_message = true if msg.nil?
+    def poll(max_wait_time_ms = @config.max_wait_time)
+      batch_poll(max_wait_time_ms, 1).first
     end
 
-    # XXX: messages are not guaranteed to be from the same partition
-    def batch_poll(timeout_ms)
-      @batch_started_at = Time.now
-      @messages = []
-      while collect_messages_for_batch? do
-        remain = remaining_time_ms(timeout_ms, @batch_started_at)
-        break if remain <= 0
-        msg = poll(remain)
+    # batch_poll collects messages until any of the following occurs:
+    # - max_wait_time_ms time has passed
+    # - max_messages have been collected
+    # - a nil message was polled (end of topic, Kafka stalled, etc.)
+    #
+    # The messages are from a single topic, but potentially from more than one partition.
+    #
+    # Any errors during polling are retried in an exponential backoff fashion. If an error
+    # occurs, but there is no time left for a backoff and retry, it will return the
+    # already collected messages and only retry on the next call.
+    def batch_poll(max_wait_time_ms = @config.max_wait_time, max_messages = @config.fetch_messages)
+      started_at = Time.now
+      remain_ms = max_wait_time_ms
+      maybe_select_next_consumer
+      messages = []
+
+      while remain_ms > 0 && messages.size < max_messages
+        remain_ms = remaining_time_ms(max_wait_time_ms, started_at)
+        msg = poll_with_retries(remain_ms)
         break if msg.nil?
-        @messages << msg
+        messages << msg
       end
-      @messages
+
+      messages
     end
 
     def store_offset(message)
@@ -125,6 +115,52 @@ module Racecar
 
     private
 
+    # polls a single message from the current consumer, retrying errors with exponential
+    # backoff. The sleep time is capped by max_wait_time_ms. If there's enough time budget
+    # left, it will retry before returning. If there isn't, the retry will only occur on
+    # the next call. It tries up to MAX_POLL_TRIES before passing on the exception.
+    def poll_with_retries(max_wait_time_ms)
+      try ||= @previous_retries
+      @previous_retries = 0
+      started_at ||= Time.now
+      remain_ms = remaining_time_ms(max_wait_time_ms, started_at)
+
+      wait_ms = try == 0 ? 0 : 50 * (2**try) # 0ms, 100ms, 200ms, 400ms, …
+      if wait_ms >= max_wait_time_ms && remain_ms > 1
+        @logger.debug "Capping #{wait_ms}ms to #{max_wait_time_ms-1}ms."
+        sleep (max_wait_time_ms-1)/1000.0
+        remain_ms = 1
+      elsif wait_ms >= remain_ms
+        @logger.error "Only #{remain_ms}ms left, but want to wait for #{wait_ms}ms before poll. Will retry on next call."
+        @previous_retries = try
+        return nil
+      elsif wait_ms > 0
+        sleep wait_ms/1000.0
+        remain_ms -= wait_ms
+      end
+
+      poll_current_consumer(remain_ms)
+    rescue Rdkafka::RdkafkaError => e
+      try += 1
+      @instrumenter.instrument("poll_retry", try: try, rdkafka_time_limit: remain_ms, exception: e)
+      @logger.error "(try #{try}/#{MAX_POLL_TRIES}): Error for topic subscription #{current_subscription}: #{e}"
+      raise if try >= MAX_POLL_TRIES
+      retry
+    end
+
+    # polls a message for the current consumer, handling any API edge cases.
+    def poll_current_consumer(max_wait_time_ms)
+      msg = current.poll(max_wait_time_ms)
+    rescue Rdkafka::RdkafkaError => e
+      case e.code
+      when :max_poll_exceeded, :transport # -147, -195
+        reset_current_consumer
+      end
+      raise
+    ensure
+      @last_poll_read_nil_message = msg.nil?
+    end
+
     def find_consumer_by(topic, partition)
       each do |consumer|
         tpl = consumer.assignment.to_h
@@ -142,7 +178,12 @@ module Racecar
     end
 
     def reset_current_consumer
-      @consumers[@consumer_id_iterator.peek] = nil
+      current_consumer_id = @consumer_id_iterator.peek
+      @logger.info "Resetting consumer with id: #{current_consumer_id}"
+
+      consumer = @consumers[current_consumer_id]
+      consumer.close unless consumer.nil?
+      @consumers[current_consumer_id] = nil
     end
 
     def maybe_select_next_consumer
@@ -162,11 +203,6 @@ module Racecar
       @logger.debug "Nothing to commit."
     end
 
-    def collect_messages_for_batch?
-      @messages.size < @config.fetch_messages &&
-      (Time.now - @batch_started_at) < @config.max_wait_time
-    end
-
     def rdkafka_config(subscription)
       # https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
       config = {

data/lib/racecar/ctl.rb

@@ -100,7 +100,8 @@ module Racecar
         "client.id":         Racecar.config.client_id,
       }.merge(Racecar.config.rdkafka_producer)).producer
 
-      producer.produce(payload: message.value, key: message.key, topic: message.topic).wait
+      handle = producer.produce(payload: message.value, key: message.key, topic: message.topic)
+      handle.wait(max_wait_timeout: 5)
 
       $stderr.puts "=> Delivered message to Kafka cluster"
     end

data/lib/racecar/datadog.rb

@@ -157,6 +157,15 @@ module Racecar
         end
       end
 
+      def poll_retry(event)
+        tags = {
+          client: event.payload.fetch(:client_id),
+          group_id: event.payload.fetch(:group_id),
+        }
+        rdkafka_error_code = event.payload.fetch(:exception).code.to_s.gsub(/\W/, '')
+        increment("consumer.poll.rdkafka_error.#{rdkafka_error_code}", tags: tags)
+      end
+
       def main_loop(event)
         tags = {
           client: event.payload.fetch(:client_id),

data/lib/racecar/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Racecar
-  VERSION = "2.1.0"
+  VERSION = "2.1.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 2.1.1
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-09-30 00:00:00.000000000 Z
+date: 2020-11-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: king_konf