racecar 2.9.0 → 2.10.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b394d31edcb9c83e562811ba748b6c6915388d0227a4e7425aa9aa13f64e3890
-  data.tar.gz: 1dfcd7046b6d932f716246d0685589c1635af115a4e126067c9775b13fe05464
+  metadata.gz: e0057ec2861db783b83e34a63c0a09a0be476ed6cb57b760dea3e2cc5ef57d1f
+  data.tar.gz: 22dac9ed21af6f9abfd50f22d9dc86032a68af556e1be77bff2f59cbf505ccf2
 SHA512:
-  metadata.gz: 59043be21e411e680c11815b583236556413a539f5aa9508a460eefe82cee6199f56a30b9fa07a3f206886d5dc811b210cea6130813a7073a68fe6612343ca0d
-  data.tar.gz: d6692b9bb7cdc27efe5272a10fa1d9061920084a7b517699e5133f5e31e9fd50415e72ee40550e07e5a2fed028aeabb741422fef47372c7a53e0a96bdfc1b090
+  metadata.gz: f3baeec4de02a68b55e4a302d2b713de67d0245564aae57cbca363aec8b34633f8dd2220d44c1e212347857d7fde35217644dae8afdbde06103e11a52394997d
+  data.tar.gz: 66a2168ca6f7a4ec7a6d63e8cf767397502c02487d87445b59e4d7fbdd0269abee3279019e36ab1f343e246d4c58146e951ad521b1bc881a4cb60b3f35076b6e

data/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## Unreleased
 
+## 2.10.0.beta1
+
+* Bump rdkafka gem version to 0.13.0
+* Support cooperative-sticky
+* Instrument produce delivery errors
+* Fix config load for liveness probe
+* Send exceptions to `process_batch` instrumenter
+* Docker test fixes
 * Test with Ruby 3.2
 
 ## v2.9.0, v2.9.0.beta1

data/Dockerfile

@@ -1,4 +1,4 @@
-FROM circleci/ruby:2.7.2
+FROM cimg/ruby:2.7.8
 
 RUN sudo apt-get update
 RUN sudo apt-get install docker

data/Gemfile.lock

@@ -1,9 +1,9 @@
 PATH
   remote: .
   specs:
-    racecar (2.9.0.beta1)
+    racecar (2.9.0)
       king_konf (~> 1.0.0)
-      rdkafka (~> 0.12.0)
+      rdkafka (~> 0.13.0)
 
 GEM
   remote: https://rubygems.org/
@@ -33,7 +33,7 @@ GEM
       byebug (~> 11.0)
       pry (>= 0.13, < 0.15)
     rake (13.0.6)
-    rdkafka (0.12.0)
+    rdkafka (0.13.0)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
       rake (> 12)

data/README.md

@@ -476,6 +476,8 @@ With Foreman, you can easily run these processes locally by executing `foreman r
 
 If you run your applications in Kubernetes, use the following [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) spec as a starting point:
 
+##### Recreate Strategy
+
 ```yaml
 apiVersion: apps/v1
 kind: Deployment
@@ -483,8 +485,8 @@ metadata:
   name: my-racecar-deployment
   labels:
     app: my-racecar
-spec:
-  replicas: 3 # <-- this will give us three consumers in the group.
+spec
+  replicas: 4 # <-- this is a good value if you have a multliple of 4 partitions
   selector:
     matchLabels:
       app: my-racecar
@@ -506,9 +508,33 @@ spec:
               value: 5
 ```
 
-The important part is the `strategy.type` value, which tells Kubernetes how to upgrade from one version of your Deployment to another. Many services use so-called _rolling updates_, where some but not all containers are replaced with the new version. This is done so that, if the new version doesn't work, the old version is still there to serve most of the requests. For Kafka consumers, this doesn't work well. The reason is that every time a consumer joins or leaves a group, every other consumer in the group needs to stop and synchronize the list of partitions assigned to each group member. So if the group is updated in a rolling fashion, this synchronization would occur over and over again, causing undesirable double-processing of messages as consumers would start only to be synchronized shortly after.
+This configuration uses the recreate strategy which completely terminates all consumers before starting new ones.
+It's simple and easy to understand but can result in significant 'downtime' where no messages are processed.
+
+##### Rolling Updates and 'sticky-cooperative' Assignment
+
+A newer alternative is to use the consumer's "cooperative-sticky" assignment strategy which allows healthy consumers to keep processing their partitions while others are terminated.
+This can be combined with a restricted rolling update to minimize processing downtime.
+
+Add to your Racecar config:
+```ruby
+Racecar.configure do |c|
+  c.partition_assignment_strategy = "cooperative-sticky"
+end
+```
+
+Replace the Kubernetes deployment strategy with:
+```yaml
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+        maxSurge: 0 # <- Never boot an excess consumer
+        maxUnavailable: 1 # <- The deploy 'rolls' one consumer at a time
+```
+
+These two configurations should be deployed together.
 
-Instead, the `Recreate` update strategy should be used. It completely tears down the existing containers before starting all of the new containers simultaneously, allowing for a single synchronization stage and a much faster, more stable deployment update.
+While `maxSurge` should always be 0, `maxUnavailable` can be increased to reduce deployment times in exchange for longer pauses in message processing.
 
 #### Liveness Probe
 
@@ -663,7 +689,7 @@ In order to introspect the configuration of a consumer process, send it the `SIG
 
 ### Upgrading from v1 to v2
 
-In order to safely upgrade from Racecar v1 to v2, you need to completely shut down your consumer group before starting it up again with the v2 Racecar dependency. In general, you should avoid rolling deploys for consumers groups, so it is likely the case that this will just work for you, but it's a good idea to check first. 
+In order to safely upgrade from Racecar v1 to v2, you need to completely shut down your consumer group before starting it up again with the v2 Racecar dependency.
 
 ### Compression
 
@@ -681,12 +707,12 @@ The integration tests run against a Kafka instance that is not automatically sta
 
 ### Running RSpec within Docker
 
-There can be behavioural inconsistencies between running the specs on your machine, and in the CI pipeline. Due to this, there is now a Dockerfile included in the project, which is based on the CircleCI ruby 2.7.2 image. This could easily be extended with more Dockerfiles to cover different Ruby versions if desired. In order to run the specs via Docker:
+There can be behavioural inconsistencies between running the specs on your machine, and in the CI pipeline. Due to this, there is now a Dockerfile included in the project, which is based on the CircleCI ruby 2.7.8 image. This could easily be extended with more Dockerfiles to cover different Ruby versions if desired. In order to run the specs via Docker:
 
 - Uncomment the `tests` service from the docker-compose.yml
 - Bring up the stack with `docker-compose up -d`
-- Execute the entire suite with `docker-compose run --rm tests rspec`
-- Execute a single spec or directory with `docker-compose run --rm tests rspec spec/integration/consumer_spec.rb`
+- Execute the entire suite with `docker-compose run --rm tests bundle exec rspec`
+- Execute a single spec or directory with `docker-compose run --rm tests bundle exec rspec spec/integration/consumer_spec.rb`
 
 Please note - your code directory is mounted as a volume, so you can make code changes without needing to rebuild
 

data/lib/racecar/config.rb

@@ -29,6 +29,9 @@ module Racecar
     desc "The minimum number of messages in the local consumer queue"
     integer :min_message_queue_size, default: 2000
 
+    desc "Which partition assignment strategy to use, range, roundrobin or cooperative-sticky. -- https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md"
+    string :partition_assignment_strategy, default: "range,roundrobin"
+
     desc "Kafka consumer configuration options, separated with '=' -- https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md"
     list :consumer, default: []
 
@@ -296,10 +299,6 @@ module Racecar
       )
     end
 
-    def rebalance_listener
-      RebalanceListener.new(self)
-    end
-
     private
 
     def rdkafka_security_config

data/lib/racecar/consumer.rb

@@ -37,6 +37,10 @@ module Racecar
           subscriptions << Subscription.new(topic, start_from_beginning, max_bytes_per_partition, additional_config)
         end
       end
+
+      # Rebalance hooks for subclasses to override
+      def on_partitions_assigned(rebalance_event); end
+      def on_partitions_revoked(rebalance_event); end
     end
 
     def configure(producer:, consumer:, instrumenter: NullInstrumenter, config: Racecar.config)

data/lib/racecar/consumer_set.rb

@@ -15,6 +15,7 @@ module Racecar
       @previous_retries = 0
 
       @last_poll_read_nil_message = false
+      @paused_tpls = Hash.new { |h, k| h[k] = {} }
     end
 
     def poll(max_wait_time_ms = @config.max_wait_time_ms)
@@ -65,14 +66,17 @@ module Racecar
 
     def close
       each_subscribed(&:close)
+      @paused_tpls.clear
     end
 
     def current
       @consumers[@consumer_id_iterator.peek] ||= begin
         consumer_config = Rdkafka::Config.new(rdkafka_config(current_subscription))
-        consumer_config.consumer_rebalance_listener = @config.rebalance_listener
-
+        listener = RebalanceListener.new(@config.consumer_class, @instrumenter)
+        consumer_config.consumer_rebalance_listener = listener
         consumer = consumer_config.consumer
+        listener.rdkafka_consumer = consumer
+
         @instrumenter.instrument('join_group') do
           consumer.subscribe current_subscription.topic
         end
@@ -98,16 +102,25 @@ module Racecar
       consumer.pause(filtered_tpl)
       fake_msg = OpenStruct.new(topic: topic, partition: partition, offset: offset)
       consumer.seek(fake_msg)
+
+      @paused_tpls[topic][partition] = [consumer, filtered_tpl]
     end
 
     def resume(topic, partition)
       consumer, filtered_tpl = find_consumer_by(topic, partition)
+
+      if !consumer && @paused_tpls[topic][partition]
+        consumer, filtered_tpl = @paused_tpls[topic][partition]
+      end
+
       if !consumer
         @logger.info "Attempted to resume #{topic}/#{partition}, but we're not subscribed to it"
         return
       end
 
       consumer.resume(filtered_tpl)
+      @paused_tpls[topic].delete(partition)
+      @paused_tpls.delete(topic) if @paused_tpls[topic].empty?
     end
 
     alias :each :each_subscribed
@@ -218,6 +231,11 @@ module Racecar
     def rdkafka_config(subscription)
       # https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
       config = {
+        # Manually store offset after messages have been processed successfully
+        # to avoid marking failed messages as committed. The call just updates
+        # a value within librdkafka and is asynchronously written to proper
+        # storage through auto commits.
+        "enable.auto.offset.store" => false,
         "auto.commit.interval.ms" => @config.offset_commit_interval * 1000,
         "auto.offset.reset"       => subscription.start_from_beginning ? "earliest" : "largest",
         "bootstrap.servers"       => @config.brokers.join(","),
@@ -233,7 +251,8 @@ module Racecar
         "queued.min.messages"     => @config.min_message_queue_size,
         "session.timeout.ms"      => @config.session_timeout * 1000,
         "socket.timeout.ms"       => @config.socket_timeout * 1000,
-        "statistics.interval.ms"  => @config.statistics_interval_ms
+        "statistics.interval.ms"  => @config.statistics_interval_ms,
+        "partition.assignment.strategy" => @config.partition_assignment_strategy,
       }
       config.merge! @config.rdkafka_consumer
       config.merge! subscription.additional_config

data/lib/racecar/ctl.rb

@@ -36,8 +36,10 @@ module Racecar
       require "racecar/liveness_probe"
       parse_options!(args)
 
-      if ENV["RAILS_ENV"]
-        Racecar.config.load_file("config/racecar.yml", ENV["RAILS_ENV"])
+      RailsConfigFileLoader.load! unless config.without_rails?
+
+      if File.exist?("config/racecar.rb")
+        require "./config/racecar"
       end
 
       Racecar.config.liveness_probe.check_liveness_within_interval!
@@ -136,5 +138,9 @@ module Racecar
 
       parser.parse!(args)
     end
+
+    def config
+      Racecar.config
+    end
   end
 end

data/lib/racecar/rebalance_listener.rb

@@ -1,22 +1,58 @@
 module Racecar
   class RebalanceListener
-    def initialize(config)
-      @config = config
-      @consumer_class = config.consumer_class
+    def initialize(consumer_class, instrumenter)
+      @consumer_class = consumer_class
+      @instrumenter = instrumenter
+      @rdkafka_consumer = nil
     end
 
-    attr_reader :config, :consumer_class
+    attr_writer :rdkafka_consumer
 
-    def on_partitions_assigned(_consumer, topic_partition_list)
-      consumer_class.respond_to?(:on_partitions_assigned) &&
-        consumer_class.on_partitions_assigned(topic_partition_list.to_h)
-    rescue
+    attr_reader :consumer_class, :instrumenter, :rdkafka_consumer
+    private     :consumer_class, :instrumenter, :rdkafka_consumer
+
+    def on_partitions_assigned(rdkafka_topic_partition_list)
+      event = Event.new(rdkafka_consumer: rdkafka_consumer, rdkafka_topic_partition_list: rdkafka_topic_partition_list)
+
+      instrument("partitions_assigned", partitions: event.partition_numbers) do
+        consumer_class.on_partitions_assigned(event)
+      end
+    end
+
+    def on_partitions_revoked(rdkafka_topic_partition_list)
+      event = Event.new(rdkafka_consumer: rdkafka_consumer, rdkafka_topic_partition_list: rdkafka_topic_partition_list)
+
+      instrument("partitions_revoked", partitions: event.partition_numbers) do
+        consumer_class.on_partitions_revoked(event)
+      end
+    end
+
+    private
+
+    def instrument(event, payload, &block)
+      instrumenter.instrument(event, payload, &block)
     end
 
-    def on_partitions_revoked(_consumer, topic_partition_list)
-      consumer_class.respond_to?(:on_partitions_revoked) &&
-        consumer_class.on_partitions_revoked(topic_partition_list.to_h)
-    rescue
+    class Event
+      def initialize(rdkafka_topic_partition_list:, rdkafka_consumer:)
+        @__rdkafka_topic_partition_list = rdkafka_topic_partition_list
+        @__rdkafka_consumer = rdkafka_consumer
+      end
+
+      def topic_name
+        __rdkafka_topic_partition_list.to_h.keys.first
+      end
+
+      def partition_numbers
+        __rdkafka_topic_partition_list.to_h.values.flatten.map(&:partition)
+      end
+
+      def empty?
+        __rdkafka_topic_partition_list.empty?
+      end
+
+      # API private and not guaranteed stable
+      attr_reader :__rdkafka_topic_partition_list, :__rdkafka_consumer
     end
   end
 end

data/lib/racecar/runner.rb

@@ -96,7 +96,7 @@ module Racecar
       end
     ensure
       producer.close
-      Racecar::Datadog.close if Object.const_defined?("Racecar::Datadog")
+      Racecar::Datadog.close if config.datadog_enabled
       @instrumenter.instrument("shut_down", instrumentation_payload || {})
     end
 
@@ -131,11 +131,6 @@ module Racecar
 
     def consumer
       @consumer ||= begin
-        # Manually store offset after messages have been processed successfully
-        # to avoid marking failed messages as committed. The call just updates
-        # a value within librdkafka and is asynchronously written to proper
-        # storage through auto commits.
-        config.consumer << "enable.auto.offset.store=false"
         ConsumerSet.new(config, logger, @instrumenter)
       end
     end
@@ -213,21 +208,21 @@ module Racecar
       }
 
       @instrumenter.instrument("start_process_batch", instrumentation_payload)
-      @instrumenter.instrument("process_batch", instrumentation_payload) do
-        with_pause(first.topic, first.partition, first.offset..last.offset) do |pause|
-          begin
+      with_pause(first.topic, first.partition, first.offset..last.offset) do |pause|
+        begin
+          @instrumenter.instrument("process_batch", instrumentation_payload) do
             racecar_messages = messages.map do |message|
               Racecar::Message.new(message, retries_count: pause.pauses_count)
             end
             processor.process_batch(racecar_messages)
             processor.deliver!
             consumer.store_offset(messages.last)
-          rescue => e
-            instrumentation_payload[:unrecoverable_delivery_error] = reset_producer_on_unrecoverable_delivery_errors(e)
-            instrumentation_payload[:retries_count] = pause.pauses_count
-            config.error_handler.call(e, instrumentation_payload)
-            raise e
           end
+        rescue => e
+          instrumentation_payload[:unrecoverable_delivery_error] = reset_producer_on_unrecoverable_delivery_errors(e)
+          instrumentation_payload[:retries_count] = pause.pauses_count
+          config.error_handler.call(e, instrumentation_payload)
+          raise e
         end
       end
     end

data/lib/racecar/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Racecar
-  VERSION = "2.9.0"
+  VERSION = "2.10.0.beta1"
 end

data/racecar.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
   spec.required_ruby_version = '>= 2.6'
 
   spec.add_runtime_dependency "king_konf", "~> 1.0.0"
-  spec.add_runtime_dependency "rdkafka",   "~> 0.12.0"
+  spec.add_runtime_dependency "rdkafka",   "~> 0.13.0"
 
   spec.add_development_dependency "bundler", [">= 1.13", "< 3"]
   spec.add_development_dependency "pry-byebug"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 2.9.0
+  version: 2.10.0.beta1
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2023-09-25 00:00:00.000000000 Z
+date: 2023-10-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: king_konf
@@ -31,14 +31,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.12.0
+        version: 0.13.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.12.0
+        version: 0.13.0
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -225,9 +225,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.6'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubygems_version: 3.0.3.1
 signing_key: