racecar 3.0.0.alpha.3 → 3.0.0.beta.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 043070ffc30189dd4f500a691316db7a2a55422078b7bf5f31b81ff45aa69cab
-  data.tar.gz: 73a2d774f56f7f14d18c5c43691f185ab64fd7e2f3cd6a29dfcad8b9111f00b5
+  metadata.gz: b0b740ff066f0f1aee2cd6bffac700badbbf684d2d9575c298aefe6a92a50d75
+  data.tar.gz: 022b4e5aea481beb637407cebe248c7890713a9ad64563edd316efca3f8ddfd6
 SHA512:
-  metadata.gz: d0ea838d6e6381660a88b6e1d71c95d0fbe741609dc216b18830d8a74eee78b0d56a6e05713b4f12ac28f60146b566f4f125d53b4c4bd4c52c3c13459941fac4
-  data.tar.gz: 213281a3b10ad91e3d7bcbf3a8974fbf15d8e1b5905518d3a4f13505ede101d254a433f86b1e7f875ccdad3c7d4d59fbc570b0d9db63f146615f4f454a65bd85
+  metadata.gz: fb0f983b58a63ec97ae7f7a47b42cb47ff3a03907055b2c855305918b30964983c15882f806bff173acbbff224d22763331c8503366b0b7783af3ad9e7535dac
+  data.tar.gz: 4f95b4582e091c8543aeaf683f915a55880dc622d49f67a56c8dbac160531babcff640919804e555d971022035dfb865931c8d603433c2b5748cb9c11c5b7642

data/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## Unreleased
 
+## 3.0.0
+
+* Introduce multithreaded processing: when enabled, Racecar processes each assigned partition on its own dedicated thread. Disabled by default and gated behind the `multithreaded_processing_enabled` config.
+* Refactor of the Racecar architecture: introduction of `PartitionProcessor` and `AsyncPartitionProcessor` handling the processing of messages.
+* [Racecar::Config] Add `multithreaded_processing_enabled` (default `false`) to enable multithreaded processing. Can be set via `RACECAR_MULTITHREADED_PROCESSING_ENABLED=1`.
+* [Racecar::Config] Add `multithreaded_processing_max_queue_size` (default `1000`) to cap the number of messages queued per partition before backpressure is applied.
+* [Racecar::Config] Add `multithreaded_processing_resume_threshold` (default `0.5`) controlling the queue fill ratio at which a paused partition is resumed.
+* [Racecar::Config] Add `multithreaded_processing_shutdown_timeout` (default `300`) for how long the main thread waits on each processing thread during graceful shutdown.
+* Apply backpressure when multithreaded processing is enabled: a partition is paused once its queue reaches `multithreaded_processing_max_queue_size` and resumed once it drains below `multithreaded_processing_resume_threshold` of that size.
+* Gracefully drain queued messages and exit per-partition threads on rebalance and shutdown.
+
 ## 2.12.0
 
 * Add tests against Ruby 3.4

data/Gemfile.lock

@@ -1,7 +1,8 @@
 PATH
   remote: .
   specs:
-    racecar (3.0.0.alpha.3)
+    racecar (3.0.0.beta.2)
+      concurrent-ruby (~> 1.3)
       king_konf (~> 1.0.0)
       rdkafka (>= 0.15.0)
 
@@ -81,4 +82,4 @@ DEPENDENCIES
   timecop
 
 BUNDLED WITH
-   2.6.2
+   2.6.9

data/README.md

@@ -134,6 +134,42 @@ end
 
 This is useful to do any one-off work that you wouldn't want to do for each and every message.
 
+#### Multithreaded message processing (experimental)
+
+Warning - limited battle testing in production environments; use at your own risk!
+
+By default a Racecar consumer processes all of its assigned partitions on a single thread. When `multithreaded_processing_enabled` is set, the consumer instead spins up one dedicated thread per assigned partition, so partitions are processed concurrently within a single process. This is an alternative to [parallel workers](#running-consumers-in-parallel-experimental) that avoids forking extra processes (and the associated memory overhead), at the cost of running your consumer code on multiple threads.
+
+Each partition thread gets its own instance of your consumer class, so **your consumer code does not need to be thread-safe** - a thread never shares its instance (or its instance state) with another partition. The main thread keeps polling Kafka and hands each partition's messages off to the relevant thread via a bounded queue; if a thread falls behind and its queue fills up, the partition is paused until the queue drains, applying backpressure rather than growing memory without bound.
+
+**Warning:** the number of threads scales with the number of assigned partitions, which can be large. Since each thread runs its own consumer instance, every resource that instance acquires (database connections, file handles, network sockets, HTTP clients, etc.) is multiplied by the number of partition threads. Make sure your consumer releases any resource it grabs and that any connection pools or other shared limits are sized to accommodate the resulting concurrency.
+
+Enable it via config (or the `RACECAR_MULTITHREADED_PROCESSING_ENABLED=1` environment variable):
+
+```ruby
+Racecar.configure do |config|
+  config.multithreaded_processing_enabled = true
+end
+
+class ResizeImagesConsumer < Racecar::Consumer
+  subscribes_to "images"
+
+  def process(message)
+    # This runs on a thread dedicated to message.partition.
+    # @state below is private to this partition's thread.
+    @state ||= {}
+    Image.resize(message.value)
+  end
+end
+```
+
+The behaviour can be tuned with the following options:
+
+- `multithreaded_processing_enabled` – Enable per-partition threads. Default is `false`.
+- `multithreaded_processing_max_queue_size` – Maximum number of queued message batches per partition before the partition is paused to apply backpressure. Default is `1000`.
+- `multithreaded_processing_resume_threshold` – A paused partition is resumed once its queue drains below this fraction of `multithreaded_processing_max_queue_size`. Default is `0.5` (50%).
+- `multithreaded_processing_shutdown_timeout` – How many seconds to wait for each partition thread to finish during graceful shutdown. Default is `300`.
+
 #### Setting the starting position
 
 When a consumer is started for the first time, it needs to decide where in each partition to start. By default, it will start at the _beginning_, meaning that all past messages will be processed. If you want to instead start at the _end_ of each partition, change your `subscribes_to` like this:

data/lib/racecar/async_partition_processor.rb

@@ -7,12 +7,14 @@ module Racecar
   class AsyncPartitionProcessor
     attr_reader :thread
 
-    THREAD_KEY_IDENTIFIER = 'racecar_topic_partition_identifier'.freeze
-
     def self.thread_key(topic, partition)
       "#{topic}/#{partition}"
     end
 
+    def thread_key
+      self.class.thread_key(@topic, @partition)
+    end
+
     def initialize(topic:, partition:, logger:, config:, consumer:, consumer_class:, instrumenter:, rdkafka_consumer:)
       @topic      = topic
       @partition  = partition
@@ -70,7 +72,6 @@ module Racecar
         rdkafka_consumer: @rdkafka_consumer,
       )
       @queue  = Queue.new
-      @thread = nil
 
       use_process_batch = consumer_class.method_defined?(:process_batch)
 
@@ -90,7 +91,6 @@ module Racecar
     def spawn_thread(&block)
       @thread = Thread.new do
         Thread.current.name = "Racecar thread for #{thread_key}"
-        Thread.current[AsyncPartitionProcessor::THREAD_KEY_IDENTIFIER] = thread_key
         main_processing_loop(block)
       end
     end
@@ -121,10 +121,6 @@ module Racecar
       end
     end
 
-    def thread_key
-      self.class.thread_key(@topic, @partition)
-    end
-
     def main_processing_loop(block)
       loop do
         msgs = @queue.pop

data/lib/racecar/partition_processor.rb

@@ -48,9 +48,8 @@ module Racecar
 
       with_error_handling(message, payload) do |pause|
         @instrumenter.instrument("process_message", payload) do
-          if @config.multithreaded_processing_enabled && consumer_class_instance.instance_variable_get(:@producer)&.closed?
-            reconfigure_consumer_class_instance!
-          end
+          reconfigure_if_producer_closed!
+          exit_if_rebalancing!
           consumer_class_instance.process(Racecar::Message.new(message, retries_count: pause.pauses_count))
           consumer_class_instance.deliver!
           consumer.store_offset(message, @rdkafka_consumer) unless rebalancing
@@ -76,9 +75,8 @@ module Racecar
           racecar_messages = messages.map do |message|
             Racecar::Message.new(message, retries_count: pause.pauses_count)
           end
-          if @config.multithreaded_processing_enabled && consumer_class_instance.instance_variable_get(:@producer)&.closed?
-            reconfigure_consumer_class_instance!
-          end
+          reconfigure_if_producer_closed!
+          exit_if_rebalancing!
           consumer_class_instance.process_batch(racecar_messages)
           consumer_class_instance.deliver!
           consumer.store_offset(messages.last, @rdkafka_consumer) unless rebalancing
@@ -146,14 +144,20 @@ module Racecar
           elsif !shutting_down
             handle_processing_error(e, payload, pause: pause)
             pause.pause!
-            unless config.pause_timeout <= 0
-              @sleep_mutex.synchronize do
-                next if rebalancing || shutting_down
+
+            break if config.pause_timeout == 0
+
+            @sleep_mutex.synchronize do
+              next if rebalancing || shutting_down
+              if config.pause_timeout == -1
+                # Pause indefinitely. backoff_interval is Float::INFINITY here,
+                @sleep_cv.wait(@sleep_mutex)
+              else
                 @sleep_cv.wait(@sleep_mutex, pause.backoff_interval)
               end
             end
             Thread.exit if rebalancing
-            break if shutting_down || config.pause_timeout <= 0
+            break if shutting_down
           else
             handle_processing_error(e, payload, pause: pause)
             break
@@ -205,6 +209,21 @@ module Racecar
       reconfigure_consumer_class_instance!
     end
 
+    def reconfigure_if_producer_closed!
+      return unless @config.multithreaded_processing_enabled
+      return unless consumer_class_instance.instance_variable_get(:@producer)&.closed?
+
+      reconfigure_consumer_class_instance!
+    end
+
+    def exit_if_rebalancing!
+      return unless @config.multithreaded_processing_enabled
+      if rebalancing
+        logger.info "Exiting processing thread for #{topic}/#{partition} due to rebalancing"
+        Thread.exit
+      end
+    end
+
     def reconfigure_consumer_class_instance!
       consumer_class_instance.configure(
         producer:     consumer.producer,

data/lib/racecar/rebalance_listener.rb

@@ -2,7 +2,6 @@ module Racecar
   class RebalanceListener
     def initialize(config, instrumenter, partition_processors)
       @consumer_class = config.consumer_class
-      @config = config
       @instrumenter = instrumenter
       @partition_processors = partition_processors
       @rdkafka_consumer = nil

data/lib/racecar/runner.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "rdkafka"
+require "timeout"
 require "racecar/pause"
 require "racecar/message"
 require "racecar/message_delivery_error"
@@ -168,7 +169,9 @@ module Racecar
         processors_snapshot.each do |processor|
           if processor.respond_to?(:thread)
             begin
-              processor.thread.join(config.multithreaded_processing_shutdown_timeout)
+              raise Timeout::Error unless processor.thread.join(config.multithreaded_processing_shutdown_timeout)
+            rescue Timeout::Error
+              logger.error "Processor thread for #{processor.thread_key} did not finish within #{config.multithreaded_processing_shutdown_timeout} seconds. It may be stuck in a long-running process or blocked on I/O."
             rescue => e
               logger.error "Error while waiting for processor thread to finish: #{e}"
             end

data/lib/racecar/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Racecar
-  VERSION = "3.0.0.alpha.3"
+  VERSION = "3.0.0.beta.2"
 end

data/racecar.gemspec

@@ -24,6 +24,7 @@ Gem::Specification.new do |spec|
 
   spec.add_runtime_dependency "king_konf", "~> 1.0.0"
   spec.add_runtime_dependency "rdkafka",   ">= 0.15.0"
+  spec.add_runtime_dependency "concurrent-ruby", "~> 1.3"
 
   spec.add_development_dependency "bundler", [">= 1.13", "< 3"]
   spec.add_development_dependency "pry-byebug"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 3.0.0.alpha.3
+  version: 3.0.0.beta.2
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -38,6 +38,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.15.0
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement