racecar 3.0.0.alpha.3 → 3.0.0.beta.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 043070ffc30189dd4f500a691316db7a2a55422078b7bf5f31b81ff45aa69cab
-  data.tar.gz: 73a2d774f56f7f14d18c5c43691f185ab64fd7e2f3cd6a29dfcad8b9111f00b5
+  metadata.gz: da583e5664d52a322de9d5d1e94c21b1b95a5ddeecfa31c5c52878867c4568e5
+  data.tar.gz: f35e00ca35831da9cfc0c5dbb6eebbe2404d8963ad62f2fb785a20e14aa5df08
 SHA512:
-  metadata.gz: d0ea838d6e6381660a88b6e1d71c95d0fbe741609dc216b18830d8a74eee78b0d56a6e05713b4f12ac28f60146b566f4f125d53b4c4bd4c52c3c13459941fac4
-  data.tar.gz: 213281a3b10ad91e3d7bcbf3a8974fbf15d8e1b5905518d3a4f13505ede101d254a433f86b1e7f875ccdad3c7d4d59fbc570b0d9db63f146615f4f454a65bd85
+  metadata.gz: 0cbdae52223f76c0d7dbe6fb000522071789ee22deee87603cdab5947eca5842eafc8cc4053662d2152c8f18c841fb53a5872bc7e48c5b0596b82cf2ead0af0b
+  data.tar.gz: 2ee11ff529c0043ec2b9bbc680f2da257775e9e6273236d9066df400d17c97ab02a7347f5a2c4687d56b00dc928178a9f1ec5c5c179c639fe6a5ae2ff2b505f5

data/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## Unreleased
 
+## 3.0.0
+
+* Introduce multithreaded processing: when enabled, Racecar processes each assigned partition on its own dedicated thread. Disabled by default and gated behind the `multithreaded_processing_enabled` config.
+* Refactor of the Racecar architecture: introduction of `PartitionProcessor` and `AsyncPartitionProcessor` handling the processing of messages.
+* [Racecar::Config] Add `multithreaded_processing_enabled` (default `false`) to enable multithreaded processing. Can be set via `RACECAR_MULTITHREADED_PROCESSING_ENABLED=1`.
+* [Racecar::Config] Add `multithreaded_processing_max_queue_size` (default `1000`) to cap the number of messages queued per partition before backpressure is applied.
+* [Racecar::Config] Add `multithreaded_processing_resume_threshold` (default `0.5`) controlling the queue fill ratio at which a paused partition is resumed.
+* [Racecar::Config] Add `multithreaded_processing_shutdown_timeout` (default `300`) for how long the main thread waits on each processing thread during graceful shutdown.
+* Apply backpressure when multithreaded processing is enabled: a partition is paused once its queue reaches `multithreaded_processing_max_queue_size` and resumed once it drains below `multithreaded_processing_resume_threshold` of that size.
+* Gracefully drain queued messages and exit per-partition threads on rebalance and shutdown.
+
 ## 2.12.0
 
 * Add tests against Ruby 3.4

data/Gemfile.lock

@@ -1,7 +1,8 @@
 PATH
   remote: .
   specs:
-    racecar (3.0.0.alpha.3)
+    racecar (3.0.0.beta.1)
+      concurrent-ruby (~> 1.3)
       king_konf (~> 1.0.0)
       rdkafka (>= 0.15.0)
 

data/README.md

@@ -134,6 +134,42 @@ end
 
 This is useful to do any one-off work that you wouldn't want to do for each and every message.
 
+#### Multithreaded message processing (experimental)
+
+Warning - limited battle testing in production environments; use at your own risk!
+
+By default a Racecar consumer processes all of its assigned partitions on a single thread. When `multithreaded_processing_enabled` is set, the consumer instead spins up one dedicated thread per assigned partition, so partitions are processed concurrently within a single process. This is an alternative to [parallel workers](#running-consumers-in-parallel-experimental) that avoids forking extra processes (and the associated memory overhead), at the cost of running your consumer code on multiple threads.
+
+Each partition thread gets its own instance of your consumer class, so **your consumer code does not need to be thread-safe** - a thread never shares its instance (or its instance state) with another partition. The main thread keeps polling Kafka and hands each partition's messages off to the relevant thread via a bounded queue; if a thread falls behind and its queue fills up, the partition is paused until the queue drains, applying backpressure rather than growing memory without bound.
+
+**Warning:** the number of threads scales with the number of assigned partitions, which can be large. Since each thread runs its own consumer instance, every resource that instance acquires (database connections, file handles, network sockets, HTTP clients, etc.) is multiplied by the number of partition threads. Make sure your consumer releases any resource it grabs and that any connection pools or other shared limits are sized to accommodate the resulting concurrency.
+
+Enable it via config (or the `RACECAR_MULTITHREADED_PROCESSING_ENABLED=1` environment variable):
+
+```ruby
+Racecar.configure do |config|
+  config.multithreaded_processing_enabled = true
+end
+
+class ResizeImagesConsumer < Racecar::Consumer
+  subscribes_to "images"
+
+  def process(message)
+    # This runs on a thread dedicated to message.partition.
+    # @state below is private to this partition's thread.
+    @state ||= {}
+    Image.resize(message.value)
+  end
+end
+```
+
+The behaviour can be tuned with the following options:
+
+- `multithreaded_processing_enabled` – Enable per-partition threads. Default is `false`.
+- `multithreaded_processing_max_queue_size` – Maximum number of queued message batches per partition before the partition is paused to apply backpressure. Default is `1000`.
+- `multithreaded_processing_resume_threshold` – A paused partition is resumed once its queue drains below this fraction of `multithreaded_processing_max_queue_size`. Default is `0.5` (50%).
+- `multithreaded_processing_shutdown_timeout` – How many seconds to wait for each partition thread to finish during graceful shutdown. Default is `300`.
+
 #### Setting the starting position
 
 When a consumer is started for the first time, it needs to decide where in each partition to start. By default, it will start at the _beginning_, meaning that all past messages will be processed. If you want to instead start at the _end_ of each partition, change your `subscribes_to` like this:

data/lib/racecar/async_partition_processor.rb

@@ -7,8 +7,6 @@ module Racecar
   class AsyncPartitionProcessor
     attr_reader :thread
 
-    THREAD_KEY_IDENTIFIER = 'racecar_topic_partition_identifier'.freeze
-
     def self.thread_key(topic, partition)
       "#{topic}/#{partition}"
     end
@@ -70,7 +68,6 @@ module Racecar
         rdkafka_consumer: @rdkafka_consumer,
       )
       @queue  = Queue.new
-      @thread = nil
 
       use_process_batch = consumer_class.method_defined?(:process_batch)
 
@@ -90,7 +87,6 @@ module Racecar
     def spawn_thread(&block)
       @thread = Thread.new do
         Thread.current.name = "Racecar thread for #{thread_key}"
-        Thread.current[AsyncPartitionProcessor::THREAD_KEY_IDENTIFIER] = thread_key
         main_processing_loop(block)
       end
     end

data/lib/racecar/partition_processor.rb

@@ -48,9 +48,7 @@ module Racecar
 
       with_error_handling(message, payload) do |pause|
         @instrumenter.instrument("process_message", payload) do
-          if @config.multithreaded_processing_enabled && consumer_class_instance.instance_variable_get(:@producer)&.closed?
-            reconfigure_consumer_class_instance!
-          end
+          reconfigure_if_producer_closed!
           consumer_class_instance.process(Racecar::Message.new(message, retries_count: pause.pauses_count))
           consumer_class_instance.deliver!
           consumer.store_offset(message, @rdkafka_consumer) unless rebalancing
@@ -76,9 +74,7 @@ module Racecar
           racecar_messages = messages.map do |message|
             Racecar::Message.new(message, retries_count: pause.pauses_count)
           end
-          if @config.multithreaded_processing_enabled && consumer_class_instance.instance_variable_get(:@producer)&.closed?
-            reconfigure_consumer_class_instance!
-          end
+          reconfigure_if_producer_closed!
           consumer_class_instance.process_batch(racecar_messages)
           consumer_class_instance.deliver!
           consumer.store_offset(messages.last, @rdkafka_consumer) unless rebalancing
@@ -205,6 +201,13 @@ module Racecar
       reconfigure_consumer_class_instance!
     end
 
+    def reconfigure_if_producer_closed!
+      return unless @config.multithreaded_processing_enabled
+      return unless consumer_class_instance.instance_variable_get(:@producer)&.closed?
+
+      reconfigure_consumer_class_instance!
+    end
+
     def reconfigure_consumer_class_instance!
       consumer_class_instance.configure(
         producer:     consumer.producer,

data/lib/racecar/rebalance_listener.rb

@@ -2,7 +2,6 @@ module Racecar
   class RebalanceListener
     def initialize(config, instrumenter, partition_processors)
       @consumer_class = config.consumer_class
-      @config = config
       @instrumenter = instrumenter
       @partition_processors = partition_processors
       @rdkafka_consumer = nil

data/lib/racecar/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Racecar
-  VERSION = "3.0.0.alpha.3"
+  VERSION = "3.0.0.beta.1"
 end

data/racecar.gemspec

@@ -24,6 +24,7 @@ Gem::Specification.new do |spec|
 
   spec.add_runtime_dependency "king_konf", "~> 1.0.0"
   spec.add_runtime_dependency "rdkafka",   ">= 0.15.0"
+  spec.add_runtime_dependency "concurrent-ruby", "~> 1.3"
 
   spec.add_development_dependency "bundler", [">= 1.13", "< 3"]
   spec.add_development_dependency "pry-byebug"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 3.0.0.alpha.3
+  version: 3.0.0.beta.1
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -38,6 +38,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.15.0
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement