racecar 0.4.2 → 0.5.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c45d8677d5b6355960cdef16cddc745909b28145bb3243cfa64170f6f284609a
-  data.tar.gz: dfed28e1acb40d781aedc6c0ae1d6d916b04efa48919cad6a05c4310720fee0b
+  metadata.gz: 5806641d4fcbd87d7625458de4ed60b19db3b4a8695d52dc062c72e1db281136
+  data.tar.gz: '08f9d6c439bcbc97bc05bcffb37d97b6eaf3b0a375deb85d1ad33dd2c004c9f7'
 SHA512:
-  metadata.gz: 5a8f1c5077c03fb3dcca7124d6465ba6386f5a9a0dbe7ef5bf752073c58f04cc36c2665c1b5595b692289dfacddbb83e5cc022f07fbf05fc83dfcca266e743fb
-  data.tar.gz: 75dc3b0d158f0a4b8fb8a8242dc7f4b2c043217792f6351a27d2a8698361e7e86fe04315a4b167202eb1703045b58183533019ef9dd48563db954c076ddb4b01
+  metadata.gz: 18b8f848e4830f538b679cb3301f9bdcff36f0ccf4a4dfd0b7edc125aaa4a738164dd879d10366878ca3b74876506471809880d5ce23a607e22c8fcf0de1768d
+  data.tar.gz: 32af67f7bafd1320423e60f18895e2264c8fe623a1ff3ef96d457255c286a795dcee35ed83e1dd6b581baa14f9defc889757daedaf516623f46e7a7273d98a10

data/CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## Unreleased
 
+## racecar v0.5.0
+
+* Add support for manually sending heartbeats with `heartbeat` (#105).
+
 ## racecar v0.4.2
 
 * Allow configuring `max_bytes` and `max_fetch_queue_size`.

data/README.md

@@ -147,6 +147,32 @@ end
 
 An important detail is that, if an exception is raised while processing a batch, the _whole batch_ is re-processed.
 
+#### Message headers
+
+Any headers set on the message will be available when consuming the message:
+
+```ruby
+message.headers #=> { "Header-A" => 42, ... }
+```
+
+#### Heartbeats
+
+In order to avoid your consumer being kicked out of its group during long-running message processing operations, it may be a good idea to periodically send so-called _heartbeats_ back to Kafka. This is done automatically for you after each message has been processed, but if the processing of a _single_ message takes a long time you may run into group stability issues.
+
+If possible, intersperse `heartbeat` calls in between long-running operations in your consumer, e.g.
+
+```ruby
+def process(message)
+  long_running_op_one(message)
+
+  # Signals back to Kafka that we're still alive!
+  heartbeat
+
+  long_running_op_two(message)
+end
+```
+
+
 #### Tearing down resources when stopping
 
 When a Racecar consumer shuts down, it gets the opportunity to tear down any resources held by the consumer instance. For example, it may make sense to close any open files or network connections. Doing so is simple: just implement a `#teardown` method in your consumer class and it will be called during the shutdown procedure.
@@ -203,6 +229,8 @@ class GeoCodingConsumer < Racecar::Consumer
 end
 ```
 
+You can set message headers by passing a `headers:` option with a Hash of headers.
+
 ### Configuration
 
 Racecar provides a flexible way to configure your consumer in a way that feels at home in a Rails application. If you haven't already, run `bundle exec rails generate racecar:install` in order to generate a config file. You'll get a separate section for each Rails environment, with the common configuration values in a shared `common` section.
@@ -227,6 +255,11 @@ end
 * `group_id` – The group id to use for a given group of consumers. Note that this _must_ be different for each consumer class. If left blank a group id is generated based on the consumer class name.
 * `group_id_prefix` – A prefix used when generating consumer group names. For instance, if you set the prefix to be `kevin.` and your consumer class is named `BaconConsumer`, the resulting consumer group will be named `kevin.bacon_consumer`.
 
+#### Logging
+
+* `logfile` – A filename that log messages should be written to. Default is `nil`, which means logs will be written to standard output.
+* `log_level` – The log level for the Racecar logs, one of `debug`, `info`, `warn`, or `error`. Default is `info`.
+
 #### Consumer checkpointing
 
 The consumers will checkpoint their positions from time to time in order to be able to recover from failures. This is called _committing offsets_, since it's done by tracking the offset reached in each partition being processed, and committing those offset numbers to the Kafka offset storage API. If you can tolerate more double-processing after a failure, you can increase the interval between commits in order to better performance. You can also do the opposite if you prefer less chance of double-processing.
@@ -253,7 +286,7 @@ Kafka is _really_ good at throwing data at consumers, so you may want to tune th
 Racecar uses ruby-kafka under the hood, which fetches messages from the Kafka brokers in a background thread. This thread pushes fetch responses, possible containing messages from many partitions, into a queue that is read by the processing thread (AKA your code). The main way to control the fetcher thread is to control the size of those responses and the size of the queue.
 
 * `max_bytes` — The maximum size of message sets returned from a single fetch request.
-* `max_fetch_queue_size` — The maximum number of fetch responses to keep in the queue.
+* `max_fetch_queue_size` — The maximum number of fetch responses to keep in the queue. Once reached, the fetcher will back off until the queue gets back down under to limit.
 
 The memory usage limit is roughly estimated as `max_bytes * max_fetch_queue_size`, plus whatever your application uses.
 
@@ -372,7 +405,7 @@ Again, the recommended approach is to manage the processes using process manager
 
 ### Handling errors
 
-When processing messages from a Kafka topic, your code may encounter an error and raise an exception. The cause is typically on of two things:
+When processing messages from a Kafka topic, your code may encounter an error and raise an exception. The cause is typically one of two things:
 
 1. The message being processed is somehow malformed or doesn't conform with the assumptions made by the processing code.
 2. You're using some external resource such as a database or a network API that is temporarily unavailable.

data/lib/racecar/config.rb

@@ -59,7 +59,7 @@ module Racecar
     string :logfile
 
     desc "The log level for the Racecar logs"
-    string :log_level
+    string :log_level, default: "info"
 
     desc "A valid SSL certificate authority"
     string :ssl_ca_cert

data/lib/racecar/consumer.rb

@@ -28,7 +28,8 @@ module Racecar
       end
     end
 
-    def configure(producer:)
+    def configure(consumer:, producer:)
+      @_consumer = consumer
       @_producer = producer
     end
 
@@ -36,6 +37,10 @@ module Racecar
 
     protected
 
+    def heartbeat
+      @_consumer.trigger_heartbeat
+    end
+
     def produce(value, **options)
       @_producer.produce(value, **options)
     end

data/lib/racecar/runner.rb

@@ -10,8 +10,10 @@ module Racecar
     end
 
     def stop
-      processor.teardown
-      consumer.stop unless consumer.nil?
+      Thread.new do
+        processor.teardown
+        consumer.stop unless consumer.nil?
+      end.join
     end
 
     def run
@@ -64,7 +66,7 @@ module Racecar
         compression_codec: config.producer_compression_codec,
       )
 
-      processor.configure(producer: producer)
+      processor.configure(consumer: consumer, producer: producer)
 
       begin
         if processor.respond_to?(:process)

data/lib/racecar/version.rb

@@ -1,3 +1,3 @@
 module Racecar
-  VERSION = "0.4.2"
+  VERSION = "0.5.0.beta1"
 end

data/racecar.gemspec

@@ -20,7 +20,7 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_runtime_dependency "king_konf", "~> 0.3.6"
+  spec.add_runtime_dependency "king_konf", "~> 0.3.7"
   spec.add_runtime_dependency "ruby-kafka", "~> 0.6"
 
   spec.add_development_dependency "bundler", "~> 1.13"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 0.4.2
+  version: 0.5.0.beta1
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-06-07 00:00:00.000000000 Z
+date: 2019-01-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: king_konf
@@ -17,14 +17,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.6
+        version: 0.3.7
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.6
+        version: 0.3.7
 - !ruby/object:Gem::Dependency
   name: ruby-kafka
   requirement: !ruby/object:Gem::Requirement
@@ -135,9 +135,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
 rubygems_version: 2.7.6