racecar 2.0.0.beta2 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 40511d6f40dafe30b33700935c54aea628eecdeae41fa4c1f25c1af26a9f398c
-  data.tar.gz: 41d62e586808c359db8761b872fe8c2ddbfd116b10e703c347b4b4452bdaea57
+  metadata.gz: dbf41d42dfa928893b72e569837c86cdab99e180a4c182c6b7ef7194553f57f5
+  data.tar.gz: 4e67b732e59c59431c34a19d32b0a5d0ffae64a31fd858e35d2673b26c3ac192
 SHA512:
-  metadata.gz: 92479e9d06a5c724041fc1b6556a329eadca406f909c767e9aa1f3f09f42470b9e67d2444ec9aa019459ed325d516d59f84772b49cf26fceded1c157e5937edf
-  data.tar.gz: e6f60c739ad0cd1db7d833fd32dd7df2537a26830b8d87ac19382aa2bc2d98f1289fe07544159d35d6cdd3dd586a80397068256a36ad5228a3bdd4b76ca143c8
+  metadata.gz: a501912a2c4f9f8d32c943b1c9516042998b241ce743449c3db48623507e360bc14af96c70a2f3b1448550de9e5e6a45d917bf31b3a2f6270f4401eed9c1d70a
+  data.tar.gz: 8e13043d7f41ff8fcaa47d7e5b1cdd4f5a2fd75eba04579172e31dbe76c7831a0d8fc9ed1dd976baec2a8d782c6ad4656f96709d7b3161f7186004aa633a5324

data/CHANGELOG.md

@@ -2,29 +2,30 @@
 
 ## racecar v2.0.0
 
-* Replace `ruby-kafka` with `rdkafka-ruby`
-* Removed config option `sasl_over_ssl`
-* [Racecar::Consumer] Do not pause consuming partitions on exception
-* [Racecar::Consumer] `topic`, `payload` and `key` are mandadory to method `produce`
-* [Racecar::Consumer] `process_batch` retrieves an array of messages instead of batch object
-* [Racecar::Consumer] Remove `offset_retention_time`
-* [Racecar::Consumer] Allow providing `additional_config` for subscriptions
-* [Racecar::Consumer] Provide access to `producer` and `consumer`
-* [Racecar::Consumer] Enforce delivering messages with method `deliver!`
-* [Racecar::Consumer] instead of raising when a partition EOF is reached, the result can be queried through `consumer.last_poll_read_partition_eof?`
-* [Racecar::Config] Remove `offset_retention_time`, `connect_timeout` and `offset_commit_threshold`
-* [Racecar::Config] Pass config to `rdkafka-ruby` via `producer` and `consumer`
-* [Racecar::Config] Replace `max_fetch_queue_size` with `min_message_queue_size`
-* [Racecar::Config] Add `synchronous_commits` to control blocking of `consumer.commit` (default `false`)
-* [Racecar::Config] Add `security_protocol` to control protocol between client and broker
-* [Racecar::Config] SSL configuration via `ssl_ca_location`, `ssl_crl_location`, `ssl_keystore_location` and `ssl_keystore_password`
-* [Racecar::Config] SASL configuration via `sasl_mechanism`, `sasl_kerberos_service_name`, `sasl_kerberos_principal`, `sasl_kerberos_kinit_cmd`, `sasl_kerberos_keytab`, `sasl_kerberos_min_time_before_relogin`, `sasl_username` and `sasl_password`
+* Replace `ruby-kafka` with `rdkafka-ruby` as the low-level library underneath Racecar.
+* Fix `max_wait_time` usage (#179).
+* Removed config option `sasl_over_ssl`.
+* [Racecar::Consumer] Do not pause consuming partitions on exception.
+* [Racecar::Consumer] `topic`, `payload` and `key` are mandadory to method `produce`.
+* [Racecar::Consumer] `process_batch` retrieves an array of messages instead of batch object.
+* [Racecar::Consumer] Remove `offset_retention_time`.
+* [Racecar::Consumer] Allow providing `additional_config` for subscriptions.
+* [Racecar::Consumer] Provide access to `producer` and `consumer`.
+* [Racecar::Consumer] Enforce delivering messages with method `deliver!`.
+* [Racecar::Consumer] instead of raising when a partition EOF is reached, the result can be queried through `consumer.last_poll_read_partition_eof?`.
+* [Racecar::Config] Remove `offset_retention_time`, `connect_timeout` and `offset_commit_threshold`.
+* [Racecar::Config] Pass config to `rdkafka-ruby` via `producer` and `consumer`.
+* [Racecar::Config] Replace `max_fetch_queue_size` with `min_message_queue_size`.
+* [Racecar::Config] Add `synchronous_commits` to control blocking of `consumer.commit` (default `false`).
+* [Racecar::Config] Add `security_protocol` to control protocol between client and broker.
+* [Racecar::Config] SSL configuration via `ssl_ca_location`, `ssl_crl_location`, `ssl_keystore_location` and `ssl_keystore_password`.
+* [Racecar::Config] SASL configuration via `sasl_mechanism`, `sasl_kerberos_service_name`, `sasl_kerberos_principal`, `sasl_kerberos_kinit_cmd`, `sasl_kerberos_keytab`, `sasl_kerberos_min_time_before_relogin`, `sasl_username` and `sasl_password`.
 * [Instrumentation] `produce_message.racecar` sent whenever a produced message is queued. Payload includes `topic`, `key`, `value` and `create_time`.
 * [Instrumentation] `acknowledged_message.racecar` send whenever a produced message was successfully received by Kafka. Payload includes `offset` and `partition`, but no message details.
-* [Instrumentation] `rdkafka-ruby` does not yet provide instrumentation [rdkafka-ruby#54](https://github.com/appsignal/rdkafka-ruby/issues/54)
-* [Instrumentation] if processors define a `statistics_callback`, it will be called once every second for every subscription or producer connection. The first argument will be a Hash, for contents see [librdkafka STATISTICS.md](https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md)
+* [Instrumentation] `rdkafka-ruby` does not yet provide instrumentation [rdkafka-ruby#54](https://github.com/appsignal/rdkafka-ruby/issues/54).
+* [Instrumentation] if processors define a `statistics_callback`, it will be called once every second for every subscription or producer connection. The first argument will be a Hash, for contents see [librdkafka STATISTICS.md](https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md).
 * Add current directory to `$LOAD_PATH` only when `--require` option is used (#117).
-* Remove manual heartbeat support, see [Long-running message processing section in README](README.md#long-running-message-processing)
+* Remove manual heartbeat support, see [Long-running message processing section in README](README.md#long-running-message-processing).
 
 ## racecar v1.0.0
 

data/README.md

@@ -1,5 +1,3 @@
-**IMPORTANT:** The `master` branch is unstable, working towards a v2 release that breaks a lot of stuff. Use the `v1-stable` branch if you want to suggest changes.
-
 # Racecar
 
 Racecar is a friendly and easy-to-approach Kafka consumer framework. It allows you to write small applications that process messages stored in Kafka topics while optionally integrating with your Rails models.
@@ -21,6 +19,7 @@ The framework is based on [rdkafka-ruby](https://github.com/appsignal/rdkafka-ru
     7. [Handling errors](#handling-errors)
     8. [Logging](#logging)
     9. [Operations](#operations)
+    10. [Upgrading from v1 to v2](#upgrading-from-v1-to-v2)
 3. [Development](#development)
 4. [Contributing](#contributing)
 5. [Support and Discussion](#support-and-discussion)
@@ -288,6 +287,9 @@ The memory usage limit is roughly estimated as `max_bytes * min_message_queue_si
 * `ssl_crl_location` – Path to CRL for verifying broker's certificate validity
 * `ssl_keystore_location` – Path to client's keystore (PKCS#12) used for authentication
 * `ssl_keystore_password` – Client's keystore (PKCS#12) password
+* `ssl_certificate_location` – Path to the certificate
+* `ssl_key_location` – Path to client's certificate used for authentication
+* `ssl_key_password` – Client's certificate password
 
 #### SASL encryption, authentication & authorization
 
@@ -464,7 +466,7 @@ Racecar.config.on_error do |exception, info|
 end
 ```
 
-It is highly recommended that you set up an error handler.
+It is highly recommended that you set up an error handler. Please note that the `info` object contains different keys and values depending on whether you are using `process` or `process_batch`. See the `instrumentation_payload` object in the `process` and `process_batch` methods in the `Runner` class for the complete list.
 
 
 ### Logging
@@ -481,6 +483,11 @@ In order to gracefully shut down a Racecar consumer process, send it the `SIGTER
 In order to introspect the configuration of a consumer process, send it the `SIGUSR1` signal. This will make Racecar print its configuration to the standard error file descriptor associated with the consumer process, so you'll need to know where that is written to.
 
 
+### Upgrading from v1 to v2
+
+In order to safely upgrade from Racecar v1 to v2, you need to completely shut down your consumer group before starting it up again with the v2 Racecar dependency. In general, you should avoid rolling deploys for consumers groups, so it is likely the case that this will just work for you, but it's a good idea to check first.
+
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rspec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/exe/racecar

@@ -13,6 +13,8 @@ rescue SignalException => e
     raise
   end
 rescue
+  # Exceptions are printed to STDERR and sent to the error handler
+  # in `Racecar::Cli#run`, so we don't need to do anything here.
   exit(1)
 else
   exit(0)

data/lib/racecar/cli.rb

@@ -102,7 +102,12 @@ module Racecar
         opts.on("-r", "--require STRING", "Require a library before starting the consumer") do |lib|
           $LOAD_PATH.unshift(Dir.pwd) unless load_path_modified
           load_path_modified = true
-          require lib
+          begin
+            require lib
+          rescue => e
+            $stderr.puts "=> #{lib} failed to load: #{e.message}"
+            exit
+          end
         end
 
         opts.on("-l", "--log STRING", "Log to the specified file") do |logfile|

data/lib/racecar/config.rb

@@ -28,6 +28,9 @@ module Racecar
     desc "The maxium number of messages that get consumed within one batch"
     integer :fetch_messages, default: 1000
 
+    desc "Minimum number of bytes the broker responds with"
+    integer :fetch_min_bytes, default: 1
+
     desc "Automatically store offset of last message provided to application"
     boolean :synchronous_commits, default: false
 
@@ -82,6 +85,15 @@ module Racecar
     desc "Client's keystore (PKCS#12) password"
     string :ssl_keystore_password
 
+    desc "Path to the certificate used for authentication"
+    string :ssl_certificate_location
+
+    desc "Path to client's certificate used for authentication"
+    string :ssl_key_location
+
+    desc "Client's certificate password"
+    string :ssl_key_password
+
     desc "SASL mechanism to use for authentication"
     string :sasl_mechanism, allowed_values: %w{GSSAPI PLAIN SCRAM-SHA-256 SCRAM-SHA-512}
 
@@ -144,6 +156,10 @@ module Racecar
 
     attr_accessor :subscriptions, :logger
 
+    def max_wait_time_ms
+      max_wait_time * 1000
+    end
+
     def initialize(env: ENV)
       super(env: env)
       @error_handler = proc {}
@@ -217,6 +233,9 @@ module Racecar
         "ssl.crl.location" => ssl_crl_location,
         "ssl.keystore.location" => ssl_keystore_location,
         "ssl.keystore.password" => ssl_keystore_password,
+        "ssl.certificate.location" => ssl_certificate_location,
+        "ssl.key.location" => ssl_key_location,
+        "ssl.key.password" => ssl_key_password,
         "sasl.mechanism" => sasl_mechanism,
         "sasl.kerberos.service.name" => sasl_kerberos_service_name,
         "sasl.kerberos.principal" => sasl_kerberos_principal,

data/lib/racecar/consumer.rb

@@ -54,21 +54,29 @@ module Racecar
     protected
 
     # https://github.com/appsignal/rdkafka-ruby#producing-messages
-    def produce(payload, topic:, key:, headers: nil, create_time: nil)
+    def produce(payload, topic:, key:, partition_key: nil, headers: nil, create_time: nil)
       @delivery_handles ||= []
       message_size = payload.respond_to?(:bytesize) ? payload.bytesize : 0
       instrumentation_payload = {
-        value:        payload,
-        headers:      headers,
-        key:          key,
-        topic:        topic,
+        value: payload,
+        headers: headers,
+        key: key,
+        partition_key: partition_key,
+        topic: topic,
         message_size: message_size,
-        create_time:  Time.now,
-        buffer_size:  @delivery_handles.size
+        create_time: Time.now,
+        buffer_size: @delivery_handles.size,
       }
 
       @instrumenter.instrument("produce_message", instrumentation_payload) do
-        @delivery_handles << @producer.produce(topic: topic, payload: payload, key: key, timestamp: create_time, headers: headers)
+        @delivery_handles << @producer.produce(
+          topic: topic,
+          payload: payload,
+          key: key,
+          partition_key: partition_key,
+          timestamp: create_time,
+          headers: headers,
+        )
       end
     end
 

data/lib/racecar/consumer_set.rb

@@ -174,8 +174,9 @@ module Racecar
         "client.id"               => @config.client_id,
         "enable.partition.eof"    => false,
         "fetch.max.bytes"         => @config.max_bytes,
-        "fetch.message.max.bytes" => subscription.max_bytes_per_partition,
-        "fetch.wait.max.ms"       => @config.max_wait_time * 1000,
+        "message.max.bytes"       => subscription.max_bytes_per_partition,
+        "fetch.min.bytes"         => @config.fetch_min_bytes,
+        "fetch.wait.max.ms"       => @config.max_wait_time_ms,
         "group.id"                => @config.group_id,
         "heartbeat.interval.ms"   => @config.heartbeat_interval * 1000,
         "max.poll.interval.ms"    => @config.max_poll_interval * 1000,

data/lib/racecar/datadog.rb

@@ -26,7 +26,7 @@ module Racecar
       end
 
       def host
-        @host ||= default_host
+        @host
       end
 
       def host=(host)
@@ -35,7 +35,7 @@ module Racecar
       end
 
       def port
-        @port ||= default_port
+        @port
       end
 
       def port=(port)
@@ -63,22 +63,6 @@ module Racecar
 
       private
 
-      def default_host
-        if ::Datadog::Statsd.const_defined?(:Connection)
-          ::Datadog::Statsd::Connection::DEFAULT_HOST
-        else
-          ::Datadog::Statsd::DEFAULT_HOST
-        end
-      end
-
-      def default_port
-        if ::Datadog::Statsd.const_defined?(:Connection)
-          ::Datadog::Statsd::Connection::DEFAULT_PORT
-        else
-          ::Datadog::Statsd::DEFAULT_PORT
-        end
-      end
-
       def clear
         @statsd && @statsd.close
         @statsd = nil
@@ -126,6 +110,8 @@ module Racecar
       def process_batch(event)
         offset = event.payload.fetch(:last_offset)
         messages = event.payload.fetch(:message_count)
+        last_create_time = event.payload.fetch(:last_create_time)
+        time_lag = last_create_time && ((Time.now - last_create_time) * 1000).to_i
         tags = default_tags(event)
 
         if event.payload.key?(:exception)
@@ -135,7 +121,12 @@ module Racecar
           count("consumer.messages", messages, tags: tags)
         end
 
+        histogram("consumer.batch_size", messages, tags: tags)
         gauge("consumer.offset", offset, tags: tags)
+
+        if time_lag
+          gauge("consumer.time_lag", time_lag, tags: tags)
+        end
       end
 
       def join_group(event)
@@ -239,7 +230,7 @@ module Racecar
         increment("producer.ack.messages", tags: tags)
       end
 
-      attach_to "producer.racecar"
+      attach_to "racecar"
     end
   end
 end

data/lib/racecar/runner.rb

@@ -65,12 +65,12 @@ module Racecar
         @instrumenter.instrument("main_loop", instrumentation_payload) do
           case process_method
           when :batch then
-            msg_per_part = consumer.batch_poll(config.max_wait_time).group_by(&:partition)
+            msg_per_part = consumer.batch_poll(config.max_wait_time_ms).group_by(&:partition)
             msg_per_part.each_value do |messages|
               process_batch(messages)
             end
           when :single then
-            message = consumer.poll(config.max_wait_time)
+            message = consumer.poll(config.max_wait_time_ms)
             process(message) if message
           end
         end
@@ -166,11 +166,16 @@ module Racecar
       }
 
       @instrumenter.instrument("start_process_message", instrumentation_payload)
-      @instrumenter.instrument("process_message", instrumentation_payload) do
-        with_pause(message.topic, message.partition, message.offset..message.offset) do
-          processor.process(Racecar::Message.new(message))
-          processor.deliver!
-          consumer.store_offset(message)
+      with_pause(message.topic, message.partition, message.offset..message.offset) do
+        begin
+          @instrumenter.instrument("process_message", instrumentation_payload) do
+            processor.process(Racecar::Message.new(message))
+            processor.deliver!
+            consumer.store_offset(message)
+          end
+        rescue => e
+          config.error_handler.call(e, instrumentation_payload)
+          raise e
         end
       end
     end
@@ -183,15 +188,21 @@ module Racecar
         partition:      first.partition,
         first_offset:   first.offset,
         last_offset:    last.offset,
+        last_create_time: last.timestamp,
         message_count:  messages.size
       }
 
       @instrumenter.instrument("start_process_batch", instrumentation_payload)
       @instrumenter.instrument("process_batch", instrumentation_payload) do
         with_pause(first.topic, first.partition, first.offset..last.offset) do
-          processor.process_batch(messages.map {|message| Racecar::Message.new(message) })
-          processor.deliver!
-          consumer.store_offset(messages.last)
+          begin
+            processor.process_batch(messages.map {|message| Racecar::Message.new(message) })
+            processor.deliver!
+            consumer.store_offset(messages.last)
+          rescue => e
+            config.error_handler.call(e, instrumentation_payload)
+            raise e
+          end
         end
       end
     end

data/lib/racecar/version.rb

@@ -1,3 +1,3 @@
 module Racecar
-  VERSION = "2.0.0.beta2"
+  VERSION = "2.0.0"
 end

data/racecar.gemspec

@@ -21,12 +21,12 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_runtime_dependency "king_konf", "~> 0.3.7"
-  spec.add_runtime_dependency "rdkafka",   "~> 0.6.0"
+  spec.add_runtime_dependency "rdkafka",   "~> 0.8.0.beta.1"
 
   spec.add_development_dependency "bundler", [">= 1.13", "< 3"]
   spec.add_development_dependency "rake", "> 10.0"
   spec.add_development_dependency "rspec", "~> 3.0"
   spec.add_development_dependency "timecop"
-  spec.add_development_dependency "dogstatsd-ruby", ">= 3.0.0", "< 5.0.0"
+  spec.add_development_dependency "dogstatsd-ruby", ">= 4.0.0", "< 5.0.0"
   spec.add_development_dependency "activesupport", ">= 4.0", "< 6.1"
 end

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 2.0.0.beta2
+  version: 2.0.0
 platform: ruby
 authors:
 - Daniel Schierbeck
 - Benjamin Quorning
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-02-05 00:00:00.000000000 Z
+date: 2020-08-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: king_konf
@@ -31,14 +31,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.6.0
+        version: 0.8.0.beta.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.6.0
+        version: 0.8.0.beta.1
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -107,7 +107,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 3.0.0
+        version: 4.0.0
     - - "<"
       - !ruby/object:Gem::Version
         version: 5.0.0
@@ -117,7 +117,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 3.0.0
+        version: 4.0.0
     - - "<"
       - !ruby/object:Gem::Version
         version: 5.0.0
@@ -141,7 +141,7 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: '6.1'
-description: 
+description:
 email:
 - dschierbeck@zendesk.com
 - bquorning@zendesk.com
@@ -192,7 +192,7 @@ homepage: https://github.com/zendesk/racecar
 licenses:
 - Apache License Version 2.0
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -203,13 +203,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6
-signing_key: 
+rubygems_version: 3.1.2
+signing_key:
 specification_version: 4
 summary: A framework for running Kafka consumers
 test_files: []