racecar 2.9.0 → 2.10.0.beta1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b394d31edcb9c83e562811ba748b6c6915388d0227a4e7425aa9aa13f64e3890
4
- data.tar.gz: 1dfcd7046b6d932f716246d0685589c1635af115a4e126067c9775b13fe05464
3
+ metadata.gz: e0057ec2861db783b83e34a63c0a09a0be476ed6cb57b760dea3e2cc5ef57d1f
4
+ data.tar.gz: 22dac9ed21af6f9abfd50f22d9dc86032a68af556e1be77bff2f59cbf505ccf2
5
5
  SHA512:
6
- metadata.gz: 59043be21e411e680c11815b583236556413a539f5aa9508a460eefe82cee6199f56a30b9fa07a3f206886d5dc811b210cea6130813a7073a68fe6612343ca0d
7
- data.tar.gz: d6692b9bb7cdc27efe5272a10fa1d9061920084a7b517699e5133f5e31e9fd50415e72ee40550e07e5a2fed028aeabb741422fef47372c7a53e0a96bdfc1b090
6
+ metadata.gz: f3baeec4de02a68b55e4a302d2b713de67d0245564aae57cbca363aec8b34633f8dd2220d44c1e212347857d7fde35217644dae8afdbde06103e11a52394997d
7
+ data.tar.gz: 66a2168ca6f7a4ec7a6d63e8cf767397502c02487d87445b59e4d7fbdd0269abee3279019e36ab1f343e246d4c58146e951ad521b1bc881a4cb60b3f35076b6e
data/CHANGELOG.md CHANGED
@@ -2,6 +2,14 @@
2
2
 
3
3
  ## Unreleased
4
4
 
5
+ ## 2.10.0.beta1
6
+
7
+ * Bump rdkafka gem version to 0.13.0
8
+ * Support cooperative-sticky
9
+ * Instrument produce delivery errors
10
+ * Fix config load for liveness probe
11
+ * Send exceptions to `process_batch` instrumenter
12
+ * Docker test fixes
5
13
  * Test with Ruby 3.2
6
14
 
7
15
  ## v2.9.0, v2.9.0.beta1
data/Dockerfile CHANGED
@@ -1,4 +1,4 @@
1
- FROM circleci/ruby:2.7.2
1
+ FROM cimg/ruby:2.7.8
2
2
 
3
3
  RUN sudo apt-get update
4
4
  RUN sudo apt-get install docker
data/Gemfile.lock CHANGED
@@ -1,9 +1,9 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- racecar (2.9.0.beta1)
4
+ racecar (2.9.0)
5
5
  king_konf (~> 1.0.0)
6
- rdkafka (~> 0.12.0)
6
+ rdkafka (~> 0.13.0)
7
7
 
8
8
  GEM
9
9
  remote: https://rubygems.org/
@@ -33,7 +33,7 @@ GEM
33
33
  byebug (~> 11.0)
34
34
  pry (>= 0.13, < 0.15)
35
35
  rake (13.0.6)
36
- rdkafka (0.12.0)
36
+ rdkafka (0.13.0)
37
37
  ffi (~> 1.15)
38
38
  mini_portile2 (~> 2.6)
39
39
  rake (> 12)
data/README.md CHANGED
@@ -476,6 +476,8 @@ With Foreman, you can easily run these processes locally by executing `foreman r
476
476
 
477
477
  If you run your applications in Kubernetes, use the following [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) spec as a starting point:
478
478
 
479
+ ##### Recreate Strategy
480
+
479
481
  ```yaml
480
482
  apiVersion: apps/v1
481
483
  kind: Deployment
@@ -483,8 +485,8 @@ metadata:
483
485
  name: my-racecar-deployment
484
486
  labels:
485
487
  app: my-racecar
486
- spec:
487
- replicas: 3 # <-- this will give us three consumers in the group.
488
+ spec
489
+ replicas: 4 # <-- this is a good value if you have a multliple of 4 partitions
488
490
  selector:
489
491
  matchLabels:
490
492
  app: my-racecar
@@ -506,9 +508,33 @@ spec:
506
508
  value: 5
507
509
  ```
508
510
 
509
- The important part is the `strategy.type` value, which tells Kubernetes how to upgrade from one version of your Deployment to another. Many services use so-called _rolling updates_, where some but not all containers are replaced with the new version. This is done so that, if the new version doesn't work, the old version is still there to serve most of the requests. For Kafka consumers, this doesn't work well. The reason is that every time a consumer joins or leaves a group, every other consumer in the group needs to stop and synchronize the list of partitions assigned to each group member. So if the group is updated in a rolling fashion, this synchronization would occur over and over again, causing undesirable double-processing of messages as consumers would start only to be synchronized shortly after.
511
+ This configuration uses the recreate strategy which completely terminates all consumers before starting new ones.
512
+ It's simple and easy to understand but can result in significant 'downtime' where no messages are processed.
513
+
514
+ ##### Rolling Updates and 'sticky-cooperative' Assignment
515
+
516
+ A newer alternative is to use the consumer's "cooperative-sticky" assignment strategy which allows healthy consumers to keep processing their partitions while others are terminated.
517
+ This can be combined with a restricted rolling update to minimize processing downtime.
518
+
519
+ Add to your Racecar config:
520
+ ```ruby
521
+ Racecar.configure do |c|
522
+ c.partition_assignment_strategy = "cooperative-sticky"
523
+ end
524
+ ```
525
+
526
+ Replace the Kubernetes deployment strategy with:
527
+ ```yaml
528
+ strategy:
529
+ type: RollingUpdate
530
+ rollingUpdate:
531
+ maxSurge: 0 # <- Never boot an excess consumer
532
+ maxUnavailable: 1 # <- The deploy 'rolls' one consumer at a time
533
+ ```
534
+
535
+ These two configurations should be deployed together.
510
536
 
511
- Instead, the `Recreate` update strategy should be used. It completely tears down the existing containers before starting all of the new containers simultaneously, allowing for a single synchronization stage and a much faster, more stable deployment update.
537
+ While `maxSurge` should always be 0, `maxUnavailable` can be increased to reduce deployment times in exchange for longer pauses in message processing.
512
538
 
513
539
  #### Liveness Probe
514
540
 
@@ -663,7 +689,7 @@ In order to introspect the configuration of a consumer process, send it the `SIG
663
689
 
664
690
  ### Upgrading from v1 to v2
665
691
 
666
- In order to safely upgrade from Racecar v1 to v2, you need to completely shut down your consumer group before starting it up again with the v2 Racecar dependency. In general, you should avoid rolling deploys for consumers groups, so it is likely the case that this will just work for you, but it's a good idea to check first.
692
+ In order to safely upgrade from Racecar v1 to v2, you need to completely shut down your consumer group before starting it up again with the v2 Racecar dependency.
667
693
 
668
694
  ### Compression
669
695
 
@@ -681,12 +707,12 @@ The integration tests run against a Kafka instance that is not automatically sta
681
707
 
682
708
  ### Running RSpec within Docker
683
709
 
684
- There can be behavioural inconsistencies between running the specs on your machine, and in the CI pipeline. Due to this, there is now a Dockerfile included in the project, which is based on the CircleCI ruby 2.7.2 image. This could easily be extended with more Dockerfiles to cover different Ruby versions if desired. In order to run the specs via Docker:
710
+ There can be behavioural inconsistencies between running the specs on your machine, and in the CI pipeline. Due to this, there is now a Dockerfile included in the project, which is based on the CircleCI ruby 2.7.8 image. This could easily be extended with more Dockerfiles to cover different Ruby versions if desired. In order to run the specs via Docker:
685
711
 
686
712
  - Uncomment the `tests` service from the docker-compose.yml
687
713
  - Bring up the stack with `docker-compose up -d`
688
- - Execute the entire suite with `docker-compose run --rm tests rspec`
689
- - Execute a single spec or directory with `docker-compose run --rm tests rspec spec/integration/consumer_spec.rb`
714
+ - Execute the entire suite with `docker-compose run --rm tests bundle exec rspec`
715
+ - Execute a single spec or directory with `docker-compose run --rm tests bundle exec rspec spec/integration/consumer_spec.rb`
690
716
 
691
717
  Please note - your code directory is mounted as a volume, so you can make code changes without needing to rebuild
692
718
 
@@ -29,6 +29,9 @@ module Racecar
29
29
  desc "The minimum number of messages in the local consumer queue"
30
30
  integer :min_message_queue_size, default: 2000
31
31
 
32
+ desc "Which partition assignment strategy to use, range, roundrobin or cooperative-sticky. -- https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md"
33
+ string :partition_assignment_strategy, default: "range,roundrobin"
34
+
32
35
  desc "Kafka consumer configuration options, separated with '=' -- https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md"
33
36
  list :consumer, default: []
34
37
 
@@ -296,10 +299,6 @@ module Racecar
296
299
  )
297
300
  end
298
301
 
299
- def rebalance_listener
300
- RebalanceListener.new(self)
301
- end
302
-
303
302
  private
304
303
 
305
304
  def rdkafka_security_config
@@ -37,6 +37,10 @@ module Racecar
37
37
  subscriptions << Subscription.new(topic, start_from_beginning, max_bytes_per_partition, additional_config)
38
38
  end
39
39
  end
40
+
41
+ # Rebalance hooks for subclasses to override
42
+ def on_partitions_assigned(rebalance_event); end
43
+ def on_partitions_revoked(rebalance_event); end
40
44
  end
41
45
 
42
46
  def configure(producer:, consumer:, instrumenter: NullInstrumenter, config: Racecar.config)
@@ -15,6 +15,7 @@ module Racecar
15
15
  @previous_retries = 0
16
16
 
17
17
  @last_poll_read_nil_message = false
18
+ @paused_tpls = Hash.new { |h, k| h[k] = {} }
18
19
  end
19
20
 
20
21
  def poll(max_wait_time_ms = @config.max_wait_time_ms)
@@ -65,14 +66,17 @@ module Racecar
65
66
 
66
67
  def close
67
68
  each_subscribed(&:close)
69
+ @paused_tpls.clear
68
70
  end
69
71
 
70
72
  def current
71
73
  @consumers[@consumer_id_iterator.peek] ||= begin
72
74
  consumer_config = Rdkafka::Config.new(rdkafka_config(current_subscription))
73
- consumer_config.consumer_rebalance_listener = @config.rebalance_listener
74
-
75
+ listener = RebalanceListener.new(@config.consumer_class, @instrumenter)
76
+ consumer_config.consumer_rebalance_listener = listener
75
77
  consumer = consumer_config.consumer
78
+ listener.rdkafka_consumer = consumer
79
+
76
80
  @instrumenter.instrument('join_group') do
77
81
  consumer.subscribe current_subscription.topic
78
82
  end
@@ -98,16 +102,25 @@ module Racecar
98
102
  consumer.pause(filtered_tpl)
99
103
  fake_msg = OpenStruct.new(topic: topic, partition: partition, offset: offset)
100
104
  consumer.seek(fake_msg)
105
+
106
+ @paused_tpls[topic][partition] = [consumer, filtered_tpl]
101
107
  end
102
108
 
103
109
  def resume(topic, partition)
104
110
  consumer, filtered_tpl = find_consumer_by(topic, partition)
111
+
112
+ if !consumer && @paused_tpls[topic][partition]
113
+ consumer, filtered_tpl = @paused_tpls[topic][partition]
114
+ end
115
+
105
116
  if !consumer
106
117
  @logger.info "Attempted to resume #{topic}/#{partition}, but we're not subscribed to it"
107
118
  return
108
119
  end
109
120
 
110
121
  consumer.resume(filtered_tpl)
122
+ @paused_tpls[topic].delete(partition)
123
+ @paused_tpls.delete(topic) if @paused_tpls[topic].empty?
111
124
  end
112
125
 
113
126
  alias :each :each_subscribed
@@ -218,6 +231,11 @@ module Racecar
218
231
  def rdkafka_config(subscription)
219
232
  # https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
220
233
  config = {
234
+ # Manually store offset after messages have been processed successfully
235
+ # to avoid marking failed messages as committed. The call just updates
236
+ # a value within librdkafka and is asynchronously written to proper
237
+ # storage through auto commits.
238
+ "enable.auto.offset.store" => false,
221
239
  "auto.commit.interval.ms" => @config.offset_commit_interval * 1000,
222
240
  "auto.offset.reset" => subscription.start_from_beginning ? "earliest" : "largest",
223
241
  "bootstrap.servers" => @config.brokers.join(","),
@@ -233,7 +251,8 @@ module Racecar
233
251
  "queued.min.messages" => @config.min_message_queue_size,
234
252
  "session.timeout.ms" => @config.session_timeout * 1000,
235
253
  "socket.timeout.ms" => @config.socket_timeout * 1000,
236
- "statistics.interval.ms" => @config.statistics_interval_ms
254
+ "statistics.interval.ms" => @config.statistics_interval_ms,
255
+ "partition.assignment.strategy" => @config.partition_assignment_strategy,
237
256
  }
238
257
  config.merge! @config.rdkafka_consumer
239
258
  config.merge! subscription.additional_config
data/lib/racecar/ctl.rb CHANGED
@@ -36,8 +36,10 @@ module Racecar
36
36
  require "racecar/liveness_probe"
37
37
  parse_options!(args)
38
38
 
39
- if ENV["RAILS_ENV"]
40
- Racecar.config.load_file("config/racecar.yml", ENV["RAILS_ENV"])
39
+ RailsConfigFileLoader.load! unless config.without_rails?
40
+
41
+ if File.exist?("config/racecar.rb")
42
+ require "./config/racecar"
41
43
  end
42
44
 
43
45
  Racecar.config.liveness_probe.check_liveness_within_interval!
@@ -136,5 +138,9 @@ module Racecar
136
138
 
137
139
  parser.parse!(args)
138
140
  end
141
+
142
+ def config
143
+ Racecar.config
144
+ end
139
145
  end
140
146
  end
@@ -1,22 +1,58 @@
1
1
  module Racecar
2
2
  class RebalanceListener
3
- def initialize(config)
4
- @config = config
5
- @consumer_class = config.consumer_class
3
+ def initialize(consumer_class, instrumenter)
4
+ @consumer_class = consumer_class
5
+ @instrumenter = instrumenter
6
+ @rdkafka_consumer = nil
6
7
  end
7
8
 
8
- attr_reader :config, :consumer_class
9
+ attr_writer :rdkafka_consumer
9
10
 
10
- def on_partitions_assigned(_consumer, topic_partition_list)
11
- consumer_class.respond_to?(:on_partitions_assigned) &&
12
- consumer_class.on_partitions_assigned(topic_partition_list.to_h)
13
- rescue
11
+ attr_reader :consumer_class, :instrumenter, :rdkafka_consumer
12
+ private :consumer_class, :instrumenter, :rdkafka_consumer
13
+
14
+ def on_partitions_assigned(rdkafka_topic_partition_list)
15
+ event = Event.new(rdkafka_consumer: rdkafka_consumer, rdkafka_topic_partition_list: rdkafka_topic_partition_list)
16
+
17
+ instrument("partitions_assigned", partitions: event.partition_numbers) do
18
+ consumer_class.on_partitions_assigned(event)
19
+ end
20
+ end
21
+
22
+ def on_partitions_revoked(rdkafka_topic_partition_list)
23
+ event = Event.new(rdkafka_consumer: rdkafka_consumer, rdkafka_topic_partition_list: rdkafka_topic_partition_list)
24
+
25
+ instrument("partitions_revoked", partitions: event.partition_numbers) do
26
+ consumer_class.on_partitions_revoked(event)
27
+ end
28
+ end
29
+
30
+ private
31
+
32
+ def instrument(event, payload, &block)
33
+ instrumenter.instrument(event, payload, &block)
14
34
  end
15
35
 
16
- def on_partitions_revoked(_consumer, topic_partition_list)
17
- consumer_class.respond_to?(:on_partitions_revoked) &&
18
- consumer_class.on_partitions_revoked(topic_partition_list.to_h)
19
- rescue
36
+ class Event
37
+ def initialize(rdkafka_topic_partition_list:, rdkafka_consumer:)
38
+ @__rdkafka_topic_partition_list = rdkafka_topic_partition_list
39
+ @__rdkafka_consumer = rdkafka_consumer
40
+ end
41
+
42
+ def topic_name
43
+ __rdkafka_topic_partition_list.to_h.keys.first
44
+ end
45
+
46
+ def partition_numbers
47
+ __rdkafka_topic_partition_list.to_h.values.flatten.map(&:partition)
48
+ end
49
+
50
+ def empty?
51
+ __rdkafka_topic_partition_list.empty?
52
+ end
53
+
54
+ # API private and not guaranteed stable
55
+ attr_reader :__rdkafka_topic_partition_list, :__rdkafka_consumer
20
56
  end
21
57
  end
22
58
  end
@@ -96,7 +96,7 @@ module Racecar
96
96
  end
97
97
  ensure
98
98
  producer.close
99
- Racecar::Datadog.close if Object.const_defined?("Racecar::Datadog")
99
+ Racecar::Datadog.close if config.datadog_enabled
100
100
  @instrumenter.instrument("shut_down", instrumentation_payload || {})
101
101
  end
102
102
 
@@ -131,11 +131,6 @@ module Racecar
131
131
 
132
132
  def consumer
133
133
  @consumer ||= begin
134
- # Manually store offset after messages have been processed successfully
135
- # to avoid marking failed messages as committed. The call just updates
136
- # a value within librdkafka and is asynchronously written to proper
137
- # storage through auto commits.
138
- config.consumer << "enable.auto.offset.store=false"
139
134
  ConsumerSet.new(config, logger, @instrumenter)
140
135
  end
141
136
  end
@@ -213,21 +208,21 @@ module Racecar
213
208
  }
214
209
 
215
210
  @instrumenter.instrument("start_process_batch", instrumentation_payload)
216
- @instrumenter.instrument("process_batch", instrumentation_payload) do
217
- with_pause(first.topic, first.partition, first.offset..last.offset) do |pause|
218
- begin
211
+ with_pause(first.topic, first.partition, first.offset..last.offset) do |pause|
212
+ begin
213
+ @instrumenter.instrument("process_batch", instrumentation_payload) do
219
214
  racecar_messages = messages.map do |message|
220
215
  Racecar::Message.new(message, retries_count: pause.pauses_count)
221
216
  end
222
217
  processor.process_batch(racecar_messages)
223
218
  processor.deliver!
224
219
  consumer.store_offset(messages.last)
225
- rescue => e
226
- instrumentation_payload[:unrecoverable_delivery_error] = reset_producer_on_unrecoverable_delivery_errors(e)
227
- instrumentation_payload[:retries_count] = pause.pauses_count
228
- config.error_handler.call(e, instrumentation_payload)
229
- raise e
230
220
  end
221
+ rescue => e
222
+ instrumentation_payload[:unrecoverable_delivery_error] = reset_producer_on_unrecoverable_delivery_errors(e)
223
+ instrumentation_payload[:retries_count] = pause.pauses_count
224
+ config.error_handler.call(e, instrumentation_payload)
225
+ raise e
231
226
  end
232
227
  end
233
228
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Racecar
4
- VERSION = "2.9.0"
4
+ VERSION = "2.10.0.beta1"
5
5
  end
data/racecar.gemspec CHANGED
@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
23
23
  spec.required_ruby_version = '>= 2.6'
24
24
 
25
25
  spec.add_runtime_dependency "king_konf", "~> 1.0.0"
26
- spec.add_runtime_dependency "rdkafka", "~> 0.12.0"
26
+ spec.add_runtime_dependency "rdkafka", "~> 0.13.0"
27
27
 
28
28
  spec.add_development_dependency "bundler", [">= 1.13", "< 3"]
29
29
  spec.add_development_dependency "pry-byebug"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: racecar
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.9.0
4
+ version: 2.10.0.beta1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Daniel Schierbeck
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: exe
11
11
  cert_chain: []
12
- date: 2023-09-25 00:00:00.000000000 Z
12
+ date: 2023-10-09 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: king_konf
@@ -31,14 +31,14 @@ dependencies:
31
31
  requirements:
32
32
  - - "~>"
33
33
  - !ruby/object:Gem::Version
34
- version: 0.12.0
34
+ version: 0.13.0
35
35
  type: :runtime
36
36
  prerelease: false
37
37
  version_requirements: !ruby/object:Gem::Requirement
38
38
  requirements:
39
39
  - - "~>"
40
40
  - !ruby/object:Gem::Version
41
- version: 0.12.0
41
+ version: 0.13.0
42
42
  - !ruby/object:Gem::Dependency
43
43
  name: bundler
44
44
  requirement: !ruby/object:Gem::Requirement
@@ -225,9 +225,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
225
225
  version: '2.6'
226
226
  required_rubygems_version: !ruby/object:Gem::Requirement
227
227
  requirements:
228
- - - ">="
228
+ - - ">"
229
229
  - !ruby/object:Gem::Version
230
- version: '0'
230
+ version: 1.3.1
231
231
  requirements: []
232
232
  rubygems_version: 3.0.3.1
233
233
  signing_key: