karafka 2.0.0.beta1 → 2.0.0.beta2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/CHANGELOG.md +13 -0
- data/Gemfile.lock +1 -1
- data/config/errors.yml +1 -0
- data/lib/active_job/karafka.rb +2 -2
- data/lib/karafka/active_job/routing/extensions.rb +21 -0
- data/lib/karafka/base_consumer.rb +1 -1
- data/lib/karafka/connection/client.rb +1 -1
- data/lib/karafka/connection/listener.rb +88 -27
- data/lib/karafka/connection/listeners_batch.rb +24 -0
- data/lib/karafka/connection/messages_buffer.rb +50 -54
- data/lib/karafka/connection/raw_messages_buffer.rb +101 -0
- data/lib/karafka/contracts/config.rb +7 -0
- data/lib/karafka/helpers/async.rb +33 -0
- data/lib/karafka/messages/batch_metadata.rb +26 -3
- data/lib/karafka/messages/builders/batch_metadata.rb +17 -29
- data/lib/karafka/messages/builders/message.rb +1 -0
- data/lib/karafka/messages/builders/messages.rb +4 -12
- data/lib/karafka/pro/active_job/consumer.rb +21 -0
- data/lib/karafka/pro/active_job/dispatcher.rb +1 -1
- data/lib/karafka/pro/loader.rb +5 -1
- data/lib/karafka/pro/processing/jobs/consume_non_blocking.rb +38 -0
- data/lib/karafka/pro/scheduler.rb +54 -0
- data/lib/karafka/processing/executor.rb +5 -2
- data/lib/karafka/processing/executors_buffer.rb +15 -7
- data/lib/karafka/processing/jobs/base.rb +13 -1
- data/lib/karafka/processing/jobs/consume.rb +4 -2
- data/lib/karafka/processing/jobs_queue.rb +15 -12
- data/lib/karafka/processing/worker.rb +7 -9
- data/lib/karafka/processing/workers_batch.rb +5 -0
- data/lib/karafka/routing/consumer_group.rb +1 -1
- data/lib/karafka/routing/subscription_group.rb +1 -1
- data/lib/karafka/routing/subscription_groups_builder.rb +3 -2
- data/lib/karafka/routing/topics.rb +38 -0
- data/lib/karafka/runner.rb +19 -27
- data/lib/karafka/scheduler.rb +10 -11
- data/lib/karafka/server.rb +24 -23
- data/lib/karafka/setup/config.rb +1 -0
- data/lib/karafka/version.rb +1 -1
- data.tar.gz.sig +1 -3
- metadata +10 -3
- metadata.gz.sig +0 -0
- data/lib/karafka/active_job/routing_extensions.rb +0 -18
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 7f75623e7d9cdcc4ba151ad551079275528c56bf66cd9c32ecc585756a8d505c
|
4
|
+
data.tar.gz: e0becf53133b579f581ddfdf947bbff21221fe69a8c73a0406174aecd0155f3a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2ef2ac59f1ea60136abbaccf460206a0c2f6d4fe3124eda520f3a19568702acc774e0d9e02eae24cfe6bb6cb8ee8aa74602588aa818dc3537cd6bbc8409f159d
|
7
|
+
data.tar.gz: e29e964e777e2bd8a551458f591b92aea69a3ac2eafa1bc1d75bb42d7cd6bb904abf997724be4268ae1f9d429627d7b92d5c38e51817d36b0f27c6499a062af3
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,18 @@
|
|
1
1
|
# Karafka framework changelog
|
2
2
|
|
3
|
+
## 2.0.0-beta2 (2022-06-07)
|
4
|
+
- Abstract away notion of topics groups (until now it was just an array)
|
5
|
+
- Optimize how jobs queue is closed. Since we enqueue jobs only from the listeners, we can safely close jobs queue once listeners are done. By extracting this responsibility from listeners, we remove corner cases and race conditions. Note here: for non-blocking jobs we do wait for them to finish while running the `poll`. This ensures, that for async jobs that are long-living, we do not reach `max.poll.interval`.
|
6
|
+
- `Shutdown` jobs are executed in workers to align all the jobs behaviours.
|
7
|
+
- `Shutdown` jobs are always blocking.
|
8
|
+
- Notion of `ListenersBatch` was introduced similar to `WorkersBatch` to abstract this concept.
|
9
|
+
- Change default `shutdown_timeout` to be more than `max_wait_time` not to cause forced shutdown when no messages are being received from Kafka.
|
10
|
+
- Abstract away scheduling of revocation and shutdown jobs for both default and pro schedulers
|
11
|
+
- Introduce a second (internal) messages buffer to distinguish between raw messages buffer and karafka messages buffer
|
12
|
+
- Move messages and their metadata remap process to the listener thread to allow for their inline usage
|
13
|
+
- Change how we wait in the shutdown phase, so shutdown jobs can still use Kafka connection even if they run for a longer period of time. This will prevent us from being kicked out from the group early.
|
14
|
+
- Introduce validation that ensures, that `shutdown_timeout` is more than `max_wait_time`. This will prevent users from ending up with a config that could lead to frequent forceful shutdowns.
|
15
|
+
|
3
16
|
## 2.0.0-beta1 (2022-05-22)
|
4
17
|
- Update the jobs queue blocking engine and allow for non-blocking jobs execution
|
5
18
|
- Provide `#prepared` hook that always runs before the fetching loop is unblocked
|
data/Gemfile.lock
CHANGED
data/config/errors.yml
CHANGED
@@ -2,6 +2,7 @@ en:
|
|
2
2
|
dry_validation:
|
3
3
|
errors:
|
4
4
|
max_timeout_vs_pause_max_timeout: pause_timeout must be less or equal to pause_max_timeout
|
5
|
+
shutdown_timeout_vs_max_wait_time: shutdown_timeout must be more than max_wait_time
|
5
6
|
topics_names_not_unique: all topic names within a single consumer group must be unique
|
6
7
|
required_usage_count: Given topic must be used at least once
|
7
8
|
consumer_groups_inclusion: Unknown consumer group
|
data/lib/active_job/karafka.rb
CHANGED
@@ -14,8 +14,8 @@ begin
|
|
14
14
|
# We extend routing builder by adding a simple wrapper for easier jobs topics defining
|
15
15
|
# This needs to be extended here as it is going to be used in karafka routes, hence doing that in
|
16
16
|
# the railtie initializer would be too late
|
17
|
-
::Karafka::Routing::Builder.include ::Karafka::ActiveJob::
|
18
|
-
::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::
|
17
|
+
::Karafka::Routing::Builder.include ::Karafka::ActiveJob::Routing::Extensions
|
18
|
+
::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::Routing::Extensions
|
19
19
|
rescue LoadError
|
20
20
|
# We extend ActiveJob stuff in the railtie
|
21
21
|
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
# ActiveJob related Karafka stuff
|
5
|
+
module ActiveJob
|
6
|
+
# Karafka routing ActiveJob related components
|
7
|
+
module Routing
|
8
|
+
# Routing extensions for ActiveJob
|
9
|
+
module Extensions
|
10
|
+
# This method simplifies routes definition for ActiveJob topics / queues by auto-injecting
|
11
|
+
# the consumer class
|
12
|
+
# @param name [String, Symbol] name of the topic where ActiveJobs jobs should go
|
13
|
+
def active_job_topic(name)
|
14
|
+
topic(name) do
|
15
|
+
consumer App.config.internal.active_job.consumer
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -27,7 +27,7 @@ module Karafka
|
|
27
27
|
pause_tracker.reset
|
28
28
|
|
29
29
|
# Mark as consumed only if manual offset management is not on
|
30
|
-
|
30
|
+
next if topic.manual_offset_management
|
31
31
|
|
32
32
|
# We use the non-blocking one here. If someone needs the blocking one, can implement it
|
33
33
|
# with manual offset management
|
@@ -30,7 +30,7 @@ module Karafka
|
|
30
30
|
@mutex = Mutex.new
|
31
31
|
@closed = false
|
32
32
|
@subscription_group = subscription_group
|
33
|
-
@buffer =
|
33
|
+
@buffer = RawMessagesBuffer.new
|
34
34
|
@rebalance_manager = RebalanceManager.new
|
35
35
|
@kafka = build_consumer
|
36
36
|
# Marks if we need to offset. If we did not store offsets, we should not commit the offset
|
@@ -3,9 +3,13 @@
|
|
3
3
|
module Karafka
|
4
4
|
module Connection
|
5
5
|
# A single listener that listens to incoming messages from a single subscription group.
|
6
|
-
# It polls the messages and then enqueues. It also takes care of potential recovery from
|
6
|
+
# It polls the messages and then enqueues jobs. It also takes care of potential recovery from
|
7
7
|
# critical errors by restarting everything in a safe manner.
|
8
|
+
#
|
9
|
+
# This is the heart of the consumption process.
|
8
10
|
class Listener
|
11
|
+
include Helpers::Async
|
12
|
+
|
9
13
|
# @param subscription_group [Karafka::Routing::SubscriptionGroup]
|
10
14
|
# @param jobs_queue [Karafka::Processing::JobsQueue] queue where we should push work
|
11
15
|
# @return [Karafka::Connection::Listener] listener instance
|
@@ -17,6 +21,10 @@ module Karafka
|
|
17
21
|
@executors = Processing::ExecutorsBuffer.new(@client, subscription_group)
|
18
22
|
# We reference scheduler here as it is much faster than fetching this each time
|
19
23
|
@scheduler = ::Karafka::App.config.internal.scheduler
|
24
|
+
# We keep one buffer for messages to preserve memory and not allocate extra objects
|
25
|
+
# We can do this that way because we always first schedule jobs using messages before we
|
26
|
+
# fetch another batch.
|
27
|
+
@messages_buffer = MessagesBuffer.new(subscription_group)
|
20
28
|
end
|
21
29
|
|
22
30
|
# Runs the main listener fetch loop.
|
@@ -53,33 +61,55 @@ module Karafka
|
|
53
61
|
)
|
54
62
|
|
55
63
|
resume_paused_partitions
|
64
|
+
|
56
65
|
# We need to fetch data before we revoke lost partitions details as during the polling
|
57
66
|
# the callbacks for tracking lost partitions are triggered. Otherwise we would be always
|
58
67
|
# one batch behind.
|
59
|
-
|
68
|
+
poll_and_remap_messages
|
60
69
|
|
61
70
|
Karafka.monitor.instrument(
|
62
71
|
'connection.listener.fetch_loop.received',
|
63
72
|
caller: self,
|
64
|
-
messages_buffer: messages_buffer
|
73
|
+
messages_buffer: @messages_buffer
|
65
74
|
)
|
66
75
|
|
67
76
|
# If there were revoked partitions, we need to wait on their jobs to finish before
|
68
77
|
# distributing consuming jobs as upon revoking, we might get assigned to the same
|
69
78
|
# partitions, thus getting their jobs. The revoking jobs need to finish before
|
70
79
|
# appropriate consumers are taken down and re-created
|
71
|
-
|
72
|
-
|
73
|
-
schedule_partitions_jobs(messages_buffer)
|
80
|
+
build_and_schedule_revoke_lost_partitions_jobs
|
74
81
|
|
75
82
|
# We wait only on jobs from our subscription group. Other groups are independent.
|
76
|
-
wait
|
83
|
+
wait
|
84
|
+
|
85
|
+
build_and_schedule_consumption_jobs
|
86
|
+
|
87
|
+
wait
|
77
88
|
|
78
89
|
# We don't use the `#commit_offsets!` here for performance reasons. This can be achieved
|
79
90
|
# if needed by using manual offset management.
|
80
91
|
@client.commit_offsets
|
81
92
|
end
|
82
93
|
|
94
|
+
# If we are stopping we will no longer schedule any jobs despite polling.
|
95
|
+
# We need to keep polling not to exceed the `max.poll.interval` for long-running
|
96
|
+
# non-blocking jobs and we need to allow them to finish. We however do not want to
|
97
|
+
# enqueue any new jobs. It's worth keeping in mind that it is the end user responsibility
|
98
|
+
# to detect shutdown in their long-running logic or else Karafka will force shutdown
|
99
|
+
# after a while.
|
100
|
+
#
|
101
|
+
# We do not care about resuming any partitions or lost jobs as we do not plan to do
|
102
|
+
# anything with them as we're in the shutdown phase.
|
103
|
+
wait_with_poll
|
104
|
+
|
105
|
+
# We do not want to schedule the shutdown jobs prior to finishing all the jobs
|
106
|
+
# (including non-blocking) as there might be a long-running job with a shutdown and then
|
107
|
+
# we would run two jobs in parallel for the same executor and consumer. We do not want that
|
108
|
+
# as it could create a race-condition.
|
109
|
+
build_and_schedule_shutdown_jobs
|
110
|
+
|
111
|
+
wait_with_poll
|
112
|
+
|
83
113
|
shutdown
|
84
114
|
|
85
115
|
# This is on purpose - see the notes for this method
|
@@ -100,57 +130,88 @@ module Karafka
|
|
100
130
|
|
101
131
|
# Resumes processing of partitions that were paused due to an error.
|
102
132
|
def resume_paused_partitions
|
103
|
-
@pauses_manager.resume
|
133
|
+
@pauses_manager.resume do |topic, partition|
|
134
|
+
@client.resume(topic, partition)
|
135
|
+
end
|
104
136
|
end
|
105
137
|
|
106
138
|
# Enqueues revoking jobs for partitions that were taken away from the running process.
|
107
|
-
|
108
|
-
# @note We do not use scheduler here as those jobs are not meant to be order optimized in
|
109
|
-
# any way. Since they operate occasionally it is irrelevant.
|
110
|
-
def schedule_revoke_lost_partitions_jobs
|
139
|
+
def build_and_schedule_revoke_lost_partitions_jobs
|
111
140
|
revoked_partitions = @client.rebalance_manager.revoked_partitions
|
112
141
|
|
113
|
-
|
142
|
+
# Stop early to save on some execution and array allocation
|
143
|
+
return if revoked_partitions.empty?
|
144
|
+
|
145
|
+
jobs = []
|
114
146
|
|
115
147
|
revoked_partitions.each do |topic, partitions|
|
116
148
|
partitions.each do |partition|
|
117
149
|
pause_tracker = @pauses_manager.fetch(topic, partition)
|
118
150
|
executor = @executors.fetch(topic, partition, pause_tracker)
|
119
|
-
|
151
|
+
jobs << Processing::Jobs::Revoked.new(executor)
|
120
152
|
end
|
121
153
|
end
|
122
154
|
|
123
|
-
|
155
|
+
@scheduler.schedule_revocation(@jobs_queue, jobs)
|
124
156
|
end
|
125
157
|
|
126
|
-
#
|
158
|
+
# Enqueues the shutdown jobs for all the executors that exist in our subscription group
|
159
|
+
def build_and_schedule_shutdown_jobs
|
160
|
+
jobs = []
|
161
|
+
|
162
|
+
@executors.each do |_, _, executor|
|
163
|
+
jobs << Processing::Jobs::Shutdown.new(executor)
|
164
|
+
end
|
165
|
+
|
166
|
+
@scheduler.schedule_shutdown(@jobs_queue, jobs)
|
167
|
+
end
|
168
|
+
|
169
|
+
# Polls messages within the time and amount boundaries defined in the settings and then
|
170
|
+
# builds karafka messages based on the raw rdkafka messages buffer returned by the
|
171
|
+
# `#batch_poll` method.
|
127
172
|
#
|
128
|
-
# @
|
129
|
-
def
|
130
|
-
@
|
173
|
+
# @note There are two buffers, one for raw messages and one for "built" karafka messages
|
174
|
+
def poll_and_remap_messages
|
175
|
+
@messages_buffer.remap(
|
176
|
+
@client.batch_poll
|
177
|
+
)
|
178
|
+
end
|
179
|
+
|
180
|
+
# Takes the messages per topic partition and enqueues processing jobs in threads using
|
181
|
+
# given scheduler.
|
182
|
+
def build_and_schedule_consumption_jobs
|
183
|
+
return if @messages_buffer.empty?
|
184
|
+
|
185
|
+
jobs = []
|
186
|
+
|
187
|
+
@messages_buffer.each do |topic, partition, messages|
|
131
188
|
pause = @pauses_manager.fetch(topic, partition)
|
132
189
|
|
133
190
|
next if pause.paused?
|
134
191
|
|
135
192
|
executor = @executors.fetch(topic, partition, pause)
|
136
193
|
|
137
|
-
|
194
|
+
jobs << Processing::Jobs::Consume.new(executor, messages)
|
138
195
|
end
|
196
|
+
|
197
|
+
@scheduler.schedule_consumption(@jobs_queue, jobs)
|
139
198
|
end
|
140
199
|
|
141
200
|
# Waits for all the jobs from a given subscription group to finish before moving forward
|
142
|
-
|
143
|
-
|
144
|
-
|
201
|
+
def wait
|
202
|
+
@jobs_queue.wait(@subscription_group.id)
|
203
|
+
end
|
204
|
+
|
205
|
+
# Waits without blocking the polling
|
206
|
+
# This should be used only when we no longer plan to use any incoming data and we can safely
|
207
|
+
# discard it
|
208
|
+
def wait_with_poll
|
209
|
+
@client.batch_poll until @jobs_queue.empty?(@subscription_group.id)
|
145
210
|
end
|
146
211
|
|
147
212
|
# Stops the jobs queue, triggers shutdown on all the executors (sync), commits offsets and
|
148
213
|
# stops kafka client.
|
149
214
|
def shutdown
|
150
|
-
@jobs_queue.close
|
151
|
-
# This runs synchronously, making sure we finish all the shutdowns before we stop the
|
152
|
-
# client.
|
153
|
-
@executors.shutdown
|
154
215
|
@client.commit_offsets!
|
155
216
|
@client.stop
|
156
217
|
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module Connection
|
5
|
+
# Abstraction layer around listeners batch.
|
6
|
+
class ListenersBatch
|
7
|
+
include Enumerable
|
8
|
+
|
9
|
+
# @param jobs_queue [JobsQueue]
|
10
|
+
# @return [ListenersBatch]
|
11
|
+
def initialize(jobs_queue)
|
12
|
+
@batch = App.subscription_groups.map do |subscription_group|
|
13
|
+
Connection::Listener.new(subscription_group, jobs_queue)
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
# Iterates over available listeners and yields each listener
|
18
|
+
# @param block [Proc] block we want to run
|
19
|
+
def each(&block)
|
20
|
+
@batch.each(&block)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -2,20 +2,26 @@
|
|
2
2
|
|
3
3
|
module Karafka
|
4
4
|
module Connection
|
5
|
-
# Buffer
|
6
|
-
# When message is added to this buffer, it gets assigned to an array with other messages from
|
7
|
-
# the same topic and partition.
|
5
|
+
# Buffer used to build and store karafka messages built based on raw librdkafka messages.
|
8
6
|
#
|
9
|
-
#
|
7
|
+
# Why do we have two buffers? `RawMessagesBuffer` is used to store raw messages and to handle
|
8
|
+
# cases related to partition revocation and reconnections. It is "internal" to the listening
|
9
|
+
# process. `MessagesBuffer` on the other hand is used to "translate" those raw messages that
|
10
|
+
# we know that are ok into Karafka messages and to simplify further work with them.
|
11
|
+
#
|
12
|
+
# While it adds a bit of overhead, it makes conceptual things much easier and it adds only two
|
13
|
+
# simple hash iterations over messages batch.
|
14
|
+
#
|
15
|
+
# @note This buffer is NOT thread safe. We do not worry about it as we do not use it outside
|
16
|
+
# of the main listener loop. It can be cleared after the jobs are scheduled with messages
|
17
|
+
# it stores, because messages arrays are not "cleared" in any way directly and their
|
18
|
+
# reference stays.
|
10
19
|
class MessagesBuffer
|
11
20
|
attr_reader :size
|
12
21
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
# @return [Karafka::Connection::MessagesBuffer] buffer instance
|
18
|
-
def initialize
|
22
|
+
# @param subscription_group [Karafka::Routing::SubscriptionGroup]
|
23
|
+
def initialize(subscription_group)
|
24
|
+
@subscription_group = subscription_group
|
19
25
|
@size = 0
|
20
26
|
@groups = Hash.new do |topic_groups, topic|
|
21
27
|
topic_groups[topic] = Hash.new do |partition_groups, partition|
|
@@ -24,64 +30,54 @@ module Karafka
|
|
24
30
|
end
|
25
31
|
end
|
26
32
|
|
27
|
-
#
|
28
|
-
#
|
29
|
-
|
30
|
-
|
31
|
-
def <<(message)
|
32
|
-
@size += 1
|
33
|
-
@groups[message.topic][message.partition] << message
|
34
|
-
end
|
33
|
+
# Remaps raw messages from the raw messages buffer to Karafka messages
|
34
|
+
# @param raw_messages_buffer [RawMessagesBuffer] buffer with raw messages
|
35
|
+
def remap(raw_messages_buffer)
|
36
|
+
clear unless @size.zero?
|
35
37
|
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
# @param partition [Integer] partition of which data we want to remove
|
40
|
-
def delete(topic, partition)
|
41
|
-
return unless @groups.key?(topic)
|
42
|
-
return unless @groups.fetch(topic).key?(partition)
|
38
|
+
# Since it happens "right after" we've received the messages, it is close enough it time
|
39
|
+
# to be used as the moment we received messages.
|
40
|
+
received_at = Time.now
|
43
41
|
|
44
|
-
|
45
|
-
|
42
|
+
raw_messages_buffer.each do |topic, partition, messages|
|
43
|
+
@size += messages.count
|
46
44
|
|
47
|
-
|
45
|
+
ktopic = @subscription_group.topics.find(topic)
|
48
46
|
|
49
|
-
|
50
|
-
|
47
|
+
@groups[topic][partition] = messages.map do |message|
|
48
|
+
Messages::Builders::Message.call(
|
49
|
+
message,
|
50
|
+
ktopic,
|
51
|
+
received_at
|
52
|
+
)
|
53
|
+
end
|
54
|
+
end
|
51
55
|
end
|
52
56
|
|
53
|
-
#
|
54
|
-
#
|
55
|
-
#
|
56
|
-
#
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
57
|
+
# Allows to iterate over all the topics and partitions messages
|
58
|
+
#
|
59
|
+
# @yieldparam [String] topic name
|
60
|
+
# @yieldparam [Integer] partition number
|
61
|
+
# @yieldparam [Array<Karafka::Messages::Message>] messages from a given topic partition
|
62
|
+
def each
|
63
|
+
@groups.each do |topic, partitions|
|
64
|
+
partitions.each do |partition, messages|
|
65
|
+
yield(topic, partition, messages)
|
61
66
|
end
|
62
67
|
end
|
63
|
-
|
64
|
-
recount!
|
65
68
|
end
|
66
69
|
|
67
|
-
#
|
68
|
-
|
69
|
-
|
70
|
-
# we save ourselves some objects allocations. We cannot clear the underlying arrays as they
|
71
|
-
# may be used in other threads for data processing, thus if we would clear it, we could
|
72
|
-
# potentially clear a raw messages array for a job that is in the jobs queue.
|
73
|
-
def clear
|
74
|
-
@size = 0
|
75
|
-
@groups.each_value(&:clear)
|
70
|
+
# @return [Boolean] is the buffer empty or does it contain any messages
|
71
|
+
def empty?
|
72
|
+
@size.zero?
|
76
73
|
end
|
77
74
|
|
78
75
|
private
|
79
76
|
|
80
|
-
#
|
81
|
-
def
|
82
|
-
@size =
|
83
|
-
|
84
|
-
end
|
77
|
+
# Clears the buffer completely
|
78
|
+
def clear
|
79
|
+
@size = 0
|
80
|
+
@groups.clear
|
85
81
|
end
|
86
82
|
end
|
87
83
|
end
|
@@ -0,0 +1,101 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module Connection
|
5
|
+
# Buffer for raw librdkafka messages.
|
6
|
+
#
|
7
|
+
# When message is added to this buffer, it gets assigned to an array with other messages from
|
8
|
+
# the same topic and partition.
|
9
|
+
#
|
10
|
+
# @note This buffer is NOT threadsafe.
|
11
|
+
#
|
12
|
+
# @note We store data here in groups per topic partition to handle the revocation case, where
|
13
|
+
# we may need to remove messages from a single topic partition.
|
14
|
+
class RawMessagesBuffer
|
15
|
+
attr_reader :size
|
16
|
+
|
17
|
+
# @return [Karafka::Connection::MessagesBuffer] buffer instance
|
18
|
+
def initialize
|
19
|
+
@size = 0
|
20
|
+
@groups = Hash.new do |topic_groups, topic|
|
21
|
+
topic_groups[topic] = Hash.new do |partition_groups, partition|
|
22
|
+
partition_groups[partition] = []
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
# Adds a message to the buffer.
|
28
|
+
#
|
29
|
+
# @param message [Rdkafka::Consumer::Message] raw rdkafka message
|
30
|
+
# @return [Array<Rdkafka::Consumer::Message>] given partition topic sub-buffer array
|
31
|
+
def <<(message)
|
32
|
+
@size += 1
|
33
|
+
@groups[message.topic][message.partition] << message
|
34
|
+
end
|
35
|
+
|
36
|
+
# Allows to iterate over all the topics and partitions messages
|
37
|
+
#
|
38
|
+
# @yieldparam [String] topic name
|
39
|
+
# @yieldparam [Integer] partition number
|
40
|
+
# @yieldparam [Array<Rdkafka::Consumer::Message>] topic partition aggregated results
|
41
|
+
def each
|
42
|
+
@groups.each do |topic, partitions|
|
43
|
+
partitions.each do |partition, messages|
|
44
|
+
yield(topic, partition, messages)
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
# Removes given topic and partition data out of the buffer
|
50
|
+
# This is used when there's a partition revocation
|
51
|
+
# @param topic [String] topic we're interested in
|
52
|
+
# @param partition [Integer] partition of which data we want to remove
|
53
|
+
def delete(topic, partition)
|
54
|
+
return unless @groups.key?(topic)
|
55
|
+
return unless @groups.fetch(topic).key?(partition)
|
56
|
+
|
57
|
+
topic_data = @groups.fetch(topic)
|
58
|
+
topic_data.delete(partition)
|
59
|
+
|
60
|
+
recount!
|
61
|
+
|
62
|
+
# If there are no more partitions to handle in a given topic, remove it completely
|
63
|
+
@groups.delete(topic) if topic_data.empty?
|
64
|
+
end
|
65
|
+
|
66
|
+
# Removes duplicated messages from the same partitions
|
67
|
+
# This should be used only when rebalance occurs, as we may get data again we already have
|
68
|
+
# due to the processing from the last offset. In cases like this, we may get same data
|
69
|
+
# again and we do want to ensure as few duplications as possible
|
70
|
+
def uniq!
|
71
|
+
@groups.each_value do |partitions|
|
72
|
+
partitions.each_value do |messages|
|
73
|
+
messages.uniq!(&:offset)
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
recount!
|
78
|
+
end
|
79
|
+
|
80
|
+
# Removes all the data from the buffer.
|
81
|
+
#
|
82
|
+
# @note We do not clear the whole groups hash but rather we clear the partition hashes, so
|
83
|
+
# we save ourselves some objects allocations. We cannot clear the underlying arrays as they
|
84
|
+
# may be used in other threads for data processing, thus if we would clear it, we could
|
85
|
+
# potentially clear a raw messages array for a job that is in the jobs queue.
|
86
|
+
def clear
|
87
|
+
@size = 0
|
88
|
+
@groups.each_value(&:clear)
|
89
|
+
end
|
90
|
+
|
91
|
+
private
|
92
|
+
|
93
|
+
# Updates the messages count if we performed any operations that could change the state
|
94
|
+
def recount!
|
95
|
+
@size = @groups.each_value.sum do |partitions|
|
96
|
+
partitions.each_value.map(&:count).sum
|
97
|
+
end
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
end
|
@@ -25,6 +25,7 @@ module Karafka
|
|
25
25
|
required(:pause_max_timeout) { int? & gt?(0) }
|
26
26
|
required(:pause_with_exponential_backoff).filled(:bool?)
|
27
27
|
required(:shutdown_timeout) { int? & gt?(0) }
|
28
|
+
required(:max_wait_time) { int? & gt?(0) }
|
28
29
|
required(:kafka).filled(:hash)
|
29
30
|
|
30
31
|
# We validate internals just to be sure, that they are present and working
|
@@ -53,6 +54,12 @@ module Karafka
|
|
53
54
|
key(:pause_timeout).failure(:max_timeout_vs_pause_max_timeout)
|
54
55
|
end
|
55
56
|
end
|
57
|
+
|
58
|
+
rule(:shutdown_timeout, :max_wait_time) do
|
59
|
+
if values[:max_wait_time].to_i >= values[:shutdown_timeout].to_i
|
60
|
+
key(:shutdown_timeout).failure(:shutdown_timeout_vs_max_wait_time)
|
61
|
+
end
|
62
|
+
end
|
56
63
|
end
|
57
64
|
end
|
58
65
|
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module Helpers
|
5
|
+
# Allows a given class to run async in a separate thread. Provides also few methods we may
|
6
|
+
# want to use to control the underlying thread
|
7
|
+
#
|
8
|
+
# @note Thread running code needs to manage it's own exceptions. If they leak out, they will
|
9
|
+
# abort thread on exception.
|
10
|
+
module Async
|
11
|
+
class << self
|
12
|
+
# Adds forwardable to redirect thread-based control methods to the underlying thread that
|
13
|
+
# runs the async operations
|
14
|
+
#
|
15
|
+
# @param base [Class] class we're including this module in
|
16
|
+
def included(base)
|
17
|
+
base.extend ::Forwardable
|
18
|
+
|
19
|
+
base.def_delegators :@thread, :join, :terminate, :alive?
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
# Runs the `#call` method in a new thread
|
24
|
+
def async_call
|
25
|
+
@thread = Thread.new do
|
26
|
+
Thread.current.abort_on_exception = true
|
27
|
+
|
28
|
+
call
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
@@ -13,10 +13,33 @@ module Karafka
|
|
13
13
|
:deserializer,
|
14
14
|
:partition,
|
15
15
|
:topic,
|
16
|
+
:created_at,
|
16
17
|
:scheduled_at,
|
17
|
-
:
|
18
|
-
:processing_lag,
|
18
|
+
:processed_at,
|
19
19
|
keyword_init: true
|
20
|
-
)
|
20
|
+
) do
|
21
|
+
# This lag describes how long did it take for a message to be consumed from the moment it was
|
22
|
+
# created
|
23
|
+
def consumption_lag
|
24
|
+
time_distance_in_ms(processed_at, created_at)
|
25
|
+
end
|
26
|
+
|
27
|
+
# This lag describes how long did a batch have to wait before it was picked up by one of the
|
28
|
+
# workers
|
29
|
+
def processing_lag
|
30
|
+
time_distance_in_ms(processed_at, scheduled_at)
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
# Computes time distance in between two times in ms
|
36
|
+
#
|
37
|
+
# @param time1 [Time]
|
38
|
+
# @param time2 [Time]
|
39
|
+
# @return [Integer] distance in between two times in ms
|
40
|
+
def time_distance_in_ms(time1, time2)
|
41
|
+
((time1 - time2) * 1_000).round
|
42
|
+
end
|
43
|
+
end
|
21
44
|
end
|
22
45
|
end
|