karafka 2.0.3 → 2.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/CHANGELOG.md +18 -1
- data/Gemfile.lock +1 -1
- data/README.md +9 -9
- data/bin/integrations +14 -4
- data/bin/karafka +6 -0
- data/certs/cert_chain.pem +26 -0
- data/config/errors.yml +1 -0
- data/karafka.gemspec +2 -2
- data/lib/karafka/admin.rb +13 -7
- data/lib/karafka/base_consumer.rb +21 -5
- data/lib/karafka/connection/client.rb +6 -6
- data/lib/karafka/connection/listener.rb +9 -5
- data/lib/karafka/contracts/consumer_group.rb +2 -2
- data/lib/karafka/contracts/consumer_group_topic.rb +10 -9
- data/lib/karafka/instrumentation/vendors/datadog/dashboard.json +1 -1
- data/lib/karafka/messages/builders/batch_metadata.rb +2 -3
- data/lib/karafka/messages/builders/messages.rb +3 -1
- data/lib/karafka/pro/base_consumer.rb +32 -3
- data/lib/karafka/pro/processing/coordinator.rb +14 -0
- data/lib/karafka/pro/processing/jobs/consume_non_blocking.rb +3 -2
- data/lib/karafka/processing/executor.rb +14 -6
- data/lib/karafka/processing/jobs/base.rb +4 -0
- data/lib/karafka/processing/jobs/consume.rb +7 -2
- data/lib/karafka/processing/result.rb +9 -1
- data/lib/karafka/processing/worker.rb +0 -1
- data/lib/karafka/routing/subscription_groups_builder.rb +1 -0
- data/lib/karafka/routing/topic.rb +3 -1
- data/lib/karafka/templates/karafka.rb.erb +1 -3
- data/lib/karafka/version.rb +1 -1
- data.tar.gz.sig +0 -0
- metadata +28 -27
- metadata.gz.sig +0 -0
- data/certs/mensfeld.pem +0 -25
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f7c672af31943cc38566d944bc4cb0466145e98a2130295f3ca24c275dccdc64
|
4
|
+
data.tar.gz: 7eeaea273e18f31a29c79591a2627c2f79f0915663157cc9e31e1dd354c9deb0
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: '032765549abe7a8afb1866d6b8ac89ba711a8a04f1b7370018663fc140aa0d30a3966151d1524f49d12b1dfe7a4fafb72a008d0af2fe39ffc959e34c385f5469'
|
7
|
+
data.tar.gz: 10194fcd2ada3015b1bcd549ce286dafd5f94051b98a1752e2b5d9767d7c99ceaff47ce84429e345a99fb920f51cfabd4b5779d29da99b1e11fa6981e33b437f
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,22 @@
|
|
1
1
|
# Karafka framework changelog
|
2
2
|
|
3
|
+
## 2.0.6 (2022-09-02)
|
4
|
+
- Improve client closing.
|
5
|
+
- Fix for: Multiple LRJ topics fetched concurrently block ability for LRJ to kick in (#1002)
|
6
|
+
- Introduce a pre-enqueue sync execution layer to prevent starvation cases for LRJ
|
7
|
+
- Close admin upon critical errors to prevent segmentation faults
|
8
|
+
- Add support for manual subscription group management (#852)
|
9
|
+
|
10
|
+
## 2.0.5 (2022-08-23)
|
11
|
+
- Fix unnecessary double new line in the `karafka.rb` template for Ruby on Rails
|
12
|
+
- Fix a case where a manually paused partition would not be processed after rebalance (#988)
|
13
|
+
- Increase specs stability.
|
14
|
+
- Lower concurrency of execution of specs in Github CI.
|
15
|
+
|
16
|
+
## 2.0.4 (2022-08-19)
|
17
|
+
- Fix hanging topic creation (#964)
|
18
|
+
- Fix conflict with other Rails loading libraries like `gruf` (#974)
|
19
|
+
|
3
20
|
## 2.0.3 (2022-08-09)
|
4
21
|
- Update boot info on server startup.
|
5
22
|
- Update `karafka info` with more descriptive Ruby version info.
|
@@ -45,7 +62,7 @@ Karafka 2.0:
|
|
45
62
|
- Introduces official [EOL policies](https://github.com/karafka/karafka/wiki/Versions-Lifecycle-and-EOL).
|
46
63
|
- Introduces [benchmarks](https://github.com/karafka/karafka/tree/master/spec/benchmarks) that can be used to profile Karafka.
|
47
64
|
- Introduces a requirement that the end user code **needs** to be [thread-safe](https://github.com/karafka/karafka/wiki/FAQ#does-karafka-require-gems-to-be-thread-safe).
|
48
|
-
- Introduces a [Pro subscription](https://github.com/karafka/karafka/wiki/Build-vs
|
65
|
+
- Introduces a [Pro subscription](https://github.com/karafka/karafka/wiki/Build-vs-Buy) with a [commercial license](https://github.com/karafka/karafka/blob/master/LICENSE-COMM) to fund further ecosystem development.
|
49
66
|
|
50
67
|
### Deletions
|
51
68
|
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -8,12 +8,12 @@
|
|
8
8
|
|
9
9
|
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
|
10
10
|
|
11
|
-
- Supports parallel processing in [multiple threads](https://
|
12
|
-
- Has [ActiveJob backend](https://
|
13
|
-
- [Automatically integrates](https://
|
14
|
-
- Supports in-development [code reloading](https://
|
11
|
+
- Supports parallel processing in [multiple threads](https://karafka.io/docs/Concurrency-and-multithreading) (also for a [single topic partition](https://karafka.io/docs/Pro-Virtual-Partitions) work)
|
12
|
+
- Has [ActiveJob backend](https://karafka.io/docs/Active-Job) support (including [ordered jobs](https://karafka.io/docs/Pro-Enhanced-Active-Job#ordered-jobs))
|
13
|
+
- [Automatically integrates](https://karafka.io/docs/Integrating-with-Ruby-on-Rails-and-other-frameworks#integrating-with-ruby-on-rails) with Ruby on Rails
|
14
|
+
- Supports in-development [code reloading](https://karafka.io/docs/Auto-reload-of-code-changes-in-development)
|
15
15
|
- Is powered by [librdkafka](https://github.com/edenhill/librdkafka) (the Apache Kafka C/C++ client library)
|
16
|
-
- Has an out-of the box [StatsD/DataDog monitoring](https://
|
16
|
+
- Has an out-of the box [StatsD/DataDog monitoring](https://karafka.io/docs/Monitoring-and-logging) with a dashboard template.
|
17
17
|
|
18
18
|
```ruby
|
19
19
|
# Define what topics you want to consume with which consumers in karafka.rb
|
@@ -42,13 +42,13 @@ If you're entirely new to the subject, you can start with our "Kafka on Rails" a
|
|
42
42
|
- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages](https://mensfeld.pl/2017/11/kafka-on-rails-using-kafka-with-ruby-on-rails-part-1-kafka-basics-and-its-advantages/)
|
43
43
|
- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Rails and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
|
44
44
|
|
45
|
-
If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit our [Getting started](https://
|
45
|
+
If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit our [Getting started](https://karafka.io/docs/Getting-Started) guides and the [example apps repository](https://github.com/karafka/example-apps).
|
46
46
|
|
47
47
|
We also maintain many [integration specs](https://github.com/karafka/karafka/tree/master/spec/integrations) illustrating various use-cases and features of the framework.
|
48
48
|
|
49
49
|
### TL;DR (1 minute from setup to publishing and consuming messages)
|
50
50
|
|
51
|
-
**Prerequisites**: Kafka running. You can start it by following instructions from [here](https://
|
51
|
+
**Prerequisites**: Kafka running. You can start it by following instructions from [here](https://karafka.io/docs/Setting-up-Kafka).
|
52
52
|
|
53
53
|
1. Add and install Karafka:
|
54
54
|
|
@@ -85,8 +85,8 @@ Help me provide high-quality open-source software. Please see the Karafka [homep
|
|
85
85
|
|
86
86
|
## Support
|
87
87
|
|
88
|
-
Karafka has [Wiki pages](https://
|
88
|
+
Karafka has [Wiki pages](https://karafka.io/docs) for almost everything and a pretty decent [FAQ](https://karafka.io/docs/FAQ). It covers the installation, setup, and deployment, along with other useful details on how to run Karafka.
|
89
89
|
|
90
90
|
If you have questions about using Karafka, feel free to join our [Slack](https://slack.karafka.io) channel.
|
91
91
|
|
92
|
-
Karafka has priority support for technical and architectural questions that is part of the Karafka Pro subscription.
|
92
|
+
Karafka has [priority support](https://karafka.io/docs/Pro-Support) for technical and architectural questions that is part of the Karafka Pro subscription.
|
data/bin/integrations
CHANGED
@@ -19,7 +19,7 @@ ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../
|
|
19
19
|
# When the value is high, there's a problem with thread allocation on Github CI, tht is why
|
20
20
|
# we limit it. Locally we can run a lot of those, as many of them have sleeps and do not use a lot
|
21
21
|
# of CPU
|
22
|
-
CONCURRENCY = ENV.key?('CI') ?
|
22
|
+
CONCURRENCY = ENV.key?('CI') ? 3 : Etc.nprocessors * 2
|
23
23
|
|
24
24
|
# How may bytes do we want to keep from the stdout in the buffer for when we need to print it
|
25
25
|
MAX_BUFFER_OUTPUT = 51_200
|
@@ -28,7 +28,7 @@ MAX_BUFFER_OUTPUT = 51_200
|
|
28
28
|
class Scenario
|
29
29
|
# How long a scenario can run before we kill it
|
30
30
|
# This is a fail-safe just in case something would hang
|
31
|
-
MAX_RUN_TIME =
|
31
|
+
MAX_RUN_TIME = 5 * 60 # 5 minutes tops
|
32
32
|
|
33
33
|
# There are rare cases where Karafka may force shutdown for some of the integration cases
|
34
34
|
# This includes exactly those
|
@@ -47,6 +47,8 @@ class Scenario
|
|
47
47
|
# @param path [String] path to the scenarios file
|
48
48
|
def initialize(path)
|
49
49
|
@path = path
|
50
|
+
# First 1024 characters from stdout
|
51
|
+
@stdout_head = ''
|
50
52
|
# Last 1024 characters from stdout
|
51
53
|
@stdout_tail = ''
|
52
54
|
end
|
@@ -75,8 +77,6 @@ class Scenario
|
|
75
77
|
def finished?
|
76
78
|
# If the thread is running too long, kill it
|
77
79
|
if current_time - @started_at > MAX_RUN_TIME
|
78
|
-
@wait_thr.kill
|
79
|
-
|
80
80
|
begin
|
81
81
|
Process.kill('TERM', pid)
|
82
82
|
# It may finish right after we want to kill it, that's why we ignore this
|
@@ -88,6 +88,7 @@ class Scenario
|
|
88
88
|
# to stdout. Otherwise after reaching the buffer size, it would hang
|
89
89
|
buffer = ''
|
90
90
|
@stdout.read_nonblock(MAX_BUFFER_OUTPUT, buffer, exception: false)
|
91
|
+
@stdout_head = buffer if @stdout_head.empty?
|
91
92
|
@stdout_tail << buffer
|
92
93
|
@stdout_tail = @stdout_tail[-MAX_BUFFER_OUTPUT..-1] || @stdout_tail
|
93
94
|
|
@@ -112,6 +113,11 @@ class Scenario
|
|
112
113
|
@wait_thr.value&.exitstatus || 123
|
113
114
|
end
|
114
115
|
|
116
|
+
# @return [String] exit status of the process
|
117
|
+
def exit_status
|
118
|
+
@wait_thr.value.to_s
|
119
|
+
end
|
120
|
+
|
115
121
|
# Prints a status report when scenario is finished and stdout if it failed
|
116
122
|
def report
|
117
123
|
if success?
|
@@ -123,7 +129,11 @@ class Scenario
|
|
123
129
|
|
124
130
|
puts
|
125
131
|
puts "\e[#{31}m#{'[FAILED]'}\e[0m #{name}"
|
132
|
+
puts "Time taken: #{current_time - @started_at} seconds"
|
126
133
|
puts "Exit code: #{exit_code}"
|
134
|
+
puts "Exit status: #{exit_status}"
|
135
|
+
puts @stdout_head
|
136
|
+
puts '...'
|
127
137
|
puts @stdout_tail
|
128
138
|
puts buffer
|
129
139
|
puts
|
data/bin/karafka
CHANGED
@@ -9,6 +9,12 @@ ENV['KARAFKA_CLI'] = 'true'
|
|
9
9
|
# If there is a boot file, we need to require it as we expect it to contain
|
10
10
|
# Karafka app setup, routes, etc
|
11
11
|
if File.exist?(Karafka.boot_file)
|
12
|
+
rails_env_rb = File.join(Dir.pwd, 'config/environment.rb')
|
13
|
+
|
14
|
+
# Load Rails environment file that starts Rails, so we can reference consumers and other things
|
15
|
+
# from `karafka.rb` file. This will work only for Rails, for non-rails a manual setup is needed
|
16
|
+
require rails_env_rb if Kernel.const_defined?(:Rails) && File.exist?(rails_env_rb)
|
17
|
+
|
12
18
|
require Karafka.boot_file.to_s
|
13
19
|
else
|
14
20
|
# However when it is unavailable, we still want to be able to run help command
|
@@ -0,0 +1,26 @@
|
|
1
|
+
-----BEGIN CERTIFICATE-----
|
2
|
+
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
3
|
+
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
4
|
+
MB4XDTIyMDgxOTE3MjEzN1oXDTIzMDgxOTE3MjEzN1owPzEQMA4GA1UEAwwHY29u
|
5
|
+
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
6
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAODzeO3L6lxdATzMHKNW
|
7
|
+
jFA/GGunoPuylO/BMzy8RiQHh7VIvysAKs0tHhTx3g2D0STDpF+hcQcPELFikiT2
|
8
|
+
F+1wOHj/SsrK7VKqfA8+gq04hKc5sQoX2Egf9k3V0YJ3eZ6R/koHkQ8A0TVt0w6F
|
9
|
+
ZQckoV4MqnEAx0g/FZN3mnHTlJ3VFLSBqJEIe+S6FZMl92mSv+hTrlUG8VaYxSfN
|
10
|
+
lTCvnKk284F6QZq5XIENLRmcDd/3aPBLnLwNnyMyhB+6gK8cUO+CFlDO5tjo/aBA
|
11
|
+
rUnl++wGG0JooF1ed0v+evOn9KoMBG6rHewcf79qJbVOscbD8qSAmo+sCXtcFryr
|
12
|
+
KRMTB8gNbowJkFRJDEe8tfRy11u1fYzFg/qNO82FJd62rKAw2wN0C29yCeQOPRb1
|
13
|
+
Cw9Y4ZwK9VFNEcV9L+3pHTHn2XfuZHtDaG198VweiF6raFO4yiEYccodH/USP0L5
|
14
|
+
cbcCFtmu/4HDSxL1ByQXO84A0ybJuk3/+aPUSXe9C9U8fwIDAQABo3cwdTAJBgNV
|
15
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQUSlcEakb7gfn/5E2WY6z73BF/
|
16
|
+
iZkwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
17
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEA1aS+E7RXJ1w9g9mJ
|
18
|
+
G0NzFxe64OEuENosNlvYQCbRKGCXAU1qqelYkBQHseRgRKxLICrnypRo9IEobyHa
|
19
|
+
vDnJ4r7Tsb34dleqQW2zY/obG+cia3Ym2JsegXWF7dDOzCXJ4FN8MFoT2jHlqLLw
|
20
|
+
yrap0YO5zx0GSQ0Dwy8h2n2v2vanMEeCx7iNm3ERgR5WuN5sjzWoz2A/JLEEcK0C
|
21
|
+
EnAGKCWAd1fuG8IemDjT1edsd5FyYR4bIX0m+99oDuFZyPiiIbalmyYiSBBp59Yb
|
22
|
+
Q0P8zeBi4OfwCZNcxqz0KONmw9JLNv6DgyEAH5xe/4JzhMEgvIRiPj0pHfA7oqQF
|
23
|
+
KUNqvD1KlxbEC+bZfE5IZhnqYLdld/Ksqd22FI1RBhiS1Ejfsj99LVIm9cBuZEY2
|
24
|
+
Qf04B9ceLUaC4fPVEz10FyobjaFoY4i32xRto3XnrzeAgfEe4swLq8bQsR3w/EF3
|
25
|
+
MGU0FeSV2Yj7Xc2x/7BzLK8xQn5l7Yy75iPF+KP3vVmDHnNl
|
26
|
+
-----END CERTIFICATE-----
|
data/config/errors.yml
CHANGED
@@ -35,6 +35,7 @@ en:
|
|
35
35
|
consumer_format: needs to be present
|
36
36
|
id_format: 'needs to be a string with a Kafka accepted format'
|
37
37
|
initial_offset_format: needs to be either earliest or latest
|
38
|
+
subscription_group_format: must be nil or a non-empty string
|
38
39
|
|
39
40
|
consumer_group:
|
40
41
|
missing: needs to be present
|
data/karafka.gemspec
CHANGED
@@ -10,7 +10,7 @@ Gem::Specification.new do |spec|
|
|
10
10
|
spec.version = ::Karafka::VERSION
|
11
11
|
spec.platform = Gem::Platform::RUBY
|
12
12
|
spec.authors = ['Maciej Mensfeld']
|
13
|
-
spec.email = %w[
|
13
|
+
spec.email = %w[contact@karafka.io]
|
14
14
|
spec.homepage = 'https://karafka.io'
|
15
15
|
spec.summary = 'Efficient Kafka processing framework for Ruby and Rails'
|
16
16
|
spec.description = 'Framework used to simplify Apache Kafka based Ruby applications development'
|
@@ -28,7 +28,7 @@ Gem::Specification.new do |spec|
|
|
28
28
|
spec.signing_key = File.expand_path('~/.ssh/gem-private_key.pem')
|
29
29
|
end
|
30
30
|
|
31
|
-
spec.cert_chain = %w[certs/
|
31
|
+
spec.cert_chain = %w[certs/cert_chain.pem]
|
32
32
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
|
33
33
|
spec.executables = %w[karafka]
|
34
34
|
spec.require_paths = %w[lib]
|
data/lib/karafka/admin.rb
CHANGED
@@ -19,9 +19,9 @@ module Karafka
|
|
19
19
|
# https://kafka.apache.org/documentation/#topicconfigs
|
20
20
|
def create_topic(name, partitions, replication_factor, topic_config = {})
|
21
21
|
with_admin do |admin|
|
22
|
-
admin
|
23
|
-
|
24
|
-
|
22
|
+
admin.create_topic(name, partitions, replication_factor, topic_config)
|
23
|
+
|
24
|
+
sleep(0.1) until topics_names.include?(name)
|
25
25
|
end
|
26
26
|
end
|
27
27
|
|
@@ -30,9 +30,9 @@ module Karafka
|
|
30
30
|
# @param name [String] topic name
|
31
31
|
def delete_topic(name)
|
32
32
|
with_admin do |admin|
|
33
|
-
admin
|
34
|
-
|
35
|
-
|
33
|
+
admin.delete_topic(name)
|
34
|
+
|
35
|
+
sleep(0.1) while topics_names.include?(name)
|
36
36
|
end
|
37
37
|
end
|
38
38
|
|
@@ -45,12 +45,18 @@ module Karafka
|
|
45
45
|
|
46
46
|
private
|
47
47
|
|
48
|
+
# @return [Array<String>] topics names
|
49
|
+
def topics_names
|
50
|
+
cluster_info.topics.map { |topic| topic.fetch(:topic_name) }
|
51
|
+
end
|
52
|
+
|
48
53
|
# Creates admin instance and yields it. After usage it closes the admin instance
|
49
54
|
def with_admin
|
50
55
|
admin = ::Rdkafka::Config.new(Karafka::App.config.kafka).admin
|
51
56
|
result = yield(admin)
|
52
|
-
admin.close
|
53
57
|
result
|
58
|
+
ensure
|
59
|
+
admin&.close
|
54
60
|
end
|
55
61
|
end
|
56
62
|
end
|
@@ -15,13 +15,24 @@ module Karafka
|
|
15
15
|
# @return [Waterdrop::Producer] producer instance
|
16
16
|
attr_accessor :producer
|
17
17
|
|
18
|
-
# Can be used to run preparation code
|
18
|
+
# Can be used to run preparation code prior to the job being enqueued
|
19
19
|
#
|
20
20
|
# @private
|
21
|
-
# @note This should not be used by the end users as it is part of the lifecycle of things
|
21
|
+
# @note This should not be used by the end users as it is part of the lifecycle of things and
|
22
|
+
# not as a part of the public api. This should not perform any extensive operations as it is
|
23
|
+
# blocking and running in the listener thread.
|
24
|
+
def on_before_enqueue; end
|
25
|
+
|
26
|
+
# Can be used to run preparation code in the worker
|
27
|
+
#
|
28
|
+
# @private
|
29
|
+
# @note This should not be used by the end users as it is part of the lifecycle of things and
|
22
30
|
# not as part of the public api. This can act as a hook when creating non-blocking
|
23
31
|
# consumers and doing other advanced stuff
|
24
|
-
def on_before_consume
|
32
|
+
def on_before_consume
|
33
|
+
messages.metadata.processed_at = Time.now
|
34
|
+
messages.metadata.freeze
|
35
|
+
end
|
25
36
|
|
26
37
|
# Executes the default consumer flow.
|
27
38
|
#
|
@@ -37,7 +48,7 @@ module Karafka
|
|
37
48
|
|
38
49
|
coordinator.consumption(self).success!
|
39
50
|
rescue StandardError => e
|
40
|
-
coordinator.consumption(self).failure!
|
51
|
+
coordinator.consumption(self).failure!(e)
|
41
52
|
|
42
53
|
Karafka.monitor.instrument(
|
43
54
|
'error.occurred',
|
@@ -70,10 +81,15 @@ module Karafka
|
|
70
81
|
end
|
71
82
|
end
|
72
83
|
|
73
|
-
# Trigger method for running on
|
84
|
+
# Trigger method for running on partition revocation.
|
74
85
|
#
|
75
86
|
# @private
|
76
87
|
def on_revoked
|
88
|
+
# We need to always un-pause the processing in case we have lost a given partition.
|
89
|
+
# Otherwise the underlying librdkafka would not know we may want to continue processing and
|
90
|
+
# the pause could in theory last forever
|
91
|
+
resume
|
92
|
+
|
77
93
|
coordinator.revoke
|
78
94
|
|
79
95
|
Karafka.monitor.instrument('consumer.revoked', caller: self) do
|
@@ -275,16 +275,16 @@ module Karafka
|
|
275
275
|
|
276
276
|
# Commits the stored offsets in a sync way and closes the consumer.
|
277
277
|
def close
|
278
|
-
# Once client is closed, we should not close it again
|
279
|
-
# This could only happen in case of a race-condition when forceful shutdown happens
|
280
|
-
# and triggers this from a different thread
|
281
|
-
return if @closed
|
282
|
-
|
283
278
|
@mutex.synchronize do
|
284
|
-
|
279
|
+
# Once client is closed, we should not close it again
|
280
|
+
# This could only happen in case of a race-condition when forceful shutdown happens
|
281
|
+
# and triggers this from a different thread
|
282
|
+
return if @closed
|
285
283
|
|
286
284
|
@closed = true
|
287
285
|
|
286
|
+
internal_commit_offsets(async: false)
|
287
|
+
|
288
288
|
# Remove callbacks runners that were registered
|
289
289
|
::Karafka::Instrumentation.statistics_callbacks.delete(@subscription_group.id)
|
290
290
|
::Karafka::Instrumentation.error_callbacks.delete(@subscription_group.id)
|
@@ -185,7 +185,9 @@ module Karafka
|
|
185
185
|
# processed (if it was assigned and revoked really fast), thus we may not have it
|
186
186
|
# here. In cases like this, we do not run a revocation job
|
187
187
|
@executors.find_all(topic, partition).each do |executor|
|
188
|
-
|
188
|
+
job = @jobs_builder.revoked(executor)
|
189
|
+
job.before_enqueue
|
190
|
+
jobs << job
|
189
191
|
end
|
190
192
|
|
191
193
|
# We need to remove all the executors of a given topic partition that we have lost, so
|
@@ -205,7 +207,9 @@ module Karafka
|
|
205
207
|
jobs = []
|
206
208
|
|
207
209
|
@executors.each do |_, _, executor|
|
208
|
-
|
210
|
+
job = @jobs_builder.shutdown(executor)
|
211
|
+
job.before_enqueue
|
212
|
+
jobs << job
|
209
213
|
end
|
210
214
|
|
211
215
|
@scheduler.schedule_shutdown(@jobs_queue, jobs)
|
@@ -238,10 +242,10 @@ module Karafka
|
|
238
242
|
@partitioner.call(topic, messages) do |group_id, partition_messages|
|
239
243
|
# Count the job we're going to create here
|
240
244
|
coordinator.increment
|
241
|
-
|
242
245
|
executor = @executors.find_or_create(topic, partition, group_id)
|
243
|
-
|
244
|
-
|
246
|
+
job = @jobs_builder.consume(executor, partition_messages, coordinator)
|
247
|
+
job.before_enqueue
|
248
|
+
jobs << job
|
245
249
|
end
|
246
250
|
end
|
247
251
|
|
@@ -12,8 +12,8 @@ module Karafka
|
|
12
12
|
).fetch('en').fetch('validations').fetch('consumer_group')
|
13
13
|
end
|
14
14
|
|
15
|
-
required(:id) { |
|
16
|
-
required(:topics) { |
|
15
|
+
required(:id) { |val| val.is_a?(String) && Contracts::TOPIC_REGEXP.match?(val) }
|
16
|
+
required(:topics) { |val| val.is_a?(Array) && !val.empty? }
|
17
17
|
|
18
18
|
virtual do |data, errors|
|
19
19
|
next unless errors.empty?
|
@@ -12,15 +12,16 @@ module Karafka
|
|
12
12
|
).fetch('en').fetch('validations').fetch('consumer_group_topic')
|
13
13
|
end
|
14
14
|
|
15
|
-
required(:consumer) { |
|
16
|
-
required(:deserializer) { |
|
17
|
-
required(:id) { |
|
18
|
-
required(:kafka) { |
|
19
|
-
required(:max_messages) { |
|
20
|
-
required(:initial_offset) { |
|
21
|
-
required(:max_wait_time) { |
|
22
|
-
required(:manual_offset_management) { |
|
23
|
-
required(:name) { |
|
15
|
+
required(:consumer) { |val| !val.nil? }
|
16
|
+
required(:deserializer) { |val| !val.nil? }
|
17
|
+
required(:id) { |val| val.is_a?(String) && Contracts::TOPIC_REGEXP.match?(val) }
|
18
|
+
required(:kafka) { |val| val.is_a?(Hash) && !val.empty? }
|
19
|
+
required(:max_messages) { |val| val.is_a?(Integer) && val >= 1 }
|
20
|
+
required(:initial_offset) { |val| %w[earliest latest].include?(val) }
|
21
|
+
required(:max_wait_time) { |val| val.is_a?(Integer) && val >= 10 }
|
22
|
+
required(:manual_offset_management) { |val| [true, false].include?(val) }
|
23
|
+
required(:name) { |val| val.is_a?(String) && Contracts::TOPIC_REGEXP.match?(val) }
|
24
|
+
required(:subscription_group) { |val| val.nil? || (val.is_a?(String) && !val.empty?) }
|
24
25
|
|
25
26
|
virtual do |data, errors|
|
26
27
|
next unless errors.empty?
|
@@ -1 +1 @@
|
|
1
|
-
{"title":"Karafka monitoring dashboard","description":"","widgets":[{"id":7444969424381053,"definition":{"title":"Stability & errors","type":"group","show_title":true,"layout_type":"ordered","widgets":[{"id":8304008422587936,"definition":{"title":"Client connects and disconnects","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Connects","formula":"query1"},{"alias":"Disconnects","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.connection.connects{*} by {host}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.connection.disconnects{*} by {host}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"bars"}]},"layout":{"x":0,"y":0,"width":4,"height":2}},{"id":3722865443336921,"definition":{"title":"Errors encountered (any)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"in-karafka errors","formula":"query1"},{"alias":"librdkafka consume errors","formula":"query2"},{"alias":"librdkafka receive errors","formula":"query3"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.error_occurred{*} by {type}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consume.errors{*}.as_count()","data_source":"metrics","name":"query2"},{"query":"sum:karafka.receive.errors{*}.as_count()","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":0,"width":4,"height":2}},{"id":5477381252952760,"definition":{"title":"Processing errors","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"query1"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.error_occurred{type:consumer.consume.error} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":0,"width":4,"height":2}},{"id":2357301680769076,"definition":{"title":"Processing errors rate per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"% error rate per topic","formula":"(query1 / (query1 + query2)) * 100"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.error_occurred{type:consumer.consume.error} by {topic,partition}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batches{*} by {topic,partition}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"bars"}]},"layout":{"x":0,"y":2,"width":4,"height":2}},{"id":3902930069982135,"definition":{"title":"Batches successful vs failures","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Successfully processed batch","formula":"query1"},{"alias":"Batch processing with error","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.batches{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"},{"query":"avg:karafka.error_occurred{type:consumer.consume.error} by {partition,topic}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":2,"width":4,"height":2}},{"id":718749162159145,"definition":{"title":"Consumer instances revocations and shutdowns","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Consumer instances revokations","formula":"query1"},{"alias":"Consumer instances shutdowns","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.revoked{*}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.shutdown{*}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":2,"width":4,"height":2}}]},"layout":{"x":0,"y":0,"width":12,"height":5}},{"id":5988438511387100,"definition":{"title":"Workers poll","type":"group","show_title":true,"layout_type":"ordered","widgets":[{"id":8769294644934352,"definition":{"title":"Enqueued jobs","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Enqueued jobs","formula":"query1"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.worker.enqueued_jobs.avg{*}","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":0,"width":4,"height":2}},{"id":2714502141463873,"definition":{"title":"Workers usage","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Busy workers (p95)","formula":"query1"},{"alias":"Total workers","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.worker.processing.95percentile{*}","data_source":"metrics","name":"query1"},{"query":"sum:karafka.worker.total_threads{*}","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":0,"width":4,"height":2}},{"id":5370086629441984,"definition":{"title":"Workers % utilization","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"% workers utilization","formula":"(query1 / query2) * 100"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.worker.processing.95percentile{*}","data_source":"metrics","name":"query1"},{"query":"sum:karafka.worker.total_threads{*}","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":0,"width":4,"height":2}}]},"layout":{"x":0,"y":5,"width":12,"height":3}},{"id":8544040083223278,"definition":{"title":"Throughput ","type":"group","show_title":true,"layout_type":"ordered","widgets":[{"id":3740207481939733,"definition":{"title":"Offset lag changes","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"derivative(query1)"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.offset{*} by {topic,partition}","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":0,"width":4,"height":2}},{"id":6319110548544878,"definition":{"title":"Batches processed per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"query1"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.batches{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":0,"width":4,"height":2}},{"id":6232784865331443,"definition":{"title":"Messages consumed per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Messages consumed","formula":"query1"},{"alias":"Average batch size","formula":"query1 / query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.messages{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batches{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":0,"width":4,"height":2}},{"id":2321394598982770,"definition":{"title":"Consumption lag (in seconds)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Consumption lag in s (max)","formula":"query2 / 1000"},{"alias":"Consumption lag in s (avg)","formula":"query3 / 1000"},{"alias":"Consumption lag in s (p95)","formula":"query1 / 1000"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.consumption_lag.max{*}","data_source":"metrics","name":"query2"},{"query":"max:karafka.consumer.consumption_lag.avg{*}","data_source":"metrics","name":"query3"},{"query":"max:karafka.consumer.consumption_lag.95percentile{*}","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":2,"width":4,"height":2}},{"id":1062074781483741,"definition":{"title":"Processing lag (in ms)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Processing lag in ms (p95)","formula":"query1"},{"alias":"Processing lag in ms (max)","formula":"query2"},{"alias":"Processing lag in ms (avg)","formula":"query3"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.processing_lag.95percentile{*}","data_source":"metrics","name":"query1"},{"query":"max:karafka.consumer.processing_lag.max{*}","data_source":"metrics","name":"query2"},{"query":"max:karafka.consumer.processing_lag.avg{*}","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":2,"width":4,"height":2}},{"id":7497794728674267,"definition":{"title":"Batch processing time","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"query1"},{"formula":"query2"},{"formula":"query3"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.consumed.time_taken.95percentile{*} by {topic,partition}","data_source":"metrics","name":"query1"},{"query":"max:karafka.consumer.consumed.time_taken.max{*} by {topic,partition}","data_source":"metrics","name":"query2"},{"query":"max:karafka.consumer.consumed.time_taken.avg{*} by {topic,partition}","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":2,"width":4,"height":2}},{"id":4192833027984161,"definition":{"title":"Batch size per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Batch size p95","formula":"query1"},{"alias":"Batch size avg","formula":"query2"},{"alias":"Batch size max","formula":"query3"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.batch_size.95percentile{*} by {partition,topic}","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batch_size.avg{*} by {partition,topic}","data_source":"metrics","name":"query2"},{"query":"sum:karafka.consumer.batch_size.max{*} by {partition,topic}","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":4,"width":4,"height":2}},{"id":4741598444771147,"definition":{"title":"Messages consumed overall","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Messages consumed","formula":"query1"},{"alias":"Average batch size","formula":"query1 / query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.messages{*}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batches{*}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":4,"width":4,"height":2}},{"id":4502534794102513,"definition":{"title":"Polling times (ms)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"time":{},"type":"timeseries","requests":[{"formulas":[{"alias":"p95 ms polling time","formula":"query1"},{"alias":"max ms polling time","formula":"query2"},{"alias":"average ms polling time","formula":"query3"}],"queries":[{"name":"query1","data_source":"metrics","query":"avg:karafka.listener.polling.time_taken.95percentile{*}"},{"name":"query2","data_source":"metrics","query":"avg:karafka.listener.polling.time_taken.max{*}"},{"name":"query3","data_source":"metrics","query":"avg:karafka.listener.polling.time_taken.avg{*}"}],"response_format":"timeseries","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":4,"width":4,"height":2}}]},"layout":{"x":0,"y":0,"width":12,"height":7,"is_column_break":true}}],"template_variables":[],"layout_type":"ordered","is_read_only":false,"notify_list":[],"reflow_type":"fixed","id":"s3u-z47-i6u"}
|
1
|
+
{"title":"Karafka monitoring dashboard","description":"","widgets":[{"id":7444969424381053,"definition":{"title":"Stability & errors","type":"group","show_title":true,"layout_type":"ordered","widgets":[{"id":8304008422587936,"definition":{"title":"Client connects and disconnects","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Connects","formula":"query1"},{"alias":"Disconnects","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.connection.connects{*} by {host}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.connection.disconnects{*} by {host}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"bars"}]},"layout":{"x":0,"y":0,"width":4,"height":2}},{"id":3722865443336921,"definition":{"title":"Errors encountered (any)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"in-karafka errors","formula":"query1"},{"alias":"librdkafka consume errors","formula":"query2"},{"alias":"librdkafka receive errors","formula":"query3"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.error_occurred{*} by {type}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consume.errors{*}.as_count()","data_source":"metrics","name":"query2"},{"query":"sum:karafka.receive.errors{*}.as_count()","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":0,"width":4,"height":2}},{"id":5477381252952760,"definition":{"title":"Processing errors","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"query1"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.error_occurred{type:consumer.consume.error} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":0,"width":4,"height":2}},{"id":2357301680769076,"definition":{"title":"Processing errors rate per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"% error rate per topic","formula":"(query1 / (query1 + query2)) * 100"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.error_occurred{type:consumer.consume.error} by {topic,partition}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batches{*} by {topic,partition}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"bars"}]},"layout":{"x":0,"y":2,"width":4,"height":2}},{"id":3902930069982135,"definition":{"title":"Batches successful vs failures","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Successfully processed batch","formula":"query1"},{"alias":"Batch processing with error","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.batches{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"},{"query":"avg:karafka.error_occurred{type:consumer.consume.error} by {partition,topic}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":2,"width":4,"height":2}},{"id":718749162159145,"definition":{"title":"Consumer instances revocations and shutdowns","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Consumer instances revokations","formula":"query1"},{"alias":"Consumer instances shutdowns","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.revoked{*}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.shutdown{*}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":2,"width":4,"height":2}}]},"layout":{"x":0,"y":0,"width":12,"height":5}},{"id":5988438511387100,"definition":{"title":"Workers poll","type":"group","show_title":true,"layout_type":"ordered","widgets":[{"id":8769294644934352,"definition":{"title":"Enqueued jobs","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Enqueued jobs","formula":"query1"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.worker.enqueued_jobs.avg{*}","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":0,"width":4,"height":2}},{"id":2714502141463873,"definition":{"title":"Workers usage","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Busy workers (p95)","formula":"query1"},{"alias":"Total workers","formula":"query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.worker.processing.95percentile{*}","data_source":"metrics","name":"query1"},{"query":"sum:karafka.worker.total_threads{*}","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":0,"width":4,"height":2}},{"id":5370086629441984,"definition":{"title":"Workers % utilization","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"% workers utilization","formula":"(query1 / query2) * 100"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.worker.processing.95percentile{*}","data_source":"metrics","name":"query1"},{"query":"sum:karafka.worker.total_threads{*}","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":0,"width":4,"height":2}}]},"layout":{"x":0,"y":5,"width":12,"height":3}},{"id":8544040083223278,"definition":{"title":"Throughput ","type":"group","show_title":true,"layout_type":"ordered","widgets":[{"id":3740207481939733,"definition":{"title":"Offset lag changes","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"derivative(query1)"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.offset{*} by {topic,partition}","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":0,"width":4,"height":2}},{"id":6319110548544878,"definition":{"title":"Batches processed per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"query1"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.batches{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":0,"width":4,"height":2}},{"id":6232784865331443,"definition":{"title":"Messages consumed per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Messages consumed","formula":"query1"},{"alias":"Average batch size","formula":"query1 / query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.messages{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batches{*} by {partition,topic}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":0,"width":4,"height":2}},{"id":2321394598982770,"definition":{"title":"Consumption lag (in seconds)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Consumption lag in s (max)","formula":"query2 / 1000"},{"alias":"Consumption lag in s (avg)","formula":"query3 / 1000"},{"alias":"Consumption lag in s (p95)","formula":"query1 / 1000"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.consumption_lag.max{*}","data_source":"metrics","name":"query2"},{"query":"max:karafka.consumer.consumption_lag.avg{*}","data_source":"metrics","name":"query3"},{"query":"max:karafka.consumer.consumption_lag.95percentile{*}","data_source":"metrics","name":"query1"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":2,"width":4,"height":2}},{"id":1062074781483741,"definition":{"title":"Processing lag (in ms)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Processing lag in ms (p95)","formula":"query1"},{"alias":"Processing lag in ms (max)","formula":"query2"},{"alias":"Processing lag in ms (avg)","formula":"query3"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.processing_lag.95percentile{*}","data_source":"metrics","name":"query1"},{"query":"max:karafka.consumer.processing_lag.max{*}","data_source":"metrics","name":"query2"},{"query":"max:karafka.consumer.processing_lag.avg{*}","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":2,"width":4,"height":2}},{"id":7497794728674267,"definition":{"title":"Batch processing time","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"formula":"query1"},{"formula":"query2"},{"formula":"query3"}],"response_format":"timeseries","queries":[{"query":"max:karafka.consumer.consumed.time_taken.95percentile{*} by {topic,partition}","data_source":"metrics","name":"query1"},{"query":"max:karafka.consumer.consumed.time_taken.max{*} by {topic,partition}","data_source":"metrics","name":"query2"},{"query":"max:karafka.consumer.consumed.time_taken.avg{*} by {topic,partition}","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":2,"width":4,"height":2}},{"id":4192833027984161,"definition":{"title":"Batch size per topic","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Batch size p95","formula":"query1"},{"alias":"Batch size avg","formula":"query2"},{"alias":"Batch size max","formula":"query3"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.batch_size.95percentile{*} by {partition,topic}","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batch_size.avg{*} by {partition,topic}","data_source":"metrics","name":"query2"},{"query":"sum:karafka.consumer.batch_size.max{*} by {partition,topic}","data_source":"metrics","name":"query3"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":0,"y":4,"width":4,"height":2}},{"id":4741598444771147,"definition":{"title":"Messages consumed overall","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"type":"timeseries","requests":[{"formulas":[{"alias":"Messages consumed","formula":"query1"},{"alias":"Average batch size","formula":"query1 / query2"}],"response_format":"timeseries","queries":[{"query":"sum:karafka.consumer.messages{*}.as_count()","data_source":"metrics","name":"query1"},{"query":"sum:karafka.consumer.batches{*}.as_count()","data_source":"metrics","name":"query2"}],"style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":4,"y":4,"width":4,"height":2}},{"id":4502534794102513,"definition":{"title":"Polling times (ms)","title_size":"16","title_align":"left","show_legend":true,"legend_layout":"auto","legend_columns":["avg","min","max","value","sum"],"time":{},"type":"timeseries","requests":[{"formulas":[{"alias":"p95 ms polling time","formula":"query1"},{"alias":"max ms polling time","formula":"query2"},{"alias":"average ms polling time","formula":"query3"}],"queries":[{"name":"query1","data_source":"metrics","query":"avg:karafka.listener.polling.time_taken.95percentile{*}"},{"name":"query2","data_source":"metrics","query":"avg:karafka.listener.polling.time_taken.max{*}"},{"name":"query3","data_source":"metrics","query":"avg:karafka.listener.polling.time_taken.avg{*}"}],"response_format":"timeseries","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}]},"layout":{"x":8,"y":4,"width":4,"height":2}}]},"layout":{"x":0,"y":0,"width":12,"height":7,"is_column_break":true}}],"template_variables":[],"layout_type":"ordered","is_read_only":false,"notify_list":[],"reflow_type":"fixed","id":"s3u-z47-i6u"}
|
@@ -28,9 +28,8 @@ module Karafka
|
|
28
28
|
created_at: messages.last.timestamp,
|
29
29
|
# When this batch was built and scheduled for execution
|
30
30
|
scheduled_at: scheduled_at,
|
31
|
-
#
|
32
|
-
|
33
|
-
processed_at: Time.now
|
31
|
+
# This needs to be set to a correct value prior to processing starting
|
32
|
+
processed_at: nil
|
34
33
|
)
|
35
34
|
end
|
36
35
|
end
|
@@ -14,11 +14,13 @@ module Karafka
|
|
14
14
|
# @param received_at [Time] moment in time when the messages were received
|
15
15
|
# @return [Karafka::Messages::Messages] messages batch object
|
16
16
|
def call(messages, topic, received_at)
|
17
|
+
# We cannot freeze the batch metadata because it is altered with the processed_at time
|
18
|
+
# prior to the consumption. It is being frozen there
|
17
19
|
metadata = BatchMetadata.call(
|
18
20
|
messages,
|
19
21
|
topic,
|
20
22
|
received_at
|
21
|
-
)
|
23
|
+
)
|
22
24
|
|
23
25
|
Karafka::Messages::Messages.new(
|
24
26
|
messages,
|
@@ -23,13 +23,17 @@ module Karafka
|
|
23
23
|
|
24
24
|
private_constant :MAX_PAUSE_TIME
|
25
25
|
|
26
|
-
# Pauses processing of a given partition until we're done with the processing
|
26
|
+
# Pauses processing of a given partition until we're done with the processing.
|
27
27
|
# This ensures, that we can easily poll not reaching the `max.poll.interval`
|
28
|
-
|
28
|
+
# @note This needs to happen in the listener thread, because we cannot wait on this being
|
29
|
+
# executed in the workers. Workers may be already running some LRJ jobs that are blocking
|
30
|
+
# all the threads until finished, yet unless we pause the incoming partitions information,
|
31
|
+
# we may be kicked out of the consumer group due to not polling often enough
|
32
|
+
def on_before_enqueue
|
29
33
|
return unless topic.long_running_job?
|
30
34
|
|
31
35
|
# This ensures, that when running LRJ with VP, things operate as expected
|
32
|
-
coordinator.
|
36
|
+
coordinator.on_enqueued do |first_group_message|
|
33
37
|
# Pause at the first message in a batch. That way in case of a crash, we will not loose
|
34
38
|
# any messages
|
35
39
|
pause(first_group_message.offset, MAX_PAUSE_TIME)
|
@@ -44,6 +48,29 @@ module Karafka
|
|
44
48
|
end
|
45
49
|
end
|
46
50
|
|
51
|
+
# Trigger method for running on partition revocation.
|
52
|
+
#
|
53
|
+
# @private
|
54
|
+
def on_revoked
|
55
|
+
# We do not want to resume on revocation in case of a LRJ.
|
56
|
+
# For LRJ we resume after the successful processing or do a backoff pause in case of a
|
57
|
+
# failure. Double non-blocking resume could cause problems in coordination.
|
58
|
+
resume unless topic.long_running_job?
|
59
|
+
|
60
|
+
coordinator.revoke
|
61
|
+
|
62
|
+
Karafka.monitor.instrument('consumer.revoked', caller: self) do
|
63
|
+
revoked
|
64
|
+
end
|
65
|
+
rescue StandardError => e
|
66
|
+
Karafka.monitor.instrument(
|
67
|
+
'error.occurred',
|
68
|
+
error: e,
|
69
|
+
caller: self,
|
70
|
+
type: 'consumer.revoked.error'
|
71
|
+
)
|
72
|
+
end
|
73
|
+
|
47
74
|
private
|
48
75
|
|
49
76
|
# Handles the post-consumption flow depending on topic settings
|
@@ -74,6 +101,8 @@ module Karafka
|
|
74
101
|
resume
|
75
102
|
else
|
76
103
|
# If processing failed, we need to pause
|
104
|
+
# For long running job this will overwrite the default never-ending pause and will cause
|
105
|
+
# the processing th keep going after the error backoff
|
77
106
|
pause(@seek_offset || first_message.offset)
|
78
107
|
end
|
79
108
|
end
|
@@ -18,6 +18,7 @@ module Karafka
|
|
18
18
|
# @param args [Object] anything the base coordinator accepts
|
19
19
|
def initialize(*args)
|
20
20
|
super
|
21
|
+
@on_enqueued_invoked = false
|
21
22
|
@on_started_invoked = false
|
22
23
|
@on_finished_invoked = false
|
23
24
|
@flow_lock = Mutex.new
|
@@ -30,6 +31,7 @@ module Karafka
|
|
30
31
|
super
|
31
32
|
|
32
33
|
@mutex.synchronize do
|
34
|
+
@on_enqueued_invoked = false
|
33
35
|
@on_started_invoked = false
|
34
36
|
@on_finished_invoked = false
|
35
37
|
@first_message = messages.first
|
@@ -42,6 +44,18 @@ module Karafka
|
|
42
44
|
@running_jobs.zero?
|
43
45
|
end
|
44
46
|
|
47
|
+
# Runs synchronized code once for a collective of virtual partitions prior to work being
|
48
|
+
# enqueued
|
49
|
+
def on_enqueued
|
50
|
+
@flow_lock.synchronize do
|
51
|
+
return if @on_enqueued_invoked
|
52
|
+
|
53
|
+
@on_enqueued_invoked = true
|
54
|
+
|
55
|
+
yield(@first_message, @last_message)
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
45
59
|
# Runs given code only once per all the coordinated jobs upon starting first of them
|
46
60
|
def on_started
|
47
61
|
@flow_lock.synchronize do
|
@@ -25,8 +25,9 @@ module Karafka
|
|
25
25
|
# @note It needs to be working with a proper consumer that will handle the partition
|
26
26
|
# management. This layer of the framework knows nothing about Kafka messages consumption.
|
27
27
|
class ConsumeNonBlocking < ::Karafka::Processing::Jobs::Consume
|
28
|
-
#
|
29
|
-
|
28
|
+
# Makes this job non-blocking from the start
|
29
|
+
# @param args [Array] any arguments accepted by `::Karafka::Processing::Jobs::Consume`
|
30
|
+
def initialize(*args)
|
30
31
|
super
|
31
32
|
@non_blocking = true
|
32
33
|
end
|
@@ -37,14 +37,17 @@ module Karafka
|
|
37
37
|
@topic = topic
|
38
38
|
end
|
39
39
|
|
40
|
-
#
|
41
|
-
#
|
40
|
+
# Allows us to prepare the consumer in the listener thread prior to the job being send to
|
41
|
+
# the queue. It also allows to run some code that is time sensitive and cannot wait in the
|
42
|
+
# queue as it could cause starvation.
|
42
43
|
#
|
43
44
|
# @param messages [Array<Karafka::Messages::Message>]
|
44
|
-
# @param received_at [Time] the moment we've received the batch (actually the moment we've)
|
45
|
-
# enqueued it, but good enough
|
46
45
|
# @param coordinator [Karafka::Processing::Coordinator] coordinator for processing management
|
47
|
-
def
|
46
|
+
def before_enqueue(messages, coordinator)
|
47
|
+
# the moment we've received the batch or actually the moment we've enqueued it,
|
48
|
+
# but good enough
|
49
|
+
@enqueued_at = Time.now
|
50
|
+
|
48
51
|
# Recreate consumer with each batch if persistence is not enabled
|
49
52
|
# We reload the consumers with each batch instead of relying on some external signals
|
50
53
|
# when needed for consistency. That way devs may have it on or off and not in this
|
@@ -57,9 +60,14 @@ module Karafka
|
|
57
60
|
consumer.messages = Messages::Builders::Messages.call(
|
58
61
|
messages,
|
59
62
|
@topic,
|
60
|
-
|
63
|
+
@enqueued_at
|
61
64
|
)
|
62
65
|
|
66
|
+
consumer.on_before_enqueue
|
67
|
+
end
|
68
|
+
|
69
|
+
# Runs setup and warm-up code in the worker prior to running the consumption
|
70
|
+
def before_consume
|
63
71
|
consumer.on_before_consume
|
64
72
|
end
|
65
73
|
|
@@ -22,6 +22,10 @@ module Karafka
|
|
22
22
|
@non_blocking = false
|
23
23
|
end
|
24
24
|
|
25
|
+
# When redefined can run any code prior to the job being enqueued
|
26
|
+
# @note This will run in the listener thread and not in the worker
|
27
|
+
def before_enqueue; end
|
28
|
+
|
25
29
|
# When redefined can run any code that should run before executing the proper code
|
26
30
|
def before_call; end
|
27
31
|
|
@@ -18,13 +18,18 @@ module Karafka
|
|
18
18
|
@executor = executor
|
19
19
|
@messages = messages
|
20
20
|
@coordinator = coordinator
|
21
|
-
@created_at = Time.now
|
22
21
|
super()
|
23
22
|
end
|
24
23
|
|
24
|
+
# Runs all the preparation code on the executor that needs to happen before the job is
|
25
|
+
# enqueued.
|
26
|
+
def before_enqueue
|
27
|
+
executor.before_enqueue(@messages, @coordinator)
|
28
|
+
end
|
29
|
+
|
25
30
|
# Runs the before consumption preparations on the executor
|
26
31
|
def before_call
|
27
|
-
executor.before_consume
|
32
|
+
executor.before_consume
|
28
33
|
end
|
29
34
|
|
30
35
|
# Runs the given executor
|
@@ -6,8 +6,11 @@ module Karafka
|
|
6
6
|
# It allows to indicate if given thing moved from success to a failure or the other way around
|
7
7
|
# Useful for tracking consumption state
|
8
8
|
class Result
|
9
|
+
attr_reader :cause
|
10
|
+
|
9
11
|
def initialize
|
10
12
|
@success = true
|
13
|
+
@cause = false
|
11
14
|
end
|
12
15
|
|
13
16
|
# @return [Boolean]
|
@@ -18,11 +21,16 @@ module Karafka
|
|
18
21
|
# Marks state as successful
|
19
22
|
def success!
|
20
23
|
@success = true
|
24
|
+
# We set cause to false so the previous error that occurred does not leak when error is
|
25
|
+
# no longer present
|
26
|
+
@cause = false
|
21
27
|
end
|
22
28
|
|
23
29
|
# Marks state as failure
|
24
|
-
|
30
|
+
# @param cause [StandardError] error that occurred and caused failure
|
31
|
+
def failure!(cause)
|
25
32
|
@success = false
|
33
|
+
@cause = cause
|
26
34
|
end
|
27
35
|
end
|
28
36
|
end
|
@@ -8,6 +8,7 @@ module Karafka
|
|
8
8
|
class Topic
|
9
9
|
attr_reader :id, :name, :consumer_group
|
10
10
|
attr_writer :consumer
|
11
|
+
attr_accessor :subscription_group
|
11
12
|
|
12
13
|
# Attributes we can inherit from the root unless they were defined on this level
|
13
14
|
INHERITABLE_ATTRIBUTES = %i[
|
@@ -91,7 +92,8 @@ module Karafka
|
|
91
92
|
id: id,
|
92
93
|
name: name,
|
93
94
|
consumer: consumer,
|
94
|
-
consumer_group_id: consumer_group.id
|
95
|
+
consumer_group_id: consumer_group.id,
|
96
|
+
subscription_group: subscription_group
|
95
97
|
).freeze
|
96
98
|
end
|
97
99
|
end
|
@@ -1,8 +1,6 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
<% unless rails? -%>
|
2
3
|
|
3
|
-
<% if rails? -%>
|
4
|
-
require ::File.expand_path('../config/environment', __FILE__)
|
5
|
-
<% else -%>
|
6
4
|
# This file is auto-generated during the install process.
|
7
5
|
# If by any chance you've wanted a setup for Rails app, either run the `karafka:install`
|
8
6
|
# command again or refer to the install templates available in the source codes
|
data/lib/karafka/version.rb
CHANGED
data.tar.gz.sig
CHANGED
Binary file
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: karafka
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.0.
|
4
|
+
version: 2.0.6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Maciej Mensfeld
|
@@ -10,31 +10,32 @@ bindir: bin
|
|
10
10
|
cert_chain:
|
11
11
|
- |
|
12
12
|
-----BEGIN CERTIFICATE-----
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
/
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
13
|
+
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
14
|
+
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
15
|
+
MB4XDTIyMDgxOTE3MjEzN1oXDTIzMDgxOTE3MjEzN1owPzEQMA4GA1UEAwwHY29u
|
16
|
+
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
17
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAODzeO3L6lxdATzMHKNW
|
18
|
+
jFA/GGunoPuylO/BMzy8RiQHh7VIvysAKs0tHhTx3g2D0STDpF+hcQcPELFikiT2
|
19
|
+
F+1wOHj/SsrK7VKqfA8+gq04hKc5sQoX2Egf9k3V0YJ3eZ6R/koHkQ8A0TVt0w6F
|
20
|
+
ZQckoV4MqnEAx0g/FZN3mnHTlJ3VFLSBqJEIe+S6FZMl92mSv+hTrlUG8VaYxSfN
|
21
|
+
lTCvnKk284F6QZq5XIENLRmcDd/3aPBLnLwNnyMyhB+6gK8cUO+CFlDO5tjo/aBA
|
22
|
+
rUnl++wGG0JooF1ed0v+evOn9KoMBG6rHewcf79qJbVOscbD8qSAmo+sCXtcFryr
|
23
|
+
KRMTB8gNbowJkFRJDEe8tfRy11u1fYzFg/qNO82FJd62rKAw2wN0C29yCeQOPRb1
|
24
|
+
Cw9Y4ZwK9VFNEcV9L+3pHTHn2XfuZHtDaG198VweiF6raFO4yiEYccodH/USP0L5
|
25
|
+
cbcCFtmu/4HDSxL1ByQXO84A0ybJuk3/+aPUSXe9C9U8fwIDAQABo3cwdTAJBgNV
|
26
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQUSlcEakb7gfn/5E2WY6z73BF/
|
27
|
+
iZkwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
28
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEA1aS+E7RXJ1w9g9mJ
|
29
|
+
G0NzFxe64OEuENosNlvYQCbRKGCXAU1qqelYkBQHseRgRKxLICrnypRo9IEobyHa
|
30
|
+
vDnJ4r7Tsb34dleqQW2zY/obG+cia3Ym2JsegXWF7dDOzCXJ4FN8MFoT2jHlqLLw
|
31
|
+
yrap0YO5zx0GSQ0Dwy8h2n2v2vanMEeCx7iNm3ERgR5WuN5sjzWoz2A/JLEEcK0C
|
32
|
+
EnAGKCWAd1fuG8IemDjT1edsd5FyYR4bIX0m+99oDuFZyPiiIbalmyYiSBBp59Yb
|
33
|
+
Q0P8zeBi4OfwCZNcxqz0KONmw9JLNv6DgyEAH5xe/4JzhMEgvIRiPj0pHfA7oqQF
|
34
|
+
KUNqvD1KlxbEC+bZfE5IZhnqYLdld/Ksqd22FI1RBhiS1Ejfsj99LVIm9cBuZEY2
|
35
|
+
Qf04B9ceLUaC4fPVEz10FyobjaFoY4i32xRto3XnrzeAgfEe4swLq8bQsR3w/EF3
|
36
|
+
MGU0FeSV2Yj7Xc2x/7BzLK8xQn5l7Yy75iPF+KP3vVmDHnNl
|
36
37
|
-----END CERTIFICATE-----
|
37
|
-
date: 2022-
|
38
|
+
date: 2022-09-02 00:00:00.000000000 Z
|
38
39
|
dependencies:
|
39
40
|
- !ruby/object:Gem::Dependency
|
40
41
|
name: karafka-core
|
@@ -120,7 +121,7 @@ dependencies:
|
|
120
121
|
version: '2.3'
|
121
122
|
description: Framework used to simplify Apache Kafka based Ruby applications development
|
122
123
|
email:
|
123
|
-
-
|
124
|
+
- contact@karafka.io
|
124
125
|
executables:
|
125
126
|
- karafka
|
126
127
|
extensions: []
|
@@ -152,8 +153,8 @@ files:
|
|
152
153
|
- bin/scenario
|
153
154
|
- bin/stress_many
|
154
155
|
- bin/stress_one
|
156
|
+
- certs/cert_chain.pem
|
155
157
|
- certs/karafka-pro.pem
|
156
|
-
- certs/mensfeld.pem
|
157
158
|
- config/errors.yml
|
158
159
|
- docker-compose.yml
|
159
160
|
- karafka.gemspec
|
metadata.gz.sig
CHANGED
Binary file
|
data/certs/mensfeld.pem
DELETED
@@ -1,25 +0,0 @@
|
|
1
|
-
-----BEGIN CERTIFICATE-----
|
2
|
-
MIIEODCCAqCgAwIBAgIBATANBgkqhkiG9w0BAQsFADAjMSEwHwYDVQQDDBhtYWNp
|
3
|
-
ZWovREM9bWVuc2ZlbGQvREM9cGwwHhcNMjEwODExMTQxNTEzWhcNMjIwODExMTQx
|
4
|
-
NTEzWjAjMSEwHwYDVQQDDBhtYWNpZWovREM9bWVuc2ZlbGQvREM9cGwwggGiMA0G
|
5
|
-
CSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDV2jKH4Ti87GM6nyT6D+ESzTI0MZDj
|
6
|
-
ak2/TEwnxvijMJyCCPKT/qIkbW4/f0VHM4rhPr1nW73sb5SZBVFCLlJcOSKOBdUY
|
7
|
-
TMY+SIXN2EtUaZuhAOe8LxtxjHTgRHvHcqUQMBENXTISNzCo32LnUxweu66ia4Pd
|
8
|
-
1mNRhzOqNv9YiBZvtBf7IMQ+sYdOCjboq2dlsWmJiwiDpY9lQBTnWORnT3mQxU5x
|
9
|
-
vPSwnLB854cHdCS8fQo4DjeJBRZHhEbcE5sqhEMB3RZA3EtFVEXOxlNxVTS3tncI
|
10
|
-
qyNXiWDaxcipaens4ObSY1C2HTV7OWb7OMqSCIybeYTSfkaSdqmcl4S6zxXkjH1J
|
11
|
-
tnjayAVzD+QVXGijsPLE2PFnJAh9iDET2cMsjabO1f6l1OQNyAtqpcyQcgfnyW0z
|
12
|
-
g7tGxTYD+6wJHffM9d9txOUw6djkF6bDxyqB8lo4Z3IObCx18AZjI9XPS9QG7w6q
|
13
|
-
LCWuMG2lkCcRgASqaVk9fEf9yMc2xxz5o3kCAwEAAaN3MHUwCQYDVR0TBAIwADAL
|
14
|
-
BgNVHQ8EBAMCBLAwHQYDVR0OBBYEFBqUFCKCOe5IuueUVqOB991jyCLLMB0GA1Ud
|
15
|
-
EQQWMBSBEm1hY2llakBtZW5zZmVsZC5wbDAdBgNVHRIEFjAUgRJtYWNpZWpAbWVu
|
16
|
-
c2ZlbGQucGwwDQYJKoZIhvcNAQELBQADggGBADD0/UuTTFgW+CGk2U0RDw2RBOca
|
17
|
-
W2LTF/G7AOzuzD0Tc4voc7WXyrgKwJREv8rgBimLnNlgmFJLmtUCh2U/MgxvcilH
|
18
|
-
yshYcbseNvjkrtYnLRlWZR4SSB6Zei5AlyGVQLPkvdsBpNegcG6w075YEwzX/38a
|
19
|
-
8V9B/Yri2OGELBz8ykl7BsXUgNoUPA/4pHF6YRLz+VirOaUIQ4JfY7xGj6fSOWWz
|
20
|
-
/rQ/d77r6o1mfJYM/3BRVg73a3b7DmRnE5qjwmSaSQ7u802pJnLesmArch0xGCT/
|
21
|
-
fMmRli1Qb+6qOTl9mzD6UDMAyFR4t6MStLm0mIEqM0nBO5nUdUWbC7l9qXEf8XBE
|
22
|
-
2DP28p3EqSuS+lKbAWKcqv7t0iRhhmaod+Yn9mcrLN1sa3q3KSQ9BCyxezCD4Mk2
|
23
|
-
R2P11bWoCtr70BsccVrN8jEhzwXngMyI2gVt750Y+dbTu1KgRqZKp/ECe7ZzPzXj
|
24
|
-
pIy9vHxTANKYVyI4qj8OrFdEM5BQNu8oQpL0iQ==
|
25
|
-
-----END CERTIFICATE-----
|