karafka 2.0.0.alpha1 → 2.0.0.alpha4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/.github/workflows/ci.yml +3 -1
- data/CHANGELOG.md +14 -1
- data/CONTRIBUTING.md +6 -6
- data/Gemfile.lock +24 -24
- data/LICENSE +3 -0
- data/bin/integrations +44 -8
- data/bin/karafka +4 -0
- data/bin/stress +1 -1
- data/config/errors.yml +1 -0
- data/docker-compose.yml +1 -0
- data/karafka.gemspec +1 -1
- data/lib/active_job/karafka.rb +16 -13
- data/lib/active_job/queue_adapters/karafka_adapter.rb +3 -6
- data/lib/karafka/active_job/consumer.rb +24 -0
- data/lib/karafka/active_job/dispatcher.rb +38 -0
- data/lib/karafka/active_job/job_extensions.rb +34 -0
- data/lib/karafka/active_job/job_options_contract.rb +15 -0
- data/lib/karafka/active_job/routing_extensions.rb +18 -0
- data/lib/karafka/app.rb +1 -0
- data/lib/karafka/cli/info.rb +3 -3
- data/lib/karafka/cli/install.rb +1 -0
- data/lib/karafka/cli/server.rb +2 -16
- data/lib/karafka/contracts/base.rb +23 -0
- data/lib/karafka/contracts/config.rb +21 -3
- data/lib/karafka/contracts/consumer_group.rb +1 -3
- data/lib/karafka/contracts/consumer_group_topic.rb +2 -3
- data/lib/karafka/contracts/server_cli_options.rb +1 -3
- data/lib/karafka/errors.rb +4 -0
- data/lib/karafka/instrumentation/monitor.rb +1 -0
- data/lib/karafka/instrumentation/stdout_listener.rb +3 -0
- data/lib/karafka/licenser.rb +20 -9
- data/lib/karafka/messages/batch_metadata.rb +2 -0
- data/lib/karafka/messages/builders/batch_metadata.rb +23 -1
- data/lib/karafka/pro/active_job/dispatcher.rb +58 -0
- data/lib/karafka/pro/active_job/job_options_contract.rb +27 -0
- data/lib/karafka/pro/loader.rb +29 -0
- data/lib/karafka/pro.rb +13 -0
- data/lib/karafka/processing/worker.rb +1 -1
- data/lib/karafka/railtie.rb +55 -19
- data/lib/karafka/routing/builder.rb +1 -11
- data/lib/karafka/routing/subscription_group.rb +5 -5
- data/lib/karafka/routing/subscription_groups_builder.rb +1 -0
- data/lib/karafka/routing/topic.rb +1 -0
- data/lib/karafka/setup/config.rb +25 -20
- data/lib/karafka/status.rb +1 -0
- data/lib/karafka/templates/karafka.rb.erb +1 -1
- data/lib/karafka/version.rb +1 -1
- data/lib/karafka.rb +7 -2
- data.tar.gz.sig +0 -0
- metadata +14 -7
- metadata.gz.sig +0 -0
- data/.github/FUNDING.yml +0 -3
- data/lib/active_job/consumer.rb +0 -22
- data/lib/active_job/routing_extensions.rb +0 -15
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f8ed5005d3288abb4f9f6389fa126b5434b5a7a6df729e924695d38624e167e5
|
4
|
+
data.tar.gz: aa802a441d9c9cb1275260ba9b290c8c4c72acf125ca72efaafc58538a6c8b20
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1ad458acdb42c28d04895db1aa348d0ab10177c0829e7b19fc9fd8b23aaac5fc17feac5e1366cd6444d1e60903b087be1b24298b7764a98f8f372a325afb7cee
|
7
|
+
data.tar.gz: fb1de540a2c50d26467585bb720a636726955e65f1cea1ca055ef4a9f10d66db233928ca8b067d225d6c935f779d50f2381660ddcaa0d86d6c5b0b8a8ed991e1
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/.github/workflows/ci.yml
CHANGED
@@ -113,6 +113,7 @@ jobs:
|
|
113
113
|
- name: Install latest Bundler
|
114
114
|
run: |
|
115
115
|
gem install bundler --no-document
|
116
|
+
gem update --system --no-document
|
116
117
|
bundle config set without 'tools benchmarks docs'
|
117
118
|
|
118
119
|
- name: Bundle install
|
@@ -122,5 +123,6 @@ jobs:
|
|
122
123
|
|
123
124
|
- name: Run integration tests
|
124
125
|
env:
|
126
|
+
KARAFKA_PRO_LICENSE_TOKEN: ${{ secrets.KARAFKA_PRO_LICENSE_TOKEN }}
|
125
127
|
GITHUB_COVERAGE: ${{matrix.coverage}}
|
126
|
-
run:
|
128
|
+
run: bin/integrations
|
data/CHANGELOG.md
CHANGED
@@ -1,6 +1,19 @@
|
|
1
1
|
# Karafka framework changelog
|
2
2
|
|
3
|
-
## 2.0.0-
|
3
|
+
## 2.0.0-alpha4 (Unreleased)
|
4
|
+
- Rails support without ActiveJob queue adapter usage (#805)
|
5
|
+
|
6
|
+
## 2.0.0-alpha3 (2022-03-16)
|
7
|
+
- Restore 'app.initialized' state and add notification on it
|
8
|
+
- Fix the installation flow for Rails and add integration tests for this scenario
|
9
|
+
- Add more integration tests covering some edge cases
|
10
|
+
|
11
|
+
## 2.0.0-alpha2 (2022-02-19)
|
12
|
+
- Require `kafka` keys to be symbols
|
13
|
+
- Added ActiveJob Pro adapter
|
14
|
+
- Small updates to the license and docs
|
15
|
+
|
16
|
+
## 2.0.0-alpha1 (2022-01-30)
|
4
17
|
- Change license to `LGPL-3.0`
|
5
18
|
- Introduce a Pro subscription
|
6
19
|
- Switch from `ruby-kafka` to `librdkafka` as an underlying driver
|
data/CONTRIBUTING.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
## Introduction
|
4
4
|
|
5
|
-
First, thank you for considering contributing to
|
5
|
+
First, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community! 😊
|
6
6
|
|
7
7
|
We welcome any type of contribution, not only code. You can help with:
|
8
8
|
- **QA**: file bug reports, the more details you can give the better (e.g. screenshots with the console open)
|
@@ -18,15 +18,17 @@ Working on your first Pull Request? You can learn how from this *free* series, [
|
|
18
18
|
|
19
19
|
Any code change should be submitted as a pull request. The description should explain what the code does and give steps to execute it. The pull request should also contain tests.
|
20
20
|
|
21
|
-
|
21
|
+
### Code review process
|
22
22
|
|
23
23
|
Each pull request must pass all the rspec specs and meet our quality requirements.
|
24
24
|
|
25
25
|
To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
|
26
26
|
|
27
|
-
|
27
|
+
### Contributing to Pro components
|
28
28
|
|
29
|
-
|
29
|
+
All of Karafka components are open-source. However, the `Pro` components are licenses under `LICENSE-COMM`.
|
30
|
+
|
31
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your code to Maciej Mensfeld.
|
30
32
|
|
31
33
|
## Questions
|
32
34
|
|
@@ -35,7 +37,5 @@ You can also reach us at hello@karafka.opencollective.com.
|
|
35
37
|
|
36
38
|
## Credits
|
37
39
|
|
38
|
-
### Contributors
|
39
|
-
|
40
40
|
Thank you to all the people who have already contributed to karafka!
|
41
41
|
<a href="graphs/contributors"><img src="https://opencollective.com/karafka/contributors.svg?width=890" /></a>
|
data/Gemfile.lock
CHANGED
@@ -1,22 +1,22 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
karafka (2.0.0.
|
4
|
+
karafka (2.0.0.alpha4)
|
5
5
|
dry-configurable (~> 0.13)
|
6
6
|
dry-monitor (~> 0.5)
|
7
7
|
dry-validation (~> 1.7)
|
8
8
|
rdkafka (>= 0.10)
|
9
9
|
thor (>= 0.20)
|
10
|
-
waterdrop (>= 2.
|
10
|
+
waterdrop (>= 2.2.0, < 3.0.0)
|
11
11
|
zeitwerk (~> 2.3)
|
12
12
|
|
13
13
|
GEM
|
14
14
|
remote: https://rubygems.org/
|
15
15
|
specs:
|
16
|
-
activejob (7.0.
|
17
|
-
activesupport (= 7.0.
|
16
|
+
activejob (7.0.2.2)
|
17
|
+
activesupport (= 7.0.2.2)
|
18
18
|
globalid (>= 0.3.6)
|
19
|
-
activesupport (7.0.
|
19
|
+
activesupport (7.0.2.2)
|
20
20
|
concurrent-ruby (~> 1.0, >= 1.0.2)
|
21
21
|
i18n (>= 1.6, < 2)
|
22
22
|
minitest (>= 5.1)
|
@@ -45,7 +45,7 @@ GEM
|
|
45
45
|
dry-configurable (~> 0.13, >= 0.13.0)
|
46
46
|
dry-core (~> 0.5, >= 0.5)
|
47
47
|
dry-events (~> 0.2)
|
48
|
-
dry-schema (1.
|
48
|
+
dry-schema (1.9.1)
|
49
49
|
concurrent-ruby (~> 1.0)
|
50
50
|
dry-configurable (~> 0.13, >= 0.13.0)
|
51
51
|
dry-core (~> 0.5, >= 0.5)
|
@@ -58,49 +58,49 @@ GEM
|
|
58
58
|
dry-core (~> 0.5, >= 0.5)
|
59
59
|
dry-inflector (~> 0.1, >= 0.1.2)
|
60
60
|
dry-logic (~> 1.0, >= 1.0.2)
|
61
|
-
dry-validation (1.
|
61
|
+
dry-validation (1.8.0)
|
62
62
|
concurrent-ruby (~> 1.0)
|
63
63
|
dry-container (~> 0.7, >= 0.7.1)
|
64
64
|
dry-core (~> 0.5, >= 0.5)
|
65
65
|
dry-initializer (~> 3.0)
|
66
|
-
dry-schema (~> 1.
|
66
|
+
dry-schema (~> 1.9, >= 1.9.1)
|
67
67
|
factory_bot (6.2.0)
|
68
68
|
activesupport (>= 5.0.0)
|
69
69
|
ffi (1.15.5)
|
70
70
|
globalid (1.0.0)
|
71
71
|
activesupport (>= 5.0)
|
72
|
-
i18n (1.
|
72
|
+
i18n (1.10.0)
|
73
73
|
concurrent-ruby (~> 1.0)
|
74
|
-
mini_portile2 (2.
|
74
|
+
mini_portile2 (2.8.0)
|
75
75
|
minitest (5.15.0)
|
76
76
|
rake (13.0.6)
|
77
77
|
rdkafka (0.11.1)
|
78
78
|
ffi (~> 1.15)
|
79
79
|
mini_portile2 (~> 2.6)
|
80
80
|
rake (> 12)
|
81
|
-
rspec (3.
|
82
|
-
rspec-core (~> 3.
|
83
|
-
rspec-expectations (~> 3.
|
84
|
-
rspec-mocks (~> 3.
|
85
|
-
rspec-core (3.
|
86
|
-
rspec-support (~> 3.
|
87
|
-
rspec-expectations (3.
|
81
|
+
rspec (3.11.0)
|
82
|
+
rspec-core (~> 3.11.0)
|
83
|
+
rspec-expectations (~> 3.11.0)
|
84
|
+
rspec-mocks (~> 3.11.0)
|
85
|
+
rspec-core (3.11.0)
|
86
|
+
rspec-support (~> 3.11.0)
|
87
|
+
rspec-expectations (3.11.0)
|
88
88
|
diff-lcs (>= 1.2.0, < 2.0)
|
89
|
-
rspec-support (~> 3.
|
90
|
-
rspec-mocks (3.
|
89
|
+
rspec-support (~> 3.11.0)
|
90
|
+
rspec-mocks (3.11.0)
|
91
91
|
diff-lcs (>= 1.2.0, < 2.0)
|
92
|
-
rspec-support (~> 3.
|
93
|
-
rspec-support (3.
|
92
|
+
rspec-support (~> 3.11.0)
|
93
|
+
rspec-support (3.11.0)
|
94
94
|
simplecov (0.21.2)
|
95
95
|
docile (~> 1.1)
|
96
96
|
simplecov-html (~> 0.11)
|
97
97
|
simplecov_json_formatter (~> 0.1)
|
98
98
|
simplecov-html (0.12.3)
|
99
|
-
simplecov_json_formatter (0.1.
|
99
|
+
simplecov_json_formatter (0.1.4)
|
100
100
|
thor (1.2.1)
|
101
101
|
tzinfo (2.0.4)
|
102
102
|
concurrent-ruby (~> 1.0)
|
103
|
-
waterdrop (2.
|
103
|
+
waterdrop (2.2.0)
|
104
104
|
concurrent-ruby (>= 1.1)
|
105
105
|
dry-configurable (~> 0.13)
|
106
106
|
dry-monitor (~> 0.5)
|
@@ -121,4 +121,4 @@ DEPENDENCIES
|
|
121
121
|
simplecov
|
122
122
|
|
123
123
|
BUNDLED WITH
|
124
|
-
2.3.
|
124
|
+
2.3.7
|
data/LICENSE
CHANGED
@@ -9,6 +9,9 @@ Karafka has also commercial-friendly license, commercial support and commercial
|
|
9
9
|
All of the commercial components are present in the lib/karafka/pro directory of this repository
|
10
10
|
and their usage requires commercial license agreement.
|
11
11
|
|
12
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
|
13
|
+
code to Maciej Mensfeld.
|
14
|
+
|
12
15
|
You can find the commercial license in LICENSE-COM.
|
13
16
|
|
14
17
|
Please see https://karafka.io for purchasing options.
|
data/bin/integrations
CHANGED
@@ -2,9 +2,15 @@
|
|
2
2
|
|
3
3
|
# Runner to run integration specs in parallel
|
4
4
|
|
5
|
+
# Part of integration specs run pristine without bundler.
|
6
|
+
# If we would run bundle exec when running this code, bundler would inject its own context
|
7
|
+
# into them, messing things up heavily
|
8
|
+
raise 'This code needs to be executed WITHOUT bundle exec' if Kernel.const_defined?(:Bundler)
|
9
|
+
|
5
10
|
require 'open3'
|
6
11
|
require 'fileutils'
|
7
12
|
require 'pathname'
|
13
|
+
require 'tmpdir'
|
8
14
|
|
9
15
|
ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
|
10
16
|
|
@@ -13,13 +19,13 @@ IntegrationTestError = Class.new(StandardError)
|
|
13
19
|
|
14
20
|
# How many child processes with integration specs do we want to run in parallel
|
15
21
|
# When the value is high, there's a problem with thread allocation on Github
|
16
|
-
CONCURRENCY =
|
22
|
+
CONCURRENCY = 4
|
17
23
|
|
18
24
|
# Abstraction around a single test scenario execution process
|
19
25
|
class Scenario
|
20
26
|
# How long a scenario can run before we kill it
|
21
27
|
# This is a fail-safe just in case something would hang
|
22
|
-
MAX_RUN_TIME = 60 *
|
28
|
+
MAX_RUN_TIME = 60 * 2
|
23
29
|
|
24
30
|
# There are rare cases where Karafka may force shutdown for some of the integration cases
|
25
31
|
# This includes exactly those
|
@@ -38,9 +44,7 @@ class Scenario
|
|
38
44
|
# @param path [String] path to the scenarios file
|
39
45
|
def initialize(path)
|
40
46
|
@path = path
|
41
|
-
@stdin, @stdout, @stderr, @wait_thr = Open3.popen3(
|
42
|
-
"bundle exec ruby -r ./spec/integrations_helper.rb #{path}"
|
43
|
-
)
|
47
|
+
@stdin, @stdout, @stderr, @wait_thr = Open3.popen3(init_and_build_cmd)
|
44
48
|
@started_at = current_time
|
45
49
|
# Last 1024 characters from stdout
|
46
50
|
@stdout_tail = ''
|
@@ -102,6 +106,35 @@ class Scenario
|
|
102
106
|
|
103
107
|
private
|
104
108
|
|
109
|
+
# Sets up a proper environment for a given spec to run and returns the run command
|
110
|
+
# @return [String] run command
|
111
|
+
def init_and_build_cmd
|
112
|
+
scenario_dir = File.dirname(@path)
|
113
|
+
|
114
|
+
# If there is a Gemfile in a scenario directory, it means it is a pristine spec and we need
|
115
|
+
# to run bundle install, etc in order to run it
|
116
|
+
if File.exist?(File.join(scenario_dir, 'Gemfile'))
|
117
|
+
# We copy the spec into a temp dir, not to pollute the spec location with logs, etc
|
118
|
+
temp_dir = Dir.mktmpdir
|
119
|
+
file_name = File.basename(@path)
|
120
|
+
|
121
|
+
FileUtils.cp_r("#{scenario_dir}/.", temp_dir)
|
122
|
+
|
123
|
+
<<~CMD
|
124
|
+
cd #{temp_dir} &&
|
125
|
+
KARAFKA_GEM_DIR=#{ROOT_PATH} \
|
126
|
+
BUNDLE_AUTO_INSTALL=true \
|
127
|
+
PRISTINE_MODE=true \
|
128
|
+
bundle exec ruby -r #{ROOT_PATH}/spec/integrations_helper.rb #{file_name}
|
129
|
+
CMD
|
130
|
+
else
|
131
|
+
<<~CMD
|
132
|
+
KARAFKA_GEM_DIR=#{ROOT_PATH} \
|
133
|
+
bundle exec ruby -r ./spec/integrations_helper.rb #{@path}
|
134
|
+
CMD
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
105
138
|
# @return [Float] current machine time
|
106
139
|
def current_time
|
107
140
|
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
@@ -136,10 +169,13 @@ end
|
|
136
169
|
# Load all the specs
|
137
170
|
specs = Dir[ROOT_PATH.join('spec/integrations/**/*.rb')]
|
138
171
|
|
139
|
-
# If
|
140
|
-
|
172
|
+
# If filters is provided, apply
|
173
|
+
# Allows to provide several filters one after another and applies all of them
|
174
|
+
ARGV.each do |filter|
|
175
|
+
specs.delete_if { |name| !name.include?(filter) }
|
176
|
+
end
|
141
177
|
|
142
|
-
raise ArgumentError, "No integration specs with
|
178
|
+
raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?
|
143
179
|
|
144
180
|
# Randomize order
|
145
181
|
seed = (ENV['SEED'] || rand(0..10_000)).to_i
|
data/bin/karafka
CHANGED
@@ -2,6 +2,10 @@
|
|
2
2
|
|
3
3
|
require 'karafka'
|
4
4
|
|
5
|
+
# We set this to indicate, that the process in which we are (whatever it does) was started using
|
6
|
+
# our bin/karafka cli
|
7
|
+
ENV['KARAFKA_CLI'] = 'true'
|
8
|
+
|
5
9
|
# If there is a boot file, we need to require it as we expect it to contain
|
6
10
|
# Karafka app setup, routes, etc
|
7
11
|
if File.exist?(Karafka.boot_file)
|
data/bin/stress
CHANGED
data/config/errors.yml
CHANGED
@@ -5,3 +5,4 @@ en:
|
|
5
5
|
topics_names_not_unique: all topic names within a single consumer group must be unique
|
6
6
|
required_usage_count: Given topic must be used at least once
|
7
7
|
consumer_groups_inclusion: Unknown consumer group
|
8
|
+
kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
|
data/docker-compose.yml
CHANGED
data/karafka.gemspec
CHANGED
@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
|
|
21
21
|
spec.add_dependency 'dry-validation', '~> 1.7'
|
22
22
|
spec.add_dependency 'rdkafka', '>= 0.10'
|
23
23
|
spec.add_dependency 'thor', '>= 0.20'
|
24
|
-
spec.add_dependency 'waterdrop', '>= 2.
|
24
|
+
spec.add_dependency 'waterdrop', '>= 2.2.0', '< 3.0.0'
|
25
25
|
spec.add_dependency 'zeitwerk', '~> 2.3'
|
26
26
|
|
27
27
|
spec.required_ruby_version = '>= 2.6.0'
|
data/lib/active_job/karafka.rb
CHANGED
@@ -1,18 +1,21 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
|
4
|
-
require 'active_job
|
5
|
-
|
6
|
-
require 'active_job/routing_extensions'
|
7
|
-
require 'active_job/queue_adapters/karafka_adapter'
|
3
|
+
begin
|
4
|
+
require 'active_job'
|
5
|
+
require_relative 'queue_adapters/karafka_adapter'
|
8
6
|
|
9
|
-
module ActiveJob
|
10
|
-
|
11
|
-
|
12
|
-
|
7
|
+
module ActiveJob
|
8
|
+
# Namespace for usage simplification outside of Rails where Railtie will not kick in.
|
9
|
+
# That way a require 'active_job/karafka' should be enough to use it
|
10
|
+
module Karafka
|
11
|
+
end
|
13
12
|
end
|
14
|
-
end
|
15
13
|
|
16
|
-
# We extend routing builder by adding a simple wrapper for easier jobs topics defining
|
17
|
-
|
18
|
-
|
14
|
+
# We extend routing builder by adding a simple wrapper for easier jobs topics defining
|
15
|
+
# This needs to be extended here as it is going to be used in karafka routes, hence doing that in
|
16
|
+
# the railtie initializer would be too late
|
17
|
+
::Karafka::Routing::Builder.include ::Karafka::ActiveJob::RoutingExtensions
|
18
|
+
::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::RoutingExtensions
|
19
|
+
rescue LoadError
|
20
|
+
# We extend ActiveJob stuff in the railtie
|
21
|
+
end
|
@@ -5,16 +5,13 @@ module ActiveJob
|
|
5
5
|
# ActiveJob queue adapters
|
6
6
|
module QueueAdapters
|
7
7
|
# Karafka adapter for enqueuing jobs
|
8
|
+
# This is here for ease of integration with ActiveJob.
|
8
9
|
class KarafkaAdapter
|
9
|
-
# Enqueues the job
|
10
|
-
# later on consumed by a special ActiveJob consumer
|
10
|
+
# Enqueues the job using the configured dispatcher
|
11
11
|
#
|
12
12
|
# @param job [Object] job that should be enqueued
|
13
13
|
def enqueue(job)
|
14
|
-
::Karafka.
|
15
|
-
topic: job.queue_name,
|
16
|
-
payload: ActiveSupport::JSON.encode(job.serialize)
|
17
|
-
)
|
14
|
+
::Karafka::App.config.internal.active_job.dispatcher.call(job)
|
18
15
|
end
|
19
16
|
|
20
17
|
# Raises info, that Karafka backend does not support scheduling jobs
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# This is the consumer for ActiveJob that eats the messages enqueued with it one after another.
|
6
|
+
# It marks the offset after each message, so we make sure, none of the jobs is executed twice
|
7
|
+
class Consumer < ::Karafka::BaseConsumer
|
8
|
+
# Executes the ActiveJob logic
|
9
|
+
# @note ActiveJob does not support batches, so we just run one message after another
|
10
|
+
def consume
|
11
|
+
messages.each do |message|
|
12
|
+
::ActiveJob::Base.execute(
|
13
|
+
# We technically speaking could set this as deserializer and reference it from the
|
14
|
+
# message instead of using the `#raw_payload`. This is not done on purpose to simplify
|
15
|
+
# the ActiveJob setup here
|
16
|
+
::ActiveSupport::JSON.decode(message.raw_payload)
|
17
|
+
)
|
18
|
+
|
19
|
+
mark_as_consumed(message)
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# Dispatcher that sends the ActiveJob job to a proper topic based on the queue name
|
6
|
+
class Dispatcher
|
7
|
+
# Defaults for dispatching
|
8
|
+
# The can be updated by using `#karafka_options` on the job
|
9
|
+
DEFAULTS = {
|
10
|
+
dispatch_method: :produce_async
|
11
|
+
}.freeze
|
12
|
+
|
13
|
+
private_constant :DEFAULTS
|
14
|
+
|
15
|
+
# @param job [ActiveJob::Base] job
|
16
|
+
def call(job)
|
17
|
+
::Karafka.producer.public_send(
|
18
|
+
fetch_option(job, :dispatch_method, DEFAULTS),
|
19
|
+
topic: job.queue_name,
|
20
|
+
payload: ::ActiveSupport::JSON.encode(job.serialize)
|
21
|
+
)
|
22
|
+
end
|
23
|
+
|
24
|
+
private
|
25
|
+
|
26
|
+
# @param job [ActiveJob::Base] job
|
27
|
+
# @param key [Symbol] key we want to fetch
|
28
|
+
# @param defaults [Hash]
|
29
|
+
# @return [Object] options we are interested in
|
30
|
+
def fetch_option(job, key, defaults)
|
31
|
+
job
|
32
|
+
.class
|
33
|
+
.karafka_options
|
34
|
+
.fetch(key, defaults.fetch(key))
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# Allows for setting karafka specific options in ActiveJob jobs
|
6
|
+
module JobExtensions
|
7
|
+
class << self
|
8
|
+
# Defines all the needed accessors and sets defaults
|
9
|
+
# @param klass [ActiveJob::Base] active job base
|
10
|
+
def extended(klass)
|
11
|
+
klass.class_attribute :_karafka_options
|
12
|
+
klass._karafka_options = {}
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
# @param new_options [Hash] additional options that allow for jobs Karafka related options
|
17
|
+
# customization
|
18
|
+
# @return [Hash] karafka options
|
19
|
+
def karafka_options(new_options = {})
|
20
|
+
return _karafka_options if new_options.empty?
|
21
|
+
|
22
|
+
# Make sure, that karafka options that someone wants to use are valid before assigning
|
23
|
+
# them
|
24
|
+
App.config.internal.active_job.job_options_contract.validate!(new_options)
|
25
|
+
|
26
|
+
new_options.each do |name, value|
|
27
|
+
_karafka_options[name] = value
|
28
|
+
end
|
29
|
+
|
30
|
+
_karafka_options
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
@@ -0,0 +1,15 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# Contract for validating the options that can be altered with `#karafka_options` per job class
|
6
|
+
# @note We keep this in the `Karafka::ActiveJob` namespace instead of `Karafka::Contracts` as
|
7
|
+
# we want to keep ActiveJob related Karafka components outside of the core Karafka code and
|
8
|
+
# all in the same place
|
9
|
+
class JobOptionsContract < Contracts::Base
|
10
|
+
params do
|
11
|
+
optional(:dispatch_method).value(included_in?: %i[produce_async produce_sync])
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
# ActiveJob related Karafka stuff
|
5
|
+
module ActiveJob
|
6
|
+
# Routing extensions for ActiveJob
|
7
|
+
module RoutingExtensions
|
8
|
+
# This method simplifies routes definition for ActiveJob topics / queues by auto-injecting
|
9
|
+
# the consumer class
|
10
|
+
# @param name [String, Symbol] name of the topic where ActiveJobs jobs should go
|
11
|
+
def active_job_topic(name)
|
12
|
+
topic(name) do
|
13
|
+
consumer App.config.internal.active_job.consumer
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
data/lib/karafka/app.rb
CHANGED
data/lib/karafka/cli/info.rb
CHANGED
@@ -31,9 +31,9 @@ module Karafka
|
|
31
31
|
def core_info
|
32
32
|
config = Karafka::App.config
|
33
33
|
|
34
|
-
postfix =
|
34
|
+
postfix = Karafka.pro? ? ' + Pro' : ''
|
35
35
|
|
36
|
-
|
36
|
+
[
|
37
37
|
"Karafka version: #{Karafka::VERSION}#{postfix}",
|
38
38
|
"Ruby version: #{RUBY_VERSION}",
|
39
39
|
"Rdkafka version: #{::Rdkafka::VERSION}",
|
@@ -49,7 +49,7 @@ module Karafka
|
|
49
49
|
def license_info
|
50
50
|
config = Karafka::App.config
|
51
51
|
|
52
|
-
if
|
52
|
+
if Karafka.pro?
|
53
53
|
[
|
54
54
|
'License: Commercial',
|
55
55
|
"License entity: #{config.license.entity}",
|
data/lib/karafka/cli/install.rb
CHANGED
data/lib/karafka/cli/server.rb
CHANGED
@@ -5,11 +5,6 @@ module Karafka
|
|
5
5
|
class Cli < Thor
|
6
6
|
# Server Karafka Cli action
|
7
7
|
class Server < Base
|
8
|
-
# Server config settings contract
|
9
|
-
CONTRACT = Contracts::ServerCliOptions.new.freeze
|
10
|
-
|
11
|
-
private_constant :CONTRACT
|
12
|
-
|
13
8
|
desc 'Start the Karafka server (short-cut alias: "s")'
|
14
9
|
option aliases: 's'
|
15
10
|
option :consumer_groups, type: :array, default: nil, aliases: :g
|
@@ -19,7 +14,7 @@ module Karafka
|
|
19
14
|
# Print our banner and info in the dev mode
|
20
15
|
print_marketing_info if Karafka::App.env.development?
|
21
16
|
|
22
|
-
validate!
|
17
|
+
Contracts::ServerCliOptions.new.validate!(cli.options)
|
23
18
|
|
24
19
|
# We assign active topics on a server level, as only server is expected to listen on
|
25
20
|
# part of the topics
|
@@ -34,7 +29,7 @@ module Karafka
|
|
34
29
|
def print_marketing_info
|
35
30
|
Karafka.logger.info Info::BANNER
|
36
31
|
|
37
|
-
if Karafka
|
32
|
+
if Karafka.pro?
|
38
33
|
Karafka.logger.info(
|
39
34
|
"\033[0;32mThank you for investing in the Karafka Pro subscription!\033[0m\n"
|
40
35
|
)
|
@@ -44,15 +39,6 @@ module Karafka
|
|
44
39
|
)
|
45
40
|
end
|
46
41
|
end
|
47
|
-
|
48
|
-
# Checks the server cli configuration
|
49
|
-
# options validations in terms of app setup (topics, pid existence, etc)
|
50
|
-
def validate!
|
51
|
-
result = CONTRACT.call(cli.options)
|
52
|
-
return if result.success?
|
53
|
-
|
54
|
-
raise Errors::InvalidConfigurationError, result.errors.to_h
|
55
|
-
end
|
56
42
|
end
|
57
43
|
end
|
58
44
|
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module Contracts
|
5
|
+
# Base contract for all Karafka contracts
|
6
|
+
class Base < Dry::Validation::Contract
|
7
|
+
config.messages.load_paths << File.join(Karafka.gem_root, 'config', 'errors.yml')
|
8
|
+
|
9
|
+
# @param data [Hash] data for validation
|
10
|
+
# @return [Boolean] true if all good
|
11
|
+
# @raise [Errors::InvalidConfigurationError] invalid configuration error
|
12
|
+
# @note We use contracts only in the config validation context, so no need to add support
|
13
|
+
# for multiple error classes. It will be added when it will be needed.
|
14
|
+
def validate!(data)
|
15
|
+
result = call(data)
|
16
|
+
|
17
|
+
return true if result.success?
|
18
|
+
|
19
|
+
raise Errors::InvalidConfigurationError, result.errors.to_h
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|