karafka 2.0.0.alpha1 → 2.0.0.alpha4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/.github/workflows/ci.yml +3 -1
- data/CHANGELOG.md +14 -1
- data/CONTRIBUTING.md +6 -6
- data/Gemfile.lock +24 -24
- data/LICENSE +3 -0
- data/bin/integrations +44 -8
- data/bin/karafka +4 -0
- data/bin/stress +1 -1
- data/config/errors.yml +1 -0
- data/docker-compose.yml +1 -0
- data/karafka.gemspec +1 -1
- data/lib/active_job/karafka.rb +16 -13
- data/lib/active_job/queue_adapters/karafka_adapter.rb +3 -6
- data/lib/karafka/active_job/consumer.rb +24 -0
- data/lib/karafka/active_job/dispatcher.rb +38 -0
- data/lib/karafka/active_job/job_extensions.rb +34 -0
- data/lib/karafka/active_job/job_options_contract.rb +15 -0
- data/lib/karafka/active_job/routing_extensions.rb +18 -0
- data/lib/karafka/app.rb +1 -0
- data/lib/karafka/cli/info.rb +3 -3
- data/lib/karafka/cli/install.rb +1 -0
- data/lib/karafka/cli/server.rb +2 -16
- data/lib/karafka/contracts/base.rb +23 -0
- data/lib/karafka/contracts/config.rb +21 -3
- data/lib/karafka/contracts/consumer_group.rb +1 -3
- data/lib/karafka/contracts/consumer_group_topic.rb +2 -3
- data/lib/karafka/contracts/server_cli_options.rb +1 -3
- data/lib/karafka/errors.rb +4 -0
- data/lib/karafka/instrumentation/monitor.rb +1 -0
- data/lib/karafka/instrumentation/stdout_listener.rb +3 -0
- data/lib/karafka/licenser.rb +20 -9
- data/lib/karafka/messages/batch_metadata.rb +2 -0
- data/lib/karafka/messages/builders/batch_metadata.rb +23 -1
- data/lib/karafka/pro/active_job/dispatcher.rb +58 -0
- data/lib/karafka/pro/active_job/job_options_contract.rb +27 -0
- data/lib/karafka/pro/loader.rb +29 -0
- data/lib/karafka/pro.rb +13 -0
- data/lib/karafka/processing/worker.rb +1 -1
- data/lib/karafka/railtie.rb +55 -19
- data/lib/karafka/routing/builder.rb +1 -11
- data/lib/karafka/routing/subscription_group.rb +5 -5
- data/lib/karafka/routing/subscription_groups_builder.rb +1 -0
- data/lib/karafka/routing/topic.rb +1 -0
- data/lib/karafka/setup/config.rb +25 -20
- data/lib/karafka/status.rb +1 -0
- data/lib/karafka/templates/karafka.rb.erb +1 -1
- data/lib/karafka/version.rb +1 -1
- data/lib/karafka.rb +7 -2
- data.tar.gz.sig +0 -0
- metadata +14 -7
- metadata.gz.sig +0 -0
- data/.github/FUNDING.yml +0 -3
- data/lib/active_job/consumer.rb +0 -22
- data/lib/active_job/routing_extensions.rb +0 -15
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f8ed5005d3288abb4f9f6389fa126b5434b5a7a6df729e924695d38624e167e5
|
4
|
+
data.tar.gz: aa802a441d9c9cb1275260ba9b290c8c4c72acf125ca72efaafc58538a6c8b20
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1ad458acdb42c28d04895db1aa348d0ab10177c0829e7b19fc9fd8b23aaac5fc17feac5e1366cd6444d1e60903b087be1b24298b7764a98f8f372a325afb7cee
|
7
|
+
data.tar.gz: fb1de540a2c50d26467585bb720a636726955e65f1cea1ca055ef4a9f10d66db233928ca8b067d225d6c935f779d50f2381660ddcaa0d86d6c5b0b8a8ed991e1
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/.github/workflows/ci.yml
CHANGED
@@ -113,6 +113,7 @@ jobs:
|
|
113
113
|
- name: Install latest Bundler
|
114
114
|
run: |
|
115
115
|
gem install bundler --no-document
|
116
|
+
gem update --system --no-document
|
116
117
|
bundle config set without 'tools benchmarks docs'
|
117
118
|
|
118
119
|
- name: Bundle install
|
@@ -122,5 +123,6 @@ jobs:
|
|
122
123
|
|
123
124
|
- name: Run integration tests
|
124
125
|
env:
|
126
|
+
KARAFKA_PRO_LICENSE_TOKEN: ${{ secrets.KARAFKA_PRO_LICENSE_TOKEN }}
|
125
127
|
GITHUB_COVERAGE: ${{matrix.coverage}}
|
126
|
-
run:
|
128
|
+
run: bin/integrations
|
data/CHANGELOG.md
CHANGED
@@ -1,6 +1,19 @@
|
|
1
1
|
# Karafka framework changelog
|
2
2
|
|
3
|
-
## 2.0.0-
|
3
|
+
## 2.0.0-alpha4 (Unreleased)
|
4
|
+
- Rails support without ActiveJob queue adapter usage (#805)
|
5
|
+
|
6
|
+
## 2.0.0-alpha3 (2022-03-16)
|
7
|
+
- Restore 'app.initialized' state and add notification on it
|
8
|
+
- Fix the installation flow for Rails and add integration tests for this scenario
|
9
|
+
- Add more integration tests covering some edge cases
|
10
|
+
|
11
|
+
## 2.0.0-alpha2 (2022-02-19)
|
12
|
+
- Require `kafka` keys to be symbols
|
13
|
+
- Added ActiveJob Pro adapter
|
14
|
+
- Small updates to the license and docs
|
15
|
+
|
16
|
+
## 2.0.0-alpha1 (2022-01-30)
|
4
17
|
- Change license to `LGPL-3.0`
|
5
18
|
- Introduce a Pro subscription
|
6
19
|
- Switch from `ruby-kafka` to `librdkafka` as an underlying driver
|
data/CONTRIBUTING.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
## Introduction
|
4
4
|
|
5
|
-
First, thank you for considering contributing to
|
5
|
+
First, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community! 😊
|
6
6
|
|
7
7
|
We welcome any type of contribution, not only code. You can help with:
|
8
8
|
- **QA**: file bug reports, the more details you can give the better (e.g. screenshots with the console open)
|
@@ -18,15 +18,17 @@ Working on your first Pull Request? You can learn how from this *free* series, [
|
|
18
18
|
|
19
19
|
Any code change should be submitted as a pull request. The description should explain what the code does and give steps to execute it. The pull request should also contain tests.
|
20
20
|
|
21
|
-
|
21
|
+
### Code review process
|
22
22
|
|
23
23
|
Each pull request must pass all the rspec specs and meet our quality requirements.
|
24
24
|
|
25
25
|
To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
|
26
26
|
|
27
|
-
|
27
|
+
### Contributing to Pro components
|
28
28
|
|
29
|
-
|
29
|
+
All of Karafka components are open-source. However, the `Pro` components are licenses under `LICENSE-COMM`.
|
30
|
+
|
31
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your code to Maciej Mensfeld.
|
30
32
|
|
31
33
|
## Questions
|
32
34
|
|
@@ -35,7 +37,5 @@ You can also reach us at hello@karafka.opencollective.com.
|
|
35
37
|
|
36
38
|
## Credits
|
37
39
|
|
38
|
-
### Contributors
|
39
|
-
|
40
40
|
Thank you to all the people who have already contributed to karafka!
|
41
41
|
<a href="graphs/contributors"><img src="https://opencollective.com/karafka/contributors.svg?width=890" /></a>
|
data/Gemfile.lock
CHANGED
@@ -1,22 +1,22 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
karafka (2.0.0.
|
4
|
+
karafka (2.0.0.alpha4)
|
5
5
|
dry-configurable (~> 0.13)
|
6
6
|
dry-monitor (~> 0.5)
|
7
7
|
dry-validation (~> 1.7)
|
8
8
|
rdkafka (>= 0.10)
|
9
9
|
thor (>= 0.20)
|
10
|
-
waterdrop (>= 2.
|
10
|
+
waterdrop (>= 2.2.0, < 3.0.0)
|
11
11
|
zeitwerk (~> 2.3)
|
12
12
|
|
13
13
|
GEM
|
14
14
|
remote: https://rubygems.org/
|
15
15
|
specs:
|
16
|
-
activejob (7.0.
|
17
|
-
activesupport (= 7.0.
|
16
|
+
activejob (7.0.2.2)
|
17
|
+
activesupport (= 7.0.2.2)
|
18
18
|
globalid (>= 0.3.6)
|
19
|
-
activesupport (7.0.
|
19
|
+
activesupport (7.0.2.2)
|
20
20
|
concurrent-ruby (~> 1.0, >= 1.0.2)
|
21
21
|
i18n (>= 1.6, < 2)
|
22
22
|
minitest (>= 5.1)
|
@@ -45,7 +45,7 @@ GEM
|
|
45
45
|
dry-configurable (~> 0.13, >= 0.13.0)
|
46
46
|
dry-core (~> 0.5, >= 0.5)
|
47
47
|
dry-events (~> 0.2)
|
48
|
-
dry-schema (1.
|
48
|
+
dry-schema (1.9.1)
|
49
49
|
concurrent-ruby (~> 1.0)
|
50
50
|
dry-configurable (~> 0.13, >= 0.13.0)
|
51
51
|
dry-core (~> 0.5, >= 0.5)
|
@@ -58,49 +58,49 @@ GEM
|
|
58
58
|
dry-core (~> 0.5, >= 0.5)
|
59
59
|
dry-inflector (~> 0.1, >= 0.1.2)
|
60
60
|
dry-logic (~> 1.0, >= 1.0.2)
|
61
|
-
dry-validation (1.
|
61
|
+
dry-validation (1.8.0)
|
62
62
|
concurrent-ruby (~> 1.0)
|
63
63
|
dry-container (~> 0.7, >= 0.7.1)
|
64
64
|
dry-core (~> 0.5, >= 0.5)
|
65
65
|
dry-initializer (~> 3.0)
|
66
|
-
dry-schema (~> 1.
|
66
|
+
dry-schema (~> 1.9, >= 1.9.1)
|
67
67
|
factory_bot (6.2.0)
|
68
68
|
activesupport (>= 5.0.0)
|
69
69
|
ffi (1.15.5)
|
70
70
|
globalid (1.0.0)
|
71
71
|
activesupport (>= 5.0)
|
72
|
-
i18n (1.
|
72
|
+
i18n (1.10.0)
|
73
73
|
concurrent-ruby (~> 1.0)
|
74
|
-
mini_portile2 (2.
|
74
|
+
mini_portile2 (2.8.0)
|
75
75
|
minitest (5.15.0)
|
76
76
|
rake (13.0.6)
|
77
77
|
rdkafka (0.11.1)
|
78
78
|
ffi (~> 1.15)
|
79
79
|
mini_portile2 (~> 2.6)
|
80
80
|
rake (> 12)
|
81
|
-
rspec (3.
|
82
|
-
rspec-core (~> 3.
|
83
|
-
rspec-expectations (~> 3.
|
84
|
-
rspec-mocks (~> 3.
|
85
|
-
rspec-core (3.
|
86
|
-
rspec-support (~> 3.
|
87
|
-
rspec-expectations (3.
|
81
|
+
rspec (3.11.0)
|
82
|
+
rspec-core (~> 3.11.0)
|
83
|
+
rspec-expectations (~> 3.11.0)
|
84
|
+
rspec-mocks (~> 3.11.0)
|
85
|
+
rspec-core (3.11.0)
|
86
|
+
rspec-support (~> 3.11.0)
|
87
|
+
rspec-expectations (3.11.0)
|
88
88
|
diff-lcs (>= 1.2.0, < 2.0)
|
89
|
-
rspec-support (~> 3.
|
90
|
-
rspec-mocks (3.
|
89
|
+
rspec-support (~> 3.11.0)
|
90
|
+
rspec-mocks (3.11.0)
|
91
91
|
diff-lcs (>= 1.2.0, < 2.0)
|
92
|
-
rspec-support (~> 3.
|
93
|
-
rspec-support (3.
|
92
|
+
rspec-support (~> 3.11.0)
|
93
|
+
rspec-support (3.11.0)
|
94
94
|
simplecov (0.21.2)
|
95
95
|
docile (~> 1.1)
|
96
96
|
simplecov-html (~> 0.11)
|
97
97
|
simplecov_json_formatter (~> 0.1)
|
98
98
|
simplecov-html (0.12.3)
|
99
|
-
simplecov_json_formatter (0.1.
|
99
|
+
simplecov_json_formatter (0.1.4)
|
100
100
|
thor (1.2.1)
|
101
101
|
tzinfo (2.0.4)
|
102
102
|
concurrent-ruby (~> 1.0)
|
103
|
-
waterdrop (2.
|
103
|
+
waterdrop (2.2.0)
|
104
104
|
concurrent-ruby (>= 1.1)
|
105
105
|
dry-configurable (~> 0.13)
|
106
106
|
dry-monitor (~> 0.5)
|
@@ -121,4 +121,4 @@ DEPENDENCIES
|
|
121
121
|
simplecov
|
122
122
|
|
123
123
|
BUNDLED WITH
|
124
|
-
2.3.
|
124
|
+
2.3.7
|
data/LICENSE
CHANGED
@@ -9,6 +9,9 @@ Karafka has also commercial-friendly license, commercial support and commercial
|
|
9
9
|
All of the commercial components are present in the lib/karafka/pro directory of this repository
|
10
10
|
and their usage requires commercial license agreement.
|
11
11
|
|
12
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
|
13
|
+
code to Maciej Mensfeld.
|
14
|
+
|
12
15
|
You can find the commercial license in LICENSE-COM.
|
13
16
|
|
14
17
|
Please see https://karafka.io for purchasing options.
|
data/bin/integrations
CHANGED
@@ -2,9 +2,15 @@
|
|
2
2
|
|
3
3
|
# Runner to run integration specs in parallel
|
4
4
|
|
5
|
+
# Part of integration specs run pristine without bundler.
|
6
|
+
# If we would run bundle exec when running this code, bundler would inject its own context
|
7
|
+
# into them, messing things up heavily
|
8
|
+
raise 'This code needs to be executed WITHOUT bundle exec' if Kernel.const_defined?(:Bundler)
|
9
|
+
|
5
10
|
require 'open3'
|
6
11
|
require 'fileutils'
|
7
12
|
require 'pathname'
|
13
|
+
require 'tmpdir'
|
8
14
|
|
9
15
|
ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
|
10
16
|
|
@@ -13,13 +19,13 @@ IntegrationTestError = Class.new(StandardError)
|
|
13
19
|
|
14
20
|
# How many child processes with integration specs do we want to run in parallel
|
15
21
|
# When the value is high, there's a problem with thread allocation on Github
|
16
|
-
CONCURRENCY =
|
22
|
+
CONCURRENCY = 4
|
17
23
|
|
18
24
|
# Abstraction around a single test scenario execution process
|
19
25
|
class Scenario
|
20
26
|
# How long a scenario can run before we kill it
|
21
27
|
# This is a fail-safe just in case something would hang
|
22
|
-
MAX_RUN_TIME = 60 *
|
28
|
+
MAX_RUN_TIME = 60 * 2
|
23
29
|
|
24
30
|
# There are rare cases where Karafka may force shutdown for some of the integration cases
|
25
31
|
# This includes exactly those
|
@@ -38,9 +44,7 @@ class Scenario
|
|
38
44
|
# @param path [String] path to the scenarios file
|
39
45
|
def initialize(path)
|
40
46
|
@path = path
|
41
|
-
@stdin, @stdout, @stderr, @wait_thr = Open3.popen3(
|
42
|
-
"bundle exec ruby -r ./spec/integrations_helper.rb #{path}"
|
43
|
-
)
|
47
|
+
@stdin, @stdout, @stderr, @wait_thr = Open3.popen3(init_and_build_cmd)
|
44
48
|
@started_at = current_time
|
45
49
|
# Last 1024 characters from stdout
|
46
50
|
@stdout_tail = ''
|
@@ -102,6 +106,35 @@ class Scenario
|
|
102
106
|
|
103
107
|
private
|
104
108
|
|
109
|
+
# Sets up a proper environment for a given spec to run and returns the run command
|
110
|
+
# @return [String] run command
|
111
|
+
def init_and_build_cmd
|
112
|
+
scenario_dir = File.dirname(@path)
|
113
|
+
|
114
|
+
# If there is a Gemfile in a scenario directory, it means it is a pristine spec and we need
|
115
|
+
# to run bundle install, etc in order to run it
|
116
|
+
if File.exist?(File.join(scenario_dir, 'Gemfile'))
|
117
|
+
# We copy the spec into a temp dir, not to pollute the spec location with logs, etc
|
118
|
+
temp_dir = Dir.mktmpdir
|
119
|
+
file_name = File.basename(@path)
|
120
|
+
|
121
|
+
FileUtils.cp_r("#{scenario_dir}/.", temp_dir)
|
122
|
+
|
123
|
+
<<~CMD
|
124
|
+
cd #{temp_dir} &&
|
125
|
+
KARAFKA_GEM_DIR=#{ROOT_PATH} \
|
126
|
+
BUNDLE_AUTO_INSTALL=true \
|
127
|
+
PRISTINE_MODE=true \
|
128
|
+
bundle exec ruby -r #{ROOT_PATH}/spec/integrations_helper.rb #{file_name}
|
129
|
+
CMD
|
130
|
+
else
|
131
|
+
<<~CMD
|
132
|
+
KARAFKA_GEM_DIR=#{ROOT_PATH} \
|
133
|
+
bundle exec ruby -r ./spec/integrations_helper.rb #{@path}
|
134
|
+
CMD
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
105
138
|
# @return [Float] current machine time
|
106
139
|
def current_time
|
107
140
|
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
@@ -136,10 +169,13 @@ end
|
|
136
169
|
# Load all the specs
|
137
170
|
specs = Dir[ROOT_PATH.join('spec/integrations/**/*.rb')]
|
138
171
|
|
139
|
-
# If
|
140
|
-
|
172
|
+
# If filters is provided, apply
|
173
|
+
# Allows to provide several filters one after another and applies all of them
|
174
|
+
ARGV.each do |filter|
|
175
|
+
specs.delete_if { |name| !name.include?(filter) }
|
176
|
+
end
|
141
177
|
|
142
|
-
raise ArgumentError, "No integration specs with
|
178
|
+
raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?
|
143
179
|
|
144
180
|
# Randomize order
|
145
181
|
seed = (ENV['SEED'] || rand(0..10_000)).to_i
|
data/bin/karafka
CHANGED
@@ -2,6 +2,10 @@
|
|
2
2
|
|
3
3
|
require 'karafka'
|
4
4
|
|
5
|
+
# We set this to indicate, that the process in which we are (whatever it does) was started using
|
6
|
+
# our bin/karafka cli
|
7
|
+
ENV['KARAFKA_CLI'] = 'true'
|
8
|
+
|
5
9
|
# If there is a boot file, we need to require it as we expect it to contain
|
6
10
|
# Karafka app setup, routes, etc
|
7
11
|
if File.exist?(Karafka.boot_file)
|
data/bin/stress
CHANGED
data/config/errors.yml
CHANGED
@@ -5,3 +5,4 @@ en:
|
|
5
5
|
topics_names_not_unique: all topic names within a single consumer group must be unique
|
6
6
|
required_usage_count: Given topic must be used at least once
|
7
7
|
consumer_groups_inclusion: Unknown consumer group
|
8
|
+
kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
|
data/docker-compose.yml
CHANGED
data/karafka.gemspec
CHANGED
@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
|
|
21
21
|
spec.add_dependency 'dry-validation', '~> 1.7'
|
22
22
|
spec.add_dependency 'rdkafka', '>= 0.10'
|
23
23
|
spec.add_dependency 'thor', '>= 0.20'
|
24
|
-
spec.add_dependency 'waterdrop', '>= 2.
|
24
|
+
spec.add_dependency 'waterdrop', '>= 2.2.0', '< 3.0.0'
|
25
25
|
spec.add_dependency 'zeitwerk', '~> 2.3'
|
26
26
|
|
27
27
|
spec.required_ruby_version = '>= 2.6.0'
|
data/lib/active_job/karafka.rb
CHANGED
@@ -1,18 +1,21 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
|
4
|
-
require 'active_job
|
5
|
-
|
6
|
-
require 'active_job/routing_extensions'
|
7
|
-
require 'active_job/queue_adapters/karafka_adapter'
|
3
|
+
begin
|
4
|
+
require 'active_job'
|
5
|
+
require_relative 'queue_adapters/karafka_adapter'
|
8
6
|
|
9
|
-
module ActiveJob
|
10
|
-
|
11
|
-
|
12
|
-
|
7
|
+
module ActiveJob
|
8
|
+
# Namespace for usage simplification outside of Rails where Railtie will not kick in.
|
9
|
+
# That way a require 'active_job/karafka' should be enough to use it
|
10
|
+
module Karafka
|
11
|
+
end
|
13
12
|
end
|
14
|
-
end
|
15
13
|
|
16
|
-
# We extend routing builder by adding a simple wrapper for easier jobs topics defining
|
17
|
-
|
18
|
-
|
14
|
+
# We extend routing builder by adding a simple wrapper for easier jobs topics defining
|
15
|
+
# This needs to be extended here as it is going to be used in karafka routes, hence doing that in
|
16
|
+
# the railtie initializer would be too late
|
17
|
+
::Karafka::Routing::Builder.include ::Karafka::ActiveJob::RoutingExtensions
|
18
|
+
::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::RoutingExtensions
|
19
|
+
rescue LoadError
|
20
|
+
# We extend ActiveJob stuff in the railtie
|
21
|
+
end
|
@@ -5,16 +5,13 @@ module ActiveJob
|
|
5
5
|
# ActiveJob queue adapters
|
6
6
|
module QueueAdapters
|
7
7
|
# Karafka adapter for enqueuing jobs
|
8
|
+
# This is here for ease of integration with ActiveJob.
|
8
9
|
class KarafkaAdapter
|
9
|
-
# Enqueues the job
|
10
|
-
# later on consumed by a special ActiveJob consumer
|
10
|
+
# Enqueues the job using the configured dispatcher
|
11
11
|
#
|
12
12
|
# @param job [Object] job that should be enqueued
|
13
13
|
def enqueue(job)
|
14
|
-
::Karafka.
|
15
|
-
topic: job.queue_name,
|
16
|
-
payload: ActiveSupport::JSON.encode(job.serialize)
|
17
|
-
)
|
14
|
+
::Karafka::App.config.internal.active_job.dispatcher.call(job)
|
18
15
|
end
|
19
16
|
|
20
17
|
# Raises info, that Karafka backend does not support scheduling jobs
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# This is the consumer for ActiveJob that eats the messages enqueued with it one after another.
|
6
|
+
# It marks the offset after each message, so we make sure, none of the jobs is executed twice
|
7
|
+
class Consumer < ::Karafka::BaseConsumer
|
8
|
+
# Executes the ActiveJob logic
|
9
|
+
# @note ActiveJob does not support batches, so we just run one message after another
|
10
|
+
def consume
|
11
|
+
messages.each do |message|
|
12
|
+
::ActiveJob::Base.execute(
|
13
|
+
# We technically speaking could set this as deserializer and reference it from the
|
14
|
+
# message instead of using the `#raw_payload`. This is not done on purpose to simplify
|
15
|
+
# the ActiveJob setup here
|
16
|
+
::ActiveSupport::JSON.decode(message.raw_payload)
|
17
|
+
)
|
18
|
+
|
19
|
+
mark_as_consumed(message)
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# Dispatcher that sends the ActiveJob job to a proper topic based on the queue name
|
6
|
+
class Dispatcher
|
7
|
+
# Defaults for dispatching
|
8
|
+
# The can be updated by using `#karafka_options` on the job
|
9
|
+
DEFAULTS = {
|
10
|
+
dispatch_method: :produce_async
|
11
|
+
}.freeze
|
12
|
+
|
13
|
+
private_constant :DEFAULTS
|
14
|
+
|
15
|
+
# @param job [ActiveJob::Base] job
|
16
|
+
def call(job)
|
17
|
+
::Karafka.producer.public_send(
|
18
|
+
fetch_option(job, :dispatch_method, DEFAULTS),
|
19
|
+
topic: job.queue_name,
|
20
|
+
payload: ::ActiveSupport::JSON.encode(job.serialize)
|
21
|
+
)
|
22
|
+
end
|
23
|
+
|
24
|
+
private
|
25
|
+
|
26
|
+
# @param job [ActiveJob::Base] job
|
27
|
+
# @param key [Symbol] key we want to fetch
|
28
|
+
# @param defaults [Hash]
|
29
|
+
# @return [Object] options we are interested in
|
30
|
+
def fetch_option(job, key, defaults)
|
31
|
+
job
|
32
|
+
.class
|
33
|
+
.karafka_options
|
34
|
+
.fetch(key, defaults.fetch(key))
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# Allows for setting karafka specific options in ActiveJob jobs
|
6
|
+
module JobExtensions
|
7
|
+
class << self
|
8
|
+
# Defines all the needed accessors and sets defaults
|
9
|
+
# @param klass [ActiveJob::Base] active job base
|
10
|
+
def extended(klass)
|
11
|
+
klass.class_attribute :_karafka_options
|
12
|
+
klass._karafka_options = {}
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
# @param new_options [Hash] additional options that allow for jobs Karafka related options
|
17
|
+
# customization
|
18
|
+
# @return [Hash] karafka options
|
19
|
+
def karafka_options(new_options = {})
|
20
|
+
return _karafka_options if new_options.empty?
|
21
|
+
|
22
|
+
# Make sure, that karafka options that someone wants to use are valid before assigning
|
23
|
+
# them
|
24
|
+
App.config.internal.active_job.job_options_contract.validate!(new_options)
|
25
|
+
|
26
|
+
new_options.each do |name, value|
|
27
|
+
_karafka_options[name] = value
|
28
|
+
end
|
29
|
+
|
30
|
+
_karafka_options
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
@@ -0,0 +1,15 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module ActiveJob
|
5
|
+
# Contract for validating the options that can be altered with `#karafka_options` per job class
|
6
|
+
# @note We keep this in the `Karafka::ActiveJob` namespace instead of `Karafka::Contracts` as
|
7
|
+
# we want to keep ActiveJob related Karafka components outside of the core Karafka code and
|
8
|
+
# all in the same place
|
9
|
+
class JobOptionsContract < Contracts::Base
|
10
|
+
params do
|
11
|
+
optional(:dispatch_method).value(included_in?: %i[produce_async produce_sync])
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
# ActiveJob related Karafka stuff
|
5
|
+
module ActiveJob
|
6
|
+
# Routing extensions for ActiveJob
|
7
|
+
module RoutingExtensions
|
8
|
+
# This method simplifies routes definition for ActiveJob topics / queues by auto-injecting
|
9
|
+
# the consumer class
|
10
|
+
# @param name [String, Symbol] name of the topic where ActiveJobs jobs should go
|
11
|
+
def active_job_topic(name)
|
12
|
+
topic(name) do
|
13
|
+
consumer App.config.internal.active_job.consumer
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
data/lib/karafka/app.rb
CHANGED
data/lib/karafka/cli/info.rb
CHANGED
@@ -31,9 +31,9 @@ module Karafka
|
|
31
31
|
def core_info
|
32
32
|
config = Karafka::App.config
|
33
33
|
|
34
|
-
postfix =
|
34
|
+
postfix = Karafka.pro? ? ' + Pro' : ''
|
35
35
|
|
36
|
-
|
36
|
+
[
|
37
37
|
"Karafka version: #{Karafka::VERSION}#{postfix}",
|
38
38
|
"Ruby version: #{RUBY_VERSION}",
|
39
39
|
"Rdkafka version: #{::Rdkafka::VERSION}",
|
@@ -49,7 +49,7 @@ module Karafka
|
|
49
49
|
def license_info
|
50
50
|
config = Karafka::App.config
|
51
51
|
|
52
|
-
if
|
52
|
+
if Karafka.pro?
|
53
53
|
[
|
54
54
|
'License: Commercial',
|
55
55
|
"License entity: #{config.license.entity}",
|
data/lib/karafka/cli/install.rb
CHANGED
data/lib/karafka/cli/server.rb
CHANGED
@@ -5,11 +5,6 @@ module Karafka
|
|
5
5
|
class Cli < Thor
|
6
6
|
# Server Karafka Cli action
|
7
7
|
class Server < Base
|
8
|
-
# Server config settings contract
|
9
|
-
CONTRACT = Contracts::ServerCliOptions.new.freeze
|
10
|
-
|
11
|
-
private_constant :CONTRACT
|
12
|
-
|
13
8
|
desc 'Start the Karafka server (short-cut alias: "s")'
|
14
9
|
option aliases: 's'
|
15
10
|
option :consumer_groups, type: :array, default: nil, aliases: :g
|
@@ -19,7 +14,7 @@ module Karafka
|
|
19
14
|
# Print our banner and info in the dev mode
|
20
15
|
print_marketing_info if Karafka::App.env.development?
|
21
16
|
|
22
|
-
validate!
|
17
|
+
Contracts::ServerCliOptions.new.validate!(cli.options)
|
23
18
|
|
24
19
|
# We assign active topics on a server level, as only server is expected to listen on
|
25
20
|
# part of the topics
|
@@ -34,7 +29,7 @@ module Karafka
|
|
34
29
|
def print_marketing_info
|
35
30
|
Karafka.logger.info Info::BANNER
|
36
31
|
|
37
|
-
if Karafka
|
32
|
+
if Karafka.pro?
|
38
33
|
Karafka.logger.info(
|
39
34
|
"\033[0;32mThank you for investing in the Karafka Pro subscription!\033[0m\n"
|
40
35
|
)
|
@@ -44,15 +39,6 @@ module Karafka
|
|
44
39
|
)
|
45
40
|
end
|
46
41
|
end
|
47
|
-
|
48
|
-
# Checks the server cli configuration
|
49
|
-
# options validations in terms of app setup (topics, pid existence, etc)
|
50
|
-
def validate!
|
51
|
-
result = CONTRACT.call(cli.options)
|
52
|
-
return if result.success?
|
53
|
-
|
54
|
-
raise Errors::InvalidConfigurationError, result.errors.to_h
|
55
|
-
end
|
56
42
|
end
|
57
43
|
end
|
58
44
|
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Karafka
|
4
|
+
module Contracts
|
5
|
+
# Base contract for all Karafka contracts
|
6
|
+
class Base < Dry::Validation::Contract
|
7
|
+
config.messages.load_paths << File.join(Karafka.gem_root, 'config', 'errors.yml')
|
8
|
+
|
9
|
+
# @param data [Hash] data for validation
|
10
|
+
# @return [Boolean] true if all good
|
11
|
+
# @raise [Errors::InvalidConfigurationError] invalid configuration error
|
12
|
+
# @note We use contracts only in the config validation context, so no need to add support
|
13
|
+
# for multiple error classes. It will be added when it will be needed.
|
14
|
+
def validate!(data)
|
15
|
+
result = call(data)
|
16
|
+
|
17
|
+
return true if result.success?
|
18
|
+
|
19
|
+
raise Errors::InvalidConfigurationError, result.errors.to_h
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|