karafka 2.0.0.alpha1 → 2.0.0.alpha4

Sign up to get free protection for your applications and to get access to all the features.
Files changed (56) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/.github/workflows/ci.yml +3 -1
  4. data/CHANGELOG.md +14 -1
  5. data/CONTRIBUTING.md +6 -6
  6. data/Gemfile.lock +24 -24
  7. data/LICENSE +3 -0
  8. data/bin/integrations +44 -8
  9. data/bin/karafka +4 -0
  10. data/bin/stress +1 -1
  11. data/config/errors.yml +1 -0
  12. data/docker-compose.yml +1 -0
  13. data/karafka.gemspec +1 -1
  14. data/lib/active_job/karafka.rb +16 -13
  15. data/lib/active_job/queue_adapters/karafka_adapter.rb +3 -6
  16. data/lib/karafka/active_job/consumer.rb +24 -0
  17. data/lib/karafka/active_job/dispatcher.rb +38 -0
  18. data/lib/karafka/active_job/job_extensions.rb +34 -0
  19. data/lib/karafka/active_job/job_options_contract.rb +15 -0
  20. data/lib/karafka/active_job/routing_extensions.rb +18 -0
  21. data/lib/karafka/app.rb +1 -0
  22. data/lib/karafka/cli/info.rb +3 -3
  23. data/lib/karafka/cli/install.rb +1 -0
  24. data/lib/karafka/cli/server.rb +2 -16
  25. data/lib/karafka/contracts/base.rb +23 -0
  26. data/lib/karafka/contracts/config.rb +21 -3
  27. data/lib/karafka/contracts/consumer_group.rb +1 -3
  28. data/lib/karafka/contracts/consumer_group_topic.rb +2 -3
  29. data/lib/karafka/contracts/server_cli_options.rb +1 -3
  30. data/lib/karafka/errors.rb +4 -0
  31. data/lib/karafka/instrumentation/monitor.rb +1 -0
  32. data/lib/karafka/instrumentation/stdout_listener.rb +3 -0
  33. data/lib/karafka/licenser.rb +20 -9
  34. data/lib/karafka/messages/batch_metadata.rb +2 -0
  35. data/lib/karafka/messages/builders/batch_metadata.rb +23 -1
  36. data/lib/karafka/pro/active_job/dispatcher.rb +58 -0
  37. data/lib/karafka/pro/active_job/job_options_contract.rb +27 -0
  38. data/lib/karafka/pro/loader.rb +29 -0
  39. data/lib/karafka/pro.rb +13 -0
  40. data/lib/karafka/processing/worker.rb +1 -1
  41. data/lib/karafka/railtie.rb +55 -19
  42. data/lib/karafka/routing/builder.rb +1 -11
  43. data/lib/karafka/routing/subscription_group.rb +5 -5
  44. data/lib/karafka/routing/subscription_groups_builder.rb +1 -0
  45. data/lib/karafka/routing/topic.rb +1 -0
  46. data/lib/karafka/setup/config.rb +25 -20
  47. data/lib/karafka/status.rb +1 -0
  48. data/lib/karafka/templates/karafka.rb.erb +1 -1
  49. data/lib/karafka/version.rb +1 -1
  50. data/lib/karafka.rb +7 -2
  51. data.tar.gz.sig +0 -0
  52. metadata +14 -7
  53. metadata.gz.sig +0 -0
  54. data/.github/FUNDING.yml +0 -3
  55. data/lib/active_job/consumer.rb +0 -22
  56. data/lib/active_job/routing_extensions.rb +0 -15
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a441a124346099aa31fb36a2241c2abfbc242ab6f3f9ad197b82d8786e78232e
4
- data.tar.gz: 81a706d58e469ecc48759dfe28c30e3779baab848122b72c194c30e06a7cdd7d
3
+ metadata.gz: f8ed5005d3288abb4f9f6389fa126b5434b5a7a6df729e924695d38624e167e5
4
+ data.tar.gz: aa802a441d9c9cb1275260ba9b290c8c4c72acf125ca72efaafc58538a6c8b20
5
5
  SHA512:
6
- metadata.gz: bf9ebff340150913cf273bc3f09c9f3dd3c02e6ffe427fd00ead842995e5f146d554609cd1ff4d1039ee96ef349b8e7712e2c557abc322a9e0300886bc765b6a
7
- data.tar.gz: 054cf8240a33a133c2c9dce5f1c3c2ac26877006c8a823c1e58c903fccd8cac7c31b5d0c0d2ec22257d8959d40bb69d49bf5676f378c33004807a78413278b3e
6
+ metadata.gz: 1ad458acdb42c28d04895db1aa348d0ab10177c0829e7b19fc9fd8b23aaac5fc17feac5e1366cd6444d1e60903b087be1b24298b7764a98f8f372a325afb7cee
7
+ data.tar.gz: fb1de540a2c50d26467585bb720a636726955e65f1cea1ca055ef4a9f10d66db233928ca8b067d225d6c935f779d50f2381660ddcaa0d86d6c5b0b8a8ed991e1
checksums.yaml.gz.sig CHANGED
Binary file
@@ -113,6 +113,7 @@ jobs:
113
113
  - name: Install latest Bundler
114
114
  run: |
115
115
  gem install bundler --no-document
116
+ gem update --system --no-document
116
117
  bundle config set without 'tools benchmarks docs'
117
118
 
118
119
  - name: Bundle install
@@ -122,5 +123,6 @@ jobs:
122
123
 
123
124
  - name: Run integration tests
124
125
  env:
126
+ KARAFKA_PRO_LICENSE_TOKEN: ${{ secrets.KARAFKA_PRO_LICENSE_TOKEN }}
125
127
  GITHUB_COVERAGE: ${{matrix.coverage}}
126
- run: bundle exec bin/integrations
128
+ run: bin/integrations
data/CHANGELOG.md CHANGED
@@ -1,6 +1,19 @@
1
1
  # Karafka framework changelog
2
2
 
3
- ## 2.0.0-alpha1 (Unreleased)
3
+ ## 2.0.0-alpha4 (Unreleased)
4
+ - Rails support without ActiveJob queue adapter usage (#805)
5
+
6
+ ## 2.0.0-alpha3 (2022-03-16)
7
+ - Restore 'app.initialized' state and add notification on it
8
+ - Fix the installation flow for Rails and add integration tests for this scenario
9
+ - Add more integration tests covering some edge cases
10
+
11
+ ## 2.0.0-alpha2 (2022-02-19)
12
+ - Require `kafka` keys to be symbols
13
+ - Added ActiveJob Pro adapter
14
+ - Small updates to the license and docs
15
+
16
+ ## 2.0.0-alpha1 (2022-01-30)
4
17
  - Change license to `LGPL-3.0`
5
18
  - Introduce a Pro subscription
6
19
  - Switch from `ruby-kafka` to `librdkafka` as an underlying driver
data/CONTRIBUTING.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  ## Introduction
4
4
 
5
- First, thank you for considering contributing to karafka! It's people like you that make the open source community such a great community! 😊
5
+ First, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community! 😊
6
6
 
7
7
  We welcome any type of contribution, not only code. You can help with:
8
8
  - **QA**: file bug reports, the more details you can give the better (e.g. screenshots with the console open)
@@ -18,15 +18,17 @@ Working on your first Pull Request? You can learn how from this *free* series, [
18
18
 
19
19
  Any code change should be submitted as a pull request. The description should explain what the code does and give steps to execute it. The pull request should also contain tests.
20
20
 
21
- ## Code review process
21
+ ### Code review process
22
22
 
23
23
  Each pull request must pass all the rspec specs and meet our quality requirements.
24
24
 
25
25
  To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
26
26
 
27
- Coditsu will automatically check your work against our quality standards. You can find your commit check results on the [builds page](https://app.coditsu.io/karafka/commit_builds) of Karafka organization.
27
+ ### Contributing to Pro components
28
28
 
29
- [![coditsu](https://coditsu.io/assets/quality_bar.svg)](https://app.coditsu.io/karafka/commit_builds)
29
+ All of Karafka components are open-source. However, the `Pro` components are licenses under `LICENSE-COMM`.
30
+
31
+ By sending a pull request to the pro components, you are agreeing to transfer the copyright of your code to Maciej Mensfeld.
30
32
 
31
33
  ## Questions
32
34
 
@@ -35,7 +37,5 @@ You can also reach us at hello@karafka.opencollective.com.
35
37
 
36
38
  ## Credits
37
39
 
38
- ### Contributors
39
-
40
40
  Thank you to all the people who have already contributed to karafka!
41
41
  <a href="graphs/contributors"><img src="https://opencollective.com/karafka/contributors.svg?width=890" /></a>
data/Gemfile.lock CHANGED
@@ -1,22 +1,22 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- karafka (2.0.0.pre.alpha1)
4
+ karafka (2.0.0.alpha4)
5
5
  dry-configurable (~> 0.13)
6
6
  dry-monitor (~> 0.5)
7
7
  dry-validation (~> 1.7)
8
8
  rdkafka (>= 0.10)
9
9
  thor (>= 0.20)
10
- waterdrop (>= 2.1.0, < 3.0.0)
10
+ waterdrop (>= 2.2.0, < 3.0.0)
11
11
  zeitwerk (~> 2.3)
12
12
 
13
13
  GEM
14
14
  remote: https://rubygems.org/
15
15
  specs:
16
- activejob (7.0.1)
17
- activesupport (= 7.0.1)
16
+ activejob (7.0.2.2)
17
+ activesupport (= 7.0.2.2)
18
18
  globalid (>= 0.3.6)
19
- activesupport (7.0.1)
19
+ activesupport (7.0.2.2)
20
20
  concurrent-ruby (~> 1.0, >= 1.0.2)
21
21
  i18n (>= 1.6, < 2)
22
22
  minitest (>= 5.1)
@@ -45,7 +45,7 @@ GEM
45
45
  dry-configurable (~> 0.13, >= 0.13.0)
46
46
  dry-core (~> 0.5, >= 0.5)
47
47
  dry-events (~> 0.2)
48
- dry-schema (1.8.0)
48
+ dry-schema (1.9.1)
49
49
  concurrent-ruby (~> 1.0)
50
50
  dry-configurable (~> 0.13, >= 0.13.0)
51
51
  dry-core (~> 0.5, >= 0.5)
@@ -58,49 +58,49 @@ GEM
58
58
  dry-core (~> 0.5, >= 0.5)
59
59
  dry-inflector (~> 0.1, >= 0.1.2)
60
60
  dry-logic (~> 1.0, >= 1.0.2)
61
- dry-validation (1.7.0)
61
+ dry-validation (1.8.0)
62
62
  concurrent-ruby (~> 1.0)
63
63
  dry-container (~> 0.7, >= 0.7.1)
64
64
  dry-core (~> 0.5, >= 0.5)
65
65
  dry-initializer (~> 3.0)
66
- dry-schema (~> 1.8, >= 1.8.0)
66
+ dry-schema (~> 1.9, >= 1.9.1)
67
67
  factory_bot (6.2.0)
68
68
  activesupport (>= 5.0.0)
69
69
  ffi (1.15.5)
70
70
  globalid (1.0.0)
71
71
  activesupport (>= 5.0)
72
- i18n (1.9.1)
72
+ i18n (1.10.0)
73
73
  concurrent-ruby (~> 1.0)
74
- mini_portile2 (2.7.1)
74
+ mini_portile2 (2.8.0)
75
75
  minitest (5.15.0)
76
76
  rake (13.0.6)
77
77
  rdkafka (0.11.1)
78
78
  ffi (~> 1.15)
79
79
  mini_portile2 (~> 2.6)
80
80
  rake (> 12)
81
- rspec (3.10.0)
82
- rspec-core (~> 3.10.0)
83
- rspec-expectations (~> 3.10.0)
84
- rspec-mocks (~> 3.10.0)
85
- rspec-core (3.10.2)
86
- rspec-support (~> 3.10.0)
87
- rspec-expectations (3.10.2)
81
+ rspec (3.11.0)
82
+ rspec-core (~> 3.11.0)
83
+ rspec-expectations (~> 3.11.0)
84
+ rspec-mocks (~> 3.11.0)
85
+ rspec-core (3.11.0)
86
+ rspec-support (~> 3.11.0)
87
+ rspec-expectations (3.11.0)
88
88
  diff-lcs (>= 1.2.0, < 2.0)
89
- rspec-support (~> 3.10.0)
90
- rspec-mocks (3.10.3)
89
+ rspec-support (~> 3.11.0)
90
+ rspec-mocks (3.11.0)
91
91
  diff-lcs (>= 1.2.0, < 2.0)
92
- rspec-support (~> 3.10.0)
93
- rspec-support (3.10.3)
92
+ rspec-support (~> 3.11.0)
93
+ rspec-support (3.11.0)
94
94
  simplecov (0.21.2)
95
95
  docile (~> 1.1)
96
96
  simplecov-html (~> 0.11)
97
97
  simplecov_json_formatter (~> 0.1)
98
98
  simplecov-html (0.12.3)
99
- simplecov_json_formatter (0.1.3)
99
+ simplecov_json_formatter (0.1.4)
100
100
  thor (1.2.1)
101
101
  tzinfo (2.0.4)
102
102
  concurrent-ruby (~> 1.0)
103
- waterdrop (2.1.0)
103
+ waterdrop (2.2.0)
104
104
  concurrent-ruby (>= 1.1)
105
105
  dry-configurable (~> 0.13)
106
106
  dry-monitor (~> 0.5)
@@ -121,4 +121,4 @@ DEPENDENCIES
121
121
  simplecov
122
122
 
123
123
  BUNDLED WITH
124
- 2.3.5
124
+ 2.3.7
data/LICENSE CHANGED
@@ -9,6 +9,9 @@ Karafka has also commercial-friendly license, commercial support and commercial
9
9
  All of the commercial components are present in the lib/karafka/pro directory of this repository
10
10
  and their usage requires commercial license agreement.
11
11
 
12
+ By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
13
+ code to Maciej Mensfeld.
14
+
12
15
  You can find the commercial license in LICENSE-COM.
13
16
 
14
17
  Please see https://karafka.io for purchasing options.
data/bin/integrations CHANGED
@@ -2,9 +2,15 @@
2
2
 
3
3
  # Runner to run integration specs in parallel
4
4
 
5
+ # Part of integration specs run pristine without bundler.
6
+ # If we would run bundle exec when running this code, bundler would inject its own context
7
+ # into them, messing things up heavily
8
+ raise 'This code needs to be executed WITHOUT bundle exec' if Kernel.const_defined?(:Bundler)
9
+
5
10
  require 'open3'
6
11
  require 'fileutils'
7
12
  require 'pathname'
13
+ require 'tmpdir'
8
14
 
9
15
  ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
10
16
 
@@ -13,13 +19,13 @@ IntegrationTestError = Class.new(StandardError)
13
19
 
14
20
  # How many child processes with integration specs do we want to run in parallel
15
21
  # When the value is high, there's a problem with thread allocation on Github
16
- CONCURRENCY = 5
22
+ CONCURRENCY = 4
17
23
 
18
24
  # Abstraction around a single test scenario execution process
19
25
  class Scenario
20
26
  # How long a scenario can run before we kill it
21
27
  # This is a fail-safe just in case something would hang
22
- MAX_RUN_TIME = 60 * 5
28
+ MAX_RUN_TIME = 60 * 2
23
29
 
24
30
  # There are rare cases where Karafka may force shutdown for some of the integration cases
25
31
  # This includes exactly those
@@ -38,9 +44,7 @@ class Scenario
38
44
  # @param path [String] path to the scenarios file
39
45
  def initialize(path)
40
46
  @path = path
41
- @stdin, @stdout, @stderr, @wait_thr = Open3.popen3(
42
- "bundle exec ruby -r ./spec/integrations_helper.rb #{path}"
43
- )
47
+ @stdin, @stdout, @stderr, @wait_thr = Open3.popen3(init_and_build_cmd)
44
48
  @started_at = current_time
45
49
  # Last 1024 characters from stdout
46
50
  @stdout_tail = ''
@@ -102,6 +106,35 @@ class Scenario
102
106
 
103
107
  private
104
108
 
109
+ # Sets up a proper environment for a given spec to run and returns the run command
110
+ # @return [String] run command
111
+ def init_and_build_cmd
112
+ scenario_dir = File.dirname(@path)
113
+
114
+ # If there is a Gemfile in a scenario directory, it means it is a pristine spec and we need
115
+ # to run bundle install, etc in order to run it
116
+ if File.exist?(File.join(scenario_dir, 'Gemfile'))
117
+ # We copy the spec into a temp dir, not to pollute the spec location with logs, etc
118
+ temp_dir = Dir.mktmpdir
119
+ file_name = File.basename(@path)
120
+
121
+ FileUtils.cp_r("#{scenario_dir}/.", temp_dir)
122
+
123
+ <<~CMD
124
+ cd #{temp_dir} &&
125
+ KARAFKA_GEM_DIR=#{ROOT_PATH} \
126
+ BUNDLE_AUTO_INSTALL=true \
127
+ PRISTINE_MODE=true \
128
+ bundle exec ruby -r #{ROOT_PATH}/spec/integrations_helper.rb #{file_name}
129
+ CMD
130
+ else
131
+ <<~CMD
132
+ KARAFKA_GEM_DIR=#{ROOT_PATH} \
133
+ bundle exec ruby -r ./spec/integrations_helper.rb #{@path}
134
+ CMD
135
+ end
136
+ end
137
+
105
138
  # @return [Float] current machine time
106
139
  def current_time
107
140
  Process.clock_gettime(Process::CLOCK_MONOTONIC)
@@ -136,10 +169,13 @@ end
136
169
  # Load all the specs
137
170
  specs = Dir[ROOT_PATH.join('spec/integrations/**/*.rb')]
138
171
 
139
- # If filter is provided, apply
140
- specs.delete_if { |name| !name.include?(ARGV[0]) } if ARGV[0]
172
+ # If filters is provided, apply
173
+ # Allows to provide several filters one after another and applies all of them
174
+ ARGV.each do |filter|
175
+ specs.delete_if { |name| !name.include?(filter) }
176
+ end
141
177
 
142
- raise ArgumentError, "No integration specs with filter: #{ARGV[0]}" if specs.empty?
178
+ raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?
143
179
 
144
180
  # Randomize order
145
181
  seed = (ENV['SEED'] || rand(0..10_000)).to_i
data/bin/karafka CHANGED
@@ -2,6 +2,10 @@
2
2
 
3
3
  require 'karafka'
4
4
 
5
+ # We set this to indicate, that the process in which we are (whatever it does) was started using
6
+ # our bin/karafka cli
7
+ ENV['KARAFKA_CLI'] = 'true'
8
+
5
9
  # If there is a boot file, we need to require it as we expect it to contain
6
10
  # Karafka app setup, routes, etc
7
11
  if File.exist?(Karafka.boot_file)
data/bin/stress CHANGED
@@ -9,5 +9,5 @@ set -e
9
9
  while :
10
10
  do
11
11
  reset
12
- bundle exec bin/integrations $1
12
+ bin/integrations $1
13
13
  done
data/config/errors.yml CHANGED
@@ -5,3 +5,4 @@ en:
5
5
  topics_names_not_unique: all topic names within a single consumer group must be unique
6
6
  required_usage_count: Given topic must be used at least once
7
7
  consumer_groups_inclusion: Unknown consumer group
8
+ kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
data/docker-compose.yml CHANGED
@@ -16,6 +16,7 @@ services:
16
16
  KAFKA_CREATE_TOPICS:
17
17
  "integrations_0_03:3:1,\
18
18
  integrations_1_03:3:1,\
19
+ integrations_2_03:3:1,\
19
20
  integrations_0_10:10:1,\
20
21
  integrations_1_10:10:1,\
21
22
  benchmarks_0_01:1:1,\
data/karafka.gemspec CHANGED
@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
21
21
  spec.add_dependency 'dry-validation', '~> 1.7'
22
22
  spec.add_dependency 'rdkafka', '>= 0.10'
23
23
  spec.add_dependency 'thor', '>= 0.20'
24
- spec.add_dependency 'waterdrop', '>= 2.1.0', '< 3.0.0'
24
+ spec.add_dependency 'waterdrop', '>= 2.2.0', '< 3.0.0'
25
25
  spec.add_dependency 'zeitwerk', '~> 2.3'
26
26
 
27
27
  spec.required_ruby_version = '>= 2.6.0'
@@ -1,18 +1,21 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require 'active_job'
4
- require 'active_job/queue_adapters'
5
- require 'active_job/consumer'
6
- require 'active_job/routing_extensions'
7
- require 'active_job/queue_adapters/karafka_adapter'
3
+ begin
4
+ require 'active_job'
5
+ require_relative 'queue_adapters/karafka_adapter'
8
6
 
9
- module ActiveJob
10
- # Namespace for usage simplification outside of Rails where Railtie will not kick in.
11
- # That way a require 'active_job/karafka' should be enough to use it
12
- module Karafka
7
+ module ActiveJob
8
+ # Namespace for usage simplification outside of Rails where Railtie will not kick in.
9
+ # That way a require 'active_job/karafka' should be enough to use it
10
+ module Karafka
11
+ end
13
12
  end
14
- end
15
13
 
16
- # We extend routing builder by adding a simple wrapper for easier jobs topics defining
17
- ::Karafka::Routing::Builder.include ActiveJob::RoutingExtensions
18
- ::Karafka::Routing::Proxy.include ActiveJob::RoutingExtensions
14
+ # We extend routing builder by adding a simple wrapper for easier jobs topics defining
15
+ # This needs to be extended here as it is going to be used in karafka routes, hence doing that in
16
+ # the railtie initializer would be too late
17
+ ::Karafka::Routing::Builder.include ::Karafka::ActiveJob::RoutingExtensions
18
+ ::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::RoutingExtensions
19
+ rescue LoadError
20
+ # We extend ActiveJob stuff in the railtie
21
+ end
@@ -5,16 +5,13 @@ module ActiveJob
5
5
  # ActiveJob queue adapters
6
6
  module QueueAdapters
7
7
  # Karafka adapter for enqueuing jobs
8
+ # This is here for ease of integration with ActiveJob.
8
9
  class KarafkaAdapter
9
- # Enqueues the job by sending all the payload to a dedicated topic in Kafka that will be
10
- # later on consumed by a special ActiveJob consumer
10
+ # Enqueues the job using the configured dispatcher
11
11
  #
12
12
  # @param job [Object] job that should be enqueued
13
13
  def enqueue(job)
14
- ::Karafka.producer.produce_async(
15
- topic: job.queue_name,
16
- payload: ActiveSupport::JSON.encode(job.serialize)
17
- )
14
+ ::Karafka::App.config.internal.active_job.dispatcher.call(job)
18
15
  end
19
16
 
20
17
  # Raises info, that Karafka backend does not support scheduling jobs
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Karafka
4
+ module ActiveJob
5
+ # This is the consumer for ActiveJob that eats the messages enqueued with it one after another.
6
+ # It marks the offset after each message, so we make sure, none of the jobs is executed twice
7
+ class Consumer < ::Karafka::BaseConsumer
8
+ # Executes the ActiveJob logic
9
+ # @note ActiveJob does not support batches, so we just run one message after another
10
+ def consume
11
+ messages.each do |message|
12
+ ::ActiveJob::Base.execute(
13
+ # We technically speaking could set this as deserializer and reference it from the
14
+ # message instead of using the `#raw_payload`. This is not done on purpose to simplify
15
+ # the ActiveJob setup here
16
+ ::ActiveSupport::JSON.decode(message.raw_payload)
17
+ )
18
+
19
+ mark_as_consumed(message)
20
+ end
21
+ end
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Karafka
4
+ module ActiveJob
5
+ # Dispatcher that sends the ActiveJob job to a proper topic based on the queue name
6
+ class Dispatcher
7
+ # Defaults for dispatching
8
+ # The can be updated by using `#karafka_options` on the job
9
+ DEFAULTS = {
10
+ dispatch_method: :produce_async
11
+ }.freeze
12
+
13
+ private_constant :DEFAULTS
14
+
15
+ # @param job [ActiveJob::Base] job
16
+ def call(job)
17
+ ::Karafka.producer.public_send(
18
+ fetch_option(job, :dispatch_method, DEFAULTS),
19
+ topic: job.queue_name,
20
+ payload: ::ActiveSupport::JSON.encode(job.serialize)
21
+ )
22
+ end
23
+
24
+ private
25
+
26
+ # @param job [ActiveJob::Base] job
27
+ # @param key [Symbol] key we want to fetch
28
+ # @param defaults [Hash]
29
+ # @return [Object] options we are interested in
30
+ def fetch_option(job, key, defaults)
31
+ job
32
+ .class
33
+ .karafka_options
34
+ .fetch(key, defaults.fetch(key))
35
+ end
36
+ end
37
+ end
38
+ end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Karafka
4
+ module ActiveJob
5
+ # Allows for setting karafka specific options in ActiveJob jobs
6
+ module JobExtensions
7
+ class << self
8
+ # Defines all the needed accessors and sets defaults
9
+ # @param klass [ActiveJob::Base] active job base
10
+ def extended(klass)
11
+ klass.class_attribute :_karafka_options
12
+ klass._karafka_options = {}
13
+ end
14
+ end
15
+
16
+ # @param new_options [Hash] additional options that allow for jobs Karafka related options
17
+ # customization
18
+ # @return [Hash] karafka options
19
+ def karafka_options(new_options = {})
20
+ return _karafka_options if new_options.empty?
21
+
22
+ # Make sure, that karafka options that someone wants to use are valid before assigning
23
+ # them
24
+ App.config.internal.active_job.job_options_contract.validate!(new_options)
25
+
26
+ new_options.each do |name, value|
27
+ _karafka_options[name] = value
28
+ end
29
+
30
+ _karafka_options
31
+ end
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,15 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Karafka
4
+ module ActiveJob
5
+ # Contract for validating the options that can be altered with `#karafka_options` per job class
6
+ # @note We keep this in the `Karafka::ActiveJob` namespace instead of `Karafka::Contracts` as
7
+ # we want to keep ActiveJob related Karafka components outside of the core Karafka code and
8
+ # all in the same place
9
+ class JobOptionsContract < Contracts::Base
10
+ params do
11
+ optional(:dispatch_method).value(included_in?: %i[produce_async produce_sync])
12
+ end
13
+ end
14
+ end
15
+ end
@@ -0,0 +1,18 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Karafka
4
+ # ActiveJob related Karafka stuff
5
+ module ActiveJob
6
+ # Routing extensions for ActiveJob
7
+ module RoutingExtensions
8
+ # This method simplifies routes definition for ActiveJob topics / queues by auto-injecting
9
+ # the consumer class
10
+ # @param name [String, Symbol] name of the topic where ActiveJobs jobs should go
11
+ def active_job_topic(name)
12
+ topic(name) do
13
+ consumer App.config.internal.active_job.consumer
14
+ end
15
+ end
16
+ end
17
+ end
18
+ end
data/lib/karafka/app.rb CHANGED
@@ -36,6 +36,7 @@ module Karafka
36
36
  logger
37
37
  producer
38
38
  monitor
39
+ pro?
39
40
  ].each do |delegated|
40
41
  define_method(delegated) do
41
42
  Karafka.send(delegated)
@@ -31,9 +31,9 @@ module Karafka
31
31
  def core_info
32
32
  config = Karafka::App.config
33
33
 
34
- postfix = config.license.token ? ' + Pro' : ''
34
+ postfix = Karafka.pro? ? ' + Pro' : ''
35
35
 
36
- info = [
36
+ [
37
37
  "Karafka version: #{Karafka::VERSION}#{postfix}",
38
38
  "Ruby version: #{RUBY_VERSION}",
39
39
  "Rdkafka version: #{::Rdkafka::VERSION}",
@@ -49,7 +49,7 @@ module Karafka
49
49
  def license_info
50
50
  config = Karafka::App.config
51
51
 
52
- if config.license.token
52
+ if Karafka.pro?
53
53
  [
54
54
  'License: Commercial',
55
55
  "License entity: #{config.license.entity}",
@@ -14,6 +14,7 @@ module Karafka
14
14
  app/consumers
15
15
  config
16
16
  log
17
+ lib
17
18
  ].freeze
18
19
 
19
20
  # Where should we map proper files from templates
@@ -5,11 +5,6 @@ module Karafka
5
5
  class Cli < Thor
6
6
  # Server Karafka Cli action
7
7
  class Server < Base
8
- # Server config settings contract
9
- CONTRACT = Contracts::ServerCliOptions.new.freeze
10
-
11
- private_constant :CONTRACT
12
-
13
8
  desc 'Start the Karafka server (short-cut alias: "s")'
14
9
  option aliases: 's'
15
10
  option :consumer_groups, type: :array, default: nil, aliases: :g
@@ -19,7 +14,7 @@ module Karafka
19
14
  # Print our banner and info in the dev mode
20
15
  print_marketing_info if Karafka::App.env.development?
21
16
 
22
- validate!
17
+ Contracts::ServerCliOptions.new.validate!(cli.options)
23
18
 
24
19
  # We assign active topics on a server level, as only server is expected to listen on
25
20
  # part of the topics
@@ -34,7 +29,7 @@ module Karafka
34
29
  def print_marketing_info
35
30
  Karafka.logger.info Info::BANNER
36
31
 
37
- if Karafka::App.config.license.token
32
+ if Karafka.pro?
38
33
  Karafka.logger.info(
39
34
  "\033[0;32mThank you for investing in the Karafka Pro subscription!\033[0m\n"
40
35
  )
@@ -44,15 +39,6 @@ module Karafka
44
39
  )
45
40
  end
46
41
  end
47
-
48
- # Checks the server cli configuration
49
- # options validations in terms of app setup (topics, pid existence, etc)
50
- def validate!
51
- result = CONTRACT.call(cli.options)
52
- return if result.success?
53
-
54
- raise Errors::InvalidConfigurationError, result.errors.to_h
55
- end
56
42
  end
57
43
  end
58
44
  end
@@ -0,0 +1,23 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Karafka
4
+ module Contracts
5
+ # Base contract for all Karafka contracts
6
+ class Base < Dry::Validation::Contract
7
+ config.messages.load_paths << File.join(Karafka.gem_root, 'config', 'errors.yml')
8
+
9
+ # @param data [Hash] data for validation
10
+ # @return [Boolean] true if all good
11
+ # @raise [Errors::InvalidConfigurationError] invalid configuration error
12
+ # @note We use contracts only in the config validation context, so no need to add support
13
+ # for multiple error classes. It will be added when it will be needed.
14
+ def validate!(data)
15
+ result = call(data)
16
+
17
+ return true if result.success?
18
+
19
+ raise Errors::InvalidConfigurationError, result.errors.to_h
20
+ end
21
+ end
22
+ end
23
+ end