karafka 1.4.13 → 2.0.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (170) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +3 -3
  3. data/.github/workflows/ci.yml +85 -30
  4. data/.ruby-version +1 -1
  5. data/CHANGELOG.md +268 -7
  6. data/CONTRIBUTING.md +10 -19
  7. data/Gemfile +6 -0
  8. data/Gemfile.lock +44 -87
  9. data/LICENSE +17 -0
  10. data/LICENSE-COMM +89 -0
  11. data/LICENSE-LGPL +165 -0
  12. data/README.md +44 -48
  13. data/bin/benchmarks +85 -0
  14. data/bin/create_token +22 -0
  15. data/bin/integrations +237 -0
  16. data/bin/karafka +4 -0
  17. data/bin/scenario +29 -0
  18. data/bin/stress_many +13 -0
  19. data/bin/stress_one +13 -0
  20. data/bin/wait_for_kafka +20 -0
  21. data/certs/karafka-pro.pem +11 -0
  22. data/config/errors.yml +55 -40
  23. data/docker-compose.yml +39 -3
  24. data/karafka.gemspec +11 -17
  25. data/lib/active_job/karafka.rb +21 -0
  26. data/lib/active_job/queue_adapters/karafka_adapter.rb +26 -0
  27. data/lib/karafka/active_job/consumer.rb +26 -0
  28. data/lib/karafka/active_job/dispatcher.rb +38 -0
  29. data/lib/karafka/active_job/job_extensions.rb +34 -0
  30. data/lib/karafka/active_job/job_options_contract.rb +21 -0
  31. data/lib/karafka/active_job/routing/extensions.rb +31 -0
  32. data/lib/karafka/app.rb +15 -20
  33. data/lib/karafka/base_consumer.rb +181 -31
  34. data/lib/karafka/cli/base.rb +4 -4
  35. data/lib/karafka/cli/info.rb +43 -9
  36. data/lib/karafka/cli/install.rb +19 -10
  37. data/lib/karafka/cli/server.rb +17 -42
  38. data/lib/karafka/cli.rb +4 -11
  39. data/lib/karafka/connection/client.rb +385 -90
  40. data/lib/karafka/connection/listener.rb +246 -38
  41. data/lib/karafka/connection/listeners_batch.rb +24 -0
  42. data/lib/karafka/connection/messages_buffer.rb +84 -0
  43. data/lib/karafka/connection/pauses_manager.rb +46 -0
  44. data/lib/karafka/connection/raw_messages_buffer.rb +101 -0
  45. data/lib/karafka/connection/rebalance_manager.rb +78 -0
  46. data/lib/karafka/contracts/base.rb +17 -0
  47. data/lib/karafka/contracts/config.rb +88 -11
  48. data/lib/karafka/contracts/consumer_group.rb +21 -189
  49. data/lib/karafka/contracts/consumer_group_topic.rb +34 -11
  50. data/lib/karafka/contracts/server_cli_options.rb +19 -18
  51. data/lib/karafka/contracts.rb +1 -1
  52. data/lib/karafka/env.rb +46 -0
  53. data/lib/karafka/errors.rb +21 -21
  54. data/lib/karafka/helpers/async.rb +33 -0
  55. data/lib/karafka/helpers/colorize.rb +20 -0
  56. data/lib/karafka/helpers/multi_delegator.rb +2 -2
  57. data/lib/karafka/instrumentation/callbacks/error.rb +40 -0
  58. data/lib/karafka/instrumentation/callbacks/statistics.rb +41 -0
  59. data/lib/karafka/instrumentation/logger_listener.rb +164 -0
  60. data/lib/karafka/instrumentation/monitor.rb +13 -61
  61. data/lib/karafka/instrumentation/notifications.rb +52 -0
  62. data/lib/karafka/instrumentation/proctitle_listener.rb +3 -3
  63. data/lib/karafka/instrumentation/vendors/datadog/dashboard.json +1 -0
  64. data/lib/karafka/instrumentation/vendors/datadog/listener.rb +232 -0
  65. data/lib/karafka/instrumentation.rb +21 -0
  66. data/lib/karafka/licenser.rb +75 -0
  67. data/lib/karafka/messages/batch_metadata.rb +45 -0
  68. data/lib/karafka/messages/builders/batch_metadata.rb +40 -0
  69. data/lib/karafka/messages/builders/message.rb +39 -0
  70. data/lib/karafka/messages/builders/messages.rb +32 -0
  71. data/lib/karafka/{params/params.rb → messages/message.rb} +7 -12
  72. data/lib/karafka/messages/messages.rb +64 -0
  73. data/lib/karafka/{params → messages}/metadata.rb +4 -6
  74. data/lib/karafka/messages/seek.rb +9 -0
  75. data/lib/karafka/patches/rdkafka/consumer.rb +22 -0
  76. data/lib/karafka/pro/active_job/consumer.rb +46 -0
  77. data/lib/karafka/pro/active_job/dispatcher.rb +61 -0
  78. data/lib/karafka/pro/active_job/job_options_contract.rb +32 -0
  79. data/lib/karafka/pro/base_consumer.rb +82 -0
  80. data/lib/karafka/pro/contracts/base.rb +21 -0
  81. data/lib/karafka/pro/contracts/consumer_group.rb +34 -0
  82. data/lib/karafka/pro/contracts/consumer_group_topic.rb +33 -0
  83. data/lib/karafka/pro/loader.rb +76 -0
  84. data/lib/karafka/pro/performance_tracker.rb +80 -0
  85. data/lib/karafka/pro/processing/coordinator.rb +72 -0
  86. data/lib/karafka/pro/processing/jobs/consume_non_blocking.rb +37 -0
  87. data/lib/karafka/pro/processing/jobs_builder.rb +32 -0
  88. data/lib/karafka/pro/processing/partitioner.rb +60 -0
  89. data/lib/karafka/pro/processing/scheduler.rb +56 -0
  90. data/lib/karafka/pro/routing/builder_extensions.rb +30 -0
  91. data/lib/karafka/pro/routing/topic_extensions.rb +38 -0
  92. data/lib/karafka/pro.rb +13 -0
  93. data/lib/karafka/process.rb +1 -0
  94. data/lib/karafka/processing/coordinator.rb +88 -0
  95. data/lib/karafka/processing/coordinators_buffer.rb +54 -0
  96. data/lib/karafka/processing/executor.rb +118 -0
  97. data/lib/karafka/processing/executors_buffer.rb +88 -0
  98. data/lib/karafka/processing/jobs/base.rb +51 -0
  99. data/lib/karafka/processing/jobs/consume.rb +42 -0
  100. data/lib/karafka/processing/jobs/revoked.rb +22 -0
  101. data/lib/karafka/processing/jobs/shutdown.rb +23 -0
  102. data/lib/karafka/processing/jobs_builder.rb +29 -0
  103. data/lib/karafka/processing/jobs_queue.rb +144 -0
  104. data/lib/karafka/processing/partitioner.rb +22 -0
  105. data/lib/karafka/processing/result.rb +29 -0
  106. data/lib/karafka/processing/scheduler.rb +22 -0
  107. data/lib/karafka/processing/worker.rb +88 -0
  108. data/lib/karafka/processing/workers_batch.rb +27 -0
  109. data/lib/karafka/railtie.rb +113 -0
  110. data/lib/karafka/routing/builder.rb +15 -24
  111. data/lib/karafka/routing/consumer_group.rb +11 -19
  112. data/lib/karafka/routing/consumer_mapper.rb +1 -2
  113. data/lib/karafka/routing/router.rb +1 -1
  114. data/lib/karafka/routing/subscription_group.rb +53 -0
  115. data/lib/karafka/routing/subscription_groups_builder.rb +53 -0
  116. data/lib/karafka/routing/topic.rb +61 -24
  117. data/lib/karafka/routing/topics.rb +38 -0
  118. data/lib/karafka/runner.rb +51 -0
  119. data/lib/karafka/serialization/json/deserializer.rb +6 -15
  120. data/lib/karafka/server.rb +67 -26
  121. data/lib/karafka/setup/config.rb +147 -175
  122. data/lib/karafka/status.rb +14 -5
  123. data/lib/karafka/templates/example_consumer.rb.erb +16 -0
  124. data/lib/karafka/templates/karafka.rb.erb +15 -51
  125. data/lib/karafka/time_trackers/base.rb +19 -0
  126. data/lib/karafka/time_trackers/pause.rb +92 -0
  127. data/lib/karafka/time_trackers/poll.rb +65 -0
  128. data/lib/karafka/version.rb +1 -1
  129. data/lib/karafka.rb +38 -17
  130. data.tar.gz.sig +0 -0
  131. metadata +118 -120
  132. metadata.gz.sig +0 -0
  133. data/MIT-LICENCE +0 -18
  134. data/lib/karafka/assignment_strategies/round_robin.rb +0 -13
  135. data/lib/karafka/attributes_map.rb +0 -63
  136. data/lib/karafka/backends/inline.rb +0 -16
  137. data/lib/karafka/base_responder.rb +0 -226
  138. data/lib/karafka/cli/flow.rb +0 -48
  139. data/lib/karafka/cli/missingno.rb +0 -19
  140. data/lib/karafka/code_reloader.rb +0 -67
  141. data/lib/karafka/connection/api_adapter.rb +0 -158
  142. data/lib/karafka/connection/batch_delegator.rb +0 -55
  143. data/lib/karafka/connection/builder.rb +0 -23
  144. data/lib/karafka/connection/message_delegator.rb +0 -36
  145. data/lib/karafka/consumers/batch_metadata.rb +0 -10
  146. data/lib/karafka/consumers/callbacks.rb +0 -71
  147. data/lib/karafka/consumers/includer.rb +0 -64
  148. data/lib/karafka/consumers/responders.rb +0 -24
  149. data/lib/karafka/consumers/single_params.rb +0 -15
  150. data/lib/karafka/contracts/responder_usage.rb +0 -54
  151. data/lib/karafka/fetcher.rb +0 -42
  152. data/lib/karafka/helpers/class_matcher.rb +0 -88
  153. data/lib/karafka/helpers/config_retriever.rb +0 -46
  154. data/lib/karafka/helpers/inflector.rb +0 -26
  155. data/lib/karafka/instrumentation/stdout_listener.rb +0 -140
  156. data/lib/karafka/params/batch_metadata.rb +0 -26
  157. data/lib/karafka/params/builders/batch_metadata.rb +0 -30
  158. data/lib/karafka/params/builders/params.rb +0 -38
  159. data/lib/karafka/params/builders/params_batch.rb +0 -25
  160. data/lib/karafka/params/params_batch.rb +0 -60
  161. data/lib/karafka/patches/ruby_kafka.rb +0 -47
  162. data/lib/karafka/persistence/client.rb +0 -29
  163. data/lib/karafka/persistence/consumers.rb +0 -45
  164. data/lib/karafka/persistence/topics.rb +0 -48
  165. data/lib/karafka/responders/builder.rb +0 -36
  166. data/lib/karafka/responders/topic.rb +0 -55
  167. data/lib/karafka/routing/topic_mapper.rb +0 -53
  168. data/lib/karafka/serialization/json/serializer.rb +0 -31
  169. data/lib/karafka/setup/configurators/water_drop.rb +0 -36
  170. data/lib/karafka/templates/application_responder.rb.erb +0 -11
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8799d5854288a36f4af9e7688241a0df2719a7d982fe4940d9d7acec80ec8389
4
- data.tar.gz: 7d17c4de581aa08467c8c810f8a2977a8e7cffa1f0f68f2ff369e39bc15f888b
3
+ metadata.gz: 06f96b3ba14b7910d3cb23a90bdc135927f1483d5af4d079d59b3d6940d391b9
4
+ data.tar.gz: 2d936d4ddac360e229004c05f6ee51d004018e3c4cf4203067a99e6df93df92e
5
5
  SHA512:
6
- metadata.gz: f43dc6e07e61671aff52a9200333945e93b3641e63a44ff5d877456278553040f5ba2e6dbf69f30570d974bcaa6010312b66da5aee478f7d356b3268a62428a2
7
- data.tar.gz: 87f1ed908bfc5f3076c42bec507c916900881e7536f23822d38fd8cd0214b0fdedb58a6d6920e8f11dd0e035c6ec301bfb217f906d129873b2b5a8df20d18e53
6
+ metadata.gz: caeb9bcf1f0301f31442025176f3da631aa8ff9c21164e112b9ce3112372309d4b7fe59863cf7cf511e6a6bc4448f61448f3bc2024b101d4ae2668a358e5bbe9
7
+ data.tar.gz: cfc422fae74512c2142ef36f9d4b5c3024bf318d79dc83456e27553aebc177cec7a3772f6378160c8ecdf5a3b2626d862ba91d823fae6e30cbb593d4198474a9
checksums.yaml.gz.sig CHANGED
@@ -1,3 +1,3 @@
1
- ������<e~��?([��]�a�#z�jq���*�����(�9�! ��Q� ��ʠ����n���N�bH{b�A`����b�6��B���X���/H��O�K?���,���^i<)�\k'f
2
- F0�>��+vG�fj�X�~>�x�5R��f��8qns����,��ݸP���q�n���,T#Ң"뻡v��m%��6n��`�Þ�Jӿ��g�����-��`�u��V��E3�,�b7K�[`y�dxJ�� -/b������RN'ɋw���m; O�̿��i�b`���/>�.�n"/��)
3
- 9t�퉛Adǰ������:�����
1
+ �����L��zi�.��.�fT�
2
+ guM��Ԉ142��B��U 0ǪqEJ\�����o�zrU_z!b�ל2_�R��iў1��t,}P��]{��l��0��om�.�$lbl��p(KW��V,�iC)��~��K��¸�Gv��m+��[By!�����h���03R'�k���Ygu�'L9u%�T���G�ĸ�e���;�KqQd�m���=���l�>��AG\t��jí�%��n���鐁Q�C. ���4s:�){+�Q0��=Nh�� ɀ����X!à�� ����Ul�ZD�����'�� 磶.j��v[
3
+
@@ -8,7 +8,43 @@ on:
8
8
  schedule:
9
9
  - cron: '0 1 * * *'
10
10
 
11
+ env:
12
+ BUNDLE_RETRY: 6
13
+ BUNDLE_JOBS: 4
14
+
11
15
  jobs:
16
+ diffend:
17
+ runs-on: ubuntu-latest
18
+ strategy:
19
+ fail-fast: false
20
+ steps:
21
+ - uses: actions/checkout@v2
22
+ with:
23
+ fetch-depth: 0
24
+
25
+ - name: Set up Ruby
26
+ uses: ruby/setup-ruby@v1
27
+ with:
28
+ ruby-version: 3.1
29
+ bundler-cache: true
30
+
31
+ - name: Install Diffend plugin
32
+ run: bundle plugin install diffend
33
+
34
+ - name: Bundle Secure
35
+ run: bundle secure
36
+
37
+ coditsu:
38
+ runs-on: ubuntu-latest
39
+ strategy:
40
+ fail-fast: false
41
+ steps:
42
+ - uses: actions/checkout@v2
43
+ with:
44
+ fetch-depth: 0
45
+ - name: Run Coditsu
46
+ run: \curl -sSL https://api.coditsu.io/run/ci | bash
47
+
12
48
  specs:
13
49
  runs-on: ubuntu-latest
14
50
  needs: diffend
@@ -26,51 +62,70 @@ jobs:
26
62
  - uses: actions/checkout@v2
27
63
  - name: Install package dependencies
28
64
  run: "[ -e $APT_DEPS ] || sudo apt-get install -y --no-install-recommends $APT_DEPS"
65
+
66
+ - name: Start Kafka with docker-compose
67
+ run: |
68
+ docker-compose up -d
69
+
29
70
  - name: Set up Ruby
30
71
  uses: ruby/setup-ruby@v1
31
72
  with:
32
73
  ruby-version: ${{matrix.ruby}}
33
- - name: Install latest bundler
34
- run: |
35
- gem install bundler --no-document
36
- bundle config set without 'tools benchmarks docs'
37
- - name: Bundle install
74
+ bundler-cache: true
75
+
76
+ - name: Ensure all needed Kafka topics are created and wait if not
38
77
  run: |
39
- bundle config set without development
40
- bundle install --jobs 4 --retry 3
41
- - name: Run Kafka with docker-compose
42
- run: docker-compose up -d
43
- - name: Run all tests
78
+ bin/wait_for_kafka
79
+
80
+ - name: Run all specs
44
81
  env:
45
82
  GITHUB_COVERAGE: ${{matrix.coverage}}
46
83
  run: bundle exec rspec
47
84
 
48
- diffend:
85
+ integrations:
49
86
  runs-on: ubuntu-latest
87
+ needs: diffend
50
88
  strategy:
51
89
  fail-fast: false
90
+ matrix:
91
+ ruby:
92
+ - '3.1'
93
+ - '3.0'
94
+ - '2.7'
95
+ include:
96
+ - ruby: '3.1'
97
+ coverage: 'true'
52
98
  steps:
53
99
  - uses: actions/checkout@v2
54
- with:
55
- fetch-depth: 0
100
+ - name: Install package dependencies
101
+ run: "[ -e $APT_DEPS ] || sudo apt-get install -y --no-install-recommends $APT_DEPS"
102
+
103
+ - name: Start Kafka with docker-compose
104
+ run: |
105
+ docker-compose up -d
106
+
56
107
  - name: Set up Ruby
57
108
  uses: ruby/setup-ruby@v1
58
109
  with:
59
- ruby-version: 3.1
60
- - name: Install latest bundler
61
- run: gem install bundler --no-document
62
- - name: Install Diffend plugin
63
- run: bundle plugin install diffend
64
- - name: Bundle Secure
65
- run: bundle secure
110
+ ruby-version: ${{matrix.ruby}}
66
111
 
67
- coditsu:
68
- runs-on: ubuntu-latest
69
- strategy:
70
- fail-fast: false
71
- steps:
72
- - uses: actions/checkout@v2
73
- with:
74
- fetch-depth: 0
75
- - name: Run Coditsu
76
- run: \curl -sSL https://api.coditsu.io/run/ci | bash
112
+ - name: Install latest Bundler
113
+ run: |
114
+ gem install bundler --no-document
115
+ gem update --system --no-document
116
+ bundle config set without 'tools benchmarks docs'
117
+
118
+ - name: Bundle install
119
+ run: |
120
+ bundle config set without development
121
+ bundle install
122
+
123
+ - name: Ensure all needed Kafka topics are created and wait if not
124
+ run: |
125
+ bin/wait_for_kafka
126
+
127
+ - name: Run integration tests
128
+ env:
129
+ KARAFKA_PRO_LICENSE_TOKEN: ${{ secrets.KARAFKA_PRO_LICENSE_TOKEN }}
130
+ GITHUB_COVERAGE: ${{matrix.coverage}}
131
+ run: bin/integrations
data/.ruby-version CHANGED
@@ -1 +1 @@
1
- 3.1.0
1
+ 3.1.2
data/CHANGELOG.md CHANGED
@@ -1,12 +1,273 @@
1
1
  # Karafka framework changelog
2
2
 
3
- ## 1.4.13 (2022-02-19)
4
- - Drop support for ruby 2.6
5
- - Add mfa requirement
6
-
7
- ## 1.4.12 (2022-01-13)
8
- - Ruby 3.1 support
9
- - `irb` dependency removal (vbyno)
3
+ ## 2.0.0 (2022-08-5)
4
+
5
+ This changelog describes changes between `1.4` and `2.0`. Please refer to appropriate release notes for changes between particular `rc` releases.
6
+
7
+ Karafka 2.0 is a **major** rewrite that brings many new things to the table but also removes specific concepts that happened not to be as good as I initially thought when I created them.
8
+
9
+ Please consider getting a Pro version if you want to **support** my work on the Karafka ecosystem!
10
+
11
+ For anyone worried that I will start converting regular features into Pro: This will **not** happen. Anything free and fully OSS in Karafka 1.4 will **forever** remain free. Most additions and improvements to the ecosystem are to its free parts. Any feature that is introduced as a free and open one will not become paid.
12
+
13
+ ### Additions
14
+
15
+ This section describes **new** things and concepts introduced with Karafka 2.0.
16
+
17
+ Karafka 2.0:
18
+
19
+ - Introduces multi-threaded support for [concurrent work](https://github.com/karafka/karafka/wiki/Concurrency-and-multithreading) consumption for separate partitions as well as for single partition work via [Virtual Partitions](https://github.com/karafka/karafka/wiki/Pro-Virtual-Partitions).
20
+ - Introduces [Active Job adapter](https://github.com/karafka/karafka/wiki/Active-Job) for using Karafka as a jobs backend with Ruby on Rails Active Job.
21
+ - Introduces fully automatic integration end-to-end [test suite](https://github.com/karafka/karafka/tree/master/spec/integrations) that checks any case I could imagine.
22
+ - Introduces [Virtual Partitions](https://github.com/karafka/karafka/wiki/Pro-Virtual-Partitions) for ability to parallelize work of a single partition.
23
+ - Introduces [Long-Running Jobs](https://github.com/karafka/karafka/wiki/Pro-Long-Running-Jobs) to allow for work that would otherwise exceed the `max.poll.interval.ms`.
24
+ - Introduces the [Enhanced Scheduler](https://github.com/karafka/karafka/wiki/Pro-Enhanced-Scheduler) that uses a non-preemptive LJF (Longest Job First) algorithm instead of a a FIFO (First-In, First-Out) one.
25
+ - Introduces [Enhanced Active Job adapter](https://github.com/karafka/karafka/wiki/Pro-Enhanced-Active-Job) that is optimized and allows for strong ordering of jobs and more.
26
+ - Introduces seamless [Ruby on Rails integration](https://github.com/karafka/karafka/wiki/Integrating-with-Ruby-on-Rails-and-other-frameworks) via `Rails::Railte` without need for any extra configuration.
27
+ - Provides `#revoked` [method](https://github.com/karafka/karafka/wiki/Consuming-messages#shutdown-and-partition-revocation-handlers) for taking actions upon topic revocation.
28
+ - Emits underlying async errors emitted from `librdkafka` via the standardized `error.occurred` [monitor channel](https://github.com/karafka/karafka/wiki/Error-handling-and-back-off-policy#error-tracking).
29
+ - Replaces `ruby-kafka` with `librdkafka` as an underlying driver.
30
+ - Introduces official [EOL policies](https://github.com/karafka/karafka/wiki/Versions-Lifecycle-and-EOL).
31
+ - Introduces [benchmarks](https://github.com/karafka/karafka/tree/master/spec/benchmarks) that can be used to profile Karafka.
32
+ - Introduces a requirement that the end user code **needs** to be [thread-safe](https://github.com/karafka/karafka/wiki/FAQ#does-karafka-require-gems-to-be-thread-safe).
33
+ - Introduces a [Pro subscription](https://github.com/karafka/karafka/wiki/Build-vs.-Buy) with a [commercial license](https://github.com/karafka/karafka/blob/master/LICENSE-COMM) to fund further ecosystem development.
34
+
35
+ ### Deletions
36
+
37
+ This section describes things that are **no longer** part of the Karafka ecosystem.
38
+
39
+ Karafka 2.0:
40
+
41
+ - Removes topics mappers concept completely.
42
+ - Removes pidfiles support.
43
+ - Removes daemonization support.
44
+ - Removes support for using `sidekiq-backend` due to introduction of [multi-threading](https://github.com/karafka/karafka/wiki/Concurrency-and-multithreading).
45
+ - Removes the `Responders` concept in favour of WaterDrop producer usage.
46
+ - Removes completely all the callbacks in favour of finalizer method `#shutdown`.
47
+ - Removes single message consumption mode in favour of [documentation](https://github.com/karafka/karafka/wiki/Consuming-messages#one-at-a-time) on how to do it easily by yourself.
48
+
49
+ ### Changes
50
+
51
+ This section describes things that were **changed** in Karafka but are still present.
52
+
53
+ Karafka 2.0:
54
+
55
+ - Uses only instrumentation that comes from Karafka. This applies also to notifications coming natively from `librdkafka`. They are now piped through Karafka prior to being dispatched.
56
+ - Integrates WaterDrop `2.x` tightly with autoconfiguration inheritance and an option to redefine it.
57
+ - Integrates with the `karafka-testing` gem for RSpec that also has been updated.
58
+ - Updates `cli info` to reflect the `2.0` details.
59
+ - Stops validating `kafka` configuration beyond minimum as the rest is handled by `librdkafka`.
60
+ - No longer uses `dry-validation`.
61
+ - No longer uses `dry-monitor`.
62
+ - No longer uses `dry-configurable`.
63
+ - Lowers general external dependencies three **heavily**.
64
+ - Renames `Karafka::Params::BatchMetadata` to `Karafka::Messages::BatchMetadata`.
65
+ - Renames `Karafka::Params::Params` to `Karafka::Messages::Message`.
66
+ - Renames `#params_batch` in consumers to `#messages`.
67
+ - Renames `Karafka::Params::Metadata` to `Karafka::Messages::Metadata`.
68
+ - Renames `Karafka::Fetcher` to `Karafka::Runner` and align notifications key names.
69
+ - Renames `StdoutListener` to `LoggerListener`.
70
+ - Reorganizes [monitoring and logging](https://github.com/karafka/karafka/wiki/Monitoring-and-logging) to match new concepts.
71
+ - Notifies on fatal worker processing errors.
72
+ - Contains updated install templates for Rails and no-non Rails.
73
+ - Changes how the routing style (`0.5`) behaves. It now builds a single consumer group instead of one per topic.
74
+ - Introduces changes that will allow me to build full web-UI in the upcoming `2.1`.
75
+ - Contains updated example apps.
76
+ - Standardizes error hooks for all error reporting (`error.occurred`).
77
+ - Changes license to `LGPL-3.0`.
78
+ - Introduces a `karafka-core` dependency that contains common code used across the ecosystem.
79
+ - Contains updated [wiki](https://github.com/karafka/karafka/wiki) on everything I could think of.
80
+
81
+ ### What's ahead
82
+
83
+ Karafka 2.0 is just the beginning.
84
+
85
+ There are several things in the plan already for 2.1 and beyond, including a web dashboard, at-rest encryption, transactions support, and more.
86
+
87
+ ## 2.0.0.rc6 (2022-08-05)
88
+ - Update licenser to use a gem based approach based on `karafka-license`.
89
+ - Do not mark intermediate jobs as consumed when Karafka runs Enhanced Active Job with Virtual Partitions.
90
+ - Improve development experience by adding fast cluster state changes refresh (#944)
91
+ - Improve the license loading.
92
+
93
+ ## 2.0.0.rc5 (2022-08-01)
94
+ - Improve specs stability
95
+ - Improve forceful shutdown
96
+ - Add support for debug `TTIN` backtrace printing
97
+ - Fix a case where logger listener would not intercept `warn` level
98
+ - Require `rdkafka` >= `0.12`
99
+ - Replace statistics decorator with the one from `karafka-core`
100
+
101
+ ## 2.0.0.rc4 (2022-07-28)
102
+ - Remove `dry-monitor`
103
+ - Use `karafka-core`
104
+ - Improve forceful shutdown resources finalization
105
+ - Cache consumer client name
106
+
107
+ ## 2.0.0.rc3 (2022-07-26)
108
+ - Fix Pro partitioner hash function may not utilize all the threads (#907).
109
+ - Improve virtual partitions messages distribution.
110
+ - Add StatsD/DataDog optional monitoring listener + dashboard template.
111
+ - Validate that Pro consumer is always used for Pro subscription.
112
+ - Improve ActiveJob consumer shutdown behaviour.
113
+ - Change default `max_wait_time` to 1 second.
114
+ - Change default `max_messages` to 100 (#915).
115
+ - Move logger listener polling reporting level to debug when no messages (#916).
116
+ - Improve stability on aggressive rebalancing (multiple rebalances in a short period).
117
+ - Improve specs stability.
118
+ - Allow using `:key` and `:partition_key` for Enhanced Active Job partitioning.
119
+
120
+ ## 2.0.0.rc2 (2022-07-19)
121
+ - Fix `example_consumer.rb.erb` `#shutdown` and `#revoked` signatures to correct once.
122
+ - Improve the install user experience (print status and created files).
123
+ - Change default `max_wait_time` from 10s to 5s.
124
+ - Remove direct dependency on `dry-configurable` in favour of a home-brew.
125
+ - Remove direct dependency on `dry-validation` in favour of a home-brew.
126
+
127
+ ## 2.0.0-rc1 (2022-07-08)
128
+ - Extract consumption partitioner out of listener inline code.
129
+ - Introduce virtual partitioner concept for parallel processing of data from a single topic partition.
130
+ - Improve stability when there kafka internal errors occur while polling.
131
+ - Fix a case where we would resume a LRJ partition upon rebalance where we would reclaim the partition while job was still running.
132
+ - Do not revoke pauses for lost partitions. This will allow to un-pause reclaimed partitions when LRJ jobs are done.
133
+ - Fail integrations by default (unless configured otherwise) if any errors occur during Karafka server execution.
134
+
135
+ ## 2.0.0-beta5 (2022-07-05)
136
+ - Always resume processing of a revoked partition upon assignment.
137
+ - Improve specs stability.
138
+ - Fix a case where revocation job would be executed on partition for which we never did any work.
139
+ - Introduce a jobs group coordinator for easier jobs management.
140
+ - Improve stability of resuming paused partitions that were revoked and re-assigned.
141
+ - Optimize reaction time on partition ownership changes.
142
+ - Fix a bug where despite setting long max wait time, we would return messages prior to it while not reaching the desired max messages count.
143
+ - Add more integration specs related to polling limits.
144
+ - Remove auto-detection of re-assigned partitions upon rebalance as for too fast rebalances it could not be accurate enough. It would also mess up in case of rebalances that would happen right after a `#seek` was issued for a partition.
145
+ - Optimize the removal of pre-buffered lost partitions data.
146
+ - Always run `#revoked` when rebalance with revocation happens.
147
+ - Evict executors upon rebalance, to prevent race-conditions.
148
+ - Align topics names for integration specs.
149
+
150
+ ## 2.0.0-beta4 (2022-06-20)
151
+ - Rename job internal api methods from `#prepare` to `#before_call` and from `#teardown` to `#after_call` to abstract away jobs execution from any type of executors and consumers logic
152
+ - Remove ability of running `before_consume` and `after_consume` completely. Those should be for internal usage only.
153
+ - Reorganize how Pro consumer and Pro AJ consumers inherit.
154
+ - Require WaterDrop `2.3.1`.
155
+ - Add more integration specs for rebalancing and max poll exceeded.
156
+ - Move `revoked?` state from PRO to regular Karafka.
157
+ - Use return value of `mark_as_consumed!` and `mark_as_consumed` as indicator of partition ownership + use it to switch the ownership state.
158
+ - Do not remove rebalance manager upon client reset and recovery. This will allow us to keep the notion of lost partitions, so we can run revocation jobs for blocking jobs that exceeded the max poll interval.
159
+ - Run revocation jobs upon reaching max poll interval for blocking jobs.
160
+ - Early exit `poll` operation upon partition lost or max poll exceeded event.
161
+ - Always reset consumer instances on timeout exceeded.
162
+ - Wait for Kafka to create all the needed topics before running specs in CI.
163
+
164
+ ## 2.0.0-beta3 (2022-06-14)
165
+ - Jobs building responsibility extracted out of the listener code base.
166
+ - Fix a case where specs supervisor would try to kill no longer running process (#868)
167
+ - Fix an instable integration spec that could misbehave under load
168
+ - Commit offsets prior to pausing partitions to ensure that the latest offset is always committed
169
+ - Fix a case where consecutive CTRL+C (non-stop) would case an exception during forced shutdown
170
+ - Add missing `consumer.prepared.error` into `LoggerListener`
171
+ - Delegate partition resuming from the consumers to listeners threads.
172
+ - Add support for Long-Running Jobs (LRJ) for ActiveJob [PRO]
173
+ - Add support for Long-Running Jobs for consumers [PRO]
174
+ - Allow `active_job_topic` to accept a block for extra topic related settings
175
+ - Remove no longer needed logger threads
176
+ - Auto-adapt number of processes for integration specs based on the number of CPUs
177
+ - Introduce an integration spec runner that prints everything to stdout (better for development)
178
+ - Introduce extra integration specs for various ActiveJob usage scenarios
179
+ - Rename consumer method `#prepared` to `#prepare` to reflect better its use-case
180
+ - For test and dev raise an error when expired license key is used (never for non dev)
181
+ - Add worker related monitor events (`worker.process` and `worker.processed`)
182
+ - Update `LoggerListener` to include more useful information about processing and polling messages
183
+
184
+ ## 2.0.0-beta2 (2022-06-07)
185
+ - Abstract away notion of topics groups (until now it was just an array)
186
+ - Optimize how jobs queue is closed. Since we enqueue jobs only from the listeners, we can safely close jobs queue once listeners are done. By extracting this responsibility from listeners, we remove corner cases and race conditions. Note here: for non-blocking jobs we do wait for them to finish while running the `poll`. This ensures, that for async jobs that are long-living, we do not reach `max.poll.interval`.
187
+ - `Shutdown` jobs are executed in workers to align all the jobs behaviours.
188
+ - `Shutdown` jobs are always blocking.
189
+ - Notion of `ListenersBatch` was introduced similar to `WorkersBatch` to abstract this concept.
190
+ - Change default `shutdown_timeout` to be more than `max_wait_time` not to cause forced shutdown when no messages are being received from Kafka.
191
+ - Abstract away scheduling of revocation and shutdown jobs for both default and pro schedulers
192
+ - Introduce a second (internal) messages buffer to distinguish between raw messages buffer and karafka messages buffer
193
+ - Move messages and their metadata remap process to the listener thread to allow for their inline usage
194
+ - Change how we wait in the shutdown phase, so shutdown jobs can still use Kafka connection even if they run for a longer period of time. This will prevent us from being kicked out from the group early.
195
+ - Introduce validation that ensures, that `shutdown_timeout` is more than `max_wait_time`. This will prevent users from ending up with a config that could lead to frequent forceful shutdowns.
196
+
197
+ ## 2.0.0-beta1 (2022-05-22)
198
+ - Update the jobs queue blocking engine and allow for non-blocking jobs execution
199
+ - Provide `#prepared` hook that always runs before the fetching loop is unblocked
200
+ - [Pro] Introduce performance tracker for scheduling optimizer
201
+ - Provide ability to pause (`#pause`) and resume (`#resume`) given partitions from the consumers
202
+ - Small integration specs refactoring + specs for pausing scenarios
203
+
204
+ ## 2.0.0-alpha6 (2022-04-17)
205
+ - Fix a bug, where upon missing boot file and Rails, railtie would fail with a generic exception (#818)
206
+ - Fix an issue with parallel pristine specs colliding with each other during `bundle install` (#820)
207
+ - Replace `consumer.consume` with `consumer.consumed` event to match the behaviour
208
+ - Make sure, that offset committing happens before the `consumer.consumed` event is propagated
209
+ - Fix for failing when not installed (just a dependency) (#817)
210
+ - Evict messages from partitions that were lost upon rebalancing (#825)
211
+ - Do **not** run `#revoked` on partitions that were lost and assigned back upon rebalancing (#825)
212
+ - Remove potential duplicated that could occur upon rebalance with re-assigned partitions (#825)
213
+ - Optimize integration test suite additional consumers shutdown process (#828)
214
+ - Optimize messages eviction and duplicates removal on poll stopped due to lack of messages
215
+ - Add static group membership integration spec
216
+
217
+ ## 2.0.0-alpha5 (2022-04-03)
218
+ - Rename StdoutListener to LoggerListener (#811)
219
+
220
+ ## 2.0.0-alpha4 (2022-03-20)
221
+ - Rails support without ActiveJob queue adapter usage (#805)
222
+
223
+ ## 2.0.0-alpha3 (2022-03-16)
224
+ - Restore 'app.initialized' state and add notification on it
225
+ - Fix the installation flow for Rails and add integration tests for this scenario
226
+ - Add more integration tests covering some edge cases
227
+
228
+ ## 2.0.0-alpha2 (2022-02-19)
229
+ - Require `kafka` keys to be symbols
230
+ - [Pro] Added ActiveJob Pro adapter
231
+ - Small updates to the license and docs
232
+
233
+ ## 2.0.0-alpha1 (2022-01-30)
234
+ - Change license to `LGPL-3.0`
235
+ - [Pro] Introduce a Pro subscription
236
+ - Switch from `ruby-kafka` to `librdkafka` as an underlying driver
237
+ - Introduce fully automatic integration tests that go through the whole server lifecycle
238
+ - Integrate WaterDrop tightly with autoconfiguration inheritance and an option to redefine it
239
+ - Multi-threaded support for concurrent jobs consumption (when in separate topics and/or partitions)
240
+ - Introduce subscriptions groups concept for better resources management
241
+ - Remove completely all the callbacks in favour of finalizer method `#on_shutdown`
242
+ - Provide `on_revoked` method for taking actions upon topic revoke
243
+ - Remove single message consumption mode in favour of documentation on how to do it easily by yourself
244
+ - Provide sync and async offset management with async preferred
245
+ - Introduce seamless Ruby on Rails integration via `Rails::Railte`
246
+ - Update `cli info` to reflect the `2.0` details
247
+ - Remove responders in favour of WaterDrop `2.0` producer
248
+ - Remove pidfiles support
249
+ - Remove daemonization support
250
+ - Stop validating `kafka` configuration beyond minimum as it is handled by `librdkafka`
251
+ - Remove topics mappers concept
252
+ - Reorganize monitoring to match new concepts
253
+ - Notify on fatal worker processing errors
254
+ - Rename `Karafka::Params::BatchMetadata` to `Karafka::Messages::BatchMetadata`
255
+ - Rename `Karafka::Params::Params` to `Karafka::Messages::Message`
256
+ - Rename `#params_batch` in consumers to `#messages`
257
+ - Rename `Karafka::Params::Metadata` to `Karafka::Messages::Metadata`
258
+ - Allow for processing work of multiple consumer groups by the same worker poll
259
+ - Rename `Karafka::Fetcher` to `Karafka::Runner` and align notifications key names
260
+ - Update install templates
261
+ - `sidekiq-backend` is no longer supported
262
+ - `testing` gem for RSpec has been updated
263
+ - `WaterDrop` `2.1+` support
264
+ - Simple routing style (`0.5`) now builds a single consumer group instead of one per topic
265
+ - Example apps were updated
266
+ - Hook for underlying statistics emitted from librdkafka have been added
267
+ - Hook for underlying async errors emitted from librdkafka have been added
268
+ - ActiveJob Rails adapter
269
+ - Added benchmarks that can be used to profile Karafka
270
+ - Standardize error hook for all error reporting
10
271
 
11
272
  ## 1.4.11 (2021-12-04)
12
273
  - Source code metadata url added to the gemspec
data/CONTRIBUTING.md CHANGED
@@ -1,41 +1,32 @@
1
- # Contribute
1
+ # Contributing
2
2
 
3
3
  ## Introduction
4
4
 
5
- First, thank you for considering contributing to karafka! It's people like you that make the open source community such a great community! 😊
5
+ First, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community! 😊
6
6
 
7
7
  We welcome any type of contribution, not only code. You can help with:
8
8
  - **QA**: file bug reports, the more details you can give the better (e.g. screenshots with the console open)
9
9
  - **Marketing**: writing blog posts, howto's, printing stickers, ...
10
10
  - **Community**: presenting the project at meetups, organizing a dedicated meetup for the local community, ...
11
- - **Code**: take a look at the [open issues](issues). Even if you can't write code, commenting on them, showing that you care about a given issue matters. It helps us triage them.
12
-
13
- ## Your First Contribution
14
-
15
- Working on your first Pull Request? You can learn how from this *free* series, [How to Contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github).
11
+ - **Code**: take a look at the [open issues](https://github.com/karafka/karafka/issues). Even if you can't write code, commenting on them, showing that you care about a given issue matters. It helps us triage them.
16
12
 
17
13
  ## Submitting code
18
14
 
19
15
  Any code change should be submitted as a pull request. The description should explain what the code does and give steps to execute it. The pull request should also contain tests.
20
16
 
21
- ## Code review process
17
+ ### Code review process
22
18
 
23
19
  Each pull request must pass all the rspec specs and meet our quality requirements.
24
20
 
25
21
  To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
26
22
 
27
- Coditsu will automatically check your work against our quality standards. You can find your commit check results on the [builds page](https://app.coditsu.io/karafka/commit_builds) of Karafka organization.
28
-
29
- [![coditsu](https://coditsu.io/assets/quality_bar.svg)](https://app.coditsu.io/karafka/commit_builds)
23
+ ### Contributing to Pro components
30
24
 
31
- ## Questions
32
-
33
- If you have any questions, create an [issue](issue) (protip: do a quick search first to see if someone else didn't ask the same question before!).
34
- You can also reach us at hello@karafka.opencollective.com.
25
+ All of Karafka components are open-source. However, the `Pro` components are licenses under `LICENSE-COMM`.
35
26
 
36
- ## Credits
27
+ By sending a pull request to the pro components, you are agreeing to transfer the copyright of your code to Maciej Mensfeld.
37
28
 
38
- ### Contributors
29
+ ## Questions
39
30
 
40
- Thank you to all the people who have already contributed to karafka!
41
- <a href="graphs/contributors"><img src="https://opencollective.com/karafka/contributors.svg?width=890" /></a>
31
+ If you have any questions, create an [issue](https://github.com/karafka/karafka/issues) (protip: do a quick search first to see if someone else didn't ask the same question before!).
32
+ You can also reach us at contact@karafka.io.
data/Gemfile CHANGED
@@ -6,6 +6,12 @@ plugin 'diffend'
6
6
 
7
7
  gemspec
8
8
 
9
+ # Karafka gem does not require this but we add it here so we can test the integration with
10
+ # ActiveJob much easier
11
+ group :integrations do
12
+ gem 'activejob'
13
+ end
14
+
9
15
  group :test do
10
16
  gem 'byebug'
11
17
  gem 'factory_bot'