waterdrop 2.6.14 → 2.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/.github/ISSUE_TEMPLATE/bug_report.md +43 -0
  4. data/.github/ISSUE_TEMPLATE/feature_request.md +20 -0
  5. data/.github/workflows/ci.yml +11 -19
  6. data/.ruby-version +1 -1
  7. data/CHANGELOG.md +256 -0
  8. data/Gemfile +1 -0
  9. data/Gemfile.lock +35 -31
  10. data/LICENSE +17 -0
  11. data/README.md +4 -4
  12. data/bin/verify_kafka_warnings +31 -0
  13. data/certs/cert.pem +26 -0
  14. data/config/locales/errors.yml +17 -3
  15. data/docker-compose.yml +1 -3
  16. data/lib/waterdrop/clients/rdkafka.rb +34 -4
  17. data/lib/waterdrop/config.rb +27 -12
  18. data/lib/waterdrop/contracts/config.rb +12 -1
  19. data/lib/waterdrop/contracts/variant.rb +107 -0
  20. data/lib/waterdrop/errors.rb +16 -0
  21. data/lib/waterdrop/instrumentation/callbacks/delivery.rb +11 -0
  22. data/lib/waterdrop/instrumentation/callbacks/error.rb +11 -0
  23. data/lib/waterdrop/instrumentation/callbacks/oauthbearer_token_refresh.rb +49 -0
  24. data/lib/waterdrop/instrumentation/callbacks/statistics.rb +11 -0
  25. data/lib/waterdrop/instrumentation/logger_listener.rb +11 -3
  26. data/lib/waterdrop/instrumentation/notifications.rb +3 -0
  27. data/lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb +10 -1
  28. data/lib/waterdrop/producer/buffer.rb +0 -6
  29. data/lib/waterdrop/producer/sync.rb +33 -9
  30. data/lib/waterdrop/producer/transactions.rb +98 -19
  31. data/lib/waterdrop/producer/variant.rb +109 -0
  32. data/lib/waterdrop/producer.rb +155 -92
  33. data/lib/waterdrop/version.rb +1 -1
  34. data/lib/waterdrop.rb +1 -0
  35. data/waterdrop.gemspec +7 -4
  36. data.tar.gz.sig +0 -0
  37. metadata +51 -33
  38. metadata.gz.sig +0 -0
  39. data/MIT-LICENSE +0 -18
  40. data/certs/cert_chain.pem +0 -26
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 24dc1ffc8d6298980ec8f0c302141a643196acbf62da017cf03ec6675552532c
4
- data.tar.gz: '08b66f7abb7f2e04fb9a9b5da566f5cefb5a00769ef14c14898f86c07003f81e'
3
+ metadata.gz: f1dd1ecba0be29bdf637575f0fe8eaea654900f7466d5840c62fbcdeb5000492
4
+ data.tar.gz: 663b683a670ec1df3700dd3440c80435b376c493bcdb7d024eedcc9a9c7b0de9
5
5
  SHA512:
6
- metadata.gz: 65d1a0b2ce58fa07edfa96dffaa97773424a53787ee05ba981ca1ed9e4edb91d7f9b25e4dbc5e054ac68abf288dd5e1a879cb09ee13fcacc9422d9a341f4bd81
7
- data.tar.gz: 2c7ce204ab9c9af43c5e916143aa5375e45157f1fd9c34acf69dc3191a646c3d41c230af82f6d1b8047836f2b7d4437d9a2cf3c173cd8beb9a7c1948c652cd54
6
+ metadata.gz: 8ff3c1d142dd9d855c7a26223ceefa973764dfde23b0ec8530999e87d52b7827b7180b30dd1f9f545a65eb7e1b00a8ddf753baa4fa365b93ec997450491ddc9b
7
+ data.tar.gz: 0fee34d4935ee757fef55ae9205570894b20894b839a1e6abd80fa918eb7d733603a0c791b3d81c73fd913699f07a1ddf1d3af9921722713accdb8bc90a17255
checksums.yaml.gz.sig CHANGED
Binary file
@@ -0,0 +1,43 @@
1
+ ---
2
+ name: Bug Report
3
+ about: Report an issue within the Karafka ecosystem you've discovered.
4
+ ---
5
+
6
+ To make this process smoother for everyone involved, please read the following information before filling out the template.
7
+
8
+ Scope of the OSS Support
9
+ ===========
10
+
11
+ We do not provide OSS support for outdated versions of Karafka and its components.
12
+
13
+ Please ensure that you are using a version that is still actively supported. We cannot assist with any no longer maintained versions unless you support us with our Pro offering (https://karafka.io/docs/Pro-Support/).
14
+
15
+ We acknowledge that understanding the specifics of your application and its configuration can be essential for resolving certain issues. However, due to the extensive time and resources such analysis can require, this may fall beyond our Open Source Support scope.
16
+
17
+ If Karafka or its components are critical to your infrastructure, we encourage you to consider our Pro Offering.
18
+
19
+ By backing us up, you can gain direct assistance and ensure your use case receives the dedicated attention it deserves.
20
+
21
+
22
+ Important Links to Read
23
+ ===========
24
+
25
+ Please take a moment to review the following resources before submitting your report:
26
+
27
+ - Issue Reporting Guide: https://karafka.io/docs/Support/#issue-reporting-guide
28
+ - Support Policy: https://karafka.io/docs/Support/
29
+ - Versions, Lifecycle, and EOL: https://karafka.io/docs/Versions-Lifecycle-and-EOL/
30
+
31
+
32
+ Bug Report Details
33
+ ===========
34
+
35
+ Please provide all the details per our Issue Reporting Guide: https://karafka.io/docs/Support/#issue-reporting-guide
36
+
37
+ Failing to provide the required details may result in the issue being closed. Please include all necessary information to help us understand and resolve your issue effectively.
38
+
39
+
40
+ Additional Context
41
+ ===========
42
+
43
+ Add any other context about the problem here.
@@ -0,0 +1,20 @@
1
+ ---
2
+ name: Feature Request
3
+ about: Suggest new WaterDrop features or improvements to existing features.
4
+ ---
5
+
6
+ ## Is your feature request related to a problem? Please describe.
7
+
8
+ A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
9
+
10
+ ## Describe the solution you'd like
11
+
12
+ A clear and concise description of what you want to happen.
13
+
14
+ ## Describe alternatives you've considered
15
+
16
+ A clear and concise description of any alternative solutions or features you've considered.
17
+
18
+ ## Additional context
19
+
20
+ Add any other context or screenshots about the feature request here.
@@ -18,13 +18,12 @@ jobs:
18
18
  fail-fast: false
19
19
  matrix:
20
20
  ruby:
21
+ - '3.4'
21
22
  - '3.3'
22
23
  - '3.2'
23
24
  - '3.1'
24
- - '3.0'
25
- - '2.7'
26
25
  include:
27
- - ruby: '3.3'
26
+ - ruby: '3.4'
28
27
  coverage: 'true'
29
28
  steps:
30
29
  - uses: actions/checkout@v4
@@ -39,9 +38,9 @@ jobs:
39
38
  bundler-cache: true
40
39
  bundler: 'latest'
41
40
 
42
- - name: Run Kafka with docker-compose
41
+ - name: Run Kafka with docker compose
43
42
  run: |
44
- docker-compose up -d || (sleep 5 && docker-compose up -d)
43
+ docker compose up -d || (sleep 5 && docker compose up -d)
45
44
 
46
45
  - name: Wait for Kafka
47
46
  run: |
@@ -49,31 +48,24 @@ jobs:
49
48
 
50
49
  - name: Install latest bundler
51
50
  run: |
52
- if [[ "$(ruby -v | awk '{print $2}')" == 2.7.8* ]]; then
53
- gem install bundler -v 2.4.22 --no-document
54
- gem update --system 3.4.22 --no-document
55
- else
56
- gem install bundler --no-document
57
- gem update --system --no-document
58
- fi
51
+ gem install bundler --no-document
52
+ gem update --system --no-document
59
53
 
60
54
  bundle config set without 'tools benchmarks docs'
61
55
 
62
56
  - name: Bundle install
63
57
  run: |
64
58
  bundle config set without development
65
-
66
- if [[ "$(ruby -v | awk '{print $2}')" == 2.7.8* ]]; then
67
- BUNDLER_VERSION=2.4.22 bundle install --jobs 4 --retry 3
68
- else
69
- bundle install --jobs 4 --retry 3
70
- fi
59
+ bundle install --jobs 4 --retry 3
71
60
 
72
61
  - name: Run all tests
73
62
  env:
74
63
  GITHUB_COVERAGE: ${{matrix.coverage}}
75
64
  run: bundle exec rspec
76
65
 
66
+ - name: Check Kafka logs for unexpected warnings
67
+ run: bin/verify_kafka_warnings
68
+
77
69
  diffend:
78
70
  runs-on: ubuntu-latest
79
71
  strategy:
@@ -85,7 +77,7 @@ jobs:
85
77
  - name: Set up Ruby
86
78
  uses: ruby/setup-ruby@v1
87
79
  with:
88
- ruby-version: 3.3
80
+ ruby-version: 3.4
89
81
  - name: Install latest bundler
90
82
  run: gem install bundler --no-document
91
83
  - name: Install Diffend plugin
data/.ruby-version CHANGED
@@ -1 +1 @@
1
- 3.3.0
1
+ 3.4.1
data/CHANGELOG.md CHANGED
@@ -1,5 +1,261 @@
1
1
  # WaterDrop changelog
2
2
 
3
+ ## 2.8.2 (2025-02-13)
4
+ - [Feature] Allow for tagging of producer instances similar to how consumers can be tagged.
5
+ - [Refactor] Ensure all test topics in the test suite start with "it-" prefix.
6
+
7
+ ## 2.8.1 (2024-12-26)
8
+ - [Enhancement] Raise `WaterDrop::ProducerNotTransactionalError` when attempting to use transactions on a non-transactional producer.
9
+ - [Fix] Disallow closing a producer from within a transaction.
10
+ - [Fix] WaterDrop should prevent opening a transaction using a closed producer.
11
+
12
+ ## 2.8.0 (2024-09-16)
13
+
14
+ This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
15
+
16
+ - **[Breaking]** Require Ruby `3.1+`.
17
+ - **[Breaking]** Remove ability to abort transactions using `throw(:abort)`. Please use `raise WaterDrop::Errors::AbortTransaction`.
18
+ - **[Breaking]** Disallow (similar to ActiveRecord) exiting transactions with `return`, `break` or `throw`.
19
+ - **[Breaking]** License changed from MIT to LGPL with an additional commercial option. Note: there is no commercial code in this repository. The commercial license is available for companies unable to use LGPL-licensed software for legal reasons.
20
+ - [Enhancement] Make variants fiber safe.
21
+ - [Enhancement] In transactional mode do not return any `dispatched` messages as none will be dispatched due to rollback.
22
+ - [Enhancement] Align the `LoggerListener` async messages to reflect, that messages are delegated to the internal queue and not dispatched.
23
+ - [Fix] Ensure, that `:dispatched` key for `#produce_many_sync` always contains delivery handles (final) and not delivery reports.
24
+
25
+ ### Upgrade Notes
26
+
27
+ **PLEASE MAKE SURE TO READ AND APPLY THEM!**
28
+
29
+ #### `throw(:abort)` No Longer Allowed To Abort Transactions
30
+
31
+ Replace:
32
+
33
+ ```ruby
34
+ producer.transaction do
35
+ messages.each do |message|
36
+ # Pipe all events
37
+ producer.produce_async(topic: 'events', payload: message.raw_payload)
38
+ end
39
+
40
+ # And abort if more events are no longer needed
41
+ throw(:abort) if KnowledgeBase.more_events_needed?
42
+ end
43
+ ```
44
+
45
+ With:
46
+
47
+ ```ruby
48
+ producer.transaction do
49
+ messages.each do |message|
50
+ # Pipe all events
51
+ producer.produce_async(topic: 'events', payload: message.raw_payload)
52
+ end
53
+
54
+ # And abort if more events are no longer needed
55
+ raise(WaterDrop::AbortTransaction) if KnowledgeBase.more_events_needed?
56
+ end
57
+ ```
58
+
59
+ #### `return`, `break` and `throw` No Longer Allowed Inside Transaction Block
60
+
61
+ Previously, transactions would abort if you exited early using `return`, `break`, or `throw`. This could create unexpected behavior, where users might not notice the rollback or have different intentions. For example, the following would trigger a rollback:
62
+
63
+ ```ruby
64
+ MAX = 10
65
+
66
+ def process(messages)
67
+ count = 0
68
+
69
+ producer.transaction do
70
+ messages.each do |message|
71
+ count += 1
72
+
73
+ producer.produce_async(topic: 'events', payload: message.raw_payload)
74
+
75
+ # This would trigger a rollback.
76
+ return if count >= MAX
77
+ end
78
+ end
79
+ end
80
+ ```
81
+
82
+ This is a source of errors, hence such exits are no longer allowed. You can implement similar flow control inside of your methods that are wrapped in a WaterDrop transaction:
83
+
84
+ ```ruby
85
+ MAX = 10
86
+
87
+ def process(messages)
88
+ producer.transaction do
89
+ # Early return from this method will not affect the transaction.
90
+ # It will be committed
91
+ insert_with_limit(messages)
92
+ end
93
+ end
94
+
95
+ def insert_with_limit(messages)
96
+ count = 0
97
+
98
+ messages.each do |message|
99
+ count += 1
100
+
101
+ producer.produce_async(topic: 'events', payload: message.raw_payload)
102
+
103
+ # This would trigger a rollback.
104
+ return if count >= MAX
105
+ end
106
+ end
107
+ ```
108
+
109
+ ## 2.7.4 (2024-07-04)
110
+ - [Maintenance] Alias `WaterDrop::Errors::AbortTransaction` with `WaterDrop::AbortTransaction`.
111
+ - [Maintenance] Lower the precision reporting to 100 microseconds in the logger listener.
112
+ - [Fix] Consumer consuming error: Local: Erroneous state (state) post break flow in transaction.
113
+ - [Change] Require 'karafka-core' `>= 2.4.3`
114
+
115
+ ## 2.7.3 (2024-06-09)
116
+ - [Enhancement] Introduce `reload_on_transaction_fatal_error` to reload the librdkafka after transactional failures
117
+ - [Enhancement] Flush on fatal transactional errors.
118
+ - [Enhancement] Add topic scope to `report_metric` (YadhuPrakash)
119
+ - [Enhancement] Cache middleware reference saving 1 object allocation on each message dispatch.
120
+ - [Enhancement] Provide `#idempotent?` similar to `#transactional?`.
121
+ - [Enhancement] Provide alias to `#with` named `#variant`.
122
+ - [Fix] Prevent from creating `acks` altering variants on idempotent producers.
123
+
124
+ ## 2.7.2 (2024-05-09)
125
+ - [Fix] Fix missing requirement of `delegate` for non-Rails use-cases. Always require delegate for variants usage (samsm)
126
+
127
+ ## 2.7.1 (2024-05-09)
128
+ - **[Feature]** Support context-base configuration with low-level topic settings alterations producer variants.
129
+ - [Enhancement] Prefix random default `SecureRandom.hex(6)` producers ids with `waterdrop-hex` to indicate type of object.
130
+
131
+ ## 2.7.0 (2024-04-26)
132
+
133
+ This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
134
+
135
+ - **[Feature]** Support custom OAuth providers.
136
+ - **[Breaking]** Drop Ruby `2.7` support.
137
+ - **[Breaking]** Change default timeouts so final delivery `message.timeout.ms` is less that `max_wait_time` so we do not end up with not final verdict.
138
+ - **[Breaking]** Update all the time related configuration settings to be in `ms` and not mixed.
139
+ - **[Breaking]** Remove no longer needed `wait_timeout` configuration option.
140
+ - **[Breaking]** Do **not** validate or morph (via middleware) messages added to the buffer prior to `flush_sync` or `flush_async`.
141
+ - [Enhancement] Provide `WaterDrop::Producer#transaction?` that returns only when producer has an active transaction running.
142
+ - [Enhancement] Introduce `instrument_on_wait_queue_full` flag (defaults to `true`) to be able to configure whether non critical (retryable) queue full errors should be instrumented in the error pipeline. Useful when building high-performance pipes with WaterDrop queue retry backoff as a throttler.
143
+ - [Enhancement] Protect critical `rdkafka` thread executable code sections.
144
+ - [Enhancement] Treat the queue size as a gauge rather than a cumulative stat (isturdy).
145
+ - [Fix] Fix a case where purge on non-initialized client would crash.
146
+ - [Fix] Middlewares run twice when using buffered produce.
147
+ - [Fix] Validations run twice when using buffered produce.
148
+
149
+ ### Upgrade Notes
150
+
151
+ **PLEASE MAKE SURE TO READ AND APPLY THEM!**
152
+
153
+ #### `wait_timeout` Configuration No Longer Needed
154
+
155
+ The `wait_timeout` WaterDrop configuration option is no longer needed. You can safely remove it.
156
+
157
+ ```ruby
158
+ producer = WaterDrop::Producer.new
159
+
160
+ producer.setup do |config|
161
+ # Other config...
162
+
163
+ # Remove this, no longer needed
164
+ config.wait_timeout = 30
165
+ end
166
+ ```
167
+
168
+ #### Time Settings Format Alignment
169
+
170
+ **All** time-related values are now configured in milliseconds instead of some being in seconds and some in milliseconds.
171
+
172
+ The values that were changed from seconds to milliseconds are:
173
+
174
+ - `max_wait_timeout`
175
+ - `wait_backoff_on_queue_full`
176
+ - `wait_timeout_on_queue_full`
177
+ - `wait_backoff_on_transaction_command, default`
178
+
179
+ If you have configured any of those yourself, please replace the seconds representation with milliseconds:
180
+
181
+ ```ruby
182
+ producer = WaterDrop::Producer.new
183
+
184
+ producer.setup do |config|
185
+ config.deliver = true
186
+
187
+ # Replace this:
188
+ config.max_wait_timeout = 30
189
+
190
+ # With
191
+ config.max_wait_timeout = 30_000
192
+ # ...
193
+ end
194
+ ```
195
+
196
+ #### Defaults Alignment
197
+
198
+ In this release, we've updated our default settings to address a crucial issue: previous defaults could lead to inconclusive outcomes in synchronous operations due to wait timeout errors. Users often mistakenly believed that a message dispatch was halted because of these errors when, in fact, the timeout was related to awaiting the final dispatch verdict, not the dispatch action itself.
199
+
200
+ The new defaults in WaterDrop 2.7.0 eliminate this confusion by ensuring synchronous operation results are always transparent and conclusive. This change aims to provide a straightforward understanding of wait timeout errors, reinforcing that they reflect the wait state, not the dispatch success.
201
+
202
+ Below, you can find a table with what has changed, the new defaults, and the current ones in case you want to retain the previous behavior:
203
+
204
+ <table>
205
+ <thead>
206
+ <tr>
207
+ <th>Config</th>
208
+ <th>Previous Default</th>
209
+ <th>New Default</th>
210
+ </tr>
211
+ </thead>
212
+ <tbody>
213
+ <tr>
214
+ <td>root <code>max_wait_timeout</code></td>
215
+ <td>5000 ms (5 seconds)</td>
216
+ <td>60000 ms (60 seconds)</td>
217
+ </tr>
218
+ <tr>
219
+ <td>kafka <code>message.timeout.ms</code></td>
220
+ <td>300000 ms (5 minutes)</td>
221
+ <td>50000 ms (50 seconds)</td>
222
+ </tr>
223
+ <tr>
224
+ <td>kafka <code>transaction.timeout.ms</code></td>
225
+ <td>60000 ms (1 minute)</td>
226
+ <td>55000 ms (55 seconds)</td>
227
+ </tr>
228
+ </tbody>
229
+ </table>
230
+
231
+ This alignment ensures that when using sync operations or invoking `#wait`, any exception you get should give you a conclusive and final delivery verdict.
232
+
233
+ #### Buffering No Longer Early Validates Messages
234
+
235
+ As of version `2.7.0`, WaterDrop has changed how message buffering works. Previously, messages underwent validation and middleware processing when they were buffered. Now, these steps are deferred until just before dispatching the messages. The buffer functions strictly as a thread-safe storage area without performing any validations or middleware operations until the messages are ready to be sent.
236
+
237
+ This adjustment was made primarily to ensure that middleware runs and validations are applied when most relevant—shortly before message dispatch. This approach addresses potential issues with buffers that might hold messages for extended periods:
238
+
239
+ - **Temporal Relevance**: Validating and processing messages near their dispatch time helps ensure that actions such as partition assignments reflect the current system state. This is crucial in dynamic environments where system states are subject to rapid changes.
240
+
241
+ - **Stale State Management**: By delaying validations and middleware to the dispatch phase, the system minimizes the risk of acting on outdated information, which could lead to incorrect processing or partitioning decisions.
242
+
243
+ ```ruby
244
+ # Prior to 2.7.0 this would raise an error
245
+ producer.buffer(topic: nil, payload: '')
246
+ # => WaterDrop::Errors::MessageInvalidError
247
+
248
+ # After 2.7.0 buffer will not, but flush_async will
249
+ producer.buffer(topic: nil, payload: '')
250
+ # => all good here
251
+ producer.flush_async(topic: nil, payload: '')
252
+ # => WaterDrop::Errors::MessageInvalidError
253
+ ```
254
+
255
+ #### Middleware Execution Prior to Flush When Buffering
256
+
257
+ The timing of middleware execution has been adjusted. Middleware, which was previously run when messages were added to the buffer, will now only execute immediately before the messages are flushed from the buffer and dispatched. This change is similar to the validation-related changes.
258
+
3
259
  ## 2.6.14 (2024-02-06)
4
260
  - [Enhancement] Instrument `producer.connected` and `producer.closing` lifecycle events.
5
261
 
data/Gemfile CHANGED
@@ -12,6 +12,7 @@ end
12
12
 
13
13
  group :test do
14
14
  gem 'factory_bot'
15
+ gem 'ostruct'
15
16
  gem 'rspec'
16
17
  gem 'simplecov'
17
18
  end
data/Gemfile.lock CHANGED
@@ -1,71 +1,74 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- waterdrop (2.6.14)
5
- karafka-core (>= 2.2.3, < 3.0.0)
4
+ waterdrop (2.8.2)
5
+ karafka-core (>= 2.4.3, < 3.0.0)
6
+ karafka-rdkafka (>= 0.17.5)
6
7
  zeitwerk (~> 2.3)
7
8
 
8
9
  GEM
9
10
  remote: https://rubygems.org/
10
11
  specs:
11
- activesupport (7.1.3)
12
+ activesupport (7.2.2.1)
12
13
  base64
14
+ benchmark (>= 0.3)
13
15
  bigdecimal
14
- concurrent-ruby (~> 1.0, >= 1.0.2)
16
+ concurrent-ruby (~> 1.0, >= 1.3.1)
15
17
  connection_pool (>= 2.2.5)
16
18
  drb
17
19
  i18n (>= 1.6, < 2)
20
+ logger (>= 1.4.2)
18
21
  minitest (>= 5.1)
19
- mutex_m
20
- tzinfo (~> 2.0)
22
+ securerandom (>= 0.3)
23
+ tzinfo (~> 2.0, >= 2.0.5)
21
24
  base64 (0.2.0)
22
- bigdecimal (3.1.6)
25
+ benchmark (0.4.0)
26
+ bigdecimal (3.1.9)
23
27
  byebug (11.1.3)
24
- concurrent-ruby (1.2.3)
28
+ concurrent-ruby (1.3.5)
25
29
  connection_pool (2.4.1)
26
30
  diff-lcs (1.5.1)
27
- docile (1.4.0)
28
- drb (2.2.0)
29
- ruby2_keywords
30
- factory_bot (6.4.5)
31
- activesupport (>= 5.0.0)
32
- ffi (1.16.3)
33
- i18n (1.14.1)
31
+ docile (1.4.1)
32
+ drb (2.2.1)
33
+ factory_bot (6.5.1)
34
+ activesupport (>= 6.1.0)
35
+ ffi (1.17.0)
36
+ i18n (1.14.7)
34
37
  concurrent-ruby (~> 1.0)
35
- karafka-core (2.2.7)
36
- concurrent-ruby (>= 1.1)
37
- karafka-rdkafka (>= 0.13.9, < 0.15.0)
38
- karafka-rdkafka (0.14.7)
38
+ karafka-core (2.4.4)
39
+ karafka-rdkafka (>= 0.15.0, < 0.18.0)
40
+ karafka-rdkafka (0.17.6)
39
41
  ffi (~> 1.15)
40
42
  mini_portile2 (~> 2.6)
41
43
  rake (> 12)
42
- mini_portile2 (2.8.5)
43
- minitest (5.21.2)
44
- mutex_m (0.2.0)
45
- rake (13.1.0)
44
+ logger (1.6.5)
45
+ mini_portile2 (2.8.7)
46
+ minitest (5.25.4)
47
+ ostruct (0.6.1)
48
+ rake (13.2.1)
46
49
  rspec (3.13.0)
47
50
  rspec-core (~> 3.13.0)
48
51
  rspec-expectations (~> 3.13.0)
49
52
  rspec-mocks (~> 3.13.0)
50
- rspec-core (3.13.0)
53
+ rspec-core (3.13.1)
51
54
  rspec-support (~> 3.13.0)
52
- rspec-expectations (3.13.0)
55
+ rspec-expectations (3.13.3)
53
56
  diff-lcs (>= 1.2.0, < 2.0)
54
57
  rspec-support (~> 3.13.0)
55
- rspec-mocks (3.13.0)
58
+ rspec-mocks (3.13.1)
56
59
  diff-lcs (>= 1.2.0, < 2.0)
57
60
  rspec-support (~> 3.13.0)
58
- rspec-support (3.13.0)
59
- ruby2_keywords (0.0.5)
61
+ rspec-support (3.13.1)
62
+ securerandom (0.3.2)
60
63
  simplecov (0.22.0)
61
64
  docile (~> 1.1)
62
65
  simplecov-html (~> 0.11)
63
66
  simplecov_json_formatter (~> 0.1)
64
- simplecov-html (0.12.3)
67
+ simplecov-html (0.13.1)
65
68
  simplecov_json_formatter (0.1.4)
66
69
  tzinfo (2.0.6)
67
70
  concurrent-ruby (~> 1.0)
68
- zeitwerk (2.6.12)
71
+ zeitwerk (2.6.18)
69
72
 
70
73
  PLATFORMS
71
74
  ruby
@@ -74,9 +77,10 @@ PLATFORMS
74
77
  DEPENDENCIES
75
78
  byebug
76
79
  factory_bot
80
+ ostruct
77
81
  rspec
78
82
  simplecov
79
83
  waterdrop!
80
84
 
81
85
  BUNDLED WITH
82
- 2.5.4
86
+ 2.5.16
data/LICENSE ADDED
@@ -0,0 +1,17 @@
1
+ Copyright (c) Maciej Mensfeld
2
+
3
+ WaterDrop is part of Karafka and it is an Open Source project licensed under the terms of
4
+ the LGPLv3 license. Please see <https://github.com/karafka/karafka/blob/master/LICENSE-LGPL>
5
+ for license text.
6
+
7
+ Karafka ecosystem can also be used under commercial-friendly license, with commercial support and commercial components.
8
+
9
+ All of the commercial components are present in the lib/karafka/pro and lib/karafka/web/pro
10
+ directory of this repository and their usage requires commercial license agreement.
11
+
12
+ By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
13
+ code to Maciej Mensfeld.
14
+
15
+ You can find the commercial license in LICENSE-COMM <https://github.com/karafka/karafka/blob/master/LICENSE-COMM>.
16
+
17
+ Please see https://karafka.io for purchasing options.
data/README.md CHANGED
@@ -9,13 +9,13 @@ WaterDrop is a standalone gem that sends messages to Kafka easily with an extra
9
9
  It:
10
10
 
11
11
  - Is thread-safe
12
- - Supports sync producing
13
- - Supports async producing
12
+ - Supports sync and async producing
14
13
  - Supports transactions
15
14
  - Supports buffering
16
- - Supports producing messages to multiple clusters
15
+ - Supports producing to multiple clusters
17
16
  - Supports multiple delivery policies
18
- - Works with Kafka `1.0+` and Ruby `2.7+`
17
+ - Supports per-topic configuration alterations (variants)
18
+ - Works with Kafka `1.0+` and Ruby `3.1+`
19
19
  - Works with and without Karafka
20
20
 
21
21
  ## Documentation
@@ -0,0 +1,31 @@
1
+ #!/bin/bash
2
+
3
+ # Checks Kafka logs for unsupported warning patterns
4
+ # Only specified warnings are allowed, all others should trigger failure
5
+
6
+ allowed_patterns=(
7
+ "Performing controller activation"
8
+ "registered with feature metadata.version"
9
+ )
10
+
11
+ # Get all warnings
12
+ warnings=$(docker logs --since=0 kafka | grep WARN)
13
+ exit_code=0
14
+
15
+ while IFS= read -r line; do
16
+ allowed=0
17
+ for pattern in "${allowed_patterns[@]}"; do
18
+ if echo "$line" | grep -q "$pattern"; then
19
+ allowed=1
20
+ break
21
+ fi
22
+ done
23
+
24
+ if [ $allowed -eq 0 ]; then
25
+ echo "Unexpected warning found:"
26
+ echo "$line"
27
+ exit_code=1
28
+ fi
29
+ done <<< "$warnings"
30
+
31
+ exit $exit_code
data/certs/cert.pem ADDED
@@ -0,0 +1,26 @@
1
+ -----BEGIN CERTIFICATE-----
2
+ MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
3
+ YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
4
+ MB4XDTI0MDgyMzEwMTkyMFoXDTQ5MDgxNzEwMTkyMFowPzEQMA4GA1UEAwwHY29u
5
+ dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
6
+ bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAKjLhLjQqUlNayxkXnO+
7
+ PsmCDs/KFIzhrsYMfLZRZNaWmzV3ujljMOdDjd4snM2X06C41iVdQPWjpe3j8vVe
8
+ ZXEWR/twSbOP6Eeg8WVH2wCOo0x5i7yhVn4UBLH4JpfEMCbemVcWQ9ry9OMg4WpH
9
+ Uu4dRwxFV7hzCz3p0QfNLRI4miAxnGWcnlD98IJRjBAksTuR1Llj0vbOrDGsL9ZT
10
+ JeXP2gdRLd8SqzAFJEWrbeTBCBU7gfSh3oMg5SVDLjaqf7Kz5wC/8bDZydzanOxB
11
+ T6CDXPsCnllmvTNx2ei2T5rGYJOzJeNTmJLLK6hJWUlAvaQSvCwZRvFJ0tVGLEoS
12
+ flqSr6uGyyl1eMUsNmsH4BqPEYcAV6P2PKTv2vUR8AP0raDvZ3xL1TKvfRb8xRpo
13
+ vPopCGlY5XBWEc6QERHfVLTIVsjnls2/Ujj4h8/TSfqqYnaHKefIMLbuD/tquMjD
14
+ iWQsW2qStBV0T+U7FijKxVfrfqZP7GxQmDAc9o1iiyAa3QIDAQABo3cwdTAJBgNV
15
+ HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQU3O4dTXmvE7YpAkszGzR9DdL9
16
+ sbEwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
17
+ bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAVKTfoLXn7mqdSxIR
18
+ eqxcR6Huudg1jes81s1+X0uiRTR3hxxKZ3Y82cPsee9zYWyBrN8TA4KA0WILTru7
19
+ Ygxvzha0SRPsSiaKLmgOJ+61ebI4+bOORzIJLpD6GxCxu1r7MI4+0r1u1xe0EWi8
20
+ agkVo1k4Vi8cKMLm6Gl9b3wG9zQBw6fcgKwmpjKiNnOLP+OytzUANrIUJjoq6oal
21
+ TC+f/Uc0TLaRqUaW/bejxzDWWHoM3SU6aoLPuerglzp9zZVzihXwx3jPLUVKDFpF
22
+ Rl2lcBDxlpYGueGo0/oNzGJAAy6js8jhtHC9+19PD53vk7wHtFTZ/0ugDQYnwQ+x
23
+ oml2fAAuVWpTBCgOVFe6XCQpMKopzoxQ1PjKztW2KYxgJdIBX87SnL3aWuBQmhRd
24
+ i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
25
+ ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
26
+ -----END CERTIFICATE-----
@@ -4,15 +4,29 @@ en:
4
4
  missing: must be present
5
5
  logger_format: must be present
6
6
  deliver_format: must be boolean
7
+ instrument_on_wait_queue_full_format: must be boolean
7
8
  id_format: must be a non-empty string
9
+ monitor_format: must be present
10
+ client_class_format: must be present
8
11
  max_payload_size_format: must be an integer that is equal or bigger than 1
9
- wait_timeout_format: must be a numeric that is bigger than 0
10
12
  max_wait_timeout_format: must be an integer that is equal or bigger than 0
11
13
  kafka_format: must be a hash with symbol based keys
12
14
  kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
13
15
  wait_on_queue_full_format: must be boolean
14
- wait_backoff_on_queue_full_format: must be a numeric that is bigger or equal to 0
15
- wait_timeout_on_queue_full_format: must be a numeric that is bigger or equal to 0
16
+ wait_backoff_on_queue_full_format: must be a numeric that is equal or bigger to 0
17
+ wait_timeout_on_queue_full_format: must be a numeric that is equal or bigger to 0
18
+ wait_backoff_on_transaction_command_format: must be a numeric that is equal or bigger to 0
19
+ max_attempts_on_transaction_command_format: must be an integer that is equal or bigger than 1
20
+ reload_on_transaction_fatal_error_format: must be boolean
21
+ oauth.token_provider_listener_format: 'must be false or respond to #on_oauthbearer_token_refresh'
22
+
23
+ variant:
24
+ missing: must be present
25
+ default_format: must be boolean
26
+ max_wait_timeout_format: must be an integer that is equal or bigger than 0
27
+ kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
28
+ kafka_key_not_per_topic: This config option cannot be set on a per topic basis
29
+ kafka_key_acks_not_changeable: Acks value cannot be redefined for a transactional or idempotent producer
16
30
 
17
31
  message:
18
32
  missing: must be present