waterdrop 2.6.14 → 2.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/.github/ISSUE_TEMPLATE/bug_report.md +43 -0
- data/.github/ISSUE_TEMPLATE/feature_request.md +20 -0
- data/.github/workflows/ci.yml +11 -19
- data/.ruby-version +1 -1
- data/CHANGELOG.md +256 -0
- data/Gemfile +1 -0
- data/Gemfile.lock +35 -31
- data/LICENSE +17 -0
- data/README.md +4 -4
- data/bin/verify_kafka_warnings +31 -0
- data/certs/cert.pem +26 -0
- data/config/locales/errors.yml +17 -3
- data/docker-compose.yml +1 -3
- data/lib/waterdrop/clients/rdkafka.rb +34 -4
- data/lib/waterdrop/config.rb +27 -12
- data/lib/waterdrop/contracts/config.rb +12 -1
- data/lib/waterdrop/contracts/variant.rb +107 -0
- data/lib/waterdrop/errors.rb +16 -0
- data/lib/waterdrop/instrumentation/callbacks/delivery.rb +11 -0
- data/lib/waterdrop/instrumentation/callbacks/error.rb +11 -0
- data/lib/waterdrop/instrumentation/callbacks/oauthbearer_token_refresh.rb +49 -0
- data/lib/waterdrop/instrumentation/callbacks/statistics.rb +11 -0
- data/lib/waterdrop/instrumentation/logger_listener.rb +11 -3
- data/lib/waterdrop/instrumentation/notifications.rb +3 -0
- data/lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb +10 -1
- data/lib/waterdrop/producer/buffer.rb +0 -6
- data/lib/waterdrop/producer/sync.rb +33 -9
- data/lib/waterdrop/producer/transactions.rb +98 -19
- data/lib/waterdrop/producer/variant.rb +109 -0
- data/lib/waterdrop/producer.rb +155 -92
- data/lib/waterdrop/version.rb +1 -1
- data/lib/waterdrop.rb +1 -0
- data/waterdrop.gemspec +7 -4
- data.tar.gz.sig +0 -0
- metadata +51 -33
- metadata.gz.sig +0 -0
- data/MIT-LICENSE +0 -18
- data/certs/cert_chain.pem +0 -26
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f1dd1ecba0be29bdf637575f0fe8eaea654900f7466d5840c62fbcdeb5000492
|
|
4
|
+
data.tar.gz: 663b683a670ec1df3700dd3440c80435b376c493bcdb7d024eedcc9a9c7b0de9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8ff3c1d142dd9d855c7a26223ceefa973764dfde23b0ec8530999e87d52b7827b7180b30dd1f9f545a65eb7e1b00a8ddf753baa4fa365b93ec997450491ddc9b
|
|
7
|
+
data.tar.gz: 0fee34d4935ee757fef55ae9205570894b20894b839a1e6abd80fa918eb7d733603a0c791b3d81c73fd913699f07a1ddf1d3af9921722713accdb8bc90a17255
|
checksums.yaml.gz.sig
CHANGED
|
Binary file
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Bug Report
|
|
3
|
+
about: Report an issue within the Karafka ecosystem you've discovered.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
To make this process smoother for everyone involved, please read the following information before filling out the template.
|
|
7
|
+
|
|
8
|
+
Scope of the OSS Support
|
|
9
|
+
===========
|
|
10
|
+
|
|
11
|
+
We do not provide OSS support for outdated versions of Karafka and its components.
|
|
12
|
+
|
|
13
|
+
Please ensure that you are using a version that is still actively supported. We cannot assist with any no longer maintained versions unless you support us with our Pro offering (https://karafka.io/docs/Pro-Support/).
|
|
14
|
+
|
|
15
|
+
We acknowledge that understanding the specifics of your application and its configuration can be essential for resolving certain issues. However, due to the extensive time and resources such analysis can require, this may fall beyond our Open Source Support scope.
|
|
16
|
+
|
|
17
|
+
If Karafka or its components are critical to your infrastructure, we encourage you to consider our Pro Offering.
|
|
18
|
+
|
|
19
|
+
By backing us up, you can gain direct assistance and ensure your use case receives the dedicated attention it deserves.
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
Important Links to Read
|
|
23
|
+
===========
|
|
24
|
+
|
|
25
|
+
Please take a moment to review the following resources before submitting your report:
|
|
26
|
+
|
|
27
|
+
- Issue Reporting Guide: https://karafka.io/docs/Support/#issue-reporting-guide
|
|
28
|
+
- Support Policy: https://karafka.io/docs/Support/
|
|
29
|
+
- Versions, Lifecycle, and EOL: https://karafka.io/docs/Versions-Lifecycle-and-EOL/
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
Bug Report Details
|
|
33
|
+
===========
|
|
34
|
+
|
|
35
|
+
Please provide all the details per our Issue Reporting Guide: https://karafka.io/docs/Support/#issue-reporting-guide
|
|
36
|
+
|
|
37
|
+
Failing to provide the required details may result in the issue being closed. Please include all necessary information to help us understand and resolve your issue effectively.
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
Additional Context
|
|
41
|
+
===========
|
|
42
|
+
|
|
43
|
+
Add any other context about the problem here.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Feature Request
|
|
3
|
+
about: Suggest new WaterDrop features or improvements to existing features.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## Is your feature request related to a problem? Please describe.
|
|
7
|
+
|
|
8
|
+
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
|
|
9
|
+
|
|
10
|
+
## Describe the solution you'd like
|
|
11
|
+
|
|
12
|
+
A clear and concise description of what you want to happen.
|
|
13
|
+
|
|
14
|
+
## Describe alternatives you've considered
|
|
15
|
+
|
|
16
|
+
A clear and concise description of any alternative solutions or features you've considered.
|
|
17
|
+
|
|
18
|
+
## Additional context
|
|
19
|
+
|
|
20
|
+
Add any other context or screenshots about the feature request here.
|
data/.github/workflows/ci.yml
CHANGED
|
@@ -18,13 +18,12 @@ jobs:
|
|
|
18
18
|
fail-fast: false
|
|
19
19
|
matrix:
|
|
20
20
|
ruby:
|
|
21
|
+
- '3.4'
|
|
21
22
|
- '3.3'
|
|
22
23
|
- '3.2'
|
|
23
24
|
- '3.1'
|
|
24
|
-
- '3.0'
|
|
25
|
-
- '2.7'
|
|
26
25
|
include:
|
|
27
|
-
- ruby: '3.
|
|
26
|
+
- ruby: '3.4'
|
|
28
27
|
coverage: 'true'
|
|
29
28
|
steps:
|
|
30
29
|
- uses: actions/checkout@v4
|
|
@@ -39,9 +38,9 @@ jobs:
|
|
|
39
38
|
bundler-cache: true
|
|
40
39
|
bundler: 'latest'
|
|
41
40
|
|
|
42
|
-
- name: Run Kafka with docker
|
|
41
|
+
- name: Run Kafka with docker compose
|
|
43
42
|
run: |
|
|
44
|
-
docker
|
|
43
|
+
docker compose up -d || (sleep 5 && docker compose up -d)
|
|
45
44
|
|
|
46
45
|
- name: Wait for Kafka
|
|
47
46
|
run: |
|
|
@@ -49,31 +48,24 @@ jobs:
|
|
|
49
48
|
|
|
50
49
|
- name: Install latest bundler
|
|
51
50
|
run: |
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
gem update --system 3.4.22 --no-document
|
|
55
|
-
else
|
|
56
|
-
gem install bundler --no-document
|
|
57
|
-
gem update --system --no-document
|
|
58
|
-
fi
|
|
51
|
+
gem install bundler --no-document
|
|
52
|
+
gem update --system --no-document
|
|
59
53
|
|
|
60
54
|
bundle config set without 'tools benchmarks docs'
|
|
61
55
|
|
|
62
56
|
- name: Bundle install
|
|
63
57
|
run: |
|
|
64
58
|
bundle config set without development
|
|
65
|
-
|
|
66
|
-
if [[ "$(ruby -v | awk '{print $2}')" == 2.7.8* ]]; then
|
|
67
|
-
BUNDLER_VERSION=2.4.22 bundle install --jobs 4 --retry 3
|
|
68
|
-
else
|
|
69
|
-
bundle install --jobs 4 --retry 3
|
|
70
|
-
fi
|
|
59
|
+
bundle install --jobs 4 --retry 3
|
|
71
60
|
|
|
72
61
|
- name: Run all tests
|
|
73
62
|
env:
|
|
74
63
|
GITHUB_COVERAGE: ${{matrix.coverage}}
|
|
75
64
|
run: bundle exec rspec
|
|
76
65
|
|
|
66
|
+
- name: Check Kafka logs for unexpected warnings
|
|
67
|
+
run: bin/verify_kafka_warnings
|
|
68
|
+
|
|
77
69
|
diffend:
|
|
78
70
|
runs-on: ubuntu-latest
|
|
79
71
|
strategy:
|
|
@@ -85,7 +77,7 @@ jobs:
|
|
|
85
77
|
- name: Set up Ruby
|
|
86
78
|
uses: ruby/setup-ruby@v1
|
|
87
79
|
with:
|
|
88
|
-
ruby-version: 3.
|
|
80
|
+
ruby-version: 3.4
|
|
89
81
|
- name: Install latest bundler
|
|
90
82
|
run: gem install bundler --no-document
|
|
91
83
|
- name: Install Diffend plugin
|
data/.ruby-version
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
3.
|
|
1
|
+
3.4.1
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,261 @@
|
|
|
1
1
|
# WaterDrop changelog
|
|
2
2
|
|
|
3
|
+
## 2.8.2 (2025-02-13)
|
|
4
|
+
- [Feature] Allow for tagging of producer instances similar to how consumers can be tagged.
|
|
5
|
+
- [Refactor] Ensure all test topics in the test suite start with "it-" prefix.
|
|
6
|
+
|
|
7
|
+
## 2.8.1 (2024-12-26)
|
|
8
|
+
- [Enhancement] Raise `WaterDrop::ProducerNotTransactionalError` when attempting to use transactions on a non-transactional producer.
|
|
9
|
+
- [Fix] Disallow closing a producer from within a transaction.
|
|
10
|
+
- [Fix] WaterDrop should prevent opening a transaction using a closed producer.
|
|
11
|
+
|
|
12
|
+
## 2.8.0 (2024-09-16)
|
|
13
|
+
|
|
14
|
+
This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
|
|
15
|
+
|
|
16
|
+
- **[Breaking]** Require Ruby `3.1+`.
|
|
17
|
+
- **[Breaking]** Remove ability to abort transactions using `throw(:abort)`. Please use `raise WaterDrop::Errors::AbortTransaction`.
|
|
18
|
+
- **[Breaking]** Disallow (similar to ActiveRecord) exiting transactions with `return`, `break` or `throw`.
|
|
19
|
+
- **[Breaking]** License changed from MIT to LGPL with an additional commercial option. Note: there is no commercial code in this repository. The commercial license is available for companies unable to use LGPL-licensed software for legal reasons.
|
|
20
|
+
- [Enhancement] Make variants fiber safe.
|
|
21
|
+
- [Enhancement] In transactional mode do not return any `dispatched` messages as none will be dispatched due to rollback.
|
|
22
|
+
- [Enhancement] Align the `LoggerListener` async messages to reflect, that messages are delegated to the internal queue and not dispatched.
|
|
23
|
+
- [Fix] Ensure, that `:dispatched` key for `#produce_many_sync` always contains delivery handles (final) and not delivery reports.
|
|
24
|
+
|
|
25
|
+
### Upgrade Notes
|
|
26
|
+
|
|
27
|
+
**PLEASE MAKE SURE TO READ AND APPLY THEM!**
|
|
28
|
+
|
|
29
|
+
#### `throw(:abort)` No Longer Allowed To Abort Transactions
|
|
30
|
+
|
|
31
|
+
Replace:
|
|
32
|
+
|
|
33
|
+
```ruby
|
|
34
|
+
producer.transaction do
|
|
35
|
+
messages.each do |message|
|
|
36
|
+
# Pipe all events
|
|
37
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# And abort if more events are no longer needed
|
|
41
|
+
throw(:abort) if KnowledgeBase.more_events_needed?
|
|
42
|
+
end
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
With:
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
producer.transaction do
|
|
49
|
+
messages.each do |message|
|
|
50
|
+
# Pipe all events
|
|
51
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# And abort if more events are no longer needed
|
|
55
|
+
raise(WaterDrop::AbortTransaction) if KnowledgeBase.more_events_needed?
|
|
56
|
+
end
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
#### `return`, `break` and `throw` No Longer Allowed Inside Transaction Block
|
|
60
|
+
|
|
61
|
+
Previously, transactions would abort if you exited early using `return`, `break`, or `throw`. This could create unexpected behavior, where users might not notice the rollback or have different intentions. For example, the following would trigger a rollback:
|
|
62
|
+
|
|
63
|
+
```ruby
|
|
64
|
+
MAX = 10
|
|
65
|
+
|
|
66
|
+
def process(messages)
|
|
67
|
+
count = 0
|
|
68
|
+
|
|
69
|
+
producer.transaction do
|
|
70
|
+
messages.each do |message|
|
|
71
|
+
count += 1
|
|
72
|
+
|
|
73
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
|
74
|
+
|
|
75
|
+
# This would trigger a rollback.
|
|
76
|
+
return if count >= MAX
|
|
77
|
+
end
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
This is a source of errors, hence such exits are no longer allowed. You can implement similar flow control inside of your methods that are wrapped in a WaterDrop transaction:
|
|
83
|
+
|
|
84
|
+
```ruby
|
|
85
|
+
MAX = 10
|
|
86
|
+
|
|
87
|
+
def process(messages)
|
|
88
|
+
producer.transaction do
|
|
89
|
+
# Early return from this method will not affect the transaction.
|
|
90
|
+
# It will be committed
|
|
91
|
+
insert_with_limit(messages)
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
def insert_with_limit(messages)
|
|
96
|
+
count = 0
|
|
97
|
+
|
|
98
|
+
messages.each do |message|
|
|
99
|
+
count += 1
|
|
100
|
+
|
|
101
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
|
102
|
+
|
|
103
|
+
# This would trigger a rollback.
|
|
104
|
+
return if count >= MAX
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
## 2.7.4 (2024-07-04)
|
|
110
|
+
- [Maintenance] Alias `WaterDrop::Errors::AbortTransaction` with `WaterDrop::AbortTransaction`.
|
|
111
|
+
- [Maintenance] Lower the precision reporting to 100 microseconds in the logger listener.
|
|
112
|
+
- [Fix] Consumer consuming error: Local: Erroneous state (state) post break flow in transaction.
|
|
113
|
+
- [Change] Require 'karafka-core' `>= 2.4.3`
|
|
114
|
+
|
|
115
|
+
## 2.7.3 (2024-06-09)
|
|
116
|
+
- [Enhancement] Introduce `reload_on_transaction_fatal_error` to reload the librdkafka after transactional failures
|
|
117
|
+
- [Enhancement] Flush on fatal transactional errors.
|
|
118
|
+
- [Enhancement] Add topic scope to `report_metric` (YadhuPrakash)
|
|
119
|
+
- [Enhancement] Cache middleware reference saving 1 object allocation on each message dispatch.
|
|
120
|
+
- [Enhancement] Provide `#idempotent?` similar to `#transactional?`.
|
|
121
|
+
- [Enhancement] Provide alias to `#with` named `#variant`.
|
|
122
|
+
- [Fix] Prevent from creating `acks` altering variants on idempotent producers.
|
|
123
|
+
|
|
124
|
+
## 2.7.2 (2024-05-09)
|
|
125
|
+
- [Fix] Fix missing requirement of `delegate` for non-Rails use-cases. Always require delegate for variants usage (samsm)
|
|
126
|
+
|
|
127
|
+
## 2.7.1 (2024-05-09)
|
|
128
|
+
- **[Feature]** Support context-base configuration with low-level topic settings alterations producer variants.
|
|
129
|
+
- [Enhancement] Prefix random default `SecureRandom.hex(6)` producers ids with `waterdrop-hex` to indicate type of object.
|
|
130
|
+
|
|
131
|
+
## 2.7.0 (2024-04-26)
|
|
132
|
+
|
|
133
|
+
This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
|
|
134
|
+
|
|
135
|
+
- **[Feature]** Support custom OAuth providers.
|
|
136
|
+
- **[Breaking]** Drop Ruby `2.7` support.
|
|
137
|
+
- **[Breaking]** Change default timeouts so final delivery `message.timeout.ms` is less that `max_wait_time` so we do not end up with not final verdict.
|
|
138
|
+
- **[Breaking]** Update all the time related configuration settings to be in `ms` and not mixed.
|
|
139
|
+
- **[Breaking]** Remove no longer needed `wait_timeout` configuration option.
|
|
140
|
+
- **[Breaking]** Do **not** validate or morph (via middleware) messages added to the buffer prior to `flush_sync` or `flush_async`.
|
|
141
|
+
- [Enhancement] Provide `WaterDrop::Producer#transaction?` that returns only when producer has an active transaction running.
|
|
142
|
+
- [Enhancement] Introduce `instrument_on_wait_queue_full` flag (defaults to `true`) to be able to configure whether non critical (retryable) queue full errors should be instrumented in the error pipeline. Useful when building high-performance pipes with WaterDrop queue retry backoff as a throttler.
|
|
143
|
+
- [Enhancement] Protect critical `rdkafka` thread executable code sections.
|
|
144
|
+
- [Enhancement] Treat the queue size as a gauge rather than a cumulative stat (isturdy).
|
|
145
|
+
- [Fix] Fix a case where purge on non-initialized client would crash.
|
|
146
|
+
- [Fix] Middlewares run twice when using buffered produce.
|
|
147
|
+
- [Fix] Validations run twice when using buffered produce.
|
|
148
|
+
|
|
149
|
+
### Upgrade Notes
|
|
150
|
+
|
|
151
|
+
**PLEASE MAKE SURE TO READ AND APPLY THEM!**
|
|
152
|
+
|
|
153
|
+
#### `wait_timeout` Configuration No Longer Needed
|
|
154
|
+
|
|
155
|
+
The `wait_timeout` WaterDrop configuration option is no longer needed. You can safely remove it.
|
|
156
|
+
|
|
157
|
+
```ruby
|
|
158
|
+
producer = WaterDrop::Producer.new
|
|
159
|
+
|
|
160
|
+
producer.setup do |config|
|
|
161
|
+
# Other config...
|
|
162
|
+
|
|
163
|
+
# Remove this, no longer needed
|
|
164
|
+
config.wait_timeout = 30
|
|
165
|
+
end
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
#### Time Settings Format Alignment
|
|
169
|
+
|
|
170
|
+
**All** time-related values are now configured in milliseconds instead of some being in seconds and some in milliseconds.
|
|
171
|
+
|
|
172
|
+
The values that were changed from seconds to milliseconds are:
|
|
173
|
+
|
|
174
|
+
- `max_wait_timeout`
|
|
175
|
+
- `wait_backoff_on_queue_full`
|
|
176
|
+
- `wait_timeout_on_queue_full`
|
|
177
|
+
- `wait_backoff_on_transaction_command, default`
|
|
178
|
+
|
|
179
|
+
If you have configured any of those yourself, please replace the seconds representation with milliseconds:
|
|
180
|
+
|
|
181
|
+
```ruby
|
|
182
|
+
producer = WaterDrop::Producer.new
|
|
183
|
+
|
|
184
|
+
producer.setup do |config|
|
|
185
|
+
config.deliver = true
|
|
186
|
+
|
|
187
|
+
# Replace this:
|
|
188
|
+
config.max_wait_timeout = 30
|
|
189
|
+
|
|
190
|
+
# With
|
|
191
|
+
config.max_wait_timeout = 30_000
|
|
192
|
+
# ...
|
|
193
|
+
end
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
#### Defaults Alignment
|
|
197
|
+
|
|
198
|
+
In this release, we've updated our default settings to address a crucial issue: previous defaults could lead to inconclusive outcomes in synchronous operations due to wait timeout errors. Users often mistakenly believed that a message dispatch was halted because of these errors when, in fact, the timeout was related to awaiting the final dispatch verdict, not the dispatch action itself.
|
|
199
|
+
|
|
200
|
+
The new defaults in WaterDrop 2.7.0 eliminate this confusion by ensuring synchronous operation results are always transparent and conclusive. This change aims to provide a straightforward understanding of wait timeout errors, reinforcing that they reflect the wait state, not the dispatch success.
|
|
201
|
+
|
|
202
|
+
Below, you can find a table with what has changed, the new defaults, and the current ones in case you want to retain the previous behavior:
|
|
203
|
+
|
|
204
|
+
<table>
|
|
205
|
+
<thead>
|
|
206
|
+
<tr>
|
|
207
|
+
<th>Config</th>
|
|
208
|
+
<th>Previous Default</th>
|
|
209
|
+
<th>New Default</th>
|
|
210
|
+
</tr>
|
|
211
|
+
</thead>
|
|
212
|
+
<tbody>
|
|
213
|
+
<tr>
|
|
214
|
+
<td>root <code>max_wait_timeout</code></td>
|
|
215
|
+
<td>5000 ms (5 seconds)</td>
|
|
216
|
+
<td>60000 ms (60 seconds)</td>
|
|
217
|
+
</tr>
|
|
218
|
+
<tr>
|
|
219
|
+
<td>kafka <code>message.timeout.ms</code></td>
|
|
220
|
+
<td>300000 ms (5 minutes)</td>
|
|
221
|
+
<td>50000 ms (50 seconds)</td>
|
|
222
|
+
</tr>
|
|
223
|
+
<tr>
|
|
224
|
+
<td>kafka <code>transaction.timeout.ms</code></td>
|
|
225
|
+
<td>60000 ms (1 minute)</td>
|
|
226
|
+
<td>55000 ms (55 seconds)</td>
|
|
227
|
+
</tr>
|
|
228
|
+
</tbody>
|
|
229
|
+
</table>
|
|
230
|
+
|
|
231
|
+
This alignment ensures that when using sync operations or invoking `#wait`, any exception you get should give you a conclusive and final delivery verdict.
|
|
232
|
+
|
|
233
|
+
#### Buffering No Longer Early Validates Messages
|
|
234
|
+
|
|
235
|
+
As of version `2.7.0`, WaterDrop has changed how message buffering works. Previously, messages underwent validation and middleware processing when they were buffered. Now, these steps are deferred until just before dispatching the messages. The buffer functions strictly as a thread-safe storage area without performing any validations or middleware operations until the messages are ready to be sent.
|
|
236
|
+
|
|
237
|
+
This adjustment was made primarily to ensure that middleware runs and validations are applied when most relevant—shortly before message dispatch. This approach addresses potential issues with buffers that might hold messages for extended periods:
|
|
238
|
+
|
|
239
|
+
- **Temporal Relevance**: Validating and processing messages near their dispatch time helps ensure that actions such as partition assignments reflect the current system state. This is crucial in dynamic environments where system states are subject to rapid changes.
|
|
240
|
+
|
|
241
|
+
- **Stale State Management**: By delaying validations and middleware to the dispatch phase, the system minimizes the risk of acting on outdated information, which could lead to incorrect processing or partitioning decisions.
|
|
242
|
+
|
|
243
|
+
```ruby
|
|
244
|
+
# Prior to 2.7.0 this would raise an error
|
|
245
|
+
producer.buffer(topic: nil, payload: '')
|
|
246
|
+
# => WaterDrop::Errors::MessageInvalidError
|
|
247
|
+
|
|
248
|
+
# After 2.7.0 buffer will not, but flush_async will
|
|
249
|
+
producer.buffer(topic: nil, payload: '')
|
|
250
|
+
# => all good here
|
|
251
|
+
producer.flush_async(topic: nil, payload: '')
|
|
252
|
+
# => WaterDrop::Errors::MessageInvalidError
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
#### Middleware Execution Prior to Flush When Buffering
|
|
256
|
+
|
|
257
|
+
The timing of middleware execution has been adjusted. Middleware, which was previously run when messages were added to the buffer, will now only execute immediately before the messages are flushed from the buffer and dispatched. This change is similar to the validation-related changes.
|
|
258
|
+
|
|
3
259
|
## 2.6.14 (2024-02-06)
|
|
4
260
|
- [Enhancement] Instrument `producer.connected` and `producer.closing` lifecycle events.
|
|
5
261
|
|
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
|
@@ -1,71 +1,74 @@
|
|
|
1
1
|
PATH
|
|
2
2
|
remote: .
|
|
3
3
|
specs:
|
|
4
|
-
waterdrop (2.
|
|
5
|
-
karafka-core (>= 2.
|
|
4
|
+
waterdrop (2.8.2)
|
|
5
|
+
karafka-core (>= 2.4.3, < 3.0.0)
|
|
6
|
+
karafka-rdkafka (>= 0.17.5)
|
|
6
7
|
zeitwerk (~> 2.3)
|
|
7
8
|
|
|
8
9
|
GEM
|
|
9
10
|
remote: https://rubygems.org/
|
|
10
11
|
specs:
|
|
11
|
-
activesupport (7.1
|
|
12
|
+
activesupport (7.2.2.1)
|
|
12
13
|
base64
|
|
14
|
+
benchmark (>= 0.3)
|
|
13
15
|
bigdecimal
|
|
14
|
-
concurrent-ruby (~> 1.0, >= 1.
|
|
16
|
+
concurrent-ruby (~> 1.0, >= 1.3.1)
|
|
15
17
|
connection_pool (>= 2.2.5)
|
|
16
18
|
drb
|
|
17
19
|
i18n (>= 1.6, < 2)
|
|
20
|
+
logger (>= 1.4.2)
|
|
18
21
|
minitest (>= 5.1)
|
|
19
|
-
|
|
20
|
-
tzinfo (~> 2.0)
|
|
22
|
+
securerandom (>= 0.3)
|
|
23
|
+
tzinfo (~> 2.0, >= 2.0.5)
|
|
21
24
|
base64 (0.2.0)
|
|
22
|
-
|
|
25
|
+
benchmark (0.4.0)
|
|
26
|
+
bigdecimal (3.1.9)
|
|
23
27
|
byebug (11.1.3)
|
|
24
|
-
concurrent-ruby (1.
|
|
28
|
+
concurrent-ruby (1.3.5)
|
|
25
29
|
connection_pool (2.4.1)
|
|
26
30
|
diff-lcs (1.5.1)
|
|
27
|
-
docile (1.4.
|
|
28
|
-
drb (2.2.
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
i18n (1.14.1)
|
|
31
|
+
docile (1.4.1)
|
|
32
|
+
drb (2.2.1)
|
|
33
|
+
factory_bot (6.5.1)
|
|
34
|
+
activesupport (>= 6.1.0)
|
|
35
|
+
ffi (1.17.0)
|
|
36
|
+
i18n (1.14.7)
|
|
34
37
|
concurrent-ruby (~> 1.0)
|
|
35
|
-
karafka-core (2.
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
karafka-rdkafka (0.14.7)
|
|
38
|
+
karafka-core (2.4.4)
|
|
39
|
+
karafka-rdkafka (>= 0.15.0, < 0.18.0)
|
|
40
|
+
karafka-rdkafka (0.17.6)
|
|
39
41
|
ffi (~> 1.15)
|
|
40
42
|
mini_portile2 (~> 2.6)
|
|
41
43
|
rake (> 12)
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
44
|
+
logger (1.6.5)
|
|
45
|
+
mini_portile2 (2.8.7)
|
|
46
|
+
minitest (5.25.4)
|
|
47
|
+
ostruct (0.6.1)
|
|
48
|
+
rake (13.2.1)
|
|
46
49
|
rspec (3.13.0)
|
|
47
50
|
rspec-core (~> 3.13.0)
|
|
48
51
|
rspec-expectations (~> 3.13.0)
|
|
49
52
|
rspec-mocks (~> 3.13.0)
|
|
50
|
-
rspec-core (3.13.
|
|
53
|
+
rspec-core (3.13.1)
|
|
51
54
|
rspec-support (~> 3.13.0)
|
|
52
|
-
rspec-expectations (3.13.
|
|
55
|
+
rspec-expectations (3.13.3)
|
|
53
56
|
diff-lcs (>= 1.2.0, < 2.0)
|
|
54
57
|
rspec-support (~> 3.13.0)
|
|
55
|
-
rspec-mocks (3.13.
|
|
58
|
+
rspec-mocks (3.13.1)
|
|
56
59
|
diff-lcs (>= 1.2.0, < 2.0)
|
|
57
60
|
rspec-support (~> 3.13.0)
|
|
58
|
-
rspec-support (3.13.
|
|
59
|
-
|
|
61
|
+
rspec-support (3.13.1)
|
|
62
|
+
securerandom (0.3.2)
|
|
60
63
|
simplecov (0.22.0)
|
|
61
64
|
docile (~> 1.1)
|
|
62
65
|
simplecov-html (~> 0.11)
|
|
63
66
|
simplecov_json_formatter (~> 0.1)
|
|
64
|
-
simplecov-html (0.
|
|
67
|
+
simplecov-html (0.13.1)
|
|
65
68
|
simplecov_json_formatter (0.1.4)
|
|
66
69
|
tzinfo (2.0.6)
|
|
67
70
|
concurrent-ruby (~> 1.0)
|
|
68
|
-
zeitwerk (2.6.
|
|
71
|
+
zeitwerk (2.6.18)
|
|
69
72
|
|
|
70
73
|
PLATFORMS
|
|
71
74
|
ruby
|
|
@@ -74,9 +77,10 @@ PLATFORMS
|
|
|
74
77
|
DEPENDENCIES
|
|
75
78
|
byebug
|
|
76
79
|
factory_bot
|
|
80
|
+
ostruct
|
|
77
81
|
rspec
|
|
78
82
|
simplecov
|
|
79
83
|
waterdrop!
|
|
80
84
|
|
|
81
85
|
BUNDLED WITH
|
|
82
|
-
2.5.
|
|
86
|
+
2.5.16
|
data/LICENSE
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
Copyright (c) Maciej Mensfeld
|
|
2
|
+
|
|
3
|
+
WaterDrop is part of Karafka and it is an Open Source project licensed under the terms of
|
|
4
|
+
the LGPLv3 license. Please see <https://github.com/karafka/karafka/blob/master/LICENSE-LGPL>
|
|
5
|
+
for license text.
|
|
6
|
+
|
|
7
|
+
Karafka ecosystem can also be used under commercial-friendly license, with commercial support and commercial components.
|
|
8
|
+
|
|
9
|
+
All of the commercial components are present in the lib/karafka/pro and lib/karafka/web/pro
|
|
10
|
+
directory of this repository and their usage requires commercial license agreement.
|
|
11
|
+
|
|
12
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
|
|
13
|
+
code to Maciej Mensfeld.
|
|
14
|
+
|
|
15
|
+
You can find the commercial license in LICENSE-COMM <https://github.com/karafka/karafka/blob/master/LICENSE-COMM>.
|
|
16
|
+
|
|
17
|
+
Please see https://karafka.io for purchasing options.
|
data/README.md
CHANGED
|
@@ -9,13 +9,13 @@ WaterDrop is a standalone gem that sends messages to Kafka easily with an extra
|
|
|
9
9
|
It:
|
|
10
10
|
|
|
11
11
|
- Is thread-safe
|
|
12
|
-
- Supports sync producing
|
|
13
|
-
- Supports async producing
|
|
12
|
+
- Supports sync and async producing
|
|
14
13
|
- Supports transactions
|
|
15
14
|
- Supports buffering
|
|
16
|
-
- Supports producing
|
|
15
|
+
- Supports producing to multiple clusters
|
|
17
16
|
- Supports multiple delivery policies
|
|
18
|
-
-
|
|
17
|
+
- Supports per-topic configuration alterations (variants)
|
|
18
|
+
- Works with Kafka `1.0+` and Ruby `3.1+`
|
|
19
19
|
- Works with and without Karafka
|
|
20
20
|
|
|
21
21
|
## Documentation
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
#!/bin/bash
|
|
2
|
+
|
|
3
|
+
# Checks Kafka logs for unsupported warning patterns
|
|
4
|
+
# Only specified warnings are allowed, all others should trigger failure
|
|
5
|
+
|
|
6
|
+
allowed_patterns=(
|
|
7
|
+
"Performing controller activation"
|
|
8
|
+
"registered with feature metadata.version"
|
|
9
|
+
)
|
|
10
|
+
|
|
11
|
+
# Get all warnings
|
|
12
|
+
warnings=$(docker logs --since=0 kafka | grep WARN)
|
|
13
|
+
exit_code=0
|
|
14
|
+
|
|
15
|
+
while IFS= read -r line; do
|
|
16
|
+
allowed=0
|
|
17
|
+
for pattern in "${allowed_patterns[@]}"; do
|
|
18
|
+
if echo "$line" | grep -q "$pattern"; then
|
|
19
|
+
allowed=1
|
|
20
|
+
break
|
|
21
|
+
fi
|
|
22
|
+
done
|
|
23
|
+
|
|
24
|
+
if [ $allowed -eq 0 ]; then
|
|
25
|
+
echo "Unexpected warning found:"
|
|
26
|
+
echo "$line"
|
|
27
|
+
exit_code=1
|
|
28
|
+
fi
|
|
29
|
+
done <<< "$warnings"
|
|
30
|
+
|
|
31
|
+
exit $exit_code
|
data/certs/cert.pem
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
-----BEGIN CERTIFICATE-----
|
|
2
|
+
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
|
3
|
+
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
|
4
|
+
MB4XDTI0MDgyMzEwMTkyMFoXDTQ5MDgxNzEwMTkyMFowPzEQMA4GA1UEAwwHY29u
|
|
5
|
+
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
|
6
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAKjLhLjQqUlNayxkXnO+
|
|
7
|
+
PsmCDs/KFIzhrsYMfLZRZNaWmzV3ujljMOdDjd4snM2X06C41iVdQPWjpe3j8vVe
|
|
8
|
+
ZXEWR/twSbOP6Eeg8WVH2wCOo0x5i7yhVn4UBLH4JpfEMCbemVcWQ9ry9OMg4WpH
|
|
9
|
+
Uu4dRwxFV7hzCz3p0QfNLRI4miAxnGWcnlD98IJRjBAksTuR1Llj0vbOrDGsL9ZT
|
|
10
|
+
JeXP2gdRLd8SqzAFJEWrbeTBCBU7gfSh3oMg5SVDLjaqf7Kz5wC/8bDZydzanOxB
|
|
11
|
+
T6CDXPsCnllmvTNx2ei2T5rGYJOzJeNTmJLLK6hJWUlAvaQSvCwZRvFJ0tVGLEoS
|
|
12
|
+
flqSr6uGyyl1eMUsNmsH4BqPEYcAV6P2PKTv2vUR8AP0raDvZ3xL1TKvfRb8xRpo
|
|
13
|
+
vPopCGlY5XBWEc6QERHfVLTIVsjnls2/Ujj4h8/TSfqqYnaHKefIMLbuD/tquMjD
|
|
14
|
+
iWQsW2qStBV0T+U7FijKxVfrfqZP7GxQmDAc9o1iiyAa3QIDAQABo3cwdTAJBgNV
|
|
15
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQU3O4dTXmvE7YpAkszGzR9DdL9
|
|
16
|
+
sbEwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
|
17
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAVKTfoLXn7mqdSxIR
|
|
18
|
+
eqxcR6Huudg1jes81s1+X0uiRTR3hxxKZ3Y82cPsee9zYWyBrN8TA4KA0WILTru7
|
|
19
|
+
Ygxvzha0SRPsSiaKLmgOJ+61ebI4+bOORzIJLpD6GxCxu1r7MI4+0r1u1xe0EWi8
|
|
20
|
+
agkVo1k4Vi8cKMLm6Gl9b3wG9zQBw6fcgKwmpjKiNnOLP+OytzUANrIUJjoq6oal
|
|
21
|
+
TC+f/Uc0TLaRqUaW/bejxzDWWHoM3SU6aoLPuerglzp9zZVzihXwx3jPLUVKDFpF
|
|
22
|
+
Rl2lcBDxlpYGueGo0/oNzGJAAy6js8jhtHC9+19PD53vk7wHtFTZ/0ugDQYnwQ+x
|
|
23
|
+
oml2fAAuVWpTBCgOVFe6XCQpMKopzoxQ1PjKztW2KYxgJdIBX87SnL3aWuBQmhRd
|
|
24
|
+
i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
|
|
25
|
+
ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
|
|
26
|
+
-----END CERTIFICATE-----
|
data/config/locales/errors.yml
CHANGED
|
@@ -4,15 +4,29 @@ en:
|
|
|
4
4
|
missing: must be present
|
|
5
5
|
logger_format: must be present
|
|
6
6
|
deliver_format: must be boolean
|
|
7
|
+
instrument_on_wait_queue_full_format: must be boolean
|
|
7
8
|
id_format: must be a non-empty string
|
|
9
|
+
monitor_format: must be present
|
|
10
|
+
client_class_format: must be present
|
|
8
11
|
max_payload_size_format: must be an integer that is equal or bigger than 1
|
|
9
|
-
wait_timeout_format: must be a numeric that is bigger than 0
|
|
10
12
|
max_wait_timeout_format: must be an integer that is equal or bigger than 0
|
|
11
13
|
kafka_format: must be a hash with symbol based keys
|
|
12
14
|
kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
|
|
13
15
|
wait_on_queue_full_format: must be boolean
|
|
14
|
-
wait_backoff_on_queue_full_format: must be a numeric that is
|
|
15
|
-
wait_timeout_on_queue_full_format: must be a numeric that is
|
|
16
|
+
wait_backoff_on_queue_full_format: must be a numeric that is equal or bigger to 0
|
|
17
|
+
wait_timeout_on_queue_full_format: must be a numeric that is equal or bigger to 0
|
|
18
|
+
wait_backoff_on_transaction_command_format: must be a numeric that is equal or bigger to 0
|
|
19
|
+
max_attempts_on_transaction_command_format: must be an integer that is equal or bigger than 1
|
|
20
|
+
reload_on_transaction_fatal_error_format: must be boolean
|
|
21
|
+
oauth.token_provider_listener_format: 'must be false or respond to #on_oauthbearer_token_refresh'
|
|
22
|
+
|
|
23
|
+
variant:
|
|
24
|
+
missing: must be present
|
|
25
|
+
default_format: must be boolean
|
|
26
|
+
max_wait_timeout_format: must be an integer that is equal or bigger than 0
|
|
27
|
+
kafka_key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
|
|
28
|
+
kafka_key_not_per_topic: This config option cannot be set on a per topic basis
|
|
29
|
+
kafka_key_acks_not_changeable: Acks value cannot be redefined for a transactional or idempotent producer
|
|
16
30
|
|
|
17
31
|
message:
|
|
18
32
|
missing: must be present
|