waterdrop 2.7.4 → 2.8.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/.github/workflows/ci.yml +0 -1
- data/.ruby-version +1 -1
- data/CHANGELOG.md +97 -0
- data/Gemfile.lock +21 -19
- data/LICENSE +17 -0
- data/README.md +1 -1
- data/certs/cert.pem +26 -0
- data/docker-compose.yml +1 -1
- data/lib/waterdrop/errors.rb +3 -0
- data/lib/waterdrop/instrumentation/logger_listener.rb +5 -2
- data/lib/waterdrop/producer/sync.rb +31 -7
- data/lib/waterdrop/producer/transactions.rb +21 -26
- data/lib/waterdrop/producer/variant.rb +3 -2
- data/lib/waterdrop/producer.rb +16 -6
- data/lib/waterdrop/version.rb +1 -1
- data/waterdrop.gemspec +4 -4
- data.tar.gz.sig +0 -0
- metadata +31 -30
- metadata.gz.sig +0 -0
- data/MIT-LICENSE +0 -18
- data/certs/cert_chain.pem +0 -26
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 96435533f730edae9dcf7bed5426641a33715e87b55c804168fa6a02465f1431
|
4
|
+
data.tar.gz: 3737e88ec3ffaa407b1cdaa523cf1d24a25aa85c4d90989a249381dbfb7a80c2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1b83e8505ca1aa9be5c137c06af1b98e10bb8f39af93e77aefa42302bf8dc2bd95f1442eb2569d5f032e80ac0374b8013e774d1f36494d23e20af0766094e2b3
|
7
|
+
data.tar.gz: 198ea59a4f7ca6a63f9816e7026c97152aaed5e6154142c47e6e96c16696a2a3d4c60cd56bf72b407b092682e9d15c08f1adfa552eb5bdc17a308f77f7424dfc
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/.github/workflows/ci.yml
CHANGED
data/.ruby-version
CHANGED
@@ -1 +1 @@
|
|
1
|
-
3.3.
|
1
|
+
3.3.5
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,102 @@
|
|
1
1
|
# WaterDrop changelog
|
2
2
|
|
3
|
+
## 2.8.0 (2024-09-16)
|
4
|
+
|
5
|
+
This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
|
6
|
+
|
7
|
+
- **[Breaking]** Require Ruby `3.1+`.
|
8
|
+
- **[Breaking]** Remove ability to abort transactions using `throw(:abort)`. Please use `raise WaterDrop::Errors::AbortTransaction`.
|
9
|
+
- **[Breaking]** Disallow (similar to ActiveRecord) exiting transactions with `return`, `break` or `throw`.
|
10
|
+
- **[Breaking]** License changed from MIT to LGPL with an additional commercial option. Note: there is no commercial code in this repository. The commercial license is available for companies unable to use LGPL-licensed software for legal reasons.
|
11
|
+
- [Enhancement] Make variants fiber safe.
|
12
|
+
- [Enhancement] In transactional mode do not return any `dispatched` messages as none will be dispatched due to rollback.
|
13
|
+
- [Enhancement] Align the `LoggerListener` async messages to reflect, that messages are delegated to the internal queue and not dispatched.
|
14
|
+
- [Fix] Ensure, that `:dispatched` key for `#produce_many_sync` always contains delivery handles (final) and not delivery reports.
|
15
|
+
|
16
|
+
### Upgrade Notes
|
17
|
+
|
18
|
+
**PLEASE MAKE SURE TO READ AND APPLY THEM!**
|
19
|
+
|
20
|
+
#### `throw(:abort)` No Longer Allowed To Abort Transactions
|
21
|
+
|
22
|
+
Replace:
|
23
|
+
|
24
|
+
```ruby
|
25
|
+
producer.transaction do
|
26
|
+
messages.each do |message|
|
27
|
+
# Pipe all events
|
28
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
29
|
+
end
|
30
|
+
|
31
|
+
# And abort if more events are no longer needed
|
32
|
+
throw(:abort) if KnowledgeBase.more_events_needed?
|
33
|
+
end
|
34
|
+
```
|
35
|
+
|
36
|
+
With:
|
37
|
+
|
38
|
+
```ruby
|
39
|
+
producer.transaction do
|
40
|
+
messages.each do |message|
|
41
|
+
# Pipe all events
|
42
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
43
|
+
end
|
44
|
+
|
45
|
+
# And abort if more events are no longer needed
|
46
|
+
raise(WaterDrop::AbortTransaction) if KnowledgeBase.more_events_needed?
|
47
|
+
end
|
48
|
+
```
|
49
|
+
|
50
|
+
#### `return`, `break` and `throw` No Longer Allowed Inside Transaction Block
|
51
|
+
|
52
|
+
Previously, transactions would abort if you exited early using `return`, `break`, or `throw`. This could create unexpected behavior, where users might not notice the rollback or have different intentions. For example, the following would trigger a rollback:
|
53
|
+
|
54
|
+
```ruby
|
55
|
+
MAX = 10
|
56
|
+
|
57
|
+
def process(messages)
|
58
|
+
count = 0
|
59
|
+
|
60
|
+
producer.transaction do
|
61
|
+
messages.each do |message|
|
62
|
+
count += 1
|
63
|
+
|
64
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
65
|
+
|
66
|
+
# This would trigger a rollback.
|
67
|
+
return if count >= MAX
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
```
|
72
|
+
|
73
|
+
This is a source of errors, hence such exits are no longer allowed. You can implement similar flow control inside of your methods that are wrapped in a WaterDrop transaction:
|
74
|
+
|
75
|
+
```ruby
|
76
|
+
MAX = 10
|
77
|
+
|
78
|
+
def process(messages)
|
79
|
+
producer.transaction do
|
80
|
+
# Early return from this method will not affect the transaction.
|
81
|
+
# It will be committed
|
82
|
+
insert_with_limit(messages)
|
83
|
+
end
|
84
|
+
end
|
85
|
+
|
86
|
+
def insert_with_limit(messages)
|
87
|
+
count = 0
|
88
|
+
|
89
|
+
messages.each do |message|
|
90
|
+
count += 1
|
91
|
+
|
92
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
93
|
+
|
94
|
+
# This would trigger a rollback.
|
95
|
+
return if count >= MAX
|
96
|
+
end
|
97
|
+
end
|
98
|
+
```
|
99
|
+
|
3
100
|
## 2.7.4 (2024-07-04)
|
4
101
|
- [Maintenance] Alias `WaterDrop::Errors::AbortTransaction` with `WaterDrop::AbortTransaction`.
|
5
102
|
- [Maintenance] Lower the precision reporting to 100 microseconds in the logger listener.
|
data/Gemfile.lock
CHANGED
@@ -1,70 +1,72 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
waterdrop (2.
|
4
|
+
waterdrop (2.8.0)
|
5
5
|
karafka-core (>= 2.4.3, < 3.0.0)
|
6
|
-
karafka-rdkafka (>= 0.
|
6
|
+
karafka-rdkafka (>= 0.17.5)
|
7
7
|
zeitwerk (~> 2.3)
|
8
8
|
|
9
9
|
GEM
|
10
10
|
remote: https://rubygems.org/
|
11
11
|
specs:
|
12
|
-
activesupport (7.1
|
12
|
+
activesupport (7.2.1)
|
13
13
|
base64
|
14
14
|
bigdecimal
|
15
|
-
concurrent-ruby (~> 1.0, >= 1.
|
15
|
+
concurrent-ruby (~> 1.0, >= 1.3.1)
|
16
16
|
connection_pool (>= 2.2.5)
|
17
17
|
drb
|
18
18
|
i18n (>= 1.6, < 2)
|
19
|
+
logger (>= 1.4.2)
|
19
20
|
minitest (>= 5.1)
|
20
|
-
|
21
|
-
tzinfo (~> 2.0)
|
21
|
+
securerandom (>= 0.3)
|
22
|
+
tzinfo (~> 2.0, >= 2.0.5)
|
22
23
|
base64 (0.2.0)
|
23
24
|
bigdecimal (3.1.8)
|
24
25
|
byebug (11.1.3)
|
25
|
-
concurrent-ruby (1.3.
|
26
|
+
concurrent-ruby (1.3.4)
|
26
27
|
connection_pool (2.4.1)
|
27
28
|
diff-lcs (1.5.1)
|
28
|
-
docile (1.4.
|
29
|
+
docile (1.4.1)
|
29
30
|
drb (2.2.1)
|
30
|
-
factory_bot (6.
|
31
|
+
factory_bot (6.5.0)
|
31
32
|
activesupport (>= 5.0.0)
|
32
33
|
ffi (1.17.0)
|
33
34
|
i18n (1.14.5)
|
34
35
|
concurrent-ruby (~> 1.0)
|
35
|
-
karafka-core (2.4.
|
36
|
-
karafka-rdkafka (>= 0.15.0, < 0.
|
37
|
-
karafka-rdkafka (0.
|
36
|
+
karafka-core (2.4.4)
|
37
|
+
karafka-rdkafka (>= 0.15.0, < 0.18.0)
|
38
|
+
karafka-rdkafka (0.17.6)
|
38
39
|
ffi (~> 1.15)
|
39
40
|
mini_portile2 (~> 2.6)
|
40
41
|
rake (> 12)
|
42
|
+
logger (1.6.1)
|
41
43
|
mini_portile2 (2.8.7)
|
42
|
-
minitest (5.
|
43
|
-
mutex_m (0.2.0)
|
44
|
+
minitest (5.25.1)
|
44
45
|
ostruct (0.6.0)
|
45
46
|
rake (13.2.1)
|
46
47
|
rspec (3.13.0)
|
47
48
|
rspec-core (~> 3.13.0)
|
48
49
|
rspec-expectations (~> 3.13.0)
|
49
50
|
rspec-mocks (~> 3.13.0)
|
50
|
-
rspec-core (3.13.
|
51
|
+
rspec-core (3.13.1)
|
51
52
|
rspec-support (~> 3.13.0)
|
52
|
-
rspec-expectations (3.13.
|
53
|
+
rspec-expectations (3.13.3)
|
53
54
|
diff-lcs (>= 1.2.0, < 2.0)
|
54
55
|
rspec-support (~> 3.13.0)
|
55
56
|
rspec-mocks (3.13.1)
|
56
57
|
diff-lcs (>= 1.2.0, < 2.0)
|
57
58
|
rspec-support (~> 3.13.0)
|
58
59
|
rspec-support (3.13.1)
|
60
|
+
securerandom (0.3.1)
|
59
61
|
simplecov (0.22.0)
|
60
62
|
docile (~> 1.1)
|
61
63
|
simplecov-html (~> 0.11)
|
62
64
|
simplecov_json_formatter (~> 0.1)
|
63
|
-
simplecov-html (0.
|
65
|
+
simplecov-html (0.13.1)
|
64
66
|
simplecov_json_formatter (0.1.4)
|
65
67
|
tzinfo (2.0.6)
|
66
68
|
concurrent-ruby (~> 1.0)
|
67
|
-
zeitwerk (2.6.
|
69
|
+
zeitwerk (2.6.18)
|
68
70
|
|
69
71
|
PLATFORMS
|
70
72
|
ruby
|
@@ -79,4 +81,4 @@ DEPENDENCIES
|
|
79
81
|
waterdrop!
|
80
82
|
|
81
83
|
BUNDLED WITH
|
82
|
-
2.5.
|
84
|
+
2.5.16
|
data/LICENSE
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
Copyright (c) Maciej Mensfeld
|
2
|
+
|
3
|
+
WaterDrop is part of Karafka and it is an Open Source project licensed under the terms of
|
4
|
+
the LGPLv3 license. Please see <https://github.com/karafka/karafka/blob/master/LICENSE-LGPL>
|
5
|
+
for license text.
|
6
|
+
|
7
|
+
Karafka ecosystem can also be used under commercial-friendly license, with commercial support and commercial components.
|
8
|
+
|
9
|
+
All of the commercial components are present in the lib/karafka/pro and lib/karafka/web/pro
|
10
|
+
directory of this repository and their usage requires commercial license agreement.
|
11
|
+
|
12
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
|
13
|
+
code to Maciej Mensfeld.
|
14
|
+
|
15
|
+
You can find the commercial license in LICENSE-COMM <https://github.com/karafka/karafka/blob/master/LICENSE-COMM>.
|
16
|
+
|
17
|
+
Please see https://karafka.io for purchasing options.
|
data/README.md
CHANGED
@@ -15,7 +15,7 @@ It:
|
|
15
15
|
- Supports producing to multiple clusters
|
16
16
|
- Supports multiple delivery policies
|
17
17
|
- Supports per-topic configuration alterations (variants)
|
18
|
-
- Works with Kafka `1.0+` and Ruby `3.
|
18
|
+
- Works with Kafka `1.0+` and Ruby `3.1+`
|
19
19
|
- Works with and without Karafka
|
20
20
|
|
21
21
|
## Documentation
|
data/certs/cert.pem
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
-----BEGIN CERTIFICATE-----
|
2
|
+
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
3
|
+
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
4
|
+
MB4XDTI0MDgyMzEwMTkyMFoXDTQ5MDgxNzEwMTkyMFowPzEQMA4GA1UEAwwHY29u
|
5
|
+
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
6
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAKjLhLjQqUlNayxkXnO+
|
7
|
+
PsmCDs/KFIzhrsYMfLZRZNaWmzV3ujljMOdDjd4snM2X06C41iVdQPWjpe3j8vVe
|
8
|
+
ZXEWR/twSbOP6Eeg8WVH2wCOo0x5i7yhVn4UBLH4JpfEMCbemVcWQ9ry9OMg4WpH
|
9
|
+
Uu4dRwxFV7hzCz3p0QfNLRI4miAxnGWcnlD98IJRjBAksTuR1Llj0vbOrDGsL9ZT
|
10
|
+
JeXP2gdRLd8SqzAFJEWrbeTBCBU7gfSh3oMg5SVDLjaqf7Kz5wC/8bDZydzanOxB
|
11
|
+
T6CDXPsCnllmvTNx2ei2T5rGYJOzJeNTmJLLK6hJWUlAvaQSvCwZRvFJ0tVGLEoS
|
12
|
+
flqSr6uGyyl1eMUsNmsH4BqPEYcAV6P2PKTv2vUR8AP0raDvZ3xL1TKvfRb8xRpo
|
13
|
+
vPopCGlY5XBWEc6QERHfVLTIVsjnls2/Ujj4h8/TSfqqYnaHKefIMLbuD/tquMjD
|
14
|
+
iWQsW2qStBV0T+U7FijKxVfrfqZP7GxQmDAc9o1iiyAa3QIDAQABo3cwdTAJBgNV
|
15
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQU3O4dTXmvE7YpAkszGzR9DdL9
|
16
|
+
sbEwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
17
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAVKTfoLXn7mqdSxIR
|
18
|
+
eqxcR6Huudg1jes81s1+X0uiRTR3hxxKZ3Y82cPsee9zYWyBrN8TA4KA0WILTru7
|
19
|
+
Ygxvzha0SRPsSiaKLmgOJ+61ebI4+bOORzIJLpD6GxCxu1r7MI4+0r1u1xe0EWi8
|
20
|
+
agkVo1k4Vi8cKMLm6Gl9b3wG9zQBw6fcgKwmpjKiNnOLP+OytzUANrIUJjoq6oal
|
21
|
+
TC+f/Uc0TLaRqUaW/bejxzDWWHoM3SU6aoLPuerglzp9zZVzihXwx3jPLUVKDFpF
|
22
|
+
Rl2lcBDxlpYGueGo0/oNzGJAAy6js8jhtHC9+19PD53vk7wHtFTZ/0ugDQYnwQ+x
|
23
|
+
oml2fAAuVWpTBCgOVFe6XCQpMKopzoxQ1PjKztW2KYxgJdIBX87SnL3aWuBQmhRd
|
24
|
+
i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
|
25
|
+
ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
|
26
|
+
-----END CERTIFICATE-----
|
data/docker-compose.yml
CHANGED
data/lib/waterdrop/errors.rb
CHANGED
@@ -47,6 +47,9 @@ module WaterDrop
|
|
47
47
|
# graceful transaction aborting
|
48
48
|
AbortTransaction = Class.new(BaseError)
|
49
49
|
|
50
|
+
# Do not use `break`, `return` or `throw` inside of the transaction blocks
|
51
|
+
EarlyTransactionExitNotAllowedError = Class.new(BaseError)
|
52
|
+
|
50
53
|
# Raised when during messages producing something bad happened inline
|
51
54
|
class ProduceManyError < ProduceError
|
52
55
|
attr_reader :dispatched
|
@@ -26,7 +26,7 @@ module WaterDrop
|
|
26
26
|
def on_message_produced_async(event)
|
27
27
|
message = event[:message]
|
28
28
|
|
29
|
-
info(event, "
|
29
|
+
info(event, "Message to '#{message[:topic]}' topic was delegated to a dispatch queue")
|
30
30
|
|
31
31
|
return unless log_messages?
|
32
32
|
|
@@ -49,7 +49,10 @@ module WaterDrop
|
|
49
49
|
messages = event[:messages]
|
50
50
|
topics_count = messages.map { |message| "'#{message[:topic]}'" }.uniq.count
|
51
51
|
|
52
|
-
info(
|
52
|
+
info(
|
53
|
+
event,
|
54
|
+
"#{messages.size} messages to #{topics_count} topics were delegated to a dispatch queue"
|
55
|
+
)
|
53
56
|
|
54
57
|
return unless log_messages?
|
55
58
|
|
@@ -61,17 +61,37 @@ module WaterDrop
|
|
61
61
|
messages.each { |message| validate_message!(message) }
|
62
62
|
|
63
63
|
dispatched = []
|
64
|
+
inline_error = nil
|
64
65
|
|
65
66
|
@monitor.instrument('messages.produced_sync', producer_id: id, messages: messages) do
|
66
|
-
|
67
|
-
|
68
|
-
|
67
|
+
# While most of the librdkafka errors are async and not inline, there are some like
|
68
|
+
# buffer overflow that can leak in during the `#produce` itself. When this happens, we
|
69
|
+
# still (since it's a sync mode) need to wait on deliveries of things that were
|
70
|
+
# successfully delegated to librdkafka. To do this, we catch the inline error and we
|
71
|
+
# wait on messages that were in the buffer to reach final state. Then if no error, we
|
72
|
+
# check each for error and if none all good. If there was an inline error, we re-raise
|
73
|
+
# it with the handlers in final states.
|
74
|
+
#
|
75
|
+
# Such flow ensures, that we never end up with handlers not being in the final states
|
76
|
+
# for the sync flow
|
77
|
+
begin
|
78
|
+
with_transaction_if_transactional do
|
79
|
+
messages.each do |message|
|
80
|
+
dispatched << produce(message)
|
81
|
+
end
|
69
82
|
end
|
83
|
+
rescue *SUPPORTED_FLOW_ERRORS => e
|
84
|
+
inline_error = e
|
70
85
|
end
|
71
86
|
|
72
|
-
|
73
|
-
|
74
|
-
|
87
|
+
# This will ensure, that we have all verdicts before raising the failure, so we pass
|
88
|
+
# all delivery handles having a final verdict
|
89
|
+
dispatched.each { |handler| wait(handler, raise_response_error: false) }
|
90
|
+
|
91
|
+
raise(inline_error) if inline_error
|
92
|
+
|
93
|
+
# This will raise an error on the first error that have happened
|
94
|
+
dispatched.each { |handler| wait(handler) }
|
75
95
|
|
76
96
|
dispatched
|
77
97
|
end
|
@@ -82,7 +102,11 @@ module WaterDrop
|
|
82
102
|
'error.occurred',
|
83
103
|
producer_id: id,
|
84
104
|
messages: messages,
|
85
|
-
dispatched
|
105
|
+
# If it is a transactional producer nothing was successfully dispatched on error, thus
|
106
|
+
# we never return any dispatched handlers. While those messages might have reached
|
107
|
+
# Kafka, in transactional mode they will not be visible to consumers with correct
|
108
|
+
# isolation level.
|
109
|
+
dispatched: transactional? ? EMPTY_ARRAY : dispatched,
|
86
110
|
error: re_raised,
|
87
111
|
type: 'messages.produce_many_sync'
|
88
112
|
)
|
@@ -35,7 +35,6 @@ module WaterDrop
|
|
35
35
|
# with a transaction. One transaction per single dispatch and for `produce_many` it will be
|
36
36
|
# a single transaction wrapping all messages dispatches (not one per message).
|
37
37
|
#
|
38
|
-
# @param block [Proc] block of code that should run
|
39
38
|
# @return Block result or `nil` in case of early break/return
|
40
39
|
#
|
41
40
|
# @example Simple transaction
|
@@ -46,7 +45,7 @@ module WaterDrop
|
|
46
45
|
# @example Aborted transaction - messages producer won't be visible by consumers
|
47
46
|
# producer.transaction do
|
48
47
|
# producer.produce_sync(topic: 'topic', payload: 'data')
|
49
|
-
#
|
48
|
+
# raise WaterDrop::AbortTransaction
|
50
49
|
# end
|
51
50
|
#
|
52
51
|
# @example Use block result last handler to wait on all messages ack
|
@@ -55,7 +54,7 @@ module WaterDrop
|
|
55
54
|
# end
|
56
55
|
#
|
57
56
|
# handler.wait
|
58
|
-
def transaction
|
57
|
+
def transaction
|
59
58
|
# This will safely allow us to support one operation transactions so a transactional
|
60
59
|
# producer can work without the transactional block if needed
|
61
60
|
return yield if @transaction_mutex.owned?
|
@@ -66,9 +65,26 @@ module WaterDrop
|
|
66
65
|
transactional_instrument(:started) { client.begin_transaction }
|
67
66
|
end
|
68
67
|
|
69
|
-
result
|
68
|
+
result = nil
|
69
|
+
finished = false
|
70
70
|
|
71
|
-
|
71
|
+
begin
|
72
|
+
result = yield
|
73
|
+
finished = true
|
74
|
+
rescue Exception => e
|
75
|
+
raise(e)
|
76
|
+
ensure
|
77
|
+
if !e && !finished
|
78
|
+
raise(
|
79
|
+
Errors::EarlyTransactionExitNotAllowedError,
|
80
|
+
<<~ERROR_MSG.tr("\n", ' ')
|
81
|
+
Using `return`, `break` or `throw` to exit a transaction block is not allowed.
|
82
|
+
If the `throw` came from `Timeout.timeout(duration)`, pass an exception class as
|
83
|
+
a second argument so it doesn't use `throw` to abort its block.
|
84
|
+
ERROR_MSG
|
85
|
+
)
|
86
|
+
end
|
87
|
+
end
|
72
88
|
|
73
89
|
with_transactional_error_handling(:commit) do
|
74
90
|
transactional_instrument(:committed) { client.commit_transaction }
|
@@ -168,27 +184,6 @@ module WaterDrop
|
|
168
184
|
transactional? ? transaction(&block) : yield
|
169
185
|
end
|
170
186
|
|
171
|
-
# Executes the requested code in a transaction with error handling and ensures, that upon
|
172
|
-
# early break we rollback the transaction instead of having it dangling and causing an issue
|
173
|
-
# where transactional producer would end up in an error state.
|
174
|
-
def transactional_execute
|
175
|
-
result = nil
|
176
|
-
commit = false
|
177
|
-
|
178
|
-
catch(:abort) do
|
179
|
-
result = yield
|
180
|
-
commit = true
|
181
|
-
end
|
182
|
-
|
183
|
-
[result, commit]
|
184
|
-
rescue Exception => e
|
185
|
-
errored = true
|
186
|
-
|
187
|
-
raise e
|
188
|
-
ensure
|
189
|
-
return [result, commit] unless errored
|
190
|
-
end
|
191
|
-
|
192
187
|
# Instruments the transactional operation with producer id
|
193
188
|
#
|
194
189
|
# @param key [Symbol] transaction operation key
|
@@ -76,11 +76,12 @@ module WaterDrop
|
|
76
76
|
scope.instance_methods(false).each do |method_name|
|
77
77
|
class_eval <<-RUBY, __FILE__, __LINE__ + 1
|
78
78
|
def #{method_name}(*args, &block)
|
79
|
-
|
79
|
+
ref = Fiber.current.waterdrop_clients ||= {}
|
80
|
+
ref[@producer.id] = self
|
80
81
|
|
81
82
|
@producer.#{method_name}(*args, &block)
|
82
83
|
ensure
|
83
|
-
|
84
|
+
ref[@producer.id] = nil
|
84
85
|
end
|
85
86
|
RUBY
|
86
87
|
end
|
data/lib/waterdrop/producer.rb
CHANGED
@@ -10,16 +10,22 @@ module WaterDrop
|
|
10
10
|
include Transactions
|
11
11
|
include ::Karafka::Core::Helpers::Time
|
12
12
|
|
13
|
+
# Local storage for given thread waterdrop client references for variants
|
14
|
+
::Fiber.send(:attr_accessor, :waterdrop_clients)
|
15
|
+
|
13
16
|
# Which of the inline flow errors do we want to intercept and re-bind
|
14
17
|
SUPPORTED_FLOW_ERRORS = [
|
15
18
|
Rdkafka::RdkafkaError,
|
16
19
|
Rdkafka::Producer::DeliveryHandle::WaitTimeoutError
|
17
20
|
].freeze
|
18
21
|
|
19
|
-
# Empty
|
22
|
+
# Empty hash to save on memory allocations
|
20
23
|
EMPTY_HASH = {}.freeze
|
21
24
|
|
22
|
-
|
25
|
+
# Empty array to save on memory allocations
|
26
|
+
EMPTY_ARRAY = [].freeze
|
27
|
+
|
28
|
+
private_constant :SUPPORTED_FLOW_ERRORS, :EMPTY_HASH, :EMPTY_ARRAY
|
23
29
|
|
24
30
|
def_delegators :config
|
25
31
|
|
@@ -269,17 +275,21 @@ module WaterDrop
|
|
269
275
|
# Waits on a given handler
|
270
276
|
#
|
271
277
|
# @param handler [Rdkafka::Producer::DeliveryHandle]
|
272
|
-
|
278
|
+
# @param raise_response_error [Boolean] should we raise the response error after we receive the
|
279
|
+
# final result and it is an error.
|
280
|
+
def wait(handler, raise_response_error: true)
|
273
281
|
handler.wait(
|
274
282
|
# rdkafka max_wait_timeout is in seconds and we use ms
|
275
|
-
max_wait_timeout: current_variant.max_wait_timeout / 1_000.0
|
283
|
+
max_wait_timeout: current_variant.max_wait_timeout / 1_000.0,
|
284
|
+
raise_response_error: raise_response_error
|
276
285
|
)
|
277
286
|
end
|
278
287
|
|
279
|
-
# @return [Producer::
|
288
|
+
# @return [Producer::Variant] the variant config. Either custom if built using `#with` or
|
280
289
|
# a default one.
|
281
290
|
def current_variant
|
282
|
-
|
291
|
+
Fiber.current.waterdrop_clients ||= {}
|
292
|
+
Fiber.current.waterdrop_clients[id] || @default_variant
|
283
293
|
end
|
284
294
|
|
285
295
|
# Runs the client produce method with a given message
|
data/lib/waterdrop/version.rb
CHANGED
data/waterdrop.gemspec
CHANGED
@@ -14,19 +14,19 @@ Gem::Specification.new do |spec|
|
|
14
14
|
spec.homepage = 'https://karafka.io'
|
15
15
|
spec.summary = 'Kafka messaging made easy!'
|
16
16
|
spec.description = spec.summary
|
17
|
-
spec.
|
17
|
+
spec.licenses = %w[LGPL-3.0-only Commercial]
|
18
18
|
|
19
19
|
spec.add_dependency 'karafka-core', '>= 2.4.3', '< 3.0.0'
|
20
|
-
spec.add_dependency 'karafka-rdkafka', '>= 0.
|
20
|
+
spec.add_dependency 'karafka-rdkafka', '>= 0.17.5'
|
21
21
|
spec.add_dependency 'zeitwerk', '~> 2.3'
|
22
22
|
|
23
|
-
spec.required_ruby_version = '>= 3.
|
23
|
+
spec.required_ruby_version = '>= 3.1.0'
|
24
24
|
|
25
25
|
if $PROGRAM_NAME.end_with?('gem')
|
26
26
|
spec.signing_key = File.expand_path('~/.ssh/gem-private_key.pem')
|
27
27
|
end
|
28
28
|
|
29
|
-
spec.cert_chain = %w[certs/
|
29
|
+
spec.cert_chain = %w[certs/cert.pem]
|
30
30
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
|
31
31
|
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
32
32
|
spec.require_paths = %w[lib]
|
data.tar.gz.sig
CHANGED
Binary file
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: waterdrop
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.8.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Maciej Mensfeld
|
@@ -12,30 +12,30 @@ cert_chain:
|
|
12
12
|
-----BEGIN CERTIFICATE-----
|
13
13
|
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
14
14
|
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
15
|
-
|
15
|
+
MB4XDTI0MDgyMzEwMTkyMFoXDTQ5MDgxNzEwMTkyMFowPzEQMA4GA1UEAwwHY29u
|
16
16
|
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
17
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAKjLhLjQqUlNayxkXnO+
|
18
|
+
PsmCDs/KFIzhrsYMfLZRZNaWmzV3ujljMOdDjd4snM2X06C41iVdQPWjpe3j8vVe
|
19
|
+
ZXEWR/twSbOP6Eeg8WVH2wCOo0x5i7yhVn4UBLH4JpfEMCbemVcWQ9ry9OMg4WpH
|
20
|
+
Uu4dRwxFV7hzCz3p0QfNLRI4miAxnGWcnlD98IJRjBAksTuR1Llj0vbOrDGsL9ZT
|
21
|
+
JeXP2gdRLd8SqzAFJEWrbeTBCBU7gfSh3oMg5SVDLjaqf7Kz5wC/8bDZydzanOxB
|
22
|
+
T6CDXPsCnllmvTNx2ei2T5rGYJOzJeNTmJLLK6hJWUlAvaQSvCwZRvFJ0tVGLEoS
|
23
|
+
flqSr6uGyyl1eMUsNmsH4BqPEYcAV6P2PKTv2vUR8AP0raDvZ3xL1TKvfRb8xRpo
|
24
|
+
vPopCGlY5XBWEc6QERHfVLTIVsjnls2/Ujj4h8/TSfqqYnaHKefIMLbuD/tquMjD
|
25
|
+
iWQsW2qStBV0T+U7FijKxVfrfqZP7GxQmDAc9o1iiyAa3QIDAQABo3cwdTAJBgNV
|
26
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQU3O4dTXmvE7YpAkszGzR9DdL9
|
27
|
+
sbEwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
28
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAVKTfoLXn7mqdSxIR
|
29
|
+
eqxcR6Huudg1jes81s1+X0uiRTR3hxxKZ3Y82cPsee9zYWyBrN8TA4KA0WILTru7
|
30
|
+
Ygxvzha0SRPsSiaKLmgOJ+61ebI4+bOORzIJLpD6GxCxu1r7MI4+0r1u1xe0EWi8
|
31
|
+
agkVo1k4Vi8cKMLm6Gl9b3wG9zQBw6fcgKwmpjKiNnOLP+OytzUANrIUJjoq6oal
|
32
|
+
TC+f/Uc0TLaRqUaW/bejxzDWWHoM3SU6aoLPuerglzp9zZVzihXwx3jPLUVKDFpF
|
33
|
+
Rl2lcBDxlpYGueGo0/oNzGJAAy6js8jhtHC9+19PD53vk7wHtFTZ/0ugDQYnwQ+x
|
34
|
+
oml2fAAuVWpTBCgOVFe6XCQpMKopzoxQ1PjKztW2KYxgJdIBX87SnL3aWuBQmhRd
|
35
|
+
i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
|
36
|
+
ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
|
37
37
|
-----END CERTIFICATE-----
|
38
|
-
date: 2024-
|
38
|
+
date: 2024-09-16 00:00:00.000000000 Z
|
39
39
|
dependencies:
|
40
40
|
- !ruby/object:Gem::Dependency
|
41
41
|
name: karafka-core
|
@@ -63,14 +63,14 @@ dependencies:
|
|
63
63
|
requirements:
|
64
64
|
- - ">="
|
65
65
|
- !ruby/object:Gem::Version
|
66
|
-
version: 0.
|
66
|
+
version: 0.17.5
|
67
67
|
type: :runtime
|
68
68
|
prerelease: false
|
69
69
|
version_requirements: !ruby/object:Gem::Requirement
|
70
70
|
requirements:
|
71
71
|
- - ">="
|
72
72
|
- !ruby/object:Gem::Version
|
73
|
-
version: 0.
|
73
|
+
version: 0.17.5
|
74
74
|
- !ruby/object:Gem::Dependency
|
75
75
|
name: zeitwerk
|
76
76
|
requirement: !ruby/object:Gem::Requirement
|
@@ -103,9 +103,9 @@ files:
|
|
103
103
|
- CHANGELOG.md
|
104
104
|
- Gemfile
|
105
105
|
- Gemfile.lock
|
106
|
-
-
|
106
|
+
- LICENSE
|
107
107
|
- README.md
|
108
|
-
- certs/
|
108
|
+
- certs/cert.pem
|
109
109
|
- config/locales/errors.yml
|
110
110
|
- docker-compose.yml
|
111
111
|
- lib/waterdrop.rb
|
@@ -144,7 +144,8 @@ files:
|
|
144
144
|
- waterdrop.gemspec
|
145
145
|
homepage: https://karafka.io
|
146
146
|
licenses:
|
147
|
-
-
|
147
|
+
- LGPL-3.0-only
|
148
|
+
- Commercial
|
148
149
|
metadata:
|
149
150
|
funding_uri: https://karafka.io/#become-pro
|
150
151
|
homepage_uri: https://karafka.io
|
@@ -161,14 +162,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
161
162
|
requirements:
|
162
163
|
- - ">="
|
163
164
|
- !ruby/object:Gem::Version
|
164
|
-
version: 3.
|
165
|
+
version: 3.1.0
|
165
166
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
166
167
|
requirements:
|
167
168
|
- - ">="
|
168
169
|
- !ruby/object:Gem::Version
|
169
170
|
version: '0'
|
170
171
|
requirements: []
|
171
|
-
rubygems_version: 3.5.
|
172
|
+
rubygems_version: 3.5.16
|
172
173
|
signing_key:
|
173
174
|
specification_version: 4
|
174
175
|
summary: Kafka messaging made easy!
|
metadata.gz.sig
CHANGED
Binary file
|
data/MIT-LICENSE
DELETED
@@ -1,18 +0,0 @@
|
|
1
|
-
Permission is hereby granted, free of charge, to any person obtaining
|
2
|
-
a copy of this software and associated documentation files (the
|
3
|
-
"Software"), to deal in the Software without restriction, including
|
4
|
-
without limitation the rights to use, copy, modify, merge, publish,
|
5
|
-
distribute, sublicense, and/or sell copies of the Software, and to
|
6
|
-
permit persons to whom the Software is furnished to do so, subject to
|
7
|
-
the following conditions:
|
8
|
-
|
9
|
-
The above copyright notice and this permission notice shall be
|
10
|
-
included in all copies or substantial portions of the Software.
|
11
|
-
|
12
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
13
|
-
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
14
|
-
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
15
|
-
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
16
|
-
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
17
|
-
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
18
|
-
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/certs/cert_chain.pem
DELETED
@@ -1,26 +0,0 @@
|
|
1
|
-
-----BEGIN CERTIFICATE-----
|
2
|
-
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
3
|
-
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
4
|
-
MB4XDTIzMDgyMTA3MjU1NFoXDTI0MDgyMDA3MjU1NFowPzEQMA4GA1UEAwwHY29u
|
5
|
-
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
6
|
-
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAOuZpyQKEwsTG9plLat7
|
7
|
-
8bUaNuNBEnouTsNMr6X+XTgvyrAxTuocdsyP1sNCjdS1B8RiiDH1/Nt9qpvlBWon
|
8
|
-
sdJ1SYhaWNVfqiYStTDnCx3PRMmHRdD4KqUWKpN6VpZ1O/Zu+9Mw0COmvXgZuuO9
|
9
|
-
wMSJkXRo6dTCfMedLAIxjMeBIxtoLR2e6Jm6MR8+8WYYVWrO9kSOOt5eKQLBY7aK
|
10
|
-
b/Dc40EcJKPg3Z30Pia1M9ZyRlb6SOj6SKpHRqc7vbVQxjEw6Jjal1lZ49m3YZMd
|
11
|
-
ArMAs9lQZNdSw5/UX6HWWURLowg6k10RnhTUtYyzO9BFev0JFJftHnmuk8vtb+SD
|
12
|
-
5VPmjFXg2VOcw0B7FtG75Vackk8QKfgVe3nSPhVpew2CSPlbJzH80wChbr19+e3+
|
13
|
-
YGr1tOiaJrL6c+PNmb0F31NXMKpj/r+n15HwlTMRxQrzFcgjBlxf2XFGnPQXHhBm
|
14
|
-
kp1OFnEq4GG9sON4glRldkwzi/f/fGcZmo5fm3d+0ZdNgwIDAQABo3cwdTAJBgNV
|
15
|
-
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQUPVH5+dLA80A1kJ2Uz5iGwfOa
|
16
|
-
1+swHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
17
|
-
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAnpa0jcN7JzREHMTQ
|
18
|
-
bfZ+xcvlrzuROMY6A3zIZmQgbnoZZNuX4cMRrT1p1HuwXpxdpHPw7dDjYqWw3+1h
|
19
|
-
3mXLeMuk7amjQpYoSWU/OIZMhIsARra22UN8qkkUlUj3AwTaChVKN/bPJOM2DzfU
|
20
|
-
kz9vUgLeYYFfQbZqeI6SsM7ltilRV4W8D9yNUQQvOxCFxtLOetJ00fC/E7zMUzbK
|
21
|
-
IBwYFQYsbI6XQzgAIPW6nGSYKgRhkfpmquXSNKZRIQ4V6bFrufa+DzD0bt2ZA3ah
|
22
|
-
fMmJguyb5L2Gf1zpDXzFSPMG7YQFLzwYz1zZZvOU7/UCpQsHpID/YxqDp4+Dgb+Y
|
23
|
-
qma0whX8UG/gXFV2pYWpYOfpatvahwi+A1TwPQsuZwkkhi1OyF1At3RY+hjSXyav
|
24
|
-
AnG1dJU+yL2BK7vaVytLTstJME5mepSZ46qqIJXMuWob/YPDmVaBF39TDSG9e34s
|
25
|
-
msG3BiCqgOgHAnL23+CN3Rt8MsuRfEtoTKpJVcCfoEoNHOkc
|
26
|
-
-----END CERTIFICATE-----
|