waterdrop 2.7.4 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/.github/workflows/ci.yml +0 -1
- data/.ruby-version +1 -1
- data/CHANGELOG.md +97 -0
- data/Gemfile.lock +21 -19
- data/LICENSE +17 -0
- data/README.md +1 -1
- data/certs/cert.pem +26 -0
- data/docker-compose.yml +1 -1
- data/lib/waterdrop/errors.rb +3 -0
- data/lib/waterdrop/instrumentation/logger_listener.rb +5 -2
- data/lib/waterdrop/producer/sync.rb +31 -7
- data/lib/waterdrop/producer/transactions.rb +21 -26
- data/lib/waterdrop/producer/variant.rb +3 -2
- data/lib/waterdrop/producer.rb +16 -6
- data/lib/waterdrop/version.rb +1 -1
- data/waterdrop.gemspec +4 -4
- data.tar.gz.sig +0 -0
- metadata +31 -30
- metadata.gz.sig +0 -0
- data/MIT-LICENSE +0 -18
- data/certs/cert_chain.pem +0 -26
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 96435533f730edae9dcf7bed5426641a33715e87b55c804168fa6a02465f1431
|
4
|
+
data.tar.gz: 3737e88ec3ffaa407b1cdaa523cf1d24a25aa85c4d90989a249381dbfb7a80c2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1b83e8505ca1aa9be5c137c06af1b98e10bb8f39af93e77aefa42302bf8dc2bd95f1442eb2569d5f032e80ac0374b8013e774d1f36494d23e20af0766094e2b3
|
7
|
+
data.tar.gz: 198ea59a4f7ca6a63f9816e7026c97152aaed5e6154142c47e6e96c16696a2a3d4c60cd56bf72b407b092682e9d15c08f1adfa552eb5bdc17a308f77f7424dfc
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/.github/workflows/ci.yml
CHANGED
data/.ruby-version
CHANGED
@@ -1 +1 @@
|
|
1
|
-
3.3.
|
1
|
+
3.3.5
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,102 @@
|
|
1
1
|
# WaterDrop changelog
|
2
2
|
|
3
|
+
## 2.8.0 (2024-09-16)
|
4
|
+
|
5
|
+
This release contains **BREAKING** changes. Make sure to read and apply upgrade notes.
|
6
|
+
|
7
|
+
- **[Breaking]** Require Ruby `3.1+`.
|
8
|
+
- **[Breaking]** Remove ability to abort transactions using `throw(:abort)`. Please use `raise WaterDrop::Errors::AbortTransaction`.
|
9
|
+
- **[Breaking]** Disallow (similar to ActiveRecord) exiting transactions with `return`, `break` or `throw`.
|
10
|
+
- **[Breaking]** License changed from MIT to LGPL with an additional commercial option. Note: there is no commercial code in this repository. The commercial license is available for companies unable to use LGPL-licensed software for legal reasons.
|
11
|
+
- [Enhancement] Make variants fiber safe.
|
12
|
+
- [Enhancement] In transactional mode do not return any `dispatched` messages as none will be dispatched due to rollback.
|
13
|
+
- [Enhancement] Align the `LoggerListener` async messages to reflect, that messages are delegated to the internal queue and not dispatched.
|
14
|
+
- [Fix] Ensure, that `:dispatched` key for `#produce_many_sync` always contains delivery handles (final) and not delivery reports.
|
15
|
+
|
16
|
+
### Upgrade Notes
|
17
|
+
|
18
|
+
**PLEASE MAKE SURE TO READ AND APPLY THEM!**
|
19
|
+
|
20
|
+
#### `throw(:abort)` No Longer Allowed To Abort Transactions
|
21
|
+
|
22
|
+
Replace:
|
23
|
+
|
24
|
+
```ruby
|
25
|
+
producer.transaction do
|
26
|
+
messages.each do |message|
|
27
|
+
# Pipe all events
|
28
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
29
|
+
end
|
30
|
+
|
31
|
+
# And abort if more events are no longer needed
|
32
|
+
throw(:abort) if KnowledgeBase.more_events_needed?
|
33
|
+
end
|
34
|
+
```
|
35
|
+
|
36
|
+
With:
|
37
|
+
|
38
|
+
```ruby
|
39
|
+
producer.transaction do
|
40
|
+
messages.each do |message|
|
41
|
+
# Pipe all events
|
42
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
43
|
+
end
|
44
|
+
|
45
|
+
# And abort if more events are no longer needed
|
46
|
+
raise(WaterDrop::AbortTransaction) if KnowledgeBase.more_events_needed?
|
47
|
+
end
|
48
|
+
```
|
49
|
+
|
50
|
+
#### `return`, `break` and `throw` No Longer Allowed Inside Transaction Block
|
51
|
+
|
52
|
+
Previously, transactions would abort if you exited early using `return`, `break`, or `throw`. This could create unexpected behavior, where users might not notice the rollback or have different intentions. For example, the following would trigger a rollback:
|
53
|
+
|
54
|
+
```ruby
|
55
|
+
MAX = 10
|
56
|
+
|
57
|
+
def process(messages)
|
58
|
+
count = 0
|
59
|
+
|
60
|
+
producer.transaction do
|
61
|
+
messages.each do |message|
|
62
|
+
count += 1
|
63
|
+
|
64
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
65
|
+
|
66
|
+
# This would trigger a rollback.
|
67
|
+
return if count >= MAX
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
```
|
72
|
+
|
73
|
+
This is a source of errors, hence such exits are no longer allowed. You can implement similar flow control inside of your methods that are wrapped in a WaterDrop transaction:
|
74
|
+
|
75
|
+
```ruby
|
76
|
+
MAX = 10
|
77
|
+
|
78
|
+
def process(messages)
|
79
|
+
producer.transaction do
|
80
|
+
# Early return from this method will not affect the transaction.
|
81
|
+
# It will be committed
|
82
|
+
insert_with_limit(messages)
|
83
|
+
end
|
84
|
+
end
|
85
|
+
|
86
|
+
def insert_with_limit(messages)
|
87
|
+
count = 0
|
88
|
+
|
89
|
+
messages.each do |message|
|
90
|
+
count += 1
|
91
|
+
|
92
|
+
producer.produce_async(topic: 'events', payload: message.raw_payload)
|
93
|
+
|
94
|
+
# This would trigger a rollback.
|
95
|
+
return if count >= MAX
|
96
|
+
end
|
97
|
+
end
|
98
|
+
```
|
99
|
+
|
3
100
|
## 2.7.4 (2024-07-04)
|
4
101
|
- [Maintenance] Alias `WaterDrop::Errors::AbortTransaction` with `WaterDrop::AbortTransaction`.
|
5
102
|
- [Maintenance] Lower the precision reporting to 100 microseconds in the logger listener.
|
data/Gemfile.lock
CHANGED
@@ -1,70 +1,72 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
waterdrop (2.
|
4
|
+
waterdrop (2.8.0)
|
5
5
|
karafka-core (>= 2.4.3, < 3.0.0)
|
6
|
-
karafka-rdkafka (>= 0.
|
6
|
+
karafka-rdkafka (>= 0.17.5)
|
7
7
|
zeitwerk (~> 2.3)
|
8
8
|
|
9
9
|
GEM
|
10
10
|
remote: https://rubygems.org/
|
11
11
|
specs:
|
12
|
-
activesupport (7.1
|
12
|
+
activesupport (7.2.1)
|
13
13
|
base64
|
14
14
|
bigdecimal
|
15
|
-
concurrent-ruby (~> 1.0, >= 1.
|
15
|
+
concurrent-ruby (~> 1.0, >= 1.3.1)
|
16
16
|
connection_pool (>= 2.2.5)
|
17
17
|
drb
|
18
18
|
i18n (>= 1.6, < 2)
|
19
|
+
logger (>= 1.4.2)
|
19
20
|
minitest (>= 5.1)
|
20
|
-
|
21
|
-
tzinfo (~> 2.0)
|
21
|
+
securerandom (>= 0.3)
|
22
|
+
tzinfo (~> 2.0, >= 2.0.5)
|
22
23
|
base64 (0.2.0)
|
23
24
|
bigdecimal (3.1.8)
|
24
25
|
byebug (11.1.3)
|
25
|
-
concurrent-ruby (1.3.
|
26
|
+
concurrent-ruby (1.3.4)
|
26
27
|
connection_pool (2.4.1)
|
27
28
|
diff-lcs (1.5.1)
|
28
|
-
docile (1.4.
|
29
|
+
docile (1.4.1)
|
29
30
|
drb (2.2.1)
|
30
|
-
factory_bot (6.
|
31
|
+
factory_bot (6.5.0)
|
31
32
|
activesupport (>= 5.0.0)
|
32
33
|
ffi (1.17.0)
|
33
34
|
i18n (1.14.5)
|
34
35
|
concurrent-ruby (~> 1.0)
|
35
|
-
karafka-core (2.4.
|
36
|
-
karafka-rdkafka (>= 0.15.0, < 0.
|
37
|
-
karafka-rdkafka (0.
|
36
|
+
karafka-core (2.4.4)
|
37
|
+
karafka-rdkafka (>= 0.15.0, < 0.18.0)
|
38
|
+
karafka-rdkafka (0.17.6)
|
38
39
|
ffi (~> 1.15)
|
39
40
|
mini_portile2 (~> 2.6)
|
40
41
|
rake (> 12)
|
42
|
+
logger (1.6.1)
|
41
43
|
mini_portile2 (2.8.7)
|
42
|
-
minitest (5.
|
43
|
-
mutex_m (0.2.0)
|
44
|
+
minitest (5.25.1)
|
44
45
|
ostruct (0.6.0)
|
45
46
|
rake (13.2.1)
|
46
47
|
rspec (3.13.0)
|
47
48
|
rspec-core (~> 3.13.0)
|
48
49
|
rspec-expectations (~> 3.13.0)
|
49
50
|
rspec-mocks (~> 3.13.0)
|
50
|
-
rspec-core (3.13.
|
51
|
+
rspec-core (3.13.1)
|
51
52
|
rspec-support (~> 3.13.0)
|
52
|
-
rspec-expectations (3.13.
|
53
|
+
rspec-expectations (3.13.3)
|
53
54
|
diff-lcs (>= 1.2.0, < 2.0)
|
54
55
|
rspec-support (~> 3.13.0)
|
55
56
|
rspec-mocks (3.13.1)
|
56
57
|
diff-lcs (>= 1.2.0, < 2.0)
|
57
58
|
rspec-support (~> 3.13.0)
|
58
59
|
rspec-support (3.13.1)
|
60
|
+
securerandom (0.3.1)
|
59
61
|
simplecov (0.22.0)
|
60
62
|
docile (~> 1.1)
|
61
63
|
simplecov-html (~> 0.11)
|
62
64
|
simplecov_json_formatter (~> 0.1)
|
63
|
-
simplecov-html (0.
|
65
|
+
simplecov-html (0.13.1)
|
64
66
|
simplecov_json_formatter (0.1.4)
|
65
67
|
tzinfo (2.0.6)
|
66
68
|
concurrent-ruby (~> 1.0)
|
67
|
-
zeitwerk (2.6.
|
69
|
+
zeitwerk (2.6.18)
|
68
70
|
|
69
71
|
PLATFORMS
|
70
72
|
ruby
|
@@ -79,4 +81,4 @@ DEPENDENCIES
|
|
79
81
|
waterdrop!
|
80
82
|
|
81
83
|
BUNDLED WITH
|
82
|
-
2.5.
|
84
|
+
2.5.16
|
data/LICENSE
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
Copyright (c) Maciej Mensfeld
|
2
|
+
|
3
|
+
WaterDrop is part of Karafka and it is an Open Source project licensed under the terms of
|
4
|
+
the LGPLv3 license. Please see <https://github.com/karafka/karafka/blob/master/LICENSE-LGPL>
|
5
|
+
for license text.
|
6
|
+
|
7
|
+
Karafka ecosystem can also be used under commercial-friendly license, with commercial support and commercial components.
|
8
|
+
|
9
|
+
All of the commercial components are present in the lib/karafka/pro and lib/karafka/web/pro
|
10
|
+
directory of this repository and their usage requires commercial license agreement.
|
11
|
+
|
12
|
+
By sending a pull request to the pro components, you are agreeing to transfer the copyright of your
|
13
|
+
code to Maciej Mensfeld.
|
14
|
+
|
15
|
+
You can find the commercial license in LICENSE-COMM <https://github.com/karafka/karafka/blob/master/LICENSE-COMM>.
|
16
|
+
|
17
|
+
Please see https://karafka.io for purchasing options.
|
data/README.md
CHANGED
@@ -15,7 +15,7 @@ It:
|
|
15
15
|
- Supports producing to multiple clusters
|
16
16
|
- Supports multiple delivery policies
|
17
17
|
- Supports per-topic configuration alterations (variants)
|
18
|
-
- Works with Kafka `1.0+` and Ruby `3.
|
18
|
+
- Works with Kafka `1.0+` and Ruby `3.1+`
|
19
19
|
- Works with and without Karafka
|
20
20
|
|
21
21
|
## Documentation
|
data/certs/cert.pem
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
-----BEGIN CERTIFICATE-----
|
2
|
+
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
3
|
+
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
4
|
+
MB4XDTI0MDgyMzEwMTkyMFoXDTQ5MDgxNzEwMTkyMFowPzEQMA4GA1UEAwwHY29u
|
5
|
+
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
6
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAKjLhLjQqUlNayxkXnO+
|
7
|
+
PsmCDs/KFIzhrsYMfLZRZNaWmzV3ujljMOdDjd4snM2X06C41iVdQPWjpe3j8vVe
|
8
|
+
ZXEWR/twSbOP6Eeg8WVH2wCOo0x5i7yhVn4UBLH4JpfEMCbemVcWQ9ry9OMg4WpH
|
9
|
+
Uu4dRwxFV7hzCz3p0QfNLRI4miAxnGWcnlD98IJRjBAksTuR1Llj0vbOrDGsL9ZT
|
10
|
+
JeXP2gdRLd8SqzAFJEWrbeTBCBU7gfSh3oMg5SVDLjaqf7Kz5wC/8bDZydzanOxB
|
11
|
+
T6CDXPsCnllmvTNx2ei2T5rGYJOzJeNTmJLLK6hJWUlAvaQSvCwZRvFJ0tVGLEoS
|
12
|
+
flqSr6uGyyl1eMUsNmsH4BqPEYcAV6P2PKTv2vUR8AP0raDvZ3xL1TKvfRb8xRpo
|
13
|
+
vPopCGlY5XBWEc6QERHfVLTIVsjnls2/Ujj4h8/TSfqqYnaHKefIMLbuD/tquMjD
|
14
|
+
iWQsW2qStBV0T+U7FijKxVfrfqZP7GxQmDAc9o1iiyAa3QIDAQABo3cwdTAJBgNV
|
15
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQU3O4dTXmvE7YpAkszGzR9DdL9
|
16
|
+
sbEwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
17
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAVKTfoLXn7mqdSxIR
|
18
|
+
eqxcR6Huudg1jes81s1+X0uiRTR3hxxKZ3Y82cPsee9zYWyBrN8TA4KA0WILTru7
|
19
|
+
Ygxvzha0SRPsSiaKLmgOJ+61ebI4+bOORzIJLpD6GxCxu1r7MI4+0r1u1xe0EWi8
|
20
|
+
agkVo1k4Vi8cKMLm6Gl9b3wG9zQBw6fcgKwmpjKiNnOLP+OytzUANrIUJjoq6oal
|
21
|
+
TC+f/Uc0TLaRqUaW/bejxzDWWHoM3SU6aoLPuerglzp9zZVzihXwx3jPLUVKDFpF
|
22
|
+
Rl2lcBDxlpYGueGo0/oNzGJAAy6js8jhtHC9+19PD53vk7wHtFTZ/0ugDQYnwQ+x
|
23
|
+
oml2fAAuVWpTBCgOVFe6XCQpMKopzoxQ1PjKztW2KYxgJdIBX87SnL3aWuBQmhRd
|
24
|
+
i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
|
25
|
+
ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
|
26
|
+
-----END CERTIFICATE-----
|
data/docker-compose.yml
CHANGED
data/lib/waterdrop/errors.rb
CHANGED
@@ -47,6 +47,9 @@ module WaterDrop
|
|
47
47
|
# graceful transaction aborting
|
48
48
|
AbortTransaction = Class.new(BaseError)
|
49
49
|
|
50
|
+
# Do not use `break`, `return` or `throw` inside of the transaction blocks
|
51
|
+
EarlyTransactionExitNotAllowedError = Class.new(BaseError)
|
52
|
+
|
50
53
|
# Raised when during messages producing something bad happened inline
|
51
54
|
class ProduceManyError < ProduceError
|
52
55
|
attr_reader :dispatched
|
@@ -26,7 +26,7 @@ module WaterDrop
|
|
26
26
|
def on_message_produced_async(event)
|
27
27
|
message = event[:message]
|
28
28
|
|
29
|
-
info(event, "
|
29
|
+
info(event, "Message to '#{message[:topic]}' topic was delegated to a dispatch queue")
|
30
30
|
|
31
31
|
return unless log_messages?
|
32
32
|
|
@@ -49,7 +49,10 @@ module WaterDrop
|
|
49
49
|
messages = event[:messages]
|
50
50
|
topics_count = messages.map { |message| "'#{message[:topic]}'" }.uniq.count
|
51
51
|
|
52
|
-
info(
|
52
|
+
info(
|
53
|
+
event,
|
54
|
+
"#{messages.size} messages to #{topics_count} topics were delegated to a dispatch queue"
|
55
|
+
)
|
53
56
|
|
54
57
|
return unless log_messages?
|
55
58
|
|
@@ -61,17 +61,37 @@ module WaterDrop
|
|
61
61
|
messages.each { |message| validate_message!(message) }
|
62
62
|
|
63
63
|
dispatched = []
|
64
|
+
inline_error = nil
|
64
65
|
|
65
66
|
@monitor.instrument('messages.produced_sync', producer_id: id, messages: messages) do
|
66
|
-
|
67
|
-
|
68
|
-
|
67
|
+
# While most of the librdkafka errors are async and not inline, there are some like
|
68
|
+
# buffer overflow that can leak in during the `#produce` itself. When this happens, we
|
69
|
+
# still (since it's a sync mode) need to wait on deliveries of things that were
|
70
|
+
# successfully delegated to librdkafka. To do this, we catch the inline error and we
|
71
|
+
# wait on messages that were in the buffer to reach final state. Then if no error, we
|
72
|
+
# check each for error and if none all good. If there was an inline error, we re-raise
|
73
|
+
# it with the handlers in final states.
|
74
|
+
#
|
75
|
+
# Such flow ensures, that we never end up with handlers not being in the final states
|
76
|
+
# for the sync flow
|
77
|
+
begin
|
78
|
+
with_transaction_if_transactional do
|
79
|
+
messages.each do |message|
|
80
|
+
dispatched << produce(message)
|
81
|
+
end
|
69
82
|
end
|
83
|
+
rescue *SUPPORTED_FLOW_ERRORS => e
|
84
|
+
inline_error = e
|
70
85
|
end
|
71
86
|
|
72
|
-
|
73
|
-
|
74
|
-
|
87
|
+
# This will ensure, that we have all verdicts before raising the failure, so we pass
|
88
|
+
# all delivery handles having a final verdict
|
89
|
+
dispatched.each { |handler| wait(handler, raise_response_error: false) }
|
90
|
+
|
91
|
+
raise(inline_error) if inline_error
|
92
|
+
|
93
|
+
# This will raise an error on the first error that have happened
|
94
|
+
dispatched.each { |handler| wait(handler) }
|
75
95
|
|
76
96
|
dispatched
|
77
97
|
end
|
@@ -82,7 +102,11 @@ module WaterDrop
|
|
82
102
|
'error.occurred',
|
83
103
|
producer_id: id,
|
84
104
|
messages: messages,
|
85
|
-
dispatched
|
105
|
+
# If it is a transactional producer nothing was successfully dispatched on error, thus
|
106
|
+
# we never return any dispatched handlers. While those messages might have reached
|
107
|
+
# Kafka, in transactional mode they will not be visible to consumers with correct
|
108
|
+
# isolation level.
|
109
|
+
dispatched: transactional? ? EMPTY_ARRAY : dispatched,
|
86
110
|
error: re_raised,
|
87
111
|
type: 'messages.produce_many_sync'
|
88
112
|
)
|
@@ -35,7 +35,6 @@ module WaterDrop
|
|
35
35
|
# with a transaction. One transaction per single dispatch and for `produce_many` it will be
|
36
36
|
# a single transaction wrapping all messages dispatches (not one per message).
|
37
37
|
#
|
38
|
-
# @param block [Proc] block of code that should run
|
39
38
|
# @return Block result or `nil` in case of early break/return
|
40
39
|
#
|
41
40
|
# @example Simple transaction
|
@@ -46,7 +45,7 @@ module WaterDrop
|
|
46
45
|
# @example Aborted transaction - messages producer won't be visible by consumers
|
47
46
|
# producer.transaction do
|
48
47
|
# producer.produce_sync(topic: 'topic', payload: 'data')
|
49
|
-
#
|
48
|
+
# raise WaterDrop::AbortTransaction
|
50
49
|
# end
|
51
50
|
#
|
52
51
|
# @example Use block result last handler to wait on all messages ack
|
@@ -55,7 +54,7 @@ module WaterDrop
|
|
55
54
|
# end
|
56
55
|
#
|
57
56
|
# handler.wait
|
58
|
-
def transaction
|
57
|
+
def transaction
|
59
58
|
# This will safely allow us to support one operation transactions so a transactional
|
60
59
|
# producer can work without the transactional block if needed
|
61
60
|
return yield if @transaction_mutex.owned?
|
@@ -66,9 +65,26 @@ module WaterDrop
|
|
66
65
|
transactional_instrument(:started) { client.begin_transaction }
|
67
66
|
end
|
68
67
|
|
69
|
-
result
|
68
|
+
result = nil
|
69
|
+
finished = false
|
70
70
|
|
71
|
-
|
71
|
+
begin
|
72
|
+
result = yield
|
73
|
+
finished = true
|
74
|
+
rescue Exception => e
|
75
|
+
raise(e)
|
76
|
+
ensure
|
77
|
+
if !e && !finished
|
78
|
+
raise(
|
79
|
+
Errors::EarlyTransactionExitNotAllowedError,
|
80
|
+
<<~ERROR_MSG.tr("\n", ' ')
|
81
|
+
Using `return`, `break` or `throw` to exit a transaction block is not allowed.
|
82
|
+
If the `throw` came from `Timeout.timeout(duration)`, pass an exception class as
|
83
|
+
a second argument so it doesn't use `throw` to abort its block.
|
84
|
+
ERROR_MSG
|
85
|
+
)
|
86
|
+
end
|
87
|
+
end
|
72
88
|
|
73
89
|
with_transactional_error_handling(:commit) do
|
74
90
|
transactional_instrument(:committed) { client.commit_transaction }
|
@@ -168,27 +184,6 @@ module WaterDrop
|
|
168
184
|
transactional? ? transaction(&block) : yield
|
169
185
|
end
|
170
186
|
|
171
|
-
# Executes the requested code in a transaction with error handling and ensures, that upon
|
172
|
-
# early break we rollback the transaction instead of having it dangling and causing an issue
|
173
|
-
# where transactional producer would end up in an error state.
|
174
|
-
def transactional_execute
|
175
|
-
result = nil
|
176
|
-
commit = false
|
177
|
-
|
178
|
-
catch(:abort) do
|
179
|
-
result = yield
|
180
|
-
commit = true
|
181
|
-
end
|
182
|
-
|
183
|
-
[result, commit]
|
184
|
-
rescue Exception => e
|
185
|
-
errored = true
|
186
|
-
|
187
|
-
raise e
|
188
|
-
ensure
|
189
|
-
return [result, commit] unless errored
|
190
|
-
end
|
191
|
-
|
192
187
|
# Instruments the transactional operation with producer id
|
193
188
|
#
|
194
189
|
# @param key [Symbol] transaction operation key
|
@@ -76,11 +76,12 @@ module WaterDrop
|
|
76
76
|
scope.instance_methods(false).each do |method_name|
|
77
77
|
class_eval <<-RUBY, __FILE__, __LINE__ + 1
|
78
78
|
def #{method_name}(*args, &block)
|
79
|
-
|
79
|
+
ref = Fiber.current.waterdrop_clients ||= {}
|
80
|
+
ref[@producer.id] = self
|
80
81
|
|
81
82
|
@producer.#{method_name}(*args, &block)
|
82
83
|
ensure
|
83
|
-
|
84
|
+
ref[@producer.id] = nil
|
84
85
|
end
|
85
86
|
RUBY
|
86
87
|
end
|
data/lib/waterdrop/producer.rb
CHANGED
@@ -10,16 +10,22 @@ module WaterDrop
|
|
10
10
|
include Transactions
|
11
11
|
include ::Karafka::Core::Helpers::Time
|
12
12
|
|
13
|
+
# Local storage for given thread waterdrop client references for variants
|
14
|
+
::Fiber.send(:attr_accessor, :waterdrop_clients)
|
15
|
+
|
13
16
|
# Which of the inline flow errors do we want to intercept and re-bind
|
14
17
|
SUPPORTED_FLOW_ERRORS = [
|
15
18
|
Rdkafka::RdkafkaError,
|
16
19
|
Rdkafka::Producer::DeliveryHandle::WaitTimeoutError
|
17
20
|
].freeze
|
18
21
|
|
19
|
-
# Empty
|
22
|
+
# Empty hash to save on memory allocations
|
20
23
|
EMPTY_HASH = {}.freeze
|
21
24
|
|
22
|
-
|
25
|
+
# Empty array to save on memory allocations
|
26
|
+
EMPTY_ARRAY = [].freeze
|
27
|
+
|
28
|
+
private_constant :SUPPORTED_FLOW_ERRORS, :EMPTY_HASH, :EMPTY_ARRAY
|
23
29
|
|
24
30
|
def_delegators :config
|
25
31
|
|
@@ -269,17 +275,21 @@ module WaterDrop
|
|
269
275
|
# Waits on a given handler
|
270
276
|
#
|
271
277
|
# @param handler [Rdkafka::Producer::DeliveryHandle]
|
272
|
-
|
278
|
+
# @param raise_response_error [Boolean] should we raise the response error after we receive the
|
279
|
+
# final result and it is an error.
|
280
|
+
def wait(handler, raise_response_error: true)
|
273
281
|
handler.wait(
|
274
282
|
# rdkafka max_wait_timeout is in seconds and we use ms
|
275
|
-
max_wait_timeout: current_variant.max_wait_timeout / 1_000.0
|
283
|
+
max_wait_timeout: current_variant.max_wait_timeout / 1_000.0,
|
284
|
+
raise_response_error: raise_response_error
|
276
285
|
)
|
277
286
|
end
|
278
287
|
|
279
|
-
# @return [Producer::
|
288
|
+
# @return [Producer::Variant] the variant config. Either custom if built using `#with` or
|
280
289
|
# a default one.
|
281
290
|
def current_variant
|
282
|
-
|
291
|
+
Fiber.current.waterdrop_clients ||= {}
|
292
|
+
Fiber.current.waterdrop_clients[id] || @default_variant
|
283
293
|
end
|
284
294
|
|
285
295
|
# Runs the client produce method with a given message
|
data/lib/waterdrop/version.rb
CHANGED
data/waterdrop.gemspec
CHANGED
@@ -14,19 +14,19 @@ Gem::Specification.new do |spec|
|
|
14
14
|
spec.homepage = 'https://karafka.io'
|
15
15
|
spec.summary = 'Kafka messaging made easy!'
|
16
16
|
spec.description = spec.summary
|
17
|
-
spec.
|
17
|
+
spec.licenses = %w[LGPL-3.0-only Commercial]
|
18
18
|
|
19
19
|
spec.add_dependency 'karafka-core', '>= 2.4.3', '< 3.0.0'
|
20
|
-
spec.add_dependency 'karafka-rdkafka', '>= 0.
|
20
|
+
spec.add_dependency 'karafka-rdkafka', '>= 0.17.5'
|
21
21
|
spec.add_dependency 'zeitwerk', '~> 2.3'
|
22
22
|
|
23
|
-
spec.required_ruby_version = '>= 3.
|
23
|
+
spec.required_ruby_version = '>= 3.1.0'
|
24
24
|
|
25
25
|
if $PROGRAM_NAME.end_with?('gem')
|
26
26
|
spec.signing_key = File.expand_path('~/.ssh/gem-private_key.pem')
|
27
27
|
end
|
28
28
|
|
29
|
-
spec.cert_chain = %w[certs/
|
29
|
+
spec.cert_chain = %w[certs/cert.pem]
|
30
30
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
|
31
31
|
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
32
32
|
spec.require_paths = %w[lib]
|
data.tar.gz.sig
CHANGED
Binary file
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: waterdrop
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.8.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Maciej Mensfeld
|
@@ -12,30 +12,30 @@ cert_chain:
|
|
12
12
|
-----BEGIN CERTIFICATE-----
|
13
13
|
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
14
14
|
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
15
|
-
|
15
|
+
MB4XDTI0MDgyMzEwMTkyMFoXDTQ5MDgxNzEwMTkyMFowPzEQMA4GA1UEAwwHY29u
|
16
16
|
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
17
|
+
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAKjLhLjQqUlNayxkXnO+
|
18
|
+
PsmCDs/KFIzhrsYMfLZRZNaWmzV3ujljMOdDjd4snM2X06C41iVdQPWjpe3j8vVe
|
19
|
+
ZXEWR/twSbOP6Eeg8WVH2wCOo0x5i7yhVn4UBLH4JpfEMCbemVcWQ9ry9OMg4WpH
|
20
|
+
Uu4dRwxFV7hzCz3p0QfNLRI4miAxnGWcnlD98IJRjBAksTuR1Llj0vbOrDGsL9ZT
|
21
|
+
JeXP2gdRLd8SqzAFJEWrbeTBCBU7gfSh3oMg5SVDLjaqf7Kz5wC/8bDZydzanOxB
|
22
|
+
T6CDXPsCnllmvTNx2ei2T5rGYJOzJeNTmJLLK6hJWUlAvaQSvCwZRvFJ0tVGLEoS
|
23
|
+
flqSr6uGyyl1eMUsNmsH4BqPEYcAV6P2PKTv2vUR8AP0raDvZ3xL1TKvfRb8xRpo
|
24
|
+
vPopCGlY5XBWEc6QERHfVLTIVsjnls2/Ujj4h8/TSfqqYnaHKefIMLbuD/tquMjD
|
25
|
+
iWQsW2qStBV0T+U7FijKxVfrfqZP7GxQmDAc9o1iiyAa3QIDAQABo3cwdTAJBgNV
|
26
|
+
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQU3O4dTXmvE7YpAkszGzR9DdL9
|
27
|
+
sbEwHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
28
|
+
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAVKTfoLXn7mqdSxIR
|
29
|
+
eqxcR6Huudg1jes81s1+X0uiRTR3hxxKZ3Y82cPsee9zYWyBrN8TA4KA0WILTru7
|
30
|
+
Ygxvzha0SRPsSiaKLmgOJ+61ebI4+bOORzIJLpD6GxCxu1r7MI4+0r1u1xe0EWi8
|
31
|
+
agkVo1k4Vi8cKMLm6Gl9b3wG9zQBw6fcgKwmpjKiNnOLP+OytzUANrIUJjoq6oal
|
32
|
+
TC+f/Uc0TLaRqUaW/bejxzDWWHoM3SU6aoLPuerglzp9zZVzihXwx3jPLUVKDFpF
|
33
|
+
Rl2lcBDxlpYGueGo0/oNzGJAAy6js8jhtHC9+19PD53vk7wHtFTZ/0ugDQYnwQ+x
|
34
|
+
oml2fAAuVWpTBCgOVFe6XCQpMKopzoxQ1PjKztW2KYxgJdIBX87SnL3aWuBQmhRd
|
35
|
+
i9zWxov0mr44TWegTVeypcWGd/0nxu1+QHVNHJrpqlPBRvwQsUm7fwmRInGpcaB8
|
36
|
+
ap8wNYvryYzrzvzUxIVFBVM5PacgkFqRmolCa8I7tdKQN+R1
|
37
37
|
-----END CERTIFICATE-----
|
38
|
-
date: 2024-
|
38
|
+
date: 2024-09-16 00:00:00.000000000 Z
|
39
39
|
dependencies:
|
40
40
|
- !ruby/object:Gem::Dependency
|
41
41
|
name: karafka-core
|
@@ -63,14 +63,14 @@ dependencies:
|
|
63
63
|
requirements:
|
64
64
|
- - ">="
|
65
65
|
- !ruby/object:Gem::Version
|
66
|
-
version: 0.
|
66
|
+
version: 0.17.5
|
67
67
|
type: :runtime
|
68
68
|
prerelease: false
|
69
69
|
version_requirements: !ruby/object:Gem::Requirement
|
70
70
|
requirements:
|
71
71
|
- - ">="
|
72
72
|
- !ruby/object:Gem::Version
|
73
|
-
version: 0.
|
73
|
+
version: 0.17.5
|
74
74
|
- !ruby/object:Gem::Dependency
|
75
75
|
name: zeitwerk
|
76
76
|
requirement: !ruby/object:Gem::Requirement
|
@@ -103,9 +103,9 @@ files:
|
|
103
103
|
- CHANGELOG.md
|
104
104
|
- Gemfile
|
105
105
|
- Gemfile.lock
|
106
|
-
-
|
106
|
+
- LICENSE
|
107
107
|
- README.md
|
108
|
-
- certs/
|
108
|
+
- certs/cert.pem
|
109
109
|
- config/locales/errors.yml
|
110
110
|
- docker-compose.yml
|
111
111
|
- lib/waterdrop.rb
|
@@ -144,7 +144,8 @@ files:
|
|
144
144
|
- waterdrop.gemspec
|
145
145
|
homepage: https://karafka.io
|
146
146
|
licenses:
|
147
|
-
-
|
147
|
+
- LGPL-3.0-only
|
148
|
+
- Commercial
|
148
149
|
metadata:
|
149
150
|
funding_uri: https://karafka.io/#become-pro
|
150
151
|
homepage_uri: https://karafka.io
|
@@ -161,14 +162,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
161
162
|
requirements:
|
162
163
|
- - ">="
|
163
164
|
- !ruby/object:Gem::Version
|
164
|
-
version: 3.
|
165
|
+
version: 3.1.0
|
165
166
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
166
167
|
requirements:
|
167
168
|
- - ">="
|
168
169
|
- !ruby/object:Gem::Version
|
169
170
|
version: '0'
|
170
171
|
requirements: []
|
171
|
-
rubygems_version: 3.5.
|
172
|
+
rubygems_version: 3.5.16
|
172
173
|
signing_key:
|
173
174
|
specification_version: 4
|
174
175
|
summary: Kafka messaging made easy!
|
metadata.gz.sig
CHANGED
Binary file
|
data/MIT-LICENSE
DELETED
@@ -1,18 +0,0 @@
|
|
1
|
-
Permission is hereby granted, free of charge, to any person obtaining
|
2
|
-
a copy of this software and associated documentation files (the
|
3
|
-
"Software"), to deal in the Software without restriction, including
|
4
|
-
without limitation the rights to use, copy, modify, merge, publish,
|
5
|
-
distribute, sublicense, and/or sell copies of the Software, and to
|
6
|
-
permit persons to whom the Software is furnished to do so, subject to
|
7
|
-
the following conditions:
|
8
|
-
|
9
|
-
The above copyright notice and this permission notice shall be
|
10
|
-
included in all copies or substantial portions of the Software.
|
11
|
-
|
12
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
13
|
-
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
14
|
-
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
15
|
-
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
16
|
-
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
17
|
-
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
18
|
-
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/certs/cert_chain.pem
DELETED
@@ -1,26 +0,0 @@
|
|
1
|
-
-----BEGIN CERTIFICATE-----
|
2
|
-
MIIEcDCCAtigAwIBAgIBATANBgkqhkiG9w0BAQsFADA/MRAwDgYDVQQDDAdjb250
|
3
|
-
YWN0MRcwFQYKCZImiZPyLGQBGRYHa2FyYWZrYTESMBAGCgmSJomT8ixkARkWAmlv
|
4
|
-
MB4XDTIzMDgyMTA3MjU1NFoXDTI0MDgyMDA3MjU1NFowPzEQMA4GA1UEAwwHY29u
|
5
|
-
dGFjdDEXMBUGCgmSJomT8ixkARkWB2thcmFma2ExEjAQBgoJkiaJk/IsZAEZFgJp
|
6
|
-
bzCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAOuZpyQKEwsTG9plLat7
|
7
|
-
8bUaNuNBEnouTsNMr6X+XTgvyrAxTuocdsyP1sNCjdS1B8RiiDH1/Nt9qpvlBWon
|
8
|
-
sdJ1SYhaWNVfqiYStTDnCx3PRMmHRdD4KqUWKpN6VpZ1O/Zu+9Mw0COmvXgZuuO9
|
9
|
-
wMSJkXRo6dTCfMedLAIxjMeBIxtoLR2e6Jm6MR8+8WYYVWrO9kSOOt5eKQLBY7aK
|
10
|
-
b/Dc40EcJKPg3Z30Pia1M9ZyRlb6SOj6SKpHRqc7vbVQxjEw6Jjal1lZ49m3YZMd
|
11
|
-
ArMAs9lQZNdSw5/UX6HWWURLowg6k10RnhTUtYyzO9BFev0JFJftHnmuk8vtb+SD
|
12
|
-
5VPmjFXg2VOcw0B7FtG75Vackk8QKfgVe3nSPhVpew2CSPlbJzH80wChbr19+e3+
|
13
|
-
YGr1tOiaJrL6c+PNmb0F31NXMKpj/r+n15HwlTMRxQrzFcgjBlxf2XFGnPQXHhBm
|
14
|
-
kp1OFnEq4GG9sON4glRldkwzi/f/fGcZmo5fm3d+0ZdNgwIDAQABo3cwdTAJBgNV
|
15
|
-
HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHQ4EFgQUPVH5+dLA80A1kJ2Uz5iGwfOa
|
16
|
-
1+swHQYDVR0RBBYwFIESY29udGFjdEBrYXJhZmthLmlvMB0GA1UdEgQWMBSBEmNv
|
17
|
-
bnRhY3RAa2FyYWZrYS5pbzANBgkqhkiG9w0BAQsFAAOCAYEAnpa0jcN7JzREHMTQ
|
18
|
-
bfZ+xcvlrzuROMY6A3zIZmQgbnoZZNuX4cMRrT1p1HuwXpxdpHPw7dDjYqWw3+1h
|
19
|
-
3mXLeMuk7amjQpYoSWU/OIZMhIsARra22UN8qkkUlUj3AwTaChVKN/bPJOM2DzfU
|
20
|
-
kz9vUgLeYYFfQbZqeI6SsM7ltilRV4W8D9yNUQQvOxCFxtLOetJ00fC/E7zMUzbK
|
21
|
-
IBwYFQYsbI6XQzgAIPW6nGSYKgRhkfpmquXSNKZRIQ4V6bFrufa+DzD0bt2ZA3ah
|
22
|
-
fMmJguyb5L2Gf1zpDXzFSPMG7YQFLzwYz1zZZvOU7/UCpQsHpID/YxqDp4+Dgb+Y
|
23
|
-
qma0whX8UG/gXFV2pYWpYOfpatvahwi+A1TwPQsuZwkkhi1OyF1At3RY+hjSXyav
|
24
|
-
AnG1dJU+yL2BK7vaVytLTstJME5mepSZ46qqIJXMuWob/YPDmVaBF39TDSG9e34s
|
25
|
-
msG3BiCqgOgHAnL23+CN3Rt8MsuRfEtoTKpJVcCfoEoNHOkc
|
26
|
-
-----END CERTIFICATE-----
|