phobos 1.9.0 → 2.0.0.pre.beta1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +52 -9
- data/lib/phobos/actions/process_batch_inline.rb +3 -22
- data/lib/phobos/actions/process_message.rb +2 -37
- data/lib/phobos/producer.rb +12 -46
- data/lib/phobos/version.rb +1 -1
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 646f966d9d307346090b88a3ef52f784da42f07a7f8fd3369131a4d1149fc293
|
4
|
+
data.tar.gz: 3631aef66014166e318f24b8b282265e0d3663ed006b915c49762301e8f02b89
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c19eedfd196604f55ea659d501323cecfca50c4c53a03ae1193457f344e4a1e580bbd7ca2b33559b20ab2c1418143b48b6d9e4942986f71b369f4a9e261ccfeb
|
7
|
+
data.tar.gz: cdbd2146b341d91bb7131eb986e24055bf73054ed12eb542e09c9d2a56b4b462bd4d3015e5ee3a00934e4e125658e5438987b402aa2edb0ca2fac2172b098f58
|
data/CHANGELOG.md
CHANGED
@@ -6,13 +6,22 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
|
|
6
6
|
``
|
7
7
|
## UNRELEASED
|
8
8
|
|
9
|
+
## [2.0.0-beta1] - 2020-5-04
|
10
|
+
|
11
|
+
- Remove deprecated patterns:
|
12
|
+
-- `before_consume` method
|
13
|
+
-- `around_consume` as a class method or without yielding values
|
14
|
+
-- `publish` and `async_publish` with positional arguments
|
15
|
+
|
9
16
|
## [1.9.0] - 2020-03-05
|
10
17
|
- Bumped version to 1.9.0.
|
11
18
|
|
12
19
|
## [1.9.0-beta3] - 2020-02-05
|
20
|
+
|
13
21
|
- Fix bug where deprecation errors would be shown when receiving nil payloads
|
14
22
|
even if `around_consume` was updated to yield them.
|
15
23
|
|
24
|
+
|
16
25
|
## [1.9.0-beta2] - 2020-01-09
|
17
26
|
|
18
27
|
- Allow `around_consume` to yield payload and metadata, and deprecate
|
data/README.md
CHANGED
@@ -21,7 +21,6 @@ With Phobos by your side, all this becomes smooth sailing.
|
|
21
21
|
## Table of Contents
|
22
22
|
|
23
23
|
1. [Installation](#installation)
|
24
|
-
1. [Upgrade Notes](#upgrade-notes)
|
25
24
|
1. [Usage](#usage)
|
26
25
|
1. [Standalone apps](#usage-standalone-apps)
|
27
26
|
1. [Consuming messages from Kafka](#usage-consuming-messages-from-kafka)
|
@@ -32,6 +31,7 @@ With Phobos by your side, all this becomes smooth sailing.
|
|
32
31
|
1. [Plugins](#plugins)
|
33
32
|
1. [Development](#development)
|
34
33
|
1. [Test](#test)
|
34
|
+
1. [Upgrade Notes](#upgrade-notes)
|
35
35
|
|
36
36
|
## <a name="installation"></a> Installation
|
37
37
|
|
@@ -53,10 +53,6 @@ Or install it yourself as:
|
|
53
53
|
$ gem install phobos
|
54
54
|
```
|
55
55
|
|
56
|
-
### <a name="upgrade-notes"></a> Upgrade Notes
|
57
|
-
|
58
|
-
Version 1.8.2 introduced a new `persistent_connections` setting for regular producers. This reduces the number of connections used to produce messages and you should consider setting it to true. This does require a manual shutdown call - please see [Producers with persistent connections](#persistent-connection).
|
59
|
-
|
60
56
|
## <a name="usage"></a> Usage
|
61
57
|
|
62
58
|
Phobos can be used in two ways: as a standalone application or to support Kafka features in your existing project - including Rails apps. It provides a CLI tool to run it.
|
@@ -197,8 +193,6 @@ class MyHandler
|
|
197
193
|
end
|
198
194
|
```
|
199
195
|
|
200
|
-
Note: Previous versions used a `before_consume` method to pre-process the payload. This is still supported, but deprecated. Going forward, `around_consume` should yield the payload and metadata back to the calling code, allowing it to act as a pre-processor, e.g. by decoding Avro messages into Ruby hashes.
|
201
|
-
|
202
196
|
Take a look at the examples folder for some ideas.
|
203
197
|
|
204
198
|
The hander life cycle can be illustrated as:
|
@@ -287,8 +281,6 @@ my = MyProducer.new
|
|
287
281
|
my.producer.publish(topic: 'topic', payload: 'message-payload', key: 'partition and message key', headers: { header_1: 'value 1' })
|
288
282
|
```
|
289
283
|
|
290
|
-
Older versions of Phobos provided a `publish` method that accepted positional arguments. That version is still supported but it's soon to be deprecated in favour of the keyword arguments version.
|
291
|
-
|
292
284
|
It is also possible to publish several messages at once:
|
293
285
|
|
294
286
|
```ruby
|
@@ -619,6 +611,57 @@ describe MyConsumer do
|
|
619
611
|
end
|
620
612
|
```
|
621
613
|
|
614
|
+
## <a name="upgrade-notes"></a> Upgrade Notes
|
615
|
+
|
616
|
+
Version 2.0 removes deprecated ways of defining producers and consumers:
|
617
|
+
* The `before_consume` method has been removed. You can have this behavior in the first part of an `around_consume` method.
|
618
|
+
* `around_consume` is now only available as an instance method, and it must yield the values to pass to the `consume` method.
|
619
|
+
* `publish` and `async_publish` now only accept keyword arguments, not positional arguments.
|
620
|
+
|
621
|
+
Example pre-2.0:
|
622
|
+
```ruby
|
623
|
+
class MyHandler
|
624
|
+
include Phobos::Handler
|
625
|
+
|
626
|
+
def before_consume(payload, metadata)
|
627
|
+
payload[:id] = 1
|
628
|
+
end
|
629
|
+
|
630
|
+
def self.around_consume(payload, metadata)
|
631
|
+
metadata[:key] = 5
|
632
|
+
yield
|
633
|
+
end
|
634
|
+
end
|
635
|
+
```
|
636
|
+
|
637
|
+
In 2.0:
|
638
|
+
```ruby
|
639
|
+
class MyHandler
|
640
|
+
include Phobos::Handler
|
641
|
+
|
642
|
+
def around_consume(payload, metadata)
|
643
|
+
new_payload = payload.dup
|
644
|
+
new_metadata = metadata.dup
|
645
|
+
new_payload[:id] = 1
|
646
|
+
new_metadata[:key] = 5
|
647
|
+
yield new_payload, new_metadata
|
648
|
+
end
|
649
|
+
end
|
650
|
+
```
|
651
|
+
|
652
|
+
Producer, 1.9:
|
653
|
+
```ruby
|
654
|
+
producer.publish('my-topic', { payload_value: 1}, 5, 3, {header_val: 5})
|
655
|
+
```
|
656
|
+
|
657
|
+
Producer 2.0:
|
658
|
+
```ruby
|
659
|
+
producer.publish(topic: 'my-topic', payload: { payload_value: 1}, key: 5,
|
660
|
+
partition_key: 3, headers: { header_val: 5})
|
661
|
+
```
|
662
|
+
|
663
|
+
Version 1.8.2 introduced a new `persistent_connections` setting for regular producers. This reduces the number of connections used to produce messages and you should consider setting it to true. This does require a manual shutdown call - please see [Producers with persistent connections](#persistent-connection).
|
664
|
+
|
622
665
|
## Contributing
|
623
666
|
|
624
667
|
Bug reports and pull requests are welcome on GitHub at https://github.com/klarna/phobos.
|
@@ -47,32 +47,13 @@ module Phobos
|
|
47
47
|
)
|
48
48
|
end
|
49
49
|
|
50
|
-
def preprocess(batch, handler)
|
51
|
-
if handler.respond_to?(:before_consume_batch)
|
52
|
-
Phobos.deprecate('before_consume_batch is deprecated and will be removed in 2.0. \
|
53
|
-
Use around_consume_batch and yield payloads and metadata objects.')
|
54
|
-
handler.before_consume_batch(batch, @metadata)
|
55
|
-
else
|
56
|
-
batch
|
57
|
-
end
|
58
|
-
end
|
59
|
-
|
60
50
|
def process_batch(batch)
|
61
51
|
instrument('listener.process_batch_inline', @metadata) do |_metadata|
|
62
52
|
handler = @listener.handler_class.new
|
63
53
|
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
handler.consume_batch(around_batch, around_metadata)
|
68
|
-
else
|
69
|
-
Phobos.deprecate('Calling around_consume_batch without yielding payloads \
|
70
|
-
and metadata is deprecated and will be removed in 2.0.')
|
71
|
-
handler.consume_batch(preprocessed_batch, @metadata)
|
72
|
-
end
|
73
|
-
}
|
74
|
-
|
75
|
-
handler.around_consume_batch(preprocessed_batch, @metadata, &consume_block)
|
54
|
+
handler.around_consume_batch(batch, @metadata) do |around_batch, around_metadata|
|
55
|
+
handler.consume_batch(around_batch, around_metadata)
|
56
|
+
end
|
76
57
|
end
|
77
58
|
end
|
78
59
|
end
|
@@ -35,47 +35,12 @@ module Phobos
|
|
35
35
|
|
36
36
|
private
|
37
37
|
|
38
|
-
def preprocess(payload, handler)
|
39
|
-
if handler.respond_to?(:before_consume)
|
40
|
-
Phobos.deprecate('before_consume is deprecated and will be removed in 2.0. \
|
41
|
-
Use around_consume and yield payload and metadata objects.')
|
42
|
-
begin
|
43
|
-
handler.before_consume(payload, @metadata)
|
44
|
-
rescue ArgumentError
|
45
|
-
handler.before_consume(payload)
|
46
|
-
end
|
47
|
-
else
|
48
|
-
payload
|
49
|
-
end
|
50
|
-
end
|
51
|
-
|
52
|
-
def consume_block(payload, handler)
|
53
|
-
proc { |around_payload, around_metadata|
|
54
|
-
if around_metadata
|
55
|
-
handler.consume(around_payload, around_metadata)
|
56
|
-
else
|
57
|
-
Phobos.deprecate('Calling around_consume without yielding payload and metadata \
|
58
|
-
is deprecated and will be removed in 2.0.')
|
59
|
-
handler.consume(payload, @metadata)
|
60
|
-
end
|
61
|
-
}
|
62
|
-
end
|
63
|
-
|
64
38
|
def process_message(payload)
|
65
39
|
instrument('listener.process_message', @metadata) do
|
66
40
|
handler = @listener.handler_class.new
|
67
41
|
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
if @listener.handler_class.respond_to?(:around_consume)
|
72
|
-
# around_consume class method implementation
|
73
|
-
Phobos.deprecate('around_consume has been moved to instance method, please update '\
|
74
|
-
'your consumer. This will not be backwards compatible in the future.')
|
75
|
-
@listener.handler_class.around_consume(preprocessed_payload, @metadata, &block)
|
76
|
-
else
|
77
|
-
# around_consume instance method implementation
|
78
|
-
handler.around_consume(preprocessed_payload, @metadata, &block)
|
42
|
+
handler.around_consume(payload, @metadata) do |around_payload, around_metadata|
|
43
|
+
handler.consume(around_payload, around_metadata)
|
79
44
|
end
|
80
45
|
end
|
81
46
|
end
|
data/lib/phobos/producer.rb
CHANGED
@@ -11,28 +11,24 @@ module Phobos
|
|
11
11
|
end
|
12
12
|
|
13
13
|
class PublicAPI
|
14
|
-
MissingRequiredArgumentsError = Class.new(StandardError) do
|
15
|
-
def initialize
|
16
|
-
super('You need to provide a topic name and a payload')
|
17
|
-
end
|
18
|
-
end
|
19
|
-
|
20
14
|
def initialize(host_obj)
|
21
15
|
@host_obj = host_obj
|
22
16
|
end
|
23
17
|
|
24
|
-
def publish(
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
18
|
+
def publish(topic:, payload:, key: nil, partition_key: nil, headers: nil)
|
19
|
+
class_producer.publish(topic: topic,
|
20
|
+
payload: payload,
|
21
|
+
key: key,
|
22
|
+
partition_key: partition_key,
|
23
|
+
headers: headers)
|
29
24
|
end
|
30
25
|
|
31
|
-
def async_publish(
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
26
|
+
def async_publish(topic:, payload:, key: nil, partition_key: nil, headers: nil)
|
27
|
+
class_producer.async_publish(topic: topic,
|
28
|
+
payload: payload,
|
29
|
+
key: key,
|
30
|
+
partition_key: partition_key,
|
31
|
+
headers: headers)
|
36
32
|
end
|
37
33
|
|
38
34
|
# @param messages [Array(Hash(:topic, :payload, :key, :headers))]
|
@@ -54,36 +50,6 @@ module Phobos
|
|
54
50
|
def class_producer
|
55
51
|
@host_obj.class.producer
|
56
52
|
end
|
57
|
-
|
58
|
-
# rubocop:disable Metrics/ParameterLists
|
59
|
-
def normalize_arguments(p_topic = nil, p_payload = nil, p_key = nil,
|
60
|
-
p_partition_key = nil, p_headers = {},
|
61
|
-
**kwargs)
|
62
|
-
{}.tap do |args|
|
63
|
-
{
|
64
|
-
topic: p_topic,
|
65
|
-
payload: p_payload,
|
66
|
-
key: p_key,
|
67
|
-
partition_key: p_partition_key,
|
68
|
-
headers: p_headers
|
69
|
-
}.each { |k, v| args[k] = kwargs[k] || v }
|
70
|
-
|
71
|
-
raise MissingRequiredArgumentsError if [:topic, :payload].any? { |k| args[k].nil? }
|
72
|
-
|
73
|
-
kwargs.each do |k, v|
|
74
|
-
next if args.key?(k)
|
75
|
-
|
76
|
-
args[:headers][k] = v
|
77
|
-
end
|
78
|
-
end
|
79
|
-
end
|
80
|
-
# rubocop:enable Metrics/ParameterLists
|
81
|
-
|
82
|
-
def deprecate_positional_args_message(method_name)
|
83
|
-
"The `#{method_name}` method should now receive keyword arguments " \
|
84
|
-
'rather than positional ones. Please update your publishers. This will ' \
|
85
|
-
'not be backwards compatible in the future.'
|
86
|
-
end
|
87
53
|
end
|
88
54
|
|
89
55
|
module ClassMethods
|
data/lib/phobos/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: phobos
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0.pre.beta1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Túlio Ornelas
|
@@ -15,7 +15,7 @@ authors:
|
|
15
15
|
autorequire:
|
16
16
|
bindir: bin
|
17
17
|
cert_chain: []
|
18
|
-
date: 2020-
|
18
|
+
date: 2020-05-04 00:00:00.000000000 Z
|
19
19
|
dependencies:
|
20
20
|
- !ruby/object:Gem::Dependency
|
21
21
|
name: bundler
|
@@ -310,9 +310,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
310
310
|
version: '2.3'
|
311
311
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
312
312
|
requirements:
|
313
|
-
- - "
|
313
|
+
- - ">"
|
314
314
|
- !ruby/object:Gem::Version
|
315
|
-
version:
|
315
|
+
version: 1.3.1
|
316
316
|
requirements: []
|
317
317
|
rubygems_version: 3.1.2
|
318
318
|
signing_key:
|