phobos 1.9.0 → 2.0.0.pre.beta1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +52 -9
- data/lib/phobos/actions/process_batch_inline.rb +3 -22
- data/lib/phobos/actions/process_message.rb +2 -37
- data/lib/phobos/producer.rb +12 -46
- data/lib/phobos/version.rb +1 -1
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 646f966d9d307346090b88a3ef52f784da42f07a7f8fd3369131a4d1149fc293
|
4
|
+
data.tar.gz: 3631aef66014166e318f24b8b282265e0d3663ed006b915c49762301e8f02b89
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c19eedfd196604f55ea659d501323cecfca50c4c53a03ae1193457f344e4a1e580bbd7ca2b33559b20ab2c1418143b48b6d9e4942986f71b369f4a9e261ccfeb
|
7
|
+
data.tar.gz: cdbd2146b341d91bb7131eb986e24055bf73054ed12eb542e09c9d2a56b4b462bd4d3015e5ee3a00934e4e125658e5438987b402aa2edb0ca2fac2172b098f58
|
data/CHANGELOG.md
CHANGED
@@ -6,13 +6,22 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
|
|
6
6
|
``
|
7
7
|
## UNRELEASED
|
8
8
|
|
9
|
+
## [2.0.0-beta1] - 2020-5-04
|
10
|
+
|
11
|
+
- Remove deprecated patterns:
|
12
|
+
-- `before_consume` method
|
13
|
+
-- `around_consume` as a class method or without yielding values
|
14
|
+
-- `publish` and `async_publish` with positional arguments
|
15
|
+
|
9
16
|
## [1.9.0] - 2020-03-05
|
10
17
|
- Bumped version to 1.9.0.
|
11
18
|
|
12
19
|
## [1.9.0-beta3] - 2020-02-05
|
20
|
+
|
13
21
|
- Fix bug where deprecation errors would be shown when receiving nil payloads
|
14
22
|
even if `around_consume` was updated to yield them.
|
15
23
|
|
24
|
+
|
16
25
|
## [1.9.0-beta2] - 2020-01-09
|
17
26
|
|
18
27
|
- Allow `around_consume` to yield payload and metadata, and deprecate
|
data/README.md
CHANGED
@@ -21,7 +21,6 @@ With Phobos by your side, all this becomes smooth sailing.
|
|
21
21
|
## Table of Contents
|
22
22
|
|
23
23
|
1. [Installation](#installation)
|
24
|
-
1. [Upgrade Notes](#upgrade-notes)
|
25
24
|
1. [Usage](#usage)
|
26
25
|
1. [Standalone apps](#usage-standalone-apps)
|
27
26
|
1. [Consuming messages from Kafka](#usage-consuming-messages-from-kafka)
|
@@ -32,6 +31,7 @@ With Phobos by your side, all this becomes smooth sailing.
|
|
32
31
|
1. [Plugins](#plugins)
|
33
32
|
1. [Development](#development)
|
34
33
|
1. [Test](#test)
|
34
|
+
1. [Upgrade Notes](#upgrade-notes)
|
35
35
|
|
36
36
|
## <a name="installation"></a> Installation
|
37
37
|
|
@@ -53,10 +53,6 @@ Or install it yourself as:
|
|
53
53
|
$ gem install phobos
|
54
54
|
```
|
55
55
|
|
56
|
-
### <a name="upgrade-notes"></a> Upgrade Notes
|
57
|
-
|
58
|
-
Version 1.8.2 introduced a new `persistent_connections` setting for regular producers. This reduces the number of connections used to produce messages and you should consider setting it to true. This does require a manual shutdown call - please see [Producers with persistent connections](#persistent-connection).
|
59
|
-
|
60
56
|
## <a name="usage"></a> Usage
|
61
57
|
|
62
58
|
Phobos can be used in two ways: as a standalone application or to support Kafka features in your existing project - including Rails apps. It provides a CLI tool to run it.
|
@@ -197,8 +193,6 @@ class MyHandler
|
|
197
193
|
end
|
198
194
|
```
|
199
195
|
|
200
|
-
Note: Previous versions used a `before_consume` method to pre-process the payload. This is still supported, but deprecated. Going forward, `around_consume` should yield the payload and metadata back to the calling code, allowing it to act as a pre-processor, e.g. by decoding Avro messages into Ruby hashes.
|
201
|
-
|
202
196
|
Take a look at the examples folder for some ideas.
|
203
197
|
|
204
198
|
The hander life cycle can be illustrated as:
|
@@ -287,8 +281,6 @@ my = MyProducer.new
|
|
287
281
|
my.producer.publish(topic: 'topic', payload: 'message-payload', key: 'partition and message key', headers: { header_1: 'value 1' })
|
288
282
|
```
|
289
283
|
|
290
|
-
Older versions of Phobos provided a `publish` method that accepted positional arguments. That version is still supported but it's soon to be deprecated in favour of the keyword arguments version.
|
291
|
-
|
292
284
|
It is also possible to publish several messages at once:
|
293
285
|
|
294
286
|
```ruby
|
@@ -619,6 +611,57 @@ describe MyConsumer do
|
|
619
611
|
end
|
620
612
|
```
|
621
613
|
|
614
|
+
## <a name="upgrade-notes"></a> Upgrade Notes
|
615
|
+
|
616
|
+
Version 2.0 removes deprecated ways of defining producers and consumers:
|
617
|
+
* The `before_consume` method has been removed. You can have this behavior in the first part of an `around_consume` method.
|
618
|
+
* `around_consume` is now only available as an instance method, and it must yield the values to pass to the `consume` method.
|
619
|
+
* `publish` and `async_publish` now only accept keyword arguments, not positional arguments.
|
620
|
+
|
621
|
+
Example pre-2.0:
|
622
|
+
```ruby
|
623
|
+
class MyHandler
|
624
|
+
include Phobos::Handler
|
625
|
+
|
626
|
+
def before_consume(payload, metadata)
|
627
|
+
payload[:id] = 1
|
628
|
+
end
|
629
|
+
|
630
|
+
def self.around_consume(payload, metadata)
|
631
|
+
metadata[:key] = 5
|
632
|
+
yield
|
633
|
+
end
|
634
|
+
end
|
635
|
+
```
|
636
|
+
|
637
|
+
In 2.0:
|
638
|
+
```ruby
|
639
|
+
class MyHandler
|
640
|
+
include Phobos::Handler
|
641
|
+
|
642
|
+
def around_consume(payload, metadata)
|
643
|
+
new_payload = payload.dup
|
644
|
+
new_metadata = metadata.dup
|
645
|
+
new_payload[:id] = 1
|
646
|
+
new_metadata[:key] = 5
|
647
|
+
yield new_payload, new_metadata
|
648
|
+
end
|
649
|
+
end
|
650
|
+
```
|
651
|
+
|
652
|
+
Producer, 1.9:
|
653
|
+
```ruby
|
654
|
+
producer.publish('my-topic', { payload_value: 1}, 5, 3, {header_val: 5})
|
655
|
+
```
|
656
|
+
|
657
|
+
Producer 2.0:
|
658
|
+
```ruby
|
659
|
+
producer.publish(topic: 'my-topic', payload: { payload_value: 1}, key: 5,
|
660
|
+
partition_key: 3, headers: { header_val: 5})
|
661
|
+
```
|
662
|
+
|
663
|
+
Version 1.8.2 introduced a new `persistent_connections` setting for regular producers. This reduces the number of connections used to produce messages and you should consider setting it to true. This does require a manual shutdown call - please see [Producers with persistent connections](#persistent-connection).
|
664
|
+
|
622
665
|
## Contributing
|
623
666
|
|
624
667
|
Bug reports and pull requests are welcome on GitHub at https://github.com/klarna/phobos.
|
@@ -47,32 +47,13 @@ module Phobos
|
|
47
47
|
)
|
48
48
|
end
|
49
49
|
|
50
|
-
def preprocess(batch, handler)
|
51
|
-
if handler.respond_to?(:before_consume_batch)
|
52
|
-
Phobos.deprecate('before_consume_batch is deprecated and will be removed in 2.0. \
|
53
|
-
Use around_consume_batch and yield payloads and metadata objects.')
|
54
|
-
handler.before_consume_batch(batch, @metadata)
|
55
|
-
else
|
56
|
-
batch
|
57
|
-
end
|
58
|
-
end
|
59
|
-
|
60
50
|
def process_batch(batch)
|
61
51
|
instrument('listener.process_batch_inline', @metadata) do |_metadata|
|
62
52
|
handler = @listener.handler_class.new
|
63
53
|
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
handler.consume_batch(around_batch, around_metadata)
|
68
|
-
else
|
69
|
-
Phobos.deprecate('Calling around_consume_batch without yielding payloads \
|
70
|
-
and metadata is deprecated and will be removed in 2.0.')
|
71
|
-
handler.consume_batch(preprocessed_batch, @metadata)
|
72
|
-
end
|
73
|
-
}
|
74
|
-
|
75
|
-
handler.around_consume_batch(preprocessed_batch, @metadata, &consume_block)
|
54
|
+
handler.around_consume_batch(batch, @metadata) do |around_batch, around_metadata|
|
55
|
+
handler.consume_batch(around_batch, around_metadata)
|
56
|
+
end
|
76
57
|
end
|
77
58
|
end
|
78
59
|
end
|
@@ -35,47 +35,12 @@ module Phobos
|
|
35
35
|
|
36
36
|
private
|
37
37
|
|
38
|
-
def preprocess(payload, handler)
|
39
|
-
if handler.respond_to?(:before_consume)
|
40
|
-
Phobos.deprecate('before_consume is deprecated and will be removed in 2.0. \
|
41
|
-
Use around_consume and yield payload and metadata objects.')
|
42
|
-
begin
|
43
|
-
handler.before_consume(payload, @metadata)
|
44
|
-
rescue ArgumentError
|
45
|
-
handler.before_consume(payload)
|
46
|
-
end
|
47
|
-
else
|
48
|
-
payload
|
49
|
-
end
|
50
|
-
end
|
51
|
-
|
52
|
-
def consume_block(payload, handler)
|
53
|
-
proc { |around_payload, around_metadata|
|
54
|
-
if around_metadata
|
55
|
-
handler.consume(around_payload, around_metadata)
|
56
|
-
else
|
57
|
-
Phobos.deprecate('Calling around_consume without yielding payload and metadata \
|
58
|
-
is deprecated and will be removed in 2.0.')
|
59
|
-
handler.consume(payload, @metadata)
|
60
|
-
end
|
61
|
-
}
|
62
|
-
end
|
63
|
-
|
64
38
|
def process_message(payload)
|
65
39
|
instrument('listener.process_message', @metadata) do
|
66
40
|
handler = @listener.handler_class.new
|
67
41
|
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
if @listener.handler_class.respond_to?(:around_consume)
|
72
|
-
# around_consume class method implementation
|
73
|
-
Phobos.deprecate('around_consume has been moved to instance method, please update '\
|
74
|
-
'your consumer. This will not be backwards compatible in the future.')
|
75
|
-
@listener.handler_class.around_consume(preprocessed_payload, @metadata, &block)
|
76
|
-
else
|
77
|
-
# around_consume instance method implementation
|
78
|
-
handler.around_consume(preprocessed_payload, @metadata, &block)
|
42
|
+
handler.around_consume(payload, @metadata) do |around_payload, around_metadata|
|
43
|
+
handler.consume(around_payload, around_metadata)
|
79
44
|
end
|
80
45
|
end
|
81
46
|
end
|
data/lib/phobos/producer.rb
CHANGED
@@ -11,28 +11,24 @@ module Phobos
|
|
11
11
|
end
|
12
12
|
|
13
13
|
class PublicAPI
|
14
|
-
MissingRequiredArgumentsError = Class.new(StandardError) do
|
15
|
-
def initialize
|
16
|
-
super('You need to provide a topic name and a payload')
|
17
|
-
end
|
18
|
-
end
|
19
|
-
|
20
14
|
def initialize(host_obj)
|
21
15
|
@host_obj = host_obj
|
22
16
|
end
|
23
17
|
|
24
|
-
def publish(
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
18
|
+
def publish(topic:, payload:, key: nil, partition_key: nil, headers: nil)
|
19
|
+
class_producer.publish(topic: topic,
|
20
|
+
payload: payload,
|
21
|
+
key: key,
|
22
|
+
partition_key: partition_key,
|
23
|
+
headers: headers)
|
29
24
|
end
|
30
25
|
|
31
|
-
def async_publish(
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
26
|
+
def async_publish(topic:, payload:, key: nil, partition_key: nil, headers: nil)
|
27
|
+
class_producer.async_publish(topic: topic,
|
28
|
+
payload: payload,
|
29
|
+
key: key,
|
30
|
+
partition_key: partition_key,
|
31
|
+
headers: headers)
|
36
32
|
end
|
37
33
|
|
38
34
|
# @param messages [Array(Hash(:topic, :payload, :key, :headers))]
|
@@ -54,36 +50,6 @@ module Phobos
|
|
54
50
|
def class_producer
|
55
51
|
@host_obj.class.producer
|
56
52
|
end
|
57
|
-
|
58
|
-
# rubocop:disable Metrics/ParameterLists
|
59
|
-
def normalize_arguments(p_topic = nil, p_payload = nil, p_key = nil,
|
60
|
-
p_partition_key = nil, p_headers = {},
|
61
|
-
**kwargs)
|
62
|
-
{}.tap do |args|
|
63
|
-
{
|
64
|
-
topic: p_topic,
|
65
|
-
payload: p_payload,
|
66
|
-
key: p_key,
|
67
|
-
partition_key: p_partition_key,
|
68
|
-
headers: p_headers
|
69
|
-
}.each { |k, v| args[k] = kwargs[k] || v }
|
70
|
-
|
71
|
-
raise MissingRequiredArgumentsError if [:topic, :payload].any? { |k| args[k].nil? }
|
72
|
-
|
73
|
-
kwargs.each do |k, v|
|
74
|
-
next if args.key?(k)
|
75
|
-
|
76
|
-
args[:headers][k] = v
|
77
|
-
end
|
78
|
-
end
|
79
|
-
end
|
80
|
-
# rubocop:enable Metrics/ParameterLists
|
81
|
-
|
82
|
-
def deprecate_positional_args_message(method_name)
|
83
|
-
"The `#{method_name}` method should now receive keyword arguments " \
|
84
|
-
'rather than positional ones. Please update your publishers. This will ' \
|
85
|
-
'not be backwards compatible in the future.'
|
86
|
-
end
|
87
53
|
end
|
88
54
|
|
89
55
|
module ClassMethods
|
data/lib/phobos/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: phobos
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0.pre.beta1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Túlio Ornelas
|
@@ -15,7 +15,7 @@ authors:
|
|
15
15
|
autorequire:
|
16
16
|
bindir: bin
|
17
17
|
cert_chain: []
|
18
|
-
date: 2020-
|
18
|
+
date: 2020-05-04 00:00:00.000000000 Z
|
19
19
|
dependencies:
|
20
20
|
- !ruby/object:Gem::Dependency
|
21
21
|
name: bundler
|
@@ -310,9 +310,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
310
310
|
version: '2.3'
|
311
311
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
312
312
|
requirements:
|
313
|
-
- - "
|
313
|
+
- - ">"
|
314
314
|
- !ruby/object:Gem::Version
|
315
|
-
version:
|
315
|
+
version: 1.3.1
|
316
316
|
requirements: []
|
317
317
|
rubygems_version: 3.1.2
|
318
318
|
signing_key:
|