deimos-ruby 1.8.1.pre.beta7 → 1.8.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +36 -0
- data/Gemfile.lock +1 -1
- data/README.md +43 -3
- data/docs/CONFIGURATION.md +1 -0
- data/docs/UPGRADING.md +128 -0
- data/lib/deimos/config/configurable.rb +7 -0
- data/lib/deimos/config/configuration.rb +4 -0
- data/lib/deimos/config/phobos_config.rb +4 -1
- data/lib/deimos/producer.rb +14 -9
- data/lib/deimos/schema_backends/avro_schema_registry.rb +1 -1
- data/lib/deimos/test_helpers.rb +8 -7
- data/lib/deimos/utils/schema_controller_mixin.rb +5 -1
- data/lib/deimos/version.rb +1 -1
- data/spec/config/configuration_spec.rb +15 -0
- data/spec/producer_spec.rb +76 -0
- data/spec/schemas/com/my-namespace/request/CreateTopic.avsc +11 -0
- data/spec/schemas/com/my-namespace/response/CreateTopic.avsc +11 -0
- data/spec/spec_helper.rb +1 -0
- data/spec/utils/schema_controller_mixin_spec.rb +16 -0
- metadata +9 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 237cb6cd22b6a057003fedae1caeec81c999a3d9b90d837bb8b0a4add43ca4e2
|
4
|
+
data.tar.gz: fab51933cdcd5c5fd10b8c92bbc67854fe56f23cdbec41660579f18b9075e09c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 193b9b593b4f92edecc6c239eb6106fbe40e0acd1453694508f3fb02525166c39e4f1eba71230362493cd1647ab19e98a63ecf55f5371b844e9d0c38ffa5b2ea
|
7
|
+
data.tar.gz: e7d42ec7ec2bc864735c9230a925685935678d2dd0f3918426926dc9f358d5eb810dd5b324bcacaf9201c78bdfbbfa58037a4cac4cf4ca66257737440b3d2c97
|
data/CHANGELOG.md
CHANGED
@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
7
7
|
|
8
8
|
## UNRELEASED
|
9
9
|
|
10
|
+
## 1.8.2 - 2020-09-25
|
11
|
+
|
12
|
+
### Features :star:
|
13
|
+
- Add "disabled" config field to consumers to allow disabling
|
14
|
+
individual consumers without having to comment out their
|
15
|
+
entries and possibly affecting unit tests.
|
16
|
+
|
17
|
+
### Fixes :wrench:
|
18
|
+
- Prepend topic_prefix while encoding messages
|
19
|
+
(fixes [#37](https://github.com/flipp-oss/deimos/issues/37))
|
20
|
+
- Raise error if producing without a topic
|
21
|
+
(fixes [#50](https://github.com/flipp-oss/deimos/issues/50))
|
22
|
+
- Don't try to load producers/consumers when running rake tasks involving webpacker or assets
|
23
|
+
|
24
|
+
## 1.8.2-beta2 - 2020-09-15
|
25
|
+
|
26
|
+
### Features :star:
|
27
|
+
|
28
|
+
- Add details on using schema backend directly in README.
|
29
|
+
- Default to the provided schema if topic is not provided when
|
30
|
+
encoding to `AvroSchemaRegistry`.
|
31
|
+
- Add mapping syntax for the `schema` call in `SchemaControllerMixin`.
|
32
|
+
|
33
|
+
## 1.8.2-beta1 - 2020-09-09
|
34
|
+
|
35
|
+
### Features :star:
|
36
|
+
|
37
|
+
- Added the ability to specify the topic for `publish`
|
38
|
+
and `publish_list` in a producer
|
39
|
+
|
40
|
+
## 1.8.1-beta9 - 2020-08-27
|
41
|
+
|
42
|
+
### Fixes :wrench:
|
43
|
+
- Moved the TestHelpers hook to `before(:suite)` to allow for
|
44
|
+
overriding e.g. in integration tests.
|
45
|
+
|
10
46
|
## 1.8.1-beta7 - 2020-08-25
|
11
47
|
|
12
48
|
### Fixes :wrench:
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -11,6 +11,7 @@ a useful toolbox of goodies for Ruby-based Kafka development.
|
|
11
11
|
Built on Phobos and hence Ruby-Kafka.
|
12
12
|
|
13
13
|
<!--ts-->
|
14
|
+
* [Additional Documentation](#additional-documentation)
|
14
15
|
* [Installation](#installation)
|
15
16
|
* [Versioning](#versioning)
|
16
17
|
* [Configuration](#configuration)
|
@@ -29,9 +30,19 @@ Built on Phobos and hence Ruby-Kafka.
|
|
29
30
|
* [Metrics](#metrics)
|
30
31
|
* [Testing](#testing)
|
31
32
|
* [Integration Test Helpers](#integration-test-helpers)
|
33
|
+
* [Utilities](#utilities)
|
32
34
|
* [Contributing](#contributing)
|
33
35
|
<!--te-->
|
34
36
|
|
37
|
+
# Additional Documentation
|
38
|
+
|
39
|
+
Please see the following for further information not covered by this readme:
|
40
|
+
|
41
|
+
* [Architecture Design](docs/ARCHITECTURE.md)
|
42
|
+
* [Configuration Reference](docs/CONFIGURATION.md)
|
43
|
+
* [Database Backend Feature](docs/DATABASE_BACKEND.md)
|
44
|
+
* [Upgrading Deimos](docs/UPGRADING.md)
|
45
|
+
|
35
46
|
# Installation
|
36
47
|
|
37
48
|
Add this line to your application's Gemfile:
|
@@ -108,6 +119,7 @@ class MyProducer < Deimos::Producer
|
|
108
119
|
'some-key2' => an_object.bar
|
109
120
|
}
|
110
121
|
# You can also publish an array with self.publish_list(payloads)
|
122
|
+
# You may specify the topic here with self.publish(payload, topic: 'my-topic')
|
111
123
|
self.publish(payload)
|
112
124
|
end
|
113
125
|
|
@@ -481,6 +493,12 @@ class WhateverController < ApplicationController
|
|
481
493
|
# will look for: my.namespace.requests.Index.avsc
|
482
494
|
# my.namespace.responses.Index.avsc
|
483
495
|
|
496
|
+
# Can use mapping to change the schema but keep the namespaces,
|
497
|
+
# i.e. use the same schema name across the two namespaces
|
498
|
+
schemas create: 'CreateTopic'
|
499
|
+
# will look for: my.namespace.requests.CreateTopic.avsc
|
500
|
+
# my.namespace.responses.CreateTopic.avsc
|
501
|
+
|
484
502
|
# If all routes use the default, you can add them all at once
|
485
503
|
schemas :index, :show, :update
|
486
504
|
|
@@ -921,6 +939,17 @@ Deimos::TestHelpers.schemas_compatible?(schema1, schema2)
|
|
921
939
|
|
922
940
|
### Integration Test Helpers
|
923
941
|
|
942
|
+
When running integration tests, you'll want to override the default test helper settings:
|
943
|
+
|
944
|
+
```ruby
|
945
|
+
config.before(:each, :my_integration_metadata) do
|
946
|
+
Deimos.configure do
|
947
|
+
producers.backend :kafka
|
948
|
+
schema.backend :avro_schema_registry
|
949
|
+
end
|
950
|
+
end
|
951
|
+
```
|
952
|
+
|
924
953
|
You can use the `InlineConsumer` class to help with integration testing,
|
925
954
|
with a full external Kafka running.
|
926
955
|
|
@@ -977,13 +1006,24 @@ Deimos::Utils::InlineConsumer.get_messages_for(
|
|
977
1006
|
)
|
978
1007
|
```
|
979
1008
|
|
1009
|
+
## Utilities
|
1010
|
+
|
1011
|
+
You can use your configured schema backend directly if you want to
|
1012
|
+
encode and decode payloads outside of the context of sending messages.
|
1013
|
+
|
1014
|
+
```ruby
|
1015
|
+
backend = Deimos.schema_backend(schema: 'MySchema', namespace: 'com.my-namespace')
|
1016
|
+
encoded = backend.encode(my_payload)
|
1017
|
+
decoded = backend.decode(my_encoded_payload)
|
1018
|
+
coerced = backend.coerce(my_payload) # coerce to correct types
|
1019
|
+
backend.validate(my_payload) # throws an error if not valid
|
1020
|
+
fields = backend.schema_fields # list of fields defined in the schema
|
1021
|
+
```
|
1022
|
+
|
980
1023
|
## Contributing
|
981
1024
|
|
982
1025
|
Bug reports and pull requests are welcome on GitHub at https://github.com/flipp-oss/deimos .
|
983
1026
|
|
984
|
-
We have more information on the [internal architecture](docs/ARCHITECTURE.md) of Deimos
|
985
|
-
for contributors!
|
986
|
-
|
987
1027
|
### Linting
|
988
1028
|
|
989
1029
|
Deimos uses Rubocop to lint the code. Please run Rubocop on your code
|
data/docs/CONFIGURATION.md
CHANGED
@@ -79,6 +79,7 @@ topic|nil|Topic to produce to.
|
|
79
79
|
schema|nil|This is optional but strongly recommended for testing purposes; this will validate against a local schema file used as the reader schema, as well as being able to write tests against this schema. This is recommended since it ensures you are always getting the values you expect.
|
80
80
|
namespace|nil|Namespace of the schema to use when finding it locally.
|
81
81
|
key_config|nil|Configuration hash for message keys. See [Kafka Message Keys](../README.md#installation)
|
82
|
+
disabled|false|Set to true to skip starting an actual listener for this consumer on startup.
|
82
83
|
group_id|nil|ID of the consumer group.
|
83
84
|
max_concurrency|1|Number of threads created for this listener. Each thread will behave as an independent consumer. They don't share any state.
|
84
85
|
start_from_beginning|true|Once the consumer group has checkpointed its progress in the topic's partitions, the consumers will always start from the checkpointed offsets, regardless of config. As such, this setting only applies when the consumer initially starts consuming from a topic
|
data/docs/UPGRADING.md
ADDED
@@ -0,0 +1,128 @@
|
|
1
|
+
# Upgrading Deimos
|
2
|
+
|
3
|
+
## Upgrading from < 1.5.0 to >= 1.5.0
|
4
|
+
|
5
|
+
If you are using Confluent's schema registry to Avro-encode your
|
6
|
+
messages, you will need to manually include the `avro_turf` gem
|
7
|
+
in your Gemfile now.
|
8
|
+
|
9
|
+
This update changes how to interact with Deimos's schema classes.
|
10
|
+
Although these are meant to be internal, they are still "public"
|
11
|
+
and can be used by calling code.
|
12
|
+
|
13
|
+
Before 1.5.0:
|
14
|
+
|
15
|
+
```ruby
|
16
|
+
encoder = Deimos::AvroDataEncoder.new(schema: 'MySchema',
|
17
|
+
namespace: 'com.my-namespace')
|
18
|
+
encoder.encode(my_payload)
|
19
|
+
|
20
|
+
decoder = Deimos::AvroDataDecoder.new(schema: 'MySchema',
|
21
|
+
namespace: 'com.my-namespace')
|
22
|
+
decoder.decode(my_payload)
|
23
|
+
```
|
24
|
+
|
25
|
+
After 1.5.0:
|
26
|
+
```ruby
|
27
|
+
backend = Deimos.schema_backend(schema: 'MySchema', namespace: 'com.my-namespace')
|
28
|
+
backend.encode(my_payload)
|
29
|
+
backend.decode(my_payload)
|
30
|
+
```
|
31
|
+
|
32
|
+
The two classes are different and if you are using them to e.g.
|
33
|
+
inspect Avro schema fields, please look at the source code for the following:
|
34
|
+
* `Deimos::SchemaBackends::Base`
|
35
|
+
* `Deimos::SchemaBackends::AvroBase`
|
36
|
+
* `Deimos::SchemaBackends::AvroSchemaRegistry`
|
37
|
+
|
38
|
+
Deprecated `Deimos::TestHelpers.sent_messages` in favor of
|
39
|
+
`Deimos::Backends::Test.sent_messages`.
|
40
|
+
|
41
|
+
## Upgrading from < 1.4.0 to >= 1.4.0
|
42
|
+
|
43
|
+
Previously, configuration was handled as follows:
|
44
|
+
* Kafka configuration, including listeners, lived in `phobos.yml`
|
45
|
+
* Additional Deimos configuration would live in an initializer, e.g. `kafka.rb`
|
46
|
+
* Producer and consumer configuration lived in each individual producer and consumer
|
47
|
+
|
48
|
+
As of 1.4.0, all configuration is centralized in one initializer
|
49
|
+
file, using default configuration.
|
50
|
+
|
51
|
+
Before 1.4.0:
|
52
|
+
```yaml
|
53
|
+
# config/phobos.yml
|
54
|
+
logger:
|
55
|
+
file: log/phobos.log
|
56
|
+
level: debug
|
57
|
+
ruby_kafka:
|
58
|
+
level: debug
|
59
|
+
|
60
|
+
kafka:
|
61
|
+
client_id: phobos
|
62
|
+
connect_timeout: 15
|
63
|
+
socket_timeout: 15
|
64
|
+
|
65
|
+
producer:
|
66
|
+
ack_timeout: 5
|
67
|
+
required_acks: :all
|
68
|
+
...
|
69
|
+
|
70
|
+
listeners:
|
71
|
+
- handler: ConsumerTest::MyConsumer
|
72
|
+
topic: my_consume_topic
|
73
|
+
group_id: my_group_id
|
74
|
+
- handler: ConsumerTest::MyBatchConsumer
|
75
|
+
topic: my_batch_consume_topic
|
76
|
+
group_id: my_batch_group_id
|
77
|
+
delivery: inline_batch
|
78
|
+
```
|
79
|
+
|
80
|
+
```ruby
|
81
|
+
# kafka.rb
|
82
|
+
Deimos.configure do |config|
|
83
|
+
config.reraise_consumer_errors = true
|
84
|
+
config.logger = Rails.logger
|
85
|
+
...
|
86
|
+
end
|
87
|
+
|
88
|
+
# my_consumer.rb
|
89
|
+
class ConsumerTest::MyConsumer < Deimos::Producer
|
90
|
+
namespace 'com.my-namespace'
|
91
|
+
schema 'MySchema'
|
92
|
+
topic 'MyTopic'
|
93
|
+
key_config field: :id
|
94
|
+
end
|
95
|
+
```
|
96
|
+
|
97
|
+
After 1.4.0:
|
98
|
+
```ruby
|
99
|
+
kafka.rb
|
100
|
+
Deimos.configure do
|
101
|
+
logger Rails.logger
|
102
|
+
kafka do
|
103
|
+
client_id 'phobos'
|
104
|
+
connect_timeout 15
|
105
|
+
socket_timeout 15
|
106
|
+
end
|
107
|
+
producers.ack_timeout 5
|
108
|
+
producers.required_acks :all
|
109
|
+
...
|
110
|
+
consumer do
|
111
|
+
class_name 'ConsumerTest::MyConsumer'
|
112
|
+
topic 'my_consume_topic'
|
113
|
+
group_id 'my_group_id'
|
114
|
+
namespace 'com.my-namespace'
|
115
|
+
schema 'MySchema'
|
116
|
+
topic 'MyTopic'
|
117
|
+
key_config field: :id
|
118
|
+
end
|
119
|
+
...
|
120
|
+
end
|
121
|
+
```
|
122
|
+
|
123
|
+
Note that the old configuration way *will* work if you set
|
124
|
+
`config.phobos_config_file = "config/phobos.yml"`. You will
|
125
|
+
get a number of deprecation notices, however. You can also still
|
126
|
+
set the topic, namespace, etc. on the producer/consumer class,
|
127
|
+
but it's much more convenient to centralize these configs
|
128
|
+
in one place to see what your app does.
|
@@ -245,6 +245,13 @@ module Deimos
|
|
245
245
|
|
246
246
|
# Configure the settings with values.
|
247
247
|
def configure(&block)
|
248
|
+
if defined?(Rake) && defined?(Rake.application)
|
249
|
+
tasks = Rake.application.top_level_tasks
|
250
|
+
if tasks.any? { |t| %w(assets webpacker yarn).include?(t.split(':').first) }
|
251
|
+
puts 'Skipping Deimos configuration since we are in JS/CSS compilation'
|
252
|
+
return
|
253
|
+
end
|
254
|
+
end
|
248
255
|
config.run_callbacks(:configure) do
|
249
256
|
config.instance_eval(&block)
|
250
257
|
end
|
@@ -319,6 +319,10 @@ module Deimos
|
|
319
319
|
# Key configuration (see docs).
|
320
320
|
# @return [Hash]
|
321
321
|
setting :key_config
|
322
|
+
# Set to true to ignore the consumer in the Phobos config and not actually start up a
|
323
|
+
# listener.
|
324
|
+
# @return [Boolean]
|
325
|
+
setting :disabled, false
|
322
326
|
|
323
327
|
# These are the phobos "listener" configs. See CONFIGURATION.md for more
|
324
328
|
# info.
|
@@ -63,8 +63,10 @@ module Deimos
|
|
63
63
|
}
|
64
64
|
|
65
65
|
p_config[:listeners] = self.consumer_objects.map do |consumer|
|
66
|
+
next nil if consumer.disabled
|
67
|
+
|
66
68
|
hash = consumer.to_h.reject do |k, _|
|
67
|
-
%i(class_name schema namespace key_config backoff).include?(k)
|
69
|
+
%i(class_name schema namespace key_config backoff disabled).include?(k)
|
68
70
|
end
|
69
71
|
hash = hash.map { |k, v| [k, v.is_a?(Symbol) ? v.to_s : v] }.to_h
|
70
72
|
hash[:handler] = consumer.class_name
|
@@ -73,6 +75,7 @@ module Deimos
|
|
73
75
|
end
|
74
76
|
hash
|
75
77
|
end
|
78
|
+
p_config[:listeners].compact!
|
76
79
|
|
77
80
|
if self.kafka.ssl.enabled
|
78
81
|
%w(ca_cert client_cert client_cert_key).each do |key|
|
data/lib/deimos/producer.rb
CHANGED
@@ -87,8 +87,9 @@ module Deimos
|
|
87
87
|
|
88
88
|
# Publish the payload to the topic.
|
89
89
|
# @param payload [Hash] with an optional payload_key hash key.
|
90
|
-
|
91
|
-
|
90
|
+
# @param topic [String] if specifying the topic
|
91
|
+
def publish(payload, topic: self.topic)
|
92
|
+
publish_list([payload], topic: topic)
|
92
93
|
end
|
93
94
|
|
94
95
|
# Publish a list of messages.
|
@@ -97,11 +98,14 @@ module Deimos
|
|
97
98
|
# whether to publish synchronously.
|
98
99
|
# @param force_send [Boolean] if true, ignore the configured backend
|
99
100
|
# and send immediately to Kafka.
|
100
|
-
|
101
|
+
# @param topic [String] if specifying the topic
|
102
|
+
def publish_list(payloads, sync: nil, force_send: false, topic: self.topic)
|
101
103
|
return if Deimos.config.kafka.seed_brokers.blank? ||
|
102
104
|
Deimos.config.producers.disabled ||
|
103
105
|
Deimos.producers_disabled?(self)
|
104
106
|
|
107
|
+
raise 'Topic not specified. Please specify the topic.' if topic.blank?
|
108
|
+
|
105
109
|
backend_class = determine_backend_class(sync, force_send)
|
106
110
|
Deimos.instrument(
|
107
111
|
'encode_messages',
|
@@ -110,7 +114,7 @@ module Deimos
|
|
110
114
|
payloads: payloads
|
111
115
|
) do
|
112
116
|
messages = Array(payloads).map { |p| Deimos::Message.new(p, self) }
|
113
|
-
messages.each(
|
117
|
+
messages.each { |m| _process_message(m, topic) }
|
114
118
|
messages.in_groups_of(MAX_BATCH_SIZE, false) do |batch|
|
115
119
|
self.produce_batch(backend_class, batch)
|
116
120
|
end
|
@@ -163,7 +167,8 @@ module Deimos
|
|
163
167
|
private
|
164
168
|
|
165
169
|
# @param message [Message]
|
166
|
-
|
170
|
+
# @param topic [String]
|
171
|
+
def _process_message(message, topic)
|
167
172
|
# this violates the Law of Demeter but it has to happen in a very
|
168
173
|
# specific order and requires a bunch of methods on the producer
|
169
174
|
# to work correctly.
|
@@ -175,12 +180,12 @@ module Deimos
|
|
175
180
|
message.payload = nil if message.payload.blank?
|
176
181
|
message.coerce_fields(encoder)
|
177
182
|
message.encoded_key = _encode_key(message.key)
|
178
|
-
message.topic =
|
183
|
+
message.topic = topic
|
179
184
|
message.encoded_payload = if message.payload.nil?
|
180
185
|
nil
|
181
186
|
else
|
182
187
|
encoder.encode(message.payload,
|
183
|
-
topic: "#{config[:topic]}-value")
|
188
|
+
topic: "#{Deimos.config.producers.topic_prefix}#{config[:topic]}-value")
|
184
189
|
end
|
185
190
|
end
|
186
191
|
|
@@ -198,9 +203,9 @@ module Deimos
|
|
198
203
|
end
|
199
204
|
|
200
205
|
if config[:key_field]
|
201
|
-
encoder.encode_key(config[:key_field], key, topic: "#{config[:topic]}-key")
|
206
|
+
encoder.encode_key(config[:key_field], key, topic: "#{Deimos.config.producers.topic_prefix}#{config[:topic]}-key")
|
202
207
|
elsif config[:key_schema]
|
203
|
-
key_encoder.encode(key, topic: "#{config[:topic]}-key")
|
208
|
+
key_encoder.encode(key, topic: "#{Deimos.config.producers.topic_prefix}#{config[:topic]}-key")
|
204
209
|
else
|
205
210
|
key
|
206
211
|
end
|
@@ -15,7 +15,7 @@ module Deimos
|
|
15
15
|
|
16
16
|
# @override
|
17
17
|
def encode_payload(payload, schema: nil, topic: nil)
|
18
|
-
avro_turf_messaging.encode(payload, schema_name: schema, subject: topic)
|
18
|
+
avro_turf_messaging.encode(payload, schema_name: schema, subject: topic || schema)
|
19
19
|
end
|
20
20
|
|
21
21
|
private
|
data/lib/deimos/test_helpers.rb
CHANGED
@@ -30,18 +30,19 @@ module Deimos
|
|
30
30
|
d_config.consumers.reraise_errors = true
|
31
31
|
d_config.kafka.seed_brokers ||= ['test_broker']
|
32
32
|
d_config.schema.backend = Deimos.schema_backend_class.mock_backend
|
33
|
+
d_config.producers.backend = :test
|
33
34
|
end
|
34
35
|
end
|
35
|
-
end
|
36
36
|
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
37
|
+
config.before(:each) do
|
38
|
+
client = double('client').as_null_object
|
39
|
+
allow(client).to receive(:time) do |*_args, &block|
|
40
|
+
block.call
|
41
|
+
end
|
42
|
+
Deimos::Backends::Test.sent_messages.clear
|
41
43
|
end
|
42
|
-
Deimos.configure { |c| c.producers.backend = :test }
|
43
|
-
Deimos::Backends::Test.sent_messages.clear
|
44
44
|
end
|
45
|
+
|
45
46
|
end
|
46
47
|
|
47
48
|
# @deprecated
|
@@ -28,14 +28,18 @@ module Deimos
|
|
28
28
|
|
29
29
|
# Indicate which schemas should be assigned to actions.
|
30
30
|
# @param actions [Symbol]
|
31
|
+
# @param kwactions [String]
|
31
32
|
# @param request [String]
|
32
33
|
# @param response [String]
|
33
|
-
def schemas(*actions, request: nil, response: nil)
|
34
|
+
def schemas(*actions, request: nil, response: nil, **kwactions)
|
34
35
|
actions.each do |action|
|
35
36
|
request ||= action.to_s.titleize
|
36
37
|
response ||= action.to_s.titleize
|
37
38
|
schema_mapping[action.to_s] = { request: request, response: response }
|
38
39
|
end
|
40
|
+
kwactions.each do |key, val|
|
41
|
+
schema_mapping[key.to_s] = { request: val, response: val }
|
42
|
+
end
|
39
43
|
end
|
40
44
|
|
41
45
|
# @return [Hash<Symbol, String>]
|
data/lib/deimos/version.rb
CHANGED
@@ -6,6 +6,14 @@ class MyConfigConsumer < Deimos::Consumer
|
|
6
6
|
def consume
|
7
7
|
end
|
8
8
|
end
|
9
|
+
|
10
|
+
# Mock consumer 2
|
11
|
+
class MyConfigConsumer2 < Deimos::Consumer
|
12
|
+
# :no-doc:
|
13
|
+
def consume
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
9
17
|
describe Deimos, 'configuration' do
|
10
18
|
it 'should configure with deprecated fields' do
|
11
19
|
logger = Logger.new(nil)
|
@@ -171,6 +179,13 @@ describe Deimos, 'configuration' do
|
|
171
179
|
offset_retention_time 13
|
172
180
|
heartbeat_interval 13
|
173
181
|
end
|
182
|
+
consumer do
|
183
|
+
disabled true
|
184
|
+
class_name 'MyConfigConsumer2'
|
185
|
+
schema 'blah2'
|
186
|
+
topic 'blah2'
|
187
|
+
group_id 'myconsumerid2'
|
188
|
+
end
|
174
189
|
end
|
175
190
|
|
176
191
|
expect(described_class.config.phobos_config).
|
data/spec/producer_spec.rb
CHANGED
@@ -64,6 +64,14 @@ module ProducerTest
|
|
64
64
|
end
|
65
65
|
stub_const('MyErrorProducer', producer_class)
|
66
66
|
|
67
|
+
producer_class = Class.new(Deimos::Producer) do
|
68
|
+
schema 'MySchema'
|
69
|
+
namespace 'com.my-namespace'
|
70
|
+
topic nil
|
71
|
+
key_config none: true
|
72
|
+
end
|
73
|
+
stub_const('MyNoTopicProducer', producer_class)
|
74
|
+
|
67
75
|
end
|
68
76
|
|
69
77
|
it 'should fail on invalid message with error handler' do
|
@@ -102,6 +110,33 @@ module ProducerTest
|
|
102
110
|
expect('my-topic').not_to have_sent('test_id' => 'foo2', 'some_int' => 123)
|
103
111
|
end
|
104
112
|
|
113
|
+
it 'should allow setting the topic from publish_list' do
|
114
|
+
expect(described_class).to receive(:produce_batch).once.with(
|
115
|
+
Deimos::Backends::Test,
|
116
|
+
[
|
117
|
+
Deimos::Message.new({ 'test_id' => 'foo', 'some_int' => 123 },
|
118
|
+
MyProducer,
|
119
|
+
topic: 'a-new-topic',
|
120
|
+
partition_key: 'foo',
|
121
|
+
key: 'foo'),
|
122
|
+
Deimos::Message.new({ 'test_id' => 'bar', 'some_int' => 124 },
|
123
|
+
MyProducer,
|
124
|
+
topic: 'a-new-topic',
|
125
|
+
partition_key: 'bar',
|
126
|
+
key: 'bar')
|
127
|
+
]
|
128
|
+
).and_call_original
|
129
|
+
|
130
|
+
MyProducer.publish_list(
|
131
|
+
[{ 'test_id' => 'foo', 'some_int' => 123 },
|
132
|
+
{ 'test_id' => 'bar', 'some_int' => 124 }],
|
133
|
+
topic: 'a-new-topic'
|
134
|
+
)
|
135
|
+
expect('a-new-topic').to have_sent('test_id' => 'foo', 'some_int' => 123)
|
136
|
+
expect('my-topic').not_to have_sent('test_id' => 'foo', 'some_int' => 123)
|
137
|
+
expect('my-topic').not_to have_sent('test_id' => 'foo2', 'some_int' => 123)
|
138
|
+
end
|
139
|
+
|
105
140
|
it 'should add a message ID' do
|
106
141
|
payload = { 'test_id' => 'foo',
|
107
142
|
'some_int' => 123,
|
@@ -201,6 +236,7 @@ module ProducerTest
|
|
201
236
|
end
|
202
237
|
|
203
238
|
it 'should encode the key' do
|
239
|
+
Deimos.configure { |c| c.producers.topic_prefix = nil }
|
204
240
|
expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'foo', topic: 'my-topic-key')
|
205
241
|
expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'bar', topic: 'my-topic-key')
|
206
242
|
expect(MyProducer.encoder).to receive(:encode).with({
|
@@ -218,6 +254,21 @@ module ProducerTest
|
|
218
254
|
)
|
219
255
|
end
|
220
256
|
|
257
|
+
it 'should encode the key with topic prefix' do
|
258
|
+
Deimos.configure { |c| c.producers.topic_prefix = 'prefix.' }
|
259
|
+
expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'foo', topic: 'prefix.my-topic-key')
|
260
|
+
expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'bar', topic: 'prefix.my-topic-key')
|
261
|
+
expect(MyProducer.encoder).to receive(:encode).with({ 'test_id' => 'foo',
|
262
|
+
'some_int' => 123 },
|
263
|
+
{ topic: 'prefix.my-topic-value' })
|
264
|
+
expect(MyProducer.encoder).to receive(:encode).with({ 'test_id' => 'bar',
|
265
|
+
'some_int' => 124 },
|
266
|
+
{ topic: 'prefix.my-topic-value' })
|
267
|
+
|
268
|
+
MyProducer.publish_list([{ 'test_id' => 'foo', 'some_int' => 123 },
|
269
|
+
{ 'test_id' => 'bar', 'some_int' => 124 }])
|
270
|
+
end
|
271
|
+
|
221
272
|
it 'should not encode with plaintext key' do
|
222
273
|
expect(MyNonEncodedProducer.key_encoder).not_to receive(:encode_key)
|
223
274
|
|
@@ -269,6 +320,31 @@ module ProducerTest
|
|
269
320
|
)
|
270
321
|
end
|
271
322
|
|
323
|
+
it 'should raise error if blank topic is passed in explicitly' do
|
324
|
+
expect {
|
325
|
+
MyProducer.publish_list(
|
326
|
+
[{ 'test_id' => 'foo',
|
327
|
+
'some_int' => 123 },
|
328
|
+
{ 'test_id' => 'bar',
|
329
|
+
'some_int' => 124 }],
|
330
|
+
topic: ''
|
331
|
+
)
|
332
|
+
}.to raise_error(RuntimeError,
|
333
|
+
'Topic not specified. Please specify the topic.')
|
334
|
+
end
|
335
|
+
|
336
|
+
it 'should raise error if the producer has not been initialized with a topic' do
|
337
|
+
expect {
|
338
|
+
MyNoTopicProducer.publish_list(
|
339
|
+
[{ 'test_id' => 'foo',
|
340
|
+
'some_int' => 123 },
|
341
|
+
{ 'test_id' => 'bar',
|
342
|
+
'some_int' => 124 }]
|
343
|
+
)
|
344
|
+
}.to raise_error(RuntimeError,
|
345
|
+
'Topic not specified. Please specify the topic.')
|
346
|
+
end
|
347
|
+
|
272
348
|
it 'should error with nothing set' do
|
273
349
|
expect {
|
274
350
|
MyErrorProducer.publish_list(
|
data/spec/spec_helper.rb
CHANGED
@@ -183,6 +183,7 @@ RSpec.configure do |config|
|
|
183
183
|
config.before(:each) do
|
184
184
|
Deimos.config.reset!
|
185
185
|
Deimos.configure do |deimos_config|
|
186
|
+
deimos_config.producers.backend = :test
|
186
187
|
deimos_config.phobos_config_file = File.join(File.dirname(__FILE__), 'phobos.yml')
|
187
188
|
deimos_config.schema.path = File.join(File.expand_path(__dir__), 'schemas')
|
188
189
|
deimos_config.consumers.reraise_errors = true
|
@@ -17,6 +17,7 @@ RSpec.describe Deimos::Utils::SchemaControllerMixin, type: :controller do
|
|
17
17
|
request_namespace 'com.my-namespace.request'
|
18
18
|
response_namespace 'com.my-namespace.response'
|
19
19
|
schemas :index, :show
|
20
|
+
schemas create: 'CreateTopic'
|
20
21
|
schemas :update, request: 'UpdateRequest', response: 'UpdateResponse'
|
21
22
|
|
22
23
|
# :nodoc:
|
@@ -29,6 +30,11 @@ RSpec.describe Deimos::Utils::SchemaControllerMixin, type: :controller do
|
|
29
30
|
render_schema({ 'response_id' => payload[:request_id] + ' dad' })
|
30
31
|
end
|
31
32
|
|
33
|
+
# :nodoc:
|
34
|
+
def create
|
35
|
+
render_schema({ 'response_id' => payload[:request_id] + ' bro' })
|
36
|
+
end
|
37
|
+
|
32
38
|
# :nodoc:
|
33
39
|
def update
|
34
40
|
render_schema({ 'update_response_id' => payload[:update_request_id] + ' sis' })
|
@@ -65,4 +71,14 @@ RSpec.describe Deimos::Utils::SchemaControllerMixin, type: :controller do
|
|
65
71
|
expect(response_backend.decode(response.body)).to eq({ 'update_response_id' => 'hi sis' })
|
66
72
|
end
|
67
73
|
|
74
|
+
it 'should render the correct response for create' do
|
75
|
+
request_backend = Deimos.schema_backend(schema: 'CreateTopic',
|
76
|
+
namespace: 'com.my-namespace.request')
|
77
|
+
response_backend = Deimos.schema_backend(schema: 'CreateTopic',
|
78
|
+
namespace: 'com.my-namespace.response')
|
79
|
+
request.content_type = 'avro/binary'
|
80
|
+
post :create, params: { id: 1 }, body: request_backend.encode({ 'request_id' => 'hi' })
|
81
|
+
expect(response_backend.decode(response.body)).to eq({ 'response_id' => 'hi bro' })
|
82
|
+
end
|
83
|
+
|
68
84
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: deimos-ruby
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.8.
|
4
|
+
version: 1.8.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Daniel Orner
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-09-25 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: avro_turf
|
@@ -349,6 +349,7 @@ files:
|
|
349
349
|
- docs/CONFIGURATION.md
|
350
350
|
- docs/DATABASE_BACKEND.md
|
351
351
|
- docs/PULL_REQUEST_TEMPLATE.md
|
352
|
+
- docs/UPGRADING.md
|
352
353
|
- lib/deimos.rb
|
353
354
|
- lib/deimos/active_record_consume/batch_consumption.rb
|
354
355
|
- lib/deimos/active_record_consume/batch_slicer.rb
|
@@ -453,8 +454,10 @@ files:
|
|
453
454
|
- spec/schemas/com/my-namespace/Wibble.avsc
|
454
455
|
- spec/schemas/com/my-namespace/Widget.avsc
|
455
456
|
- spec/schemas/com/my-namespace/WidgetTheSecond.avsc
|
457
|
+
- spec/schemas/com/my-namespace/request/CreateTopic.avsc
|
456
458
|
- spec/schemas/com/my-namespace/request/Index.avsc
|
457
459
|
- spec/schemas/com/my-namespace/request/UpdateRequest.avsc
|
460
|
+
- spec/schemas/com/my-namespace/response/CreateTopic.avsc
|
458
461
|
- spec/schemas/com/my-namespace/response/Index.avsc
|
459
462
|
- spec/schemas/com/my-namespace/response/UpdateResponse.avsc
|
460
463
|
- spec/spec_helper.rb
|
@@ -483,9 +486,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
483
486
|
version: '0'
|
484
487
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
485
488
|
requirements:
|
486
|
-
- - "
|
489
|
+
- - ">="
|
487
490
|
- !ruby/object:Gem::Version
|
488
|
-
version:
|
491
|
+
version: '0'
|
489
492
|
requirements: []
|
490
493
|
rubygems_version: 3.1.3
|
491
494
|
signing_key:
|
@@ -534,8 +537,10 @@ test_files:
|
|
534
537
|
- spec/schemas/com/my-namespace/Wibble.avsc
|
535
538
|
- spec/schemas/com/my-namespace/Widget.avsc
|
536
539
|
- spec/schemas/com/my-namespace/WidgetTheSecond.avsc
|
540
|
+
- spec/schemas/com/my-namespace/request/CreateTopic.avsc
|
537
541
|
- spec/schemas/com/my-namespace/request/Index.avsc
|
538
542
|
- spec/schemas/com/my-namespace/request/UpdateRequest.avsc
|
543
|
+
- spec/schemas/com/my-namespace/response/CreateTopic.avsc
|
539
544
|
- spec/schemas/com/my-namespace/response/Index.avsc
|
540
545
|
- spec/schemas/com/my-namespace/response/UpdateResponse.avsc
|
541
546
|
- spec/spec_helper.rb
|