deimos-ruby 1.8.1.pre.beta7 → 1.8.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6ed6bdbc70b4e29dfd893853d2df53ae6779fe3a95adc85a099b270674908135
4
- data.tar.gz: ca1da1180acac7a76cb3abd4ce9c4ab16712b16e94f88b79ae5c952d98a74500
3
+ metadata.gz: 237cb6cd22b6a057003fedae1caeec81c999a3d9b90d837bb8b0a4add43ca4e2
4
+ data.tar.gz: fab51933cdcd5c5fd10b8c92bbc67854fe56f23cdbec41660579f18b9075e09c
5
5
  SHA512:
6
- metadata.gz: fd3b5728ce13e60b35ffc0b7dd025e402fa4bbd760edc000ca8cd587bca0c7f39d136eba951ebd631a711cf79171716021f04bda38c8262318ab344d8452be34
7
- data.tar.gz: '0305200088b9d463c92562be1d9bf27d2727a06c10a147fdb6bace097adc795b2cfca6d5942eb4f06ee0da316e5a41db26b1fd04698d63987e2b45fdf7aea4c7'
6
+ metadata.gz: 193b9b593b4f92edecc6c239eb6106fbe40e0acd1453694508f3fb02525166c39e4f1eba71230362493cd1647ab19e98a63ecf55f5371b844e9d0c38ffa5b2ea
7
+ data.tar.gz: e7d42ec7ec2bc864735c9230a925685935678d2dd0f3918426926dc9f358d5eb810dd5b324bcacaf9201c78bdfbbfa58037a4cac4cf4ca66257737440b3d2c97
@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## UNRELEASED
9
9
 
10
+ ## 1.8.2 - 2020-09-25
11
+
12
+ ### Features :star:
13
+ - Add "disabled" config field to consumers to allow disabling
14
+ individual consumers without having to comment out their
15
+ entries and possibly affecting unit tests.
16
+
17
+ ### Fixes :wrench:
18
+ - Prepend topic_prefix while encoding messages
19
+ (fixes [#37](https://github.com/flipp-oss/deimos/issues/37))
20
+ - Raise error if producing without a topic
21
+ (fixes [#50](https://github.com/flipp-oss/deimos/issues/50))
22
+ - Don't try to load producers/consumers when running rake tasks involving webpacker or assets
23
+
24
+ ## 1.8.2-beta2 - 2020-09-15
25
+
26
+ ### Features :star:
27
+
28
+ - Add details on using schema backend directly in README.
29
+ - Default to the provided schema if topic is not provided when
30
+ encoding to `AvroSchemaRegistry`.
31
+ - Add mapping syntax for the `schema` call in `SchemaControllerMixin`.
32
+
33
+ ## 1.8.2-beta1 - 2020-09-09
34
+
35
+ ### Features :star:
36
+
37
+ - Added the ability to specify the topic for `publish`
38
+ and `publish_list` in a producer
39
+
40
+ ## 1.8.1-beta9 - 2020-08-27
41
+
42
+ ### Fixes :wrench:
43
+ - Moved the TestHelpers hook to `before(:suite)` to allow for
44
+ overriding e.g. in integration tests.
45
+
10
46
  ## 1.8.1-beta7 - 2020-08-25
11
47
 
12
48
  ### Fixes :wrench:
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- deimos-ruby (1.8.1.pre.beta7)
4
+ deimos-ruby (1.8.2)
5
5
  avro_turf (~> 0.11)
6
6
  phobos (~> 1.9)
7
7
  ruby-kafka (~> 0.7)
data/README.md CHANGED
@@ -11,6 +11,7 @@ a useful toolbox of goodies for Ruby-based Kafka development.
11
11
  Built on Phobos and hence Ruby-Kafka.
12
12
 
13
13
  <!--ts-->
14
+ * [Additional Documentation](#additional-documentation)
14
15
  * [Installation](#installation)
15
16
  * [Versioning](#versioning)
16
17
  * [Configuration](#configuration)
@@ -29,9 +30,19 @@ Built on Phobos and hence Ruby-Kafka.
29
30
  * [Metrics](#metrics)
30
31
  * [Testing](#testing)
31
32
  * [Integration Test Helpers](#integration-test-helpers)
33
+ * [Utilities](#utilities)
32
34
  * [Contributing](#contributing)
33
35
  <!--te-->
34
36
 
37
+ # Additional Documentation
38
+
39
+ Please see the following for further information not covered by this readme:
40
+
41
+ * [Architecture Design](docs/ARCHITECTURE.md)
42
+ * [Configuration Reference](docs/CONFIGURATION.md)
43
+ * [Database Backend Feature](docs/DATABASE_BACKEND.md)
44
+ * [Upgrading Deimos](docs/UPGRADING.md)
45
+
35
46
  # Installation
36
47
 
37
48
  Add this line to your application's Gemfile:
@@ -108,6 +119,7 @@ class MyProducer < Deimos::Producer
108
119
  'some-key2' => an_object.bar
109
120
  }
110
121
  # You can also publish an array with self.publish_list(payloads)
122
+ # You may specify the topic here with self.publish(payload, topic: 'my-topic')
111
123
  self.publish(payload)
112
124
  end
113
125
 
@@ -481,6 +493,12 @@ class WhateverController < ApplicationController
481
493
  # will look for: my.namespace.requests.Index.avsc
482
494
  # my.namespace.responses.Index.avsc
483
495
 
496
+ # Can use mapping to change the schema but keep the namespaces,
497
+ # i.e. use the same schema name across the two namespaces
498
+ schemas create: 'CreateTopic'
499
+ # will look for: my.namespace.requests.CreateTopic.avsc
500
+ # my.namespace.responses.CreateTopic.avsc
501
+
484
502
  # If all routes use the default, you can add them all at once
485
503
  schemas :index, :show, :update
486
504
 
@@ -921,6 +939,17 @@ Deimos::TestHelpers.schemas_compatible?(schema1, schema2)
921
939
 
922
940
  ### Integration Test Helpers
923
941
 
942
+ When running integration tests, you'll want to override the default test helper settings:
943
+
944
+ ```ruby
945
+ config.before(:each, :my_integration_metadata) do
946
+ Deimos.configure do
947
+ producers.backend :kafka
948
+ schema.backend :avro_schema_registry
949
+ end
950
+ end
951
+ ```
952
+
924
953
  You can use the `InlineConsumer` class to help with integration testing,
925
954
  with a full external Kafka running.
926
955
 
@@ -977,13 +1006,24 @@ Deimos::Utils::InlineConsumer.get_messages_for(
977
1006
  )
978
1007
  ```
979
1008
 
1009
+ ## Utilities
1010
+
1011
+ You can use your configured schema backend directly if you want to
1012
+ encode and decode payloads outside of the context of sending messages.
1013
+
1014
+ ```ruby
1015
+ backend = Deimos.schema_backend(schema: 'MySchema', namespace: 'com.my-namespace')
1016
+ encoded = backend.encode(my_payload)
1017
+ decoded = backend.decode(my_encoded_payload)
1018
+ coerced = backend.coerce(my_payload) # coerce to correct types
1019
+ backend.validate(my_payload) # throws an error if not valid
1020
+ fields = backend.schema_fields # list of fields defined in the schema
1021
+ ```
1022
+
980
1023
  ## Contributing
981
1024
 
982
1025
  Bug reports and pull requests are welcome on GitHub at https://github.com/flipp-oss/deimos .
983
1026
 
984
- We have more information on the [internal architecture](docs/ARCHITECTURE.md) of Deimos
985
- for contributors!
986
-
987
1027
  ### Linting
988
1028
 
989
1029
  Deimos uses Rubocop to lint the code. Please run Rubocop on your code
@@ -79,6 +79,7 @@ topic|nil|Topic to produce to.
79
79
  schema|nil|This is optional but strongly recommended for testing purposes; this will validate against a local schema file used as the reader schema, as well as being able to write tests against this schema. This is recommended since it ensures you are always getting the values you expect.
80
80
  namespace|nil|Namespace of the schema to use when finding it locally.
81
81
  key_config|nil|Configuration hash for message keys. See [Kafka Message Keys](../README.md#installation)
82
+ disabled|false|Set to true to skip starting an actual listener for this consumer on startup.
82
83
  group_id|nil|ID of the consumer group.
83
84
  max_concurrency|1|Number of threads created for this listener. Each thread will behave as an independent consumer. They don't share any state.
84
85
  start_from_beginning|true|Once the consumer group has checkpointed its progress in the topic's partitions, the consumers will always start from the checkpointed offsets, regardless of config. As such, this setting only applies when the consumer initially starts consuming from a topic
@@ -0,0 +1,128 @@
1
+ # Upgrading Deimos
2
+
3
+ ## Upgrading from < 1.5.0 to >= 1.5.0
4
+
5
+ If you are using Confluent's schema registry to Avro-encode your
6
+ messages, you will need to manually include the `avro_turf` gem
7
+ in your Gemfile now.
8
+
9
+ This update changes how to interact with Deimos's schema classes.
10
+ Although these are meant to be internal, they are still "public"
11
+ and can be used by calling code.
12
+
13
+ Before 1.5.0:
14
+
15
+ ```ruby
16
+ encoder = Deimos::AvroDataEncoder.new(schema: 'MySchema',
17
+ namespace: 'com.my-namespace')
18
+ encoder.encode(my_payload)
19
+
20
+ decoder = Deimos::AvroDataDecoder.new(schema: 'MySchema',
21
+ namespace: 'com.my-namespace')
22
+ decoder.decode(my_payload)
23
+ ```
24
+
25
+ After 1.5.0:
26
+ ```ruby
27
+ backend = Deimos.schema_backend(schema: 'MySchema', namespace: 'com.my-namespace')
28
+ backend.encode(my_payload)
29
+ backend.decode(my_payload)
30
+ ```
31
+
32
+ The two classes are different and if you are using them to e.g.
33
+ inspect Avro schema fields, please look at the source code for the following:
34
+ * `Deimos::SchemaBackends::Base`
35
+ * `Deimos::SchemaBackends::AvroBase`
36
+ * `Deimos::SchemaBackends::AvroSchemaRegistry`
37
+
38
+ Deprecated `Deimos::TestHelpers.sent_messages` in favor of
39
+ `Deimos::Backends::Test.sent_messages`.
40
+
41
+ ## Upgrading from < 1.4.0 to >= 1.4.0
42
+
43
+ Previously, configuration was handled as follows:
44
+ * Kafka configuration, including listeners, lived in `phobos.yml`
45
+ * Additional Deimos configuration would live in an initializer, e.g. `kafka.rb`
46
+ * Producer and consumer configuration lived in each individual producer and consumer
47
+
48
+ As of 1.4.0, all configuration is centralized in one initializer
49
+ file, using default configuration.
50
+
51
+ Before 1.4.0:
52
+ ```yaml
53
+ # config/phobos.yml
54
+ logger:
55
+ file: log/phobos.log
56
+ level: debug
57
+ ruby_kafka:
58
+ level: debug
59
+
60
+ kafka:
61
+ client_id: phobos
62
+ connect_timeout: 15
63
+ socket_timeout: 15
64
+
65
+ producer:
66
+ ack_timeout: 5
67
+ required_acks: :all
68
+ ...
69
+
70
+ listeners:
71
+ - handler: ConsumerTest::MyConsumer
72
+ topic: my_consume_topic
73
+ group_id: my_group_id
74
+ - handler: ConsumerTest::MyBatchConsumer
75
+ topic: my_batch_consume_topic
76
+ group_id: my_batch_group_id
77
+ delivery: inline_batch
78
+ ```
79
+
80
+ ```ruby
81
+ # kafka.rb
82
+ Deimos.configure do |config|
83
+ config.reraise_consumer_errors = true
84
+ config.logger = Rails.logger
85
+ ...
86
+ end
87
+
88
+ # my_consumer.rb
89
+ class ConsumerTest::MyConsumer < Deimos::Producer
90
+ namespace 'com.my-namespace'
91
+ schema 'MySchema'
92
+ topic 'MyTopic'
93
+ key_config field: :id
94
+ end
95
+ ```
96
+
97
+ After 1.4.0:
98
+ ```ruby
99
+ kafka.rb
100
+ Deimos.configure do
101
+ logger Rails.logger
102
+ kafka do
103
+ client_id 'phobos'
104
+ connect_timeout 15
105
+ socket_timeout 15
106
+ end
107
+ producers.ack_timeout 5
108
+ producers.required_acks :all
109
+ ...
110
+ consumer do
111
+ class_name 'ConsumerTest::MyConsumer'
112
+ topic 'my_consume_topic'
113
+ group_id 'my_group_id'
114
+ namespace 'com.my-namespace'
115
+ schema 'MySchema'
116
+ topic 'MyTopic'
117
+ key_config field: :id
118
+ end
119
+ ...
120
+ end
121
+ ```
122
+
123
+ Note that the old configuration way *will* work if you set
124
+ `config.phobos_config_file = "config/phobos.yml"`. You will
125
+ get a number of deprecation notices, however. You can also still
126
+ set the topic, namespace, etc. on the producer/consumer class,
127
+ but it's much more convenient to centralize these configs
128
+ in one place to see what your app does.
@@ -245,6 +245,13 @@ module Deimos
245
245
 
246
246
  # Configure the settings with values.
247
247
  def configure(&block)
248
+ if defined?(Rake) && defined?(Rake.application)
249
+ tasks = Rake.application.top_level_tasks
250
+ if tasks.any? { |t| %w(assets webpacker yarn).include?(t.split(':').first) }
251
+ puts 'Skipping Deimos configuration since we are in JS/CSS compilation'
252
+ return
253
+ end
254
+ end
248
255
  config.run_callbacks(:configure) do
249
256
  config.instance_eval(&block)
250
257
  end
@@ -319,6 +319,10 @@ module Deimos
319
319
  # Key configuration (see docs).
320
320
  # @return [Hash]
321
321
  setting :key_config
322
+ # Set to true to ignore the consumer in the Phobos config and not actually start up a
323
+ # listener.
324
+ # @return [Boolean]
325
+ setting :disabled, false
322
326
 
323
327
  # These are the phobos "listener" configs. See CONFIGURATION.md for more
324
328
  # info.
@@ -63,8 +63,10 @@ module Deimos
63
63
  }
64
64
 
65
65
  p_config[:listeners] = self.consumer_objects.map do |consumer|
66
+ next nil if consumer.disabled
67
+
66
68
  hash = consumer.to_h.reject do |k, _|
67
- %i(class_name schema namespace key_config backoff).include?(k)
69
+ %i(class_name schema namespace key_config backoff disabled).include?(k)
68
70
  end
69
71
  hash = hash.map { |k, v| [k, v.is_a?(Symbol) ? v.to_s : v] }.to_h
70
72
  hash[:handler] = consumer.class_name
@@ -73,6 +75,7 @@ module Deimos
73
75
  end
74
76
  hash
75
77
  end
78
+ p_config[:listeners].compact!
76
79
 
77
80
  if self.kafka.ssl.enabled
78
81
  %w(ca_cert client_cert client_cert_key).each do |key|
@@ -87,8 +87,9 @@ module Deimos
87
87
 
88
88
  # Publish the payload to the topic.
89
89
  # @param payload [Hash] with an optional payload_key hash key.
90
- def publish(payload)
91
- publish_list([payload])
90
+ # @param topic [String] if specifying the topic
91
+ def publish(payload, topic: self.topic)
92
+ publish_list([payload], topic: topic)
92
93
  end
93
94
 
94
95
  # Publish a list of messages.
@@ -97,11 +98,14 @@ module Deimos
97
98
  # whether to publish synchronously.
98
99
  # @param force_send [Boolean] if true, ignore the configured backend
99
100
  # and send immediately to Kafka.
100
- def publish_list(payloads, sync: nil, force_send: false)
101
+ # @param topic [String] if specifying the topic
102
+ def publish_list(payloads, sync: nil, force_send: false, topic: self.topic)
101
103
  return if Deimos.config.kafka.seed_brokers.blank? ||
102
104
  Deimos.config.producers.disabled ||
103
105
  Deimos.producers_disabled?(self)
104
106
 
107
+ raise 'Topic not specified. Please specify the topic.' if topic.blank?
108
+
105
109
  backend_class = determine_backend_class(sync, force_send)
106
110
  Deimos.instrument(
107
111
  'encode_messages',
@@ -110,7 +114,7 @@ module Deimos
110
114
  payloads: payloads
111
115
  ) do
112
116
  messages = Array(payloads).map { |p| Deimos::Message.new(p, self) }
113
- messages.each(&method(:_process_message))
117
+ messages.each { |m| _process_message(m, topic) }
114
118
  messages.in_groups_of(MAX_BATCH_SIZE, false) do |batch|
115
119
  self.produce_batch(backend_class, batch)
116
120
  end
@@ -163,7 +167,8 @@ module Deimos
163
167
  private
164
168
 
165
169
  # @param message [Message]
166
- def _process_message(message)
170
+ # @param topic [String]
171
+ def _process_message(message, topic)
167
172
  # this violates the Law of Demeter but it has to happen in a very
168
173
  # specific order and requires a bunch of methods on the producer
169
174
  # to work correctly.
@@ -175,12 +180,12 @@ module Deimos
175
180
  message.payload = nil if message.payload.blank?
176
181
  message.coerce_fields(encoder)
177
182
  message.encoded_key = _encode_key(message.key)
178
- message.topic = self.topic
183
+ message.topic = topic
179
184
  message.encoded_payload = if message.payload.nil?
180
185
  nil
181
186
  else
182
187
  encoder.encode(message.payload,
183
- topic: "#{config[:topic]}-value")
188
+ topic: "#{Deimos.config.producers.topic_prefix}#{config[:topic]}-value")
184
189
  end
185
190
  end
186
191
 
@@ -198,9 +203,9 @@ module Deimos
198
203
  end
199
204
 
200
205
  if config[:key_field]
201
- encoder.encode_key(config[:key_field], key, topic: "#{config[:topic]}-key")
206
+ encoder.encode_key(config[:key_field], key, topic: "#{Deimos.config.producers.topic_prefix}#{config[:topic]}-key")
202
207
  elsif config[:key_schema]
203
- key_encoder.encode(key, topic: "#{config[:topic]}-key")
208
+ key_encoder.encode(key, topic: "#{Deimos.config.producers.topic_prefix}#{config[:topic]}-key")
204
209
  else
205
210
  key
206
211
  end
@@ -15,7 +15,7 @@ module Deimos
15
15
 
16
16
  # @override
17
17
  def encode_payload(payload, schema: nil, topic: nil)
18
- avro_turf_messaging.encode(payload, schema_name: schema, subject: topic)
18
+ avro_turf_messaging.encode(payload, schema_name: schema, subject: topic || schema)
19
19
  end
20
20
 
21
21
  private
@@ -30,18 +30,19 @@ module Deimos
30
30
  d_config.consumers.reraise_errors = true
31
31
  d_config.kafka.seed_brokers ||= ['test_broker']
32
32
  d_config.schema.backend = Deimos.schema_backend_class.mock_backend
33
+ d_config.producers.backend = :test
33
34
  end
34
35
  end
35
- end
36
36
 
37
- before(:each) do
38
- client = double('client').as_null_object
39
- allow(client).to receive(:time) do |*_args, &block|
40
- block.call
37
+ config.before(:each) do
38
+ client = double('client').as_null_object
39
+ allow(client).to receive(:time) do |*_args, &block|
40
+ block.call
41
+ end
42
+ Deimos::Backends::Test.sent_messages.clear
41
43
  end
42
- Deimos.configure { |c| c.producers.backend = :test }
43
- Deimos::Backends::Test.sent_messages.clear
44
44
  end
45
+
45
46
  end
46
47
 
47
48
  # @deprecated
@@ -28,14 +28,18 @@ module Deimos
28
28
 
29
29
  # Indicate which schemas should be assigned to actions.
30
30
  # @param actions [Symbol]
31
+ # @param kwactions [String]
31
32
  # @param request [String]
32
33
  # @param response [String]
33
- def schemas(*actions, request: nil, response: nil)
34
+ def schemas(*actions, request: nil, response: nil, **kwactions)
34
35
  actions.each do |action|
35
36
  request ||= action.to_s.titleize
36
37
  response ||= action.to_s.titleize
37
38
  schema_mapping[action.to_s] = { request: request, response: response }
38
39
  end
40
+ kwactions.each do |key, val|
41
+ schema_mapping[key.to_s] = { request: val, response: val }
42
+ end
39
43
  end
40
44
 
41
45
  # @return [Hash<Symbol, String>]
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Deimos
4
- VERSION = '1.8.1-beta7'
4
+ VERSION = '1.8.2'
5
5
  end
@@ -6,6 +6,14 @@ class MyConfigConsumer < Deimos::Consumer
6
6
  def consume
7
7
  end
8
8
  end
9
+
10
+ # Mock consumer 2
11
+ class MyConfigConsumer2 < Deimos::Consumer
12
+ # :no-doc:
13
+ def consume
14
+ end
15
+ end
16
+
9
17
  describe Deimos, 'configuration' do
10
18
  it 'should configure with deprecated fields' do
11
19
  logger = Logger.new(nil)
@@ -171,6 +179,13 @@ describe Deimos, 'configuration' do
171
179
  offset_retention_time 13
172
180
  heartbeat_interval 13
173
181
  end
182
+ consumer do
183
+ disabled true
184
+ class_name 'MyConfigConsumer2'
185
+ schema 'blah2'
186
+ topic 'blah2'
187
+ group_id 'myconsumerid2'
188
+ end
174
189
  end
175
190
 
176
191
  expect(described_class.config.phobos_config).
@@ -64,6 +64,14 @@ module ProducerTest
64
64
  end
65
65
  stub_const('MyErrorProducer', producer_class)
66
66
 
67
+ producer_class = Class.new(Deimos::Producer) do
68
+ schema 'MySchema'
69
+ namespace 'com.my-namespace'
70
+ topic nil
71
+ key_config none: true
72
+ end
73
+ stub_const('MyNoTopicProducer', producer_class)
74
+
67
75
  end
68
76
 
69
77
  it 'should fail on invalid message with error handler' do
@@ -102,6 +110,33 @@ module ProducerTest
102
110
  expect('my-topic').not_to have_sent('test_id' => 'foo2', 'some_int' => 123)
103
111
  end
104
112
 
113
+ it 'should allow setting the topic from publish_list' do
114
+ expect(described_class).to receive(:produce_batch).once.with(
115
+ Deimos::Backends::Test,
116
+ [
117
+ Deimos::Message.new({ 'test_id' => 'foo', 'some_int' => 123 },
118
+ MyProducer,
119
+ topic: 'a-new-topic',
120
+ partition_key: 'foo',
121
+ key: 'foo'),
122
+ Deimos::Message.new({ 'test_id' => 'bar', 'some_int' => 124 },
123
+ MyProducer,
124
+ topic: 'a-new-topic',
125
+ partition_key: 'bar',
126
+ key: 'bar')
127
+ ]
128
+ ).and_call_original
129
+
130
+ MyProducer.publish_list(
131
+ [{ 'test_id' => 'foo', 'some_int' => 123 },
132
+ { 'test_id' => 'bar', 'some_int' => 124 }],
133
+ topic: 'a-new-topic'
134
+ )
135
+ expect('a-new-topic').to have_sent('test_id' => 'foo', 'some_int' => 123)
136
+ expect('my-topic').not_to have_sent('test_id' => 'foo', 'some_int' => 123)
137
+ expect('my-topic').not_to have_sent('test_id' => 'foo2', 'some_int' => 123)
138
+ end
139
+
105
140
  it 'should add a message ID' do
106
141
  payload = { 'test_id' => 'foo',
107
142
  'some_int' => 123,
@@ -201,6 +236,7 @@ module ProducerTest
201
236
  end
202
237
 
203
238
  it 'should encode the key' do
239
+ Deimos.configure { |c| c.producers.topic_prefix = nil }
204
240
  expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'foo', topic: 'my-topic-key')
205
241
  expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'bar', topic: 'my-topic-key')
206
242
  expect(MyProducer.encoder).to receive(:encode).with({
@@ -218,6 +254,21 @@ module ProducerTest
218
254
  )
219
255
  end
220
256
 
257
+ it 'should encode the key with topic prefix' do
258
+ Deimos.configure { |c| c.producers.topic_prefix = 'prefix.' }
259
+ expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'foo', topic: 'prefix.my-topic-key')
260
+ expect(MyProducer.encoder).to receive(:encode_key).with('test_id', 'bar', topic: 'prefix.my-topic-key')
261
+ expect(MyProducer.encoder).to receive(:encode).with({ 'test_id' => 'foo',
262
+ 'some_int' => 123 },
263
+ { topic: 'prefix.my-topic-value' })
264
+ expect(MyProducer.encoder).to receive(:encode).with({ 'test_id' => 'bar',
265
+ 'some_int' => 124 },
266
+ { topic: 'prefix.my-topic-value' })
267
+
268
+ MyProducer.publish_list([{ 'test_id' => 'foo', 'some_int' => 123 },
269
+ { 'test_id' => 'bar', 'some_int' => 124 }])
270
+ end
271
+
221
272
  it 'should not encode with plaintext key' do
222
273
  expect(MyNonEncodedProducer.key_encoder).not_to receive(:encode_key)
223
274
 
@@ -269,6 +320,31 @@ module ProducerTest
269
320
  )
270
321
  end
271
322
 
323
+ it 'should raise error if blank topic is passed in explicitly' do
324
+ expect {
325
+ MyProducer.publish_list(
326
+ [{ 'test_id' => 'foo',
327
+ 'some_int' => 123 },
328
+ { 'test_id' => 'bar',
329
+ 'some_int' => 124 }],
330
+ topic: ''
331
+ )
332
+ }.to raise_error(RuntimeError,
333
+ 'Topic not specified. Please specify the topic.')
334
+ end
335
+
336
+ it 'should raise error if the producer has not been initialized with a topic' do
337
+ expect {
338
+ MyNoTopicProducer.publish_list(
339
+ [{ 'test_id' => 'foo',
340
+ 'some_int' => 123 },
341
+ { 'test_id' => 'bar',
342
+ 'some_int' => 124 }]
343
+ )
344
+ }.to raise_error(RuntimeError,
345
+ 'Topic not specified. Please specify the topic.')
346
+ end
347
+
272
348
  it 'should error with nothing set' do
273
349
  expect {
274
350
  MyErrorProducer.publish_list(
@@ -0,0 +1,11 @@
1
+ {
2
+ "namespace": "com.my-namespace.request",
3
+ "name": "CreateTopic",
4
+ "type": "record",
5
+ "fields": [
6
+ {
7
+ "name": "request_id",
8
+ "type": "string"
9
+ }
10
+ ]
11
+ }
@@ -0,0 +1,11 @@
1
+ {
2
+ "namespace": "com.my-namespace.response",
3
+ "name": "CreateTopic",
4
+ "type": "record",
5
+ "fields": [
6
+ {
7
+ "name": "response_id",
8
+ "type": "string"
9
+ }
10
+ ]
11
+ }
@@ -183,6 +183,7 @@ RSpec.configure do |config|
183
183
  config.before(:each) do
184
184
  Deimos.config.reset!
185
185
  Deimos.configure do |deimos_config|
186
+ deimos_config.producers.backend = :test
186
187
  deimos_config.phobos_config_file = File.join(File.dirname(__FILE__), 'phobos.yml')
187
188
  deimos_config.schema.path = File.join(File.expand_path(__dir__), 'schemas')
188
189
  deimos_config.consumers.reraise_errors = true
@@ -17,6 +17,7 @@ RSpec.describe Deimos::Utils::SchemaControllerMixin, type: :controller do
17
17
  request_namespace 'com.my-namespace.request'
18
18
  response_namespace 'com.my-namespace.response'
19
19
  schemas :index, :show
20
+ schemas create: 'CreateTopic'
20
21
  schemas :update, request: 'UpdateRequest', response: 'UpdateResponse'
21
22
 
22
23
  # :nodoc:
@@ -29,6 +30,11 @@ RSpec.describe Deimos::Utils::SchemaControllerMixin, type: :controller do
29
30
  render_schema({ 'response_id' => payload[:request_id] + ' dad' })
30
31
  end
31
32
 
33
+ # :nodoc:
34
+ def create
35
+ render_schema({ 'response_id' => payload[:request_id] + ' bro' })
36
+ end
37
+
32
38
  # :nodoc:
33
39
  def update
34
40
  render_schema({ 'update_response_id' => payload[:update_request_id] + ' sis' })
@@ -65,4 +71,14 @@ RSpec.describe Deimos::Utils::SchemaControllerMixin, type: :controller do
65
71
  expect(response_backend.decode(response.body)).to eq({ 'update_response_id' => 'hi sis' })
66
72
  end
67
73
 
74
+ it 'should render the correct response for create' do
75
+ request_backend = Deimos.schema_backend(schema: 'CreateTopic',
76
+ namespace: 'com.my-namespace.request')
77
+ response_backend = Deimos.schema_backend(schema: 'CreateTopic',
78
+ namespace: 'com.my-namespace.response')
79
+ request.content_type = 'avro/binary'
80
+ post :create, params: { id: 1 }, body: request_backend.encode({ 'request_id' => 'hi' })
81
+ expect(response_backend.decode(response.body)).to eq({ 'response_id' => 'hi bro' })
82
+ end
83
+
68
84
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: deimos-ruby
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.8.1.pre.beta7
4
+ version: 1.8.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Daniel Orner
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-08-25 00:00:00.000000000 Z
11
+ date: 2020-09-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: avro_turf
@@ -349,6 +349,7 @@ files:
349
349
  - docs/CONFIGURATION.md
350
350
  - docs/DATABASE_BACKEND.md
351
351
  - docs/PULL_REQUEST_TEMPLATE.md
352
+ - docs/UPGRADING.md
352
353
  - lib/deimos.rb
353
354
  - lib/deimos/active_record_consume/batch_consumption.rb
354
355
  - lib/deimos/active_record_consume/batch_slicer.rb
@@ -453,8 +454,10 @@ files:
453
454
  - spec/schemas/com/my-namespace/Wibble.avsc
454
455
  - spec/schemas/com/my-namespace/Widget.avsc
455
456
  - spec/schemas/com/my-namespace/WidgetTheSecond.avsc
457
+ - spec/schemas/com/my-namespace/request/CreateTopic.avsc
456
458
  - spec/schemas/com/my-namespace/request/Index.avsc
457
459
  - spec/schemas/com/my-namespace/request/UpdateRequest.avsc
460
+ - spec/schemas/com/my-namespace/response/CreateTopic.avsc
458
461
  - spec/schemas/com/my-namespace/response/Index.avsc
459
462
  - spec/schemas/com/my-namespace/response/UpdateResponse.avsc
460
463
  - spec/spec_helper.rb
@@ -483,9 +486,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
483
486
  version: '0'
484
487
  required_rubygems_version: !ruby/object:Gem::Requirement
485
488
  requirements:
486
- - - ">"
489
+ - - ">="
487
490
  - !ruby/object:Gem::Version
488
- version: 1.3.1
491
+ version: '0'
489
492
  requirements: []
490
493
  rubygems_version: 3.1.3
491
494
  signing_key:
@@ -534,8 +537,10 @@ test_files:
534
537
  - spec/schemas/com/my-namespace/Wibble.avsc
535
538
  - spec/schemas/com/my-namespace/Widget.avsc
536
539
  - spec/schemas/com/my-namespace/WidgetTheSecond.avsc
540
+ - spec/schemas/com/my-namespace/request/CreateTopic.avsc
537
541
  - spec/schemas/com/my-namespace/request/Index.avsc
538
542
  - spec/schemas/com/my-namespace/request/UpdateRequest.avsc
543
+ - spec/schemas/com/my-namespace/response/CreateTopic.avsc
539
544
  - spec/schemas/com/my-namespace/response/Index.avsc
540
545
  - spec/schemas/com/my-namespace/response/UpdateResponse.avsc
541
546
  - spec/spec_helper.rb