avromatic 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.overcommit.yml +14 -0
- data/.rubocop.yml +8 -0
- data/CHANGELOG.md +6 -0
- data/README.md +85 -30
- data/Rakefile +4 -4
- data/avromatic.gemspec +22 -20
- data/lib/avromatic/model/attributes.rb +11 -9
- data/lib/avromatic/model/builder.rb +4 -2
- data/lib/avromatic/model/{decoder.rb → message_decoder.rb} +11 -10
- data/lib/avromatic/model/messaging_serialization.rb +62 -0
- data/lib/avromatic/model/raw_serialization.rb +111 -0
- data/lib/avromatic/model/value_object.rb +2 -2
- data/lib/avromatic/model.rb +1 -1
- data/lib/avromatic/version.rb +1 -1
- data/lib/avromatic.rb +4 -2
- metadata +35 -4
- data/lib/avromatic/model/serialization.rb +0 -86
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4d18b7d880de4f22e4615d3d547551953f24bec7
|
4
|
+
data.tar.gz: 8a970619e0c045513276a54544d2c3e061c20fe9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 57486cdd4f3828110aaba56755cf9e75f69c4f3fcfc409f139e2d8e4fc8153e68950b8e98bc2dcc32f0d81a3f42e23d985712235bc4e30cf95f2d20c7a18814a
|
7
|
+
data.tar.gz: a50d377790330e4196597ca6163d4f0f953e686a910b2dd5a58bb2878f88d5eca9dad23e674251677810e9bc0bdb45d2416862b1d05a1a2f104c6d7f70379b7b
|
data/.overcommit.yml
ADDED
data/.rubocop.yml
ADDED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,11 @@
|
|
1
1
|
# avromatic changelog
|
2
2
|
|
3
|
+
## v0.5.0
|
4
|
+
- Rename `Avromatic::Model::Decoder` to `MessageDecoder`.
|
5
|
+
- Rename `.deserialize` on generated models to `.avro_message_decode`.
|
6
|
+
- Add `#avro_raw_key`, `#avro_raw_value` and `.avro_raw_decode` methods to
|
7
|
+
generated models to support encoding and decoding without a schema registry.
|
8
|
+
|
3
9
|
## v0.4.0
|
4
10
|
- Allow the specification of a custom type, including conversion to/from Avro,
|
5
11
|
for named types.
|
data/README.md
CHANGED
@@ -29,28 +29,39 @@ Or install it yourself as:
|
|
29
29
|
|
30
30
|
`Avromatic` supports the following configuration:
|
31
31
|
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
* registry_url: URL for the schema registry. The schema registry is used to store
|
36
|
-
Avro schemas so that they can be referenced by id. Either `schema_registry` or
|
37
|
-
`registry_url` must be configured.
|
38
|
-
* schema_store: The schema store is used to load Avro schemas from the filesystem.
|
32
|
+
#### Model Generation
|
33
|
+
|
34
|
+
* **schema_store**: A schema store is required to load Avro schemas from the filesystem.
|
39
35
|
It should be an object that responds to `find(name, namespace = nil)` and
|
40
36
|
returns an `Avro::Schema` object. An `AvroTurf::SchemaStore` can be used.
|
41
|
-
|
37
|
+
The `schema_store` is unnecessary if models are generated directly from
|
38
|
+
`Avro::Schema` objects. See [Models](#models).
|
39
|
+
|
40
|
+
#### Using a Schema Registry/Messaging API
|
41
|
+
|
42
|
+
The configuration options below are required when using a schema registry
|
43
|
+
(see [Confluent Schema Registry](http://docs.confluent.io/2.0.1/schema-registry/docs/intro.html))
|
44
|
+
and the [Messaging API](#messaging-api).
|
45
|
+
|
46
|
+
* **schema_registry**: An `AvroTurf::SchemaRegistry` object used to store Avro schemas
|
47
|
+
so that they can be referenced by id. Either `schema_registry` or
|
48
|
+
`registry_url` must be configured.
|
49
|
+
* **registry_url**: URL for the schema registry. Either `schema_registry` or
|
50
|
+
`registry_url` must be configured.
|
51
|
+
* **messaging**: An `AvroTurf::Messaging` object to be shared by all generated models.
|
42
52
|
The `build_messaging!` method may be used to create a `Messaging` instance based
|
43
53
|
on the other configuration values.
|
44
|
-
* logger
|
54
|
+
* **logger**: The logger to use for the schema registry client.
|
45
55
|
* [Custom Types](#custom-types)
|
46
56
|
|
47
|
-
Example:
|
57
|
+
Example using a schema registry:
|
48
58
|
|
49
59
|
```ruby
|
50
60
|
Avromatic.configure do |config|
|
51
61
|
config.schema_store = AvroTurf::SchemaStore.new(path: 'avro/schemas')
|
52
62
|
config.registry_url = Rails.configuration.x.avro_schema_registry_url
|
53
63
|
config.build_messaging!
|
64
|
+
end
|
54
65
|
```
|
55
66
|
|
56
67
|
### Models
|
@@ -136,51 +147,95 @@ inbound or outbound value is nil.
|
|
136
147
|
If a custom type is registered for a record-type field, then any `to_avro`
|
137
148
|
method/Proc should return a Hash with string keys for encoding using Avro.
|
138
149
|
|
139
|
-
|
150
|
+
### Encoding and Decoding
|
151
|
+
|
152
|
+
`Avromatic` provides two different interfaces for encoding the key (optional)
|
153
|
+
and value associated with a model.
|
154
|
+
|
155
|
+
#### Manually Managed Schemas
|
156
|
+
|
157
|
+
The attributes for the value schema used to define a model can be encoded using:
|
158
|
+
|
159
|
+
```ruby
|
160
|
+
encoded_value = model.avro_raw_value
|
161
|
+
```
|
162
|
+
|
163
|
+
In order to decode this data, a copy of the value schema is required.
|
164
|
+
|
165
|
+
If a model also has an Avro schema for a key, then the key attributes can be
|
166
|
+
encoded using:
|
167
|
+
|
168
|
+
```ruby
|
169
|
+
encoded_key = model.avro_raw_key
|
170
|
+
```
|
171
|
+
|
172
|
+
If attributes were encoded using the same schema(s) used to define a model, then
|
173
|
+
the data can be decoded to create a new model instance:
|
174
|
+
|
175
|
+
```ruby
|
176
|
+
MyModel.avro_raw_decode(key: encoded_key, value: encoded_value)
|
177
|
+
```
|
178
|
+
|
179
|
+
If the attributes where encoded using a different version of the model's schemas,
|
180
|
+
then a new model instance can be created by also providing the schemas used to
|
181
|
+
encode the data:
|
182
|
+
|
183
|
+
```ruby
|
184
|
+
MyModel.avro_raw_decode(key: encoded_key,
|
185
|
+
key_schema: writers_key_schema,
|
186
|
+
value: encoded_value,
|
187
|
+
value_schema: writers_value_schema)
|
188
|
+
```
|
189
|
+
|
190
|
+
#### Messaging API
|
191
|
+
|
192
|
+
The other interface for encoding and decoding attributes uses the
|
193
|
+
`AvroTurf::Messaging` API. This interface leverages a schema registry and
|
194
|
+
prefixes the encoded data with an id to identify the schema. In this approach,
|
195
|
+
a schema registry is used to ensure that the correct schemas are available during
|
196
|
+
decoding.
|
140
197
|
|
141
|
-
|
142
|
-
id at the beginning of the value.
|
198
|
+
The attributes for the value schema can be encoded with a schema id prefix using:
|
143
199
|
|
144
200
|
```ruby
|
145
|
-
model.avro_message_value
|
201
|
+
message_value = model.avro_message_value
|
146
202
|
```
|
147
203
|
|
148
|
-
If a model has an Avro schema for a key, then
|
149
|
-
prefixed with a schema id
|
204
|
+
If a model has an Avro schema for a key, then those attributes can also be encoded
|
205
|
+
prefixed with a schema id:
|
150
206
|
|
151
207
|
```ruby
|
152
|
-
model.avro_message_key
|
208
|
+
message_key = model.avro_message_key
|
153
209
|
```
|
154
210
|
|
155
|
-
A model instance can be created from
|
156
|
-
optional key:
|
211
|
+
A model instance can be created from a key and value encoded in this manner:
|
157
212
|
|
158
213
|
```ruby
|
159
|
-
MyTopic.
|
214
|
+
MyTopic.avro_message_decode(message_key, message_value)
|
160
215
|
```
|
161
216
|
|
162
217
|
Or just a value if only one schema is used:
|
163
218
|
|
164
219
|
```ruby
|
165
|
-
MyValue.
|
220
|
+
MyValue.avro_message_decode(message_value)
|
166
221
|
```
|
167
222
|
|
168
|
-
####
|
223
|
+
#### Avromatric::Model::MessageDecoder
|
169
224
|
|
170
|
-
A stream of messages encoded from various models
|
171
|
-
`Avromatic::Model::
|
172
|
-
of models to decode:
|
225
|
+
A stream of messages encoded from various models using the messaging approach
|
226
|
+
can be decoded using `Avromatic::Model::MessageDecoder`. The decoder must be
|
227
|
+
initialized with the list of models to decode:
|
173
228
|
|
174
229
|
```ruby
|
175
|
-
decoder = Avromatic::Model::
|
230
|
+
decoder = Avromatic::Model::MessageDecoder.new(MyModel1, MyModel2)
|
176
231
|
|
177
|
-
decoder.decode(
|
232
|
+
decoder.decode(model1_messge_key, model1_message_value)
|
178
233
|
# => instance of MyModel1
|
179
|
-
decoder.decode(
|
234
|
+
decoder.decode(model2_message_value)
|
180
235
|
# => instance of MyModel2
|
181
236
|
```
|
182
237
|
|
183
|
-
|
238
|
+
### Validations
|
184
239
|
|
185
240
|
The following validations are supported:
|
186
241
|
|
@@ -188,7 +243,7 @@ The following validations are supported:
|
|
188
243
|
- The value for an enum type field is in the declared set of values.
|
189
244
|
- Presence of a value for required fields.
|
190
245
|
|
191
|
-
|
246
|
+
### Unsupported/Future
|
192
247
|
|
193
248
|
The following types/features are not supported for generated models:
|
194
249
|
|
data/Rakefile
CHANGED
@@ -1,6 +1,6 @@
|
|
1
|
-
require
|
2
|
-
require
|
3
|
-
require
|
1
|
+
require 'bundler/gem_tasks'
|
2
|
+
require 'rspec/core/rake_task'
|
3
|
+
require 'avro/builder'
|
4
4
|
|
5
5
|
RSpec::Core::RakeTask.new(:spec)
|
6
6
|
|
@@ -19,4 +19,4 @@ namespace :avro do
|
|
19
19
|
end
|
20
20
|
end
|
21
21
|
|
22
|
-
task :
|
22
|
+
task default: :spec
|
data/avromatic.gemspec
CHANGED
@@ -4,33 +4,35 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
require 'avromatic/version'
|
5
5
|
|
6
6
|
Gem::Specification.new do |spec|
|
7
|
-
spec.name =
|
7
|
+
spec.name = 'avromatic'
|
8
8
|
spec.version = Avromatic::VERSION
|
9
|
-
spec.authors = [
|
10
|
-
spec.email = [
|
9
|
+
spec.authors = ['Salsify Engineering']
|
10
|
+
spec.email = ['engineering@salsify.com']
|
11
11
|
|
12
|
-
spec.summary =
|
12
|
+
spec.summary = 'Generate Ruby models from Avro schemas'
|
13
13
|
spec.description = spec.summary
|
14
|
-
spec.homepage =
|
15
|
-
spec.license =
|
14
|
+
spec.homepage = 'https://github.com/salsify/avromatic.git'
|
15
|
+
spec.license = 'MIT'
|
16
16
|
|
17
17
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
18
|
-
spec.bindir =
|
18
|
+
spec.bindir = 'exe'
|
19
19
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
20
|
-
spec.require_paths = [
|
20
|
+
spec.require_paths = ['lib']
|
21
21
|
|
22
|
-
spec.add_runtime_dependency
|
23
|
-
spec.add_runtime_dependency
|
24
|
-
spec.add_runtime_dependency
|
25
|
-
spec.add_runtime_dependency
|
26
|
-
spec.add_runtime_dependency
|
22
|
+
spec.add_runtime_dependency 'avro', '>= 1.7.7'
|
23
|
+
spec.add_runtime_dependency 'virtus'
|
24
|
+
spec.add_runtime_dependency 'activesupport'
|
25
|
+
spec.add_runtime_dependency 'activemodel'
|
26
|
+
spec.add_runtime_dependency 'avro_turf'
|
27
27
|
|
28
|
-
spec.add_development_dependency
|
29
|
-
spec.add_development_dependency
|
30
|
-
spec.add_development_dependency
|
31
|
-
spec.add_development_dependency
|
32
|
-
spec.add_development_dependency
|
33
|
-
spec.add_development_dependency
|
28
|
+
spec.add_development_dependency 'bundler', '~> 1.11'
|
29
|
+
spec.add_development_dependency 'rake', '~> 10.0'
|
30
|
+
spec.add_development_dependency 'rspec', '~> 3.0'
|
31
|
+
spec.add_development_dependency 'simplecov'
|
32
|
+
spec.add_development_dependency 'webmock'
|
33
|
+
spec.add_development_dependency 'avro-builder', '>= 0.3.2'
|
34
34
|
# For FakeSchemaRegistryServer
|
35
|
-
spec.add_development_dependency
|
35
|
+
spec.add_development_dependency 'sinatra'
|
36
|
+
spec.add_development_dependency 'salsify_rubocop', '~> 0.40.0'
|
37
|
+
spec.add_development_dependency 'overcommit'
|
36
38
|
end
|
@@ -30,15 +30,17 @@ module Avromatic
|
|
30
30
|
private
|
31
31
|
|
32
32
|
def check_for_field_conflicts!
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
33
|
+
conflicts =
|
34
|
+
(key_avro_field_names & value_avro_field_names).each_with_object([]) do |name, msgs|
|
35
|
+
next unless schema_fields_differ?(name)
|
36
|
+
msgs << "Field '#{name}' has a different type in each schema: "\
|
37
|
+
"value #{value_avro_fields_by_name[name]}, "\
|
38
|
+
"key #{key_avro_fields_by_name[name]}"
|
39
|
+
end
|
40
|
+
|
41
|
+
raise conflicts.join("\n") if conflicts.any?
|
42
|
+
|
43
|
+
conflicts
|
42
44
|
end
|
43
45
|
|
44
46
|
# The Avro::Schema::Field#== method is lame. It just compares
|
@@ -5,7 +5,8 @@ require 'avromatic/model/configuration'
|
|
5
5
|
require 'avromatic/model/value_object'
|
6
6
|
require 'avromatic/model/configurable'
|
7
7
|
require 'avromatic/model/attributes'
|
8
|
-
require 'avromatic/model/
|
8
|
+
require 'avromatic/model/raw_serialization'
|
9
|
+
require 'avromatic/model/messaging_serialization'
|
9
10
|
|
10
11
|
module Avromatic
|
11
12
|
module Model
|
@@ -41,7 +42,8 @@ module Avromatic
|
|
41
42
|
Avromatic::Model::Configurable,
|
42
43
|
Avromatic::Model::Attributes,
|
43
44
|
Avromatic::Model::ValueObject,
|
44
|
-
Avromatic::Model::
|
45
|
+
Avromatic::Model::RawSerialization,
|
46
|
+
Avromatic::Model::MessagingSerialization
|
45
47
|
]
|
46
48
|
end
|
47
49
|
|
@@ -3,9 +3,10 @@ require 'avro_turf/schema_registry'
|
|
3
3
|
module Avromatic
|
4
4
|
module Model
|
5
5
|
|
6
|
-
# This class is used to decode Avro
|
7
|
-
|
8
|
-
|
6
|
+
# This class is used to decode messages encoded using Avro to their
|
7
|
+
# corresponding models.
|
8
|
+
class MessageDecoder
|
9
|
+
MAGIC_BYTE = [0].pack('C').freeze
|
9
10
|
|
10
11
|
class UnexpectedKeyError < StandardError
|
11
12
|
def initialize(schema_key)
|
@@ -13,16 +14,16 @@ module Avromatic
|
|
13
14
|
end
|
14
15
|
end
|
15
16
|
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
17
|
+
class MagicByteError < StandardError
|
18
|
+
def initialize(magic_byte)
|
19
|
+
super("Expected data to begin with a magic byte, got '#{magic_byte}'")
|
20
|
+
end
|
21
|
+
end
|
21
22
|
|
22
23
|
class DuplicateKeyError < StandardError
|
23
24
|
def initialize(*models)
|
24
25
|
super("Multiple models #{models} have the same key "\
|
25
|
-
"'#{Avromatic::Model::
|
26
|
+
"'#{Avromatic::Model::MessageDecoder.model_key(models.first)}'")
|
26
27
|
end
|
27
28
|
end
|
28
29
|
|
@@ -63,7 +64,7 @@ module Avromatic
|
|
63
64
|
|
64
65
|
def deserialize(model_key, message_key, message_value)
|
65
66
|
raise UnexpectedKeyError.new(model_key) unless model_map.key?(model_key)
|
66
|
-
model_map[model_key].
|
67
|
+
model_map[model_key].avro_message_decode(message_key, message_value)
|
67
68
|
end
|
68
69
|
|
69
70
|
def schema_name_for_data(data)
|
@@ -0,0 +1,62 @@
|
|
1
|
+
require 'avro_turf/messaging'
|
2
|
+
|
3
|
+
module Avromatic
|
4
|
+
module Model
|
5
|
+
|
6
|
+
# This concern adds support for serialization based on AvroTurf::Messaging.
|
7
|
+
# This serialization leverages a schema registry to prefix encoded values
|
8
|
+
# with an id for the schema.
|
9
|
+
module MessagingSerialization
|
10
|
+
extend ActiveSupport::Concern
|
11
|
+
|
12
|
+
delegate :avro_messaging, to: :class
|
13
|
+
private :avro_messaging
|
14
|
+
|
15
|
+
module Encode
|
16
|
+
def avro_message_value
|
17
|
+
avro_messaging.encode(
|
18
|
+
value_attributes_for_avro,
|
19
|
+
schema_name: value_avro_schema.fullname
|
20
|
+
)
|
21
|
+
end
|
22
|
+
|
23
|
+
def avro_message_key
|
24
|
+
raise 'Model has no key schema' unless key_avro_schema
|
25
|
+
avro_messaging.encode(
|
26
|
+
key_attributes_for_avro,
|
27
|
+
schema_name: key_avro_schema.fullname
|
28
|
+
)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
include Encode
|
32
|
+
|
33
|
+
# This module provides methods to decode an Avro-encoded value and
|
34
|
+
# an optional Avro-encoded key as a new model instance.
|
35
|
+
module Decode
|
36
|
+
|
37
|
+
# If two arguments are specified then the first is interpreted as the
|
38
|
+
# message key and the second is the message value. If there is only one
|
39
|
+
# arg then it is used as the message value.
|
40
|
+
def avro_message_decode(*args)
|
41
|
+
message_key, message_value = args.size > 1 ? args : [nil, args.first]
|
42
|
+
key_attributes = message_key &&
|
43
|
+
avro_messaging.decode(message_key, schema_name: key_avro_schema.fullname)
|
44
|
+
value_attributes = avro_messaging
|
45
|
+
.decode(message_value, schema_name: avro_schema.fullname)
|
46
|
+
|
47
|
+
new(value_attributes.merge!(key_attributes || {}))
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
module ClassMethods
|
52
|
+
# The messaging object acts as an intermediary talking to the schema
|
53
|
+
# registry and using returned/specified schemas to decode/encode.
|
54
|
+
def avro_messaging
|
55
|
+
Avromatic.messaging
|
56
|
+
end
|
57
|
+
|
58
|
+
include Decode
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
@@ -0,0 +1,111 @@
|
|
1
|
+
require 'avromatic/model/passthrough_serializer'
|
2
|
+
|
3
|
+
module Avromatic
|
4
|
+
module Model
|
5
|
+
|
6
|
+
# This module provides serialization support for encoding directly to Avro
|
7
|
+
# without dependency on a schema registry.
|
8
|
+
module RawSerialization
|
9
|
+
extend ActiveSupport::Concern
|
10
|
+
|
11
|
+
module Encode
|
12
|
+
delegate :avro_serializer, :datum_writer, :datum_reader, to: :class
|
13
|
+
private :avro_serializer, :datum_writer, :datum_reader
|
14
|
+
|
15
|
+
def avro_raw_value
|
16
|
+
avro_raw_encode(value_attributes_for_avro, :value)
|
17
|
+
end
|
18
|
+
|
19
|
+
def avro_raw_key
|
20
|
+
raise 'Model has no key schema' unless key_avro_schema
|
21
|
+
avro_raw_encode(key_attributes_for_avro, :key)
|
22
|
+
end
|
23
|
+
|
24
|
+
protected
|
25
|
+
|
26
|
+
def value_attributes_for_avro
|
27
|
+
avro_hash(value_avro_field_names)
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def key_attributes_for_avro
|
33
|
+
avro_hash(key_avro_field_names)
|
34
|
+
end
|
35
|
+
|
36
|
+
def avro_hash(fields)
|
37
|
+
attributes.slice(*fields).each_with_object(Hash.new) do |(key, value), result|
|
38
|
+
result[key.to_s] = if value.is_a?(Avromatic::Model::Attributes)
|
39
|
+
value.value_attributes_for_avro
|
40
|
+
else
|
41
|
+
avro_serializer[key].call(value)
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
def avro_raw_encode(data, key_or_value = :value)
|
47
|
+
stream = StringIO.new
|
48
|
+
encoder = Avro::IO::BinaryEncoder.new(stream)
|
49
|
+
datum_writer[key_or_value].write(data, encoder)
|
50
|
+
stream.string
|
51
|
+
end
|
52
|
+
end
|
53
|
+
include Encode
|
54
|
+
|
55
|
+
module Decode
|
56
|
+
|
57
|
+
# Create a new instance based on an encoded value and optional encoded key.
|
58
|
+
# If supplied then the key_schema and value_schema are used as the writer's
|
59
|
+
# schema for the corresponding value. The model's schemas are always used
|
60
|
+
# as the reader's schemas.
|
61
|
+
def avro_raw_decode(key: nil, value:, key_schema: nil, value_schema: nil)
|
62
|
+
key_attributes = key && decode_avro_datum(key, key_schema, :key)
|
63
|
+
value_attributes = decode_avro_datum(value, value_schema, :value)
|
64
|
+
|
65
|
+
new(value_attributes.merge!(key_attributes || {}))
|
66
|
+
end
|
67
|
+
|
68
|
+
private
|
69
|
+
|
70
|
+
def decode_avro_datum(data, schema = nil, key_or_value = :value)
|
71
|
+
stream = StringIO.new(data)
|
72
|
+
decoder = Avro::IO::BinaryDecoder.new(stream)
|
73
|
+
reader = schema ? custom_datum_reader(schema, key_or_value) : datum_reader[key_or_value]
|
74
|
+
reader.read(decoder)
|
75
|
+
end
|
76
|
+
|
77
|
+
def custom_datum_reader(schema, key_or_value)
|
78
|
+
Avro::IO::DatumReader.new(schema, send("#{key_or_value}_avro_schema"))
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
module ClassMethods
|
83
|
+
|
84
|
+
# Store a hash of Procs by field name (as a symbol) to convert
|
85
|
+
# the value before Avro serialization.
|
86
|
+
# Returns the default PassthroughSerializer if a key is not present.
|
87
|
+
def avro_serializer
|
88
|
+
@avro_serializer ||= Hash.new(PassthroughSerializer)
|
89
|
+
end
|
90
|
+
|
91
|
+
def datum_writer
|
92
|
+
@datum_writer ||= begin
|
93
|
+
hash = { value: Avro::IO::DatumWriter.new(value_avro_schema) }
|
94
|
+
hash[:key] = Avro::IO::DatumWriter.new(key_avro_schema) if key_avro_schema
|
95
|
+
hash
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
def datum_reader
|
100
|
+
@datum_reader ||= begin
|
101
|
+
hash = { value: Avro::IO::DatumReader.new(value_avro_schema) }
|
102
|
+
hash[:key] = Avro::IO::DatumReader.new(key_avro_schema) if key_avro_schema
|
103
|
+
hash
|
104
|
+
end
|
105
|
+
end
|
106
|
+
|
107
|
+
include Decode
|
108
|
+
end
|
109
|
+
end
|
110
|
+
end
|
111
|
+
end
|
@@ -14,11 +14,11 @@ module Avromatic
|
|
14
14
|
end
|
15
15
|
|
16
16
|
def inspect
|
17
|
-
"#<#{self.class.name} #{attributes.map { |k, v| "#{k}: #{v.inspect}" }.join(', ')
|
17
|
+
"#<#{self.class.name} #{attributes.map { |k, v| "#{k}: #{v.inspect}" }.join(', ')}>"
|
18
18
|
end
|
19
19
|
|
20
20
|
def to_s
|
21
|
-
|
21
|
+
format('#<%s:0x00%x>', self.class.name, object_id.abs * 2)
|
22
22
|
end
|
23
23
|
end
|
24
24
|
end
|
data/lib/avromatic/model.rb
CHANGED
data/lib/avromatic/version.rb
CHANGED
data/lib/avromatic.rb
CHANGED
@@ -21,7 +21,8 @@ module Avromatic
|
|
21
21
|
def self.build_schema_registry
|
22
22
|
raise 'Avromatic must be configured with a registry_url' unless registry_url
|
23
23
|
AvroTurf::CachedSchemaRegistry.new(
|
24
|
-
AvroTurf::SchemaRegistry.new(registry_url, logger: logger)
|
24
|
+
AvroTurf::SchemaRegistry.new(registry_url, logger: logger)
|
25
|
+
)
|
25
26
|
end
|
26
27
|
|
27
28
|
def self.build_messaging
|
@@ -29,7 +30,8 @@ module Avromatic
|
|
29
30
|
AvroTurf::Messaging.new(
|
30
31
|
registry: schema_registry || build_schema_registry,
|
31
32
|
schema_store: schema_store,
|
32
|
-
logger: logger
|
33
|
+
logger: logger
|
34
|
+
)
|
33
35
|
end
|
34
36
|
|
35
37
|
def self.build_messaging!
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: avromatic
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Salsify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-05-
|
11
|
+
date: 2016-05-25 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: avro
|
@@ -178,6 +178,34 @@ dependencies:
|
|
178
178
|
- - ">="
|
179
179
|
- !ruby/object:Gem::Version
|
180
180
|
version: '0'
|
181
|
+
- !ruby/object:Gem::Dependency
|
182
|
+
name: salsify_rubocop
|
183
|
+
requirement: !ruby/object:Gem::Requirement
|
184
|
+
requirements:
|
185
|
+
- - "~>"
|
186
|
+
- !ruby/object:Gem::Version
|
187
|
+
version: 0.40.0
|
188
|
+
type: :development
|
189
|
+
prerelease: false
|
190
|
+
version_requirements: !ruby/object:Gem::Requirement
|
191
|
+
requirements:
|
192
|
+
- - "~>"
|
193
|
+
- !ruby/object:Gem::Version
|
194
|
+
version: 0.40.0
|
195
|
+
- !ruby/object:Gem::Dependency
|
196
|
+
name: overcommit
|
197
|
+
requirement: !ruby/object:Gem::Requirement
|
198
|
+
requirements:
|
199
|
+
- - ">="
|
200
|
+
- !ruby/object:Gem::Version
|
201
|
+
version: '0'
|
202
|
+
type: :development
|
203
|
+
prerelease: false
|
204
|
+
version_requirements: !ruby/object:Gem::Requirement
|
205
|
+
requirements:
|
206
|
+
- - ">="
|
207
|
+
- !ruby/object:Gem::Version
|
208
|
+
version: '0'
|
181
209
|
description: Generate Ruby models from Avro schemas
|
182
210
|
email:
|
183
211
|
- engineering@salsify.com
|
@@ -186,7 +214,9 @@ extensions: []
|
|
186
214
|
extra_rdoc_files: []
|
187
215
|
files:
|
188
216
|
- ".gitignore"
|
217
|
+
- ".overcommit.yml"
|
189
218
|
- ".rspec"
|
219
|
+
- ".rubocop.yml"
|
190
220
|
- ".travis.yml"
|
191
221
|
- CHANGELOG.md
|
192
222
|
- Gemfile
|
@@ -203,10 +233,11 @@ files:
|
|
203
233
|
- lib/avromatic/model/configurable.rb
|
204
234
|
- lib/avromatic/model/configuration.rb
|
205
235
|
- lib/avromatic/model/custom_type.rb
|
206
|
-
- lib/avromatic/model/
|
236
|
+
- lib/avromatic/model/message_decoder.rb
|
237
|
+
- lib/avromatic/model/messaging_serialization.rb
|
207
238
|
- lib/avromatic/model/null_custom_type.rb
|
208
239
|
- lib/avromatic/model/passthrough_serializer.rb
|
209
|
-
- lib/avromatic/model/
|
240
|
+
- lib/avromatic/model/raw_serialization.rb
|
210
241
|
- lib/avromatic/model/type_registry.rb
|
211
242
|
- lib/avromatic/model/value_object.rb
|
212
243
|
- lib/avromatic/railtie.rb
|
@@ -1,86 +0,0 @@
|
|
1
|
-
require 'avro_turf/messaging'
|
2
|
-
require 'avromatic/model/passthrough_serializer'
|
3
|
-
|
4
|
-
module Avromatic
|
5
|
-
module Model
|
6
|
-
|
7
|
-
# This concern adds support for serialization to a model
|
8
|
-
# generated from Avro schema(s).
|
9
|
-
module Serialization
|
10
|
-
extend ActiveSupport::Concern
|
11
|
-
|
12
|
-
delegate :messaging, to: :Avromatic
|
13
|
-
|
14
|
-
module Encode
|
15
|
-
def avro_message_value
|
16
|
-
messaging.encode(
|
17
|
-
value_attributes_for_avro,
|
18
|
-
schema_name: value_avro_schema.fullname)
|
19
|
-
end
|
20
|
-
|
21
|
-
def avro_message_key
|
22
|
-
raise 'Model has no key schema' unless key_avro_schema
|
23
|
-
messaging.encode(
|
24
|
-
key_attributes_for_avro,
|
25
|
-
schema_name: key_avro_schema.fullname)
|
26
|
-
end
|
27
|
-
|
28
|
-
protected
|
29
|
-
|
30
|
-
def value_attributes_for_avro
|
31
|
-
avro_hash(value_avro_field_names)
|
32
|
-
end
|
33
|
-
|
34
|
-
private
|
35
|
-
|
36
|
-
delegate :avro_serializer, to: :class
|
37
|
-
|
38
|
-
def key_attributes_for_avro
|
39
|
-
avro_hash(key_avro_field_names)
|
40
|
-
end
|
41
|
-
|
42
|
-
def avro_hash(fields)
|
43
|
-
attributes.slice(*fields).each_with_object(Hash.new) do |(key, value), result|
|
44
|
-
result[key.to_s] = if value.is_a?(Avromatic::Model::Attributes)
|
45
|
-
value.value_attributes_for_avro
|
46
|
-
else
|
47
|
-
avro_serializer[key].call(value)
|
48
|
-
end
|
49
|
-
end
|
50
|
-
end
|
51
|
-
end
|
52
|
-
include Encode
|
53
|
-
|
54
|
-
# This module provides methods to deserialize an Avro-encoded value and
|
55
|
-
# an optional Avro-encoded key as a new model instance.
|
56
|
-
module Decode
|
57
|
-
|
58
|
-
# If two arguments are specified then the first is interpreted as the
|
59
|
-
# message key and the second is the message value. If there is only one
|
60
|
-
# arg then it is used as the message value.
|
61
|
-
def deserialize(*args)
|
62
|
-
message_key, message_value = args.size > 1 ? args : [nil, args.first]
|
63
|
-
key_attributes = message_key && messaging.decode(message_key, schema_name: key_avro_schema.fullname)
|
64
|
-
value_attributes = messaging.decode(message_value, schema_name: avro_schema.fullname)
|
65
|
-
|
66
|
-
new(value_attributes.merge!(key_attributes || {}))
|
67
|
-
end
|
68
|
-
end
|
69
|
-
|
70
|
-
module ClassMethods
|
71
|
-
# The messaging object acts as an intermediary talking to the schema
|
72
|
-
# registry and using returned/specified schemas to decode/encode.
|
73
|
-
delegate :messaging, to: :Avromatic
|
74
|
-
|
75
|
-
include Decode
|
76
|
-
|
77
|
-
# Store a hash of Procs by field name (as a symbol) to convert
|
78
|
-
# the value before Avro serialization.
|
79
|
-
# Returns the default PassthroughSerializer if a key is not present.
|
80
|
-
def avro_serializer
|
81
|
-
@avro_serializer ||= Hash.new(PassthroughSerializer)
|
82
|
-
end
|
83
|
-
end
|
84
|
-
end
|
85
|
-
end
|
86
|
-
end
|