avromatic 0.4.0 → 0.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.overcommit.yml +14 -0
- data/.rubocop.yml +8 -0
- data/CHANGELOG.md +6 -0
- data/README.md +85 -30
- data/Rakefile +4 -4
- data/avromatic.gemspec +22 -20
- data/lib/avromatic/model/attributes.rb +11 -9
- data/lib/avromatic/model/builder.rb +4 -2
- data/lib/avromatic/model/{decoder.rb → message_decoder.rb} +11 -10
- data/lib/avromatic/model/messaging_serialization.rb +62 -0
- data/lib/avromatic/model/raw_serialization.rb +111 -0
- data/lib/avromatic/model/value_object.rb +2 -2
- data/lib/avromatic/model.rb +1 -1
- data/lib/avromatic/version.rb +1 -1
- data/lib/avromatic.rb +4 -2
- metadata +35 -4
- data/lib/avromatic/model/serialization.rb +0 -86
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4d18b7d880de4f22e4615d3d547551953f24bec7
|
4
|
+
data.tar.gz: 8a970619e0c045513276a54544d2c3e061c20fe9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 57486cdd4f3828110aaba56755cf9e75f69c4f3fcfc409f139e2d8e4fc8153e68950b8e98bc2dcc32f0d81a3f42e23d985712235bc4e30cf95f2d20c7a18814a
|
7
|
+
data.tar.gz: a50d377790330e4196597ca6163d4f0f953e686a910b2dd5a58bb2878f88d5eca9dad23e674251677810e9bc0bdb45d2416862b1d05a1a2f104c6d7f70379b7b
|
data/.overcommit.yml
ADDED
data/.rubocop.yml
ADDED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,11 @@
|
|
1
1
|
# avromatic changelog
|
2
2
|
|
3
|
+
## v0.5.0
|
4
|
+
- Rename `Avromatic::Model::Decoder` to `MessageDecoder`.
|
5
|
+
- Rename `.deserialize` on generated models to `.avro_message_decode`.
|
6
|
+
- Add `#avro_raw_key`, `#avro_raw_value` and `.avro_raw_decode` methods to
|
7
|
+
generated models to support encoding and decoding without a schema registry.
|
8
|
+
|
3
9
|
## v0.4.0
|
4
10
|
- Allow the specification of a custom type, including conversion to/from Avro,
|
5
11
|
for named types.
|
data/README.md
CHANGED
@@ -29,28 +29,39 @@ Or install it yourself as:
|
|
29
29
|
|
30
30
|
`Avromatic` supports the following configuration:
|
31
31
|
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
* registry_url: URL for the schema registry. The schema registry is used to store
|
36
|
-
Avro schemas so that they can be referenced by id. Either `schema_registry` or
|
37
|
-
`registry_url` must be configured.
|
38
|
-
* schema_store: The schema store is used to load Avro schemas from the filesystem.
|
32
|
+
#### Model Generation
|
33
|
+
|
34
|
+
* **schema_store**: A schema store is required to load Avro schemas from the filesystem.
|
39
35
|
It should be an object that responds to `find(name, namespace = nil)` and
|
40
36
|
returns an `Avro::Schema` object. An `AvroTurf::SchemaStore` can be used.
|
41
|
-
|
37
|
+
The `schema_store` is unnecessary if models are generated directly from
|
38
|
+
`Avro::Schema` objects. See [Models](#models).
|
39
|
+
|
40
|
+
#### Using a Schema Registry/Messaging API
|
41
|
+
|
42
|
+
The configuration options below are required when using a schema registry
|
43
|
+
(see [Confluent Schema Registry](http://docs.confluent.io/2.0.1/schema-registry/docs/intro.html))
|
44
|
+
and the [Messaging API](#messaging-api).
|
45
|
+
|
46
|
+
* **schema_registry**: An `AvroTurf::SchemaRegistry` object used to store Avro schemas
|
47
|
+
so that they can be referenced by id. Either `schema_registry` or
|
48
|
+
`registry_url` must be configured.
|
49
|
+
* **registry_url**: URL for the schema registry. Either `schema_registry` or
|
50
|
+
`registry_url` must be configured.
|
51
|
+
* **messaging**: An `AvroTurf::Messaging` object to be shared by all generated models.
|
42
52
|
The `build_messaging!` method may be used to create a `Messaging` instance based
|
43
53
|
on the other configuration values.
|
44
|
-
* logger
|
54
|
+
* **logger**: The logger to use for the schema registry client.
|
45
55
|
* [Custom Types](#custom-types)
|
46
56
|
|
47
|
-
Example:
|
57
|
+
Example using a schema registry:
|
48
58
|
|
49
59
|
```ruby
|
50
60
|
Avromatic.configure do |config|
|
51
61
|
config.schema_store = AvroTurf::SchemaStore.new(path: 'avro/schemas')
|
52
62
|
config.registry_url = Rails.configuration.x.avro_schema_registry_url
|
53
63
|
config.build_messaging!
|
64
|
+
end
|
54
65
|
```
|
55
66
|
|
56
67
|
### Models
|
@@ -136,51 +147,95 @@ inbound or outbound value is nil.
|
|
136
147
|
If a custom type is registered for a record-type field, then any `to_avro`
|
137
148
|
method/Proc should return a Hash with string keys for encoding using Avro.
|
138
149
|
|
139
|
-
|
150
|
+
### Encoding and Decoding
|
151
|
+
|
152
|
+
`Avromatic` provides two different interfaces for encoding the key (optional)
|
153
|
+
and value associated with a model.
|
154
|
+
|
155
|
+
#### Manually Managed Schemas
|
156
|
+
|
157
|
+
The attributes for the value schema used to define a model can be encoded using:
|
158
|
+
|
159
|
+
```ruby
|
160
|
+
encoded_value = model.avro_raw_value
|
161
|
+
```
|
162
|
+
|
163
|
+
In order to decode this data, a copy of the value schema is required.
|
164
|
+
|
165
|
+
If a model also has an Avro schema for a key, then the key attributes can be
|
166
|
+
encoded using:
|
167
|
+
|
168
|
+
```ruby
|
169
|
+
encoded_key = model.avro_raw_key
|
170
|
+
```
|
171
|
+
|
172
|
+
If attributes were encoded using the same schema(s) used to define a model, then
|
173
|
+
the data can be decoded to create a new model instance:
|
174
|
+
|
175
|
+
```ruby
|
176
|
+
MyModel.avro_raw_decode(key: encoded_key, value: encoded_value)
|
177
|
+
```
|
178
|
+
|
179
|
+
If the attributes where encoded using a different version of the model's schemas,
|
180
|
+
then a new model instance can be created by also providing the schemas used to
|
181
|
+
encode the data:
|
182
|
+
|
183
|
+
```ruby
|
184
|
+
MyModel.avro_raw_decode(key: encoded_key,
|
185
|
+
key_schema: writers_key_schema,
|
186
|
+
value: encoded_value,
|
187
|
+
value_schema: writers_value_schema)
|
188
|
+
```
|
189
|
+
|
190
|
+
#### Messaging API
|
191
|
+
|
192
|
+
The other interface for encoding and decoding attributes uses the
|
193
|
+
`AvroTurf::Messaging` API. This interface leverages a schema registry and
|
194
|
+
prefixes the encoded data with an id to identify the schema. In this approach,
|
195
|
+
a schema registry is used to ensure that the correct schemas are available during
|
196
|
+
decoding.
|
140
197
|
|
141
|
-
|
142
|
-
id at the beginning of the value.
|
198
|
+
The attributes for the value schema can be encoded with a schema id prefix using:
|
143
199
|
|
144
200
|
```ruby
|
145
|
-
model.avro_message_value
|
201
|
+
message_value = model.avro_message_value
|
146
202
|
```
|
147
203
|
|
148
|
-
If a model has an Avro schema for a key, then
|
149
|
-
prefixed with a schema id
|
204
|
+
If a model has an Avro schema for a key, then those attributes can also be encoded
|
205
|
+
prefixed with a schema id:
|
150
206
|
|
151
207
|
```ruby
|
152
|
-
model.avro_message_key
|
208
|
+
message_key = model.avro_message_key
|
153
209
|
```
|
154
210
|
|
155
|
-
A model instance can be created from
|
156
|
-
optional key:
|
211
|
+
A model instance can be created from a key and value encoded in this manner:
|
157
212
|
|
158
213
|
```ruby
|
159
|
-
MyTopic.
|
214
|
+
MyTopic.avro_message_decode(message_key, message_value)
|
160
215
|
```
|
161
216
|
|
162
217
|
Or just a value if only one schema is used:
|
163
218
|
|
164
219
|
```ruby
|
165
|
-
MyValue.
|
220
|
+
MyValue.avro_message_decode(message_value)
|
166
221
|
```
|
167
222
|
|
168
|
-
####
|
223
|
+
#### Avromatric::Model::MessageDecoder
|
169
224
|
|
170
|
-
A stream of messages encoded from various models
|
171
|
-
`Avromatic::Model::
|
172
|
-
of models to decode:
|
225
|
+
A stream of messages encoded from various models using the messaging approach
|
226
|
+
can be decoded using `Avromatic::Model::MessageDecoder`. The decoder must be
|
227
|
+
initialized with the list of models to decode:
|
173
228
|
|
174
229
|
```ruby
|
175
|
-
decoder = Avromatic::Model::
|
230
|
+
decoder = Avromatic::Model::MessageDecoder.new(MyModel1, MyModel2)
|
176
231
|
|
177
|
-
decoder.decode(
|
232
|
+
decoder.decode(model1_messge_key, model1_message_value)
|
178
233
|
# => instance of MyModel1
|
179
|
-
decoder.decode(
|
234
|
+
decoder.decode(model2_message_value)
|
180
235
|
# => instance of MyModel2
|
181
236
|
```
|
182
237
|
|
183
|
-
|
238
|
+
### Validations
|
184
239
|
|
185
240
|
The following validations are supported:
|
186
241
|
|
@@ -188,7 +243,7 @@ The following validations are supported:
|
|
188
243
|
- The value for an enum type field is in the declared set of values.
|
189
244
|
- Presence of a value for required fields.
|
190
245
|
|
191
|
-
|
246
|
+
### Unsupported/Future
|
192
247
|
|
193
248
|
The following types/features are not supported for generated models:
|
194
249
|
|
data/Rakefile
CHANGED
@@ -1,6 +1,6 @@
|
|
1
|
-
require
|
2
|
-
require
|
3
|
-
require
|
1
|
+
require 'bundler/gem_tasks'
|
2
|
+
require 'rspec/core/rake_task'
|
3
|
+
require 'avro/builder'
|
4
4
|
|
5
5
|
RSpec::Core::RakeTask.new(:spec)
|
6
6
|
|
@@ -19,4 +19,4 @@ namespace :avro do
|
|
19
19
|
end
|
20
20
|
end
|
21
21
|
|
22
|
-
task :
|
22
|
+
task default: :spec
|
data/avromatic.gemspec
CHANGED
@@ -4,33 +4,35 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
require 'avromatic/version'
|
5
5
|
|
6
6
|
Gem::Specification.new do |spec|
|
7
|
-
spec.name =
|
7
|
+
spec.name = 'avromatic'
|
8
8
|
spec.version = Avromatic::VERSION
|
9
|
-
spec.authors = [
|
10
|
-
spec.email = [
|
9
|
+
spec.authors = ['Salsify Engineering']
|
10
|
+
spec.email = ['engineering@salsify.com']
|
11
11
|
|
12
|
-
spec.summary =
|
12
|
+
spec.summary = 'Generate Ruby models from Avro schemas'
|
13
13
|
spec.description = spec.summary
|
14
|
-
spec.homepage =
|
15
|
-
spec.license =
|
14
|
+
spec.homepage = 'https://github.com/salsify/avromatic.git'
|
15
|
+
spec.license = 'MIT'
|
16
16
|
|
17
17
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
18
|
-
spec.bindir =
|
18
|
+
spec.bindir = 'exe'
|
19
19
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
20
|
-
spec.require_paths = [
|
20
|
+
spec.require_paths = ['lib']
|
21
21
|
|
22
|
-
spec.add_runtime_dependency
|
23
|
-
spec.add_runtime_dependency
|
24
|
-
spec.add_runtime_dependency
|
25
|
-
spec.add_runtime_dependency
|
26
|
-
spec.add_runtime_dependency
|
22
|
+
spec.add_runtime_dependency 'avro', '>= 1.7.7'
|
23
|
+
spec.add_runtime_dependency 'virtus'
|
24
|
+
spec.add_runtime_dependency 'activesupport'
|
25
|
+
spec.add_runtime_dependency 'activemodel'
|
26
|
+
spec.add_runtime_dependency 'avro_turf'
|
27
27
|
|
28
|
-
spec.add_development_dependency
|
29
|
-
spec.add_development_dependency
|
30
|
-
spec.add_development_dependency
|
31
|
-
spec.add_development_dependency
|
32
|
-
spec.add_development_dependency
|
33
|
-
spec.add_development_dependency
|
28
|
+
spec.add_development_dependency 'bundler', '~> 1.11'
|
29
|
+
spec.add_development_dependency 'rake', '~> 10.0'
|
30
|
+
spec.add_development_dependency 'rspec', '~> 3.0'
|
31
|
+
spec.add_development_dependency 'simplecov'
|
32
|
+
spec.add_development_dependency 'webmock'
|
33
|
+
spec.add_development_dependency 'avro-builder', '>= 0.3.2'
|
34
34
|
# For FakeSchemaRegistryServer
|
35
|
-
spec.add_development_dependency
|
35
|
+
spec.add_development_dependency 'sinatra'
|
36
|
+
spec.add_development_dependency 'salsify_rubocop', '~> 0.40.0'
|
37
|
+
spec.add_development_dependency 'overcommit'
|
36
38
|
end
|
@@ -30,15 +30,17 @@ module Avromatic
|
|
30
30
|
private
|
31
31
|
|
32
32
|
def check_for_field_conflicts!
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
33
|
+
conflicts =
|
34
|
+
(key_avro_field_names & value_avro_field_names).each_with_object([]) do |name, msgs|
|
35
|
+
next unless schema_fields_differ?(name)
|
36
|
+
msgs << "Field '#{name}' has a different type in each schema: "\
|
37
|
+
"value #{value_avro_fields_by_name[name]}, "\
|
38
|
+
"key #{key_avro_fields_by_name[name]}"
|
39
|
+
end
|
40
|
+
|
41
|
+
raise conflicts.join("\n") if conflicts.any?
|
42
|
+
|
43
|
+
conflicts
|
42
44
|
end
|
43
45
|
|
44
46
|
# The Avro::Schema::Field#== method is lame. It just compares
|
@@ -5,7 +5,8 @@ require 'avromatic/model/configuration'
|
|
5
5
|
require 'avromatic/model/value_object'
|
6
6
|
require 'avromatic/model/configurable'
|
7
7
|
require 'avromatic/model/attributes'
|
8
|
-
require 'avromatic/model/
|
8
|
+
require 'avromatic/model/raw_serialization'
|
9
|
+
require 'avromatic/model/messaging_serialization'
|
9
10
|
|
10
11
|
module Avromatic
|
11
12
|
module Model
|
@@ -41,7 +42,8 @@ module Avromatic
|
|
41
42
|
Avromatic::Model::Configurable,
|
42
43
|
Avromatic::Model::Attributes,
|
43
44
|
Avromatic::Model::ValueObject,
|
44
|
-
Avromatic::Model::
|
45
|
+
Avromatic::Model::RawSerialization,
|
46
|
+
Avromatic::Model::MessagingSerialization
|
45
47
|
]
|
46
48
|
end
|
47
49
|
|
@@ -3,9 +3,10 @@ require 'avro_turf/schema_registry'
|
|
3
3
|
module Avromatic
|
4
4
|
module Model
|
5
5
|
|
6
|
-
# This class is used to decode Avro
|
7
|
-
|
8
|
-
|
6
|
+
# This class is used to decode messages encoded using Avro to their
|
7
|
+
# corresponding models.
|
8
|
+
class MessageDecoder
|
9
|
+
MAGIC_BYTE = [0].pack('C').freeze
|
9
10
|
|
10
11
|
class UnexpectedKeyError < StandardError
|
11
12
|
def initialize(schema_key)
|
@@ -13,16 +14,16 @@ module Avromatic
|
|
13
14
|
end
|
14
15
|
end
|
15
16
|
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
17
|
+
class MagicByteError < StandardError
|
18
|
+
def initialize(magic_byte)
|
19
|
+
super("Expected data to begin with a magic byte, got '#{magic_byte}'")
|
20
|
+
end
|
21
|
+
end
|
21
22
|
|
22
23
|
class DuplicateKeyError < StandardError
|
23
24
|
def initialize(*models)
|
24
25
|
super("Multiple models #{models} have the same key "\
|
25
|
-
"'#{Avromatic::Model::
|
26
|
+
"'#{Avromatic::Model::MessageDecoder.model_key(models.first)}'")
|
26
27
|
end
|
27
28
|
end
|
28
29
|
|
@@ -63,7 +64,7 @@ module Avromatic
|
|
63
64
|
|
64
65
|
def deserialize(model_key, message_key, message_value)
|
65
66
|
raise UnexpectedKeyError.new(model_key) unless model_map.key?(model_key)
|
66
|
-
model_map[model_key].
|
67
|
+
model_map[model_key].avro_message_decode(message_key, message_value)
|
67
68
|
end
|
68
69
|
|
69
70
|
def schema_name_for_data(data)
|
@@ -0,0 +1,62 @@
|
|
1
|
+
require 'avro_turf/messaging'
|
2
|
+
|
3
|
+
module Avromatic
|
4
|
+
module Model
|
5
|
+
|
6
|
+
# This concern adds support for serialization based on AvroTurf::Messaging.
|
7
|
+
# This serialization leverages a schema registry to prefix encoded values
|
8
|
+
# with an id for the schema.
|
9
|
+
module MessagingSerialization
|
10
|
+
extend ActiveSupport::Concern
|
11
|
+
|
12
|
+
delegate :avro_messaging, to: :class
|
13
|
+
private :avro_messaging
|
14
|
+
|
15
|
+
module Encode
|
16
|
+
def avro_message_value
|
17
|
+
avro_messaging.encode(
|
18
|
+
value_attributes_for_avro,
|
19
|
+
schema_name: value_avro_schema.fullname
|
20
|
+
)
|
21
|
+
end
|
22
|
+
|
23
|
+
def avro_message_key
|
24
|
+
raise 'Model has no key schema' unless key_avro_schema
|
25
|
+
avro_messaging.encode(
|
26
|
+
key_attributes_for_avro,
|
27
|
+
schema_name: key_avro_schema.fullname
|
28
|
+
)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
include Encode
|
32
|
+
|
33
|
+
# This module provides methods to decode an Avro-encoded value and
|
34
|
+
# an optional Avro-encoded key as a new model instance.
|
35
|
+
module Decode
|
36
|
+
|
37
|
+
# If two arguments are specified then the first is interpreted as the
|
38
|
+
# message key and the second is the message value. If there is only one
|
39
|
+
# arg then it is used as the message value.
|
40
|
+
def avro_message_decode(*args)
|
41
|
+
message_key, message_value = args.size > 1 ? args : [nil, args.first]
|
42
|
+
key_attributes = message_key &&
|
43
|
+
avro_messaging.decode(message_key, schema_name: key_avro_schema.fullname)
|
44
|
+
value_attributes = avro_messaging
|
45
|
+
.decode(message_value, schema_name: avro_schema.fullname)
|
46
|
+
|
47
|
+
new(value_attributes.merge!(key_attributes || {}))
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
module ClassMethods
|
52
|
+
# The messaging object acts as an intermediary talking to the schema
|
53
|
+
# registry and using returned/specified schemas to decode/encode.
|
54
|
+
def avro_messaging
|
55
|
+
Avromatic.messaging
|
56
|
+
end
|
57
|
+
|
58
|
+
include Decode
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
@@ -0,0 +1,111 @@
|
|
1
|
+
require 'avromatic/model/passthrough_serializer'
|
2
|
+
|
3
|
+
module Avromatic
|
4
|
+
module Model
|
5
|
+
|
6
|
+
# This module provides serialization support for encoding directly to Avro
|
7
|
+
# without dependency on a schema registry.
|
8
|
+
module RawSerialization
|
9
|
+
extend ActiveSupport::Concern
|
10
|
+
|
11
|
+
module Encode
|
12
|
+
delegate :avro_serializer, :datum_writer, :datum_reader, to: :class
|
13
|
+
private :avro_serializer, :datum_writer, :datum_reader
|
14
|
+
|
15
|
+
def avro_raw_value
|
16
|
+
avro_raw_encode(value_attributes_for_avro, :value)
|
17
|
+
end
|
18
|
+
|
19
|
+
def avro_raw_key
|
20
|
+
raise 'Model has no key schema' unless key_avro_schema
|
21
|
+
avro_raw_encode(key_attributes_for_avro, :key)
|
22
|
+
end
|
23
|
+
|
24
|
+
protected
|
25
|
+
|
26
|
+
def value_attributes_for_avro
|
27
|
+
avro_hash(value_avro_field_names)
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def key_attributes_for_avro
|
33
|
+
avro_hash(key_avro_field_names)
|
34
|
+
end
|
35
|
+
|
36
|
+
def avro_hash(fields)
|
37
|
+
attributes.slice(*fields).each_with_object(Hash.new) do |(key, value), result|
|
38
|
+
result[key.to_s] = if value.is_a?(Avromatic::Model::Attributes)
|
39
|
+
value.value_attributes_for_avro
|
40
|
+
else
|
41
|
+
avro_serializer[key].call(value)
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
def avro_raw_encode(data, key_or_value = :value)
|
47
|
+
stream = StringIO.new
|
48
|
+
encoder = Avro::IO::BinaryEncoder.new(stream)
|
49
|
+
datum_writer[key_or_value].write(data, encoder)
|
50
|
+
stream.string
|
51
|
+
end
|
52
|
+
end
|
53
|
+
include Encode
|
54
|
+
|
55
|
+
module Decode
|
56
|
+
|
57
|
+
# Create a new instance based on an encoded value and optional encoded key.
|
58
|
+
# If supplied then the key_schema and value_schema are used as the writer's
|
59
|
+
# schema for the corresponding value. The model's schemas are always used
|
60
|
+
# as the reader's schemas.
|
61
|
+
def avro_raw_decode(key: nil, value:, key_schema: nil, value_schema: nil)
|
62
|
+
key_attributes = key && decode_avro_datum(key, key_schema, :key)
|
63
|
+
value_attributes = decode_avro_datum(value, value_schema, :value)
|
64
|
+
|
65
|
+
new(value_attributes.merge!(key_attributes || {}))
|
66
|
+
end
|
67
|
+
|
68
|
+
private
|
69
|
+
|
70
|
+
def decode_avro_datum(data, schema = nil, key_or_value = :value)
|
71
|
+
stream = StringIO.new(data)
|
72
|
+
decoder = Avro::IO::BinaryDecoder.new(stream)
|
73
|
+
reader = schema ? custom_datum_reader(schema, key_or_value) : datum_reader[key_or_value]
|
74
|
+
reader.read(decoder)
|
75
|
+
end
|
76
|
+
|
77
|
+
def custom_datum_reader(schema, key_or_value)
|
78
|
+
Avro::IO::DatumReader.new(schema, send("#{key_or_value}_avro_schema"))
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
module ClassMethods
|
83
|
+
|
84
|
+
# Store a hash of Procs by field name (as a symbol) to convert
|
85
|
+
# the value before Avro serialization.
|
86
|
+
# Returns the default PassthroughSerializer if a key is not present.
|
87
|
+
def avro_serializer
|
88
|
+
@avro_serializer ||= Hash.new(PassthroughSerializer)
|
89
|
+
end
|
90
|
+
|
91
|
+
def datum_writer
|
92
|
+
@datum_writer ||= begin
|
93
|
+
hash = { value: Avro::IO::DatumWriter.new(value_avro_schema) }
|
94
|
+
hash[:key] = Avro::IO::DatumWriter.new(key_avro_schema) if key_avro_schema
|
95
|
+
hash
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
def datum_reader
|
100
|
+
@datum_reader ||= begin
|
101
|
+
hash = { value: Avro::IO::DatumReader.new(value_avro_schema) }
|
102
|
+
hash[:key] = Avro::IO::DatumReader.new(key_avro_schema) if key_avro_schema
|
103
|
+
hash
|
104
|
+
end
|
105
|
+
end
|
106
|
+
|
107
|
+
include Decode
|
108
|
+
end
|
109
|
+
end
|
110
|
+
end
|
111
|
+
end
|
@@ -14,11 +14,11 @@ module Avromatic
|
|
14
14
|
end
|
15
15
|
|
16
16
|
def inspect
|
17
|
-
"#<#{self.class.name} #{attributes.map { |k, v| "#{k}: #{v.inspect}" }.join(', ')
|
17
|
+
"#<#{self.class.name} #{attributes.map { |k, v| "#{k}: #{v.inspect}" }.join(', ')}>"
|
18
18
|
end
|
19
19
|
|
20
20
|
def to_s
|
21
|
-
|
21
|
+
format('#<%s:0x00%x>', self.class.name, object_id.abs * 2)
|
22
22
|
end
|
23
23
|
end
|
24
24
|
end
|
data/lib/avromatic/model.rb
CHANGED
data/lib/avromatic/version.rb
CHANGED
data/lib/avromatic.rb
CHANGED
@@ -21,7 +21,8 @@ module Avromatic
|
|
21
21
|
def self.build_schema_registry
|
22
22
|
raise 'Avromatic must be configured with a registry_url' unless registry_url
|
23
23
|
AvroTurf::CachedSchemaRegistry.new(
|
24
|
-
AvroTurf::SchemaRegistry.new(registry_url, logger: logger)
|
24
|
+
AvroTurf::SchemaRegistry.new(registry_url, logger: logger)
|
25
|
+
)
|
25
26
|
end
|
26
27
|
|
27
28
|
def self.build_messaging
|
@@ -29,7 +30,8 @@ module Avromatic
|
|
29
30
|
AvroTurf::Messaging.new(
|
30
31
|
registry: schema_registry || build_schema_registry,
|
31
32
|
schema_store: schema_store,
|
32
|
-
logger: logger
|
33
|
+
logger: logger
|
34
|
+
)
|
33
35
|
end
|
34
36
|
|
35
37
|
def self.build_messaging!
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: avromatic
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Salsify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-05-
|
11
|
+
date: 2016-05-25 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: avro
|
@@ -178,6 +178,34 @@ dependencies:
|
|
178
178
|
- - ">="
|
179
179
|
- !ruby/object:Gem::Version
|
180
180
|
version: '0'
|
181
|
+
- !ruby/object:Gem::Dependency
|
182
|
+
name: salsify_rubocop
|
183
|
+
requirement: !ruby/object:Gem::Requirement
|
184
|
+
requirements:
|
185
|
+
- - "~>"
|
186
|
+
- !ruby/object:Gem::Version
|
187
|
+
version: 0.40.0
|
188
|
+
type: :development
|
189
|
+
prerelease: false
|
190
|
+
version_requirements: !ruby/object:Gem::Requirement
|
191
|
+
requirements:
|
192
|
+
- - "~>"
|
193
|
+
- !ruby/object:Gem::Version
|
194
|
+
version: 0.40.0
|
195
|
+
- !ruby/object:Gem::Dependency
|
196
|
+
name: overcommit
|
197
|
+
requirement: !ruby/object:Gem::Requirement
|
198
|
+
requirements:
|
199
|
+
- - ">="
|
200
|
+
- !ruby/object:Gem::Version
|
201
|
+
version: '0'
|
202
|
+
type: :development
|
203
|
+
prerelease: false
|
204
|
+
version_requirements: !ruby/object:Gem::Requirement
|
205
|
+
requirements:
|
206
|
+
- - ">="
|
207
|
+
- !ruby/object:Gem::Version
|
208
|
+
version: '0'
|
181
209
|
description: Generate Ruby models from Avro schemas
|
182
210
|
email:
|
183
211
|
- engineering@salsify.com
|
@@ -186,7 +214,9 @@ extensions: []
|
|
186
214
|
extra_rdoc_files: []
|
187
215
|
files:
|
188
216
|
- ".gitignore"
|
217
|
+
- ".overcommit.yml"
|
189
218
|
- ".rspec"
|
219
|
+
- ".rubocop.yml"
|
190
220
|
- ".travis.yml"
|
191
221
|
- CHANGELOG.md
|
192
222
|
- Gemfile
|
@@ -203,10 +233,11 @@ files:
|
|
203
233
|
- lib/avromatic/model/configurable.rb
|
204
234
|
- lib/avromatic/model/configuration.rb
|
205
235
|
- lib/avromatic/model/custom_type.rb
|
206
|
-
- lib/avromatic/model/
|
236
|
+
- lib/avromatic/model/message_decoder.rb
|
237
|
+
- lib/avromatic/model/messaging_serialization.rb
|
207
238
|
- lib/avromatic/model/null_custom_type.rb
|
208
239
|
- lib/avromatic/model/passthrough_serializer.rb
|
209
|
-
- lib/avromatic/model/
|
240
|
+
- lib/avromatic/model/raw_serialization.rb
|
210
241
|
- lib/avromatic/model/type_registry.rb
|
211
242
|
- lib/avromatic/model/value_object.rb
|
212
243
|
- lib/avromatic/railtie.rb
|
@@ -1,86 +0,0 @@
|
|
1
|
-
require 'avro_turf/messaging'
|
2
|
-
require 'avromatic/model/passthrough_serializer'
|
3
|
-
|
4
|
-
module Avromatic
|
5
|
-
module Model
|
6
|
-
|
7
|
-
# This concern adds support for serialization to a model
|
8
|
-
# generated from Avro schema(s).
|
9
|
-
module Serialization
|
10
|
-
extend ActiveSupport::Concern
|
11
|
-
|
12
|
-
delegate :messaging, to: :Avromatic
|
13
|
-
|
14
|
-
module Encode
|
15
|
-
def avro_message_value
|
16
|
-
messaging.encode(
|
17
|
-
value_attributes_for_avro,
|
18
|
-
schema_name: value_avro_schema.fullname)
|
19
|
-
end
|
20
|
-
|
21
|
-
def avro_message_key
|
22
|
-
raise 'Model has no key schema' unless key_avro_schema
|
23
|
-
messaging.encode(
|
24
|
-
key_attributes_for_avro,
|
25
|
-
schema_name: key_avro_schema.fullname)
|
26
|
-
end
|
27
|
-
|
28
|
-
protected
|
29
|
-
|
30
|
-
def value_attributes_for_avro
|
31
|
-
avro_hash(value_avro_field_names)
|
32
|
-
end
|
33
|
-
|
34
|
-
private
|
35
|
-
|
36
|
-
delegate :avro_serializer, to: :class
|
37
|
-
|
38
|
-
def key_attributes_for_avro
|
39
|
-
avro_hash(key_avro_field_names)
|
40
|
-
end
|
41
|
-
|
42
|
-
def avro_hash(fields)
|
43
|
-
attributes.slice(*fields).each_with_object(Hash.new) do |(key, value), result|
|
44
|
-
result[key.to_s] = if value.is_a?(Avromatic::Model::Attributes)
|
45
|
-
value.value_attributes_for_avro
|
46
|
-
else
|
47
|
-
avro_serializer[key].call(value)
|
48
|
-
end
|
49
|
-
end
|
50
|
-
end
|
51
|
-
end
|
52
|
-
include Encode
|
53
|
-
|
54
|
-
# This module provides methods to deserialize an Avro-encoded value and
|
55
|
-
# an optional Avro-encoded key as a new model instance.
|
56
|
-
module Decode
|
57
|
-
|
58
|
-
# If two arguments are specified then the first is interpreted as the
|
59
|
-
# message key and the second is the message value. If there is only one
|
60
|
-
# arg then it is used as the message value.
|
61
|
-
def deserialize(*args)
|
62
|
-
message_key, message_value = args.size > 1 ? args : [nil, args.first]
|
63
|
-
key_attributes = message_key && messaging.decode(message_key, schema_name: key_avro_schema.fullname)
|
64
|
-
value_attributes = messaging.decode(message_value, schema_name: avro_schema.fullname)
|
65
|
-
|
66
|
-
new(value_attributes.merge!(key_attributes || {}))
|
67
|
-
end
|
68
|
-
end
|
69
|
-
|
70
|
-
module ClassMethods
|
71
|
-
# The messaging object acts as an intermediary talking to the schema
|
72
|
-
# registry and using returned/specified schemas to decode/encode.
|
73
|
-
delegate :messaging, to: :Avromatic
|
74
|
-
|
75
|
-
include Decode
|
76
|
-
|
77
|
-
# Store a hash of Procs by field name (as a symbol) to convert
|
78
|
-
# the value before Avro serialization.
|
79
|
-
# Returns the default PassthroughSerializer if a key is not present.
|
80
|
-
def avro_serializer
|
81
|
-
@avro_serializer ||= Hash.new(PassthroughSerializer)
|
82
|
-
end
|
83
|
-
end
|
84
|
-
end
|
85
|
-
end
|
86
|
-
end
|