avromatic 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -0
- data/README.md +57 -9
- data/lib/avromatic/model/attributes.rb +30 -12
- data/lib/avromatic/model/configuration.rb +1 -2
- data/lib/avromatic/model/custom_type.rb +55 -0
- data/lib/avromatic/model/null_custom_type.rb +21 -0
- data/lib/avromatic/model/passthrough_serializer.rb +10 -0
- data/lib/avromatic/model/serialization.rb +11 -1
- data/lib/avromatic/model/type_registry.rb +42 -0
- data/lib/avromatic/model.rb +1 -0
- data/lib/avromatic/version.rb +1 -1
- data/lib/avromatic.rb +5 -1
- metadata +6 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ca2b490c9b6978f339e2f6ea50c5a3476796fcae
|
4
|
+
data.tar.gz: 083c69aace3a37ceb0cce8f4c88fc5e5ade94611
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: aa9927585c4e68af3905fd6b54f219bc1b4831c3f2d1b85e600fe8a44ea84367bafd248dc1e007dbdbdd45e39a6aa6b98deef727a587de13a39c48b4fe1384de
|
7
|
+
data.tar.gz: 46086b401b88aedc6ddef32ecd0fb946d5f8767536686cf6807d95631e9095d070b591268eccc162f87f000ca52b17327c52616f741e31dedfa40f30880a00a7
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -4,8 +4,8 @@
|
|
4
4
|
|
5
5
|
[travis]: http://travis-ci.org/salsify/avromatic
|
6
6
|
|
7
|
-
`Avromatic` generates Ruby models from Avro schemas
|
8
|
-
encode and decode them.
|
7
|
+
`Avromatic` generates Ruby models from [Avro](http://avro.apache.org/) schemas
|
8
|
+
and provides utilities to encode and decode them.
|
9
9
|
|
10
10
|
## Installation
|
11
11
|
|
@@ -31,21 +31,31 @@ Or install it yourself as:
|
|
31
31
|
|
32
32
|
* schema_registry: An `AvroTurf::SchemaRegistry` object used to store Avro schemas
|
33
33
|
so that they can be referenced by id. Either `schema_registry` or
|
34
|
-
`registry_url` must be configured.
|
34
|
+
`registry_url` must be configured. See [Confluent Schema Registry](http://docs.confluent.io/2.0.1/schema-registry/docs/intro.html).
|
35
35
|
* registry_url: URL for the schema registry. The schema registry is used to store
|
36
36
|
Avro schemas so that they can be referenced by id. Either `schema_registry` or
|
37
37
|
`registry_url` must be configured.
|
38
38
|
* schema_store: The schema store is used to load Avro schemas from the filesystem.
|
39
39
|
It should be an object that responds to `find(name, namespace = nil)` and
|
40
|
-
returns an `Avro::Schema` object.
|
40
|
+
returns an `Avro::Schema` object. An `AvroTurf::SchemaStore` can be used.
|
41
41
|
* messaging: An `AvroTurf::Messaging` object to be shared by all generated models.
|
42
42
|
The `build_messaging!` method may be used to create a `Messaging` instance based
|
43
43
|
on the other configuration values.
|
44
|
-
* logger: The logger
|
44
|
+
* logger: The logger to use for the schema registry client.
|
45
|
+
* [Custom Types](#custom-types)
|
46
|
+
|
47
|
+
Example:
|
48
|
+
|
49
|
+
```ruby
|
50
|
+
Avromatic.configure do |config|
|
51
|
+
config.schema_store = AvroTurf::SchemaStore.new(path: 'avro/schemas')
|
52
|
+
config.registry_url = Rails.configuration.x.avro_schema_registry_url
|
53
|
+
config.build_messaging!
|
54
|
+
```
|
45
55
|
|
46
56
|
### Models
|
47
57
|
|
48
|
-
Models
|
58
|
+
Models are defined based on an Avro schema for a record.
|
49
59
|
|
50
60
|
The Avro schema can be specified by name and loaded using the schema store:
|
51
61
|
|
@@ -61,13 +71,12 @@ Or an `Avro::Schema` object can be specified directly:
|
|
61
71
|
class MyModel
|
62
72
|
include Avromatic::Model.build(schema: schema_object)
|
63
73
|
end
|
64
|
-
|
65
74
|
```
|
66
75
|
|
67
76
|
Models are generated as [Virtus](https://github.com/solnic/virtus) value
|
68
77
|
objects. `Virtus` attributes are added for each field in the Avro schema
|
69
78
|
including any default values defined in the schema. `ActiveModel` validations
|
70
|
-
are used to define validations on certain types of fields.
|
79
|
+
are used to define validations on certain types of fields ([see below](#validations)).
|
71
80
|
|
72
81
|
A model may be defined with both a key and a value schema:
|
73
82
|
|
@@ -88,6 +97,45 @@ constant:
|
|
88
97
|
MyModel = Avromatic::Model.model(schema_name :my_model)
|
89
98
|
```
|
90
99
|
|
100
|
+
#### Custom Types
|
101
|
+
|
102
|
+
Custom types can be configured for fields of named types (record, enum, fixed).
|
103
|
+
These customizations are registered on the `Avromatic` module. Once a custom type
|
104
|
+
is registered, it is used for all models with a schema that references that type.
|
105
|
+
It is recommended to register types within a block passed to `Avromatic.configure`:
|
106
|
+
|
107
|
+
```ruby
|
108
|
+
Avromatic.configure do |config|
|
109
|
+
config.register_type('com.example.my_string', MyString)
|
110
|
+
end
|
111
|
+
```
|
112
|
+
|
113
|
+
The full name of the type and an optional class may be specified. When a class is
|
114
|
+
provided then values for attributes of that type are defined using the specified
|
115
|
+
class.
|
116
|
+
|
117
|
+
If the provided class responds to the class methods `from_avro` and `to_avro`
|
118
|
+
then those methods are used to convert values when assigning to the model and
|
119
|
+
before encoding using Avro respectively.
|
120
|
+
|
121
|
+
`from_avro` and `to_avro` methods may be also be specified as Procs when
|
122
|
+
registering the type:
|
123
|
+
|
124
|
+
```ruby
|
125
|
+
Avromatic.configure do |config|
|
126
|
+
config.register_type('com.example.updown_string') do |type|
|
127
|
+
type.from_avro = ->(value) { value.upcase }
|
128
|
+
type.to_avro = ->(value) { value.downcase }
|
129
|
+
end
|
130
|
+
end
|
131
|
+
```
|
132
|
+
|
133
|
+
Nil handling is not required as the conversion methods are not be called if the
|
134
|
+
inbound or outbound value is nil.
|
135
|
+
|
136
|
+
If a custom type is registered for a record-type field, then any `to_avro`
|
137
|
+
method/Proc should return a Hash with string keys for encoding using Avro.
|
138
|
+
|
91
139
|
#### Encode/Decode
|
92
140
|
|
93
141
|
Models can be encoded using Avro leveraging a schema registry to encode a schema
|
@@ -97,7 +145,7 @@ id at the beginning of the value.
|
|
97
145
|
model.avro_message_value
|
98
146
|
```
|
99
147
|
|
100
|
-
If a model has
|
148
|
+
If a model has an Avro schema for a key, then the key can also be encoded
|
101
149
|
prefixed with a schema id.
|
102
150
|
|
103
151
|
```ruby
|
@@ -9,6 +9,15 @@ module Avromatic
|
|
9
9
|
module Attributes
|
10
10
|
extend ActiveSupport::Concern
|
11
11
|
|
12
|
+
def self.first_union_schema(field_type)
|
13
|
+
# TODO: This is a hack until I find a better solution for unions with
|
14
|
+
# Virtus. This only handles a union for an optional field with :null
|
15
|
+
# and one other type.
|
16
|
+
schemas = field_type.schemas.reject { |schema| schema.type_sym == :null }
|
17
|
+
raise "Only the union of null with one other type is supported #{field_type}" if schemas.size > 1
|
18
|
+
schemas.first
|
19
|
+
end
|
20
|
+
|
12
21
|
module ClassMethods
|
13
22
|
def add_avro_fields
|
14
23
|
if key_avro_schema
|
@@ -48,6 +57,7 @@ module Avromatic
|
|
48
57
|
avro_field_options(field))
|
49
58
|
|
50
59
|
add_validation(field)
|
60
|
+
add_serializer(field)
|
51
61
|
end
|
52
62
|
end
|
53
63
|
|
@@ -81,6 +91,9 @@ module Avromatic
|
|
81
91
|
end
|
82
92
|
|
83
93
|
def avro_field_class(field_type)
|
94
|
+
custom_type = Avromatic.type_registry.fetch(field_type)
|
95
|
+
return custom_type.value_class if custom_type.value_class
|
96
|
+
|
84
97
|
case field_type.type_sym
|
85
98
|
when :string, :bytes, :fixed
|
86
99
|
String
|
@@ -112,23 +125,28 @@ module Avromatic
|
|
112
125
|
end
|
113
126
|
|
114
127
|
def union_field_class(field_type)
|
115
|
-
|
116
|
-
# Virtus. This only handles a union for a optional field with :null
|
117
|
-
# and one other type.
|
118
|
-
schemas = field_type.schemas.reject { |schema| schema.type_sym == :null }
|
119
|
-
raise "Only the union of null with one other type is supported #{field_type}" if schemas.size > 1
|
120
|
-
avro_field_class(schemas.first)
|
128
|
+
avro_field_class(Avromatic::Model::Attributes.first_union_schema(field_type))
|
121
129
|
end
|
122
130
|
|
123
131
|
def avro_field_options(field)
|
132
|
+
options = {}
|
133
|
+
|
134
|
+
custom_type = Avromatic.type_registry.fetch(field)
|
135
|
+
coercer = custom_type.deserializer
|
136
|
+
options[:coercer] = coercer if coercer
|
137
|
+
|
124
138
|
if field.default
|
125
|
-
|
126
|
-
default: default_for(field.default),
|
127
|
-
lazy: true
|
128
|
-
}
|
129
|
-
else
|
130
|
-
{ }
|
139
|
+
options.merge!(default: default_for(field.default), lazy: true)
|
131
140
|
end
|
141
|
+
|
142
|
+
options
|
143
|
+
end
|
144
|
+
|
145
|
+
def add_serializer(field)
|
146
|
+
custom_type = Avromatic.type_registry.fetch(field)
|
147
|
+
serializer = custom_type.serializer
|
148
|
+
|
149
|
+
avro_serializer[field.name.to_sym] = serializer if serializer
|
132
150
|
end
|
133
151
|
|
134
152
|
def default_for(value)
|
@@ -1,7 +1,7 @@
|
|
1
1
|
module Avromatic
|
2
2
|
module Model
|
3
3
|
|
4
|
-
# This class holds configuration for a model
|
4
|
+
# This class holds configuration for a model built from Avro schema(s).
|
5
5
|
class Configuration
|
6
6
|
|
7
7
|
attr_reader :avro_schema, :key_avro_schema
|
@@ -17,7 +17,6 @@ module Avromatic
|
|
17
17
|
# @option options [String, Symbol] :value_schema_name
|
18
18
|
# @option options [Avro::Schema] :key_schema
|
19
19
|
# @option options [String, Symbol] :key_schema_name
|
20
|
-
# @option options [schema store] :schema_store
|
21
20
|
def initialize(**options)
|
22
21
|
@avro_schema = find_avro_schema(**options)
|
23
22
|
raise ArgumentError.new('value_schema(_name) or schema(_name) must be specified') unless avro_schema
|
@@ -0,0 +1,55 @@
|
|
1
|
+
require 'avromatic/model/null_custom_type'
|
2
|
+
|
3
|
+
module Avromatic
|
4
|
+
module Model
|
5
|
+
|
6
|
+
# Instances of this class contains the configuration for custom handling of
|
7
|
+
# a named type (record, enum, fixed).
|
8
|
+
class CustomType
|
9
|
+
|
10
|
+
attr_accessor :to_avro, :from_avro, :value_class
|
11
|
+
|
12
|
+
def initialize(value_class)
|
13
|
+
@value_class = value_class
|
14
|
+
end
|
15
|
+
|
16
|
+
# A deserializer method is used when assigning to the model. It is used both when
|
17
|
+
# deserializing a model instance from Avro and when directly instantiating
|
18
|
+
# an instance. The deserializer method must accept a single argument and return
|
19
|
+
# the value to store in the model for the attribute.
|
20
|
+
def deserializer
|
21
|
+
proc = from_avro_proc
|
22
|
+
wrap_proc(proc) if proc
|
23
|
+
end
|
24
|
+
|
25
|
+
# A serializer method is used when preparing attributes to be serialized using
|
26
|
+
# Avro. The serializer method must accept a single argument of the model value
|
27
|
+
# for the attribute and return a value in a form that Avro can serialize
|
28
|
+
# for the attribute.
|
29
|
+
def serializer
|
30
|
+
proc = to_avro_proc
|
31
|
+
wrap_proc(proc) if proc
|
32
|
+
end
|
33
|
+
|
34
|
+
private
|
35
|
+
|
36
|
+
def to_avro_proc
|
37
|
+
to_avro || value_class_method(:to_avro)
|
38
|
+
end
|
39
|
+
|
40
|
+
def from_avro_proc
|
41
|
+
from_avro || value_class_method(:from_avro)
|
42
|
+
end
|
43
|
+
|
44
|
+
def value_class_method(method_name)
|
45
|
+
value_class && value_class.respond_to?(method_name) &&
|
46
|
+
value_class.method(method_name).to_proc
|
47
|
+
end
|
48
|
+
|
49
|
+
# Wrap the supplied Proc to handle nil.
|
50
|
+
def wrap_proc(proc)
|
51
|
+
->(value) { proc.call(value) if value }
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
module Avromatic
|
2
|
+
module Model
|
3
|
+
|
4
|
+
# This module is used to implement the null object pattern for a CustomType.
|
5
|
+
module NullCustomType
|
6
|
+
class << self
|
7
|
+
def value_class
|
8
|
+
nil
|
9
|
+
end
|
10
|
+
|
11
|
+
def deserializer
|
12
|
+
nil
|
13
|
+
end
|
14
|
+
|
15
|
+
def serializer
|
16
|
+
nil
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -1,4 +1,5 @@
|
|
1
1
|
require 'avro_turf/messaging'
|
2
|
+
require 'avromatic/model/passthrough_serializer'
|
2
3
|
|
3
4
|
module Avromatic
|
4
5
|
module Model
|
@@ -32,6 +33,8 @@ module Avromatic
|
|
32
33
|
|
33
34
|
private
|
34
35
|
|
36
|
+
delegate :avro_serializer, to: :class
|
37
|
+
|
35
38
|
def key_attributes_for_avro
|
36
39
|
avro_hash(key_avro_field_names)
|
37
40
|
end
|
@@ -41,7 +44,7 @@ module Avromatic
|
|
41
44
|
result[key.to_s] = if value.is_a?(Avromatic::Model::Attributes)
|
42
45
|
value.value_attributes_for_avro
|
43
46
|
else
|
44
|
-
value
|
47
|
+
avro_serializer[key].call(value)
|
45
48
|
end
|
46
49
|
end
|
47
50
|
end
|
@@ -70,6 +73,13 @@ module Avromatic
|
|
70
73
|
delegate :messaging, to: :Avromatic
|
71
74
|
|
72
75
|
include Decode
|
76
|
+
|
77
|
+
# Store a hash of Procs by field name (as a symbol) to convert
|
78
|
+
# the value before Avro serialization.
|
79
|
+
# Returns the default PassthroughSerializer if a key is not present.
|
80
|
+
def avro_serializer
|
81
|
+
@avro_serializer ||= Hash.new(PassthroughSerializer)
|
82
|
+
end
|
73
83
|
end
|
74
84
|
end
|
75
85
|
end
|
@@ -0,0 +1,42 @@
|
|
1
|
+
require 'avromatic/model/custom_type'
|
2
|
+
|
3
|
+
module Avromatic
|
4
|
+
module Model
|
5
|
+
class TypeRegistry
|
6
|
+
|
7
|
+
delegate :clear, to: :custom_types
|
8
|
+
|
9
|
+
def initialize
|
10
|
+
@custom_types = {}
|
11
|
+
end
|
12
|
+
|
13
|
+
# @param fullname [String] The fullname of the Avro type.
|
14
|
+
# @param value_class [Class] Optional class to use for the attribute.
|
15
|
+
# If unspecified then the default class for the Avro field is used.
|
16
|
+
# @block If a block is specified then the CustomType is yielded for
|
17
|
+
# additional configuration.
|
18
|
+
def register_type(fullname, value_class = nil)
|
19
|
+
custom_types[fullname.to_s] = Avromatic::Model::CustomType.new(value_class).tap do |type|
|
20
|
+
yield(type) if block_given?
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
# @object [Avro::Schema] Custom type may be fetched based on a Avro field
|
25
|
+
# or schema. If there is no custom type, then NullCustomType is returned.
|
26
|
+
def fetch(object)
|
27
|
+
field_type = object.is_a?(Avro::Schema::Field) ? object.type : object
|
28
|
+
|
29
|
+
if field_type.type_sym == :union
|
30
|
+
field_type = Avromatic::Model::Attributes.first_union_schema(field_type)
|
31
|
+
end
|
32
|
+
|
33
|
+
fullname = field_type.fullname if field_type.is_a?(Avro::Schema::NamedSchema)
|
34
|
+
custom_types.fetch(fullname, NullCustomType)
|
35
|
+
end
|
36
|
+
|
37
|
+
private
|
38
|
+
|
39
|
+
attr_reader :custom_types
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
data/lib/avromatic/model.rb
CHANGED
data/lib/avromatic/version.rb
CHANGED
data/lib/avromatic.rb
CHANGED
@@ -5,10 +5,14 @@ require 'avro_turf/messaging'
|
|
5
5
|
|
6
6
|
module Avromatic
|
7
7
|
class << self
|
8
|
-
attr_accessor :schema_registry, :registry_url, :schema_store, :logger,
|
8
|
+
attr_accessor :schema_registry, :registry_url, :schema_store, :logger,
|
9
|
+
:messaging, :type_registry
|
10
|
+
|
11
|
+
delegate :register_type, to: :type_registry
|
9
12
|
end
|
10
13
|
|
11
14
|
self.logger = Logger.new($stdout)
|
15
|
+
self.type_registry = Avromatic::Model::TypeRegistry.new
|
12
16
|
|
13
17
|
def self.configure
|
14
18
|
yield self
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: avromatic
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Salsify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-05-
|
11
|
+
date: 2016-05-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: avro
|
@@ -202,8 +202,12 @@ files:
|
|
202
202
|
- lib/avromatic/model/builder.rb
|
203
203
|
- lib/avromatic/model/configurable.rb
|
204
204
|
- lib/avromatic/model/configuration.rb
|
205
|
+
- lib/avromatic/model/custom_type.rb
|
205
206
|
- lib/avromatic/model/decoder.rb
|
207
|
+
- lib/avromatic/model/null_custom_type.rb
|
208
|
+
- lib/avromatic/model/passthrough_serializer.rb
|
206
209
|
- lib/avromatic/model/serialization.rb
|
210
|
+
- lib/avromatic/model/type_registry.rb
|
207
211
|
- lib/avromatic/model/value_object.rb
|
208
212
|
- lib/avromatic/railtie.rb
|
209
213
|
- lib/avromatic/version.rb
|