avromatic 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: eda3c903923a23dfb499d2a2f0d60a063b1b305d
4
- data.tar.gz: 5b7cb13c8615dc6e8549892c80ec1dab6ccc8f3f
3
+ metadata.gz: ca2b490c9b6978f339e2f6ea50c5a3476796fcae
4
+ data.tar.gz: 083c69aace3a37ceb0cce8f4c88fc5e5ade94611
5
5
  SHA512:
6
- metadata.gz: c1d8cb9fcafb46abb997d1d4e72d4abc9a2fbb7171022f90c5207a466e3f254ea76e1d5e96b9a48013b66ab2b15a96e285795895a07415f9a67a26d24664a3dd
7
- data.tar.gz: 5c04568b634ac3c7be07662100d71281c62d06cc923bd74ddeca8bf47907ead80d787384593b7a0d5cbc23dd96ca144e2a32ce97f68f64459470693d0267ebe0
6
+ metadata.gz: aa9927585c4e68af3905fd6b54f219bc1b4831c3f2d1b85e600fe8a44ea84367bafd248dc1e007dbdbdd45e39a6aa6b98deef727a587de13a39c48b4fe1384de
7
+ data.tar.gz: 46086b401b88aedc6ddef32ecd0fb946d5f8767536686cf6807d95631e9095d070b591268eccc162f87f000ca52b17327c52616f741e31dedfa40f30880a00a7
data/CHANGELOG.md CHANGED
@@ -1,5 +1,9 @@
1
1
  # avromatic changelog
2
2
 
3
+ ## v0.4.0
4
+ - Allow the specification of a custom type, including conversion to/from Avro,
5
+ for named types.
6
+
3
7
  ## v0.3.0
4
8
  - Remove dependency on the `private_attr` gem.
5
9
 
data/README.md CHANGED
@@ -4,8 +4,8 @@
4
4
 
5
5
  [travis]: http://travis-ci.org/salsify/avromatic
6
6
 
7
- `Avromatic` generates Ruby models from Avro schemas and provides utilities to
8
- encode and decode them.
7
+ `Avromatic` generates Ruby models from [Avro](http://avro.apache.org/) schemas
8
+ and provides utilities to encode and decode them.
9
9
 
10
10
  ## Installation
11
11
 
@@ -31,21 +31,31 @@ Or install it yourself as:
31
31
 
32
32
  * schema_registry: An `AvroTurf::SchemaRegistry` object used to store Avro schemas
33
33
  so that they can be referenced by id. Either `schema_registry` or
34
- `registry_url` must be configured.
34
+ `registry_url` must be configured. See [Confluent Schema Registry](http://docs.confluent.io/2.0.1/schema-registry/docs/intro.html).
35
35
  * registry_url: URL for the schema registry. The schema registry is used to store
36
36
  Avro schemas so that they can be referenced by id. Either `schema_registry` or
37
37
  `registry_url` must be configured.
38
38
  * schema_store: The schema store is used to load Avro schemas from the filesystem.
39
39
  It should be an object that responds to `find(name, namespace = nil)` and
40
- returns an `Avro::Schema` object.
40
+ returns an `Avro::Schema` object. An `AvroTurf::SchemaStore` can be used.
41
41
  * messaging: An `AvroTurf::Messaging` object to be shared by all generated models.
42
42
  The `build_messaging!` method may be used to create a `Messaging` instance based
43
43
  on the other configuration values.
44
- * logger: The logger is for the schema registry client.
44
+ * logger: The logger to use for the schema registry client.
45
+ * [Custom Types](#custom-types)
46
+
47
+ Example:
48
+
49
+ ```ruby
50
+ Avromatic.configure do |config|
51
+ config.schema_store = AvroTurf::SchemaStore.new(path: 'avro/schemas')
52
+ config.registry_url = Rails.configuration.x.avro_schema_registry_url
53
+ config.build_messaging!
54
+ ```
45
55
 
46
56
  ### Models
47
57
 
48
- Models may be defined based on an Avro schema for a record.
58
+ Models are defined based on an Avro schema for a record.
49
59
 
50
60
  The Avro schema can be specified by name and loaded using the schema store:
51
61
 
@@ -61,13 +71,12 @@ Or an `Avro::Schema` object can be specified directly:
61
71
  class MyModel
62
72
  include Avromatic::Model.build(schema: schema_object)
63
73
  end
64
-
65
74
  ```
66
75
 
67
76
  Models are generated as [Virtus](https://github.com/solnic/virtus) value
68
77
  objects. `Virtus` attributes are added for each field in the Avro schema
69
78
  including any default values defined in the schema. `ActiveModel` validations
70
- are used to define validations on certain types of fields.
79
+ are used to define validations on certain types of fields ([see below](#validations)).
71
80
 
72
81
  A model may be defined with both a key and a value schema:
73
82
 
@@ -88,6 +97,45 @@ constant:
88
97
  MyModel = Avromatic::Model.model(schema_name :my_model)
89
98
  ```
90
99
 
100
+ #### Custom Types
101
+
102
+ Custom types can be configured for fields of named types (record, enum, fixed).
103
+ These customizations are registered on the `Avromatic` module. Once a custom type
104
+ is registered, it is used for all models with a schema that references that type.
105
+ It is recommended to register types within a block passed to `Avromatic.configure`:
106
+
107
+ ```ruby
108
+ Avromatic.configure do |config|
109
+ config.register_type('com.example.my_string', MyString)
110
+ end
111
+ ```
112
+
113
+ The full name of the type and an optional class may be specified. When a class is
114
+ provided then values for attributes of that type are defined using the specified
115
+ class.
116
+
117
+ If the provided class responds to the class methods `from_avro` and `to_avro`
118
+ then those methods are used to convert values when assigning to the model and
119
+ before encoding using Avro respectively.
120
+
121
+ `from_avro` and `to_avro` methods may be also be specified as Procs when
122
+ registering the type:
123
+
124
+ ```ruby
125
+ Avromatic.configure do |config|
126
+ config.register_type('com.example.updown_string') do |type|
127
+ type.from_avro = ->(value) { value.upcase }
128
+ type.to_avro = ->(value) { value.downcase }
129
+ end
130
+ end
131
+ ```
132
+
133
+ Nil handling is not required as the conversion methods are not be called if the
134
+ inbound or outbound value is nil.
135
+
136
+ If a custom type is registered for a record-type field, then any `to_avro`
137
+ method/Proc should return a Hash with string keys for encoding using Avro.
138
+
91
139
  #### Encode/Decode
92
140
 
93
141
  Models can be encoded using Avro leveraging a schema registry to encode a schema
@@ -97,7 +145,7 @@ id at the beginning of the value.
97
145
  model.avro_message_value
98
146
  ```
99
147
 
100
- If a model has a Avro schema for a key, then the key can also be encoded
148
+ If a model has an Avro schema for a key, then the key can also be encoded
101
149
  prefixed with a schema id.
102
150
 
103
151
  ```ruby
@@ -9,6 +9,15 @@ module Avromatic
9
9
  module Attributes
10
10
  extend ActiveSupport::Concern
11
11
 
12
+ def self.first_union_schema(field_type)
13
+ # TODO: This is a hack until I find a better solution for unions with
14
+ # Virtus. This only handles a union for an optional field with :null
15
+ # and one other type.
16
+ schemas = field_type.schemas.reject { |schema| schema.type_sym == :null }
17
+ raise "Only the union of null with one other type is supported #{field_type}" if schemas.size > 1
18
+ schemas.first
19
+ end
20
+
12
21
  module ClassMethods
13
22
  def add_avro_fields
14
23
  if key_avro_schema
@@ -48,6 +57,7 @@ module Avromatic
48
57
  avro_field_options(field))
49
58
 
50
59
  add_validation(field)
60
+ add_serializer(field)
51
61
  end
52
62
  end
53
63
 
@@ -81,6 +91,9 @@ module Avromatic
81
91
  end
82
92
 
83
93
  def avro_field_class(field_type)
94
+ custom_type = Avromatic.type_registry.fetch(field_type)
95
+ return custom_type.value_class if custom_type.value_class
96
+
84
97
  case field_type.type_sym
85
98
  when :string, :bytes, :fixed
86
99
  String
@@ -112,23 +125,28 @@ module Avromatic
112
125
  end
113
126
 
114
127
  def union_field_class(field_type)
115
- # TODO: This is a hack until I find a better solution for unions with
116
- # Virtus. This only handles a union for a optional field with :null
117
- # and one other type.
118
- schemas = field_type.schemas.reject { |schema| schema.type_sym == :null }
119
- raise "Only the union of null with one other type is supported #{field_type}" if schemas.size > 1
120
- avro_field_class(schemas.first)
128
+ avro_field_class(Avromatic::Model::Attributes.first_union_schema(field_type))
121
129
  end
122
130
 
123
131
  def avro_field_options(field)
132
+ options = {}
133
+
134
+ custom_type = Avromatic.type_registry.fetch(field)
135
+ coercer = custom_type.deserializer
136
+ options[:coercer] = coercer if coercer
137
+
124
138
  if field.default
125
- {
126
- default: default_for(field.default),
127
- lazy: true
128
- }
129
- else
130
- { }
139
+ options.merge!(default: default_for(field.default), lazy: true)
131
140
  end
141
+
142
+ options
143
+ end
144
+
145
+ def add_serializer(field)
146
+ custom_type = Avromatic.type_registry.fetch(field)
147
+ serializer = custom_type.serializer
148
+
149
+ avro_serializer[field.name.to_sym] = serializer if serializer
132
150
  end
133
151
 
134
152
  def default_for(value)
@@ -1,7 +1,7 @@
1
1
  module Avromatic
2
2
  module Model
3
3
 
4
- # This class holds configuration for a model build from Avro schema(s).
4
+ # This class holds configuration for a model built from Avro schema(s).
5
5
  class Configuration
6
6
 
7
7
  attr_reader :avro_schema, :key_avro_schema
@@ -17,7 +17,6 @@ module Avromatic
17
17
  # @option options [String, Symbol] :value_schema_name
18
18
  # @option options [Avro::Schema] :key_schema
19
19
  # @option options [String, Symbol] :key_schema_name
20
- # @option options [schema store] :schema_store
21
20
  def initialize(**options)
22
21
  @avro_schema = find_avro_schema(**options)
23
22
  raise ArgumentError.new('value_schema(_name) or schema(_name) must be specified') unless avro_schema
@@ -0,0 +1,55 @@
1
+ require 'avromatic/model/null_custom_type'
2
+
3
+ module Avromatic
4
+ module Model
5
+
6
+ # Instances of this class contains the configuration for custom handling of
7
+ # a named type (record, enum, fixed).
8
+ class CustomType
9
+
10
+ attr_accessor :to_avro, :from_avro, :value_class
11
+
12
+ def initialize(value_class)
13
+ @value_class = value_class
14
+ end
15
+
16
+ # A deserializer method is used when assigning to the model. It is used both when
17
+ # deserializing a model instance from Avro and when directly instantiating
18
+ # an instance. The deserializer method must accept a single argument and return
19
+ # the value to store in the model for the attribute.
20
+ def deserializer
21
+ proc = from_avro_proc
22
+ wrap_proc(proc) if proc
23
+ end
24
+
25
+ # A serializer method is used when preparing attributes to be serialized using
26
+ # Avro. The serializer method must accept a single argument of the model value
27
+ # for the attribute and return a value in a form that Avro can serialize
28
+ # for the attribute.
29
+ def serializer
30
+ proc = to_avro_proc
31
+ wrap_proc(proc) if proc
32
+ end
33
+
34
+ private
35
+
36
+ def to_avro_proc
37
+ to_avro || value_class_method(:to_avro)
38
+ end
39
+
40
+ def from_avro_proc
41
+ from_avro || value_class_method(:from_avro)
42
+ end
43
+
44
+ def value_class_method(method_name)
45
+ value_class && value_class.respond_to?(method_name) &&
46
+ value_class.method(method_name).to_proc
47
+ end
48
+
49
+ # Wrap the supplied Proc to handle nil.
50
+ def wrap_proc(proc)
51
+ ->(value) { proc.call(value) if value }
52
+ end
53
+ end
54
+ end
55
+ end
@@ -0,0 +1,21 @@
1
+ module Avromatic
2
+ module Model
3
+
4
+ # This module is used to implement the null object pattern for a CustomType.
5
+ module NullCustomType
6
+ class << self
7
+ def value_class
8
+ nil
9
+ end
10
+
11
+ def deserializer
12
+ nil
13
+ end
14
+
15
+ def serializer
16
+ nil
17
+ end
18
+ end
19
+ end
20
+ end
21
+ end
@@ -0,0 +1,10 @@
1
+ module Avromatic
2
+ module Model
3
+ # This trivial serializer simply returns the value provided.
4
+ module PassthroughSerializer
5
+ def self.call(value)
6
+ value
7
+ end
8
+ end
9
+ end
10
+ end
@@ -1,4 +1,5 @@
1
1
  require 'avro_turf/messaging'
2
+ require 'avromatic/model/passthrough_serializer'
2
3
 
3
4
  module Avromatic
4
5
  module Model
@@ -32,6 +33,8 @@ module Avromatic
32
33
 
33
34
  private
34
35
 
36
+ delegate :avro_serializer, to: :class
37
+
35
38
  def key_attributes_for_avro
36
39
  avro_hash(key_avro_field_names)
37
40
  end
@@ -41,7 +44,7 @@ module Avromatic
41
44
  result[key.to_s] = if value.is_a?(Avromatic::Model::Attributes)
42
45
  value.value_attributes_for_avro
43
46
  else
44
- value
47
+ avro_serializer[key].call(value)
45
48
  end
46
49
  end
47
50
  end
@@ -70,6 +73,13 @@ module Avromatic
70
73
  delegate :messaging, to: :Avromatic
71
74
 
72
75
  include Decode
76
+
77
+ # Store a hash of Procs by field name (as a symbol) to convert
78
+ # the value before Avro serialization.
79
+ # Returns the default PassthroughSerializer if a key is not present.
80
+ def avro_serializer
81
+ @avro_serializer ||= Hash.new(PassthroughSerializer)
82
+ end
73
83
  end
74
84
  end
75
85
  end
@@ -0,0 +1,42 @@
1
+ require 'avromatic/model/custom_type'
2
+
3
+ module Avromatic
4
+ module Model
5
+ class TypeRegistry
6
+
7
+ delegate :clear, to: :custom_types
8
+
9
+ def initialize
10
+ @custom_types = {}
11
+ end
12
+
13
+ # @param fullname [String] The fullname of the Avro type.
14
+ # @param value_class [Class] Optional class to use for the attribute.
15
+ # If unspecified then the default class for the Avro field is used.
16
+ # @block If a block is specified then the CustomType is yielded for
17
+ # additional configuration.
18
+ def register_type(fullname, value_class = nil)
19
+ custom_types[fullname.to_s] = Avromatic::Model::CustomType.new(value_class).tap do |type|
20
+ yield(type) if block_given?
21
+ end
22
+ end
23
+
24
+ # @object [Avro::Schema] Custom type may be fetched based on a Avro field
25
+ # or schema. If there is no custom type, then NullCustomType is returned.
26
+ def fetch(object)
27
+ field_type = object.is_a?(Avro::Schema::Field) ? object.type : object
28
+
29
+ if field_type.type_sym == :union
30
+ field_type = Avromatic::Model::Attributes.first_union_schema(field_type)
31
+ end
32
+
33
+ fullname = field_type.fullname if field_type.is_a?(Avro::Schema::NamedSchema)
34
+ custom_types.fetch(fullname, NullCustomType)
35
+ end
36
+
37
+ private
38
+
39
+ attr_reader :custom_types
40
+ end
41
+ end
42
+ end
@@ -1,5 +1,6 @@
1
1
  require 'avromatic/model/builder'
2
2
  require 'avromatic/model/decoder'
3
+ require 'avromatic/model/type_registry'
3
4
 
4
5
  module Avromatic
5
6
  module Model
@@ -1,3 +1,3 @@
1
1
  module Avromatic
2
- VERSION = "0.3.0"
2
+ VERSION = "0.4.0"
3
3
  end
data/lib/avromatic.rb CHANGED
@@ -5,10 +5,14 @@ require 'avro_turf/messaging'
5
5
 
6
6
  module Avromatic
7
7
  class << self
8
- attr_accessor :schema_registry, :registry_url, :schema_store, :logger, :messaging
8
+ attr_accessor :schema_registry, :registry_url, :schema_store, :logger,
9
+ :messaging, :type_registry
10
+
11
+ delegate :register_type, to: :type_registry
9
12
  end
10
13
 
11
14
  self.logger = Logger.new($stdout)
15
+ self.type_registry = Avromatic::Model::TypeRegistry.new
12
16
 
13
17
  def self.configure
14
18
  yield self
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: avromatic
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.0
4
+ version: 0.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Salsify Engineering
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2016-05-16 00:00:00.000000000 Z
11
+ date: 2016-05-17 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: avro
@@ -202,8 +202,12 @@ files:
202
202
  - lib/avromatic/model/builder.rb
203
203
  - lib/avromatic/model/configurable.rb
204
204
  - lib/avromatic/model/configuration.rb
205
+ - lib/avromatic/model/custom_type.rb
205
206
  - lib/avromatic/model/decoder.rb
207
+ - lib/avromatic/model/null_custom_type.rb
208
+ - lib/avromatic/model/passthrough_serializer.rb
206
209
  - lib/avromatic/model/serialization.rb
210
+ - lib/avromatic/model/type_registry.rb
207
211
  - lib/avromatic/model/value_object.rb
208
212
  - lib/avromatic/railtie.rb
209
213
  - lib/avromatic/version.rb