avro_turf 0.7.2 → 0.8.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +33 -15
- data/avro_turf.gemspec +10 -0
- data/lib/avro_turf/cached_confluent_schema_registry.rb +27 -0
- data/lib/avro_turf/cached_schema_registry.rb +4 -24
- data/lib/avro_turf/confluent_schema_registry.rb +106 -0
- data/lib/avro_turf/messaging.rb +7 -2
- data/lib/avro_turf/schema_registry.rb +4 -77
- data/lib/avro_turf/test/fake_confluent_schema_registry_server.rb +129 -0
- data/lib/avro_turf/test/fake_schema_registry_server.rb +4 -82
- data/lib/avro_turf/version.rb +1 -1
- data/spec/{cached_schema_registry_spec.rb → cached_confluent_schema_registry_spec.rb} +6 -6
- data/spec/confluent_schema_registry_spec.rb +9 -0
- data/spec/messaging_spec.rb +4 -4
- data/spec/support/{schema_registry_context.rb → confluent_schema_registry_context.rb} +64 -3
- metadata +21 -10
- data/spec/schema_registry_spec.rb +0 -9
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a708b9aabeca7d45e1db532e180b2d80e4a5aecb
|
4
|
+
data.tar.gz: acb21f2435fd5126efed47803395d84eb4f5a220
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4bcc5e9832804eafb4f295a6fc85273a5a52c17a515f87ab3da0deb31034ed428c6aaee3754abcaa45d78d37daf6062c6c1d1007644f8578e2561b17f70f1614
|
7
|
+
data.tar.gz: 89071ed406d0be937344cc70814499e29206f05e52484fd4ba4f797f12bcb9236c36c4b080f22ffedd815ac3f6081e13405cb10fa1dc4184cfe73c8bbb279952
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,9 @@
|
|
1
|
+
# avro_turf
|
2
|
+
|
3
|
+
## v0.8.0
|
4
|
+
- The names `AvroTurf::SchemaRegistry`, `AvroTurf::CachedSchemaRegistry`, and
|
5
|
+
`FakeSchemaRegistryServer` are deprecated and will be removed in a future release.
|
6
|
+
Use `AvroTurf::ConfluentSchemaRegistry`, `AvroTurf::CachedConfluentSchemaRegistry`,
|
7
|
+
and `FakeConfluentSchemaRegistryServer` instead.
|
8
|
+
- Add support for the Config API (http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#config)
|
9
|
+
to `AvroTurf::ConfluentSchemaRegistry`.
|
data/README.md
CHANGED
@@ -5,6 +5,17 @@ AvroTurf is a library that makes it easier to encode and decode data using the [
|
|
5
5
|
* Provides an idiomatic Ruby interface.
|
6
6
|
* Allows referencing schemas defined in another file.
|
7
7
|
|
8
|
+
## Deprecation Notice
|
9
|
+
|
10
|
+
The `AvroTurf::SchemaRegistry`, `AvroTurf::CachedSchemaRegistry`,
|
11
|
+
and `FakeSchemaRegistryServer` names have been deprecated because the Avro spec recently
|
12
|
+
introduced an incompatible [single-message encoding format](https://github.com/apache/avro/commit/30408a9c192c5f4eaaf42f01f0ffbfffd705aa57).
|
13
|
+
|
14
|
+
These classes have been renamed to `AvroTurf::ConfluentSchemaRegistry`,
|
15
|
+
`AvroTurf::CachedConfluentSchemaRegistry`, and `FakeConfluentSchemaRegistry`.
|
16
|
+
|
17
|
+
The aliases for the original names will be removed in a future release.
|
18
|
+
|
8
19
|
## Installation
|
9
20
|
|
10
21
|
Add this line to your application's Gemfile:
|
@@ -80,11 +91,8 @@ In the example above, the `person` schema references the `address` schema, even
|
|
80
91
|
```json
|
81
92
|
// person_list.avsc
|
82
93
|
{
|
83
|
-
"
|
84
|
-
"
|
85
|
-
"type": "array",
|
86
|
-
"items": "person"
|
87
|
-
}
|
94
|
+
"type": "array",
|
95
|
+
"items": "person"
|
88
96
|
}
|
89
97
|
```
|
90
98
|
|
@@ -121,11 +129,14 @@ data = avro.encode({ "title" => "hello, world" }, schema_name: "greeting")
|
|
121
129
|
avro.decode(data) #=> { "title" => "hello, world" }
|
122
130
|
```
|
123
131
|
|
124
|
-
|
125
|
-
|
132
|
+
### Confluent Schema Registry Client
|
133
|
+
|
134
|
+
The ConfluentSchemaRegistry client used by the Messaging API can also be used directly.
|
135
|
+
It can check whether a schema is compatible with a subject in the registry using the [Compatibility API](http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#compatibility):
|
126
136
|
|
127
137
|
```ruby
|
128
|
-
require 'avro_turf
|
138
|
+
require 'avro_turf'
|
139
|
+
require 'avro_turf/confluent_schema_registry'
|
129
140
|
|
130
141
|
schema = <<-JSON
|
131
142
|
{
|
@@ -144,15 +155,22 @@ schema = <<-JSON
|
|
144
155
|
}
|
145
156
|
JSON
|
146
157
|
|
147
|
-
|
158
|
+
registry = AvroTurf::ConfluentSchemaRegistry.new("http://my-registry:8081/")
|
159
|
+
|
160
|
+
# Returns true if the schema is compatible, nil if the subject or version is not registered, and false if incompatible.
|
161
|
+
registry.compatible?("person", schema)
|
162
|
+
```
|
148
163
|
|
149
|
-
|
150
|
-
|
164
|
+
The ConfluentSchemaRegistry client can also change the global compatibility level or the compatibility level for an individual subject using the [Config API](http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#config):
|
165
|
+
|
166
|
+
```ruby
|
167
|
+
registry.update_global_config(compatibility: 'FULL')
|
168
|
+
registry.update_subject_config("person", compatibility: 'NONE')
|
151
169
|
```
|
152
170
|
|
153
171
|
### Testing Support
|
154
172
|
|
155
|
-
AvroTurf includes a `
|
173
|
+
AvroTurf includes a `FakeConfluentSchemaRegistryServer` that can be used in tests. The
|
156
174
|
fake schema registry server depends on Sinatra but it is _not_ listed as a runtime
|
157
175
|
dependency for AvroTurf. Sinatra must be added to your Gemfile or gemspec in order
|
158
176
|
to use the fake server.
|
@@ -160,14 +178,14 @@ to use the fake server.
|
|
160
178
|
Example using RSpec:
|
161
179
|
|
162
180
|
```ruby
|
163
|
-
require 'avro_turf/test/
|
181
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
164
182
|
require 'webmock/rspec'
|
165
183
|
|
166
184
|
# within an example
|
167
185
|
let(:registry_url) { "http://registry.example.com" }
|
168
186
|
before do
|
169
|
-
stub_request(:any, /^#{registry_url}/).to_rack(
|
170
|
-
|
187
|
+
stub_request(:any, /^#{registry_url}/).to_rack(FakeConfluentSchemaRegistryServer)
|
188
|
+
FakeConfluentSchemaRegistryServer.clear
|
171
189
|
end
|
172
190
|
|
173
191
|
# Messaging objects created with the same registry_url will now use the fake server.
|
data/avro_turf.gemspec
CHANGED
@@ -27,4 +27,14 @@ Gem::Specification.new do |spec|
|
|
27
27
|
spec.add_development_dependency "webmock"
|
28
28
|
spec.add_development_dependency "sinatra"
|
29
29
|
spec.add_development_dependency "json_spec"
|
30
|
+
|
31
|
+
spec.post_install_message = %{
|
32
|
+
avro_turf v0.8.0 deprecates the names AvroTurf::SchemaRegistry,
|
33
|
+
AvroTurf::CachedSchemaRegistry, and FakeSchemaRegistryServer.
|
34
|
+
|
35
|
+
Use AvroTurf::ConfluentSchemaRegistry, AvroTurf::CachedConfluentSchemaRegistry,
|
36
|
+
and FakeConfluentSchemaRegistryServer instead.
|
37
|
+
|
38
|
+
See https://github.com/dasch/avro_turf#deprecation-notice
|
39
|
+
}
|
30
40
|
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
require 'avro_turf/confluent_schema_registry'
|
2
|
+
|
3
|
+
# Caches registrations and lookups to the schema registry in memory.
|
4
|
+
class AvroTurf::CachedConfluentSchemaRegistry
|
5
|
+
|
6
|
+
def initialize(upstream)
|
7
|
+
@upstream = upstream
|
8
|
+
@schemas_by_id = {}
|
9
|
+
@ids_by_schema = {}
|
10
|
+
end
|
11
|
+
|
12
|
+
# Delegate the following methods to the upstream
|
13
|
+
%i(subjects subject_versions subject_version check compatible?
|
14
|
+
global_config update_global_config subject_config update_subject_config).each do |name|
|
15
|
+
define_method(name) do |*args|
|
16
|
+
instance_variable_get(:@upstream).send(name, *args)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def fetch(id)
|
21
|
+
@schemas_by_id[id] ||= @upstream.fetch(id)
|
22
|
+
end
|
23
|
+
|
24
|
+
def register(subject, schema)
|
25
|
+
@ids_by_schema[subject + schema.to_s] ||= @upstream.register(subject, schema)
|
26
|
+
end
|
27
|
+
end
|
@@ -1,26 +1,6 @@
|
|
1
|
-
require 'avro_turf/
|
1
|
+
require 'avro_turf/cached_confluent_schema_registry'
|
2
2
|
|
3
|
-
#
|
4
|
-
|
3
|
+
# AvroTurf::CachedSchemaRegistry is deprecated and will be removed in a future release.
|
4
|
+
# Use AvroTurf::CachedConfluentSchemaRegistry instead.
|
5
5
|
|
6
|
-
|
7
|
-
@upstream = upstream
|
8
|
-
@schemas_by_id = {}
|
9
|
-
@ids_by_schema = {}
|
10
|
-
end
|
11
|
-
|
12
|
-
# Delegate the following methods to the upstream
|
13
|
-
%i(subjects subject_versions subject_version check compatible?).each do |name|
|
14
|
-
define_method(name) do |*args|
|
15
|
-
instance_variable_get(:@upstream).send(name, *args)
|
16
|
-
end
|
17
|
-
end
|
18
|
-
|
19
|
-
def fetch(id)
|
20
|
-
@schemas_by_id[id] ||= @upstream.fetch(id)
|
21
|
-
end
|
22
|
-
|
23
|
-
def register(subject, schema)
|
24
|
-
@ids_by_schema[subject + schema.to_s] ||= @upstream.register(subject, schema)
|
25
|
-
end
|
26
|
-
end
|
6
|
+
AvroTurf::CachedSchemaRegistry = AvroTurf::CachedConfluentSchemaRegistry
|
@@ -0,0 +1,106 @@
|
|
1
|
+
require 'excon'
|
2
|
+
|
3
|
+
class AvroTurf::ConfluentSchemaRegistry
|
4
|
+
CONTENT_TYPE = "application/vnd.schemaregistry.v1+json".freeze
|
5
|
+
|
6
|
+
def initialize(url, logger: Logger.new($stdout))
|
7
|
+
@logger = logger
|
8
|
+
@connection = Excon.new(url, headers: {
|
9
|
+
"Content-Type" => CONTENT_TYPE,
|
10
|
+
})
|
11
|
+
end
|
12
|
+
|
13
|
+
def fetch(id)
|
14
|
+
@logger.info "Fetching schema with id #{id}"
|
15
|
+
data = get("/schemas/ids/#{id}")
|
16
|
+
data.fetch("schema")
|
17
|
+
end
|
18
|
+
|
19
|
+
def register(subject, schema)
|
20
|
+
data = post("/subjects/#{subject}/versions", body: {
|
21
|
+
schema: schema.to_s
|
22
|
+
}.to_json)
|
23
|
+
|
24
|
+
id = data.fetch("id")
|
25
|
+
|
26
|
+
@logger.info "Registered schema for subject `#{subject}`; id = #{id}"
|
27
|
+
|
28
|
+
id
|
29
|
+
end
|
30
|
+
|
31
|
+
# List all subjects
|
32
|
+
def subjects
|
33
|
+
get('/subjects')
|
34
|
+
end
|
35
|
+
|
36
|
+
# List all versions for a subject
|
37
|
+
def subject_versions(subject)
|
38
|
+
get("/subjects/#{subject}/versions")
|
39
|
+
end
|
40
|
+
|
41
|
+
# Get a specific version for a subject
|
42
|
+
def subject_version(subject, version = 'latest')
|
43
|
+
get("/subjects/#{subject}/versions/#{version}")
|
44
|
+
end
|
45
|
+
|
46
|
+
# Check if a schema exists. Returns nil if not found.
|
47
|
+
def check(subject, schema)
|
48
|
+
data = post("/subjects/#{subject}",
|
49
|
+
expects: [200, 404],
|
50
|
+
body: { schema: schema.to_s }.to_json)
|
51
|
+
data unless data.has_key?("error_code")
|
52
|
+
end
|
53
|
+
|
54
|
+
# Check if a schema is compatible with the stored version.
|
55
|
+
# Returns:
|
56
|
+
# - true if compatible
|
57
|
+
# - nil if the subject or version does not exist
|
58
|
+
# - false if incompatible
|
59
|
+
# http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#compatibility
|
60
|
+
def compatible?(subject, schema, version = 'latest')
|
61
|
+
data = post("/compatibility/subjects/#{subject}/versions/#{version}",
|
62
|
+
expects: [200, 404],
|
63
|
+
body: { schema: schema.to_s }.to_json)
|
64
|
+
data.fetch('is_compatible', false) unless data.has_key?('error_code')
|
65
|
+
end
|
66
|
+
|
67
|
+
# Get global config
|
68
|
+
def global_config
|
69
|
+
get("/config")
|
70
|
+
end
|
71
|
+
|
72
|
+
# Update global config
|
73
|
+
def update_global_config(config)
|
74
|
+
put("/config", { body: config.to_json })
|
75
|
+
end
|
76
|
+
|
77
|
+
# Get config for subject
|
78
|
+
def subject_config(subject)
|
79
|
+
get("/config/#{subject}")
|
80
|
+
end
|
81
|
+
|
82
|
+
# Update config for subject
|
83
|
+
def update_subject_config(subject, config)
|
84
|
+
put("/config/#{subject}", { body: config.to_json })
|
85
|
+
end
|
86
|
+
|
87
|
+
private
|
88
|
+
|
89
|
+
def get(path, **options)
|
90
|
+
request(path, method: :get, **options)
|
91
|
+
end
|
92
|
+
|
93
|
+
def put(path, **options)
|
94
|
+
request(path, method: :put, **options)
|
95
|
+
end
|
96
|
+
|
97
|
+
def post(path, **options)
|
98
|
+
request(path, method: :post, **options)
|
99
|
+
end
|
100
|
+
|
101
|
+
def request(path, **options)
|
102
|
+
options = { expects: 200 }.merge!(options)
|
103
|
+
response = @connection.request(path: path, **options)
|
104
|
+
JSON.parse(response.body)
|
105
|
+
end
|
106
|
+
end
|
data/lib/avro_turf/messaging.rb
CHANGED
@@ -1,6 +1,11 @@
|
|
1
1
|
require 'logger'
|
2
2
|
require 'avro_turf'
|
3
3
|
require 'avro_turf/schema_store'
|
4
|
+
require 'avro_turf/confluent_schema_registry'
|
5
|
+
require 'avro_turf/cached_confluent_schema_registry'
|
6
|
+
|
7
|
+
# For back-compatibility require the aliases along with the Messaging API.
|
8
|
+
# These names are deprecated and will be removed in a future release.
|
4
9
|
require 'avro_turf/schema_registry'
|
5
10
|
require 'avro_turf/cached_schema_registry'
|
6
11
|
|
@@ -20,7 +25,7 @@ class AvroTurf
|
|
20
25
|
# Instantiate a new Messaging instance with the given configuration.
|
21
26
|
#
|
22
27
|
# registry - A schema registry object that responds to all methods in the
|
23
|
-
# AvroTurf::
|
28
|
+
# AvroTurf::ConfluentSchemaRegistry interface.
|
24
29
|
# registry_url - The String URL of the schema registry that should be used.
|
25
30
|
# schema_store - A schema store object that responds to #find(schema_name, namespace).
|
26
31
|
# schemas_path - The String file system path where local schemas are stored.
|
@@ -30,7 +35,7 @@ class AvroTurf
|
|
30
35
|
@logger = logger || Logger.new($stderr)
|
31
36
|
@namespace = namespace
|
32
37
|
@schema_store = schema_store || SchemaStore.new(path: schemas_path || DEFAULT_SCHEMAS_PATH)
|
33
|
-
@registry = registry ||
|
38
|
+
@registry = registry || CachedConfluentSchemaRegistry.new(ConfluentSchemaRegistry.new(registry_url, logger: @logger))
|
34
39
|
@schemas_by_id = {}
|
35
40
|
end
|
36
41
|
|
@@ -1,79 +1,6 @@
|
|
1
|
-
require '
|
1
|
+
require 'avro_turf/confluent_schema_registry'
|
2
2
|
|
3
|
-
|
4
|
-
|
3
|
+
# AvroTurf::SchemaRegistry is deprecated and will be removed in a future release.
|
4
|
+
# Use AvroTurf::ConfluentSchemaRegistry instead.
|
5
5
|
|
6
|
-
|
7
|
-
@logger = logger
|
8
|
-
@connection = Excon.new(url, headers: {
|
9
|
-
"Content-Type" => CONTENT_TYPE,
|
10
|
-
})
|
11
|
-
end
|
12
|
-
|
13
|
-
def fetch(id)
|
14
|
-
@logger.info "Fetching schema with id #{id}"
|
15
|
-
data = get("/schemas/ids/#{id}")
|
16
|
-
data.fetch("schema")
|
17
|
-
end
|
18
|
-
|
19
|
-
def register(subject, schema)
|
20
|
-
data = post("/subjects/#{subject}/versions", body: {
|
21
|
-
schema: schema.to_s
|
22
|
-
}.to_json)
|
23
|
-
|
24
|
-
id = data.fetch("id")
|
25
|
-
|
26
|
-
@logger.info "Registered schema for subject `#{subject}`; id = #{id}"
|
27
|
-
|
28
|
-
id
|
29
|
-
end
|
30
|
-
|
31
|
-
# List all subjects
|
32
|
-
def subjects
|
33
|
-
get('/subjects')
|
34
|
-
end
|
35
|
-
|
36
|
-
# List all versions for a subject
|
37
|
-
def subject_versions(subject)
|
38
|
-
get("/subjects/#{subject}/versions")
|
39
|
-
end
|
40
|
-
|
41
|
-
# Get a specific version for a subject
|
42
|
-
def subject_version(subject, version = 'latest')
|
43
|
-
get("/subjects/#{subject}/versions/#{version}")
|
44
|
-
end
|
45
|
-
|
46
|
-
# Check if a schema exists. Returns nil if not found.
|
47
|
-
def check(subject, schema)
|
48
|
-
data = post("/subjects/#{subject}",
|
49
|
-
expects: [200, 404],
|
50
|
-
body: { schema: schema.to_s }.to_json)
|
51
|
-
data unless data.has_key?("error_code")
|
52
|
-
end
|
53
|
-
|
54
|
-
# Check if a schema is compatible with the stored version.
|
55
|
-
# Returns true if compatible, false otherwise
|
56
|
-
# http://docs.confluent.io/2.0.0/schema-registry/docs/api.html#compatibility
|
57
|
-
def compatible?(subject, schema, version = 'latest')
|
58
|
-
data = post("/compatibility/subjects/#{subject}/versions/#{version}",
|
59
|
-
expects: [200, 404],
|
60
|
-
body: { schema: schema.to_s }.to_json)
|
61
|
-
data.fetch('is_compatible', false) unless data.has_key?('error_code')
|
62
|
-
end
|
63
|
-
|
64
|
-
private
|
65
|
-
|
66
|
-
def get(path, **options)
|
67
|
-
request(path, method: :get, **options)
|
68
|
-
end
|
69
|
-
|
70
|
-
def post(path, **options)
|
71
|
-
request(path, method: :post, **options)
|
72
|
-
end
|
73
|
-
|
74
|
-
def request(path, **options)
|
75
|
-
options = { expects: 200 }.merge!(options)
|
76
|
-
response = @connection.request(path: path, **options)
|
77
|
-
JSON.parse(response.body)
|
78
|
-
end
|
79
|
-
end
|
6
|
+
AvroTurf::SchemaRegistry = AvroTurf::ConfluentSchemaRegistry
|
@@ -0,0 +1,129 @@
|
|
1
|
+
require 'sinatra/base'
|
2
|
+
|
3
|
+
class FakeConfluentSchemaRegistryServer < Sinatra::Base
|
4
|
+
SUBJECTS = Hash.new { Array.new }
|
5
|
+
SCHEMAS = []
|
6
|
+
CONFIGS = Hash.new
|
7
|
+
SUBJECT_NOT_FOUND = { error_code: 40401, message: 'Subject not found' }.to_json.freeze
|
8
|
+
VERSION_NOT_FOUND = { error_code: 40402, message: 'Version not found' }.to_json.freeze
|
9
|
+
SCHEMA_NOT_FOUND = { error_code: 40403, message: 'Schema not found' }.to_json.freeze
|
10
|
+
DEFAULT_GLOBAL_CONFIG = { 'compatibility' => 'BACKWARD'.freeze }.freeze
|
11
|
+
|
12
|
+
@global_config = DEFAULT_GLOBAL_CONFIG.dup
|
13
|
+
|
14
|
+
class << self
|
15
|
+
attr_reader :global_config
|
16
|
+
end
|
17
|
+
|
18
|
+
helpers do
|
19
|
+
def parse_schema
|
20
|
+
request.body.rewind
|
21
|
+
JSON.parse(request.body.read).fetch("schema").tap do |schema|
|
22
|
+
Avro::Schema.parse(schema)
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
def parse_config
|
27
|
+
request.body.rewind
|
28
|
+
JSON.parse(request.body.read)
|
29
|
+
end
|
30
|
+
|
31
|
+
def global_config
|
32
|
+
self.class.global_config
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
post "/subjects/:subject/versions" do
|
37
|
+
SCHEMAS << parse_schema
|
38
|
+
|
39
|
+
schema_id = SCHEMAS.size - 1
|
40
|
+
SUBJECTS[params[:subject]] = SUBJECTS[params[:subject]] << schema_id
|
41
|
+
{ id: schema_id }.to_json
|
42
|
+
end
|
43
|
+
|
44
|
+
get "/schemas/ids/:schema_id" do
|
45
|
+
schema = SCHEMAS.at(params[:schema_id].to_i)
|
46
|
+
halt(404, SCHEMA_NOT_FOUND) unless schema
|
47
|
+
{ schema: schema }.to_json
|
48
|
+
end
|
49
|
+
|
50
|
+
get "/subjects" do
|
51
|
+
SUBJECTS.keys.to_json
|
52
|
+
end
|
53
|
+
|
54
|
+
get "/subjects/:subject/versions" do
|
55
|
+
schema_ids = SUBJECTS[params[:subject]]
|
56
|
+
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
57
|
+
(1..schema_ids.size).to_a.to_json
|
58
|
+
end
|
59
|
+
|
60
|
+
get "/subjects/:subject/versions/:version" do
|
61
|
+
schema_ids = SUBJECTS[params[:subject]]
|
62
|
+
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
63
|
+
|
64
|
+
schema_id = if params[:version] == 'latest'
|
65
|
+
schema_ids.last
|
66
|
+
else
|
67
|
+
schema_ids.at(Integer(params[:version]) - 1)
|
68
|
+
end
|
69
|
+
halt(404, VERSION_NOT_FOUND) unless schema_id
|
70
|
+
|
71
|
+
schema = SCHEMAS.at(schema_id)
|
72
|
+
|
73
|
+
{
|
74
|
+
name: params[:subject],
|
75
|
+
version: schema_ids.index(schema_id) + 1,
|
76
|
+
schema: schema
|
77
|
+
}.to_json
|
78
|
+
end
|
79
|
+
|
80
|
+
post "/subjects/:subject" do
|
81
|
+
schema = parse_schema
|
82
|
+
|
83
|
+
# Note: this does not actually handle the same schema registered under
|
84
|
+
# multiple subjects
|
85
|
+
schema_id = SCHEMAS.index(schema)
|
86
|
+
|
87
|
+
halt(404, SCHEMA_NOT_FOUND) unless schema_id
|
88
|
+
|
89
|
+
{
|
90
|
+
subject: params[:subject],
|
91
|
+
id: schema_id,
|
92
|
+
version: SUBJECTS[params[:subject]].index(schema_id) + 1,
|
93
|
+
schema: schema
|
94
|
+
}.to_json
|
95
|
+
end
|
96
|
+
|
97
|
+
post "/compatibility/subjects/:subject/versions/:version" do
|
98
|
+
# The ruby avro gem does not yet include a compatibility check between schemas.
|
99
|
+
# See https://github.com/apache/avro/pull/170
|
100
|
+
raise NotImplementedError
|
101
|
+
end
|
102
|
+
|
103
|
+
get "/config" do
|
104
|
+
global_config.to_json
|
105
|
+
end
|
106
|
+
|
107
|
+
put "/config" do
|
108
|
+
global_config.merge!(parse_config).to_json
|
109
|
+
end
|
110
|
+
|
111
|
+
get "/config/:subject" do
|
112
|
+
CONFIGS.fetch(params[:subject], global_config).to_json
|
113
|
+
end
|
114
|
+
|
115
|
+
put "/config/:subject" do
|
116
|
+
config = parse_config
|
117
|
+
subject = params[:subject]
|
118
|
+
CONFIGS.fetch(subject) do
|
119
|
+
CONFIGS[subject] = {}
|
120
|
+
end.merge!(config).to_json
|
121
|
+
end
|
122
|
+
|
123
|
+
def self.clear
|
124
|
+
SUBJECTS.clear
|
125
|
+
SCHEMAS.clear
|
126
|
+
CONFIGS.clear
|
127
|
+
@global_config = DEFAULT_GLOBAL_CONFIG.dup
|
128
|
+
end
|
129
|
+
end
|
@@ -1,84 +1,6 @@
|
|
1
|
-
require '
|
1
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
2
2
|
|
3
|
-
|
4
|
-
|
5
|
-
SCHEMAS = []
|
6
|
-
SUBJECT_NOT_FOUND = { error_code: 40401, message: 'Subject not found' }.to_json.freeze
|
7
|
-
VERSION_NOT_FOUND = { error_code: 40402, message: 'Version not found' }.to_json.freeze
|
8
|
-
SCHEMA_NOT_FOUND = { error_code: 40403, message: 'Schema not found' }.to_json.freeze
|
3
|
+
# FakeSchemaRegistryServer is deprecated and will be removed in a future release.
|
4
|
+
# Use FakeConfluentSchemaRegistryServer instead.
|
9
5
|
|
10
|
-
|
11
|
-
def parse_schema
|
12
|
-
request.body.rewind
|
13
|
-
JSON.parse(request.body.read).fetch("schema").tap do |schema|
|
14
|
-
Avro::Schema.parse(schema)
|
15
|
-
end
|
16
|
-
end
|
17
|
-
end
|
18
|
-
|
19
|
-
post "/subjects/:subject/versions" do
|
20
|
-
SCHEMAS << parse_schema
|
21
|
-
|
22
|
-
schema_id = SCHEMAS.size - 1
|
23
|
-
SUBJECTS[params[:subject]] = SUBJECTS[params[:subject]] << schema_id
|
24
|
-
{ id: schema_id }.to_json
|
25
|
-
end
|
26
|
-
|
27
|
-
get "/schemas/ids/:schema_id" do
|
28
|
-
schema = SCHEMAS.at(params[:schema_id].to_i)
|
29
|
-
halt(404, SCHEMA_NOT_FOUND) unless schema
|
30
|
-
{ schema: schema }.to_json
|
31
|
-
end
|
32
|
-
|
33
|
-
get "/subjects" do
|
34
|
-
SUBJECTS.keys.to_json
|
35
|
-
end
|
36
|
-
|
37
|
-
get "/subjects/:subject/versions" do
|
38
|
-
schema_ids = SUBJECTS[params[:subject]]
|
39
|
-
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
40
|
-
(1..schema_ids.size).to_a.to_json
|
41
|
-
end
|
42
|
-
|
43
|
-
get "/subjects/:subject/versions/:version" do
|
44
|
-
schema_ids = SUBJECTS[params[:subject]]
|
45
|
-
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
46
|
-
|
47
|
-
schema_id = if params[:version] == 'latest'
|
48
|
-
schema_ids.last
|
49
|
-
else
|
50
|
-
schema_ids.at(Integer(params[:version]) - 1)
|
51
|
-
end
|
52
|
-
halt(404, VERSION_NOT_FOUND) unless schema_id
|
53
|
-
|
54
|
-
schema = SCHEMAS.at(schema_id)
|
55
|
-
|
56
|
-
{
|
57
|
-
name: params[:subject],
|
58
|
-
version: schema_ids.index(schema_id) + 1,
|
59
|
-
schema: schema
|
60
|
-
}.to_json
|
61
|
-
end
|
62
|
-
|
63
|
-
post "/subjects/:subject" do
|
64
|
-
schema = parse_schema
|
65
|
-
|
66
|
-
# Note: this does not actually handle the same schema registered under
|
67
|
-
# multiple subjects
|
68
|
-
schema_id = SCHEMAS.index(schema)
|
69
|
-
|
70
|
-
halt(404, SCHEMA_NOT_FOUND) unless schema_id
|
71
|
-
|
72
|
-
{
|
73
|
-
subject: params[:subject],
|
74
|
-
id: schema_id,
|
75
|
-
version: SUBJECTS[params[:subject]].index(schema_id) + 1,
|
76
|
-
schema: schema
|
77
|
-
}.to_json
|
78
|
-
end
|
79
|
-
|
80
|
-
def self.clear
|
81
|
-
SUBJECTS.clear
|
82
|
-
SCHEMAS.clear
|
83
|
-
end
|
84
|
-
end
|
6
|
+
FakeSchemaRegistryServer = FakeConfluentSchemaRegistryServer
|
data/lib/avro_turf/version.rb
CHANGED
@@ -1,9 +1,9 @@
|
|
1
1
|
require 'webmock/rspec'
|
2
|
-
require 'avro_turf/
|
3
|
-
require 'avro_turf/test/
|
2
|
+
require 'avro_turf/cached_confluent_schema_registry'
|
3
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
4
4
|
|
5
|
-
describe AvroTurf::
|
6
|
-
let(:upstream) { instance_double(AvroTurf::
|
5
|
+
describe AvroTurf::CachedConfluentSchemaRegistry do
|
6
|
+
let(:upstream) { instance_double(AvroTurf::ConfluentSchemaRegistry) }
|
7
7
|
let(:registry) { described_class.new(upstream) }
|
8
8
|
let(:id) { rand(999) }
|
9
9
|
let(:schema) do
|
@@ -34,8 +34,8 @@ describe AvroTurf::CachedSchemaRegistry do
|
|
34
34
|
end
|
35
35
|
end
|
36
36
|
|
37
|
-
it_behaves_like "a schema registry client" do
|
38
|
-
let(:upstream) { AvroTurf::
|
37
|
+
it_behaves_like "a confluent schema registry client" do
|
38
|
+
let(:upstream) { AvroTurf::ConfluentSchemaRegistry.new(registry_url, logger: logger) }
|
39
39
|
let(:registry) { described_class.new(upstream) }
|
40
40
|
end
|
41
41
|
end
|
@@ -0,0 +1,9 @@
|
|
1
|
+
require 'webmock/rspec'
|
2
|
+
require 'avro_turf/confluent_schema_registry'
|
3
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
4
|
+
|
5
|
+
describe AvroTurf::ConfluentSchemaRegistry do
|
6
|
+
it_behaves_like "a confluent schema registry client" do
|
7
|
+
let(:registry) { described_class.new(registry_url, logger: logger) }
|
8
|
+
end
|
9
|
+
end
|
data/spec/messaging_spec.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
require 'webmock/rspec'
|
2
2
|
require 'avro_turf/messaging'
|
3
|
-
require 'avro_turf/test/
|
3
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
4
4
|
|
5
5
|
describe AvroTurf::Messaging do
|
6
6
|
let(:registry_url) { "http://registry.example.com" }
|
@@ -21,8 +21,8 @@ describe AvroTurf::Messaging do
|
|
21
21
|
end
|
22
22
|
|
23
23
|
before do
|
24
|
-
stub_request(:any, /^#{registry_url}/).to_rack(
|
25
|
-
|
24
|
+
stub_request(:any, /^#{registry_url}/).to_rack(FakeConfluentSchemaRegistryServer)
|
25
|
+
FakeConfluentSchemaRegistryServer.clear
|
26
26
|
end
|
27
27
|
|
28
28
|
before do
|
@@ -63,7 +63,7 @@ describe AvroTurf::Messaging do
|
|
63
63
|
it_behaves_like "encoding and decoding"
|
64
64
|
|
65
65
|
context "with a provided registry" do
|
66
|
-
let(:registry) { AvroTurf::
|
66
|
+
let(:registry) { AvroTurf::ConfluentSchemaRegistry.new(registry_url, logger: logger) }
|
67
67
|
|
68
68
|
let(:avro) do
|
69
69
|
AvroTurf::Messaging.new(
|
@@ -1,6 +1,6 @@
|
|
1
1
|
# This shared example expects a registry variable to be defined
|
2
2
|
# with an instance of the registry class being tested.
|
3
|
-
shared_examples_for "a schema registry client" do
|
3
|
+
shared_examples_for "a confluent schema registry client" do
|
4
4
|
let(:logger) { Logger.new(StringIO.new) }
|
5
5
|
let(:registry_url) { "http://registry.example.com" }
|
6
6
|
let(:subject_name) { "some-subject" }
|
@@ -15,8 +15,8 @@ shared_examples_for "a schema registry client" do
|
|
15
15
|
end
|
16
16
|
|
17
17
|
before do
|
18
|
-
stub_request(:any, /^#{registry_url}/).to_rack(
|
19
|
-
|
18
|
+
stub_request(:any, /^#{registry_url}/).to_rack(FakeConfluentSchemaRegistryServer)
|
19
|
+
FakeConfluentSchemaRegistryServer.clear
|
20
20
|
end
|
21
21
|
|
22
22
|
describe "#register and #fetch" do
|
@@ -178,6 +178,67 @@ shared_examples_for "a schema registry client" do
|
|
178
178
|
end
|
179
179
|
end
|
180
180
|
|
181
|
+
describe "#global_config" do
|
182
|
+
let(:expected) do
|
183
|
+
{ compatibility: 'BACKWARD' }.to_json
|
184
|
+
end
|
185
|
+
|
186
|
+
it "returns the global configuration" do
|
187
|
+
expect(registry.global_config).to eq(JSON.parse(expected))
|
188
|
+
end
|
189
|
+
end
|
190
|
+
|
191
|
+
describe "#update_global_config" do
|
192
|
+
let(:config) do
|
193
|
+
{ compatibility: 'FORWARD' }
|
194
|
+
end
|
195
|
+
let(:expected) { config.to_json }
|
196
|
+
|
197
|
+
it "updates the global configuration and returns it" do
|
198
|
+
expect(registry.update_global_config(config)).to eq(JSON.parse(expected))
|
199
|
+
expect(registry.global_config).to eq(JSON.parse(expected))
|
200
|
+
end
|
201
|
+
end
|
202
|
+
|
203
|
+
describe "#subject_config" do
|
204
|
+
let(:expected) do
|
205
|
+
{ compatibility: 'BACKWARD' }.to_json
|
206
|
+
end
|
207
|
+
|
208
|
+
context "when the subject config is not set" do
|
209
|
+
it "returns the global configuration" do
|
210
|
+
expect(registry.subject_config(subject_name)).to eq(JSON.parse(expected))
|
211
|
+
end
|
212
|
+
end
|
213
|
+
|
214
|
+
context "when the subject config is set" do
|
215
|
+
let(:config) do
|
216
|
+
{ compatibility: 'FULL' }
|
217
|
+
end
|
218
|
+
let(:expected) { config.to_json }
|
219
|
+
|
220
|
+
before do
|
221
|
+
registry.update_subject_config(subject_name, config)
|
222
|
+
end
|
223
|
+
|
224
|
+
it "returns the subject config" do
|
225
|
+
expect(registry.subject_config(subject_name)).to eq(JSON.parse(expected))
|
226
|
+
end
|
227
|
+
end
|
228
|
+
end
|
229
|
+
|
230
|
+
describe "#update_subject_config" do
|
231
|
+
let(:config) do
|
232
|
+
{ compatibility: 'NONE' }
|
233
|
+
end
|
234
|
+
let(:expected) { config.to_json }
|
235
|
+
|
236
|
+
it "updates the subject config and returns it" do
|
237
|
+
expect(registry.update_subject_config(subject_name, config)).to eq(JSON.parse(expected))
|
238
|
+
expect(registry.subject_config(subject_name)).to eq(JSON.parse(expected))
|
239
|
+
end
|
240
|
+
end
|
241
|
+
|
181
242
|
# Monkey patch an Avro::Schema to simulate the presence of
|
182
243
|
# active_support/core_ext.
|
183
244
|
def break_to_json(avro_schema)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: avro_turf
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.8.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Daniel Schierbeck
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2017-03-06 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: avro
|
@@ -151,6 +151,7 @@ extra_rdoc_files: []
|
|
151
151
|
files:
|
152
152
|
- ".gitignore"
|
153
153
|
- ".rspec"
|
154
|
+
- CHANGELOG.md
|
154
155
|
- Gemfile
|
155
156
|
- LICENSE.txt
|
156
157
|
- README.md
|
@@ -158,7 +159,9 @@ files:
|
|
158
159
|
- avro_turf.gemspec
|
159
160
|
- circle.yml
|
160
161
|
- lib/avro_turf.rb
|
162
|
+
- lib/avro_turf/cached_confluent_schema_registry.rb
|
161
163
|
- lib/avro_turf/cached_schema_registry.rb
|
164
|
+
- lib/avro_turf/confluent_schema_registry.rb
|
162
165
|
- lib/avro_turf/core_ext.rb
|
163
166
|
- lib/avro_turf/core_ext/date.rb
|
164
167
|
- lib/avro_turf/core_ext/enumerable.rb
|
@@ -174,6 +177,7 @@ files:
|
|
174
177
|
- lib/avro_turf/schema_registry.rb
|
175
178
|
- lib/avro_turf/schema_store.rb
|
176
179
|
- lib/avro_turf/schema_to_avro_patch.rb
|
180
|
+
- lib/avro_turf/test/fake_confluent_schema_registry_server.rb
|
177
181
|
- lib/avro_turf/test/fake_schema_registry_server.rb
|
178
182
|
- lib/avro_turf/version.rb
|
179
183
|
- perf/address.avsc
|
@@ -181,7 +185,8 @@ files:
|
|
181
185
|
- perf/encoding_speed.rb
|
182
186
|
- perf/person.avsc
|
183
187
|
- spec/avro_turf_spec.rb
|
184
|
-
- spec/
|
188
|
+
- spec/cached_confluent_schema_registry_spec.rb
|
189
|
+
- spec/confluent_schema_registry_spec.rb
|
185
190
|
- spec/core_ext/date_spec.rb
|
186
191
|
- spec/core_ext/enumerable_spec.rb
|
187
192
|
- spec/core_ext/false_class_spec.rb
|
@@ -193,16 +198,23 @@ files:
|
|
193
198
|
- spec/core_ext/time_spec.rb
|
194
199
|
- spec/core_ext/true_class_spec.rb
|
195
200
|
- spec/messaging_spec.rb
|
196
|
-
- spec/schema_registry_spec.rb
|
197
201
|
- spec/schema_store_spec.rb
|
198
202
|
- spec/schema_to_avro_patch_spec.rb
|
199
203
|
- spec/spec_helper.rb
|
200
|
-
- spec/support/
|
204
|
+
- spec/support/confluent_schema_registry_context.rb
|
201
205
|
homepage: https://github.com/dasch/avro_turf
|
202
206
|
licenses:
|
203
207
|
- MIT
|
204
208
|
metadata: {}
|
205
|
-
post_install_message:
|
209
|
+
post_install_message: |2
|
210
|
+
|
211
|
+
avro_turf v0.8.0 deprecates the names AvroTurf::SchemaRegistry,
|
212
|
+
AvroTurf::CachedSchemaRegistry, and FakeSchemaRegistryServer.
|
213
|
+
|
214
|
+
Use AvroTurf::ConfluentSchemaRegistry, AvroTurf::CachedConfluentSchemaRegistry,
|
215
|
+
and FakeConfluentSchemaRegistryServer instead.
|
216
|
+
|
217
|
+
See https://github.com/dasch/avro_turf#deprecation-notice
|
206
218
|
rdoc_options: []
|
207
219
|
require_paths:
|
208
220
|
- lib
|
@@ -225,7 +237,8 @@ summary: A library that makes it easier to use the Avro serialization format fro
|
|
225
237
|
Ruby
|
226
238
|
test_files:
|
227
239
|
- spec/avro_turf_spec.rb
|
228
|
-
- spec/
|
240
|
+
- spec/cached_confluent_schema_registry_spec.rb
|
241
|
+
- spec/confluent_schema_registry_spec.rb
|
229
242
|
- spec/core_ext/date_spec.rb
|
230
243
|
- spec/core_ext/enumerable_spec.rb
|
231
244
|
- spec/core_ext/false_class_spec.rb
|
@@ -237,9 +250,7 @@ test_files:
|
|
237
250
|
- spec/core_ext/time_spec.rb
|
238
251
|
- spec/core_ext/true_class_spec.rb
|
239
252
|
- spec/messaging_spec.rb
|
240
|
-
- spec/schema_registry_spec.rb
|
241
253
|
- spec/schema_store_spec.rb
|
242
254
|
- spec/schema_to_avro_patch_spec.rb
|
243
255
|
- spec/spec_helper.rb
|
244
|
-
- spec/support/
|
245
|
-
has_rdoc:
|
256
|
+
- spec/support/confluent_schema_registry_context.rb
|
@@ -1,9 +0,0 @@
|
|
1
|
-
require 'webmock/rspec'
|
2
|
-
require 'avro_turf/schema_registry'
|
3
|
-
require 'avro_turf/test/fake_schema_registry_server'
|
4
|
-
|
5
|
-
describe AvroTurf::SchemaRegistry do
|
6
|
-
it_behaves_like "a schema registry client" do
|
7
|
-
let(:registry) { described_class.new(registry_url, logger: logger) }
|
8
|
-
end
|
9
|
-
end
|