avro_turf 0.7.2 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +33 -15
- data/avro_turf.gemspec +10 -0
- data/lib/avro_turf/cached_confluent_schema_registry.rb +27 -0
- data/lib/avro_turf/cached_schema_registry.rb +4 -24
- data/lib/avro_turf/confluent_schema_registry.rb +106 -0
- data/lib/avro_turf/messaging.rb +7 -2
- data/lib/avro_turf/schema_registry.rb +4 -77
- data/lib/avro_turf/test/fake_confluent_schema_registry_server.rb +129 -0
- data/lib/avro_turf/test/fake_schema_registry_server.rb +4 -82
- data/lib/avro_turf/version.rb +1 -1
- data/spec/{cached_schema_registry_spec.rb → cached_confluent_schema_registry_spec.rb} +6 -6
- data/spec/confluent_schema_registry_spec.rb +9 -0
- data/spec/messaging_spec.rb +4 -4
- data/spec/support/{schema_registry_context.rb → confluent_schema_registry_context.rb} +64 -3
- metadata +21 -10
- data/spec/schema_registry_spec.rb +0 -9
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a708b9aabeca7d45e1db532e180b2d80e4a5aecb
|
4
|
+
data.tar.gz: acb21f2435fd5126efed47803395d84eb4f5a220
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4bcc5e9832804eafb4f295a6fc85273a5a52c17a515f87ab3da0deb31034ed428c6aaee3754abcaa45d78d37daf6062c6c1d1007644f8578e2561b17f70f1614
|
7
|
+
data.tar.gz: 89071ed406d0be937344cc70814499e29206f05e52484fd4ba4f797f12bcb9236c36c4b080f22ffedd815ac3f6081e13405cb10fa1dc4184cfe73c8bbb279952
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,9 @@
|
|
1
|
+
# avro_turf
|
2
|
+
|
3
|
+
## v0.8.0
|
4
|
+
- The names `AvroTurf::SchemaRegistry`, `AvroTurf::CachedSchemaRegistry`, and
|
5
|
+
`FakeSchemaRegistryServer` are deprecated and will be removed in a future release.
|
6
|
+
Use `AvroTurf::ConfluentSchemaRegistry`, `AvroTurf::CachedConfluentSchemaRegistry`,
|
7
|
+
and `FakeConfluentSchemaRegistryServer` instead.
|
8
|
+
- Add support for the Config API (http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#config)
|
9
|
+
to `AvroTurf::ConfluentSchemaRegistry`.
|
data/README.md
CHANGED
@@ -5,6 +5,17 @@ AvroTurf is a library that makes it easier to encode and decode data using the [
|
|
5
5
|
* Provides an idiomatic Ruby interface.
|
6
6
|
* Allows referencing schemas defined in another file.
|
7
7
|
|
8
|
+
## Deprecation Notice
|
9
|
+
|
10
|
+
The `AvroTurf::SchemaRegistry`, `AvroTurf::CachedSchemaRegistry`,
|
11
|
+
and `FakeSchemaRegistryServer` names have been deprecated because the Avro spec recently
|
12
|
+
introduced an incompatible [single-message encoding format](https://github.com/apache/avro/commit/30408a9c192c5f4eaaf42f01f0ffbfffd705aa57).
|
13
|
+
|
14
|
+
These classes have been renamed to `AvroTurf::ConfluentSchemaRegistry`,
|
15
|
+
`AvroTurf::CachedConfluentSchemaRegistry`, and `FakeConfluentSchemaRegistry`.
|
16
|
+
|
17
|
+
The aliases for the original names will be removed in a future release.
|
18
|
+
|
8
19
|
## Installation
|
9
20
|
|
10
21
|
Add this line to your application's Gemfile:
|
@@ -80,11 +91,8 @@ In the example above, the `person` schema references the `address` schema, even
|
|
80
91
|
```json
|
81
92
|
// person_list.avsc
|
82
93
|
{
|
83
|
-
"
|
84
|
-
"
|
85
|
-
"type": "array",
|
86
|
-
"items": "person"
|
87
|
-
}
|
94
|
+
"type": "array",
|
95
|
+
"items": "person"
|
88
96
|
}
|
89
97
|
```
|
90
98
|
|
@@ -121,11 +129,14 @@ data = avro.encode({ "title" => "hello, world" }, schema_name: "greeting")
|
|
121
129
|
avro.decode(data) #=> { "title" => "hello, world" }
|
122
130
|
```
|
123
131
|
|
124
|
-
|
125
|
-
|
132
|
+
### Confluent Schema Registry Client
|
133
|
+
|
134
|
+
The ConfluentSchemaRegistry client used by the Messaging API can also be used directly.
|
135
|
+
It can check whether a schema is compatible with a subject in the registry using the [Compatibility API](http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#compatibility):
|
126
136
|
|
127
137
|
```ruby
|
128
|
-
require 'avro_turf
|
138
|
+
require 'avro_turf'
|
139
|
+
require 'avro_turf/confluent_schema_registry'
|
129
140
|
|
130
141
|
schema = <<-JSON
|
131
142
|
{
|
@@ -144,15 +155,22 @@ schema = <<-JSON
|
|
144
155
|
}
|
145
156
|
JSON
|
146
157
|
|
147
|
-
|
158
|
+
registry = AvroTurf::ConfluentSchemaRegistry.new("http://my-registry:8081/")
|
159
|
+
|
160
|
+
# Returns true if the schema is compatible, nil if the subject or version is not registered, and false if incompatible.
|
161
|
+
registry.compatible?("person", schema)
|
162
|
+
```
|
148
163
|
|
149
|
-
|
150
|
-
|
164
|
+
The ConfluentSchemaRegistry client can also change the global compatibility level or the compatibility level for an individual subject using the [Config API](http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#config):
|
165
|
+
|
166
|
+
```ruby
|
167
|
+
registry.update_global_config(compatibility: 'FULL')
|
168
|
+
registry.update_subject_config("person", compatibility: 'NONE')
|
151
169
|
```
|
152
170
|
|
153
171
|
### Testing Support
|
154
172
|
|
155
|
-
AvroTurf includes a `
|
173
|
+
AvroTurf includes a `FakeConfluentSchemaRegistryServer` that can be used in tests. The
|
156
174
|
fake schema registry server depends on Sinatra but it is _not_ listed as a runtime
|
157
175
|
dependency for AvroTurf. Sinatra must be added to your Gemfile or gemspec in order
|
158
176
|
to use the fake server.
|
@@ -160,14 +178,14 @@ to use the fake server.
|
|
160
178
|
Example using RSpec:
|
161
179
|
|
162
180
|
```ruby
|
163
|
-
require 'avro_turf/test/
|
181
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
164
182
|
require 'webmock/rspec'
|
165
183
|
|
166
184
|
# within an example
|
167
185
|
let(:registry_url) { "http://registry.example.com" }
|
168
186
|
before do
|
169
|
-
stub_request(:any, /^#{registry_url}/).to_rack(
|
170
|
-
|
187
|
+
stub_request(:any, /^#{registry_url}/).to_rack(FakeConfluentSchemaRegistryServer)
|
188
|
+
FakeConfluentSchemaRegistryServer.clear
|
171
189
|
end
|
172
190
|
|
173
191
|
# Messaging objects created with the same registry_url will now use the fake server.
|
data/avro_turf.gemspec
CHANGED
@@ -27,4 +27,14 @@ Gem::Specification.new do |spec|
|
|
27
27
|
spec.add_development_dependency "webmock"
|
28
28
|
spec.add_development_dependency "sinatra"
|
29
29
|
spec.add_development_dependency "json_spec"
|
30
|
+
|
31
|
+
spec.post_install_message = %{
|
32
|
+
avro_turf v0.8.0 deprecates the names AvroTurf::SchemaRegistry,
|
33
|
+
AvroTurf::CachedSchemaRegistry, and FakeSchemaRegistryServer.
|
34
|
+
|
35
|
+
Use AvroTurf::ConfluentSchemaRegistry, AvroTurf::CachedConfluentSchemaRegistry,
|
36
|
+
and FakeConfluentSchemaRegistryServer instead.
|
37
|
+
|
38
|
+
See https://github.com/dasch/avro_turf#deprecation-notice
|
39
|
+
}
|
30
40
|
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
require 'avro_turf/confluent_schema_registry'
|
2
|
+
|
3
|
+
# Caches registrations and lookups to the schema registry in memory.
|
4
|
+
class AvroTurf::CachedConfluentSchemaRegistry
|
5
|
+
|
6
|
+
def initialize(upstream)
|
7
|
+
@upstream = upstream
|
8
|
+
@schemas_by_id = {}
|
9
|
+
@ids_by_schema = {}
|
10
|
+
end
|
11
|
+
|
12
|
+
# Delegate the following methods to the upstream
|
13
|
+
%i(subjects subject_versions subject_version check compatible?
|
14
|
+
global_config update_global_config subject_config update_subject_config).each do |name|
|
15
|
+
define_method(name) do |*args|
|
16
|
+
instance_variable_get(:@upstream).send(name, *args)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def fetch(id)
|
21
|
+
@schemas_by_id[id] ||= @upstream.fetch(id)
|
22
|
+
end
|
23
|
+
|
24
|
+
def register(subject, schema)
|
25
|
+
@ids_by_schema[subject + schema.to_s] ||= @upstream.register(subject, schema)
|
26
|
+
end
|
27
|
+
end
|
@@ -1,26 +1,6 @@
|
|
1
|
-
require 'avro_turf/
|
1
|
+
require 'avro_turf/cached_confluent_schema_registry'
|
2
2
|
|
3
|
-
#
|
4
|
-
|
3
|
+
# AvroTurf::CachedSchemaRegistry is deprecated and will be removed in a future release.
|
4
|
+
# Use AvroTurf::CachedConfluentSchemaRegistry instead.
|
5
5
|
|
6
|
-
|
7
|
-
@upstream = upstream
|
8
|
-
@schemas_by_id = {}
|
9
|
-
@ids_by_schema = {}
|
10
|
-
end
|
11
|
-
|
12
|
-
# Delegate the following methods to the upstream
|
13
|
-
%i(subjects subject_versions subject_version check compatible?).each do |name|
|
14
|
-
define_method(name) do |*args|
|
15
|
-
instance_variable_get(:@upstream).send(name, *args)
|
16
|
-
end
|
17
|
-
end
|
18
|
-
|
19
|
-
def fetch(id)
|
20
|
-
@schemas_by_id[id] ||= @upstream.fetch(id)
|
21
|
-
end
|
22
|
-
|
23
|
-
def register(subject, schema)
|
24
|
-
@ids_by_schema[subject + schema.to_s] ||= @upstream.register(subject, schema)
|
25
|
-
end
|
26
|
-
end
|
6
|
+
AvroTurf::CachedSchemaRegistry = AvroTurf::CachedConfluentSchemaRegistry
|
@@ -0,0 +1,106 @@
|
|
1
|
+
require 'excon'
|
2
|
+
|
3
|
+
class AvroTurf::ConfluentSchemaRegistry
|
4
|
+
CONTENT_TYPE = "application/vnd.schemaregistry.v1+json".freeze
|
5
|
+
|
6
|
+
def initialize(url, logger: Logger.new($stdout))
|
7
|
+
@logger = logger
|
8
|
+
@connection = Excon.new(url, headers: {
|
9
|
+
"Content-Type" => CONTENT_TYPE,
|
10
|
+
})
|
11
|
+
end
|
12
|
+
|
13
|
+
def fetch(id)
|
14
|
+
@logger.info "Fetching schema with id #{id}"
|
15
|
+
data = get("/schemas/ids/#{id}")
|
16
|
+
data.fetch("schema")
|
17
|
+
end
|
18
|
+
|
19
|
+
def register(subject, schema)
|
20
|
+
data = post("/subjects/#{subject}/versions", body: {
|
21
|
+
schema: schema.to_s
|
22
|
+
}.to_json)
|
23
|
+
|
24
|
+
id = data.fetch("id")
|
25
|
+
|
26
|
+
@logger.info "Registered schema for subject `#{subject}`; id = #{id}"
|
27
|
+
|
28
|
+
id
|
29
|
+
end
|
30
|
+
|
31
|
+
# List all subjects
|
32
|
+
def subjects
|
33
|
+
get('/subjects')
|
34
|
+
end
|
35
|
+
|
36
|
+
# List all versions for a subject
|
37
|
+
def subject_versions(subject)
|
38
|
+
get("/subjects/#{subject}/versions")
|
39
|
+
end
|
40
|
+
|
41
|
+
# Get a specific version for a subject
|
42
|
+
def subject_version(subject, version = 'latest')
|
43
|
+
get("/subjects/#{subject}/versions/#{version}")
|
44
|
+
end
|
45
|
+
|
46
|
+
# Check if a schema exists. Returns nil if not found.
|
47
|
+
def check(subject, schema)
|
48
|
+
data = post("/subjects/#{subject}",
|
49
|
+
expects: [200, 404],
|
50
|
+
body: { schema: schema.to_s }.to_json)
|
51
|
+
data unless data.has_key?("error_code")
|
52
|
+
end
|
53
|
+
|
54
|
+
# Check if a schema is compatible with the stored version.
|
55
|
+
# Returns:
|
56
|
+
# - true if compatible
|
57
|
+
# - nil if the subject or version does not exist
|
58
|
+
# - false if incompatible
|
59
|
+
# http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#compatibility
|
60
|
+
def compatible?(subject, schema, version = 'latest')
|
61
|
+
data = post("/compatibility/subjects/#{subject}/versions/#{version}",
|
62
|
+
expects: [200, 404],
|
63
|
+
body: { schema: schema.to_s }.to_json)
|
64
|
+
data.fetch('is_compatible', false) unless data.has_key?('error_code')
|
65
|
+
end
|
66
|
+
|
67
|
+
# Get global config
|
68
|
+
def global_config
|
69
|
+
get("/config")
|
70
|
+
end
|
71
|
+
|
72
|
+
# Update global config
|
73
|
+
def update_global_config(config)
|
74
|
+
put("/config", { body: config.to_json })
|
75
|
+
end
|
76
|
+
|
77
|
+
# Get config for subject
|
78
|
+
def subject_config(subject)
|
79
|
+
get("/config/#{subject}")
|
80
|
+
end
|
81
|
+
|
82
|
+
# Update config for subject
|
83
|
+
def update_subject_config(subject, config)
|
84
|
+
put("/config/#{subject}", { body: config.to_json })
|
85
|
+
end
|
86
|
+
|
87
|
+
private
|
88
|
+
|
89
|
+
def get(path, **options)
|
90
|
+
request(path, method: :get, **options)
|
91
|
+
end
|
92
|
+
|
93
|
+
def put(path, **options)
|
94
|
+
request(path, method: :put, **options)
|
95
|
+
end
|
96
|
+
|
97
|
+
def post(path, **options)
|
98
|
+
request(path, method: :post, **options)
|
99
|
+
end
|
100
|
+
|
101
|
+
def request(path, **options)
|
102
|
+
options = { expects: 200 }.merge!(options)
|
103
|
+
response = @connection.request(path: path, **options)
|
104
|
+
JSON.parse(response.body)
|
105
|
+
end
|
106
|
+
end
|
data/lib/avro_turf/messaging.rb
CHANGED
@@ -1,6 +1,11 @@
|
|
1
1
|
require 'logger'
|
2
2
|
require 'avro_turf'
|
3
3
|
require 'avro_turf/schema_store'
|
4
|
+
require 'avro_turf/confluent_schema_registry'
|
5
|
+
require 'avro_turf/cached_confluent_schema_registry'
|
6
|
+
|
7
|
+
# For back-compatibility require the aliases along with the Messaging API.
|
8
|
+
# These names are deprecated and will be removed in a future release.
|
4
9
|
require 'avro_turf/schema_registry'
|
5
10
|
require 'avro_turf/cached_schema_registry'
|
6
11
|
|
@@ -20,7 +25,7 @@ class AvroTurf
|
|
20
25
|
# Instantiate a new Messaging instance with the given configuration.
|
21
26
|
#
|
22
27
|
# registry - A schema registry object that responds to all methods in the
|
23
|
-
# AvroTurf::
|
28
|
+
# AvroTurf::ConfluentSchemaRegistry interface.
|
24
29
|
# registry_url - The String URL of the schema registry that should be used.
|
25
30
|
# schema_store - A schema store object that responds to #find(schema_name, namespace).
|
26
31
|
# schemas_path - The String file system path where local schemas are stored.
|
@@ -30,7 +35,7 @@ class AvroTurf
|
|
30
35
|
@logger = logger || Logger.new($stderr)
|
31
36
|
@namespace = namespace
|
32
37
|
@schema_store = schema_store || SchemaStore.new(path: schemas_path || DEFAULT_SCHEMAS_PATH)
|
33
|
-
@registry = registry ||
|
38
|
+
@registry = registry || CachedConfluentSchemaRegistry.new(ConfluentSchemaRegistry.new(registry_url, logger: @logger))
|
34
39
|
@schemas_by_id = {}
|
35
40
|
end
|
36
41
|
|
@@ -1,79 +1,6 @@
|
|
1
|
-
require '
|
1
|
+
require 'avro_turf/confluent_schema_registry'
|
2
2
|
|
3
|
-
|
4
|
-
|
3
|
+
# AvroTurf::SchemaRegistry is deprecated and will be removed in a future release.
|
4
|
+
# Use AvroTurf::ConfluentSchemaRegistry instead.
|
5
5
|
|
6
|
-
|
7
|
-
@logger = logger
|
8
|
-
@connection = Excon.new(url, headers: {
|
9
|
-
"Content-Type" => CONTENT_TYPE,
|
10
|
-
})
|
11
|
-
end
|
12
|
-
|
13
|
-
def fetch(id)
|
14
|
-
@logger.info "Fetching schema with id #{id}"
|
15
|
-
data = get("/schemas/ids/#{id}")
|
16
|
-
data.fetch("schema")
|
17
|
-
end
|
18
|
-
|
19
|
-
def register(subject, schema)
|
20
|
-
data = post("/subjects/#{subject}/versions", body: {
|
21
|
-
schema: schema.to_s
|
22
|
-
}.to_json)
|
23
|
-
|
24
|
-
id = data.fetch("id")
|
25
|
-
|
26
|
-
@logger.info "Registered schema for subject `#{subject}`; id = #{id}"
|
27
|
-
|
28
|
-
id
|
29
|
-
end
|
30
|
-
|
31
|
-
# List all subjects
|
32
|
-
def subjects
|
33
|
-
get('/subjects')
|
34
|
-
end
|
35
|
-
|
36
|
-
# List all versions for a subject
|
37
|
-
def subject_versions(subject)
|
38
|
-
get("/subjects/#{subject}/versions")
|
39
|
-
end
|
40
|
-
|
41
|
-
# Get a specific version for a subject
|
42
|
-
def subject_version(subject, version = 'latest')
|
43
|
-
get("/subjects/#{subject}/versions/#{version}")
|
44
|
-
end
|
45
|
-
|
46
|
-
# Check if a schema exists. Returns nil if not found.
|
47
|
-
def check(subject, schema)
|
48
|
-
data = post("/subjects/#{subject}",
|
49
|
-
expects: [200, 404],
|
50
|
-
body: { schema: schema.to_s }.to_json)
|
51
|
-
data unless data.has_key?("error_code")
|
52
|
-
end
|
53
|
-
|
54
|
-
# Check if a schema is compatible with the stored version.
|
55
|
-
# Returns true if compatible, false otherwise
|
56
|
-
# http://docs.confluent.io/2.0.0/schema-registry/docs/api.html#compatibility
|
57
|
-
def compatible?(subject, schema, version = 'latest')
|
58
|
-
data = post("/compatibility/subjects/#{subject}/versions/#{version}",
|
59
|
-
expects: [200, 404],
|
60
|
-
body: { schema: schema.to_s }.to_json)
|
61
|
-
data.fetch('is_compatible', false) unless data.has_key?('error_code')
|
62
|
-
end
|
63
|
-
|
64
|
-
private
|
65
|
-
|
66
|
-
def get(path, **options)
|
67
|
-
request(path, method: :get, **options)
|
68
|
-
end
|
69
|
-
|
70
|
-
def post(path, **options)
|
71
|
-
request(path, method: :post, **options)
|
72
|
-
end
|
73
|
-
|
74
|
-
def request(path, **options)
|
75
|
-
options = { expects: 200 }.merge!(options)
|
76
|
-
response = @connection.request(path: path, **options)
|
77
|
-
JSON.parse(response.body)
|
78
|
-
end
|
79
|
-
end
|
6
|
+
AvroTurf::SchemaRegistry = AvroTurf::ConfluentSchemaRegistry
|
@@ -0,0 +1,129 @@
|
|
1
|
+
require 'sinatra/base'
|
2
|
+
|
3
|
+
class FakeConfluentSchemaRegistryServer < Sinatra::Base
|
4
|
+
SUBJECTS = Hash.new { Array.new }
|
5
|
+
SCHEMAS = []
|
6
|
+
CONFIGS = Hash.new
|
7
|
+
SUBJECT_NOT_FOUND = { error_code: 40401, message: 'Subject not found' }.to_json.freeze
|
8
|
+
VERSION_NOT_FOUND = { error_code: 40402, message: 'Version not found' }.to_json.freeze
|
9
|
+
SCHEMA_NOT_FOUND = { error_code: 40403, message: 'Schema not found' }.to_json.freeze
|
10
|
+
DEFAULT_GLOBAL_CONFIG = { 'compatibility' => 'BACKWARD'.freeze }.freeze
|
11
|
+
|
12
|
+
@global_config = DEFAULT_GLOBAL_CONFIG.dup
|
13
|
+
|
14
|
+
class << self
|
15
|
+
attr_reader :global_config
|
16
|
+
end
|
17
|
+
|
18
|
+
helpers do
|
19
|
+
def parse_schema
|
20
|
+
request.body.rewind
|
21
|
+
JSON.parse(request.body.read).fetch("schema").tap do |schema|
|
22
|
+
Avro::Schema.parse(schema)
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
def parse_config
|
27
|
+
request.body.rewind
|
28
|
+
JSON.parse(request.body.read)
|
29
|
+
end
|
30
|
+
|
31
|
+
def global_config
|
32
|
+
self.class.global_config
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
post "/subjects/:subject/versions" do
|
37
|
+
SCHEMAS << parse_schema
|
38
|
+
|
39
|
+
schema_id = SCHEMAS.size - 1
|
40
|
+
SUBJECTS[params[:subject]] = SUBJECTS[params[:subject]] << schema_id
|
41
|
+
{ id: schema_id }.to_json
|
42
|
+
end
|
43
|
+
|
44
|
+
get "/schemas/ids/:schema_id" do
|
45
|
+
schema = SCHEMAS.at(params[:schema_id].to_i)
|
46
|
+
halt(404, SCHEMA_NOT_FOUND) unless schema
|
47
|
+
{ schema: schema }.to_json
|
48
|
+
end
|
49
|
+
|
50
|
+
get "/subjects" do
|
51
|
+
SUBJECTS.keys.to_json
|
52
|
+
end
|
53
|
+
|
54
|
+
get "/subjects/:subject/versions" do
|
55
|
+
schema_ids = SUBJECTS[params[:subject]]
|
56
|
+
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
57
|
+
(1..schema_ids.size).to_a.to_json
|
58
|
+
end
|
59
|
+
|
60
|
+
get "/subjects/:subject/versions/:version" do
|
61
|
+
schema_ids = SUBJECTS[params[:subject]]
|
62
|
+
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
63
|
+
|
64
|
+
schema_id = if params[:version] == 'latest'
|
65
|
+
schema_ids.last
|
66
|
+
else
|
67
|
+
schema_ids.at(Integer(params[:version]) - 1)
|
68
|
+
end
|
69
|
+
halt(404, VERSION_NOT_FOUND) unless schema_id
|
70
|
+
|
71
|
+
schema = SCHEMAS.at(schema_id)
|
72
|
+
|
73
|
+
{
|
74
|
+
name: params[:subject],
|
75
|
+
version: schema_ids.index(schema_id) + 1,
|
76
|
+
schema: schema
|
77
|
+
}.to_json
|
78
|
+
end
|
79
|
+
|
80
|
+
post "/subjects/:subject" do
|
81
|
+
schema = parse_schema
|
82
|
+
|
83
|
+
# Note: this does not actually handle the same schema registered under
|
84
|
+
# multiple subjects
|
85
|
+
schema_id = SCHEMAS.index(schema)
|
86
|
+
|
87
|
+
halt(404, SCHEMA_NOT_FOUND) unless schema_id
|
88
|
+
|
89
|
+
{
|
90
|
+
subject: params[:subject],
|
91
|
+
id: schema_id,
|
92
|
+
version: SUBJECTS[params[:subject]].index(schema_id) + 1,
|
93
|
+
schema: schema
|
94
|
+
}.to_json
|
95
|
+
end
|
96
|
+
|
97
|
+
post "/compatibility/subjects/:subject/versions/:version" do
|
98
|
+
# The ruby avro gem does not yet include a compatibility check between schemas.
|
99
|
+
# See https://github.com/apache/avro/pull/170
|
100
|
+
raise NotImplementedError
|
101
|
+
end
|
102
|
+
|
103
|
+
get "/config" do
|
104
|
+
global_config.to_json
|
105
|
+
end
|
106
|
+
|
107
|
+
put "/config" do
|
108
|
+
global_config.merge!(parse_config).to_json
|
109
|
+
end
|
110
|
+
|
111
|
+
get "/config/:subject" do
|
112
|
+
CONFIGS.fetch(params[:subject], global_config).to_json
|
113
|
+
end
|
114
|
+
|
115
|
+
put "/config/:subject" do
|
116
|
+
config = parse_config
|
117
|
+
subject = params[:subject]
|
118
|
+
CONFIGS.fetch(subject) do
|
119
|
+
CONFIGS[subject] = {}
|
120
|
+
end.merge!(config).to_json
|
121
|
+
end
|
122
|
+
|
123
|
+
def self.clear
|
124
|
+
SUBJECTS.clear
|
125
|
+
SCHEMAS.clear
|
126
|
+
CONFIGS.clear
|
127
|
+
@global_config = DEFAULT_GLOBAL_CONFIG.dup
|
128
|
+
end
|
129
|
+
end
|
@@ -1,84 +1,6 @@
|
|
1
|
-
require '
|
1
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
2
2
|
|
3
|
-
|
4
|
-
|
5
|
-
SCHEMAS = []
|
6
|
-
SUBJECT_NOT_FOUND = { error_code: 40401, message: 'Subject not found' }.to_json.freeze
|
7
|
-
VERSION_NOT_FOUND = { error_code: 40402, message: 'Version not found' }.to_json.freeze
|
8
|
-
SCHEMA_NOT_FOUND = { error_code: 40403, message: 'Schema not found' }.to_json.freeze
|
3
|
+
# FakeSchemaRegistryServer is deprecated and will be removed in a future release.
|
4
|
+
# Use FakeConfluentSchemaRegistryServer instead.
|
9
5
|
|
10
|
-
|
11
|
-
def parse_schema
|
12
|
-
request.body.rewind
|
13
|
-
JSON.parse(request.body.read).fetch("schema").tap do |schema|
|
14
|
-
Avro::Schema.parse(schema)
|
15
|
-
end
|
16
|
-
end
|
17
|
-
end
|
18
|
-
|
19
|
-
post "/subjects/:subject/versions" do
|
20
|
-
SCHEMAS << parse_schema
|
21
|
-
|
22
|
-
schema_id = SCHEMAS.size - 1
|
23
|
-
SUBJECTS[params[:subject]] = SUBJECTS[params[:subject]] << schema_id
|
24
|
-
{ id: schema_id }.to_json
|
25
|
-
end
|
26
|
-
|
27
|
-
get "/schemas/ids/:schema_id" do
|
28
|
-
schema = SCHEMAS.at(params[:schema_id].to_i)
|
29
|
-
halt(404, SCHEMA_NOT_FOUND) unless schema
|
30
|
-
{ schema: schema }.to_json
|
31
|
-
end
|
32
|
-
|
33
|
-
get "/subjects" do
|
34
|
-
SUBJECTS.keys.to_json
|
35
|
-
end
|
36
|
-
|
37
|
-
get "/subjects/:subject/versions" do
|
38
|
-
schema_ids = SUBJECTS[params[:subject]]
|
39
|
-
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
40
|
-
(1..schema_ids.size).to_a.to_json
|
41
|
-
end
|
42
|
-
|
43
|
-
get "/subjects/:subject/versions/:version" do
|
44
|
-
schema_ids = SUBJECTS[params[:subject]]
|
45
|
-
halt(404, SUBJECT_NOT_FOUND) if schema_ids.empty?
|
46
|
-
|
47
|
-
schema_id = if params[:version] == 'latest'
|
48
|
-
schema_ids.last
|
49
|
-
else
|
50
|
-
schema_ids.at(Integer(params[:version]) - 1)
|
51
|
-
end
|
52
|
-
halt(404, VERSION_NOT_FOUND) unless schema_id
|
53
|
-
|
54
|
-
schema = SCHEMAS.at(schema_id)
|
55
|
-
|
56
|
-
{
|
57
|
-
name: params[:subject],
|
58
|
-
version: schema_ids.index(schema_id) + 1,
|
59
|
-
schema: schema
|
60
|
-
}.to_json
|
61
|
-
end
|
62
|
-
|
63
|
-
post "/subjects/:subject" do
|
64
|
-
schema = parse_schema
|
65
|
-
|
66
|
-
# Note: this does not actually handle the same schema registered under
|
67
|
-
# multiple subjects
|
68
|
-
schema_id = SCHEMAS.index(schema)
|
69
|
-
|
70
|
-
halt(404, SCHEMA_NOT_FOUND) unless schema_id
|
71
|
-
|
72
|
-
{
|
73
|
-
subject: params[:subject],
|
74
|
-
id: schema_id,
|
75
|
-
version: SUBJECTS[params[:subject]].index(schema_id) + 1,
|
76
|
-
schema: schema
|
77
|
-
}.to_json
|
78
|
-
end
|
79
|
-
|
80
|
-
def self.clear
|
81
|
-
SUBJECTS.clear
|
82
|
-
SCHEMAS.clear
|
83
|
-
end
|
84
|
-
end
|
6
|
+
FakeSchemaRegistryServer = FakeConfluentSchemaRegistryServer
|
data/lib/avro_turf/version.rb
CHANGED
@@ -1,9 +1,9 @@
|
|
1
1
|
require 'webmock/rspec'
|
2
|
-
require 'avro_turf/
|
3
|
-
require 'avro_turf/test/
|
2
|
+
require 'avro_turf/cached_confluent_schema_registry'
|
3
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
4
4
|
|
5
|
-
describe AvroTurf::
|
6
|
-
let(:upstream) { instance_double(AvroTurf::
|
5
|
+
describe AvroTurf::CachedConfluentSchemaRegistry do
|
6
|
+
let(:upstream) { instance_double(AvroTurf::ConfluentSchemaRegistry) }
|
7
7
|
let(:registry) { described_class.new(upstream) }
|
8
8
|
let(:id) { rand(999) }
|
9
9
|
let(:schema) do
|
@@ -34,8 +34,8 @@ describe AvroTurf::CachedSchemaRegistry do
|
|
34
34
|
end
|
35
35
|
end
|
36
36
|
|
37
|
-
it_behaves_like "a schema registry client" do
|
38
|
-
let(:upstream) { AvroTurf::
|
37
|
+
it_behaves_like "a confluent schema registry client" do
|
38
|
+
let(:upstream) { AvroTurf::ConfluentSchemaRegistry.new(registry_url, logger: logger) }
|
39
39
|
let(:registry) { described_class.new(upstream) }
|
40
40
|
end
|
41
41
|
end
|
@@ -0,0 +1,9 @@
|
|
1
|
+
require 'webmock/rspec'
|
2
|
+
require 'avro_turf/confluent_schema_registry'
|
3
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
4
|
+
|
5
|
+
describe AvroTurf::ConfluentSchemaRegistry do
|
6
|
+
it_behaves_like "a confluent schema registry client" do
|
7
|
+
let(:registry) { described_class.new(registry_url, logger: logger) }
|
8
|
+
end
|
9
|
+
end
|
data/spec/messaging_spec.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
require 'webmock/rspec'
|
2
2
|
require 'avro_turf/messaging'
|
3
|
-
require 'avro_turf/test/
|
3
|
+
require 'avro_turf/test/fake_confluent_schema_registry_server'
|
4
4
|
|
5
5
|
describe AvroTurf::Messaging do
|
6
6
|
let(:registry_url) { "http://registry.example.com" }
|
@@ -21,8 +21,8 @@ describe AvroTurf::Messaging do
|
|
21
21
|
end
|
22
22
|
|
23
23
|
before do
|
24
|
-
stub_request(:any, /^#{registry_url}/).to_rack(
|
25
|
-
|
24
|
+
stub_request(:any, /^#{registry_url}/).to_rack(FakeConfluentSchemaRegistryServer)
|
25
|
+
FakeConfluentSchemaRegistryServer.clear
|
26
26
|
end
|
27
27
|
|
28
28
|
before do
|
@@ -63,7 +63,7 @@ describe AvroTurf::Messaging do
|
|
63
63
|
it_behaves_like "encoding and decoding"
|
64
64
|
|
65
65
|
context "with a provided registry" do
|
66
|
-
let(:registry) { AvroTurf::
|
66
|
+
let(:registry) { AvroTurf::ConfluentSchemaRegistry.new(registry_url, logger: logger) }
|
67
67
|
|
68
68
|
let(:avro) do
|
69
69
|
AvroTurf::Messaging.new(
|
@@ -1,6 +1,6 @@
|
|
1
1
|
# This shared example expects a registry variable to be defined
|
2
2
|
# with an instance of the registry class being tested.
|
3
|
-
shared_examples_for "a schema registry client" do
|
3
|
+
shared_examples_for "a confluent schema registry client" do
|
4
4
|
let(:logger) { Logger.new(StringIO.new) }
|
5
5
|
let(:registry_url) { "http://registry.example.com" }
|
6
6
|
let(:subject_name) { "some-subject" }
|
@@ -15,8 +15,8 @@ shared_examples_for "a schema registry client" do
|
|
15
15
|
end
|
16
16
|
|
17
17
|
before do
|
18
|
-
stub_request(:any, /^#{registry_url}/).to_rack(
|
19
|
-
|
18
|
+
stub_request(:any, /^#{registry_url}/).to_rack(FakeConfluentSchemaRegistryServer)
|
19
|
+
FakeConfluentSchemaRegistryServer.clear
|
20
20
|
end
|
21
21
|
|
22
22
|
describe "#register and #fetch" do
|
@@ -178,6 +178,67 @@ shared_examples_for "a schema registry client" do
|
|
178
178
|
end
|
179
179
|
end
|
180
180
|
|
181
|
+
describe "#global_config" do
|
182
|
+
let(:expected) do
|
183
|
+
{ compatibility: 'BACKWARD' }.to_json
|
184
|
+
end
|
185
|
+
|
186
|
+
it "returns the global configuration" do
|
187
|
+
expect(registry.global_config).to eq(JSON.parse(expected))
|
188
|
+
end
|
189
|
+
end
|
190
|
+
|
191
|
+
describe "#update_global_config" do
|
192
|
+
let(:config) do
|
193
|
+
{ compatibility: 'FORWARD' }
|
194
|
+
end
|
195
|
+
let(:expected) { config.to_json }
|
196
|
+
|
197
|
+
it "updates the global configuration and returns it" do
|
198
|
+
expect(registry.update_global_config(config)).to eq(JSON.parse(expected))
|
199
|
+
expect(registry.global_config).to eq(JSON.parse(expected))
|
200
|
+
end
|
201
|
+
end
|
202
|
+
|
203
|
+
describe "#subject_config" do
|
204
|
+
let(:expected) do
|
205
|
+
{ compatibility: 'BACKWARD' }.to_json
|
206
|
+
end
|
207
|
+
|
208
|
+
context "when the subject config is not set" do
|
209
|
+
it "returns the global configuration" do
|
210
|
+
expect(registry.subject_config(subject_name)).to eq(JSON.parse(expected))
|
211
|
+
end
|
212
|
+
end
|
213
|
+
|
214
|
+
context "when the subject config is set" do
|
215
|
+
let(:config) do
|
216
|
+
{ compatibility: 'FULL' }
|
217
|
+
end
|
218
|
+
let(:expected) { config.to_json }
|
219
|
+
|
220
|
+
before do
|
221
|
+
registry.update_subject_config(subject_name, config)
|
222
|
+
end
|
223
|
+
|
224
|
+
it "returns the subject config" do
|
225
|
+
expect(registry.subject_config(subject_name)).to eq(JSON.parse(expected))
|
226
|
+
end
|
227
|
+
end
|
228
|
+
end
|
229
|
+
|
230
|
+
describe "#update_subject_config" do
|
231
|
+
let(:config) do
|
232
|
+
{ compatibility: 'NONE' }
|
233
|
+
end
|
234
|
+
let(:expected) { config.to_json }
|
235
|
+
|
236
|
+
it "updates the subject config and returns it" do
|
237
|
+
expect(registry.update_subject_config(subject_name, config)).to eq(JSON.parse(expected))
|
238
|
+
expect(registry.subject_config(subject_name)).to eq(JSON.parse(expected))
|
239
|
+
end
|
240
|
+
end
|
241
|
+
|
181
242
|
# Monkey patch an Avro::Schema to simulate the presence of
|
182
243
|
# active_support/core_ext.
|
183
244
|
def break_to_json(avro_schema)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: avro_turf
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.8.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Daniel Schierbeck
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2017-03-06 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: avro
|
@@ -151,6 +151,7 @@ extra_rdoc_files: []
|
|
151
151
|
files:
|
152
152
|
- ".gitignore"
|
153
153
|
- ".rspec"
|
154
|
+
- CHANGELOG.md
|
154
155
|
- Gemfile
|
155
156
|
- LICENSE.txt
|
156
157
|
- README.md
|
@@ -158,7 +159,9 @@ files:
|
|
158
159
|
- avro_turf.gemspec
|
159
160
|
- circle.yml
|
160
161
|
- lib/avro_turf.rb
|
162
|
+
- lib/avro_turf/cached_confluent_schema_registry.rb
|
161
163
|
- lib/avro_turf/cached_schema_registry.rb
|
164
|
+
- lib/avro_turf/confluent_schema_registry.rb
|
162
165
|
- lib/avro_turf/core_ext.rb
|
163
166
|
- lib/avro_turf/core_ext/date.rb
|
164
167
|
- lib/avro_turf/core_ext/enumerable.rb
|
@@ -174,6 +177,7 @@ files:
|
|
174
177
|
- lib/avro_turf/schema_registry.rb
|
175
178
|
- lib/avro_turf/schema_store.rb
|
176
179
|
- lib/avro_turf/schema_to_avro_patch.rb
|
180
|
+
- lib/avro_turf/test/fake_confluent_schema_registry_server.rb
|
177
181
|
- lib/avro_turf/test/fake_schema_registry_server.rb
|
178
182
|
- lib/avro_turf/version.rb
|
179
183
|
- perf/address.avsc
|
@@ -181,7 +185,8 @@ files:
|
|
181
185
|
- perf/encoding_speed.rb
|
182
186
|
- perf/person.avsc
|
183
187
|
- spec/avro_turf_spec.rb
|
184
|
-
- spec/
|
188
|
+
- spec/cached_confluent_schema_registry_spec.rb
|
189
|
+
- spec/confluent_schema_registry_spec.rb
|
185
190
|
- spec/core_ext/date_spec.rb
|
186
191
|
- spec/core_ext/enumerable_spec.rb
|
187
192
|
- spec/core_ext/false_class_spec.rb
|
@@ -193,16 +198,23 @@ files:
|
|
193
198
|
- spec/core_ext/time_spec.rb
|
194
199
|
- spec/core_ext/true_class_spec.rb
|
195
200
|
- spec/messaging_spec.rb
|
196
|
-
- spec/schema_registry_spec.rb
|
197
201
|
- spec/schema_store_spec.rb
|
198
202
|
- spec/schema_to_avro_patch_spec.rb
|
199
203
|
- spec/spec_helper.rb
|
200
|
-
- spec/support/
|
204
|
+
- spec/support/confluent_schema_registry_context.rb
|
201
205
|
homepage: https://github.com/dasch/avro_turf
|
202
206
|
licenses:
|
203
207
|
- MIT
|
204
208
|
metadata: {}
|
205
|
-
post_install_message:
|
209
|
+
post_install_message: |2
|
210
|
+
|
211
|
+
avro_turf v0.8.0 deprecates the names AvroTurf::SchemaRegistry,
|
212
|
+
AvroTurf::CachedSchemaRegistry, and FakeSchemaRegistryServer.
|
213
|
+
|
214
|
+
Use AvroTurf::ConfluentSchemaRegistry, AvroTurf::CachedConfluentSchemaRegistry,
|
215
|
+
and FakeConfluentSchemaRegistryServer instead.
|
216
|
+
|
217
|
+
See https://github.com/dasch/avro_turf#deprecation-notice
|
206
218
|
rdoc_options: []
|
207
219
|
require_paths:
|
208
220
|
- lib
|
@@ -225,7 +237,8 @@ summary: A library that makes it easier to use the Avro serialization format fro
|
|
225
237
|
Ruby
|
226
238
|
test_files:
|
227
239
|
- spec/avro_turf_spec.rb
|
228
|
-
- spec/
|
240
|
+
- spec/cached_confluent_schema_registry_spec.rb
|
241
|
+
- spec/confluent_schema_registry_spec.rb
|
229
242
|
- spec/core_ext/date_spec.rb
|
230
243
|
- spec/core_ext/enumerable_spec.rb
|
231
244
|
- spec/core_ext/false_class_spec.rb
|
@@ -237,9 +250,7 @@ test_files:
|
|
237
250
|
- spec/core_ext/time_spec.rb
|
238
251
|
- spec/core_ext/true_class_spec.rb
|
239
252
|
- spec/messaging_spec.rb
|
240
|
-
- spec/schema_registry_spec.rb
|
241
253
|
- spec/schema_store_spec.rb
|
242
254
|
- spec/schema_to_avro_patch_spec.rb
|
243
255
|
- spec/spec_helper.rb
|
244
|
-
- spec/support/
|
245
|
-
has_rdoc:
|
256
|
+
- spec/support/confluent_schema_registry_context.rb
|
@@ -1,9 +0,0 @@
|
|
1
|
-
require 'webmock/rspec'
|
2
|
-
require 'avro_turf/schema_registry'
|
3
|
-
require 'avro_turf/test/fake_schema_registry_server'
|
4
|
-
|
5
|
-
describe AvroTurf::SchemaRegistry do
|
6
|
-
it_behaves_like "a schema registry client" do
|
7
|
-
let(:registry) { described_class.new(registry_url, logger: logger) }
|
8
|
-
end
|
9
|
-
end
|