logstash-codec-protobuf 0.1.1 → 0.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +18 -7
- data/lib/logstash/codecs/protobuf.rb +136 -30
- data/logstash-codec-protobuf.gemspec +1 -1
- data/spec/codecs/protobuf_spec.rb +183 -6
- data/spec/helpers/ColourTestcase.pb.rb +35 -0
- data/spec/helpers/ColourTestcase.proto +24 -0
- data/spec/helpers/human.pb.rb +26 -0
- data/spec/helpers/unicorn.pb.rb +1 -0
- data/spec/helpers/unicorn_event.pb.rb +24 -0
- metadata +10 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 32ac341961786a035ed34e5e221de5e6944c2226
|
4
|
+
data.tar.gz: 8bb727065267603b4d0d771e2ab1937e7eb83660
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ba83ef5a43fa82e25d6cb248fa8076982b43d833df1d2ee62ee69c045bedd2664ff04a5b112d880344ebaf470116a176ab008bbccad9f6aab368c30b513ce5d1
|
7
|
+
data.tar.gz: 9699d09931ca8bac5d0242c4952b8496467d8c586032decdce6213ae956e43053b08039115a3deee7db7eaa74fd2d8a9700c32a12d794dcbaf338cbb01419c82
|
data/README.md
CHANGED
@@ -10,7 +10,13 @@ This is a codec plugin for [Logstash](https://github.com/elastic/logstash) to pa
|
|
10
10
|
bin/plugin install /path/to/logstash-codec-protobuf-$VERSION.gem
|
11
11
|
* use the codec in your logstash config file. See details below.
|
12
12
|
|
13
|
-
##
|
13
|
+
## Configuration
|
14
|
+
|
15
|
+
include_path (required): an array of strings with filenames or directory names where logstash can find your protobuf definitions. Please provide absolute paths. For directories it will only try to import files ending on .rb
|
16
|
+
|
17
|
+
class_name (required): the name of the protobuf class that is to be decoded or encoded.
|
18
|
+
|
19
|
+
## Usage example: decoder
|
14
20
|
|
15
21
|
Use this as a codec in any logstash input. Just provide the name of the class that your incoming objects will be encoded in, and specify the path to the compiled definition.
|
16
22
|
Here's an example for a kafka input:
|
@@ -46,11 +52,11 @@ Set the class name to the parent class:
|
|
46
52
|
|
47
53
|
class_name => "Foods::Dairy::Cheese"
|
48
54
|
|
49
|
-
##
|
55
|
+
## Usage example: encoder
|
50
56
|
|
51
|
-
|
52
|
-
|
53
|
-
|
57
|
+
The configuration of the codec for encoding logstash events for a protobuf output is pretty much the same as for the decoder input usage as demonstrated above. There are some constraints though that you need to be aware of:
|
58
|
+
* the protobuf definition needs to contain all the fields that logstash typically adds to an event, in the corrent data type. Examples for this are @timestamp (string), @version (string), host, path, all of which depend on your input sources and filters aswell. If you do not want to add those fields to your protobuf definition then please use a [modify filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html) to [remove](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field) the undesired fields.
|
59
|
+
* object members starting with @ are somewhat problematic in protobuf definitions. Therefore those fields will automatically be renamed to remove the at character. This also effects the important @timestamp field. Please name it just "timestamp" in your definition.
|
54
60
|
|
55
61
|
|
56
62
|
## Troubleshooting
|
@@ -59,7 +65,12 @@ class_name (required): the name of the protobuf class that is to be decoded.
|
|
59
65
|
|
60
66
|
If you include more than one definition class, consider the order of inclusion. This is especially relevant if you include whole directories. A definition might refer to another definition that is not loaded yet. In this case, please specify the files in the include_path variable in reverse order of reference. See 'Example with referenced definitions' above.
|
61
67
|
|
68
|
+
### no protobuf output
|
69
|
+
|
70
|
+
Maybe your protobuf definition does not fullfill the requirements and needs additional fields. Run logstash with the --debug flag and grep for "error 2".
|
71
|
+
|
72
|
+
|
73
|
+
## Limitations and roadmap
|
62
74
|
|
63
|
-
|
75
|
+
* maybe add support for setting undefined fields from default values in the decoder
|
64
76
|
|
65
|
-
Currently the plugin supports the decode functionality only. Maybe we'll add the encoding part also.
|
@@ -12,72 +12,178 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
|
|
12
12
|
# Name of the class to decode
|
13
13
|
config :class_name, :validate => :string, :required => true
|
14
14
|
|
15
|
-
|
15
|
+
# For benchmarking only, not intended for public use: change encoder strategy.
|
16
|
+
# valid method names are: encoder_strategy_1 (the others are not implemented yet)
|
17
|
+
config :encoder_method, :validate => :string, :default => "encoder_strategy_1"
|
18
|
+
|
16
19
|
def register
|
20
|
+
@pb_metainfo = {}
|
17
21
|
include_path.each { |path| require_pb_path(path) }
|
18
22
|
@obj = create_object_from_name(class_name)
|
19
23
|
@logger.debug("Protobuf files successfully loaded.")
|
24
|
+
|
20
25
|
end
|
21
26
|
|
22
27
|
def decode(data)
|
23
28
|
decoded = @obj.parse(data.to_s)
|
24
|
-
results =
|
29
|
+
results = keys2strings(decoded.to_hash)
|
25
30
|
yield LogStash::Event.new(results) if block_given?
|
26
31
|
end # def decode
|
27
32
|
|
33
|
+
def keys2strings(data)
|
34
|
+
if data.is_a?(::Hash)
|
35
|
+
new_hash = Hash.new
|
36
|
+
data.each{|k,v| new_hash[k.to_s] = keys2strings(v)}
|
37
|
+
new_hash
|
38
|
+
else
|
39
|
+
data
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
|
28
44
|
def encode(event)
|
29
|
-
|
30
|
-
|
31
|
-
# TODO integrate
|
45
|
+
protobytes = generate_protobuf(event)
|
46
|
+
@on_event.call(event, protobytes)
|
32
47
|
end # def encode
|
33
48
|
|
49
|
+
private
|
50
|
+
def generate_protobuf(event)
|
51
|
+
meth = self.method(encoder_method)
|
52
|
+
data = meth.call(event, @class_name)
|
53
|
+
begin
|
54
|
+
msg = @obj.new(data)
|
55
|
+
msg.serialize_to_string
|
56
|
+
rescue NoMethodError
|
57
|
+
@logger.debug("error 2: NoMethodError. Maybe mismatching protobuf definition. Required fields are: " + event.to_hash.keys.join(", "))
|
58
|
+
end
|
59
|
+
end
|
34
60
|
|
61
|
+
def encoder_strategy_1(event, class_name)
|
62
|
+
_encoder_strategy_1(event.to_hash, class_name)
|
63
|
+
|
64
|
+
end
|
65
|
+
|
66
|
+
def _encoder_strategy_1(datahash, class_name)
|
67
|
+
fields = clean_hash_keys(datahash)
|
68
|
+
fields = flatten_hash_values(fields) # TODO we could merge this and the above method back into one to save one iteration, but how are we going to name it?
|
69
|
+
meta = get_complex_types(class_name) # returns a hash with member names and their protobuf class names
|
70
|
+
meta.map do | (k,typeinfo) |
|
71
|
+
if fields.include?(k)
|
72
|
+
original_value = fields[k]
|
73
|
+
proto_obj = create_object_from_name(typeinfo)
|
74
|
+
fields[k] =
|
75
|
+
if original_value.is_a?(::Array)
|
76
|
+
ecs1_list_helper(original_value, proto_obj, typeinfo)
|
77
|
+
|
78
|
+
else
|
79
|
+
recursive_fix = _encoder_strategy_1(original_value, class_name)
|
80
|
+
proto_obj.new(recursive_fix)
|
81
|
+
end # if is array
|
82
|
+
end
|
83
|
+
|
84
|
+
end
|
85
|
+
|
86
|
+
fields
|
87
|
+
end
|
88
|
+
|
89
|
+
def ecs1_list_helper(value, proto_obj, class_name)
|
90
|
+
# make this field an array/list of protobuf objects
|
91
|
+
# value is a list of hashed complex objects, each of which needs to be protobuffed and
|
92
|
+
# put back into the list.
|
93
|
+
next unless value.is_a?(::Array)
|
94
|
+
value.map { |x| _encoder_strategy_1(x, class_name) }
|
95
|
+
value
|
96
|
+
end
|
97
|
+
|
98
|
+
def flatten_hash_values(datahash)
|
99
|
+
# 2) convert timestamps and other objects to strings
|
100
|
+
next unless datahash.is_a?(::Hash)
|
101
|
+
|
102
|
+
::Hash[datahash.map{|(k,v)| [k, (convert_to_string?(v) ? v.to_s : v)] }]
|
103
|
+
end
|
104
|
+
|
105
|
+
def clean_hash_keys(datahash)
|
106
|
+
# 1) remove @ signs from keys
|
107
|
+
next unless datahash.is_a?(::Hash)
|
108
|
+
|
109
|
+
::Hash[datahash.map{|(k,v)| [remove_atchar(k.to_s), v] }]
|
110
|
+
end #clean_hash_keys
|
111
|
+
|
112
|
+
def convert_to_string?(v)
|
113
|
+
!(v.is_a?(Fixnum) || v.is_a?(::Hash) || v.is_a?(::Array) || [true, false].include?(v))
|
114
|
+
end
|
115
|
+
|
116
|
+
|
117
|
+
def remove_atchar(key) # necessary for @timestamp fields and the likes. Protobuf definition doesn't handle @ in field names well.
|
118
|
+
key.dup.gsub(/@/,'')
|
119
|
+
end
|
35
120
|
|
36
121
|
private
|
37
122
|
def create_object_from_name(name)
|
38
123
|
begin
|
39
124
|
@logger.debug("Creating instance of " + name)
|
40
|
-
|
125
|
+
name.split('::').inject(Object) { |n,c| n.const_get c }
|
41
126
|
end
|
42
127
|
end
|
43
128
|
|
129
|
+
def get_complex_types(class_name)
|
130
|
+
@pb_metainfo[class_name]
|
131
|
+
end
|
132
|
+
|
133
|
+
def require_with_metadata_analysis(filename)
|
134
|
+
require filename
|
135
|
+
regex_class_name = /\s*class\s*(?<name>.+?)\s+/
|
136
|
+
regex_module_name = /\s*module\s*(?<name>.+?)\s+/
|
137
|
+
regex_pbdefs = /\s*(optional|repeated)(\s*):(?<type>.+),(\s*):(?<name>\w+),(\s*)(?<position>\d+)/
|
138
|
+
# now we also need to find out which class it contains and the protobuf definitions in it.
|
139
|
+
# We'll unfortunately need that later so that we can create nested objects.
|
140
|
+
begin
|
141
|
+
class_name = ""
|
142
|
+
type = ""
|
143
|
+
field_name = ""
|
144
|
+
classname_found = false
|
145
|
+
File.readlines(filename).each do |line|
|
146
|
+
if ! (line =~ regex_module_name).nil? && !classname_found # because it might be declared twice in the file
|
147
|
+
class_name << $1
|
148
|
+
class_name << "::"
|
149
|
+
|
150
|
+
end
|
151
|
+
if ! (line =~ regex_class_name).nil? && !classname_found # because it might be declared twice in the file
|
152
|
+
class_name << $1
|
153
|
+
@pb_metainfo[class_name] = {}
|
154
|
+
classname_found = true
|
155
|
+
end
|
156
|
+
if ! (line =~ regex_pbdefs).nil?
|
157
|
+
type = $1
|
158
|
+
field_name = $2
|
159
|
+
if type =~ /::/
|
160
|
+
@pb_metainfo[class_name][field_name] = type.gsub!(/^:/,"")
|
161
|
+
|
162
|
+
end
|
163
|
+
end
|
164
|
+
end
|
165
|
+
rescue Exception => e
|
166
|
+
@logger.warn("error 3: unable to read pb definition from file " + filename+ ". Reason: #{e.inspect}. Last settings were: class #{class_name} field #{field_name} type #{type}. Backtrace: " + e.backtrace.inspect.to_s)
|
167
|
+
end
|
168
|
+
if class_name.nil?
|
169
|
+
@logger.warn("error 4: class name not found in file " + filename)
|
170
|
+
end
|
171
|
+
end
|
44
172
|
|
45
173
|
def require_pb_path(dir_or_file)
|
46
174
|
f = dir_or_file.end_with? ('.rb')
|
47
175
|
begin
|
48
176
|
if f
|
49
177
|
@logger.debug("Including protobuf file: " + dir_or_file)
|
50
|
-
|
178
|
+
require_with_metadata_analysis dir_or_file
|
51
179
|
else
|
52
180
|
Dir[ dir_or_file + '/*.rb'].each { |file|
|
53
181
|
@logger.debug("Including protobuf path: " + dir_or_file + "/" + file)
|
54
|
-
|
182
|
+
require_with_metadata_analysis file
|
55
183
|
}
|
56
184
|
end
|
57
185
|
end
|
58
186
|
end
|
59
187
|
|
60
188
|
|
61
|
-
def extract_vars(decoded_object)
|
62
|
-
return {} if decoded_object.nil?
|
63
|
-
results = {}
|
64
|
-
decoded_object.instance_variables.each do |key|
|
65
|
-
formatted_key = key.to_s.gsub('@', '')
|
66
|
-
next if (formatted_key == :set_fields || formatted_key == "set_fields")
|
67
|
-
instance_var = decoded_object.instance_variable_get(key)
|
68
|
-
|
69
|
-
results[formatted_key] =
|
70
|
-
if instance_var.is_a?(::ProtocolBuffers::Message)
|
71
|
-
extract_vars(instance_var)
|
72
|
-
elsif instance_var.is_a?(::Hash)
|
73
|
-
instance_var.inject([]) { |h, (k, v)| h[k.to_s] = extract_vars(v); h }
|
74
|
-
elsif instance_var.is_a?(Enumerable) # is a list
|
75
|
-
instance_var.inject([]) { |h, v| h.push(extract_vars(v)); h }
|
76
|
-
else
|
77
|
-
instance_var
|
78
|
-
end
|
79
|
-
end
|
80
|
-
results
|
81
|
-
end
|
82
|
-
|
83
189
|
end # class LogStash::Codecs::Protobuf
|
@@ -1,7 +1,7 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
2
|
|
3
3
|
s.name = 'logstash-codec-protobuf'
|
4
|
-
s.version = '0.1.
|
4
|
+
s.version = '0.1.2'
|
5
5
|
s.licenses = ['Apache License (2.0)']
|
6
6
|
s.summary = "This codec may be used to decode (via inputs) and encode (via outputs) protobuf messages"
|
7
7
|
s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
|
@@ -1,3 +1,4 @@
|
|
1
|
+
# encoding: utf-8
|
1
2
|
require "logstash/devutils/rspec/spec_helper"
|
2
3
|
require "logstash/codecs/protobuf"
|
3
4
|
require "logstash/event"
|
@@ -6,23 +7,199 @@ require "insist"
|
|
6
7
|
|
7
8
|
describe LogStash::Codecs::Protobuf do
|
8
9
|
|
10
|
+
|
9
11
|
context "#decode" do
|
10
12
|
|
11
|
-
|
13
|
+
|
14
|
+
#### Test case 1: Decode simple protobuf bytes for unicorn ####################################################################################################################
|
15
|
+
let(:plugin_unicorn) { LogStash::Codecs::Protobuf.new("class_name" => "Animal::Unicorn", "include_path" => ['spec/helpers/unicorn.pb.rb']) }
|
12
16
|
before do
|
13
|
-
|
17
|
+
plugin_unicorn.register
|
14
18
|
end
|
15
19
|
|
16
20
|
it "should return an event from protobuf encoded data" do
|
17
21
|
|
18
|
-
data = {:colour => 'rainbow', :horn_length => 18, :last_seen => 1420081471}
|
22
|
+
data = {:colour => 'rainbow', :horn_length => 18, :last_seen => 1420081471, :has_wings => true}
|
19
23
|
unicorn = Animal::Unicorn.new(data)
|
20
24
|
|
21
|
-
|
22
|
-
|
25
|
+
plugin_unicorn.decode(unicorn.serialize_to_string) do |event|
|
23
26
|
expect(event["colour"] ).to eq(data[:colour] )
|
24
27
|
expect(event["horn_length"] ).to eq(data[:horn_length] )
|
28
|
+
expect(event["last_seen"] ).to eq(data[:last_seen] )
|
29
|
+
expect(event["has_wings"] ).to eq(data[:has_wings] )
|
30
|
+
end
|
31
|
+
end # it
|
32
|
+
|
33
|
+
|
34
|
+
|
35
|
+
#### Test case 2: Decode complex protobuf bytes for human #####################################################################################################################
|
36
|
+
|
37
|
+
|
38
|
+
|
39
|
+
|
40
|
+
let(:plugin_human) { LogStash::Codecs::Protobuf.new("class_name" => "Animal::Human", "include_path" => ['spec/helpers/human.pb.rb']) }
|
41
|
+
before do
|
42
|
+
plugin_human.register
|
43
|
+
end
|
44
|
+
|
45
|
+
it "should return an event from complex nested protobuf encoded data" do
|
46
|
+
|
47
|
+
data_gm = {:first_name => 'Elisabeth', :last_name => "Oliveoil", :middle_names => ["Maria","Johanna"], :vegetarian=>true}
|
48
|
+
grandmother = Animal::Human.new(data_gm)
|
49
|
+
data_m = {:first_name => 'Annemarie', :last_name => "Smørebrød", :mother => grandmother}
|
50
|
+
mother = Animal::Human.new(data_m)
|
51
|
+
data_f = {:first_name => 'Karl', :middle_names => ["Theodor-Augustin"], :last_name => "Falkenstein"}
|
52
|
+
father = Animal::Human.new(data_f)
|
53
|
+
data = {:first_name => 'Hugo', :middle_names => ["Heinz", "Peter"], :last_name => "Smørebrød",:father => father, :mother => mother}
|
54
|
+
hugo = Animal::Human.new(data)
|
55
|
+
|
56
|
+
plugin_human.decode(hugo.serialize_to_string) do |event|
|
57
|
+
expect(event["first_name"] ).to eq(data[:first_name] )
|
58
|
+
expect(event["middle_names"] ).to eq(data[:middle_names] )
|
59
|
+
expect(event["last_name"] ).to eq(data[:last_name] )
|
60
|
+
expect(event["mother"]["first_name"] ).to eq(data_m[:first_name] )
|
61
|
+
expect(event["father"]["first_name"] ).to eq(data_f[:first_name] )
|
62
|
+
expect(event["mother"]["last_name"] ).to eq(data_m[:last_name] )
|
63
|
+
expect(event["mother"]["mother"]["last_name"] ).to eq(data_gm[:last_name] )
|
64
|
+
expect(event["mother"]["mother"]["first_name"] ).to eq(data_gm[:first_name] )
|
65
|
+
expect(event["mother"]["mother"]["middle_names"] ).to eq(data_gm[:middle_names] )
|
66
|
+
expect(event["mother"]["mother"]["vegetarian"] ).to eq(data_gm[:vegetarian] )
|
67
|
+
expect(event["father"]["last_name"] ).to eq(data_f[:last_name] )
|
68
|
+
expect(event["father"]["middle_names"] ).to eq(data_f[:middle_names] )
|
25
69
|
end
|
70
|
+
end # it
|
71
|
+
|
72
|
+
|
73
|
+
|
74
|
+
|
75
|
+
|
76
|
+
|
77
|
+
#### Test case 3: Decoder test for enums #####################################################################################################################
|
78
|
+
|
79
|
+
|
80
|
+
|
81
|
+
|
82
|
+
let(:plugin_col) { LogStash::Codecs::Protobuf.new("class_name" => "ColourProtoTest", "include_path" => ['spec/helpers/ColourTestcase.pb.rb']) }
|
83
|
+
before do
|
84
|
+
plugin_col.register
|
85
|
+
end
|
86
|
+
|
87
|
+
it "should return an event from protobuf encoded data with enums" do
|
88
|
+
|
89
|
+
data = {:least_liked => ColourProtoTest::Colour::YELLOW, :favourite_colours => \
|
90
|
+
[ColourProtoTest::Colour::BLACK, ColourProtoTest::Colour::BLUE], :booleantest => [true, false, true]}
|
91
|
+
pb = ColourProtoTest.new(data)
|
92
|
+
|
93
|
+
plugin_col.decode(pb.serialize_to_string) do |event|
|
94
|
+
expect(event["least_liked"] ).to eq(data[:least_liked] )
|
95
|
+
expect(event["favourite_colours"] ).to eq(data[:favourite_colours] )
|
96
|
+
expect(event["booleantest"] ).to eq(data[:booleantest] )
|
97
|
+
end
|
98
|
+
end # it
|
99
|
+
|
100
|
+
|
101
|
+
end # context
|
102
|
+
|
103
|
+
|
104
|
+
|
105
|
+
|
106
|
+
|
107
|
+
#### Test case 4: Encode simple protobuf bytes for unicorn ####################################################################################################################
|
108
|
+
|
109
|
+
context "#encode" do
|
110
|
+
subject do
|
111
|
+
next LogStash::Codecs::Protobuf.new("class_name" => "Animal::UnicornEvent", "include_path" => ['spec/helpers/unicorn_event.pb.rb'])
|
112
|
+
end
|
113
|
+
|
114
|
+
event = LogStash::Event.new("colour" => "pink", "horn_length" => 12, "last_seen" => 1410081999, "has_wings" => true)
|
115
|
+
|
116
|
+
it "should return protobuf encoded data from a simple event" do
|
117
|
+
subject.on_event do |event, data|
|
118
|
+
insist { data.is_a? String }
|
119
|
+
unicorn = Animal::UnicornEvent.parse(data)
|
120
|
+
|
121
|
+
expect(unicorn.colour ).to eq(event["colour"] )
|
122
|
+
expect(unicorn.horn_length ).to eq(event["horn_length"] )
|
123
|
+
expect(unicorn.last_seen ).to eq(event["last_seen"] )
|
124
|
+
expect(unicorn.has_wings ).to eq(event["has_wings"] )
|
125
|
+
|
126
|
+
end # subject.on_event
|
127
|
+
subject.encode(event)
|
128
|
+
end # it
|
129
|
+
end # context
|
130
|
+
|
131
|
+
|
132
|
+
|
133
|
+
|
134
|
+
#### Test case 5: encode complex protobuf bytes for human #####################################################################################################################
|
135
|
+
|
136
|
+
|
137
|
+
context "#encode2" do
|
138
|
+
subject do
|
139
|
+
next LogStash::Codecs::Protobuf.new("class_name" => "Animal::Human", "include_path" => ['spec/helpers/human.pb.rb'])
|
26
140
|
end
|
27
|
-
|
141
|
+
|
142
|
+
event = LogStash::Event.new("first_name" => "Jimmy", "middle_names" => ["Bob", "James"], "last_name" => "Doe" \
|
143
|
+
, "mother" => {"first_name" => "Jane", "middle_names" => ["Elizabeth"], "last_name" => "Doe" , "age" => 83, "vegetarian"=> false} \
|
144
|
+
, "father" => {"first_name" => "John", "last_name" => "Doe", "@email" => "character_replacement_test@nothing" })
|
145
|
+
|
146
|
+
it "should return protobuf encoded data from a complex event" do
|
147
|
+
|
148
|
+
subject.on_event do |event, data|
|
149
|
+
insist { data.is_a? String }
|
150
|
+
jimmy = Animal::Human.parse(data)
|
151
|
+
|
152
|
+
expect(jimmy.first_name ).to eq(event["first_name"] )
|
153
|
+
expect(jimmy.middle_names ).to eq(event["middle_names"] )
|
154
|
+
expect(jimmy.last_name ).to eq(event["last_name"] )
|
155
|
+
expect(jimmy.mother.first_name ).to eq(event["mother"]["first_name"] )
|
156
|
+
expect(jimmy.father.first_name ).to eq(event["father"]["first_name"] )
|
157
|
+
expect(jimmy.mother.middle_names ).to eq(event["mother"]["middle_names"] )
|
158
|
+
expect(jimmy.mother.age ).to eq(event["mother"]["age"] ) # recursion test for values
|
159
|
+
expect(jimmy.mother.vegetarian ).to eq(event["mother"]["vegetarian"] ) # recursion test for values
|
160
|
+
expect(jimmy.father.last_name ).to eq(event["father"]["last_name"] )
|
161
|
+
expect(jimmy.father.email ).to eq(event["father"]["@email"] ) # recursion test for keys
|
162
|
+
expect(jimmy.mother.last_name ).to eq(event["mother"]["last_name"] )
|
163
|
+
|
164
|
+
end # subject.on_event
|
165
|
+
subject.encode(event)
|
166
|
+
end # it
|
167
|
+
end # context
|
168
|
+
|
169
|
+
|
170
|
+
|
171
|
+
|
172
|
+
|
173
|
+
#### Test case 6: encode enums #########################################################################################################################
|
174
|
+
|
175
|
+
|
176
|
+
|
177
|
+
context "#encode3" do
|
178
|
+
subject do
|
179
|
+
next LogStash::Codecs::Protobuf.new("class_name" => "ColourProtoTest", "include_path" => ['spec/helpers/ColourTestcase.pb.rb'])
|
180
|
+
end
|
181
|
+
|
182
|
+
require 'spec/helpers/ColourTestcase.pb.rb' # otherwise we cant use the colour enums in the next line
|
183
|
+
event = LogStash::Event.new("booleantest" => [false, false, true], "least_liked" => ColourProtoTest::Colour::YELLOW, "favourite_colours" => \
|
184
|
+
[ColourProtoTest::Colour::BLACK, ColourProtoTest::Colour::BLUE] )
|
185
|
+
|
186
|
+
it "should return protobuf encoded data from a complex event with enums" do
|
187
|
+
|
188
|
+
subject.on_event do |event, data|
|
189
|
+
insist { data.is_a? String }
|
190
|
+
|
191
|
+
colpref = ColourProtoTest.parse(data)
|
192
|
+
|
193
|
+
expect(colpref.booleantest ).to eq(event["booleantest"] )
|
194
|
+
expect(colpref.least_liked ).to eq(event["least_liked"] )
|
195
|
+
expect(colpref.favourite_colours ).to eq(event["favourite_colours"] )
|
196
|
+
|
197
|
+
|
198
|
+
end # subject.on_event
|
199
|
+
subject.encode(event)
|
200
|
+
end # it
|
201
|
+
end # context
|
202
|
+
|
203
|
+
|
204
|
+
|
28
205
|
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
# Generated by the protocol buffer compiler. DO NOT EDIT!
|
3
|
+
|
4
|
+
require 'protocol_buffers'
|
5
|
+
|
6
|
+
# forward declarations
|
7
|
+
class ColourProtoTest < ::ProtocolBuffers::Message; end
|
8
|
+
|
9
|
+
class ColourProtoTest < ::ProtocolBuffers::Message
|
10
|
+
# forward declarations
|
11
|
+
|
12
|
+
# enums
|
13
|
+
module Colour
|
14
|
+
include ::ProtocolBuffers::Enum
|
15
|
+
|
16
|
+
set_fully_qualified_name "ColourProtoTest.Colour"
|
17
|
+
|
18
|
+
BLACK = 0
|
19
|
+
BLUE = 1
|
20
|
+
WHITE = 2
|
21
|
+
GREEN = 3
|
22
|
+
RED = 4
|
23
|
+
YELLOW = 5
|
24
|
+
AQUA = 6
|
25
|
+
end
|
26
|
+
|
27
|
+
set_fully_qualified_name "ColourProtoTest"
|
28
|
+
|
29
|
+
repeated ::ColourProtoTest::Colour, :favourite_colours, 1
|
30
|
+
repeated :bool, :booleantest, 2
|
31
|
+
optional ::ColourProtoTest::Colour, :least_liked, 3
|
32
|
+
optional :string, :timestamp, 4
|
33
|
+
optional :string, :version, 5
|
34
|
+
end
|
35
|
+
|
@@ -0,0 +1,24 @@
|
|
1
|
+
|
2
|
+
|
3
|
+
message ColourProtoTest {
|
4
|
+
|
5
|
+
enum Colour {
|
6
|
+
BLACK = 0;
|
7
|
+
BLUE = 1;
|
8
|
+
WHITE = 2;
|
9
|
+
GREEN = 3;
|
10
|
+
RED = 4;
|
11
|
+
YELLOW = 5;
|
12
|
+
AQUA = 6;
|
13
|
+
|
14
|
+
}
|
15
|
+
|
16
|
+
// most liked colours; test enums in arrays.
|
17
|
+
repeated Colour favourite_colours = 1;
|
18
|
+
|
19
|
+
// why not also test booleans in arrays while we're at it.
|
20
|
+
repeated bool booleantest = 2;
|
21
|
+
|
22
|
+
// least liked colour
|
23
|
+
optional Colour least_liked = 3;
|
24
|
+
}
|
@@ -0,0 +1,26 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
# Generated by the protocol buffer compiler. DO NOT EDIT!
|
3
|
+
|
4
|
+
require 'protocol_buffers'
|
5
|
+
|
6
|
+
module Animal
|
7
|
+
# forward declarations
|
8
|
+
class Human < ::ProtocolBuffers::Message; end
|
9
|
+
|
10
|
+
class Human < ::ProtocolBuffers::Message
|
11
|
+
set_fully_qualified_name "animal.Human"
|
12
|
+
|
13
|
+
optional :string, :first_name, 1
|
14
|
+
repeated :string, :middle_names, 2
|
15
|
+
optional :string, :last_name, 3
|
16
|
+
optional ::Animal::Human, :mother, 4
|
17
|
+
optional ::Animal::Human, :father, 5
|
18
|
+
optional :string, :path, 6
|
19
|
+
optional :string, :version, 7
|
20
|
+
optional :string, :timestamp, 8
|
21
|
+
optional :string, :email, 9
|
22
|
+
optional :bool, :vegetarian, 10
|
23
|
+
optional :int32, :age, 11
|
24
|
+
end
|
25
|
+
|
26
|
+
end
|
data/spec/helpers/unicorn.pb.rb
CHANGED
@@ -0,0 +1,24 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
# Generated by the protocol buffer compiler. DO NOT EDIT!
|
3
|
+
|
4
|
+
require 'protocol_buffers'
|
5
|
+
|
6
|
+
module Animal
|
7
|
+
# forward declarations
|
8
|
+
class UnicornEvent < ::ProtocolBuffers::Message; end
|
9
|
+
|
10
|
+
class UnicornEvent < ::ProtocolBuffers::Message
|
11
|
+
set_fully_qualified_name "animal.UnicornEvent"
|
12
|
+
|
13
|
+
optional :string, :colour, 1
|
14
|
+
optional :int32, :horn_length, 2
|
15
|
+
optional :int32, :last_seen, 3
|
16
|
+
optional :string, :timestamp, 4
|
17
|
+
optional :string, :host, 5
|
18
|
+
optional :string, :path, 6
|
19
|
+
optional :string, :version, 7
|
20
|
+
optional :bool, :has_wings, 8
|
21
|
+
|
22
|
+
end
|
23
|
+
|
24
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-codec-protobuf
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Inga Feick
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-04-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: logstash-core
|
@@ -74,7 +74,11 @@ files:
|
|
74
74
|
- lib/logstash/codecs/protobuf.rb
|
75
75
|
- logstash-codec-protobuf.gemspec
|
76
76
|
- spec/codecs/protobuf_spec.rb
|
77
|
+
- spec/helpers/ColourTestcase.pb.rb
|
78
|
+
- spec/helpers/ColourTestcase.proto
|
79
|
+
- spec/helpers/human.pb.rb
|
77
80
|
- spec/helpers/unicorn.pb.rb
|
81
|
+
- spec/helpers/unicorn_event.pb.rb
|
78
82
|
homepage:
|
79
83
|
licenses:
|
80
84
|
- Apache License (2.0)
|
@@ -103,4 +107,8 @@ specification_version: 4
|
|
103
107
|
summary: This codec may be used to decode (via inputs) and encode (via outputs) protobuf messages
|
104
108
|
test_files:
|
105
109
|
- spec/codecs/protobuf_spec.rb
|
110
|
+
- spec/helpers/ColourTestcase.pb.rb
|
111
|
+
- spec/helpers/ColourTestcase.proto
|
112
|
+
- spec/helpers/human.pb.rb
|
106
113
|
- spec/helpers/unicorn.pb.rb
|
114
|
+
- spec/helpers/unicorn_event.pb.rb
|