logstash-filter-bytes 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: fadc30e8ea226b625ef2bfe6e7ad5ba0c661ad1ca183d949ca9c008a19924606
4
+ data.tar.gz: 5fcdc6c8d58ddedceab3266a9b0ce338a93849f8b3541386532c36dd6c7913cf
5
+ SHA512:
6
+ metadata.gz: 4ea531657bcf341a9b9eded3c5eb98551b1212e43a1a3b6e41085eb5a515d5e7c0b4d3c8f4c29dd329f6acdbdc1db5bdd0135d744119a3340649daaf29fe6ab2
7
+ data.tar.gz: fa00c9bf52bd727098b3e5a1c8d54c82c7b9276e4907564af67d2ae400d62aed37aaa76c4d2633666160bebc75319f20e884a2d8e085665a31c9ecddc7241e47
@@ -0,0 +1,2 @@
1
+ ## 0.1.0
2
+ - Plugin created with the logstash plugin generator
@@ -0,0 +1,10 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Shaunak Kashyap (ycombinator)
6
+
7
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
8
+ Logstash, and you aren't on the list above and want to be, please let us know
9
+ and we'll make sure you're here. Contributions from folks like you are what make
10
+ open source awesome.
data/Gemfile ADDED
@@ -0,0 +1,10 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
3
+
4
+ logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
5
+ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
6
+
7
+ if Dir.exist?(logstash_path) && use_logstash_source
8
+ gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
9
+ gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
10
+ end
data/LICENSE ADDED
@@ -0,0 +1,12 @@
1
+ Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
@@ -0,0 +1,86 @@
1
+ # Logstash Plugin
2
+
3
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
4
+
5
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
6
+
7
+ ## Documentation
8
+
9
+ Logstash provides infrastructure to automatically build documentation for this plugin. We provide a template file, index.asciidoc, where you can add documentation. The contents of this file will be converted into html and then placed with other plugin documentation in a [central location](http://www.elastic.co/guide/en/logstash/current/).
10
+
11
+ - For formatting config examples, you can use the asciidoc `[source,json]` directive
12
+ - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
13
+
14
+ ## Need Help?
15
+
16
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
17
+
18
+ ## Developing
19
+
20
+ ### 1. Plugin Developement and Testing
21
+
22
+ #### Code
23
+ - To get started, you'll need JRuby with the Bundler gem installed.
24
+
25
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
26
+
27
+ - Install dependencies
28
+ ```sh
29
+ bundle install
30
+ ```
31
+
32
+ #### Test
33
+
34
+ - Update your dependencies
35
+
36
+ ```sh
37
+ bundle install
38
+ ```
39
+
40
+ - Run tests
41
+
42
+ ```sh
43
+ bundle exec rspec
44
+ ```
45
+
46
+ ### 2. Running your unpublished Plugin in Logstash
47
+
48
+ #### 2.1 Run in a local Logstash clone
49
+
50
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
51
+ ```ruby
52
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
53
+ ```
54
+ - Install plugin
55
+ ```sh
56
+ bin/logstash-plugin install --no-verify
57
+ ```
58
+ - Run Logstash with your plugin
59
+ ```sh
60
+ bin/logstash -e 'filter {awesome {}}'
61
+ ```
62
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
63
+
64
+ #### 2.2 Run in an installed Logstash
65
+
66
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
67
+
68
+ - Build your plugin gem
69
+ ```sh
70
+ gem build logstash-filter-awesome.gemspec
71
+ ```
72
+ - Install the plugin from the Logstash home
73
+ ```sh
74
+ bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
75
+ ```
76
+ - Start Logstash and proceed to test the plugin
77
+
78
+ ## Contributing
79
+
80
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
81
+
82
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
83
+
84
+ It is more important to the community that you are able to contribute.
85
+
86
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,116 @@
1
+ # encoding: utf-8
2
+ require "logstash/filters/base"
3
+ require "logstash/namespace"
4
+
5
+ # This filter will parse a given string as a computer storage
6
+ # value (e.g. "123 MB" or "5.3GB") and add a new numeric field with
7
+ # size in bytes
8
+ #
9
+ class LogStash::Filters::Bytes < LogStash::Filters::Base
10
+
11
+ PREFIX_POWERS = {
12
+ 'k' => 1, # 1 kilobyte = 1024 ^ 1 bytes
13
+ 'm' => 2, # 1 megabyte = 1024 ^ 2 bytes
14
+ 'g' => 3, # 1 gigabyte = 1024 ^ 3 bytes
15
+ 't' => 4, # 1 terabyte = 1024 ^ 4 bytes
16
+ 'p' => 5, # 1 petabyte = 1024 ^ 5 bytes
17
+ 'e' => 6, # 1 exabyte = 1024 ^ 6 bytes
18
+ }.freeze
19
+
20
+ DIGIT_GROUP_SEPARATORS = " _,."
21
+
22
+ # Setting the config_name here is required. This is how you
23
+ # configure this filter from your Logstash config.
24
+ #
25
+ # filter {
26
+ # bytes {
27
+ # source => "123 MB"
28
+ # target => "num_bytes"
29
+ # }
30
+ # }
31
+ #
32
+ config_name "bytes"
33
+
34
+ # Source string, e.g. "123 MB", "5.3GB", etc.
35
+ config :source, :validate => :string, :default => "message"
36
+
37
+ # Target field name
38
+ config :target, :validate => :string
39
+
40
+ # Conversion method, either "binary" (1K = 1024B) or "metric" (1K = 1000B)
41
+ config :conversion_method, :validate => [ "binary", "metric" ], :default => "binary"
42
+
43
+ # Decimal separator
44
+ config :decimal_separator, :validate => [ ".", "," ] , :default => "."
45
+
46
+ # Append values to the `tags` field when there has been no
47
+ # successful match
48
+ config :tag_on_failure, :validate => :array, :default => ["_bytesparsefailure"]
49
+
50
+ private
51
+ def normalize_number(number)
52
+ number
53
+ .tr("^0-9#{@decimal_separator}", '')
54
+ .tr(@decimal_separator, '.')
55
+ end
56
+
57
+ public
58
+ def register
59
+ # Add instance variables
60
+ end # def register
61
+
62
+ public
63
+ def filter(event)
64
+
65
+ source = event.get(@source)
66
+
67
+ if !source
68
+ @tag_on_failure.each{|tag| event.tag(tag)}
69
+ return
70
+ end
71
+ source.strip!
72
+
73
+ # Parse the source into the number part (e.g. 123),
74
+ # the unit prefix part (e.g. M), and the unit suffix part (e.g. B)
75
+ match = source.match(/^([0-9#{DIGIT_GROUP_SEPARATORS}#{@decimal_separator}]*)\s*([kKmMgGtTpPeE]?)([bB]?)$/)
76
+ if !match
77
+ @tag_on_failure.each{|tag| event.tag(tag)}
78
+ return
79
+ end
80
+
81
+ number, prefix, suffix = match.captures
82
+
83
+ # Flag error if more than one decimal separator is found
84
+ num_decimals = number.count(@decimal_separator)
85
+ if num_decimals > 1
86
+ @tag_on_failure.each{|tag| event.tag(tag)}
87
+ return
88
+ end
89
+
90
+ number = normalize_number(number.strip)
91
+ if number == ''
92
+ @tag_on_failure.each{|tag| event.tag(tag)}
93
+ return
94
+ end
95
+
96
+ if suffix == ''
97
+ suffix = 'B'
98
+ end
99
+
100
+ # Convert the number to bytes
101
+ result = number.to_f
102
+ if prefix != ''
103
+ if @conversion_method == 'binary'
104
+ result *= (1024 ** PREFIX_POWERS[prefix.downcase])
105
+ else
106
+ result *= (1000 ** PREFIX_POWERS[prefix.downcase])
107
+ end
108
+ end
109
+ result = result.round
110
+
111
+ event.set(@target, result)
112
+
113
+ # filter_matched should go in the last line of our successful code
114
+ filter_matched(event)
115
+ end # def filter
116
+ end # class LogStash::Filters::Bytes
@@ -0,0 +1,23 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-filter-bytes'
3
+ s.version = '1.0.0'
4
+ s.licenses = ['Apache-2.0']
5
+ s.summary = "This filter parses strings representing computer storage sizes into numeric bytes"
6
+ s.description = 'This filter parses strings representing computer storage sizes (e.g. "123 MB" or "6.3GB") into numeric bytes (12'
7
+ s.homepage = 'https://github.com/logstash-plugins/logstash-filter-bytes'
8
+ s.authors = ['Elastic']
9
+ s.email = 'info@elastic.co'
10
+ s.require_paths = ['lib']
11
+
12
+ # Files
13
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
14
+ # Tests
15
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
16
+
17
+ # Special flag to let us know this is actually a logstash plugin
18
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
19
+
20
+ # Gem dependencies
21
+ s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
22
+ s.add_development_dependency 'logstash-devutils'
23
+ end
@@ -0,0 +1,276 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/filters/bytes"
4
+
5
+ describe LogStash::Filters::Bytes do
6
+ let(:config) do <<-CONFIG
7
+ filter {
8
+ bytes {
9
+ target => dest
10
+ tag_on_failure => [ "boom", "_bytesparsefailure" ]
11
+ }
12
+ }
13
+ CONFIG
14
+ end
15
+
16
+ describe "empty" do
17
+ sample("") do
18
+ expect(subject.get('tags')).to eq(["boom", "_bytesparsefailure"])
19
+ end
20
+ end
21
+
22
+ describe "garbage" do
23
+ sample("abcdef") do
24
+ expect(subject.get('tags')).to eq(["boom", "_bytesparsefailure"])
25
+ end
26
+ end
27
+
28
+ describe "no number, only units" do
29
+ sample("mb") do
30
+ expect(subject.get('tags')).to eq(["boom", "_bytesparsefailure"])
31
+ end
32
+ end
33
+
34
+ describe "no units" do
35
+ sample("0") do
36
+ expect(subject).to include("dest")
37
+ expect(subject.get('dest')).to eq(0)
38
+ end
39
+
40
+ sample("123") do
41
+ expect(subject).to include("dest")
42
+ expect(subject.get('dest')).to eq(123)
43
+ end
44
+
45
+ sample("12.3") do
46
+ expect(subject).to include("dest")
47
+ expect(subject.get('dest')).to eq(12)
48
+ end
49
+ end
50
+
51
+ describe "from bytes" do
52
+ sample("32.8B") do
53
+ expect(subject).to include("dest")
54
+ expect(subject.get('dest')).to eq(33)
55
+ end
56
+ end
57
+
58
+ describe "from kilobytes" do
59
+ sample("32kB") do
60
+ expect(subject).to include("dest")
61
+ expect(subject.get('dest')).to eq(32 * 1024)
62
+ end
63
+
64
+ sample("32KB") do
65
+ expect(subject).to include("dest")
66
+ expect(subject.get('dest')).to eq(32 * 1024)
67
+ end
68
+ end
69
+
70
+ describe "from megabytes" do
71
+ sample("32mB") do
72
+ expect(subject).to include("dest")
73
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024)
74
+ end
75
+
76
+ sample("32mb") do
77
+ expect(subject).to include("dest")
78
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024)
79
+ end
80
+
81
+ sample("32MB") do
82
+ expect(subject).to include("dest")
83
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024)
84
+ end
85
+ end
86
+
87
+ describe "from gigabytes" do
88
+ sample("32gB") do
89
+ expect(subject).to include("dest")
90
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024)
91
+ end
92
+
93
+ sample("32gb") do
94
+ expect(subject).to include("dest")
95
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024)
96
+ end
97
+
98
+ sample("32GB") do
99
+ expect(subject).to include("dest")
100
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024)
101
+ end
102
+ end
103
+
104
+ describe "from terabytes" do
105
+ sample("32tB") do
106
+ expect(subject).to include("dest")
107
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024)
108
+ end
109
+
110
+ sample("32tb") do
111
+ expect(subject).to include("dest")
112
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024)
113
+ end
114
+
115
+ sample("32TB") do
116
+ expect(subject).to include("dest")
117
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024)
118
+ end
119
+ end
120
+
121
+ describe "from petabytes" do
122
+ sample("32pB") do
123
+ expect(subject).to include("dest")
124
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024 * 1024)
125
+ end
126
+
127
+ sample("32pb") do
128
+ expect(subject).to include("dest")
129
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024 * 1024)
130
+ end
131
+
132
+ sample("32PB") do
133
+ expect(subject).to include("dest")
134
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024 * 1024)
135
+ end
136
+ end
137
+
138
+ describe "from exabytes" do
139
+ sample("32eB") do
140
+ expect(subject).to include("dest")
141
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024)
142
+ end
143
+
144
+ sample("32eb") do
145
+ expect(subject).to include("dest")
146
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024)
147
+ end
148
+
149
+ sample("32EB") do
150
+ expect(subject).to include("dest")
151
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024)
152
+ end
153
+ end
154
+
155
+ describe "with spaces" do
156
+ sample("32 mb") do
157
+ expect(subject).to include("dest")
158
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024)
159
+ end
160
+
161
+ sample("32\tmb") do
162
+ expect(subject).to include("dest")
163
+ expect(subject.get('dest')).to eq(32 * 1024 * 1024)
164
+ end
165
+ end
166
+
167
+ describe "using metric conversion system" do
168
+ let(:config) do <<-CONFIG
169
+ filter {
170
+ bytes {
171
+ target => dest
172
+ conversion_method => metric
173
+ }
174
+ }
175
+ CONFIG
176
+ end
177
+
178
+ sample("32 mb") do
179
+ expect(subject).to include("dest")
180
+ expect(subject.get('dest')).to eq(32 * 1000 * 1000)
181
+ end
182
+ end
183
+
184
+ describe "no decimal separator" do
185
+ sample("3 mb") do
186
+ expect(subject).to include("dest")
187
+ expect(subject.get('dest')).to eq(3 * 1024 * 1024)
188
+ end
189
+
190
+ sample("3,124 mb") do
191
+ expect(subject).to include("dest")
192
+ expect(subject.get('dest')).to eq(3124 * 1024 * 1024)
193
+ end
194
+ end
195
+
196
+ describe "digits only to right of decimal separator" do
197
+ sample(".3124 mb") do
198
+ expect(subject).to include("dest")
199
+ expect(subject.get('dest')).to eq((0.3124 * 1024 * 1024).round)
200
+ end
201
+ end
202
+
203
+ describe "digits only to left of decimal separator" do
204
+ sample("3. mb") do
205
+ expect(subject).to include("dest")
206
+ expect(subject.get('dest')).to eq(3 * 1024 * 1024)
207
+ end
208
+
209
+ sample("3,124. mb") do
210
+ expect(subject).to include("dest")
211
+ expect(subject.get('dest')).to eq(3124 * 1024 * 1024)
212
+ end
213
+ end
214
+
215
+ describe "digits on both sides of decimal separator" do
216
+ sample("3.56 mb") do
217
+ expect(subject).to include("dest")
218
+ expect(subject.get('dest')).to eq((3.56 * 1024 * 1024).round)
219
+ end
220
+
221
+ sample("3,124.56 mb") do
222
+ expect(subject).to include("dest")
223
+ expect(subject.get('dest')).to eq((3124.56 * 1024 * 1024).round)
224
+ end
225
+ end
226
+
227
+ describe "non-default decimal separator" do
228
+ let(:config) do <<-CONFIG
229
+ filter {
230
+ bytes {
231
+ target => dest
232
+ decimal_separator => ','
233
+ }
234
+ }
235
+ CONFIG
236
+ end
237
+
238
+ sample("3,9 mb") do
239
+ expect(subject).to include("dest")
240
+ expect(subject.get('dest')).to eq((3.9 * 1024 * 1024).round)
241
+ end
242
+
243
+ sample("1.9 kb") do
244
+ expect(subject).to include("dest")
245
+ expect(subject.get('dest')).to eq((19 * 1024).round)
246
+ end
247
+
248
+ sample("3.124,56 mb") do
249
+ expect(subject).to include("dest")
250
+ expect(subject.get('dest')).to eq((3124.56 * 1024 * 1024).round)
251
+ end
252
+ end
253
+
254
+ describe "with two decimal separators" do
255
+ sample("1.000.000 mb") do
256
+ expect(subject.get('tags')).to eq(["boom", "_bytesparsefailure"])
257
+ end
258
+
259
+ context "non-default decimal separator" do
260
+ let(:config) do <<-CONFIG
261
+ filter {
262
+ bytes {
263
+ target => dest
264
+ decimal_separator => ','
265
+ tag_on_failure => [ "boom", "_bytesparsefailure" ]
266
+ }
267
+ }
268
+ CONFIG
269
+ end
270
+
271
+ sample("1,000,000 mb") do
272
+ expect(subject.get('tags')).to eq(["boom", "_bytesparsefailure"])
273
+ end
274
+ end
275
+ end
276
+ end
metadata ADDED
@@ -0,0 +1,84 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-filter-bytes
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0
5
+ platform: ruby
6
+ authors:
7
+ - Elastic
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2018-04-23 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - "~>"
17
+ - !ruby/object:Gem::Version
18
+ version: '2.0'
19
+ name: logstash-core-plugin-api
20
+ prerelease: false
21
+ type: :runtime
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '2.0'
27
+ - !ruby/object:Gem::Dependency
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - ">="
31
+ - !ruby/object:Gem::Version
32
+ version: '0'
33
+ name: logstash-devutils
34
+ prerelease: false
35
+ type: :development
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ description: This filter parses strings representing computer storage sizes (e.g.
42
+ "123 MB" or "6.3GB") into numeric bytes (12
43
+ email: info@elastic.co
44
+ executables: []
45
+ extensions: []
46
+ extra_rdoc_files: []
47
+ files:
48
+ - CHANGELOG.md
49
+ - CONTRIBUTORS
50
+ - Gemfile
51
+ - LICENSE
52
+ - README.md
53
+ - lib/logstash/filters/bytes.rb
54
+ - logstash-filter-bytes.gemspec
55
+ - spec/filters/bytes_spec.rb
56
+ homepage: https://github.com/logstash-plugins/logstash-filter-bytes
57
+ licenses:
58
+ - Apache-2.0
59
+ metadata:
60
+ logstash_plugin: 'true'
61
+ logstash_group: filter
62
+ post_install_message:
63
+ rdoc_options: []
64
+ require_paths:
65
+ - lib
66
+ required_ruby_version: !ruby/object:Gem::Requirement
67
+ requirements:
68
+ - - ">="
69
+ - !ruby/object:Gem::Version
70
+ version: '0'
71
+ required_rubygems_version: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - ">="
74
+ - !ruby/object:Gem::Version
75
+ version: '0'
76
+ requirements: []
77
+ rubyforge_project:
78
+ rubygems_version: 2.6.13
79
+ signing_key:
80
+ specification_version: 4
81
+ summary: This filter parses strings representing computer storage sizes into numeric
82
+ bytes
83
+ test_files:
84
+ - spec/filters/bytes_spec.rb