logstash-filter-dynamo_enrich 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 32bc4f215634abd8e6c5e9477e7c88903a90ba67
4
+ data.tar.gz: 4f022b96d8bf2eee5ea1757799c3f138e65c2636
5
+ SHA512:
6
+ metadata.gz: aafaf79d593c53172ee62567458b4529068a498242a982a0714711e08433440ef010415708985a0a8921c4d85d8ae1bf7d0a3ca6862b1babc66f62acca0e4f2b
7
+ data.tar.gz: 0302ba0ad9e0eaba99cff35878524b93082caacc3bf9026af0bbc5cbe79afeaca1bffe67953e43334944f24fe1c212606ee183dbc2212c6a3946678db22b1c14
@@ -0,0 +1,2 @@
1
+ ## 0.1.0
2
+ - Plugin created with the logstash plugin generator
@@ -0,0 +1,10 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * your_username - your_username@example.com
6
+
7
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
8
+ Logstash, and you aren't on the list above and want to be, please let us know
9
+ and we'll make sure you're here. Contributions from folks like you are what make
10
+ open source awesome.
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
3
+
data/LICENSE ADDED
@@ -0,0 +1,11 @@
1
+ Licensed under the Apache License, Version 2.0 (the "License");
2
+ you may not use this file except in compliance with the License.
3
+ You may obtain a copy of the License at
4
+
5
+ http://www.apache.org/licenses/LICENSE-2.0
6
+
7
+ Unless required by applicable law or agreed to in writing, software
8
+ distributed under the License is distributed on an "AS IS" BASIS,
9
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
10
+ See the License for the specific language governing permissions and
11
+ limitations under the License.
@@ -0,0 +1,83 @@
1
+ # Logstash Plugin
2
+
3
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
4
+
5
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
6
+
7
+ ## Documentation
8
+
9
+ See [Docs](./docs)
10
+
11
+ ## Need Help?
12
+
13
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
14
+
15
+ ## Developing
16
+
17
+ ### 1. Plugin Developement and Testing
18
+
19
+ #### Code
20
+ - To get started, you'll need JRuby with the Bundler gem installed.
21
+
22
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
23
+
24
+ - Install dependencies
25
+ ```sh
26
+ bundle install
27
+ ```
28
+
29
+ #### Test
30
+
31
+ - Update your dependencies
32
+
33
+ ```sh
34
+ bundle install
35
+ ```
36
+
37
+ - Run tests
38
+
39
+ ```sh
40
+ bundle exec rspec
41
+ ```
42
+
43
+ ### 2. Running your unpublished Plugin in Logstash
44
+
45
+ #### 2.1 Run in a local Logstash clone
46
+
47
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
48
+ ```ruby
49
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
50
+ ```
51
+ - Install plugin
52
+ ```sh
53
+ bin/logstash-plugin install --no-verify
54
+ ```
55
+ - Run Logstash with your plugin
56
+ ```sh
57
+ bin/logstash -e 'filter {awesome {}}'
58
+ ```
59
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
60
+
61
+ #### 2.2 Run in an installed Logstash
62
+
63
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
64
+
65
+ - Build your plugin gem
66
+ ```sh
67
+ gem build logstash-filter-awesome.gemspec
68
+ ```
69
+ - Install the plugin from the Logstash home
70
+ ```sh
71
+ bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
72
+ ```
73
+ - Start Logstash and proceed to test the plugin
74
+
75
+ ## Contributing
76
+
77
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
78
+
79
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
80
+
81
+ It is more important to the community that you are able to contribute.
82
+
83
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,77 @@
1
+ # encoding: utf-8
2
+ require "logstash/filters/base"
3
+ require "logstash/namespace"
4
+ require "lru_redux"
5
+
6
+ # Workaround to JRuby 1.7 bug that will never be fixed https://github.com/jruby/jruby/issues/3920
7
+ module Aws; module DynamoDB; end; end
8
+ require 'aws-sdk'
9
+ require 'aws-sdk-core/dynamodb'
10
+
11
+ class LogStash::Filters::DynamoEnrich < LogStash::Filters::Base
12
+ config_name "dynamo_enrich"
13
+
14
+ config :table_name, :validate => :string, :required => true
15
+ config :region, :validate => :string, :default => ""
16
+ config :primary_key_name, :validate => :string, :required => true
17
+ config :primary_key_value, :validate => :string, :required => true
18
+ config :return_attribute, :validate => :string, :required => true
19
+ config :target, :validate => :string, :required => true
20
+ config :enable_cache, :validate => :boolean, :default => true
21
+ config :cache_ttl, :validate => :number, :default => 300
22
+ config :cache_size, :validate => :number, :default => 1024
23
+
24
+ public
25
+ def register
26
+ region_set = @region != ""
27
+ if region_set
28
+ @client = Aws::DynamoDB::Client.new(region: @region)
29
+ else
30
+ @client = Aws::DynamoDB::Client.new()
31
+ end
32
+
33
+ if @enable_cache
34
+ @cache = LruRedux::TTL::ThreadSafeCache.new(@cache_size, @cache_ttl)
35
+ end
36
+ end # def register
37
+
38
+ public
39
+ def filter(event)
40
+ primary_key_value = event.get(@primary_key_value)
41
+ if @enable_cache && @cache.key?(primary_key_value)
42
+ cached_value = @cache[primary_key_value]
43
+ if cached_value
44
+ event.set(@target, ::LogStash::Util.deep_clone(cached_value))
45
+ else
46
+ event.tag("_dynamoenrichitemnotfound")
47
+ end
48
+ else
49
+ lookup_value = fetch_dynamo(primary_key_value)
50
+ @cache.getset(primary_key_value){lookup_value} if @enable_cache
51
+
52
+ if lookup_value
53
+ event.set(@target, ::LogStash::Util.deep_clone(lookup_value))
54
+ else
55
+ event.tag("_dynamoenrichitemnotfound")
56
+ end
57
+ end
58
+ filter_matched(event)
59
+ rescue Aws::DynamoDB::Errors::ServiceError => e
60
+ @logger.warn("DynamoDB Service Exception", :exception => e, :event => event)
61
+ event.tag("_dynamoenrichserviceerror")
62
+ end # def filter
63
+
64
+ def fetch_dynamo(primary_key_value)
65
+ resp = @client.get_item({
66
+ key: {
67
+ @primary_key_name => primary_key_value,
68
+ },
69
+ table_name: @table_name,
70
+ })
71
+
72
+ if resp.item && !resp.item.empty?
73
+ return resp.item[@return_attribute]
74
+ end
75
+ # return false
76
+ end
77
+ end # class LogStash::Filters::DynamoEnrich
@@ -0,0 +1,25 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-filter-dynamo_enrich'
3
+ s.version = '0.1.0'
4
+ s.licenses = ['Apache License (2.0)']
5
+ s.summary = 'Enrich Logstash events with data from DynamoDB'
6
+ s.description = 'Query data from DynamoDB and populate Logstash events with the returned values'
7
+ s.homepage = 'https://github.com/envato/logstash-filter-dynamo_enrich'
8
+ s.authors = ['nemski']
9
+ s.email = 'patrick.robinson@envato.com'
10
+ s.require_paths = ['lib']
11
+
12
+ # Files
13
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
14
+ # Tests
15
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
16
+
17
+ # Special flag to let us know this is actually a logstash plugin
18
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
19
+
20
+ # Gem dependencies
21
+ s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
22
+ s.add_runtime_dependency "aws-sdk", "~> 2.3"
23
+ s.add_runtime_dependency "lru_redux", "~> 1.0"
24
+ s.add_development_dependency 'logstash-devutils'
25
+ end
@@ -0,0 +1,122 @@
1
+ # encoding: utf-8
2
+ require_relative '../spec_helper'
3
+ require "logstash/filters/dynamo_enrich"
4
+
5
+ describe LogStash::Filters::DynamoEnrich do
6
+ describe "Lookup Table" do
7
+ let(:config) do <<-CONFIG
8
+ filter {
9
+ dynamo_enrich {
10
+ table_name => "id_lookup"
11
+ region => "us-east-1"
12
+ primary_key_name => "id"
13
+ primary_key_value => "user_id"
14
+ return_attribute => "name"
15
+ target => "user_name"
16
+ enable_cache => false
17
+ }
18
+ }
19
+ CONFIG
20
+ end
21
+ let(:dynamo_client) { instance_double(Aws::DynamoDB::Client) }
22
+ let(:dynamo_response) { instance_double(Aws::DynamoDB::Types::GetItemOutput) }
23
+
24
+ before do
25
+ allow(Aws::DynamoDB::Client).to receive(:new).and_return(dynamo_client)
26
+ allow(dynamo_response).to receive(:item).and_return(item)
27
+ end
28
+
29
+ context "Item exists" do
30
+ let(:item) { {"name" => "Jane Doe"} }
31
+
32
+ before do
33
+ allow(dynamo_client).to receive(:get_item).and_return(dynamo_response)
34
+ end
35
+
36
+ sample("user_id" => "123") do
37
+ expect(subject).to include("user_name")
38
+ expect(subject.get('user_name')).to eq('Jane Doe')
39
+ end
40
+ end
41
+
42
+ context "Item doesn't exist" do
43
+ let(:item) { {} }
44
+
45
+ before do
46
+ allow(dynamo_client).to receive(:get_item).and_return(dynamo_response)
47
+ end
48
+
49
+ sample("user_id" => "123") do
50
+ expect(subject.get('tags')).to include('_dynamoenrichitemnotfound')
51
+ end
52
+ end
53
+
54
+ context "Dynamo returns a service error" do
55
+ let(:item) { {} }
56
+
57
+ before do
58
+ expect(dynamo_client).to receive(:get_item).and_raise(Aws::DynamoDB::Errors::ServiceError.new(Seahorse::Client::RequestContext.new, "Internal Error"))
59
+ end
60
+
61
+ sample("user_id" => "123") do
62
+ expect(subject.get('tags')).to include('_dynamoenrichserviceerror')
63
+ end
64
+ end
65
+ end
66
+
67
+ describe "Cached result" do
68
+ let(:config) do <<-CONFIG
69
+ filter {
70
+ dynamo_enrich {
71
+ table_name => "id_lookup"
72
+ region => "us-east-1"
73
+ primary_key_name => "id"
74
+ primary_key_value => "user_id"
75
+ return_attribute => "name"
76
+ target => "user_name"
77
+ }
78
+ }
79
+ CONFIG
80
+ end
81
+ let(:dynamo_client) { instance_double(Aws::DynamoDB::Client) }
82
+ let(:dynamo_response) { instance_double(Aws::DynamoDB::Types::GetItemOutput) }
83
+ let(:cache_double) { instance_double(LruRedux::TTL::ThreadSafeCache) }
84
+
85
+ before do
86
+ allow(Aws::DynamoDB::Client).to receive(:new).and_return(dynamo_client)
87
+ allow(LruRedux::TTL::ThreadSafeCache).to receive(:new).and_return(cache_double)
88
+ end
89
+
90
+ context "Item is cached" do
91
+ before do
92
+ allow(cache_double).to receive(:key?).with("123").and_return(true)
93
+ allow(cache_double).to receive(:[]).with("123").and_return("Jane Doe")
94
+ end
95
+
96
+ sample("user_id" => "123") do
97
+ expect(dynamo_client).not_to receive(:get_item)
98
+ expect(subject).to include("user_name")
99
+ expect(subject.get('user_name')).to eq('Jane Doe')
100
+ end
101
+ end
102
+
103
+ context "Item is not cached" do
104
+ let(:item) { {} }
105
+
106
+ before do
107
+ allow(cache_double).to receive(:key?).with("123").and_return(false)
108
+ allow(cache_double).to receive(:getset)
109
+ allow(dynamo_response).to receive(:item).and_return(item)
110
+ allow(dynamo_client).to receive(:get_item).and_return(dynamo_response)
111
+ end
112
+
113
+ sample("user_id" => "123") do
114
+ expect(cache_double).not_to receive(:[])
115
+ expect(dynamo_client).to receive(:get_item)
116
+
117
+ subject
118
+ end
119
+ end
120
+ end
121
+
122
+ end
@@ -0,0 +1,2 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
metadata ADDED
@@ -0,0 +1,112 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-filter-dynamo_enrich
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - nemski
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2017-12-19 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - "~>"
17
+ - !ruby/object:Gem::Version
18
+ version: '2.0'
19
+ name: logstash-core-plugin-api
20
+ prerelease: false
21
+ type: :runtime
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '2.0'
27
+ - !ruby/object:Gem::Dependency
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - "~>"
31
+ - !ruby/object:Gem::Version
32
+ version: '2.3'
33
+ name: aws-sdk
34
+ prerelease: false
35
+ type: :runtime
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '2.3'
41
+ - !ruby/object:Gem::Dependency
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - "~>"
45
+ - !ruby/object:Gem::Version
46
+ version: '1.0'
47
+ name: lru_redux
48
+ prerelease: false
49
+ type: :runtime
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '1.0'
55
+ - !ruby/object:Gem::Dependency
56
+ requirement: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - ">="
59
+ - !ruby/object:Gem::Version
60
+ version: '0'
61
+ name: logstash-devutils
62
+ prerelease: false
63
+ type: :development
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ description: Query data from DynamoDB and populate Logstash events with the returned values
70
+ email: patrick.robinson@envato.com
71
+ executables: []
72
+ extensions: []
73
+ extra_rdoc_files: []
74
+ files:
75
+ - CHANGELOG.md
76
+ - CONTRIBUTORS
77
+ - Gemfile
78
+ - LICENSE
79
+ - README.md
80
+ - lib/logstash/filters/dynamo_enrich.rb
81
+ - logstash-filter-dynamo_enrich.gemspec
82
+ - spec/filters/dynamo_enrich_spec.rb
83
+ - spec/spec_helper.rb
84
+ homepage: https://github.com/envato/logstash-filter-dynamo_enrich
85
+ licenses:
86
+ - Apache License (2.0)
87
+ metadata:
88
+ logstash_plugin: 'true'
89
+ logstash_group: filter
90
+ post_install_message:
91
+ rdoc_options: []
92
+ require_paths:
93
+ - lib
94
+ required_ruby_version: !ruby/object:Gem::Requirement
95
+ requirements:
96
+ - - ">="
97
+ - !ruby/object:Gem::Version
98
+ version: '0'
99
+ required_rubygems_version: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ requirements: []
105
+ rubyforge_project:
106
+ rubygems_version: 2.4.8
107
+ signing_key:
108
+ specification_version: 4
109
+ summary: Enrich Logstash events with data from DynamoDB
110
+ test_files:
111
+ - spec/filters/dynamo_enrich_spec.rb
112
+ - spec/spec_helper.rb