logstash-input-azurewadeventhub 0.9.5-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 61414c27c9a429ced5bbf1df8a0321542d308644
4
+ data.tar.gz: 0aaf66f20d863cf65bdce23b3c8c8c9ca76d06b4
5
+ SHA512:
6
+ metadata.gz: 15463ebe49eafd7edbd436761b9158dce3013a1f27a06880e1cdbfa4d81ca5aaf347f6f89cc829f66947e821281bd16b4ee585e89b1c8ef528e5d943d786e8eb
7
+ data.tar.gz: 6c31e0c73b0ef5605019a8f60fb08d060d1682227b5d15a5a6548287366ad4b6dee17a2b9a307c6220ec2e53c3d947c9377899bbc91afcfe3f6836c9a6e07eee
data/CHANGELOG.md ADDED
File without changes
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
3
+ gem 'azure', '~> 0.7.1'
data/LICENSE ADDED
@@ -0,0 +1,17 @@
1
+
2
+ Copyright (c) Microsoft. All rights reserved.
3
+ Microsoft would like to thank its contributors, a list
4
+ of whom are at http://aka.ms/entlib-contributors
5
+
6
+ Licensed under the Apache License, Version 2.0 (the "License"); you
7
+ may not use this file except in compliance with the License. You may
8
+ obtain a copy of the License at
9
+
10
+ http://www.apache.org/licenses/LICENSE-2.0
11
+
12
+ Unless required by applicable law or agreed to in writing, software
13
+ distributed under the License is distributed on an "AS IS" BASIS,
14
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
15
+ implied. See the License for the specific language governing permissions
16
+ and limitations under the License.
17
+
data/README.md ADDED
@@ -0,0 +1,91 @@
1
+ # Notice
2
+ This plugin is a part of [Microsoft Azure Diagnostics with ELK](https://github.com/Azure/azure-diagnostics-tools).
3
+
4
+ [See more documentation.](https://github.com/mspnp/semantic-logging/blob/v3/ELK/md/LogstashExtensions.md#azure-wad-table)
5
+
6
+ # Logstash Plugin
7
+
8
+ This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
9
+
10
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
11
+
12
+ ## Documentation
13
+
14
+ Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
15
+
16
+ - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
17
+ - For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
18
+
19
+ ## Need Help?
20
+
21
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
22
+
23
+ ## Developing
24
+
25
+ ### 1. Plugin Developement and Testing
26
+
27
+ #### Code
28
+ - To get started, you'll need JRuby with the Bundler gem installed.
29
+
30
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
31
+
32
+ - Install dependencies
33
+ ```sh
34
+ bundle install
35
+ ```
36
+
37
+ #### Test
38
+
39
+ - Update your dependencies
40
+
41
+ ```sh
42
+ bundle install
43
+ ```
44
+
45
+ - Run tests
46
+
47
+ ```sh
48
+ bundle exec rspec
49
+ ```
50
+
51
+ ### 2. Running your unpublished Plugin in Logstash
52
+
53
+ #### 2.1 Run in a local Logstash clone
54
+
55
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
56
+ ```ruby
57
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
58
+ ```
59
+ - Install plugin
60
+ ```sh
61
+ bin/plugin install --no-verify
62
+ ```
63
+ - Run Logstash with your plugin
64
+ ```sh
65
+ bin/logstash -e 'filter {awesome {}}'
66
+ ```
67
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
68
+
69
+ #### 2.2 Run in an installed Logstash
70
+
71
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
72
+
73
+ - Build your plugin gem
74
+ ```sh
75
+ gem build logstash-filter-awesome.gemspec
76
+ ```
77
+ - Install the plugin from the Logstash home
78
+ ```sh
79
+ bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
80
+ ```
81
+ - Start Logstash and proceed to test the plugin
82
+
83
+ ## Contributing
84
+
85
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
86
+
87
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
88
+
89
+ It is more important to the community that you are able to contribute.
90
+
91
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
data/Rakefile ADDED
@@ -0,0 +1 @@
1
+ require "logstash/devutils/rake"
@@ -0,0 +1,187 @@
1
+ # encoding: utf-8
2
+ require "logstash/inputs/base"
3
+ require "logstash/namespace"
4
+
5
+ require "securerandom"
6
+ require "open-uri"
7
+ require "thread"
8
+
9
+ require "geronimo-jms_1.1_spec-1.1.1.jar"
10
+ require "qpid-amqp-1-0-client-0.32.jar"
11
+ require "qpid-amqp-1-0-client-jms-0.32.jar"
12
+ require "qpid-amqp-1-0-common-0.32.jar"
13
+
14
+
15
+ # Reads events from Azure event-hub for Windows Azure Diagnostics
16
+ class LogStash::Inputs::Azurewadeventhub < LogStash::Inputs::Base
17
+
18
+ config_name "azurewadeventhub"
19
+ milestone 0
20
+
21
+ default :codec, "json"
22
+
23
+ config :key, :validate => :string
24
+ config :username, :validate => :string
25
+ config :namespace, :validate => :string
26
+ config :domain, :validate => :string, :default => "servicebus.windows.net"
27
+ config :port, :validate => :number, :default => 5671
28
+ config :receive_credits, :validate => :number, :default => 1000
29
+
30
+ config :eventhub, :validate => :string
31
+ config :partitions, :validate => :number
32
+ config :consumer_group, :validate => :string, :default => "$default"
33
+
34
+ config :time_since_epoch_millis, :validate => :number, :default => Time.now.to_i * 1000
35
+ config :thread_wait_sec, :validate => :number, :default => 5
36
+
37
+
38
+ def initialize(*args)
39
+ super(*args)
40
+ end # def initialize
41
+
42
+ public
43
+ def register
44
+ end # def register
45
+
46
+ def get_pay_load(message, partition)
47
+ return nil if not message
48
+ message.getPayload().each do |section|
49
+ if section.java_kind_of? org::apache::qpid::amqp_1_0::type::messaging::Data
50
+ data = ""
51
+ begin
52
+ event = LogStash::Event.new()
53
+ section.getValue().getArray().each do |byte|
54
+ data = data + byte.chr
55
+ end
56
+ json = JSON.parse(data)
57
+ # Check if the records field is there. All messages written by WAD should have
58
+ # "records" as the root element
59
+ if !json["records"].nil?
60
+ recordArray = json["records"]
61
+ recordArray.each do |record|
62
+ record.each do |name, value|
63
+ event[name] = value
64
+ end
65
+ end
66
+ end
67
+ return event
68
+ rescue => e
69
+ if data != ""
70
+ @logger.error(" " + partition.to_s.rjust(2,"0") + " --- " + "Error: Unable to JSON parse '" + data + "'.", :exception => e)
71
+ else
72
+ @logger.error(" " + partition.to_s.rjust(2,"0") + " --- " + "Error: Unable to get the message body for message", :exception => e)
73
+ end
74
+ end
75
+ end
76
+ end
77
+ return nil
78
+ end
79
+
80
+ def process(output_queue, receiver, partition)
81
+ while true
82
+ begin
83
+ msg = receiver.receive(10)
84
+ if msg
85
+ event = get_pay_load(msg, partition)
86
+ if event
87
+ output_queue << event
88
+ end
89
+ receiver.acknowledge(msg)
90
+ else
91
+ @logger.debug(" " + partition.to_s.rjust(2,"0") + " --- " + "No message")
92
+ sleep(@thread_wait_sec)
93
+ end
94
+ rescue LogStash::ShutdownSignal => e
95
+ raise e
96
+ rescue org::apache::qpid::amqp_1_0::client::ConnectionErrorException => e
97
+ raise e
98
+ rescue => e
99
+ @logger.error(" " + partition.to_s.rjust(2,"0") + " --- " + "Oh My, An error occurred.", :exception => e)
100
+ end
101
+ end # process
102
+ end # process
103
+
104
+ def process_partition(output_queue, partition)
105
+ while true
106
+ begin
107
+ filter = SelectorFilter.new "amqp.annotation.x-opt-enqueuedtimeutc > '" + @time_since_epoch_millis.to_s + "'"
108
+ filters = { org::apache::qpid::amqp_1_0::type::Symbol.valueOf("apache.org:selector-filter:string") => filter }
109
+ host = @namespace + "." + @domain
110
+ connection = org::apache::qpid::amqp_1_0::client::Connection.new(host, @port, @username, @key, host, true)
111
+ connection.getEndpoint().getDescribedTypeRegistry().register(filter.java_class, WriterFactory.new)
112
+ receiveSession = connection.createSession()
113
+ receiver = receiveSession.createReceiver(@eventhub + "/ConsumerGroups/" + @consumer_group + "/Partitions/" + partition.to_s, org::apache::qpid::amqp_1_0::client::AcknowledgeMode::ALO, "eventhubs-receiver-link-" + partition.to_s, false, filters, nil)
114
+ receiver.setCredit(org::apache::qpid::amqp_1_0::type::UnsignedInteger.valueOf(@receive_credits), true)
115
+ process(output_queue,receiver,partition)
116
+ rescue org::apache::qpid::amqp_1_0::client::ConnectionErrorException => e
117
+ @logger.debug(" " + partition.to_s.rjust(2,"0") + " --- " + "resetting connection")
118
+ @time_since_epoch_millis = Time.now.to_i * 1000
119
+ end
120
+ end
121
+ rescue LogStash::ShutdownSignal => e
122
+ receiver.close()
123
+ raise e
124
+ rescue => e
125
+ @logger.error(" " + partition.to_s.rjust(2,"0") + " --- Oh My, An error occurred.", :exception => e)
126
+ end # process
127
+
128
+ public
129
+ def run(output_queue)
130
+ threads = []
131
+ (0..(@partitions-1)).each do |p_id|
132
+ threads << Thread.new { process_partition(output_queue, p_id) }
133
+ end
134
+ threads.each { |thr| thr.join }
135
+ end # def run
136
+
137
+ public
138
+ def teardown
139
+ end # def teardown
140
+ end # class LogStash::Inputs::Azurewadeventhub
141
+
142
+
143
+ class SelectorFilter
144
+ include org::apache::qpid::amqp_1_0::type::messaging::Filter
145
+
146
+ def initialize(value)
147
+ @value = value
148
+ end
149
+
150
+ def getValue
151
+ return @value
152
+ end
153
+
154
+ def toString
155
+ return @value
156
+ end
157
+ end
158
+
159
+ class SelectorFilterWriter < org::apache::qpid::amqp_1_0::codec::AbstractDescribedTypeWriter
160
+ def initialize(registry)
161
+ super(registry)
162
+ end
163
+
164
+ def onSetValue(value)
165
+ @value = value
166
+ end
167
+
168
+ def clear
169
+ @value = nil
170
+ end
171
+
172
+ def getDescriptor
173
+ return org::apache::qpid::amqp_1_0::type::UnsignedLong.valueOf(83483426826);
174
+ end
175
+
176
+ def createDescribedWriter
177
+ return getRegistry().getValueWriter(@value.getValue());
178
+ end
179
+ end
180
+
181
+ class WriterFactory
182
+ include org::apache::qpid::amqp_1_0::codec::ValueWriter::Factory
183
+
184
+ def newInstance(registry)
185
+ return SelectorFilterWriter.new registry
186
+ end
187
+ end
@@ -0,0 +1,32 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-input-azurewadeventhub'
3
+ s.version = '0.9.5'
4
+ s.platform = "java"
5
+ s.licenses = ['Apache License (2.0)']
6
+ s.summary = "This plugin will collect Microsoft Azure Diagnostics data from Azure Event Hubs."
7
+ s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program."
8
+ s.authors = ["Microsoft Corporation"]
9
+ s.email = 'azdiag@microsoft.com'
10
+ s.homepage = "https://github.com/Azure/azure-diagnostics-tools"
11
+ s.require_paths = ["lib"]
12
+
13
+ # Files
14
+ s.files = `git ls-files`.split($\)
15
+ # Tests
16
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
17
+
18
+ # Special flag to let us know this is actually a logstash plugin
19
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
20
+
21
+ # Gem dependencies
22
+ s.add_runtime_dependency 'logstash-core', '>= 1.4.0'
23
+ s.add_runtime_dependency 'azure', '~> 0.7.1'
24
+ s.add_development_dependency 'logstash-devutils'
25
+
26
+ #Jar dependencies
27
+ s.requirements << "jar 'org.apache.qpid:qpid-amqp-1-0-common', '0.32'"
28
+ s.requirements << "jar 'org.apache.qpid:qpid-amqp-1-0-client-jms', '0.32'"
29
+ s.requirements << "jar 'org.apache.qpid:qpid-client', '0.32'"
30
+ s.requirements << "jar 'org.apache.geronimo.specs:geronimo-jms_1.1_spec', '1.1.1'"
31
+ s.add_runtime_dependency 'jar-dependencies'
32
+ end
@@ -0,0 +1 @@
1
+ require "logstash/devutils/rspec/spec_helper"
metadata ADDED
@@ -0,0 +1,114 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-input-azurewadeventhub
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.9.5
5
+ platform: java
6
+ authors:
7
+ - Microsoft Corporation
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2016-04-27 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - '>='
17
+ - !ruby/object:Gem::Version
18
+ version: 1.4.0
19
+ name: logstash-core
20
+ prerelease: false
21
+ type: :runtime
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - '>='
25
+ - !ruby/object:Gem::Version
26
+ version: 1.4.0
27
+ - !ruby/object:Gem::Dependency
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - ~>
31
+ - !ruby/object:Gem::Version
32
+ version: 0.7.1
33
+ name: azure
34
+ prerelease: false
35
+ type: :runtime
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ~>
39
+ - !ruby/object:Gem::Version
40
+ version: 0.7.1
41
+ - !ruby/object:Gem::Dependency
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - '>='
45
+ - !ruby/object:Gem::Version
46
+ version: '0'
47
+ name: logstash-devutils
48
+ prerelease: false
49
+ type: :development
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - '>='
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ requirement: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - '>='
59
+ - !ruby/object:Gem::Version
60
+ version: '0'
61
+ name: jar-dependencies
62
+ prerelease: false
63
+ type: :runtime
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - '>='
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program.
70
+ email: azdiag@microsoft.com
71
+ executables: []
72
+ extensions: []
73
+ extra_rdoc_files: []
74
+ files:
75
+ - CHANGELOG.md
76
+ - Gemfile
77
+ - LICENSE
78
+ - README.md
79
+ - Rakefile
80
+ - lib/logstash/inputs/azurewadeventhub.rb
81
+ - logstash-input-azurewadeventhub.gemspec
82
+ - spec/inputs/azurewadeventhub_spec.rb
83
+ homepage: https://github.com/Azure/azure-diagnostics-tools
84
+ licenses:
85
+ - Apache License (2.0)
86
+ metadata:
87
+ logstash_plugin: 'true'
88
+ logstash_group: input
89
+ post_install_message:
90
+ rdoc_options: []
91
+ require_paths:
92
+ - lib
93
+ required_ruby_version: !ruby/object:Gem::Requirement
94
+ requirements:
95
+ - - '>='
96
+ - !ruby/object:Gem::Version
97
+ version: '0'
98
+ required_rubygems_version: !ruby/object:Gem::Requirement
99
+ requirements:
100
+ - - '>='
101
+ - !ruby/object:Gem::Version
102
+ version: '0'
103
+ requirements:
104
+ - jar 'org.apache.qpid:qpid-amqp-1-0-common', '0.32'
105
+ - jar 'org.apache.qpid:qpid-amqp-1-0-client-jms', '0.32'
106
+ - jar 'org.apache.qpid:qpid-client', '0.32'
107
+ - jar 'org.apache.geronimo.specs:geronimo-jms_1.1_spec', '1.1.1'
108
+ rubyforge_project:
109
+ rubygems_version: 2.4.8
110
+ signing_key:
111
+ specification_version: 4
112
+ summary: This plugin will collect Microsoft Azure Diagnostics data from Azure Event Hubs.
113
+ test_files:
114
+ - spec/inputs/azurewadeventhub_spec.rb