logstash-input-azuretopicthreadable 0.9.4 → 0.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +4 -0
  3. data/Gemfile +0 -1
  4. data/README.md +60 -91
  5. data/lib/logstash/inputs/azuretopicthreadable.rb +77 -78
  6. data/logstash-input-azuretopicthreadable.gemspec +6 -6
  7. metadata +25 -32
  8. data/Rakefile +0 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: eab44087211786edf86b50c29a10ad7624e0bc05
4
- data.tar.gz: 080965dacdf000fc5447e1c36f2c4c981dd051a2
3
+ metadata.gz: fecf30d90dc023f7d2c2349fcbc64c4dc70c57eb
4
+ data.tar.gz: d762ad0321bba4b9698c57cf34f0fed1f8ee3391
5
5
  SHA512:
6
- metadata.gz: 22aa9895e67c75d0e6560114910e69ced23fc8a33f73f17af4e253ed562b34b2182bf218b25b1e6dbf4c1a27e3063008459d8cfb40de441a76cd26abe7a9cccb
7
- data.tar.gz: 570286a844078687db1a1a06495bf5994f0e9b55b6a2ae5d6e755b0cbb199e50c4c93dd041c0822032caf55e36d893be8e320839361eb51f391f53e41c6988f7
6
+ metadata.gz: f0000ca97d12e22de2faf2409f7dd3c3df38c42e43a1e893056fd2aa6f2ce95172e48cee4d7e7c90c9df6c69e0901a70341d0b1858166460fb566cfc2cc64214
7
+ data.tar.gz: 1d9438970c38b766a60858bce0ca5a6f912d46b00b7d27be489908c04603cbcb9cbf1331945f462b9086e9a7361a1fa061b76886a1e7c2a1dcabaa733bba1402
data/CHANGELOG.md CHANGED
@@ -0,0 +1,4 @@
1
+ ## 2016.05.03
2
+ * Made the plugin to respect Logstash shutdown signal.
3
+ * Updated the *logstash-core* runtime dependency requirement to '~> 2.0'.
4
+ * Updated the *logstash-devutils* development dependency requirement to '>= 0.0.16'
data/Gemfile CHANGED
@@ -1,3 +1,2 @@
1
1
  source 'https://rubygems.org'
2
2
  gemspec
3
- gem 'azure', '~> 0.7.1'
data/README.md CHANGED
@@ -1,91 +1,60 @@
1
- # Notice
2
- This plugin is a part of [Microsoft Azure Diagnostics with ELK](https://github.com/mspnp/semantic-logging/tree/v3/ELK).
3
-
4
- [See more documentation.](https://github.com/mspnp/semantic-logging/blob/v3/ELK/md/LogstashExtensions.md#azure-wad-table)
5
-
6
- # Logstash Plugin
7
-
8
- This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
9
-
10
- It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
11
-
12
- ## Documentation
13
-
14
- Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
15
-
16
- - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
17
- - For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
18
-
19
- ## Need Help?
20
-
21
- Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
22
-
23
- ## Developing
24
-
25
- ### 1. Plugin Developement and Testing
26
-
27
- #### Code
28
- - To get started, you'll need JRuby with the Bundler gem installed.
29
-
30
- - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
31
-
32
- - Install dependencies
33
- ```sh
34
- bundle install
35
- ```
36
-
37
- #### Test
38
-
39
- - Update your dependencies
40
-
41
- ```sh
42
- bundle install
43
- ```
44
-
45
- - Run tests
46
-
47
- ```sh
48
- bundle exec rspec
49
- ```
50
-
51
- ### 2. Running your unpublished Plugin in Logstash
52
-
53
- #### 2.1 Run in a local Logstash clone
54
-
55
- - Edit Logstash `Gemfile` and add the local plugin path, for example:
56
- ```ruby
57
- gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
58
- ```
59
- - Install plugin
60
- ```sh
61
- bin/plugin install --no-verify
62
- ```
63
- - Run Logstash with your plugin
64
- ```sh
65
- bin/logstash -e 'filter {awesome {}}'
66
- ```
67
- At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
68
-
69
- #### 2.2 Run in an installed Logstash
70
-
71
- You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
72
-
73
- - Build your plugin gem
74
- ```sh
75
- gem build logstash-filter-awesome.gemspec
76
- ```
77
- - Install the plugin from the Logstash home
78
- ```sh
79
- bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
80
- ```
81
- - Start Logstash and proceed to test the plugin
82
-
83
- ## Contributing
84
-
85
- All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
86
-
87
- Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
88
-
89
- It is more important to the community that you are able to contribute.
90
-
91
- For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
1
+ # Logstash input plugin for Azure Service Bus Topics
2
+
3
+ ## Summary
4
+ This plugin reads messages from Azure Service Bus Topics using multiple threads.
5
+
6
+ ## Installation
7
+ You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
8
+ ```sh
9
+ logstash-plugin install logstash-input-azuretopicthreadable
10
+ ```
11
+ For more information, see Logstash reference [Working with plugins](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html).
12
+
13
+ ## Configuration
14
+ ### Required Parameters
15
+ __*namespace*__
16
+
17
+ The Service Bus namespace.
18
+
19
+ __*access_key*__
20
+
21
+ The access key to the Service Bus resource.
22
+
23
+ __*subscription*__
24
+
25
+ The name of the Topic Subscription.
26
+
27
+ __*topic*__
28
+
29
+ The name of the Topic.
30
+
31
+ ### Optional Parameters
32
+ __*deliverycount*__
33
+
34
+ Specifies the number of times to try (and retry) to process a message before the message shall be deleted. The default value is 10.
35
+
36
+ __*threads*__
37
+
38
+ Specifies the number of threads to use to read the messages. The default value is 1.
39
+
40
+ __*thread_sleep_time*__
41
+
42
+ Specifies the number of seconds each thread should sleep before starting another loop of processing. The default value is 1/50.
43
+
44
+ ### Examples
45
+ ```
46
+ input
47
+ {
48
+ azuretopic
49
+ {
50
+ namespace => "mysbns"
51
+ access_key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
52
+ subscription => "mytopicsubscription"
53
+ topic => "mytopic"
54
+ threads => 3
55
+ }
56
+ }
57
+ ```
58
+
59
+ ## More information
60
+ The source code of this plugin is hosted in GitHub repo [Microsoft Azure Diagnostics with ELK](https://github.com/Azure/azure-diagnostics-tools). We welcome you to provide feedback and/or contribute to the project.
data/lib/logstash/inputs/azuretopicthreadable.rb CHANGED
@@ -1,78 +1,77 @@
1
- # encoding: utf-8
2
- require "logstash/inputs/base"
3
- require "logstash/namespace"
4
-
5
- require "thread"
6
-
7
- require "azure"
8
-
9
- # Reads events from Azure topics
10
- class LogStash::Inputs::Azuretopicthreadable < LogStash::Inputs::Base
11
- class Interrupted < StandardError; end
12
-
13
- config_name "azuretopicthreadable"
14
- milestone 0
15
-
16
- default :codec, "json" # default json codec
17
-
18
- config :namespace, :validate => :string
19
- config :access_key, :validate => :string
20
- config :subscription, :validate => :string
21
- config :topic, :validate => :string
22
- config :deliverycount, :validate => :number, :default => 10
23
- config :threads, :validate => :number, :default => 1
24
- config :thread_sleep_time, :validate => :number, :default => 1.0/50.0
25
-
26
- def initialize(*args)
27
- super(*args)
28
- end # def initialize
29
-
30
- public
31
- def register
32
- # Configure credentials
33
- Azure.configure do |config|
34
- config.sb_namespace = @namespace
35
- config.sb_access_key = @access_key
36
- end
37
- end # def register
38
-
39
- def process(output_queue, pid)
40
- # Get a new instance of a service
41
- azure_service_bus = Azure::ServiceBus::ServiceBusService.new
42
- while true
43
- begin
44
- # check if we have a message in the subscription
45
- message = azure_service_bus.receive_subscription_message(@topic ,@subscription, { :peek_lock => true, :timeout => 1 } )
46
- if message
47
- # decoding returns a yield
48
- codec.decode(message.body) do |event|
49
- output_queue << event
50
- end # codec.decode
51
- # delete the message after reading it
52
- azure_service_bus.delete_subscription_message(message)
53
- end
54
- rescue LogStash::ShutdownSignal => e
55
- raise e
56
- rescue => e
57
- @logger.error("Oh My, An error occurred. Thread id:" + pid.to_s, :exception => e)
58
- if message and message.delivery_count > @deliverycount
59
- azure_service_bus.delete_subscription_message(message)
60
- end
61
- end
62
- sleep(@thread_sleep_time)
63
- end
64
- end # def process
65
-
66
- public
67
- def run(output_queue)
68
- threads = []
69
- (0..(@threads-1)).each do |pid|
70
- threads << Thread.new { process(output_queue, pid) }
71
- end
72
- threads.each { |thr| thr.join }
73
- end # def run
74
-
75
- public
76
- def teardown
77
- end # def teardown
78
- end # class LogStash::Inputs::Azuretopic
1
+ # encoding: utf-8
2
+ require "logstash/inputs/base"
3
+ require "logstash/namespace"
4
+
5
+ require "thread"
6
+ require "azure"
7
+
8
+ # Reads events from Azure topics
9
+ class LogStash::Inputs::Azuretopicthreadable < LogStash::Inputs::Base
10
+ class Interrupted < StandardError; end
11
+
12
+ config_name "azuretopicthreadable"
13
+ milestone 1
14
+
15
+ default :codec, "json" # default json codec
16
+
17
+ config :namespace, :validate => :string
18
+ config :access_key, :validate => :string
19
+ config :subscription, :validate => :string
20
+ config :topic, :validate => :string
21
+ config :deliverycount, :validate => :number, :default => 10
22
+ config :threads, :validate => :number, :default => 1
23
+ config :thread_sleep_time, :validate => :number, :default => 1.0/50.0
24
+
25
+ def initialize(*args)
26
+ super(*args)
27
+ end # def initialize
28
+
29
+ public
30
+ def register
31
+ # Configure credentials
32
+ Azure.configure do |config|
33
+ config.sb_namespace = @namespace
34
+ config.sb_access_key = @access_key
35
+ end
36
+ end # def register
37
+
38
+ def process(output_queue, pid)
39
+ # Get a new instance of a service
40
+ azure_service_bus = Azure::ServiceBus::ServiceBusService.new
41
+ while !stop?
42
+ begin
43
+ # check if we have a message in the subscription
44
+ message = azure_service_bus.receive_subscription_message(@topic ,@subscription, { :peek_lock => true, :timeout => 1 } )
45
+ if message
46
+ # decoding returns a yield
47
+ codec.decode(message.body) do |event|
48
+ output_queue << event
49
+ end # codec.decode
50
+ # delete the message after reading it
51
+ azure_service_bus.delete_subscription_message(message)
52
+ end
53
+ rescue LogStash::ShutdownSignal => e
54
+ raise e
55
+ rescue => e
56
+ @logger.error("Oh My, An error occurred. Thread id:" + pid.to_s, :exception => e)
57
+ if message and message.delivery_count > @deliverycount
58
+ azure_service_bus.delete_subscription_message(message)
59
+ end
60
+ end
61
+ sleep(@thread_sleep_time)
62
+ end
63
+ end # def process
64
+
65
+ public
66
+ def run(output_queue)
67
+ threads = []
68
+ (0..(@threads-1)).each do |pid|
69
+ threads << Thread.new { process(output_queue, pid) }
70
+ end
71
+ threads.each { |thr| thr.join }
72
+ end # def run
73
+
74
+ public
75
+ def teardown
76
+ end # def teardown
77
+ end # class LogStash::Inputs::Azuretopic
data/logstash-input-azuretopicthreadable.gemspec CHANGED
@@ -1,16 +1,16 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-input-azuretopicthreadable'
3
- s.version = '0.9.4'
3
+ s.version = '0.9.5'
4
4
  s.licenses = ['Apache License (2.0)']
5
- s.summary = "This plugin will collect Microsoft Azure Diagnostics data from Azure Service Bus."
6
- s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program."
5
+ s.summary = "This plugin reads messages from Azure Service Bus Topics."
6
+ s.description = "This gem is a Logstash plugin. It reads messages from Azure Service Bus Topics using multiple threads."
7
7
  s.authors = ["Microsoft Corporation"]
8
8
  s.email = 'azdiag@microsoft.com'
9
9
  s.homepage = "https://github.com/Azure/azure-diagnostics-tools"
10
10
  s.require_paths = ["lib"]
11
11
 
12
12
  # Files
13
- s.files = `git ls-files`.split($\)
13
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','Gemfile','LICENSE']
14
14
  # Tests
15
15
  s.test_files = s.files.grep(%r{^(test|spec|features)/})
16
16
 
@@ -18,7 +18,7 @@ Gem::Specification.new do |s|
18
18
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
19
19
 
20
20
  # Gem dependencies
21
- s.add_runtime_dependency 'logstash-core', '>= 1.4.0', '< 2.0.0'
21
+ s.add_runtime_dependency 'logstash-core', '~> 2.0'
22
22
  s.add_runtime_dependency 'azure', '~> 0.7.1'
23
- s.add_development_dependency 'logstash-devutils'
23
+ s.add_development_dependency 'logstash-devutils', '>= 0.0.16'
24
24
  end
metadata CHANGED
@@ -1,64 +1,58 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-azuretopicthreadable
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.9.4
4
+ version: 0.9.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Microsoft Corporation
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2016-03-09 00:00:00.000000000 Z
11
+ date: 2016-05-05 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
- requirement: !ruby/object:Gem::Requirement
14
+ name: logstash-core
15
+ version_requirements: !ruby/object:Gem::Requirement
15
16
  requirements:
16
- - - '>='
17
+ - - "~>"
17
18
  - !ruby/object:Gem::Version
18
- version: 1.4.0
19
- - - <
19
+ version: '2.0'
20
+ requirement: !ruby/object:Gem::Requirement
21
+ requirements:
22
+ - - "~>"
20
23
  - !ruby/object:Gem::Version
21
- version: 2.0.0
22
- name: logstash-core
24
+ version: '2.0'
23
25
  prerelease: false
24
26
  type: :runtime
27
+ - !ruby/object:Gem::Dependency
28
+ name: azure
25
29
  version_requirements: !ruby/object:Gem::Requirement
26
30
  requirements:
27
- - - '>='
31
+ - - "~>"
28
32
  - !ruby/object:Gem::Version
29
- version: 1.4.0
30
- - - <
31
- - !ruby/object:Gem::Version
32
- version: 2.0.0
33
- - !ruby/object:Gem::Dependency
33
+ version: 0.7.1
34
34
  requirement: !ruby/object:Gem::Requirement
35
35
  requirements:
36
- - - ~>
36
+ - - "~>"
37
37
  - !ruby/object:Gem::Version
38
38
  version: 0.7.1
39
- name: azure
40
39
  prerelease: false
41
40
  type: :runtime
41
+ - !ruby/object:Gem::Dependency
42
+ name: logstash-devutils
42
43
  version_requirements: !ruby/object:Gem::Requirement
43
44
  requirements:
44
- - - ~>
45
+ - - ">="
45
46
  - !ruby/object:Gem::Version
46
- version: 0.7.1
47
- - !ruby/object:Gem::Dependency
47
+ version: 0.0.16
48
48
  requirement: !ruby/object:Gem::Requirement
49
49
  requirements:
50
- - - '>='
50
+ - - ">="
51
51
  - !ruby/object:Gem::Version
52
- version: '0'
53
- name: logstash-devutils
52
+ version: 0.0.16
54
53
  prerelease: false
55
54
  type: :development
56
- version_requirements: !ruby/object:Gem::Requirement
57
- requirements:
58
- - - '>='
59
- - !ruby/object:Gem::Version
60
- version: '0'
61
- description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program.
55
+ description: This gem is a Logstash plugin. It reads messages from Azure Service Bus Topics using multiple threads.
62
56
  email: azdiag@microsoft.com
63
57
  executables: []
64
58
  extensions: []
@@ -68,7 +62,6 @@ files:
68
62
  - Gemfile
69
63
  - LICENSE
70
64
  - README.md
71
- - Rakefile
72
65
  - lib/logstash/inputs/azuretopicthreadable.rb
73
66
  - logstash-input-azuretopicthreadable.gemspec
74
67
  - spec/inputs/azuretopicthreadable_spec.rb
@@ -84,12 +77,12 @@ require_paths:
84
77
  - lib
85
78
  required_ruby_version: !ruby/object:Gem::Requirement
86
79
  requirements:
87
- - - '>='
80
+ - - ">="
88
81
  - !ruby/object:Gem::Version
89
82
  version: '0'
90
83
  required_rubygems_version: !ruby/object:Gem::Requirement
91
84
  requirements:
92
- - - '>='
85
+ - - ">="
93
86
  - !ruby/object:Gem::Version
94
87
  version: '0'
95
88
  requirements: []
@@ -97,6 +90,6 @@ rubyforge_project:
97
90
  rubygems_version: 2.4.8
98
91
  signing_key:
99
92
  specification_version: 4
100
- summary: This plugin will collect Microsoft Azure Diagnostics data from Azure Service Bus.
93
+ summary: This plugin reads messages from Azure Service Bus Topics.
101
94
  test_files:
102
95
  - spec/inputs/azuretopicthreadable_spec.rb
data/Rakefile DELETED
@@ -1 +0,0 @@
1
- require "logstash/devutils/rake"