logstash-input-google-cloud-pubsub 2.0.1

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: b5a77ddf0f2c116211a2efe4d50205cd37243f7c
4
+ data.tar.gz: 4d6833e88b18d401a7d6e65ec3e536214890696e
5
+ SHA512:
6
+ metadata.gz: c886650f9f12809879597bf77f9176685248945582e9bb7f5e6c810960b4075937e272ce4691ea107ac2f08e83b97ef98e69d050be784d93576f68c0d57b24d0
7
+ data.tar.gz: 3b8ca12dab7fd871291e4b7cd57c48e1f57abc8ca39c198c51fc458bb7f6ad75d8f788052528dc837d532144308930233b8668dba30040575ef9c25f9fec9b89
@@ -0,0 +1,9 @@
1
+ ## 2.0.4
2
+ - Add encoding: utf-8 line to spec file. This can prevent issues with tests.
3
+ ## 2.0.1
4
+ - Simplify the shutdown implementation a bit for easier understanding
5
+ ## 2.0.0
6
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
7
+ instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
8
+ - Dependency on logstash-core update to 2.0
9
+
@@ -0,0 +1,2 @@
1
+ # logstash-input-example
2
+ Example input plugin. This should help bootstrap your effort to write your own input plugin!
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
@@ -0,0 +1,5 @@
1
+ Elasticsearch
2
+ Copyright 2012-2015 Elasticsearch
3
+
4
+ This product includes software developed by The Apache Software
5
+ Foundation (http://www.apache.org/).
@@ -0,0 +1,89 @@
1
+ # Logstash Plugin
2
+
3
+ [![Build
4
+ Status](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Inputs/job/logstash-plugin-input-example-unit/badge/icon)](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Inputs/job/logstash-plugin-input-example-unit/)
5
+
6
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
7
+
8
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
9
+
10
+ ## Documentation
11
+
12
+ Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
13
+
14
+ - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
15
+ - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
16
+
17
+ ## Need Help?
18
+
19
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
20
+
21
+ ## Developing
22
+
23
+ ### 1. Plugin Developement and Testing
24
+
25
+ #### Code
26
+ - To get started, you'll need JRuby with the Bundler gem installed.
27
+
28
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
29
+
30
+ - Install dependencies
31
+ ```sh
32
+ bundle install
33
+ ```
34
+
35
+ #### Test
36
+
37
+ - Update your dependencies
38
+
39
+ ```sh
40
+ bundle install
41
+ ```
42
+
43
+ - Run tests
44
+
45
+ ```sh
46
+ bundle exec rspec
47
+ ```
48
+
49
+ ### 2. Running your unpublished Plugin in Logstash
50
+
51
+ #### 2.1 Run in a local Logstash clone
52
+
53
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
54
+ ```ruby
55
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
56
+ ```
57
+ - Install plugin
58
+ ```sh
59
+ bin/plugin install --no-verify
60
+ ```
61
+ - Run Logstash with your plugin
62
+ ```sh
63
+ bin/logstash -e 'filter {awesome {}}'
64
+ ```
65
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
66
+
67
+ #### 2.2 Run in an installed Logstash
68
+
69
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
70
+
71
+ - Build your plugin gem
72
+ ```sh
73
+ gem build logstash-filter-awesome.gemspec
74
+ ```
75
+ - Install the plugin from the Logstash home
76
+ ```sh
77
+ bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
78
+ ```
79
+ - Start Logstash and proceed to test the plugin
80
+
81
+ ## Contributing
82
+
83
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
84
+
85
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
86
+
87
+ It is more important to the community that you are able to contribute.
88
+
89
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,96 @@
1
+ # encoding: utf-8
2
+ require 'gcloud'
3
+
4
+ require 'faraday'
5
+ module Faraday
6
+ class Adapter
7
+ class NetHttp < Faraday::Adapter
8
+ def ssl_verify_mode(ssl)
9
+ OpenSSL::SSL::VERIFY_NONE
10
+ end
11
+ end
12
+ end
13
+ end
14
+
15
+ require "logstash/inputs/base"
16
+ require "logstash/namespace"
17
+
18
+ # Stream events from files from a Google Cloud PubSub subscription
19
+ class LogStash::Inputs::GoogleCloudPubSub < LogStash::Inputs::Base
20
+ config_name "google_cloud_pubsub"
21
+
22
+ config :project
23
+
24
+ # Path to JSON file containing the Service Account credentials (not needed when running inside GCE)
25
+ config :keyfile
26
+
27
+ # The name of the PubSub topic
28
+ config :topic, :validate => :string, :required => true
29
+
30
+ # Autocreate the topic if it doesn't exist
31
+ config :autocreate_topic, :validate => :boolean, :default => true
32
+
33
+ # The name of the project that the topic is in (if it's not the current project)
34
+ config :topic_project, :validate => :string
35
+
36
+ # The name of the PubSub subscription. The subscription should already exist otherwise
37
+ config :subscription, :validate => :string, :required => true, :default => 'logstash'
38
+
39
+ # Autocreate the subscription if it doesn't exist
40
+ config :autocreate_subscription, :validate => :boolean, :default => true
41
+
42
+ # Maximum number of messages to pull in a single call
43
+ config :batch_size, :validate => :number, :default => 10
44
+
45
+ default :codec, "plain"
46
+
47
+ public
48
+ def register
49
+ @logger.info("Registering Google Cloud PubSub input", :project => @project, :keyfile => @keyfile, :topic => @topic,
50
+ :subscription => @subscription)
51
+
52
+ @pubsub = Gcloud.new(project=@project, keyfile=@keyfile).pubsub
53
+ _topic = @pubsub.topic(@topic, { :autocreate => @autocreate_topic, :project => @topic_project })
54
+
55
+ raise "Topic #{@topic} not found" if not _topic
56
+ @logger.debug("Topic: ", :topic => _topic)
57
+
58
+ @sub = _topic.subscription(@subscription)
59
+
60
+ if (not @sub or not @sub.exists?)
61
+ if @autocreate_subscription
62
+ @logger.debug("Creating Subscription: ", :subscription => @subscription)
63
+ @sub = _topic.subscribe(@subscription)
64
+ else
65
+ raise "Subscription #{@subscription} not found"
66
+ end
67
+ end
68
+
69
+ @logger.debug("Subscription: ", :subscription => @sub)
70
+
71
+ end # def register
72
+
73
+ public
74
+ def run(queue)
75
+ while !stop do
76
+ msgs = @sub.wait_for_messages max: @batch_size
77
+ successful_messages = []
78
+
79
+ msgs.each do |msg|
80
+ begin
81
+ @codec.decode(msg.data) do |event|
82
+ msg.attributes.each_pair{|k, v| event[k] = v}
83
+ decorate(event)
84
+ queue << event
85
+ end
86
+ successful_messages.append(msg)
87
+ rescue StandardError
88
+ @logger.warn("Error processing message", :message => msg)
89
+ end
90
+ end
91
+
92
+ @sub.acknowledge successful_messages if successful_messages.length > 0
93
+ end
94
+ end # def run
95
+
96
+ end # class LogStash::Inputs::GCS
@@ -0,0 +1,25 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-input-google-cloud-pubsub'
3
+ s.version = '2.0.1'
4
+ s.licenses = ['Apache License (2.0)']
5
+ s.summary = "This example input streams a string at a definable interval."
6
+ s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
7
+ s.authors = ["Barak Cohen"]
8
+ s.email = 'barak@wondermall.com'
9
+ s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
10
+ s.require_paths = ["lib"]
11
+
12
+ # Files
13
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
14
+ # Tests
15
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
16
+
17
+ # Special flag to let us know this is actually a logstash plugin
18
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
19
+
20
+ # Gem dependencies
21
+ s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
22
+ s.add_runtime_dependency 'gcloud'
23
+ s.add_runtime_dependency 'stud', '>= 0.0.22'
24
+ s.add_development_dependency 'logstash-devutils', '>= 0.0.16'
25
+ end
@@ -0,0 +1,11 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/inputs/example"
4
+
5
+ describe LogStash::Inputs::Example do
6
+
7
+ it_behaves_like "an interruptible input plugin" do
8
+ let(:config) { { "interval" => 100 } }
9
+ end
10
+
11
+ end
metadata ADDED
@@ -0,0 +1,119 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-input-google-cloud-pubsub
3
+ version: !ruby/object:Gem::Version
4
+ version: 2.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Barak Cohen
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-12-09 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: logstash-core
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - '>='
18
+ - !ruby/object:Gem::Version
19
+ version: 2.0.0
20
+ - - <
21
+ - !ruby/object:Gem::Version
22
+ version: 3.0.0
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - '>='
28
+ - !ruby/object:Gem::Version
29
+ version: 2.0.0
30
+ - - <
31
+ - !ruby/object:Gem::Version
32
+ version: 3.0.0
33
+ - !ruby/object:Gem::Dependency
34
+ name: gcloud
35
+ requirement: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - '>='
38
+ - !ruby/object:Gem::Version
39
+ version: '0'
40
+ type: :runtime
41
+ prerelease: false
42
+ version_requirements: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - '>='
45
+ - !ruby/object:Gem::Version
46
+ version: '0'
47
+ - !ruby/object:Gem::Dependency
48
+ name: stud
49
+ requirement: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - '>='
52
+ - !ruby/object:Gem::Version
53
+ version: 0.0.22
54
+ type: :runtime
55
+ prerelease: false
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - '>='
59
+ - !ruby/object:Gem::Version
60
+ version: 0.0.22
61
+ - !ruby/object:Gem::Dependency
62
+ name: logstash-devutils
63
+ requirement: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - '>='
66
+ - !ruby/object:Gem::Version
67
+ version: 0.0.16
68
+ type: :development
69
+ prerelease: false
70
+ version_requirements: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - '>='
73
+ - !ruby/object:Gem::Version
74
+ version: 0.0.16
75
+ description: This gem is a logstash plugin required to be installed on top of the
76
+ Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not
77
+ a stand-alone program
78
+ email: barak@wondermall.com
79
+ executables: []
80
+ extensions: []
81
+ extra_rdoc_files: []
82
+ files:
83
+ - CHANGELOG.md
84
+ - DEVELOPER.md
85
+ - Gemfile
86
+ - LICENSE
87
+ - NOTICE.TXT
88
+ - README.md
89
+ - lib/logstash/inputs/google_cloud_pubsub.rb
90
+ - logstash-input-google-cloud-pubsub.gemspec
91
+ - spec/inputs/example_spec.rb
92
+ homepage: http://www.elastic.co/guide/en/logstash/current/index.html
93
+ licenses:
94
+ - Apache License (2.0)
95
+ metadata:
96
+ logstash_plugin: 'true'
97
+ logstash_group: input
98
+ post_install_message:
99
+ rdoc_options: []
100
+ require_paths:
101
+ - lib
102
+ required_ruby_version: !ruby/object:Gem::Requirement
103
+ requirements:
104
+ - - '>='
105
+ - !ruby/object:Gem::Version
106
+ version: '0'
107
+ required_rubygems_version: !ruby/object:Gem::Requirement
108
+ requirements:
109
+ - - '>='
110
+ - !ruby/object:Gem::Version
111
+ version: '0'
112
+ requirements: []
113
+ rubyforge_project:
114
+ rubygems_version: 2.2.2
115
+ signing_key:
116
+ specification_version: 4
117
+ summary: This example input streams a string at a definable interval.
118
+ test_files:
119
+ - spec/inputs/example_spec.rb