logstash-output-google_pubsub 1.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b690a5feb720aadd58e937e7f5dbe8637008b019cceb99422f395c23a2d9e878
+  data.tar.gz: dfd102d278a4519dd0ea69800dd90bd9aa8fb5a6b8bb3410f464d509376b0465
+SHA512:
+  metadata.gz: a6e973f797df417d33e95cfd3b42c746cfa252de38f777aaddb19096a1c63ffa47f9d84b47b10f20c441340f61afe414f39a927afdf312ae34c76f9fc2e78ad1
+  data.tar.gz: 4d87a5b5ac2ad897aaff09f3b1f9e9a61aeb9dcc50392853184e546d1a61f0ca794fdbba933dd8e0cbccbf2a262c278c11c1a7b84fcba2377ba9ac134e3b6a1c

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 1.0.0
+  - Initial implementation

data/CONTRIBUTING.md

@@ -0,0 +1,9 @@
+# Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/CONTRIBUTORS

@@ -0,0 +1,13 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+
+* Google LLC
+* Eric Johnson - erjohnso@google.com
+* Joseph Lewis III - jlewisiii@google.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,10 @@
+source 'https://rubygems.org'
+gemspec
+
+logstash_path = ENV['LOGSTASH_PATH'] || '../../logstash'
+use_logstash_source = ENV['LOGSTASH_SOURCE'] && ENV['LOGSTASH_SOURCE'].to_s == '1'
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,96 @@
+# Logstash Output Google Pubsub
+
+A Logstash plugin to upload log events to https://cloud.google.com/pubsub/[Google Cloud Pubsub].
+Events are batched and uploaded in the background for the sake of efficiency.
+Message payloads are serialized JSON representations of the events.
+
+Example use-cases:
+
+ * Stream events to Dataproc via Pub/Sub for real-time analysis.
+ * Forward events from an on-prem datacenter to the Logstash in the cloud.
+ * Use Pub/Sub as an scalable buffer to even out event flow between processing steps.
+
+Note: While this project is partially maintained by Google, this is not an official Google product.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+bundle exec rake vendor
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+bundle exec rake vendor
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/docs/index.asciidoc

@@ -0,0 +1,231 @@
+:plugin: google_pubsub
+:type: output
+:default_codec: json
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Google Cloud Pub/Sub Output Plugin
+
+include::{include_path}/plugin_header.asciidoc[]
+
+==== Description
+
+A Logstash plugin to upload log events to https://cloud.google.com/pubsub/[Google Cloud Pubsub].
+Events are batched and uploaded in the background for the sake of efficiency.
+Message payloads are serialized JSON representations of the events.
+
+Example use-cases:
+
+ * Stream events to Dataproc via Pub/Sub for real-time analysis.
+ * Forward events from an on-prem datacenter to the Logstash in the cloud.
+ * Use Pub/Sub as an scalable buffer to even out event flow between processing steps.
+
+Note: While this project is partially maintained by Google, this is not an official Google product.
+
+==== Environment Configuration
+
+To use this plugin, you must create a
+https://developers.google.com/storage/docs/authentication#service_accounts[service account]
+and grant it the publish permission on a topic.
+You MAY also use the https://cloud.google.com/docs/authentication/production[Application Default Credentials]
+assigned to a compute instance.
+
+The Pub/Sub topic __must__ exist before you run the plugin.
+
+
+==== Example Configurations
+
+===== Basic
+
+A basic configuration which only includes a project, topic, and JSON key file:
+
+[source,ruby]
+------------------------------------------------------------------------------
+output {
+  google_pubsub {
+    # Required attributes
+    project_id => "my_project"
+    topic => "my_topic"
+
+    # Optional if you're using app default credentials
+    json_key_file => "service_account_key.json"
+  }
+}
+------------------------------------------------------------------------------
+
+
+===== High Volume
+
+If you find that uploads are going too slowly, you can increase the message batching:
+
+[source,ruby]
+------------------------------------------------------------------------------
+output {
+  google_pubsub {
+    project_id => "my_project"
+    topic => "my_topic"
+    json_key_file => "service_account_key.json"
+
+    # Options for configuring the upload
+    message_count_threshold => 10000
+    delay_threshold_secs => 10
+    request_byte_threshold => 5MB
+  }
+}
+------------------------------------------------------------------------------
+
+
+===== Attributes
+
+You can attach additional attributes to each request.
+For example, you could attach a datacenter label to a log message to help with debugging:
+
+[source,ruby]
+------------------------------------------------------------------------------
+output {
+  google_pubsub {
+    project_id => "my_project"
+    topic => "my_topic"
+    json_key_file => "service_account_key.json"
+
+
+    attributes => {"origin" => "pacific-datacenter"}
+  }
+}
+------------------------------------------------------------------------------
+
+
+===== Different Codecs
+
+You can use codecs with this plugin to change the body of the events:
+
+[source,ruby]
+------------------------------------------------------------------------------
+output {
+  google_pubsub {
+    project_id => "my_project"
+    topic => "my_topic"
+    json_key_file => "service_account_key.json"
+
+
+    codec => plain {format => "%{[time]}: %{[message]}"}
+  }
+}
+------------------------------------------------------------------------------
+
+
+
+==== Additional Resources
+
+ * https://cloud.google.com/pubsub/[Cloud Pub/Sub Homepage]
+ * https://cloud.google.com/pubsub/pricing-summary/[Cloud Pub/Sub Pricing]
+ * https://cloud.google.com/iam/docs/service-accounts[IAM Service Accounts]
+ * https://cloud.google.com/docs/authentication/production[Application Default Credentials]
+
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Google Cloud Pub/Sub Output Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-project_id>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-topic>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-json_key_file>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-delay_threshold_secs>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-message_count_threshold>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-request_byte_threshold>> |<<bytes,bytes>>|No
+| <<plugins-{type}s-{plugin}-attributes>> |<<hash,hash>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+input plugins.
+
+[id="plugins-{type}s-{plugin}-project_id"]
+===== `project_id`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Google Cloud Project ID (name, not number).
+
+[id="plugins-{type}s-{plugin}-topic"]
+===== `topic`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Google Cloud Pub/Sub Topic. You must create the topic manually before running this plugin.
+
+[id="plugins-{type}s-{plugin}-json_key_file"]
+===== `json_key_file`
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+The path to the key to authenticate your user to the bucket.
+This service user _must_ have the `pubsub.topics.publish` permission so it can publish to the topic.
+
+
+If Logstash is running within Google Compute Engine and no `json_key_file` is defined,
+the plugin will use GCE's Application Default Credentials.
+Outside of GCE, you must to specify a Service Account JSON key file.
+
+
+[id="plugins-{type}s-{plugin}-delay_threshold_secs"]
+===== `delay_threshold_secs`
+
+  * Value type is <<number,number>>
+  * Default is: `5`
+
+Send the batch once this delay has passed, from the time the first message is queued.
+Must be greater than 0.
+
+
+[id="plugins-{type}s-{plugin}-message_count_threshold"]
+===== `message_count_threshold`
+
+  * Value type is <<number,number>>
+  * Default is: `100`
+
+Once this many messages are queued, send all the messages in a single call, even if the delay threshold hasn't elapsed yet.
+Must be < 1000.
+A value of 0 will cause messages to instantly be sent but will reduce total throughput due to overhead.
+
+[id="plugins-{type}s-{plugin}-request_byte_threshold"]
+===== `request_byte_threshold`
+
+  * Value type is <<bytes,bytes>>
+  * Default is: `1_000_000`
+
+Once the number of bytes in the batched request reaches this threshold, send all of the messages in
+a single call, even if neither the delay or message count thresholds have been exceeded yet.
+This includes full message payload size, including any attributes set.
+
+
+[id="plugins-{type}s-{plugin}-attributes"]
+===== `attributes`
+
+  * Value type is <<hash,hash>>
+  * Default is: `{}`
+
+Attributes to add to the message in key: value formats.
+Keys and values MUST be strings.
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
+
+:default_codec!:

data/lib/logstash-output-google_pubsub_jars.rb

@@ -0,0 +1,41 @@
+# AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
+
+require 'jar_dependencies'
+require_jar('com.google.cloud', 'google-cloud-pubsub', '1.37.1')
+require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.1.3')
+require_jar('com.google.api', 'api-common', '1.6.0')
+require_jar('com.google.api', 'gax', '1.29.0')
+require_jar('com.google.api', 'gax-grpc', '1.29.0')
+require_jar('com.google.api.grpc', 'grpc-google-cloud-pubsub-v1', '1.19.1')
+require_jar('com.google.api.grpc', 'proto-google-cloud-pubsub-v1', '1.19.1')
+require_jar('com.google.api.grpc', 'proto-google-common-protos', '1.12.0')
+require_jar('com.google.api.grpc', 'proto-google-iam-v1', '0.12.0')
+require_jar('com.google.auth', 'google-auth-library-credentials', '0.9.1')
+require_jar('com.google.auth', 'google-auth-library-oauth2-http', '0.9.1')
+require_jar('com.google.auto.value', 'auto-value', '1.4')
+require_jar('com.google.cloud', 'google-cloud-core', '1.37.1')
+require_jar('com.google.cloud', 'google-cloud-core-grpc', '1.37.1')
+require_jar('com.google.code.findbugs', 'jsr305', '3.0.2')
+require_jar('com.google.code.gson', 'gson', '2.7')
+require_jar('com.google.errorprone', 'error_prone_annotations', '2.2.0')
+require_jar('com.google.guava', 'guava', '20.0')
+require_jar('com.google.http-client', 'google-http-client', '1.23.0')
+require_jar('com.google.http-client', 'google-http-client-jackson2', '1.19.0')
+require_jar('com.google.protobuf', 'protobuf-java', '3.6.0')
+require_jar('com.google.protobuf', 'protobuf-java-util', '3.6.0')
+require_jar('com.google.protobuf', 'protobuf-lite', '3.0.1')
+require_jar('commons-codec', 'commons-codec', '1.3')
+require_jar('commons-logging', 'commons-logging', '1.1.1')
+require_jar('io.grpc', 'grpc-auth', '1.13.1')
+require_jar('io.grpc', 'grpc-context', '1.13.1')
+require_jar('io.grpc', 'grpc-core', '1.13.1')
+require_jar('io.grpc', 'grpc-netty-shaded', '1.13.1')
+require_jar('io.grpc', 'grpc-protobuf', '1.13.1')
+require_jar('io.grpc', 'grpc-protobuf-lite', '1.13.1')
+require_jar('io.grpc', 'grpc-stub', '1.13.1')
+require_jar('io.opencensus', 'opencensus-api', '0.12.3')
+require_jar('io.opencensus', 'opencensus-contrib-grpc-metrics', '0.12.3')
+require_jar('joda-time', 'joda-time', '2.9.2')
+require_jar('org.apache.httpcomponents', 'httpclient', '4.0.1')
+require_jar('org.apache.httpcomponents', 'httpcore', '4.0.1')
+require_jar('org.threeten', 'threetenbp', '1.3.3')

data/lib/logstash/outputs/google_pubsub.rb

@@ -0,0 +1,80 @@
+require 'logstash/outputs/base'
+require 'logstash/namespace'
+require 'logstash/outputs/pubsub/client'
+
+# A Logstash plugin to upload log events to https://cloud.google.com/pubsub/[Google Cloud Pubsub].
+class LogStash::Outputs::GooglePubsub < LogStash::Outputs::Base
+  config_name 'google_pubsub'
+
+  concurrency :shared
+
+  # Google Cloud Project ID (name, not number)
+  config :project_id, validate: :string, required: true
+
+  # Google Cloud Pub/Sub Topic, expected to exist before the plugin starts
+  config :topic, validate: :string, required: true
+
+  # A full path to the JSON key file, if empty it's assumed Application Default Credentials
+  # will be used.
+  config :json_key_file, validate: :path, required: false
+
+  # Send the batch once this delay has passed, from the time the first message
+  # is queued. (> 0, default: 5)
+  config :delay_threshold_secs, validate: :number, default: 5
+
+  # Once this many messages are queued, send all the messages in a single call, < 1000
+  config :message_count_threshold, validate: :number, default: 100
+
+  # Once the number of bytes in the batched request reaches this threshold,
+  # send all of the messages in a single call, even if neither the delay or
+  # message count thresholds have been exceeded yet.
+  config :request_byte_threshold, validate: :bytes, default: 1_000_000
+
+  # Attributes to add to the message in key: value formats.
+  config :attributes, validate: :hash, default: {}
+
+  # By default, we serialize messages with JSON.
+  default :codec, 'json'
+
+  def register
+    @logger.info("Registering Google PubSub Output plugin: #{full_topic}")
+
+    batch_settings = LogStash::Outputs::Pubsub::Client.build_batch_settings(
+      @request_byte_threshold,
+      @delay_threshold_secs,
+      @message_count_threshold
+    )
+
+    @pubsub = LogStash::Outputs::Pubsub::Client.new(
+        @json_key_file,
+        full_topic,
+        batch_settings,
+        @logger
+    )
+
+    # Test that the attributes don't cause errors when they're set.
+    begin
+      @pubsub.build_message('', @attributes)
+    rescue TypeError => e
+      message = 'Make sure the attributes are string:string pairs'
+      @logger.error(message, error: e, attributes: @attributes)
+      raise message
+    end
+  end
+
+  def multi_receive_encoded(events_and_encoded)
+    events_and_encoded.each do |event, encoded|
+      @logger.debug("Sending message #{encoded}")
+
+      @pubsub.publish_message(encoded, @attributes)
+    end
+  end
+
+  def stop
+    @pubsub.shutdown
+  end
+
+  def full_topic
+    "projects/#{@project_id}/topics/#{@topic}"
+  end
+end

data/lib/logstash/outputs/pubsub/client.rb

@@ -0,0 +1,107 @@
+require 'java'
+require 'logstash-output-google_pubsub_jars.rb'
+require 'logstash/outputs/pubsub/message_future_callback'
+
+module LogStash
+  module Outputs
+    module Pubsub
+
+      # A wrapper around PubSub's Java API.
+      class Client
+        def initialize(json_key_file, topic_name, batch_settings, logger, client=nil)
+          @logger = logger
+
+          @pubsub = client || initialize_google_client(json_key_file, topic_name, batch_settings)
+        end
+
+        # Creates a Java BatchSettings object given user-defined thresholds.
+        def self.build_batch_settings(byte_threshold, delay_threshold_secs, count_threshold)
+          com.google.api.gax.batching.BatchingSettings.newBuilder
+              .setElementCountThreshold(count_threshold)
+              .setRequestByteThreshold(byte_threshold)
+              .setDelayThreshold(org.threeten.bp.Duration.ofSeconds(delay_threshold_secs))
+              .build
+        end
+
+        # Creates a Java PubsubMessage given the message body as a string and a
+        # string:string hash of attributes
+        def build_message(message_string, attributes)
+          attributes ||= {}
+
+          data = com.google.protobuf.ByteString.copyFromUtf8(message_string)
+          builder = com.google.pubsub.v1.PubsubMessage.newBuilder
+                       .setData(data)
+
+          attributes.each { |k, v| builder.putAttributes(k, v) }
+
+          builder.build
+        end
+
+        # Creates a PubsubMessage from the string and attributes
+        # then queues it up to be sent.
+        def publish_message(message_string, attributes)
+          message = build_message(message_string, attributes)
+          messageIdFuture = @pubsub.publish(message)
+          setup_callback(message_string, messageIdFuture)
+        end
+
+        # Sets up the Google pubsub client.
+        # It's unlikely this is needed out of initialize, but it's left public
+        # for the purposes of mocking.
+        def initialize_google_client(json_key_file, topic_name, batch_settings)
+          @logger.info("Initializing Google API client on #{topic_name} key: #{json_key_file}")
+
+          if use_default_credential? json_key_file
+            credentials = com.google.cloud.pubsub.v1.TopicAdminSettings.defaultCredentialsProviderBuilder().build()
+          else
+            raise_key_file_error(json_key_file)
+
+            key_file = java.io.FileInputStream.new(json_key_file)
+            sac = com.google.auth.oauth2.ServiceAccountCredentials.fromStream(key_file)
+            credentials = com.google.api.gax.core.FixedCredentialsProvider.create(sac)
+          end
+
+          com.google.cloud.pubsub.v1.Publisher.newBuilder(topic_name)
+             .setCredentialsProvider(credentials)
+             .setHeaderProvider(construct_headers)
+             .setBatchingSettings(batch_settings)
+             .build
+        end
+
+        # Schedules immediate publishing of any outstanding messages and waits
+        # until all are processed.
+        def shutdown
+          @pubsub.shutdown
+        end
+
+        private
+
+        def setup_callback(message_string, messageIdFuture)
+          callback = LogStash::Outputs::Pubsub::MessageFutureCallback.new message_string, @logger
+
+          com.google.api.core.ApiFutures.addCallback(messageIdFuture, callback)
+        end
+
+        def construct_headers
+          gem_name = 'logstash-output-google_pubsub'
+          user_agent = "Elastic/#{gem_name}"
+
+          com.google.api.gax.rpc.FixedHeaderProvider.create('User-Agent', user_agent)
+        end
+
+        def use_default_credential?(key_file)
+          key_file.nil? || key_file == ''
+        end
+
+        # raises an exception if the key file is invalid
+        def raise_key_file_error(key_file)
+          is_abs = ::File.absolute_path(key_file) == key_file
+          raise "json_key_file must be an absolute path: #{key_file}" unless is_abs
+
+          exists = ::File.exist? key_file
+          raise "json_key_file does not exist: #{key_file}" unless exists
+        end
+      end
+    end
+  end
+end

data/lib/logstash/outputs/pubsub/message_future_callback.rb

@@ -0,0 +1,23 @@
+module LogStash
+  module Outputs
+    module Pubsub
+      # Implements ApiFutureCallback<String>
+      class MessageFutureCallback
+        def initialize(message_body, logger)
+          @message_body = message_body
+          @logger = logger
+        end
+
+        # Implements public void onSuccess(String messageId)
+        def on_success(message_id)
+          @logger.debug("Published #{@message_body} with id: #{message_id}")
+        end
+
+        # Implements public void onFailure(Throwable t)
+        def on_failure(throwable)
+          @logger.error("Failed to send message.", message: @message_body, error: throwable.getMessage)
+        end
+      end
+    end
+  end
+end

data/logstash-output-google_pubsub.gemspec

@@ -0,0 +1,35 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-output-google_pubsub'
+  s.version       = '1.0.0'
+  s.licenses      = ['Apache-2.0']
+  s.summary       = 'Emit output messages to Google Pubsub topic.'
+  s.description   = 'This gem is a Logstash output plugin to be installed on top of the Logstash'\
+                    'core pipeline using $LS_HOME/bin/logstash-plugin install gemname.'\
+                    'This gem is not a stand-alone program.'
+  s.homepage      = 'https://cloud.google.com/pubsub/overview'
+  s.authors       = ['Eric Johnson']
+  s.email         = 'erjohnso@google.com'
+  s.require_paths = ['lib', 'vendor/jar-dependencies']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE',
+                'NOTICE.TXT', 'vendor/jar-dependencies/**/*.jar', 'vendor/jar-dependencies/**/*.rb',
+                'VERSION', 'docs/**/*']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { 'logstash_plugin' => 'true', 'logstash_group' => 'output' }
+
+  # Gem dependencies
+  s.add_runtime_dependency 'logstash-codec-json', '~> 3.0'
+  s.add_runtime_dependency 'logstash-codec-plain', '~> 3.0'
+  s.add_runtime_dependency 'logstash-core-plugin-api', '>= 1.60', '<= 2.99'
+  s.add_development_dependency 'logstash-devutils', '~> 1.3'
+
+  # JARs
+  s.add_development_dependency 'jar-dependencies', '~> 0.3.4'
+
+  s.platform = 'java'
+end

data/spec/outputs/google_pubsub_spec.rb

@@ -0,0 +1,67 @@
+require 'logstash/devutils/rspec/spec_helper'
+require 'logstash/outputs/google_pubsub'
+require 'logstash/outputs/pubsub/client'
+require 'logstash/codecs/plain'
+require 'logstash/event'
+require 'json'
+
+describe LogStash::Outputs::GooglePubsub do
+  let(:config) { {
+      'project_id' => 'my-project',
+      'topic' => 'my-topic',
+      'delay_threshold_secs' => 1,
+      'message_count_threshold' => 2,
+      'request_byte_threshold' => 3,
+      'attributes' => {'foo' => 'bar'}
+  } }
+  let(:sample_event) { LogStash::Event.new({'key'=>'value'}) }
+
+  let(:pubsub_client) { double('pubsub-api-client') }
+  let(:batching_settings) { double('batching-settings') }
+
+  subject { LogStash::Outputs::GooglePubsub.new(config) }
+
+
+  before(:each) do
+    delay = config['delay_threshold_secs']
+    count = config['message_count_threshold']
+    bytes = config['request_byte_threshold']
+
+    allow(LogStash::Outputs::Pubsub::Client).to receive(:build_batch_settings).and_return(batching_settings)
+    expect(LogStash::Outputs::Pubsub::Client).to receive(:build_batch_settings).with(bytes, delay, count)
+
+    allow(LogStash::Outputs::Pubsub::Client).to receive(:new).and_return(pubsub_client)
+    expect(LogStash::Outputs::Pubsub::Client).to receive(:new)
+
+    allow(pubsub_client).to receive(:build_message)
+    subject.register
+  end
+
+  describe '#multi_receive_encoded' do
+    it 'sends the message as JSON text by default' do
+      expect(pubsub_client).to receive(:publish_message).with(/"key":"value"/, anything)
+
+      subject.multi_receive([sample_event])
+    end
+
+    it 'sends attributes' do
+      expect(pubsub_client).to receive(:publish_message).with(anything, config['attributes'])
+
+      subject.multi_receive([sample_event])
+    end
+  end
+
+  describe '#stop' do
+    it 'calls shutdown on the pubsub client' do
+      expect(pubsub_client).to receive(:shutdown)
+
+      subject.stop
+    end
+  end
+
+  describe '#full_topic' do
+    it 'formats the topic correctly' do
+      expect(subject.full_topic).to eq('projects/my-project/topics/my-topic')
+    end
+  end
+end

data/spec/outputs/pubsub/client_spec.rb

@@ -0,0 +1,106 @@
+require 'logstash/devutils/rspec/spec_helper'
+require 'logstash/outputs/pubsub/client'
+require 'logstash/codecs/plain'
+require 'logstash/event'
+require 'java'
+
+describe LogStash::Outputs::Pubsub::Client do
+  let(:config) { {
+      'project_id' => 'my-project',
+      'topic' => 'my-topic',
+      'delay_threshold_secs' => 1,
+      'message_count_threshold' => 2,
+      'request_byte_threshold' => 3,
+      'attributes' => {'foo' => 'bar'}
+  } }
+  let(:sample_event) { LogStash::Event.new({'key'=>'value'}) }
+
+  let(:logger) { double('logger') }
+  let(:api_client) { spy('api-client') }
+  let(:batch_settings) { double('api-client') }
+
+  subject { LogStash::Outputs::Pubsub::Client.new(nil, nil, batch_settings, logger, api_client) }
+
+
+  describe '#build_message' do
+    it 'creates a Java PubsubMessage' do
+      msg = subject.build_message('message-body', {})
+      expect(msg.getClass().getName()).to eq('com.google.pubsub.v1.PubsubMessage')
+    end
+
+    it 'sets the message body' do
+      msg = subject.build_message('message-body', {})
+      expect(msg.get_data.to_string_utf8).to eq('message-body')
+    end
+
+    it 'adds all attributes' do
+      msg = subject.build_message('message-body', {'a'=>'b','c'=>'d'})
+      expect(msg.get_attributes_count).to eq(2)
+      expect(msg.get_attributes_or_default('a', nil)).to eq('b')
+      expect(msg.get_attributes_or_default('c', nil)).to eq('d')
+    end
+
+    it 'does not fail with nil attributes' do
+      expect{subject.build_message('message-body', nil)}.to_not raise_error
+    end
+  end
+
+  describe '#build_batch_settings' do
+    it 'creates a Java BatchingSettings object' do
+      b = LogStash::Outputs::Pubsub::Client.build_batch_settings(1,2,3)
+
+      expect(b.getClass().getName()).to eq('com.google.api.gax.batching.AutoValue_BatchingSettings')
+    end
+
+    it 'sets byte threshold correctly' do
+      b = LogStash::Outputs::Pubsub::Client.build_batch_settings(1,2,3)
+
+      expect(b.getRequestByteThreshold).to eq(1)
+    end
+
+    it 'sets delay correctly' do
+      b = LogStash::Outputs::Pubsub::Client.build_batch_settings(1,2,3)
+
+      expect(b.getDelayThreshold().getSeconds()).to eq(2)
+    end
+
+    it 'sets count_threshold correctly' do
+      b = LogStash::Outputs::Pubsub::Client.build_batch_settings(1,2,3)
+
+      expect(b.getElementCountThreshold()).to eq(3)
+    end
+  end
+
+  describe '#publish_message' do
+    before(:each) do
+      allow(com.google.api.core.ApiFutures).to receive(:addCallback)
+    end
+
+    it 'builds a message with passed in attributes' do
+      allow(subject).to receive(:build_message).and_return(double('message'))
+      expect(subject).to receive(:build_message).with('foo', {'a'=>'b'})
+
+      subject.publish_message 'foo', {'a' => 'b'}
+    end
+
+    it 'publishes the message' do
+      expect(api_client).to receive(:publish)
+
+      subject.publish_message 'foo', {'a' => 'b'}
+    end
+
+    it 'creates a callback' do
+      expect(com.google.api.core.ApiFutures).to receive(:addCallback)
+
+      subject.publish_message 'foo', {'a' => 'b'}
+    end
+  end
+
+  describe '#shutdown' do
+    it 'calls shutdown on the client' do
+      expect(api_client).to receive(:shutdown)
+
+      subject.shutdown
+    end
+  end
+end

metadata

@@ -0,0 +1,178 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-google_pubsub
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: java
+authors:
+- Eric Johnson
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2018-07-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  name: logstash-codec-json
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.3.4
+  name: jar-dependencies
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.3.4
+description: This gem is a Logstash output plugin to be installed on top of the Logstashcore
+  pipeline using $LS_HOME/bin/logstash-plugin install gemname.This gem is not a stand-alone
+  program.
+email: erjohnso@google.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTING.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- docs/index.asciidoc
+- lib/logstash-output-google_pubsub_jars.rb
+- lib/logstash/outputs/google_pubsub.rb
+- lib/logstash/outputs/pubsub/client.rb
+- lib/logstash/outputs/pubsub/message_future_callback.rb
+- logstash-output-google_pubsub.gemspec
+- spec/outputs/google_pubsub_spec.rb
+- spec/outputs/pubsub/client_spec.rb
+- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.1.3/jackson-core-2.1.3.jar
+- vendor/jar-dependencies/com/google/api/api-common/1.6.0/api-common-1.6.0.jar
+- vendor/jar-dependencies/com/google/api/gax-grpc/1.29.0/gax-grpc-1.29.0.jar
+- vendor/jar-dependencies/com/google/api/gax/1.29.0/gax-1.29.0.jar
+- vendor/jar-dependencies/com/google/api/grpc/grpc-google-cloud-pubsub-v1/1.19.1/grpc-google-cloud-pubsub-v1-1.19.1.jar
+- vendor/jar-dependencies/com/google/api/grpc/proto-google-cloud-pubsub-v1/1.19.1/proto-google-cloud-pubsub-v1-1.19.1.jar
+- vendor/jar-dependencies/com/google/api/grpc/proto-google-common-protos/1.12.0/proto-google-common-protos-1.12.0.jar
+- vendor/jar-dependencies/com/google/api/grpc/proto-google-iam-v1/0.12.0/proto-google-iam-v1-0.12.0.jar
+- vendor/jar-dependencies/com/google/auth/google-auth-library-credentials/0.9.1/google-auth-library-credentials-0.9.1.jar
+- vendor/jar-dependencies/com/google/auth/google-auth-library-oauth2-http/0.9.1/google-auth-library-oauth2-http-0.9.1.jar
+- vendor/jar-dependencies/com/google/auto/value/auto-value/1.4/auto-value-1.4.jar
+- vendor/jar-dependencies/com/google/cloud/google-cloud-core-grpc/1.37.1/google-cloud-core-grpc-1.37.1.jar
+- vendor/jar-dependencies/com/google/cloud/google-cloud-core/1.37.1/google-cloud-core-1.37.1.jar
+- vendor/jar-dependencies/com/google/cloud/google-cloud-pubsub/1.37.1/google-cloud-pubsub-1.37.1.jar
+- vendor/jar-dependencies/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar
+- vendor/jar-dependencies/com/google/code/gson/gson/2.7/gson-2.7.jar
+- vendor/jar-dependencies/com/google/errorprone/error_prone_annotations/2.2.0/error_prone_annotations-2.2.0.jar
+- vendor/jar-dependencies/com/google/guava/guava/20.0/guava-20.0.jar
+- vendor/jar-dependencies/com/google/http-client/google-http-client-jackson2/1.19.0/google-http-client-jackson2-1.19.0.jar
+- vendor/jar-dependencies/com/google/http-client/google-http-client/1.23.0/google-http-client-1.23.0.jar
+- vendor/jar-dependencies/com/google/protobuf/protobuf-java-util/3.6.0/protobuf-java-util-3.6.0.jar
+- vendor/jar-dependencies/com/google/protobuf/protobuf-java/3.6.0/protobuf-java-3.6.0.jar
+- vendor/jar-dependencies/com/google/protobuf/protobuf-lite/3.0.1/protobuf-lite-3.0.1.jar
+- vendor/jar-dependencies/commons-codec/commons-codec/1.3/commons-codec-1.3.jar
+- vendor/jar-dependencies/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-auth/1.13.1/grpc-auth-1.13.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-context/1.13.1/grpc-context-1.13.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-core/1.13.1/grpc-core-1.13.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-netty-shaded/1.13.1/grpc-netty-shaded-1.13.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-protobuf-lite/1.13.1/grpc-protobuf-lite-1.13.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-protobuf/1.13.1/grpc-protobuf-1.13.1.jar
+- vendor/jar-dependencies/io/grpc/grpc-stub/1.13.1/grpc-stub-1.13.1.jar
+- vendor/jar-dependencies/io/opencensus/opencensus-api/0.12.3/opencensus-api-0.12.3.jar
+- vendor/jar-dependencies/io/opencensus/opencensus-contrib-grpc-metrics/0.12.3/opencensus-contrib-grpc-metrics-0.12.3.jar
+- vendor/jar-dependencies/joda-time/joda-time/2.9.2/joda-time-2.9.2.jar
+- vendor/jar-dependencies/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar
+- vendor/jar-dependencies/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar
+- vendor/jar-dependencies/org/threeten/threetenbp/1.3.3/threetenbp-1.3.3.jar
+homepage: https://cloud.google.com/pubsub/overview
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+- vendor/jar-dependencies
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.6.13
+signing_key:
+specification_version: 4
+summary: Emit output messages to Google Pubsub topic.
+test_files:
+- spec/outputs/google_pubsub_spec.rb
+- spec/outputs/pubsub/client_spec.rb