logstash-output-google_cloud_storage 4.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 78d40dc9f25fa3b8dd031f56c3aac1fe8c50f265172ed60d455424006d29fb53
+  data.tar.gz: 22e58b552acc6099c14838559139ec3b78247c335bf7d386d115ad940b242e14
+SHA512:
+  metadata.gz: ffa0273458eba309b372f23d2db63398bee76fd4e79efe084fb5b7b931aaa1666603a22313c88b5b31c82c923ba3965c3434edecad9f78aca457f03c0ecb176d
+  data.tar.gz: f4b3cd0a25cb1972f196be3b73d002c782f22397248d5921f087ef0c30ba8099eb7ebb17a0ae183a29ac18d6cf58cd1d4d8411166bccae8c7f158ddeaf4ff2ff

data/CHANGELOG.md

@@ -0,0 +1,77 @@
+## 4.0.0
+  - Move to the Java Google Cloud client library for increased performance and stability.
+  - **Breaking** If you use the old PKCS12 authentication keys, you will need to upgrade to
+    the new JSON keys. Application Default Credentials will continue to work.
+
+### Configuration Changes
+
+**New Options**
+
+ - `json_key_file` - The JSON IAM service account credentials to use with the plugin.
+
+**Deprecations**
+
+ - `key_password` - No longer used with `json_key_file`
+ - `service_account` - No longer used with `json_key_file`
+
+**Obsoletions**
+
+ - `key_path` - Use `json_key_file` or Application Default Credentials (ADC) instead.
+   See [the documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-google_cloud_storage.html#plugins-outputs-google_cloud_storage-key_path)
+   for help about moving to JSON key files.
+
+## 3.3.0
+Added the ability to set `gzip` as `Content-Encoding`.
+This saves storage size but still allows uncompressed downloads.
+
+- Fixes [#13](https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/13) - Use `gzip` for `Content-Encoding` instead of `Content-Type`
+
+## 3.2.1
+  - Refactoring work to add locks to file rotation and writing.
+    - Fixes [#2](https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/2) - Plugin crashes on file rotation.
+    - Fixes [#19](https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/19) - Deleted files remain in use by the system eventually filling up disk space.
+
+## 3.2.0
+  - Change uploads to use a job pool for better performance
+    - Fixes [#22](https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/22) - Refactor Job Queue Architecture
+    - Fixes [#5](https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/5) - Major Performance Issues
+  - Wait for files to upload before Logstash quits
+    - Fixes [#15](https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/15) - Fails to upload files when Logstash exits
+
+## 3.1.0
+  - Add support for disabling hostname in the log file names
+  - Add support for adding a UUID to the log file names
+
+## 3.0.5
+  - Docs: Set the default_codec doc attribute.
+
+## 3.0.4
+  - Fix some documentation issues
+
+## 3.0.2
+  - Docs: Fix doc formatting
+
+## 3.0.1
+  - align the dependency on mime-type and google-api-client with the `logstash-output-google_bigquery`
+
+## 3.0.0
+  - Breaking: Updated plugin to use new Java Event APIs
+  - relax contraints on logstash-core-plugin-api
+  - Update .travis.yml
+  - Freeze google-api-client and mime-types
+  - use concurrency :single
+
+## 2.0.4
+  - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
+
+## 2.0.3
+  - New dependency requirements for logstash-core for the 5.0 release
+
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+
+## 0.2.0
+  - Changed the Google Cloud Storage API version to v1
+  - Added simple test for plugin lookup

data/CONTRIBUTORS

@@ -0,0 +1,18 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Ethan Estrada (eestrada)
+* Google LLC.
+* Jordan Sissel (jordansissel)
+* Joseph Lewis III (jlewisiii)
+* MetaPipe
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Rodrigo De Castro (rdcastro)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,100 @@
+# Logstash Plugin
+
+[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-output-google_cloud_storage.svg)](https://travis-ci.org/logstash-plugins/logstash-output-google_cloud_storage)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+bundle exec rake vendor
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+bundle exec rake vendor
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/docs/index.asciidoc

@@ -0,0 +1,305 @@
+:plugin: google_cloud_storage
+:type: output
+:default_codec: plain
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Google Cloud Storage output plugin
+
+include::{include_path}/plugin_header.asciidoc[]
+
+==== Description
+
+A plugin to upload log events to Google Cloud Storage (GCS), rolling
+files based on the date pattern provided as a configuration setting. Events
+are written to files locally and, once file is closed, this plugin uploads
+it to the configured bucket.
+
+For more info on Google Cloud Storage, please go to:
+https://cloud.google.com/products/cloud-storage
+
+In order to use this plugin, a Google service account must be used. For
+more information, please refer to:
+https://developers.google.com/storage/docs/authentication#service_accounts
+
+Recommendation: experiment with the settings depending on how much log
+data you generate, so the uploader can keep up with the generated logs.
+Using gzip output can be a good option to reduce network traffic when
+uploading the log files and in terms of storage costs as well.
+
+==== Usage
+
+This is an example of logstash config:
+
+[source,json]
+--------------------------
+output {
+   google_cloud_storage {
+     bucket => "my_bucket"                                     (required)
+     json_key_file => "/path/to/privatekey.json"               (optional)
+     temp_directory => "/tmp/logstash-gcs"                     (optional)
+     log_file_prefix => "logstash_gcs"                         (optional)
+     max_file_size_kbytes => 1024                              (optional)
+     output_format => "plain"                                  (optional)
+     date_pattern => "%Y-%m-%dT%H:00"                          (optional)
+     flush_interval_secs => 2                                  (optional)
+     gzip => false                                             (optional)
+     gzip_content_encoding => false                            (optional)
+     uploader_interval_secs => 60                              (optional)
+     include_uuid => true                                      (optional)
+     include_hostname => true                                  (optional)
+   }
+}
+--------------------------
+
+===== Additional Resources
+
+* https://cloud.google.com/docs/authentication/production[Application Default Credentials (ADC) Overview]
+* https://cloud.google.com/storage/[Cloud Storage Introduction]
+* https://cloud.google.com/storage/pricing[Pricing Information]
+
+
+==== Improvements TODO List
+
+* Support logstash event variables to determine filename.
+* Turn Google API code into a Plugin Mixin (like AwsConfig).
+* There's no recover method, so if logstash/plugin crashes, files may not
+be uploaded to GCS.
+* Allow user to configure file name.
+* Allow parallel uploads for heavier loads (+ connection configuration if
+exposed by Ruby API client)
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Google_cloud_storage Output Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-bucket>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-date_pattern>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-flush_interval_secs>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-gzip>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-gzip_content_encoding>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-include_hostname>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-include_uuid>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-json_key_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-key_password>> |<<string,string>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-key_path>> |<<string,string>>|**Obsolete**
+| <<plugins-{type}s-{plugin}-log_file_prefix>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-max_concurrent_uploads>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-max_file_size_kbytes>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-output_format>> |<<string,string>>, one of `["json", "plain"]`|No
+| <<plugins-{type}s-{plugin}-service_account>> |<<string,string>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-temp_directory>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-uploader_interval_secs>> |<<number,number>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+output plugins.
+
+&nbsp;
+
+[id="plugins-{type}s-{plugin}-bucket"]
+===== `bucket` 
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+GCS bucket name, without "gs://" or any other prefix.
+
+[id="plugins-{type}s-{plugin}-date_pattern"]
+===== `date_pattern` 
+
+  * Value type is <<string,string>>
+  * Default value is `"%Y-%m-%dT%H:00"`
+
+Time pattern for log file, defaults to hourly files.
+Must Time.strftime patterns: www.ruby-doc.org/core-2.0/Time.html#method-i-strftime
+
+[id="plugins-{type}s-{plugin}-flush_interval_secs"]
+===== `flush_interval_secs` 
+
+  * Value type is <<number,number>>
+  * Default value is `2`
+
+Flush interval in seconds for flushing writes to log files. 0 will flush
+on every message.
+
+[id="plugins-{type}s-{plugin}-gzip"]
+===== `gzip` 
+
+  * Value type is <<boolean,boolean>>
+  * Default value is `false`
+
+Gzip output stream when writing events to log files, set
+`Content-Type` to `application/gzip` instead of `text/plain`, and
+use file suffix `.log.gz` instead of `.log`.
+
+[id="plugins-{type}s-{plugin}-gzip_content_encoding"]
+===== `gzip_content_encoding`
+
+added[3.3.0]
+
+  * Value type is <<boolean,boolean>>
+  * Default value is `false`
+
+Gzip output stream when writing events to log files and set `Content-Encoding` to `gzip`.
+This will upload your files as `gzip` saving network and storage costs, but they will be
+transparently decompressed when you read them from the storage bucket.
+
+See the Cloud Storage documentation on https://cloud.google.com/storage/docs/metadata#content-encoding[metadata] and 
+https://cloud.google.com/storage/docs/transcoding#content-type_vs_content-encoding[transcoding]
+for more information.
+
+**Note**: It is not recommended to use both `gzip_content_encoding` and `gzip`.
+This compresses your file _twice_, will increase the work your machine does and makes 
+the files larger than just compressing once.
+
+[id="plugins-{type}s-{plugin}-include_hostname"]
+===== `include_hostname`
+
+added[3.1.0]
+
+  * Value type is <<boolean,boolean>>
+  * Default value is `true`
+
+Should the hostname be included in the file name?
+You may want to turn this off for privacy reasons or if you are running multiple
+instances of Logstash and need to match the files you create with a simple glob
+such as if you wanted to import files to BigQuery.
+
+
+[id="plugins-{type}s-{plugin}-include_uuid"]
+===== `include_uuid`
+
+added[3.1.0]
+
+  * Value type is <<boolean,boolean>>
+  * Default value is `false`
+
+Adds a UUID to the end of a file name.
+You may want to enable this feature so files don't clobber one another if you're
+running multiple instances of Logstash or if you expect frequent node restarts.
+
+[id="plugins-{type}s-{plugin}-json_key_file"]
+===== `json_key_file`
+
+  * Value type is <<string,string>>
+  * Default value is `nil`
+
+The plugin can use
+https://cloud.google.com/docs/authentication/production[Application Default Credentials (ADC)],
+if it's running on Compute Engine, Kubernetes Engine, App Engine, or Cloud Functions.
+
+Outside of Google Cloud, you will need create a Service Account JSON key file through the
+web interface or with the following command:
+`gcloud iam service-accounts keys create key.json --iam-account my-sa-123@my-project-123.iam.gserviceaccount.com`
+
+[id="plugins-{type}s-{plugin}-key_password"]
+===== `key_password` 
+
+  * Value type is <<string,string>>
+  * Default value is `"notasecret"`
+
+**Deprecated** this feature is no longer used, the setting is now a part of <<plugins-{type}s-{plugin}-json_key_file>>.
+
+
+[id="plugins-{type}s-{plugin}-key_path"]
+===== `key_path`
+
+  * Value type is <<string,string>>
+
+**Obsolete:** The PKCS12 key file format is no longer supported.
+Please use one of the following mechanisms:
+
+  * https://cloud.google.com/docs/authentication/production[Application Default Credentials (ADC)],
+    configured via environment variables on Compute Engine, Kubernetes Engine, App Engine, or
+    Cloud Functions.
+  * A JSON authentication key file. You can generate them in the console for the service account
+    like you did with the `.P12` file or with the following command:
+    `gcloud iam service-accounts keys create key.json --iam-account my-sa-123@my-project-123.iam.gserviceaccount.com`
+
+[id="plugins-{type}s-{plugin}-log_file_prefix"]
+===== `log_file_prefix` 
+
+  * Value type is <<string,string>>
+  * Default value is `"logstash_gcs"`
+
+Log file prefix. Log file will follow the format:
+<prefix>_hostname_date<.part?>.log
+
+[id="plugins-{type}s-{plugin}-max_concurrent_uploads"]
+===== `max_concurrent_uploads`
+
+  * Value type is <<number,number>>
+  * Default value is `5`
+
+Sets the maximum number of concurrent uploads to Cloud Storage at a time.
+Uploads are I/O bound so it makes sense to tune this paramater with regards
+to the network bandwidth available and the latency between your server and
+Cloud Storage.
+
+[id="plugins-{type}s-{plugin}-max_file_size_kbytes"]
+===== `max_file_size_kbytes` 
+
+  * Value type is <<number,number>>
+  * Default value is `10000`
+
+Sets max file size in kbytes. 0 disable max file check.
+
+[id="plugins-{type}s-{plugin}-output_format"]
+===== `output_format` 
+
+  * Value can be any of: `json`, `plain`
+  * Default value is `"plain"`
+
+The event format you want to store in files. Defaults to plain text.
+
+[id="plugins-{type}s-{plugin}-service_account"]
+===== `service_account` 
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+**Deprecated** this feature is no longer used, the setting is now a part of <<plugins-{type}s-{plugin}-json_key_file>>.
+
+[id="plugins-{type}s-{plugin}-temp_directory"]
+===== `temp_directory` 
+
+  * Value type is <<string,string>>
+  * Default value is `""`
+
+Directory where temporary files are stored.
+Defaults to /tmp/logstash-gcs-<random-suffix>
+
+[id="plugins-{type}s-{plugin}-uploader_interval_secs"]
+===== `uploader_interval_secs` 
+
+  * Value type is <<number,number>>
+  * Default value is `60`
+
+Uploader interval when uploading new files to GCS. Adjust time based
+on your time pattern (for example, for hourly files, this interval can be
+around one hour).
+
+
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
+
+:default_codec!: