logstash-output-google_bigquery 4.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 96fffb2af160bfe6dc285eaa247f4b9eec66f135f36019bacb5f7fc014116f2f
+  data.tar.gz: 3d63d628effd3399b8c7d45f6b93b2bd0c4c31935e34346dda7309cbe2b1dfcd
+SHA512:
+  metadata.gz: a274cf2d2b610451eeca3dbc78d23bf05c01f7de6fdedcc87c84e25805afef53cc840292757e172d9317565e8be2a04dd6e3d67870deec150900dffb27b460ed
+  data.tar.gz: c20e5be7ed18d9b34935eef10a444a692df62b0fc6343cde82db20ddb2a826acf99c92c961f056203f7395583c6597d720c9b5b53f46f789fc1737a9907e53eb

data/CHANGELOG.md

@@ -0,0 +1,71 @@
+## 4.0.0
+
+**Breaking**: the update to 4.0.0 requires that you use an IAM JSON credentials file
+rather than the deprecated P12 files.
+Applications using Application Default Credentials (ADC) _will_ continue to work.
+
+This plugin now uses the BigQuery Streaming API which incurs an expense on upload.
+
+ - The advantages of the streaming API are:
+    - It allows real-time incoming data analysis and queries.
+    - It allows Logstash instances to be started/stopped without worrying about failed batch jobs.
+    - The client library has better support and performance.
+ - New configuration options:
+    - `batch_size` - The number of messages to upload at once.
+    - `json_key_file` - The JSON IAM service account credentials to use with the plugin.
+    - `batch_size_bytes` - The maximum number of bytes to upload as part of a batch (approximate).
+ - Deprecated configurations:
+    - `uploader_interval_secs` - No longer used
+    - `deleter_interval_secs` - No longer used
+    - `temp_file_prefix` - No longer used
+    - `temp_directory` - No longer used
+    - `key_password` - Use `json_key_file` or Application Default Credentials (ADC) instead.
+    - `service_account` - Use `json_key_file` or Application Default Credentials (ADC) instead.
+ - Obsolete configurations:
+    - `key_path` - Use `json_key_file` or Application Default Credentials (ADC) instead.
+      See [the documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-google_bigquery.html#plugins-outputs-google_bigquery-key_path)
+      for help about moving to JSON key files.
+
+## 3.2.4
+  - Docs: Set the default_codec doc attribute.
+
+## 3.2.3
+  - Update gemspec summary
+
+## 3.2.2
+  - Fix some documentation issues
+
+# 3.2.0
+  - Add file recovery when plugin crashes
+
+# 3.1.0
+  - Fix error checking in the plugin to properly handle failed inserts
+
+# 3.0.2
+  - Docs: Fix doc formatting
+
+# 3.0.1
+  - Pin version of gems whose latest releases only work with ruby 2.x
+
+## 3.0.0
+  - Breaking: Updated plugin to use new Java Event APIs
+  - relax contrains on logstash-core-plugin-api
+  - mark this plugin as concurrency :single
+  - update .travis.yml
+
+## 2.0.5
+  - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
+
+## 2.0.4
+  - New dependency requirements for logstash-core for the 5.0 release
+
+## 2.0.3
+ - Add support for specifying schema as a hash
+ - Bubble up error message that BQ returns on an error
+ - Add the table_separator option on bigquery output
+
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+

data/CONTRIBUTORS

@@ -0,0 +1,15 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Google LLC
+* Joseph Lewis III (josephlewis42)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Rodrigo De Castro (rdcastro)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,100 @@
+# Logstash Plugin
+
+[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-output-google_bigquery.svg)](https://travis-ci.org/logstash-plugins/logstash-output-google_bigquery)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+bundle exec rake vendor
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+bundle exec rake vendor
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/docs/index.asciidoc

@@ -0,0 +1,348 @@
+:plugin: google_bigquery
+:type: output
+:default_codec: plain
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Google BigQuery output plugin
+
+include::{include_path}/plugin_header.asciidoc[]
+
+==== Description
+
+===== Summary
+
+This plugin uploads events to Google BigQuery using the streaming API
+so data can become available nearly immediately.
+
+You can configure it to flush periodically, after N events or after
+a certain amount of data is ingested.
+
+===== Environment Configuration
+
+You must enable BigQuery on your Google Cloud Storage (GCS) account and create a dataset to
+hold the tables this plugin generates.
+
+You must also grant the service account this plugin uses access to
+the dataset.
+
+You can use https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html[Logstash conditionals]
+and multiple configuration blocks to upload events with different structures.
+
+===== Usage
+This is an example of logstash config:
+
+[source,ruby]
+--------------------------
+output {
+   google_bigquery {
+     project_id => "folkloric-guru-278"                        (required)
+     dataset => "logs"                                         (required)
+     csv_schema => "path:STRING,status:INTEGER,score:FLOAT"    (required) <1>
+     json_key_file => "/path/to/key.json"                      (optional) <2>
+     error_directory => "/tmp/bigquery-errors"                 (required)
+     date_pattern => "%Y-%m-%dT%H:00"                          (optional)
+     flush_interval_secs => 30                                 (optional)
+   }
+}
+--------------------------
+
+<1> Specify either a csv_schema or a json_schema.
+
+<2> If the key is not used, then the plugin tries to find
+https://cloud.google.com/docs/authentication/production[Application Default Credentials]
+
+===== Considerations
+
+* There is a small fee to insert data into BigQuery using the streaming API
+* This plugin buffers events in-memory, so make sure the flush configurations are appropriate
+  for your use-case and consider using
+  https://www.elastic.co/guide/en/logstash/current/persistent-queues.html[Logstash Persistent Queues]
+
+===== Additional Resources
+
+* https://cloud.google.com/docs/authentication/production[Application Default Credentials (ADC) Overview]
+* https://cloud.google.com/bigquery/[BigQuery Introduction]
+* https://cloud.google.com/bigquery/docs/schemas[BigQuery Schema Formats and Types]
+* https://cloud.google.com/bigquery/pricing[Pricing Information]
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Google BigQuery Output Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-batch_size>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-batch_size_bytes>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-csv_schema>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-dataset>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-date_pattern>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-deleter_interval_secs>> |<<number,number>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-error_directory>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-flush_interval_secs>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-ignore_unknown_values>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-json_key_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-json_schema>> |<<hash,hash>>|No
+| <<plugins-{type}s-{plugin}-key_password>> |<<string,string>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-key_path>> |<<string,string>>|*Obsolete*
+| <<plugins-{type}s-{plugin}-project_id>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-service_account>> |<<string,string>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-table_prefix>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-table_separator>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-temp_directory>> |<<string,string>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-temp_file_prefix>> |<<string,string>>|__Deprecated__
+| <<plugins-{type}s-{plugin}-uploader_interval_secs>> |<<number,number>>|__Deprecated__
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+output plugins.
+
+&nbsp;
+
+[id="plugins-{type}s-{plugin}-batch_size"]
+===== `batch_size`
+
+  * Value type is <<number,number>>
+  * Default value is `128`
+
+The number of messages to upload at a single time. (< 1000, default: 128)
+
+[id="plugins-{type}s-{plugin}-batch_size_bytes"]
+===== `batch_size_bytes`
+
+  * Value type is <<number,number>>
+  * Default value is `1_000_000`
+
+An approximate number of bytes to upload as part of a batch. Default: 1MB
+
+[id="plugins-{type}s-{plugin}-csv_schema"]
+===== `csv_schema` 
+
+  * Value type is <<string,string>>
+  * Default value is `nil`
+
+Schema for log data. It must follow the format `name1:type1(,name2:type2)*`.
+For example, `path:STRING,status:INTEGER,score:FLOAT`.
+
+[id="plugins-{type}s-{plugin}-dataset"]
+===== `dataset` 
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The BigQuery dataset the tables for the events will be added to.
+
+[id="plugins-{type}s-{plugin}-date_pattern"]
+===== `date_pattern` 
+
+  * Value type is <<string,string>>
+  * Default value is `"%Y-%m-%dT%H:00"`
+
+Time pattern for BigQuery table, defaults to hourly tables.
+Must Time.strftime patterns: www.ruby-doc.org/core-2.0/Time.html#method-i-strftime
+
+[id="plugins-{type}s-{plugin}-deleter_interval_secs"]
+===== `deleter_interval_secs`
+
+  * Value type is <<number,number>>
+
+**Deprecated:** this field is no longer used because temporary files are no longer
+stored on the hard drive.
+
+[id="plugins-{type}s-{plugin}-error_directory"]
+===== `error_directory`
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * Default value is `"/tmp/bigquery"`.
+
+The location to store events that could not be uploaded due to errors.
+Consider using an additional Logstash input to pipe the contents of
+these to an alert platform so you can manually fix the events.
+
+Or use https://cloud.google.com/storage/docs/gcs-fuse[GCS FUSE] to
+transparently upload to a GCS bucket.
+
+Files names follow the pattern `[table name]-[UNIX timestamp].log`
+
+[id="plugins-{type}s-{plugin}-flush_interval_secs"]
+===== `flush_interval_secs` 
+
+  * Value type is <<number,number>>
+  * Default value is `5`
+
+Uploads all data this often even if other upload criteria aren't met. Default: 5s
+
+[id="plugins-{type}s-{plugin}-ignore_unknown_values"]
+===== `ignore_unknown_values` 
+
+  * Value type is <<boolean,boolean>>
+  * Default value is `false`
+
+Indicates if BigQuery should ignore values that are not represented in the table schema.
+If true, the extra values are discarded.
+If false, BigQuery will reject the records with extra fields and the job will fail.
+The default value is false.
+
+NOTE: You may want to add a Logstash filter like the following to remove common fields it adds:
+
+[source,ruby]
+----------------------------------
+mutate {
+    remove_field => ["@version","@timestamp","path","host","type", "message"]
+}
+----------------------------------
+
+[id="plugins-{type}s-{plugin}-json_key_file"]
+===== `json_key_file`
+
+  * Value type is <<string,string>>
+  * Default value is `nil`
+
+If logstash is running within Google Compute Engine, the plugin can use
+GCE's Application Default Credentials. Outside of GCE, you will need to
+specify a Service Account JSON key file.
+
+[id="plugins-{type}s-{plugin}-json_schema"]
+===== `json_schema` 
+
+  * Value type is <<hash,hash>>
+  * Default value is `nil`
+
+Schema for log data as a hash.
+These can include nested records, descriptions, and modes.
+
+Example:
+[source,ruby]
+--------------------------
+json_schema => {
+  fields => [{
+    name => "endpoint"
+    type => "STRING"
+    description => "Request route"
+  }, {
+    name => "status"
+    type => "INTEGER"
+    mode => "NULLABLE"
+  }, {
+    name => "params"
+    type => "RECORD"
+    mode => "REPEATED"
+    fields => [{
+      name => "key"
+      type => "STRING"
+     }, {
+      name => "value"
+      type => "STRING"
+    }]
+  }]
+}
+--------------------------
+
+[id="plugins-{type}s-{plugin}-key_password"]
+===== `key_password`
+
+  * Value type is <<string,string>>
+
+**Deprecated:** this field is no longer needed with `json_key_file` or ADC.
+
+
+[id="plugins-{type}s-{plugin}-key_path"]
+===== `key_path`
+
+  * Value type is <<string,string>>
+
+**Obsolete:** The PKCS12 key file format is no longer supported.
+Please use one of the following mechanisms:
+
+  * https://cloud.google.com/docs/authentication/production[Application Default Credentials (ADC)],
+    configured via environment variables on Compute Engine, Kubernetes Engine, App Engine, or
+    Cloud Functions.
+  * A JSON authentication key file. You can generate them in the console for the service account
+    like you did with the `.P12` file or with the following command:
+    `gcloud iam service-accounts keys create key.json --iam-account my-sa-123@my-project-123.iam.gserviceaccount.com`
+
+[id="plugins-{type}s-{plugin}-project_id"]
+===== `project_id` 
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Google Cloud Project ID (number, not Project Name!).
+
+[id="plugins-{type}s-{plugin}-service_account"]
+===== `service_account`
+
+  * Value type is <<string,string>>
+
+**Deprecated:** this field is no longer used because it is contained in the `json_key_file` or
+the Application Default Credentials (ADC) environment variables.
+
+[id="plugins-{type}s-{plugin}-table_prefix"]
+===== `table_prefix` 
+
+  * Value type is <<string,string>>
+  * Default value is `"logstash"`
+
+BigQuery table ID prefix to be used when creating new tables for log data.
+Table name will be `<table_prefix><table_separator><date>`
+
+[id="plugins-{type}s-{plugin}-table_separator"]
+===== `table_separator` 
+
+  * Value type is <<string,string>>
+  * Default value is `"_"`
+
+BigQuery table separator to be added between the table_prefix and the
+date suffix.
+
+[id="plugins-{type}s-{plugin}-temp_directory"]
+===== `temp_directory`
+
+  * Value type is <<string,string>>
+
+**Deprecated:** this field is no longer used.
+Events are uploaded in real-time without being stored to disk.
+
+ * Events that failed to be uploaded will be stored in <<plugins-{type}s-{plugin}-error_directory>>.
+ * There is a small fee to insert data into BigQuery using the streaming API.
+   https://cloud.google.com/bigquery/pricing[Pricing Information]
+ * This plugin buffers events in-memory, so make sure the flush configurations are appropriate
+  for your use-case and consider using
+  https://www.elastic.co/guide/en/logstash/current/persistent-queues.html[Logstash Persistent Queues]
+
+[id="plugins-{type}s-{plugin}-temp_file_prefix"]
+===== `temp_file_prefix`
+
+  * Value type is <<string,string>>
+
+**Deprecated:** this field is no longer used.
+Events are uploaded in real-time without being stored to disk.
+
+ * Events that failed to be uploaded will be stored in <<plugins-{type}s-{plugin}-error_directory>>.
+ * There is a small fee to insert data into BigQuery using the streaming API.
+   https://cloud.google.com/bigquery/pricing[Pricing Information]
+ * This plugin buffers events in-memory, so make sure the flush configurations are appropriate
+  for your use-case and consider using
+  https://www.elastic.co/guide/en/logstash/current/persistent-queues.html[Logstash Persistent Queues]
+
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
+
+:default_codec!: