logstash-input-google_cloud_storage 0.9.0-java → 0.10.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3c14fcc815a37372caf1deb49133e231a58107cb6b4389af67407d1af20bdca1
-  data.tar.gz: c7457cb5c5e24166c788d8222ea8fe84c65ff5f68c48c34936dafa01f739d407
+  metadata.gz: e78557c8bf094bfd04130b57015071baadd85453f4f697589352da83e40fdeb6
+  data.tar.gz: 0320d090a6bf8dc24676ab59ea59c401cb6aa18c8d4206a1df69017de7901617
 SHA512:
-  metadata.gz: 3b0be07f9543f6db508a0f29e63c1b03a191b892149eb1229f79221b909fb02601ed7dd0c429088d0bb081e459a693b15b5dfd94114b0b9ff514489736d2bf2d
-  data.tar.gz: 7d555d58756d2f735736a64dc5d4aca2d7f2a376fc62eaf10017695f2e39ccc06cae896f2939446ade623f01ac11dc7e087dcc3ea359707e87591fc28daff61e
+  metadata.gz: c9fd651ea1779ec2e824f11975ff95e3f3b3d3be809c150e590d60e2d500ab2fc78d4f1058326ba7c7058f75a1be3d693bb7f0317ab3de4f9ad30cd8e8a73188
+  data.tar.gz: 5935af6d43c06e4c766fed2b64e5d5c01ebed026edfb157ab1f27ff06af3111a5dff83df35a6f36f8fa4b0304fb04c1edc4bdfefe12de3598f3539ab24b4b2de

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.10.0
+
+- Updated JAR dependencies.
+- Changed the way lines were being read to not strip the included line endings.
+
 ## 0.9.0
 
 - Initial release

data/CONTRIBUTING.md

@@ -1,6 +1,6 @@
 All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
 
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
 
 It is more important to the community that you are able to contribute.
 

data/README.md

@@ -4,7 +4,7 @@ Extracts events from files in a Google Cloud Storage bucket.
 
 Example use-cases:
 
- * Read https://cloud.google.com/stackdriver/[Stackdriver logs] from a Cloud Storage bucket into Elastic.
+ * Read [Stackdriver logs](https://cloud.google.com/stackdriver/) from a Cloud Storage bucket into Elastic.
  * Read gzipped logs from cold-storage into Elastic.
  * Restore data from an Elastic dump.
  * Extract data from Cloud Storage, transform it with Logstash and load it into BigQuery.
@@ -36,6 +36,7 @@ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/log
 - Install dependencies
 ```sh
 bundle install
+bundle exec rake vendor
 ```
 
 #### Test
@@ -66,7 +67,7 @@ bin/logstash-plugin install --no-verify logstash-input-google_cloud_storage
 ```
 - Run Logstash with your plugin
 ```sh
-bin/logstash -e 'inpug {google_cloud_storage {}}'
+bin/logstash -e 'input {google_cloud_storage {}}'
 ```
 At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
 
@@ -88,7 +89,7 @@ bin/logstash-plugin install /your/local/plugin/logstash-input-google_cloud_stora
 
 All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
 
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
 
 It is more important to the community that you are able to contribute.
 

data/docs/index.asciidoc

@@ -19,7 +19,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
 
 include::{include_path}/plugin_header.asciidoc[]
 
-
+[id="plugins-{type}s-{plugin}-description"]
 ==== Description
 
 Extracts events from files in a Google Cloud Storage bucket.
@@ -33,11 +33,230 @@ Example use-cases:
 
 Note: While this project is partially maintained by Google, this is not an official Google product.
 
+[id="plugins-{type}s-{plugin}-metadata-attributes"]
+==== Metadata Attributes
+
+The plugin exposes several metadata attributes about the object being read.
+You can access these later in the pipeline to augment the data or perform conditional logic.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+| Key | Type | Description
+| `[@metadata][gcs][bucket]` | `string` | The name of the bucket the file was read from.
+| `[@metadata][gcs][name]` | `string` | The name of the object.
+| `[@metadata][gcs][metadata]` | `object` | A map of metadata on the object.
+| `[@metadata][gcs][md5]` | `string` | MD5 hash of the data. Encoded using base64.
+| `[@metadata][gcs][crc32c]` | `string` | CRC32c checksum, as described in RFC 4960. Encoded using base64 in big-endian byte order.
+| `[@metadata][gcs][generation]` | `long` | The content generation of the object. Used for object versioning
+| `[@metadata][gcs][line]` | `long` | The position of the event in the file. 1 indexed.
+| `[@metadata][gcs][line_id]` | `string` | A deterministic, unique ID describing this line. This lets you do idempotent inserts into Elasticsearch.
+|=======================================================================
+
+More information about object metadata can be found in the
+https://cloud.google.com/storage/docs/json_api/v1/objects[official documentation].
+
+[id="plugins-{type}s-{plugin}-example-configurations"]
+==== Example Configurations
+
+===== Basic
+
+Basic configuration to read JSON logs every minute from `my-logs-bucket`.
+For example, https://cloud.google.com/stackdriver/[Stackdriver logs].
+
+[source,ruby]
+----------------------------------
+input {
+  google_cloud_storage {
+    interval => 60
+    bucket_id => "my-logs-bucket"
+    json_key_file => "/home/user/key.json"
+    file_matches => ".*json"
+    codec => "json_lines"
+  }
+}
+output { stdout { codec => rubydebug } }
+----------------------------------
+
+===== Idempotent Inserts into Elasticsearch
+
+If your pipeline might insert the same file multiple times you can use the `line_id` metadata key as a deterministic id.
+
+The ID has the format: `gs://<bucket_id>/<object_id>:<line_num>@<generation>`.
+`line_num` represents the nth event deserialized from the file starting at 1.
+`generation` is a unique id Cloud Storage generates for the object.
+When an object is overwritten it gets a new generation.
+
+[source,ruby]
+----------------------------------
+input {
+  google_cloud_storage {
+    bucket_id => "batch-jobs-output"
+  }
+}
+
+output {
+  elasticsearch {
+    document_id => "%{[@metadata][gcs][line_id]}"
+  }
+}
+----------------------------------
+
+===== From Cloud Storage to BigQuery
+
+Extract data from Cloud Storage, transform it with Logstash and load it into BigQuery.
+
+[source,ruby]
+----------------------------------
+input {
+  google_cloud_storage {
+    interval => 60
+    bucket_id => "batch-jobs-output"
+    file_matches => "purchases.*.csv"
+    json_key_file => "/home/user/key.json"
+    codec => "plain"
+  }
+}
+
+filter {
+  csv {
+    columns => ["transaction", "sku", "price"]
+    convert => {
+      "transaction" => "integer"
+      "price" => "float"
+    }
+  }
+}
+
+output {
+  google_bigquery {
+    project_id => "my-project"
+    dataset => "logs"
+    csv_schema => "transaction:INTEGER,sku:INTEGER,price:FLOAT"
+    json_key_file => "/path/to/key.json"
+    error_directory => "/tmp/bigquery-errors"
+    ignore_unknown_values => true
+  }
+}
+----------------------------------
+
+[id="plugins-{type}s-{plugin}-additional-resources"]
+==== Additional Resources
+
+ * https://cloud.google.com/storage/[Cloud Storage Homepage]
+ * https://cloud.google.com/storage/pricing-summary/[Cloud Storage Pricing]
+ * https://cloud.google.com/iam/docs/service-accounts[IAM Service Accounts]
+ * https://cloud.google.com/docs/authentication/production[Application Default Credentials]
+
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Google Cloud Storage Input Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-bucket_id>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-json_key_file>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-interval>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-file_matches>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-file_exclude>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-metadata_key>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-processed_db_path>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-delete>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-unpack_gzip>> |<<boolean,boolean>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+input plugins.
+
+
+[id="plugins-{type}s-{plugin}-bucket_id"]
+===== `bucket_id`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The bucket containing your log files.
+
+[id="plugins-{type}s-{plugin}-json_key_file"]
+===== `json_key_file`
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+The path to the key to authenticate your user to the bucket.
+This service user _should_ have the `storage.objects.update` permission so it can
+create metadata on the object preventing it from being scanned multiple times.
+
+[id="plugins-{type}s-{plugin}-interval"]
+===== `interval`
+
+  * Value type is <<number,number>>
+  * Default is: `60`
+
+The number of seconds between looking for new files in your bucket.
+
+[id="plugins-{type}s-{plugin}-file_matches"]
+===== `file_matches`
+
+  * Value type is <<string,string>>
+  * Default is: `.*\.log(\.gz)?`
+
+A regex pattern to filter files. Only files with names matching this will be considered.
+All files match by default.
+
+[id="plugins-{type}s-{plugin}-file_exclude"]
+===== `file_exclude`
+
+  * Value type is <<string,string>>
+  * Default is: `^$`
+
+Any files matching this regex are excluded from processing.
+No files are excluded by default.
+
+[id="plugins-{type}s-{plugin}-metadata_key"]
+===== `metadata_key`
+
+  * Value type is <<string,string>>
+  * Default is: `x-goog-meta-ls-gcs-input`
+
+This key will be set on the objects after they've been processed by the plugin. That way you can
+stop the plugin and not upload files again or prevent them from being uploaded by setting the
+field manually.
+
+NOTE: the key is a flag, if a file was partially processed before Logstash exited some events will be resent.
+
+[id="plugins-{type}s-{plugin}-processed_db_path"]
+===== `processed_db_path`
+
+  * Value type is <<path,path>>
+  * Default is: `LOGSTASH_DATA/plugins/inputs/google_cloud_storage/db`.
+
+If set, the plugin will store the list of processed files locally.
+This allows you to create a service account for the plugin that does not have write permissions.
+However, the data will not be shared across multiple running instances of Logstash.
+
+[id="plugins-{type}s-{plugin}-delete"]
+===== `delete`
+
+  * Value type is <<boolean,boolean>>
+  * Default is: `false`
+
+Should the log file be deleted after its contents have been updated?
+
+[id="plugins-{type}s-{plugin}-unpack_gzip"]
+===== `unpack_gzip`
+
+  * Value type is <<boolean,boolean>>
+  * Default is: `true`
 
-include::metadata.asciidoc[]
+If set to `true`, files ending in `.gz` are decompressed before they're parsed by the codec.
+The file will be skipped if it has the suffix, but can't be opened as a gzip, e.g.
+if it has a bad magic number.
 
-include::examples.asciidoc[]
 
-include::resources.asciidoc[]
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
 
-include::configuration.asciidoc[]
+:default_codec!:

data/lib/logstash-input-google_cloud_storage_jars.rb

@@ -1,38 +1,42 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('com.google.cloud', 'google-cloud-storage', '1.25.0')
-require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.1.3')
-require_jar('com.google.api', 'api-common', '1.5.0')
-require_jar('com.google.api-client', 'google-api-client', '1.23.0')
-require_jar('com.google.api', 'gax', '1.23.0')
-require_jar('com.google.api', 'gax-httpjson', '0.40.0')
-require_jar('com.google.api.grpc', 'proto-google-common-protos', '1.7.0')
-require_jar('com.google.api.grpc', 'proto-google-iam-v1', '0.8.0')
-require_jar('com.google.apis', 'google-api-services-storage', 'v1-rev114-1.23.0')
-require_jar('com.google.auth', 'google-auth-library-credentials', '0.9.0')
-require_jar('com.google.auth', 'google-auth-library-oauth2-http', '0.9.0')
-require_jar('com.google.cloud', 'google-cloud-core', '1.25.0')
-require_jar('com.google.cloud', 'google-cloud-core-http', '1.25.0')
-require_jar('com.google.code.findbugs', 'jsr305', '3.0.1')
+require_jar('com.google.cloud', 'google-cloud-storage', '1.62.0')
+require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.9.6')
+require_jar('com.google.api', 'api-common', '1.7.0')
+require_jar('com.google.api-client', 'google-api-client', '1.27.0')
+require_jar('com.google.api', 'gax', '1.38.0')
+require_jar('com.google.api', 'gax-httpjson', '0.55.0')
+require_jar('com.google.api.grpc', 'proto-google-common-protos', '1.12.0')
+require_jar('com.google.api.grpc', 'proto-google-iam-v1', '0.12.0')
+require_jar('com.google.apis', 'google-api-services-storage', 'v1-rev20181109-1.27.0')
+require_jar('com.google.auth', 'google-auth-library-credentials', '0.12.0')
+require_jar('com.google.auth', 'google-auth-library-oauth2-http', '0.12.0')
+require_jar('com.google.cloud', 'google-cloud-core', '1.62.0')
+require_jar('com.google.cloud', 'google-cloud-core-http', '1.62.0')
+require_jar('com.google.code.findbugs', 'jsr305', '3.0.2')
 require_jar('com.google.code.gson', 'gson', '2.7')
 require_jar('com.google.errorprone', 'error_prone_annotations', '2.2.0')
-require_jar('com.google.guava', 'guava', '20.0')
+require_jar('com.google.guava', 'guava', '26.0-android')
 require_jar('com.google.guava', 'guava-jdk5', '17.0')
-require_jar('com.google.http-client', 'google-http-client', '1.23.0')
-require_jar('com.google.http-client', 'google-http-client-appengine', '1.23.0')
+require_jar('com.google.http-client', 'google-http-client', '1.27.0')
+require_jar('com.google.http-client', 'google-http-client-appengine', '1.27.0')
 require_jar('com.google.http-client', 'google-http-client-jackson', '1.23.0')
-require_jar('com.google.http-client', 'google-http-client-jackson2', '1.23.0')
-require_jar('com.google.oauth-client', 'google-oauth-client', '1.23.0')
-require_jar('com.google.protobuf', 'protobuf-java', '3.5.1')
-require_jar('com.google.protobuf', 'protobuf-java-util', '3.5.1')
-require_jar('commons-codec', 'commons-codec', '1.3')
-require_jar('commons-logging', 'commons-logging', '1.1.1')
-require_jar('io.grpc', 'grpc-context', '1.9.0')
-require_jar('io.opencensus', 'opencensus-api', '0.11.1')
-require_jar('io.opencensus', 'opencensus-contrib-http-util', '0.11.1')
+require_jar('com.google.http-client', 'google-http-client-jackson2', '1.27.0')
+require_jar('com.google.j2objc', 'j2objc-annotations', '1.1')
+require_jar('com.google.oauth-client', 'google-oauth-client', '1.27.0')
+require_jar('com.google.protobuf', 'protobuf-java', '3.6.1')
+require_jar('com.google.protobuf', 'protobuf-java-util', '3.6.1')
+require_jar('commons-codec', 'commons-codec', '1.10')
+require_jar('commons-logging', 'commons-logging', '1.2')
+require_jar('io.grpc', 'grpc-context', '1.14.0')
+require_jar('io.opencensus', 'opencensus-api', '0.17.0')
+require_jar('io.opencensus', 'opencensus-contrib-http-util', '0.17.0')
+require_jar('javax.annotation', 'javax.annotation-api', '1.2')
 require_jar('joda-time', 'joda-time', '2.9.2')
-require_jar('org.apache.httpcomponents', 'httpclient', '4.0.1')
-require_jar('org.apache.httpcomponents', 'httpcore', '4.0.1')
+require_jar('org.apache.httpcomponents', 'httpclient', '4.5.5')
+require_jar('org.apache.httpcomponents', 'httpcore', '4.4.9')
+require_jar('org.checkerframework', 'checker-compat-qual', '2.5.2')
 require_jar('org.codehaus.jackson', 'jackson-core-asl', '1.9.11')
+require_jar('org.codehaus.mojo', 'animal-sniffer-annotations', '1.14')
 require_jar('org.threeten', 'threetenbp', '1.3.3')

data/lib/logstash/inputs/cloud_storage/blob_adapter.rb

@@ -1,18 +1,5 @@
 # encoding: utf-8
 
-# Copyright 2018 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
 require 'java'
 require 'logstash-input-google_cloud_storage_jars.rb'
 

data/lib/logstash/inputs/cloud_storage/blob_filter.rb

@@ -1,19 +1,5 @@
 # encoding: utf-8
 
-# Copyright 2018 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
 module LogStash
   module Inputs
     module CloudStorage

data/lib/logstash/inputs/cloud_storage/client.rb

@@ -1,18 +1,5 @@
 # encoding: utf-8
 
-# Copyright 2018 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
 require 'thread'
 require 'java'
 require 'logstash-input-google_cloud_storage_jars.rb'

data/lib/logstash/inputs/cloud_storage/file_reader.rb

@@ -1,18 +1,5 @@
 # encoding: utf-8
 
-# Copyright 2018 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
 require 'zlib'
 
 module LogStash
@@ -41,7 +28,7 @@ module LogStash
         def self.read_plain_lines(filename, &block)
           line_num = 1
           ::File.open(filename).each do |line|
-            block.call(line.chomp, line_num)
+            block.call(line, line_num)
             line_num += 1
           end
         end
@@ -49,7 +36,7 @@ module LogStash
         def self.read_gzip_lines(filename, &block)
           line_num = 1
           Zlib::GzipReader.open(filename).each_line do |line|
-            block.call(line.chomp, line_num)
+            block.call(line, line_num)
             line_num += 1
           end
         end