logstash-input-google_cloud_storage 0.9.0-java → 0.10.0-java

data/docs/examples.asciidoc

@@ -1,84 +0,0 @@
-==== Example Configurations
-
-===== Basic
-
-Basic configuration to read JSON logs every minute from `my-logs-bucket`.
-For example, https://cloud.google.com/stackdriver/[Stackdriver logs].
-
-[source,ruby]
-----------------------------------
-input {
-  google_cloud_storage {
-    interval => 60
-    bucket_id => "my-logs-bucket"
-    json_key_file => "/home/user/key.json"
-    file_matches => ".*json"
-    codec => "json_lines"
-  }
-}
-output { stdout { codec => rubydebug } }
-----------------------------------
-
-===== Idempotent Inserts into Elasticsearch
-
-If your pipeline might insert the same file multiple times you can use the `line_id` metadata key as a deterministic id.
-
-The ID has the format: `gs://<bucket_id>/<object_id>:<line_num>@<generation>`.
-`line_num` represents the nth event deserialized from the file starting at 1.
-`generation` is a unique id Cloud Storage generates for the object.
-When an object is overwritten it gets a new generation.
-
-[source,ruby]
-----------------------------------
-input {
-  google_cloud_storage {
-    bucket_id => "batch-jobs-output"
-  }
-}
-
-output {
-  elasticsearch {
-    document_id => "%{[@metadata][gcs][line_id]}"
-  }
-}
-----------------------------------
-
-===== From Cloud Storage to BigQuery
-
-
-Extract data from Cloud Storage, transform it with Logstash and load it into BigQuery.
-
-
-[source,ruby]
-----------------------------------
-input {
-  google_cloud_storage {
-    interval => 60
-    bucket_id => "batch-jobs-output"
-    file_matches => "purchases.*.csv"
-    json_key_file => "/home/user/key.json"
-    codec => "plain"
-  }
-}
-
-filter {
-  csv {
-    columns => ["transaction", "sku", "price"]
-    convert => {
-      "transaction" => "integer"
-      "price" => "float"
-    }
-  }
-}
-
-output {
-  google_bigquery {
-    project_id => "my-project"
-    dataset => "logs"
-    csv_schema => "transaction:INTEGER,sku:INTEGER,price:FLOAT"
-    json_key_file => "/path/to/key.json"
-    error_directory => "/tmp/bigquery-errors"
-    ignore_unknown_values => true
-  }
-}
-----------------------------------

data/docs/metadata.asciidoc

@@ -1,20 +0,0 @@
-==== Metadata Attributes
-
-The plugin exposes several metadata attributes about the object being read.
-You can access these later in the pipeline to augment the data or perform conditional logic.
-
-[cols="<,<,<",options="header",]
-|=======================================================================
-| Key | Type | Description
-| `[@metadata][gcs][bucket]` | `string` | The name of the bucket the file was read from.
-| `[@metadata][gcs][name]` | `string` | The name of the object.
-| `[@metadata][gcs][metadata]` | `object` | A map of metadata on the object.
-| `[@metadata][gcs][md5]` | `string` | MD5 hash of the data. Encoded using base64.
-| `[@metadata][gcs][crc32c]` | `string` | CRC32c checksum, as described in RFC 4960. Encoded using base64 in big-endian byte order.
-| `[@metadata][gcs][generation]` | `long` | The content generation of the object. Used for object versioning
-| `[@metadata][gcs][line]` | `long` | The position of the event in the file. 1 indexed.
-| `[@metadata][gcs][line_id]` | `string` | A deterministic, unique ID describing this line. This lets you do idempotent inserts into Elasticsearch.
-|=======================================================================
-
-More information about object metadata can be found in the
-https://cloud.google.com/storage/docs/json_api/v1/objects[official documentation].

data/docs/resources.asciidoc

@@ -1,6 +0,0 @@
-==== Additional Resources
-
- * https://cloud.google.com/storage/[Cloud Storage Homepage]
- * https://cloud.google.com/storage/pricing-summary/[Cloud Storage Pricing]
- * https://cloud.google.com/iam/docs/service-accounts[IAM Service Accounts]
- * https://cloud.google.com/docs/authentication/production[Application Default Credentials]