logstash-output-kinesis 0.0.7-java → 0.0.8-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5a611c00487fc62b28645be128e51346c99ef79d
-  data.tar.gz: 9a02efc3bdeda7653fab8c244d5ebcd65b787a41
+  metadata.gz: ba5057a7a75e92ec491b6129eafc0a09742174d4
+  data.tar.gz: 9d779195310054cc17f7eeb4d492a4e5d25ee748
 SHA512:
-  metadata.gz: b9b8e63eb47505e6cef6adf49e6fddd652c1f4ebef89d88b1058f7d6749f0a85659cc08e8a0481b631f8504ffa7927951c24ec637f55ac47ff5f5c3f99f5cbaf
-  data.tar.gz: c1e89b38ad7b8e2ed67b693176976665d741deff4167624ae657548de065e988110461912cd56973bf115e3365e0efcecc4d3480240c0043e3ab4ba437e55615
+  metadata.gz: e8ca60399e520e3cd298ae1a4c40b4ee3267ed4f62498f8e8650900ef87d187001cf9fd1825bfe584de63c1c984d162725d791592986778e11f6e754dead60ab
+  data.tar.gz: eb83a4d5e8a9f77a929ac3c77a5921bb9932d8e8215dcca7c62b71973253fe16394fcbc1c485c8f142f437d6a519c6b44ee0535ab94c2795b000a020658f942b

data/README.md

@@ -5,14 +5,14 @@
 
 This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
 
-It will send log records to a [Kinesis stream](https://aws.amazon.com/kinesis/), using the [KPL](https://docs.aws.amazon.com/kinesis/latest/dev/developing-producers-with-kpl.html) library.
+It will send log records to a [Kinesis stream](https://aws.amazon.com/kinesis/), using the [Kinesis Producer Library (KPL)](https://docs.aws.amazon.com/kinesis/latest/dev/developing-producers-with-kpl.html) library.
 
 
 ## Configuration
 
 Minimum required configuration to get this plugin chugging along:
 
-```
+```nginx
 output {
   kinesis {
     stream_name => "logs-stream"
@@ -23,23 +23,79 @@ output {
 
 This plugin accepts a wide range of configuration options, most of which come from the underlying KPL library itself. [View the full list of KPL configuration options here.][kpldoc]
 
-Please note that configuration options are snake_cased instead of camelCased. So, where [KinesisProducerConfiguration][kpldoc] offers a `setMetricsLevel` option,this plugin accepts a `metrics_level` option.
+Please note that configuration options are snake_cased instead of camelCased. So, where [KinesisProducerConfiguration][kpldoc] offers a `setMetricsLevel` option, this plugin accepts a `metrics_level` option.
+
+### Metrics
+
+The underlying KPL library defaults to sending CloudWatch metrics to give insight into what it's actually doing at runtime. It's highly recommended you ensure these metrics are flowing through, and use them to monitor the health of your log shipping.
 
-### AWS Credentials
+If for some reason you want to switch them off, you can easily do so:
+
+```nginx
+output {
+  kinesis {
+    # ...
 
-There aren't many Kinesis streams out there that allow you to write to them without some AWS credentials.
+    metrics_level => "none"
+  }
+}
+```
 
-This plugin does not allow you to specify credentials directly. Instead, the AWS SDK [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) is used. It's quite flexible,you can provide your credentials via any of the following mechanisms:
+If you choose to keep metrics enabled, ensure the AWS credentials you provide to this plugin are able to write to Kinesis *and* write to CloudWatch.
 
- * `AWS_ACCESS_KEY_ID` / `AWS_SECRET_KEY` environment variables
+### Authentication
+
+By default, this plugin will use the AWS SDK [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) to obtain credentials for communication with the Kinesis stream (and CloudWatch, if metrics are enabled). The following places will be checked for credentials:
+
+ * `AWS_ACCESS_KEY_ID` / `AWS_SECRET_KEY` environment variables available to the Logstash prociess
  * `~/.aws/credentials` credentials file
- * Instance profile for your running EC2 instance
+ * Instance profile (if Logstash is running in an EC2 instance)
+
+If you want to provide credentials directly in the config file, you can do so:
+
+```nginx
+output {
+  kinesis {
+    # ...
+
+    access_key => "AKIAIDFAKECREDENTIAL"
+    secret_key => "KX0ofakeLcredentialsGrightJherepOlolPkQk"
+
+    # You can provide specific credentials for CloudWatch metrics:
+    metrics_access_key => "AKIAIDFAKECREDENTIAL"
+    metrics_secret_key => "KX0ofakeLcredentialsGrightJherepOlolPkQk"
+  }
+}
+```
+
+If `access_key` and `secret_key` are provided, they will be used for communicating with Kinesis *and* CloudWatch. If `metrics_access_key` and `metrics_secret_key` are provided, they will be used for communication with CloudWatch. If only the metrics credentials were provided, Kinesis would use the default credentials provider (explained above) and CloudWatch would use the specific credentials. Confused? Good!
+
+#### Using STS
+
+You can also configure this plugin to use [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html) to "assume" a role that has access to Kinesis and CloudWatch. If you use this in combination with EC2 instance profiles (which the defaults credentials provider explained above uses) then you can actually configure your Logstash to write to Kinesis and CloudWatch without any hardcoded credentials.
+
+```nginx
+output {
+  kinesis {
+    # ...
+
+    role_arn => "arn:aws:iam::123456789:role/my-kinesis-producer-role"
+
+    # You can also provide a specific role to assume for CloudWatch metrics:
+    metrics_role_arn => "arn:aws:iam::123456789:role/my-metrics-role"
+  }
+}
+```
+
+You can combine `role_arn` / `metrics_role_arn` with the explicit AWS credentials config explained earlier, too.
+
+All this stuff can be mixed too - if you wanted to use hardcoded credentials for Kinesis, but then assume a role via STS for accessing CloudWatch, you can do that. Vice versa would work too - assume a role for accessing Kinesis and then providing hardcoded credentials for CloudWatch. Make things as arbitrarily complicated for yourself as you like ;)
 
 ### Building a partition key
 
 Kinesis demands a [partition key](https://docs.aws.amazon.com/kinesis/latest/dev/key-concepts.html#partition-key) be provided for each record. By default, this plugin will provide a very boring partition key of `-`. However, you can configure it to compute a partition key from fields in your log events.
 
-```
+```nginx
 output {
   kinesis {
     # ...
@@ -56,7 +112,7 @@ If you are using an older version of the Amazon KCL library to consume your reco
 
 If you wish to simply disable record aggregation, that's easy:
 
-```
+```nginx
 output {
   kinesis {
     aggregation_enabled => false

data/build.gradle

@@ -7,6 +7,7 @@ repositories {
 
 dependencies {
     compile 'com.amazonaws:amazon-kinesis-producer:0.10.1'
+    compile 'com.amazonaws:aws-java-sdk-sts:1.9.37'
 }
 
 task copyLibs(type: Copy) {

data/lib/logstash-output-kinesis/version.rb

@@ -1,3 +1,3 @@
 module LogstashOutputKinesis
-  VERSION = "0.0.7"
+  VERSION = "0.0.8"
 end

data/lib/logstash/outputs/kinesis.rb

@@ -18,6 +18,20 @@ class LogStash::Outputs::Kinesis < LogStash::Outputs::Base
   # A list of event data keys to use when constructing a partition key
   config :event_partition_keys, :validate => :array, :default => []
 
+  # An AWS access key to use for authentication to Kinesis and CloudWatch
+  config :access_key, :validate => :string
+  # An AWS secret key to use for authentication to Kinesis and CloudWatch
+  config :secret_key, :validate => :string
+  # If provided, STS will be used to assume this role and use it to authenticate to Kinesis and CloudWatch
+  config :role_arn, :validate => :string
+
+  # If provided, use this AWS access key for authentication to CloudWatch
+  config :metrics_access_key, :validate => :string
+  # If provided, use this AWS secret key for authentication to CloudWatch
+  config :metrics_secret_key, :validate => :string
+  # If provided, STS will be used to assume this role and use it to authenticate to CloudWatch
+  config :metrics_role_arn, :validate => :string
+
   config :aggregation_enabled, :validate => :boolean, :default => true
   config :aggregation_max_count, :validate => :number, :default => 4294967295
   config :aggregation_max_size, :validate => :number, :default => 51200
@@ -45,10 +59,14 @@ class LogStash::Outputs::Kinesis < LogStash::Outputs::Base
   config :verify_certificate, :validate => :boolean, :default => true
 
   KPL = com.amazonaws.services.kinesis.producer
+  AWSAuth = com.amazonaws.auth
   ByteBuffer = java.nio.ByteBuffer
 
   public
   def register
+    @metrics_access_key ||= @access_key
+    @metrics_secret_key ||= @secret_key
+
     @producer = KPL.KinesisProducer::new(create_kpl_config)
     @codec.on_event(&method(:send_record))
   end
@@ -87,17 +105,22 @@ class LogStash::Outputs::Kinesis < LogStash::Outputs::Base
   def create_kpl_config
     config = KPL.KinesisProducerConfiguration::new()
 
+    credentials_provider = create_credentials_provider
+    metrics_credentials_provider = create_metrics_credentials_provider
+
     config.setAggregationEnabled(@aggregation_enabled)
     config.setAggregationMaxCount(@aggregation_max_count)
     config.setAggregationMaxSize(@aggregation_max_size)
     config.setCollectionMaxCount(@collection_max_count)
     config.setCollectionMaxSize(@collection_max_size)
     config.setConnectTimeout(@connect_timeout)
+    config.setCredentialsProvider(credentials_provider)
     config.setCredentialsRefreshDelay(@credentials_refresh_delay)
     config.setCustomEndpoint(@custom_endpoint) if !@custom_endpoint.nil?
     config.setFailIfThrottled(@fail_if_throttled)
     config.setLogLevel(@log_level)
     config.setMaxConnections(@max_connections)
+    config.setMetricsCredentialsProvider(metrics_credentials_provider)
     config.setMetricsGranularity(@metrics_granularity)
     config.setMetricsLevel(@metrics_level)
     config.setMetricsNamespace(@metrics_namespace)
@@ -116,6 +139,28 @@ class LogStash::Outputs::Kinesis < LogStash::Outputs::Base
     config
   end
 
+  def create_credentials_provider
+    provider = AWSAuth.DefaultAWSCredentialsProviderChain.new()
+    if @access_key and @secret_key
+      provider = BasicCredentialsProvider.new(AWSAuth.BasicAWSCredentials.new(@access_key, @secret_key))
+    end
+    if @role_arn
+      provider = AWSAuth.STSAssumeRoleSessionCredentialsProvider.new(provider, @role_arn, "logstash-output-kinesis")
+    end
+    provider
+  end
+
+  def create_metrics_credentials_provider
+    provider = AWSAuth.DefaultAWSCredentialsProviderChain.new()
+    if @metrics_access_key and @metrics_secret_key
+      provider = BasicCredentialsProvider.new(AWSAuth.BasicAWSCredentials.new(@metrics_access_key, @metrics_secret_key))
+    end
+    if @metrics_role_arn
+      provider = AWSAuth.STSAssumeRoleSessionCredentialsProvider.new(provider, @metrics_role_arn, "logstash-output-kinesis")
+    end
+    provider
+  end
+
   def send_record(event, payload)
     begin
       event_blob = ByteBuffer::wrap(payload.to_java_bytes)
@@ -125,3 +170,21 @@ class LogStash::Outputs::Kinesis < LogStash::Outputs::Base
     end
   end
 end
+
+class BasicCredentialsProvider
+  java_implements 'com.amazonaws.auth.AWSCredentialsProvider'
+
+  def initialize(credentials)
+    @credentials = credentials
+  end
+
+  java_signature 'com.amazonaws.auth.AWSCredentials getCredentials()'
+  def getCredentials
+    @credentials
+  end
+
+  java_signature 'void refresh()'
+  def refresh
+    # Noop.
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-kinesis
 version: !ruby/object:Gem::Version
-  version: 0.0.7
+  version: 0.0.8
 platform: java
 authors:
 - Sam Day
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2015-08-23 00:00:00.000000000 Z
+date: 2015-09-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -111,6 +111,7 @@ files:
 - spec/spec_helper.rb
 - vendor/jar-dependencies/runtime-jars/amazon-kinesis-producer-0.10.1.jar
 - vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.9.37.jar
+- vendor/jar-dependencies/runtime-jars/aws-java-sdk-sts-1.9.37.jar
 - vendor/jar-dependencies/runtime-jars/commons-codec-1.6.jar
 - vendor/jar-dependencies/runtime-jars/commons-io-2.4.jar
 - vendor/jar-dependencies/runtime-jars/commons-lang-2.6.jar