logstash-input-kinesis-jordanforks 2.0.9-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: a4dc414ade512afb167ae00c53e1b249d36eef6c
+  data.tar.gz: 5e6ec3fe52595870639712fd799fc79b319e4958
+SHA512:
+  metadata.gz: f788fc5cdcc8ecd190d21adc7f4bf56688dab1328ed7eb57853aa07fe67e4735ee708b70c5d1167161170ecd73768b6e906a05c0a3fffaaaa4fd9acce03c7926
+  data.tar.gz: f27efa5cc4d15fae4e9f503b41746e38f8e1d2e564878d32d48d8fa20dd3d910a10372f75d6bee9af92d78653145ffa4d7d209713ed0641a953c987ce29ab3a5

data/CHANGELOG.md

@@ -0,0 +1,29 @@
+## 2.0.8
+  - Changed plugin to use more recent versions of Kinesis Client library and AWS SDK[#45](https://github.com/logstash-plugins/logstash-input-kinesis/pull/45)
+
+## 2.0.7
+  - Docs: Set the default_codec doc attribute.
+
+## 2.0.7
+  - Update gemspec summary
+
+## 2.0.6
+ - Fix some documentation issues
+ - Add support for `initial_position_in_stream` config parameter. `TRIM_HORIZON` and `LATEST` are supported.
+
+## 2.0.5
+ - Docs: Add CHANGELOG.md
+ - Support for specifying an AWS credentials profile with the `profile` config parameter
+ - Docs: Remove extraneous text added during doc extract
+
+## 2.0.4
+ -  Docs: Bump version for automated doc build
+
+## 2.0.3
+ -  Fix error about failed to coerce java.util.logging.Level to org.apache.log4j.Level with logstash 5.1.1
+
+## 2.0.2
+ -  Fix error with Logstash 5.0
+ 
+## 2.0.1
+ -  Add partition_key, approximate_arrival_timestamp and sequence_number fields in the @metadata sub-has

data/CONTRIBUTORS

@@ -0,0 +1,14 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Maintainers:
+* Brian Palmer (codekitchen)
+
+Contributors:
+* Brian Palmer (codekitchen)
+* Samuel García Martínez (samuelgmartinez)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/README.md

@@ -0,0 +1,82 @@
+# Logstash AWS Kinesis Input Plugin
+
+[![Build Status](https://travis-ci.org/logstash-plugins/logstash-input-kinesis.svg)](https://travis-ci.org/logstash-plugins/logstash-input-kinesis)
+
+This is a [AWS Kinesis](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html) input plugin for [Logstash](https://github.com/elasticsearch/logstash). Under the hood uses the [Kinesis Client Library](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html).
+
+## Installation
+
+This plugin requires Logstash >= 2.0, and can be installed by Logstash
+itself.
+
+```sh
+bin/logstash-plugin install logstash-input-kinesis
+```
+
+## Usage
+
+```
+input {
+  kinesis {
+    kinesis_stream_name => "my-logging-stream"
+    codec => json { }
+  }
+}
+```
+
+### Using with CloudWatch Logs
+
+If you are looking to read a CloudWatch Logs subscription stream, you'll also want to install and configure the [CloudWatch Logs Codec](https://github.com/threadwaste/logstash-codec-cloudwatch_logs).
+
+## Configuration
+
+This are the properties you can configure and what are the default values:
+
+* `application_name`: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages.
+    * **required**: false
+    * **default value**: `logstash`
+* `kinesis_stream_name`: The Kinesis stream name.
+    * **required**: true
+* `region`: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled)
+    * **required**: false
+    * **default value**: `us-east-1`
+* `checkpoint_interval_seconds`: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
+    * **required**: false
+    * **default value**: `60`
+* `failover_time_millis`: How many milliseconds to wait since the last checkpoint before a new worker should take over the lease.
+    * **required**: false
+    * **default value**: `120000`
+* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
+    * **required**: false
+    * **default value**: `nil`
+* `profile`: The AWS profile name for authentication. This ensures that the `~/.aws/credentials` AWS auth provider is used. By default this is empty and the default chain will be used.
+    * **required**: false
+    * **default value**: `""`    
+* `initial_position_in_stream`: The value for initialPositionInStream. Accepts "TRIM_HORIZON" or "LATEST".
+    * **required**: false
+    * **default value**: `"TRIM_HORIZON"`
+
+## Authentication
+
+This plugin uses the default AWS SDK auth chain, [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html), to determine which credentials the client will use, unless `profile` is set, in which case [ProfileCredentialsProvider](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/profile/ProfileCredentialsProvider.html) is used.
+
+The default chain follows this order trying to read the credentials:
+ * `AWS_ACCESS_KEY_ID` / `AWS_SECRET_KEY` environment variables
+ * `~/.aws/credentials` credentials file
+ * EC2 instance profile
+
+The credentials will need access to the following services:
+* AWS Kinesis
+* AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition)
+* AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted.
+
+Look at the [documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) for deeper information on the default chain.
+
+## Contributing
+
+0. https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md#contribution-steps
+1. Fork it ( https://github.com/logstash-plugins/logstash-input-kinesis/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/lib/logstash/inputs/kinesis.rb

@@ -0,0 +1,135 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/errors"
+require "logstash/environment"
+require "logstash/namespace"
+
+require 'logstash-input-kinesis_jars'
+require "logstash/inputs/kinesis/version"
+
+
+# Receive events through an AWS Kinesis stream.
+#
+# This input plugin uses the Java Kinesis Client Library underneath, so the
+# documentation at https://github.com/awslabs/amazon-kinesis-client will be
+# useful.
+#
+# AWS credentials can be specified either through environment variables, or an
+# IAM instance role. The library uses a DynamoDB table for worker coordination,
+# so you'll need to grant access to that as well as to the Kinesis stream. The
+# DynamoDB table has the same name as the `application_name` configuration
+# option, which defaults to "logstash".
+#
+# The library can optionally also send worker statistics to CloudWatch.
+class LogStash::Inputs::Kinesis < LogStash::Inputs::Base
+  KCL = com.amazonaws.services.kinesis.clientlibrary.lib.worker
+  KCL_PROCESSOR_FACTORY_CLASS = com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory
+  require "logstash/inputs/kinesis/worker"
+
+  config_name 'kinesis'
+
+  attr_reader(
+    :kcl_config,
+    :kcl_worker,
+  )
+
+  # The application name used for the dynamodb coordination table. Must be
+  # unique for this kinesis stream.
+  config :application_name, :validate => :string, :default => "logstash"
+
+  # The kinesis stream name.
+  config :kinesis_stream_name, :validate => :string, :required => true
+
+  # The AWS region for Kinesis, DynamoDB, and CloudWatch (if enabled)
+  config :region, :validate => :string, :default => "us-east-1"
+
+  # How many seconds between worker checkpoints to dynamodb.
+  config :checkpoint_interval_seconds, :validate => :number, :default => 60
+
+  # How many milliseconds to wait since the last checkpoint before a new
+  # worker should take over the lease.
+  config :failover_time_millis, :validate => :number, :default => 120000
+
+  # Worker metric tracking. By default this is disabled, set it to "cloudwatch"
+  # to enable the cloudwatch integration in the Kinesis Client Library.
+  config :metrics, :validate => [nil, "cloudwatch"], :default => nil
+
+  # Select AWS profile for input
+  config :profile, :validate => :string
+
+  # Select initial_position_in_stream. Accepts TRIM_HORIZON or LATEST
+  config :initial_position_in_stream, :validate => ["TRIM_HORIZON", "LATEST"], :default => "TRIM_HORIZON"
+
+  def initialize(params = {})
+    super(params)
+  end
+
+  def register
+    # the INFO log level is extremely noisy in KCL
+    kinesis_logger = org.apache.commons.logging::LogFactory.getLog("com.amazonaws.services.kinesis").logger
+    if kinesis_logger.java_kind_of?(java.util.logging::Logger)
+      kinesis_logger.setLevel(java.util.logging::Level::WARNING)
+    else
+      kinesis_logger.setLevel(org.apache.log4j::Level::WARN)
+    end
+
+    worker_id = java.util::UUID.randomUUID.to_s
+
+    # If the AWS profile is set, use the profile credentials provider.
+    # Otherwise fall back to the default chain.
+    unless @profile.nil?
+      creds = com.amazonaws.auth.profile::ProfileCredentialsProvider.new(@profile)
+    else
+      creds = com.amazonaws.auth::DefaultAWSCredentialsProviderChain.new
+    end
+    initial_position_in_stream = if @initial_position_in_stream == "TRIM_HORIZON"
+      KCL::InitialPositionInStream::TRIM_HORIZON
+    else
+      KCL::InitialPositionInStream::LATEST
+    end
+
+    @kcl_config = KCL::KinesisClientLibConfiguration.new(
+      @application_name,
+      @kinesis_stream_name,
+      creds,
+      worker_id).
+        withInitialPositionInStream(initial_position_in_stream).
+        withRegionName(@region).
+        withFailoverTimeMillis(@failover_time_millis)
+  end
+
+  def run(output_queue)
+    @kcl_worker = kcl_builder(output_queue).build
+    @kcl_worker.run
+  end
+
+  def kcl_builder(output_queue)
+    KCL::Worker::Builder.new.tap do |builder|
+      builder.java_send(:recordProcessorFactory, [KCL_PROCESSOR_FACTORY_CLASS.java_class], worker_factory(output_queue))
+      builder.config(@kcl_config)
+
+      if metrics_factory
+        builder.metricsFactory(metrics_factory)
+      end
+    end
+  end
+
+  def stop
+    @kcl_worker.shutdown if @kcl_worker
+  end
+
+  def worker_factory(output_queue)
+    proc { Worker.new(@codec.clone, output_queue, method(:decorate), @checkpoint_interval_seconds, @logger) }
+  end
+
+  protected
+
+  def metrics_factory
+    case @metrics
+    when nil
+      com.amazonaws.services.kinesis.metrics.impl::NullMetricsFactory.new
+    when 'cloudwatch'
+      nil # default in the underlying library
+    end
+  end
+end

data/lib/logstash/inputs/kinesis/version.rb

@@ -0,0 +1,8 @@
+# encoding: utf-8
+module Logstash
+  module Input
+    module Kinesis
+      VERSION = "2.0.9"
+    end
+  end
+end

data/lib/logstash/inputs/kinesis/worker.rb

@@ -0,0 +1,67 @@
+# encoding: utf-8
+class LogStash::Inputs::Kinesis::Worker
+  include com.amazonaws.services.kinesis.clientlibrary.interfaces.v2::IRecordProcessor
+
+  attr_reader(
+    :checkpoint_interval,
+    :codec,
+    :decorator,
+    :logger,
+    :output_queue,
+  )
+
+  def initialize(*args)
+    # nasty hack, because this is the name of a method on IRecordProcessor, but also ruby's constructor
+    if !@constructed
+      @codec, @output_queue, @decorator, @checkpoint_interval, @logger = args
+      @next_checkpoint = Time.now - 600
+      @constructed = true
+    else
+      _shard_id = args[0].shardId
+    end
+  end
+  public :initialize
+
+  def processRecords(records_input)
+    records_input.records.each { |record| process_record(record) }
+    if Time.now >= @next_checkpoint
+      checkpoint(records_input.checkpointer)
+      @next_checkpoint = Time.now + @checkpoint_interval
+    end
+  end
+
+  def shutdown(shutdown_input)
+    if shutdown_input.shutdown_reason == com.amazonaws.services.kinesis.clientlibrary.lib.worker::ShutdownReason::TERMINATE
+      checkpoint(shutdown_input.checkpointer)
+    end
+  end
+
+  protected
+
+  def checkpoint(checkpointer)
+    checkpointer.checkpoint()
+  rescue => error
+    @logger.error("Kinesis worker failed checkpointing: #{error}")
+  end
+
+  def process_record(record)
+    raw = String.from_java_bytes(record.getData.array)
+    metadata = build_metadata(record)
+    @codec.decode(raw) do |event|
+      @decorator.call(event)
+      event.set('@metadata', metadata)
+      @output_queue << event
+    end
+  rescue => error
+    @logger.error("Error processing record: #{error}")
+  end
+
+  def build_metadata(record)
+    metadata = Hash.new
+    metadata['approximate_arrival_timestamp'] = record.getApproximateArrivalTimestamp.getTime
+    metadata['partition_key'] = record.getPartitionKey
+    metadata['sequence_number'] = record.getSequenceNumber
+    metadata
+  end
+
+end

data/logstash-input-kinesis.gemspec

@@ -0,0 +1,33 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'logstash/inputs/kinesis/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "logstash-input-kinesis-jordanforks"
+  spec.version       = Logstash::Input::Kinesis::VERSION
+  spec.authors       = ["Brian Palmer"]
+  spec.email         = ["brian@codekitchen.net"]
+  spec.summary       = "Receives events through an AWS Kinesis stream"
+  spec.description   = %q{This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program}
+  spec.homepage      = "https://github.com/logstash-plugins/logstash-input-kinesis"
+  spec.licenses      = ['Apache-2.0']
+
+  spec.files         = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ['lib', 'vendor/jar-dependencies/runtime-jars']
+
+  # Special flag to let us know this is actually a logstash plugin
+  spec.metadata      = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  spec.platform      = 'java'
+
+  spec.requirements << "jar 'com.amazonaws:amazon-kinesis-client', '1.9.2'"
+  spec.requirements << "jar 'com.amazonaws:aws-java-sdk-core', '1.11.414'"
+
+  spec.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+
+  spec.add_development_dependency 'logstash-devutils'
+  spec.add_development_dependency 'jar-dependencies', '~> 0.3.4'
+  spec.add_development_dependency "logstash-codec-json"
+end

data/spec/inputs/kinesis/worker_spec.rb

@@ -0,0 +1,110 @@
+require 'logstash-core/logstash-core'
+require 'logstash-input-kinesis_jars'
+require "logstash/plugin"
+require "logstash/inputs/kinesis"
+require "logstash/codecs/json"
+require "json"
+
+RSpec.describe "LogStash::Inputs::Kinesis::Worker" do
+  KCL_TYPES = com.amazonaws.services.kinesis.clientlibrary.types
+
+  subject!(:worker) { LogStash::Inputs::Kinesis::Worker.new(codec, queue, decorator, checkpoint_interval) }
+  let(:codec) { LogStash::Codecs::JSON.new() }
+  let(:queue) { Queue.new }
+  let(:decorator) { proc { |x| x.set('decorated', true); x } }
+  let(:checkpoint_interval) { 120 }
+  let(:checkpointer) { double('checkpointer', checkpoint: nil) }
+  let(:init_input) { KCL_TYPES::InitializationInput.new().withShardId("xyz") }
+
+  it "honors the initialize java interface method contract" do
+    expect { worker.initialize(init_input) }.to_not raise_error
+  end
+
+  def record(hash = { "message" => "test" }, arrival_timestamp, partition_key, sequence_number)
+    encoder = java.nio.charset::Charset.forName("UTF-8").newEncoder()
+    data = encoder.encode(java.nio.CharBuffer.wrap(JSON.generate(hash)))
+    double(
+      getData: data,
+      getApproximateArrivalTimestamp: java.util.Date.new(arrival_timestamp.to_f * 1000),
+      getPartitionKey: partition_key,
+      getSequenceNumber: sequence_number
+    )
+  end
+
+  let(:process_input) {
+    KCL_TYPES::ProcessRecordsInput.new()
+        .withRecords(java.util.Arrays.asList([
+          record(
+            {
+              id: "record1",
+              message: "test1"
+            },
+            '1.441215410867E9',
+            'partitionKey1',
+            '21269319989652663814458848515492872191'
+          ),
+          record(
+            {
+              id: "record2",
+              message: "test2"
+            },
+            '1.441215410868E9',
+            'partitionKey2',
+            '21269319989652663814458848515492872192'
+          )].to_java)
+        )
+        .withCheckpointer(checkpointer)
+  }
+  let(:empty_process_input) {
+    KCL_TYPES::ProcessRecordsInput.new()
+        .withRecords(java.util.Arrays.asList([].to_java))
+        .withCheckpointer(checkpointer)
+  }
+
+  context "initialized" do
+    before do
+      worker.initialize(init_input)
+    end
+
+    describe "#processRecords" do
+      it "decodes and queues each record with decoration" do
+        worker.processRecords(process_input)
+        expect(queue.size).to eq(2)
+        m1 = queue.pop
+        m2 = queue.pop
+        expect(m1).to be_kind_of(LogStash::Event)
+        expect(m2).to be_kind_of(LogStash::Event)
+        expect(m1.get('id')).to eq("record1")
+        expect(m1.get('message')).to eq("test1")
+        expect(m1.get('@metadata')['approximate_arrival_timestamp']).to eq(1441215410867)
+        expect(m1.get('@metadata')['partition_key']).to eq('partitionKey1')
+        expect(m1.get('@metadata')['sequence_number']).to eq('21269319989652663814458848515492872191')
+        expect(m1.get('decorated')).to eq(true)
+      end
+
+      it "checkpoints on interval" do
+        expect(checkpointer).to receive(:checkpoint).once
+        worker.processRecords(empty_process_input)
+
+        # not this time
+        worker.processRecords(empty_process_input)
+
+        allow(Time).to receive(:now).and_return(Time.now + 125)
+        expect(checkpointer).to receive(:checkpoint).once
+        worker.processRecords(empty_process_input)
+      end
+    end
+
+    describe "#shutdown" do
+      it "checkpoints on termination" do
+        input = KCL_TYPES::ShutdownInput.new
+        checkpointer = double('checkpointer')
+        expect(checkpointer).to receive(:checkpoint)
+        input.
+          with_shutdown_reason(com.amazonaws.services.kinesis.clientlibrary.lib.worker::ShutdownReason::TERMINATE).
+          with_checkpointer(checkpointer)
+        worker.shutdown(input)
+      end
+    end
+  end
+end

data/spec/inputs/kinesis_spec.rb

@@ -0,0 +1,155 @@
+require "logstash/plugin"
+require "logstash/inputs/kinesis"
+require "logstash/codecs/json"
+
+RSpec.describe "inputs/kinesis" do
+  KCL = com.amazonaws.services.kinesis.clientlibrary.lib.worker
+
+  let(:config) {{
+    "application_name" => "my-processor",
+    "kinesis_stream_name" => "run-specs",
+    "codec" => codec,
+    "metrics" => metrics,
+    "checkpoint_interval_seconds" => 120,
+    "failover_time_millis" => 30000,
+    "region" => "ap-southeast-1",
+    "profile" => nil
+  }}
+
+  # Config hash to test credentials provider to be used if profile is specified
+  let(:config_with_profile) {{
+    "application_name" => "my-processor",
+    "kinesis_stream_name" => "run-specs",
+    "codec" => codec,
+    "metrics" => metrics,
+    "checkpoint_interval_seconds" => 120,
+    "failover_time_millis" => 30000,
+    "region" => "ap-southeast-1",
+    "profile" => "my-aws-profile"
+  }}
+
+  # other config with LATEST as initial_position_in_stream
+  let(:config_with_latest) {{
+    "application_name" => "my-processor",
+    "kinesis_stream_name" => "run-specs",
+    "codec" => codec,
+    "metrics" => metrics,
+    "checkpoint_interval_seconds" => 120,
+    "failover_time_millis" => 30000,
+    "region" => "ap-southeast-1",
+    "profile" => nil,
+    "initial_position_in_stream" => "LATEST"
+  }}
+
+  subject!(:kinesis) { LogStash::Inputs::Kinesis.new(config) }
+  let(:kcl_worker) { double('kcl_worker') }
+  let(:stub_builder) { double('kcl_builder', build: kcl_worker) }
+  let(:metrics) { nil }
+  let(:codec) { LogStash::Codecs::JSON.new() }
+  let(:queue) { Queue.new }
+
+  it "registers without error" do
+    input = LogStash::Plugin.lookup("input", "kinesis").new("kinesis_stream_name" => "specs", "codec" => codec)
+    expect { input.register }.to_not raise_error
+  end
+
+  it "configures the KCL" do
+    kinesis.register
+    expect(kinesis.kcl_config.applicationName).to eq("my-processor")
+    expect(kinesis.kcl_config.streamName).to eq("run-specs")
+    expect(kinesis.kcl_config.regionName).to eq("ap-southeast-1")
+    expect(kinesis.kcl_config.initialPositionInStream).to eq(KCL::InitialPositionInStream::TRIM_HORIZON)
+    expect(kinesis.kcl_config.get_kinesis_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
+  end
+
+  subject!(:kinesis_with_profile) { LogStash::Inputs::Kinesis.new(config_with_profile) }
+
+  it "uses ProfileCredentialsProvider if profile is specified" do
+    kinesis_with_profile.register
+    expect(kinesis_with_profile.kcl_config.get_kinesis_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.profile.ProfileCredentialsProvider")
+  end
+
+  subject!(:kinesis_with_latest) { LogStash::Inputs::Kinesis.new(config_with_latest) }
+
+  it "configures the KCL" do
+    kinesis_with_latest.register
+    expect(kinesis_with_latest.kcl_config.applicationName).to eq("my-processor")
+    expect(kinesis_with_latest.kcl_config.streamName).to eq("run-specs")
+    expect(kinesis_with_latest.kcl_config.regionName).to eq("ap-southeast-1")
+    expect(kinesis_with_latest.kcl_config.initialPositionInStream).to eq(KCL::InitialPositionInStream::LATEST)
+    expect(kinesis_with_latest.kcl_config.get_kinesis_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
+  end
+
+  context "#run" do
+    it "runs the KCL worker" do
+      expect(kinesis).to receive(:kcl_builder).with(queue).and_return(stub_builder)
+      expect(kcl_worker).to receive(:run).with(no_args)
+      builder = kinesis.run(queue)
+    end
+  end
+
+  context "#stop" do
+    it "stops the KCL worker" do
+      expect(kinesis).to receive(:kcl_builder).with(queue).and_return(stub_builder)
+      expect(kcl_worker).to receive(:run).with(no_args)
+      expect(kcl_worker).to receive(:shutdown).with(no_args)
+      kinesis.run(queue)
+      kinesis.do_stop # do_stop calls stop internally
+    end
+  end
+
+  context "#worker_factory" do
+    it "clones the codec for each worker" do
+      worker = kinesis.worker_factory(queue).call()
+      expect(worker).to be_kind_of(LogStash::Inputs::Kinesis::Worker)
+      expect(worker.codec).to_not eq(kinesis.codec)
+      expect(worker.codec).to be_kind_of(codec.class)
+    end
+
+    it "generates a valid worker" do
+      worker = kinesis.worker_factory(queue).call()
+
+      expect(worker.codec).to be_kind_of(codec.class)
+      expect(worker.checkpoint_interval).to eq(120)
+      expect(worker.failover_time_millis).to eq(30000)
+      expect(worker.output_queue).to eq(queue)
+      expect(worker.decorator).to eq(kinesis.method(:decorate))
+      expect(worker.logger).to eq(kinesis.logger)
+    end
+  end
+
+  # these tests are heavily dependent on the current Worker::Builder
+  # implementation because its state is all private
+  context "#kcl_builder" do
+    let(:builder) { kinesis.kcl_builder(queue) }
+
+    it "sets the worker factory" do
+      expect(field(builder, "recordProcessorFactory")).to_not eq(nil)
+    end
+
+    it "sets the config" do
+      kinesis.register
+      config = field(builder, "config")
+      expect(config).to eq(kinesis.kcl_config)
+    end
+
+    it "disables metric tracking by default" do
+      expect(field(builder, "metricsFactory")).to be_kind_of(com.amazonaws.services.kinesis.metrics.impl::NullMetricsFactory)
+    end
+
+    context "cloudwatch" do
+      let(:metrics) { "cloudwatch" }
+      it "uses cloudwatch metrics if specified" do
+        # since the behaviour is enclosed on private methods it is not testable. So here
+        # the expected value can be tested, not the result associated to set this value
+        expect(field(builder, "metricsFactory")).to eq(nil)
+      end
+    end
+  end
+
+  def field(obj, name)
+    field = obj.java_class.declared_field(name)
+    field.accessible = true
+    field.value(obj)
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,17 @@
+RSpec.configure do |config|
+  config.expect_with :rspec do |expectations|
+    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
+  end
+
+  config.mock_with :rspec do |mocks|
+    mocks.verify_partial_doubles = true
+  end
+
+  config.disable_monkey_patching!
+  config.warnings = false
+  if config.files_to_run.one?
+    config.default_formatter = 'doc'
+  end
+  config.order = :random
+  Kernel.srand config.seed
+end

data/vendor/jar-dependencies/runtime-jars/logstash-input-kinesis-jordanforks_jars.rb

@@ -0,0 +1,50 @@
+# this is a generated file, to avoid over-writing it just delete this comment
+begin
+  require 'jar_dependencies'
+rescue LoadError
+  require 'com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar'
+  require 'com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar'
+  require 'com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar'
+  require 'org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar'
+  require 'com/amazonaws/jmespath-java/1.11.400/jmespath-java-1.11.400.jar'
+  require 'com/amazonaws/aws-java-sdk-s3/1.11.400/aws-java-sdk-s3-1.11.400.jar'
+  require 'com/google/guava/guava/18.0/guava-18.0.jar'
+  require 'commons-lang/commons-lang/2.6/commons-lang-2.6.jar'
+  require 'commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar'
+  require 'joda-time/joda-time/2.8.1/joda-time-2.8.1.jar'
+  require 'software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar'
+  require 'com/amazonaws/amazon-kinesis-client/1.9.2/amazon-kinesis-client-1.9.2.jar'
+  require 'com/amazonaws/aws-java-sdk-kinesis/1.11.400/aws-java-sdk-kinesis-1.11.400.jar'
+  require 'com/amazonaws/aws-java-sdk-cloudwatch/1.11.400/aws-java-sdk-cloudwatch-1.11.400.jar'
+  require 'org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar'
+  require 'com/amazonaws/aws-java-sdk-dynamodb/1.11.400/aws-java-sdk-dynamodb-1.11.400.jar'
+  require 'com/amazonaws/aws-java-sdk-core/1.11.414/aws-java-sdk-core-1.11.414.jar'
+  require 'commons-codec/commons-codec/1.10/commons-codec-1.10.jar'
+  require 'com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar'
+  require 'com/amazonaws/aws-java-sdk-kms/1.11.400/aws-java-sdk-kms-1.11.400.jar'
+  require 'com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar'
+end
+
+if defined? Jars
+  require_jar 'com.fasterxml.jackson.core', 'jackson-databind', '2.6.7.1'
+  require_jar 'com.fasterxml.jackson.core', 'jackson-core', '2.6.7'
+  require_jar 'com.fasterxml.jackson.dataformat', 'jackson-dataformat-cbor', '2.6.7'
+  require_jar 'org.apache.httpcomponents', 'httpclient', '4.5.5'
+  require_jar 'com.amazonaws', 'jmespath-java', '1.11.400'
+  require_jar 'com.amazonaws', 'aws-java-sdk-s3', '1.11.400'
+  require_jar 'com.google.guava', 'guava', '18.0'
+  require_jar 'commons-lang', 'commons-lang', '2.6'
+  require_jar 'commons-logging', 'commons-logging', '1.1.3'
+  require_jar 'joda-time', 'joda-time', '2.8.1'
+  require_jar 'software.amazon.ion', 'ion-java', '1.0.2'
+  require_jar 'com.amazonaws', 'amazon-kinesis-client', '1.9.2'
+  require_jar 'com.amazonaws', 'aws-java-sdk-kinesis', '1.11.400'
+  require_jar 'com.amazonaws', 'aws-java-sdk-cloudwatch', '1.11.400'
+  require_jar 'org.apache.httpcomponents', 'httpcore', '4.4.9'
+  require_jar 'com.amazonaws', 'aws-java-sdk-dynamodb', '1.11.400'
+  require_jar 'com.amazonaws', 'aws-java-sdk-core', '1.11.414'
+  require_jar 'commons-codec', 'commons-codec', '1.10'
+  require_jar 'com.fasterxml.jackson.core', 'jackson-annotations', '2.6.0'
+  require_jar 'com.amazonaws', 'aws-java-sdk-kms', '1.11.400'
+  require_jar 'com.google.protobuf', 'protobuf-java', '2.6.1'
+end

metadata

@@ -0,0 +1,149 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-kinesis-jordanforks
+version: !ruby/object:Gem::Version
+  version: 2.0.9
+platform: java
+authors:
+- Brian Palmer
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-10-25 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - <=
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - <=
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: jar-dependencies
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.3.4
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.3.4
+- !ruby/object:Gem::Dependency
+  name: logstash-codec-json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not
+  a stand-alone program
+email:
+- brian@codekitchen.net
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/logstash/inputs/kinesis/version.rb
+- lib/logstash/inputs/kinesis/worker.rb
+- lib/logstash/inputs/kinesis.rb
+- spec/inputs/kinesis/worker_spec.rb
+- spec/inputs/kinesis_spec.rb
+- spec/spec_helper.rb
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/amazon-kinesis-client/1.9.2/amazon-kinesis-client-1.9.2.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-cloudwatch/1.11.400/aws-java-sdk-cloudwatch-1.11.400.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-core/1.11.414/aws-java-sdk-core-1.11.414.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-dynamodb/1.11.400/aws-java-sdk-dynamodb-1.11.400.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-kinesis/1.11.400/aws-java-sdk-kinesis-1.11.400.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-kms/1.11.400/aws-java-sdk-kms-1.11.400.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-s3/1.11.400/aws-java-sdk-s3-1.11.400.jar
+- vendor/jar-dependencies/runtime-jars/com/amazonaws/jmespath-java/1.11.400/jmespath-java-1.11.400.jar
+- vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar
+- vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar
+- vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar
+- vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar
+- vendor/jar-dependencies/runtime-jars/com/google/guava/guava/18.0/guava-18.0.jar
+- vendor/jar-dependencies/runtime-jars/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar
+- vendor/jar-dependencies/runtime-jars/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
+- vendor/jar-dependencies/runtime-jars/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
+- vendor/jar-dependencies/runtime-jars/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar
+- vendor/jar-dependencies/runtime-jars/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar
+- vendor/jar-dependencies/runtime-jars/logstash-input-kinesis-jordanforks_jars.rb
+- vendor/jar-dependencies/runtime-jars/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar
+- vendor/jar-dependencies/runtime-jars/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar
+- vendor/jar-dependencies/runtime-jars/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar
+- logstash-input-kinesis.gemspec
+- CHANGELOG.md
+- README.md
+- CONTRIBUTORS
+- Gemfile
+homepage: https://github.com/logstash-plugins/logstash-input-kinesis
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+- vendor/jar-dependencies/runtime-jars
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements:
+- jar 'com.amazonaws:amazon-kinesis-client', '1.9.2'
+- jar 'com.amazonaws:aws-java-sdk-core', '1.11.414'
+rubyforge_project: 
+rubygems_version: 2.0.14.1
+signing_key: 
+specification_version: 4
+summary: Receives events through an AWS Kinesis stream
+test_files:
+- spec/inputs/kinesis/worker_spec.rb
+- spec/inputs/kinesis_spec.rb
+- spec/spec_helper.rb