aws-kclrb 1.0.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: d55dfffb55a3d32554c63e4636b5c4bcd7612145
-  data.tar.gz: a6257f914079ce6b9e28bfe03faba78d30a08e94
+SHA256:
+  metadata.gz: ef941c4d394cb56204cbc3bd8fde1ac88eecf7cb131cfc9f3827c3d418451629
+  data.tar.gz: aad7a5ddfbd195755f49af66b74a097e2de513ee1763e7b97509619542d0aa92
 SHA512:
-  metadata.gz: fb02f6b1830d16ec02031c6d49d07487a54af4cadfe73fe774fd30f4ce3b11d5cbe369b795767954020fb3ad0387bd852d611bc2d54656d676b0f8dc9cf2749f
-  data.tar.gz: cfb93c0609cd2b76065347622adb49372322c381199f81debd6f39593197a5b72d15e14766778695c57d7bc7ea1a35a0704be44bfae135a8dfeaa2b3aaf0b710
+  metadata.gz: d28c8c92de60f53d4b506ad3736b89396f2359c61524f088212d6a902f5c3856de7f255ebd28227ff2a97cf2752ca9b14604829d114b96373015ee15ac74840b
+  data.tar.gz: 98149d6c3f10b61f8f1072b74f6e09fdc55059ceac5e669a060a14cb8301d7ae147d77bef1e387f01652849060106c6dac1f723f34c8f29cb37c5265742018ea

data/README.md

@@ -15,17 +15,25 @@ executable. A record processor in Ruby typically looks something like:
 
     require 'aws/kclrb'
 
-    class SampleRecordProcessor < Aws::KCLrb::RecordProcessorBase
-      def init_processor(shard_id)
+    class SampleRecordProcessor < Aws::KCLrb::V2::RecordProcessorBase
+      def init_processor(initialize_input)
         # initialize
       end
 
-      def process_records(records, checkpointer)
+      def process_records(process_records_input)
         # process batch of records
       end
 
-      def shutdown(checkpointer, reason)
-        # cleanup
+      def lease_lost(lease_lost_input)
+        # lease was lost, cleanup
+      end
+
+      def shard_ended(shard_ended_input)
+        # shard has ended, cleanup
+      end
+
+      def shutdown_requested(shutdown_requested_input)
+        # shutdown has been requested
       end
     end
 
@@ -69,10 +77,11 @@ The sample application consists of two components:
 The following defaults are used in the sample application:
 
 * *Stream name*: `kclrbsample`
+* *Region*: `us-east-1`
 * *Number of shards*: 2
 * *Amazon KCL application name*: `RubyKCLSample`
-* *Amazon DynamoDB table for Amazon KCL application*: `RubyKCLSample`
-* *Sample application output directory*: `/tmp/kclrbsample/`
+* *Amazon DynamoDB table for KCL application*: `RubyKCLSample`
+* *Amazon CloudWatch metrics namespace for KCL application*: `RubyKCLSample`
 
 ### Running the Data Producer
 
@@ -105,7 +114,7 @@ To run the data processor, run the following commands:
 
 ```sh
     cd samples
-    rake run
+    rake run properties_file=sample.properties
 ```
 
 #### Notes
@@ -118,6 +127,7 @@ To run the data processor, run the following commands:
   * `executableName = samples/sample_kcl.rb`
   * `streamName = kclrbsample`
   * `applicationName = RubyKCLSample`
+  * `regionName = us-east-1`
 
 ### Cleaning Up
 
@@ -126,7 +136,7 @@ create a real DynamoDB table to track the Amazon KCL application state, thus pot
 incurring AWS costs. Once done, you can log in to AWS management console and delete these
 resources. Specifically, the sample application will create in your default AWS region
 
-* an *Amazon Kinesis stream* named `kclrbsample`
+* an *Amazon Kinesis Data Stream* named `kclrbsample`
 * an *Amazon DynamoDB table* named `RubyKCLSample`
 
 ## Running on Amazon EC2
@@ -147,7 +157,7 @@ Amazon Linux can be found at `/usr/bin/java` and should be 1.7 or greater.
     cd kclrb/samples
     rake run_producer
     # ... and in another terminal
-    rake run
+    rake run properties_file=sample.properties
 ```
 
 ## Under the Hood - What You Should Know about Amazon KCL's [MultiLangDaemon][multi-lang-daemon]
@@ -177,6 +187,33 @@ all languages.
 
 ## Release Notes
 
+### Release 2.0.0 (February 26, 2019)
+* Added support for [Enhanced Fan-Out](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/).  
+  Enhanced Fan-Out provides dedicated throughput per stream consumer, and uses an HTTP/2 push API (SubscribeToShard) to deliver records with lower latency.
+* Updated the Amazon Kinesis Client Library for Java to version 2.1.2.
+  * Version 2.1.2 uses 4 additional Kinesis API's  
+    __WARNING: These additional API's may require updating any explicit IAM policies__
+    * [`RegisterStreamConsumer`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_RegisterStreamConsumer.html)
+    * [`SubscribeToShard`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html)
+    * [`DescribeStreamConsumer`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamConsumer.html)
+    * [`DescribeStreamSummary`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamSummary.html)
+  * For more information about Enhanced Fan-Out with the Amazon Kinesis Client Library please see the [announcement](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/) and [developer documentation](https://docs.aws.amazon.com/streams/latest/dev/introduction-to-enhanced-consumers.html).
+* Added version 2 of the [`RecordProcessorBase`](https://github.com/awslabs/amazon-kinesis-client-ruby/blob/d5c2bbafb232b5e1ab947980a0bd8505c87978f9/lib/aws/kclrb/record_processor.rb#L102) which supports the new `ShardRecordProcessor` interface
+  * The `shutdown` method from version 1 has been replaced by `lease_lost` and `shard_ended`.
+  * Added the `lease_lost` method which is invoked when a lease is lost.  
+    `lease_lost` replaces `shutdown(checkpointer, 'ZOMBIE')`.
+  * Added the `shard_ended` method which is invoked when all records from a split or merge have been processed.  
+    `shard_ended` replaces `shutdown(checkpointer, 'TERMINATE')`.
+  * Added an optional method, `shutdown_requested`, which provides the record processor a last chance to checkpoint during the Amazon Kinesis Client Library shutdown process before the lease is canceled.  
+    * To control how long the Amazon Kinesis Client Library waits for the record processors to complete shutdown, add `timeoutInSeconds=<seconds to wait>` to your properties file.
+* Updated the AWS Java SDK version to 2.4.0
+* MultiLangDaemon now provides logging using Logback.
+  * MultiLangDaemon supports custom configurations for logging via a Logback XML configuration file.
+  * The example [Rakefile](https://github.com/awslabs/amazon-kinesis-client-ruby/blob/master/samples/Rakefile) supports setting the logging configuration by adding `log_configuration=<log configuration file>` to the Rake command line.
+
+### Release 1.0.1 (January 19, 2017)
+* Upgraded to use version 1.7.2 of the [Amazon Kinesis Client library][amazon-kcl-github]
+
 ### Release 1.0.0 (December 30, 2014)
 * **aws-kclrb** gem which exposes an interface to allow implementation of record processors in Ruby
   using the Amazon KCL's [MultiLangDaemon][multi-lang-daemon]

data/VERSION

@@ -1 +1 @@
-1.0.1
+2.0.0

data/lib/aws/kclrb.rb

@@ -13,4 +13,5 @@
 #  permissions and limitations under the License.
 
 require 'aws/kclrb/record_processor'
+require 'aws/kclrb/messages'
 require 'aws/kclrb/kcl_process'

data/lib/aws/kclrb/kcl_process.rb

@@ -14,13 +14,16 @@
 
 require 'aws/kclrb/io_proxy'
 require 'aws/kclrb/checkpointer'
+require 'aws/kclrb/messages'
+require 'aws/kclrb/record_processor'
 
 module Aws
   module KCLrb
     # Error raised if the {KCLProcess} received an input action that it
     # could not parse or it could not handle.
-    class MalformedAction < RuntimeError; end
-  
+    class MalformedAction < RuntimeError;
+    end
+
     # Entry point for a KCL application in Ruby.
     #
     # Implementers of KCL applications in Ruby should instantiate this
@@ -31,12 +34,16 @@ module Aws
       # @param input [IO] An `IO`-like object to read input lines from.
       # @param output [IO] An `IO`-like object to write output lines to.
       # @param error [IO] An `IO`-like object to write error lines to.
-      def initialize(processor, input=$stdin, output=$stdout, error=$stderr)
-        @processor = processor
+      def initialize(processor, input = $stdin, output = $stdout, error = $stderr)
+        if processor.version == 1
+          @processor = Aws::KCLrb::V2::V2ToV1Adapter.new(processor)
+        else
+          @processor = processor
+        end
         @io_proxy = IOProxy.new(input, output, error)
         @checkpointer = CheckpointerImpl.new(@io_proxy)
       end
-  
+
       # Starts this KCL processor's main loop.
       def run
         action = @io_proxy.read_action
@@ -45,9 +52,9 @@ module Aws
           action = @io_proxy.read_action
         end
       end
-  
+
       private
-      
+
       # @api private
       # Parses an input action and invokes the appropriate method of the
       # record processor.
@@ -63,11 +70,20 @@ module Aws
         action_name = action.fetch('action')
         case action_name
         when 'initialize'
-          dispatch_to_processor(:init_processor, action.fetch('shardId'))
+          dispatch_to_processor(:init_processor,
+                                Aws::KCLrb::V2::InitializeInput.new(action.fetch('shardId'),
+                                                                    action.fetch('sequenceNumber')))
         when 'processRecords'
-          dispatch_to_processor(:process_records, action.fetch('records'), @checkpointer)
-        when 'shutdown'
-          dispatch_to_processor(:shutdown, @checkpointer, action.fetch('reason'))
+          dispatch_to_processor(:process_records,
+                                Aws::KCLrb::V2::ProcessRecordsInput.new(action.fetch('records'),
+                                                                        action.fetch('millisBehindLatest'),
+                                                                        @checkpointer))
+        when 'leaseLost'
+          dispatch_to_processor(:lease_lost, Aws::KCLrb::V2::LeaseLostInput.new)
+        when 'shardEnded'
+          dispatch_to_processor(:shard_ended, Aws::KCLrb::V2::ShardEndedInput.new(@checkpointer))
+        when 'shutdownRequested'
+          dispatch_to_processor(:shutdown_requested, Aws::KCLrb::V2::ShutdownRequestedInput.new(@checkpointer))
         else
           raise MalformedAction.new("Received an action which couldn't be understood. Action was '#{action}'")
         end
@@ -75,7 +91,7 @@ module Aws
       rescue KeyError => ke
         raise MalformedAction.new("Action '#{action}': #{ke.message}")
       end
-  
+
       # @api private
       # Calls the specified method on the record processor, and handles
       # any resulting exceptions by writing to the error stream.
@@ -89,7 +105,7 @@ module Aws
         #   of issue.
         @io_proxy.write_error(processor_error)
       end
-  
+
     end
   end
 end

data/lib/aws/kclrb/messages.rb

@@ -0,0 +1,53 @@
+module Aws
+  module KCLrb
+    module V2
+      # @abstract
+      # Input object for RecordProcessorBase#init_processor method.
+      class InitializeInput
+        attr_reader :shard_id, :sequence_number
+
+        def initialize(shard_id, sequence_number)
+          @shard_id = shard_id
+          @sequence_number = sequence_number
+        end
+      end
+
+      # @abstract
+      # Input object for RecordProcessorBase#process_records method.
+      class ProcessRecordsInput
+        attr_reader :records, :millis_behind_latest, :checkpointer
+
+        def initialize(records, millis_behind_latest, checkpointer = nil)
+          @records = records
+          @millis_behind_latest = millis_behind_latest
+          @checkpointer = checkpointer
+        end
+      end
+
+      # @abstract
+      # Input object for RecordProcessorBase#lease_lost method.
+      class LeaseLostInput
+      end
+
+      # @abstract
+      # Input object forRecordProcessorBase#shard_ended method.
+      class ShardEndedInput
+        attr_reader :checkpointer
+
+        def initialize(checkpointer = nil)
+          @checkpointer = checkpointer
+        end
+      end
+
+      # @abstract
+      # Input object for RecordProcessorBase#shutdown_requested method.
+      class ShutdownRequestedInput
+        attr_reader :checkpointer
+
+        def initialize(checkpointer = nil)
+          @checkpointer = checkpointer
+        end
+      end
+    end
+  end
+end

data/lib/aws/kclrb/record_processor.rb

@@ -1,5 +1,5 @@
 #
-#  Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#  Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 # 
 #  Licensed under the Amazon Software License (the "License").
 #  You may not use this file except in compliance with the License.
@@ -32,7 +32,7 @@ module Aws
       def init_processor(shard_id)
         fail NotImplementedError.new
       end
-  
+
       # @abstract
       # Called by a KCLProcess with a list of records to be processed and a checkpointer
       # which accepts sequence numbers from the records to indicate where in the stream
@@ -52,7 +52,7 @@ module Aws
       def process_records(records, checkpointer)
         fail NotImplementedError.new
       end
-  
+
       # @abstract
       # Called by a KCLProcess instance to indicate that this record processor
       # should shutdown. After this is called, there will be no more calls to
@@ -72,6 +72,118 @@ module Aws
       def shutdown(checkpointer, reason)
         fail NotImplementedError.new
       end
+
+      # @abstract
+      # Called by a KCLProcess instance to indicate that this record processor
+      # is requesting a shutdown. This method should be overriden if required.
+      #
+      # @param checkpointer [Checkpointer] A checkpointer which accepts a sequence
+      #   number or no parameters.
+      def shutdown_requested(checkpointer)
+      end
+
+      def version
+        1
+      end
+    end
+
+    module V2
+      # @abstract
+      # Base class for implementing a record processor.
+      #
+      # A `RecordProcessor` processes a shard in a stream. See {https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/IRecordProcessor.java the corresponding KCL interface}.
+      # Its methods will be called as follows:
+      #
+      # 1. {#init_processor} will be called once
+      # 2. {#process_records} will be called zero or more times
+      # 3. {#lease_lost} will be called zero to one time
+      # 4. {#shard_ended} will be called zero or more times
+      # 5. {#shutdown_requested} will be called zero to one time
+      class RecordProcessorBase
+        # @abstract
+        # Called once by a KCLProcess before any calls to process_records.
+        #
+        # @param initialize_input [InitializeInput] Initialize processor input
+        #   object
+        def init_processor(initialize_input)
+          fail NotImplementedError.new
+        end
+
+        # @abstract
+        # Called by a KCLProcess with a list of records to be processed and a
+        # checkpointer which accepts sequence numbers from the records to
+        # indicate where in the stream to checkpoint.
+        #
+        # @param record_processor_input [RecordProcessorInput] Process records
+        #   input object
+        def process_records(process_records_input)
+          fail NotImplementedError.new
+        end
+
+        # @abstract
+        # Called by a KCLProcess instance to indicate that this record processor
+        # should shutdown. After this is called, there will be no more calls to
+        # any other methods of this record processor.
+        #
+        # @param lease_lost_input [LeaseLostInput] Lease lost input object
+        #
+        #    - Clients should not checkpoint because there is possibly another
+        #      record processor which has acquired the lease for this shard.
+        def lease_lost(lease_lost_input)
+          fail NotImplementedError.new
+        end
+
+        # @abstract
+        # Called by a KCLProcess instance to indicate that this record processor
+        # should shutdown. After this is called, there will be no more calls to
+        # any other methods of this record processor.
+        #
+        # @param shard_ended_input [ShardEndedInput] Shard ended input object
+        #
+        #   - Clients need to checkpoint at this time.
+        def shard_ended(shard_ended_input)
+          fail NotImplementedError.new
+        end
+
+        # @abstract
+        # Called by a KCLProcess instance to indicate that this record processor
+        # is requesting a shutdown. This method should be overriden if required.
+        #
+        # @param shutdown_requested_input [ShutdownRequestedInput]
+        def shutdown_requested(shutdown_requested_input)
+        end
+
+        def version
+          2
+        end
+      end
+
+      class V2ToV1Adapter < Aws::KCLrb::V2::RecordProcessorBase
+        def initialize(processor)
+          @processor = processor
+        end
+
+        def init_processor(initialize_input)
+          @processor.init_processor(initialize_input.shard_id)
+        end
+
+        def process_records(process_records_input)
+          @processor.process_records(process_records_input.records,
+                                     process_records_input.checkpointer)
+        end
+
+        def lease_lost(lease_lost_input)
+          @processor.shutdown(nil, 'ZOMBIE')
+        end
+
+        def shard_ended(shard_ended_input)
+          @processor.shutdown(shard_ended_input.checkpointer, 'TERMINATE')
+        end
+
+        def shutdown_requested(shutdown_requested_input)
+          @processor.shutdown_requested(shutdown_requested_input.checkpointer)
+        end
+      end
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: aws-kclrb
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 2.0.0
 platform: ruby
 authors:
 - Amazon Web Services
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-01-19 00:00:00.000000000 Z
+date: 2019-02-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: multi_json
@@ -40,6 +40,7 @@ files:
 - lib/aws/kclrb/checkpointer.rb
 - lib/aws/kclrb/io_proxy.rb
 - lib/aws/kclrb/kcl_process.rb
+- lib/aws/kclrb/messages.rb
 - lib/aws/kclrb/record_processor.rb
 - spec/checkpointer_spec.rb
 - spec/io_proxy_spec.rb
@@ -65,7 +66,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 2.7.7
 signing_key: 
 specification_version: 4
 summary: Amazon Kinesis Client Library for Ruby