telekinesis 2.0.1-java → 3.0.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: de0cb955b6e18af3b78556bb76547de2da174e79
-  data.tar.gz: f46ac116e56b5bad319e1030dabca0329ae0eebb
+  metadata.gz: b4fa22c623dd098fff1cefccc7b48d9c543858b3
+  data.tar.gz: b3063eaed5976b9744ecc6eb13fa0fa05c42a184
 SHA512:
-  metadata.gz: 79071fbac4f1136a484c53c00f0b58ab49ff133925ffc9f6c8b17c820f866b3a144536c2fc16d7f5095c73b6459e73ddfbd90516abb6208405f932f2774b426e
-  data.tar.gz: 301f45a6b0c9367172693d82da262132226dfb754665e264884f4f8ea2b4658eff4108f326e44bdf67ce790885a79d7b5f55ac52522b22d0030976237147df87
+  metadata.gz: 58272a8819dae59636c4c0827da20d244675d2a0404795d2cb33a724643cd31c195c0a692623545483563d7a920257ea8536e0496e8f8b0d938c4fba824c0752
+  data.tar.gz: 07ee614c9d050cd3b37e6d0a4d038dc3efea8e86704d4a2516b9c91ceb0c7efcbfc62d700f84b8f9234385fbd9152411253f0b286dd4c7f0c7470b4bd6627542

data/README.md

@@ -7,7 +7,7 @@
     - [SyncProducer](#syncproducer)
     - [AsyncProducer](#asyncproducer)
   - [Consumers](#consumers)
-    - [DistributedConsumer](#distributedconsumer)
+    - [KCL](#kcl)
       - [Client State](#client-state)
       - [Errors while processing records](#errors-while-processing-records)
       - [Checkpoints and `INITIAL_POSITION_IN_STREAM`](#checkpoints-and-initial_position_in_stream)
@@ -167,15 +167,15 @@ producer = Telekinesis::Producer::AsyncProducer.create(
 
 ## Consumers
 
-### DistributedConsumer
+### KCL
 
-The `DistributedConsumer` is a wrapper around Amazon's [Kinesis Client Library
-(also called the KCL)](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html#kinesis-record-processor-overview-kcl).
+`Telekinesis::Consumer::KCL` is a wrapper around Amazon's [Kinesis Client
+Library (also called the KCL)](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html#kinesis-record-processor-overview-kcl).
 
-Each `DistributedConsumer` is considered to be part of a group of consumers that
-make up an _application_. An application can be running on any number of hosts.
-Consumers identify themself uniquely within an application by specifying a
-`worker_id`.
+Each KCL instance is part of a group of consumers that make up an
+_application_. An application can be running on any number of hosts in any
+number of processes.  Consumers identify themself uniquely within an
+application by specifying a `worker_id`.
 
 All of the consumers within an application attempt to distribute work evenly
 between themselves by coordinating through a DynamoDB table. This coordination
@@ -183,12 +183,13 @@ ensures that a single consumer processes each shard, and that if one consumer
 fails for any reason, another consumer can pick up from the point at which it
 last checkpointed.
 
-This is all part of the KCL! Telekinesis just makes it easier to use from JRuby.
+This is all part of the official AWS library! Telekinesis just makes it easier
+to use from JRuby.
 
-Each `DistributedConsumer` has to know how to process all the data it's
+Each client has to know how to process all the data it's
 retreiving from Kinesis. That's done by creating a [record
 processor](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html#kinesis-record-processor-implementation-interface-java)
-and telling a `DistributedConsumer` how to create a processor when it becomes
+and telling a `KCL` how to create a processor when it becomes
 responsible for a shard.
 
 We highly recommend reading the [official
@@ -213,13 +214,15 @@ Defining and creating a simple processor might look like:
 require 'telekinesis'
 
 class MyProcessor
-  def init(shard_id)
-    @shard_id = shard_id
+  def init(init_input)
+    @shard_id = init_input.shard_id
     $stderr.puts "Started processing #{@shard_id}"
   end
 
-  def process_records(records, checkpointer)
-    records.each {|r| puts "key=#{r.partition_key} value=#{String.from_java_bytes(r.data.array)}" }
+  def process_records(process_records_input)
+    process_records_input.records.each do
+      |r| puts "key=#{r.partition_key} value=#{String.from_java_bytes(r.data.array)}"
+    end
   end
 
   def shutdown
@@ -227,7 +230,7 @@ class MyProcessor
   end
 end
 
-Telekinesis::Consumer::DistributedConsumer.new(stream: 'some-events', app: 'example') do
+Telekinesis::Consumer::KCL.new(stream: 'some-events', app: 'example') do
   MyProcessor.new
 end
 ```
@@ -240,8 +243,8 @@ processor.
 ```ruby
 require 'telekinesis'
 
-Telekinesis::Consumer::DistributedConsumer.new(stream: 'some-events', app: 'example') do
-  Telekinesis::Consumer::Block.new do |records, checkpointer|
+Telekinesis::Consumer::KCL.new(stream: 'some-events', app: 'example') do
+  Telekinesis::Consumer::Block.new do |records, checkpointer, millis_behind|
     records.each {|r| puts "key=#{r.partition_key} value=#{String.from_java_bytes(r.data.array)}" }
   end
 end
@@ -290,12 +293,12 @@ used to checkpoint all records that have been passed to the processor so far
 (by just calling `checkpointer.checkpoint`) or up to a particular sequence
 number (by calling `checkpointer.checkpoint(record.sequence_number)`).
 
-While a `DistributedConsumer` can be initialized with an
-`:initial_position_in_stream` option, any existing checkpoint for a shard will
-take precedent over that value. Furthermore, any existing STATE in DynamoDB will
-take precedent, so if you start a consumer with `initial_position_in_stream: 'LATEST'`
-and then restart with `initial_position_in_stream: 'TRIM_HORIZON'` you still end
-up starting from `LATEST`.
+While a `KCL` consumer can be initialized with an `:initial_position_in_stream`
+option, any existing checkpoint for a shard will take precedent over that
+value. Furthermore, any existing STATE in DynamoDB will take precedent, so if
+you start a consumer with `initial_position_in_stream: 'LATEST'` and then
+restart with `initial_position_in_stream: 'TRIM_HORIZON'` you still end up
+starting from `LATEST`.
 
 ## Java client logging
 

data/ext/pom.xml

@@ -4,7 +4,7 @@
 
     <groupId>com.kickstarter</groupId>
     <artifactId>telekinesis</artifactId>
-    <version>2.0.1</version>
+    <version>3.0.0</version>
 
     <!-- ================================================================== -->
     <build>

data/ext/src/main/java/com/kickstarter/jruby/Telekinesis.java

@@ -1,17 +1,25 @@
 package com.kickstarter.jruby;
 
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
+import com.amazonaws.services.kinesis.AmazonKinesis;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.leases.exceptions.LeasingException;
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLeaseManager;
 import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
 import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
 import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker;
-import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownReason;
+import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownInput;
 import com.amazonaws.services.kinesis.model.Record;
 
-import java.util.List;
-
 /**
  * A shim that makes it possible to use the Kinesis Client Library from JRuby.
  * Without the shim, {@code initialize} method in
- * {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor}
+ * {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor}
  * conflicts with the special {@code initialize} method in Ruby. The shim
  * interface renames {@code initialize} to {@code init}.
  * <p />
@@ -36,67 +44,71 @@ public class Telekinesis {
      * {@link IRecordProcessorFactory}.
      */
     public static Worker newWorker(final KinesisClientLibConfiguration config, final IRecordProcessorFactory factory) {
-        return new Worker(new com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory() {
+        com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory v2Factory = new com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory() {
             @Override
-            public com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor createProcessor() {
+            public com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor createProcessor() {
                 return new RecordProcessorShim(factory.createProcessor());
             }
-        }, config);
+        };
+        return new Worker.Builder()
+            .recordProcessorFactory(v2Factory)
+            .config(config)
+            .build();
     }
 
     // ========================================================================
     /**
      * A shim that wraps a {@link IRecordProcessor} so it can get used by the KCL.
      */
-    private static class RecordProcessorShim implements com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor {
+    private static class RecordProcessorShim implements com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor {
         private final IRecordProcessor underlying;
 
         public RecordProcessorShim(final IRecordProcessor underlying) { this.underlying = underlying; }
 
         @Override
-        public void initialize(final String shardId) {
-            underlying.init(shardId);
+        public void initialize(final InitializationInput initializationInput) {
+            underlying.init(initializationInput);
         }
 
         @Override
-        public void processRecords(final List<Record> records, final IRecordProcessorCheckpointer checkpointer) {
-            underlying.processRecords(records, checkpointer);
+        public void processRecords(final ProcessRecordsInput processRecordsInput) {
+            underlying.processRecords(processRecordsInput);
         }
 
         @Override
-        public void shutdown(final IRecordProcessorCheckpointer checkpointer, final ShutdownReason reason) {
-            underlying.shutdown(checkpointer, reason);
+        public void shutdown(final ShutdownInput shutdownInput) {
+            underlying.shutdown(shutdownInput);
         }
     }
 
     /**
-     * A parallel {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor}
+     * A parallel {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor}
      * that avoids naming conflicts with reserved words in Ruby.
      */
     public static interface IRecordProcessor {
         /**
-         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor#initialize(String)
+         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor#initialize(InitializationInput)
          */
-        void init(final String shardId);
+        void init(InitializationInput initializationInput);
 
         /**
-         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor#processRecords(List, IRecordProcessorCheckpointer)
+         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor#processRecords(ProcessRecordsInput)
          */
-        void processRecords(List<Record> records, IRecordProcessorCheckpointer checkpointer);
+        void processRecords(ProcessRecordsInput processRecordsInput);
 
         /**
-         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor#shutdown(IRecordProcessorCheckpointer, ShutdownReason)
+         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor#shutdown(ShutdownInput)
          */
-        void shutdown(IRecordProcessorCheckpointer checkpointer, ShutdownReason reason);
+        void shutdown(ShutdownInput shutdownInput);
     }
 
     /**
-     * A parallel {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory}
+     * A parallel {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory}
      * for {@link IRecordProcessor}.
      */
     public static interface IRecordProcessorFactory {
         /**
-         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory#createProcessor()
+         * @see com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory#createProcessor()
          */
         IRecordProcessor createProcessor();
     }

data/lib/telekinesis/consumer/base_processor.rb

@@ -4,9 +4,9 @@ module Telekinesis
     # IRecordProcessor methods. Override it to implement simple IRecordProcessors
     # that don't need to do anything special on init or shutdown.
     class BaseProcessor
-      def init(shard_id); end
-      def process_records(records, checkpointer); end
-      def shutdown(checkpointer, reason); end
+      def init(initialization_input); end
+      def process_records(process_records_input); end
+      def shutdown(shutdown_input); end
     end
   end
 end

data/lib/telekinesis/consumer/block.rb

@@ -4,8 +4,10 @@ module Telekinesis
     # quickly define a consumer.
     #
     # Telekinesis::Consumer::Worker.new(stream: 'my-stream', app: 'tail') do
-    #   Telekinesis::Consumer::Block.new do |records, checkpointer|
+    #   Telekinesis::Consumer::Block.new do |records, checkpointer, millis_behind_latest|
     #     records.each {|r| puts r}
+    #     $stderr.puts "#{millis_behind_latest} ms behind"
+    #     checkpointer.checkpoint
     #   end
     # end
     class Block < BaseProcessor
@@ -14,8 +16,8 @@ module Telekinesis
         @block = block
       end
 
-      def process_records(records, checkpointer)
-        @block.call(records, checkpointer)
+      def process_records(input)
+        @block.call(input.records, input.checkpointer, input.millis_behind_latest)
       end
     end
   end

data/lib/telekinesis/consumer/{distributed_consumer.rb → kcl.rb}

@@ -3,29 +3,31 @@ module Telekinesis
     java_import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
     java_import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
 
-    class DistributedConsumer
-      # Create a new consumer that consumes data from a Kinesis stream.
-      # DistributedConsumers use DynamoDB to register as part of the same
-      # application and evenly distribute work between them. See the
-      # AWS Docs for more information:
+    class KCL
+      # Create a new consumer that consumes data from a Kinesis stream using the
+      # AWS Kinesis Client Library.
+      #
+      # The KCL uses DynamoDB to register clients as part of the an application
+      # and evenly distribute work between all of the clients registered for
+      # the same application. See the AWS Docs for more information:
       #
       # http://docs.aws.amazon.com/kinesis/latest/dev/developing-consumer-apps-with-kcl.html
       #
-      # DistributedConsumers are configured with a hash. The Kinesis `:stream`
-      # to consume from is required.
+      # KCLs are configured with a hash. The Kinesis `:stream` to consume from
+      # is required.
       #
-      # DistribtuedConsumers operate in groups. All consumers with the same
-      # `:app` id use dynamo to attempt to distribute work evenly among
-      # themselves. The `:worker_id` is used to distinguish individual clients
-      # (`:worker_id` defaults to the current hostname. If you plan to run more
-      # than one DistributedConsumer in the same `:app` per host, make sure you
-      # set this to something unique!).
+      # KCL clients operate in groups. All consumers with the same `:app` id use
+      # DynamoDB to attempt to distribute work evenly among themselves. The
+      # `:worker_id` is used to distinguish individual clients (`:worker_id`
+      # defaults to the current hostname. If you plan to run more than one KCL
+      # client in the same `:app` on the same host, make sure you set this to
+      # something unique!).
       #
-      # Any other valid KCL Worker `:options` may be passed as a hash.
+      # Any other valid KCL Worker `:options` may be passed as a nested hash.
       #
       # For example, to configure a `tail` app on `some-stream` and use the
       # default `:worker_id`, you might pass the following configuration to your
-      # DistributedConsumer.
+      # KCL.
       #
       #     config = {
       #       app: 'tail',
@@ -33,12 +35,12 @@ module Telekinesis
       #       options: {initial_position_in_stream: 'TRIM_HORIZON'}
       #     }
       #
-      # To actually process the stream, a DistribtuedConsumer creates
-      # record processors. These are objects that correspond to the KCL's
-      # RecordProcessor interface - processors must implement `init`,
-      # `process_records`, and `shutdown` methods.
+      # To actually process the stream, a KCL client creates record processors.
+      # These are objects that correspond to the KCL's RecordProcessor
+      # interface - processors must implement `init`, `process_records`, and
+      # `shutdown` methods.
       #
-      # http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html#kinesis-record-processor-implementation-interface-java
+      # http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html#kcl-java-interface-v2
       #
       # To specify which record processor to create, pass a block to your
       # distribtued consumer that returns a new record processor. This block
@@ -47,17 +49,21 @@ module Telekinesis
       #
       # Telekinesis provides a BaseProcessor that implements no-op versions
       # of all of the required methods to make writing quick processors easier
-      # and a Block processor that executes the specified block every time
+      # and a Block processor that executes the given block every time
       # `process_records` is called.
       #
-      # To write a stream tailer, you might use Block as follows:
+      # To write a simple stream tailer, you might use Block as follows:
       #
-      #     Telekinesis::Consumer::DistributedConsumer.new(config) do
-      #       Telekinesis::Consumer::Block do |records, _|
-      #         records.each {|r| puts r}
+      #     kcl_worker = Telekinesis::Consumer::KCL.new(config) do
+      #       Telekinesis::Consumer::BlockProcessor.new do |records, checkpointer, millis_behind_latest|
+      #         records.each{|r| puts r}
+      #         $stderr.puts "#{millis_behind_latest} ms behind"
+      #         checkpointer.checkpoint
       #       end
       #     end
       #
+      #     kcl_worker.run
+      #
       def initialize(config, &block)
         raise ArgumentError, "No block given!" unless block_given?
         kcl_config = self.class.build_config(config)

data/lib/telekinesis/consumer.rb

@@ -1,3 +1,3 @@
-require "telekinesis/consumer/distributed_consumer"
+require "telekinesis/consumer/kcl"
 require "telekinesis/consumer/base_processor"
 require "telekinesis/consumer/block"

data/lib/telekinesis/version.rb

@@ -1,3 +1,3 @@
 module Telekinesis
-  VERSION = '2.0.1'
+  VERSION = '3.0.0'
 end

data/telekinesis.gemspec

@@ -12,7 +12,6 @@ Gem::Specification.new do |spec|
   spec.platform     = "java"
   spec.files        = `git ls-files`.split($/) + Dir.glob("lib/telekinesis/*.jar")
   spec.require_paths = ["lib"]
-  spec.add_dependency "aws-sdk"
 
   spec.add_development_dependency "rake"
   spec.add_development_dependency "nokogiri"

metadata

@@ -1,29 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: telekinesis
 version: !ruby/object:Gem::Version
-  version: 2.0.1
+  version: 3.0.0
 platform: java
 authors:
 - Ben Linsay
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2015-10-22 00:00:00.000000000 Z
+date: 2015-10-28 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - '>='
-      - !ruby/object:Gem::Version
-        version: '0'
-  name: aws-sdk
-  prerelease: false
-  type: :runtime
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - '>='
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
@@ -98,11 +84,10 @@ files:
 - lib/telekinesis/aws.rb
 - lib/telekinesis/aws/client_adapter.rb
 - lib/telekinesis/aws/java_client_adapter.rb
-- lib/telekinesis/aws/ruby_client_adapter.rb
 - lib/telekinesis/consumer.rb
 - lib/telekinesis/consumer/base_processor.rb
 - lib/telekinesis/consumer/block.rb
-- lib/telekinesis/consumer/distributed_consumer.rb
+- lib/telekinesis/consumer/kcl.rb
 - lib/telekinesis/java_util.rb
 - lib/telekinesis/logging/java_logging.rb
 - lib/telekinesis/logging/ruby_logger_handler.rb
@@ -121,7 +106,7 @@ files:
 - test/producer/test_helper.rb
 - test/producer/test_sync_producer.rb
 - test/test_helper.rb
-- lib/telekinesis/telekinesis-2.0.1.jar
+- lib/telekinesis/telekinesis-3.0.0.jar
 homepage: https://github.com/kickstarter/telekinesis
 licenses: []
 metadata: {}

data/lib/telekinesis/aws/ruby_client_adapter.rb

@@ -1,40 +0,0 @@
-module Telekinesis
-  module Aws
-    # A ClientAdapter that wraps the ruby aws-sdk gem (version 2).
-    #
-    # Since the aws-sdk gem does not appear to be thread-safe, this adapter
-    # should not be considered thread safe.
-    class RubyClientAdapter < ClientAdapter
-      # Build a new client adapter. Credentials are passed directly to the
-      # constructor for Aws::Kinesis::Client.
-      #
-      # See: http://docs.aws.amazon.com/sdkforruby/api/Aws/Kinesis/Client.html
-      def self.build(credentials)
-        new(Aws::Kinesis::Client.new(credentials))
-      end
-
-      def put_record(stream, key, value)
-        @client.put_record(stream: stream, partition_key: key, data: value)
-      rescue Aws::Errors::ServiceError => e
-        raise KinesisError.new(e)
-      end
-
-      protected
-
-      def do_put_records(stream, items)
-        @client.put_records(build_put_records_request(stream, items)).flat_map do |page|
-          page.records
-        end
-      rescue Aws::Errors::ServiceError => e
-        raise KinesisError.new(e)
-      end
-
-      def build_put_records_request(stream, items)
-        {
-          stream: stream,
-          records: items.map{|k, v| {partition_key: k, data: v}}
-        }
-      end
-    end
-  end
-end