amazon-kinesis-client-ruby 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 36265265083e58844138d431b5d5e8d0d7828008
4
+ data.tar.gz: b479c46db0375672f956e94f3b77e232c0f33db0
5
+ SHA512:
6
+ metadata.gz: c2888132cf9830a89a5217d2878aaa12e6d899a75fb48cf4c703292eb2b7257115e257e7729343de5d2c065e5cc718e2601869e26d16cc8d086f33c6844d974a
7
+ data.tar.gz: 26c9dfa9195f7c570777c92b5c4b4506b507a450e1c9d49c14a33b50db10d6a993f196f677430a9cc05fe86807683f2712f028e49973f3b07625523cf299d001
@@ -0,0 +1,15 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
10
+ *.bundle
11
+ *.so
12
+ *.o
13
+ *.a
14
+ mkmf.log
15
+ .ruby-version
@@ -0,0 +1,26 @@
1
+ AllCops:
2
+ Include:
3
+ - '**/Rakefile'
4
+ - '**/*.gemspec'
5
+
6
+ Style/Documentation:
7
+ Enabled: false
8
+
9
+ Style/Blocks:
10
+ Enabled: true
11
+ Exclude:
12
+ - spec/**/*_spec.rb
13
+
14
+ Style/WordArray:
15
+ Enabled: true
16
+ Exclude:
17
+ - spec/**/*_spec.rb
18
+
19
+ Style/Encoding:
20
+ Enabled: false
21
+
22
+ Style/MethodDefParentheses:
23
+ Enabled: false
24
+
25
+ Metrics/LineLength:
26
+ Max: 120
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in amazon-kinesis-client-ruby.gemspec
4
+ gemspec
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2014 soloman
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,226 @@
1
+ Amazon Kinesis Client Library for Ruby
2
+ ======================================
3
+
4
+ This gem provides an interface to the KCL MultiLangDaemon, which is part of the [Amazon Kinesis Client Library](https://github.com/awslabs/amazon-kinesis-client). This interface manages the interaction with the MultiLangDaemon so that developers can focus on implementing their record processor executable. A record processor executable typically looks something like:
5
+
6
+ ```ruby
7
+ class SimpleProcessor
8
+ include RecordProcessor
9
+
10
+ def process_records records, checkpointer
11
+ # process records and checkpoint
12
+ end
13
+ end
14
+ ```
15
+
16
+ Note, the initial implementation of this gem is largely based on the reference [python implementation](https://github.com/awslabs/amazon-kinesis-client-python) provided by Amazon.
17
+
18
+
19
+ Environment Setup
20
+ -----------------
21
+
22
+ Please ensure the following environment requirements are reviewed before using the gem:
23
+ - make sure that your environment is configured to allow the Amazon Kinesis Client Library to use your [AWS Security Credentials](http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html). By default the *DefaultAWSCredentialsProviderChain* is configured so you'll want to make your credentials available to one of the credentials providers in that provider chain. There are several ways to do this such as providing a ~/.aws/credentials file, or specifying the *AWS_ACCESS_KEY_ID* and
24
+ *AWS_SECRET_ACCESS_KEY* environment variables.
25
+ - ensure **JAVA** is available in the environment. This gem works by invoking the packaged *amazon-kinesis-client.jar* and which subsequently executes the target ruby record processor, therefore a compatible JVM/JDK is therefore required.
26
+
27
+
28
+ Environment Variables
29
+ ---------------------
30
+ - **AWS_ACCESS_KEY_ID** : AWS credential for accessing the target kinesis queue
31
+ - **AWS_SECRET_ACCESS_KEY** : AWS credential for accessing the target kinesis queue
32
+ - **APP_NAME** : Used by the KCL as the name of this application. It is used as the DynamoDB table name created by KCL to store checkpoints.
33
+ - **PATH_TO_JAVA** : (optional) custom java executable path (by default `which java` is used).
34
+
35
+
36
+ Example Consumer Client Setup
37
+ -----------------------------
38
+
39
+ Firstly please create the ruby script to run your kinesis consumer with structure similar to the following:
40
+
41
+ ```ruby
42
+ # FILE_NAME: run_simple_kinesis_client.rb
43
+
44
+ require 'kcl'
45
+
46
+ # define a record processor
47
+ class SimpleProcessor < Kcl::AdvancedRecordProcessor
48
+ def process_record data
49
+ p data
50
+ end
51
+ end
52
+
53
+ # config the executor
54
+ Kcl::Executor.new do |executor|
55
+ executor.config stream_name: 'data-kinesis-queue',
56
+ application_name: 'RubyKCLSample',
57
+ max_records: 5,
58
+ idle_time_between_reads_in_millis: 500
59
+
60
+ # setup the target record processor
61
+ executor.record_processor do
62
+ SimpleProcessor.new
63
+ end
64
+ end
65
+
66
+ # execute and run
67
+ Kcl::Executor.run
68
+ ```
69
+
70
+ The most essential part of this is the `Kcl::Executor.run` bit, which is required in the script that you want the consumer client to run. The configuration (i.e. `Kcl::Executor.new` bit) and record processor class (i.e. `SimpleProcessor`) can be put in other suitable places.
71
+
72
+ Next, run the script with an additional argument `exec`, e.g. `ruby run_simple_kinesis_client.rb exec`. Please note, it will **not** work without the `exec` argument, because the script is intent to be invoked by the amazon-kinesis-client java process. Specifying `exec` actually triggers the java consumer process.
73
+
74
+ The following shows an example of how the consumer worker can be specified in the Procfile:
75
+
76
+ ```bash
77
+ worker: bundle exec <your_consumer_client_script> exec
78
+ ```
79
+
80
+
81
+ Configurations
82
+ --------------
83
+
84
+ The properties required by the MultiLangDaemon (please refer to [**this**](https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/sample.properties)) can be configured through the `executor.config`. That is:
85
+
86
+
87
+ ```ruby
88
+ Kcl::Executor.new do |executor|
89
+ executor.config stream_name: 'data-kinesis-queue',
90
+ application_name: 'RubyKCLSample',
91
+ max_records: 5,
92
+ idle_time_between_reads_in_millis: 500,
93
+ region_name: 'us-east-1',
94
+ initial_position_in_stream: 'TRIM_HORIZON'
95
+
96
+ #.....
97
+ end
98
+ ```
99
+
100
+ Under the hood, the Kcl gem will translate it to the proper java properties file for the java process. Please try to use underscore key name (i.e. `stream_name` for `streamName`), so it follows good ruby convention.
101
+
102
+ Please ensure the following configuration values are specified:
103
+ - **stream_name** : the target kinesis queue name
104
+ - **application_name** : it is not required if the environment variable **APP_NAME** is set.
105
+
106
+
107
+ Record Processors
108
+ -----------------
109
+
110
+ Please also specify the record processor for the `Kcl::Executor`, i.e.
111
+
112
+ ```ruby
113
+ Kcl::Executor.new do |executor|
114
+ #.......
115
+ executor.record_processor do
116
+ YourProcessor.new
117
+ end
118
+ end
119
+ ```
120
+
121
+ The reason that why `SimpleProcessor.new` is initialised in the block instead of:
122
+
123
+ ```ruby
124
+ executor.record_processor SimpleProcessor.new
125
+ ```
126
+
127
+ is that processor should only get instantiated when invoked by the consumer client java process, and not in the first `<client_script> exec` call.
128
+
129
+
130
+ ### Kcl::RecordProcessor
131
+
132
+ The RecordProcessor module offers the most basic interface to implement a record processor. The following shows a simple example:
133
+
134
+ ```ruby
135
+ require 'kcl'
136
+
137
+ class YourProcessor
138
+ include Kcl::RecordProcessor
139
+
140
+ def init shared_id
141
+ # Called once by a KCLProcess before any calls to process_records
142
+ end
143
+
144
+ def process_records records, checkpointer
145
+ # Called by a KCLProcess with a list of records to be processed and a
146
+ # checkpointer which accepts sequence numbers from the records to indicate
147
+ # where in the stream to checkpoint.
148
+ end
149
+
150
+ def shutdown checkpointer, reason
151
+ #Called by a KCLProcess instance to indicate that this record processor
152
+ # should shutdown. After this is called, there will be no more calls to
153
+ # any other methods of this record processor.
154
+ end
155
+ end
156
+ ```
157
+
158
+ Please note, with the basic `Kcl::RecordProcessor`, it is the client's responsibility to manage the checkpoints. The client are free to decide how often the checkpoint should be made through doing:
159
+
160
+ ```ruby
161
+ def process_records records, checkpointer
162
+ checkpointer.checkpoint records.last['sequenceNumber']
163
+ end
164
+ ```
165
+
166
+ ### Kcl::AdvancedRecordProcessor
167
+
168
+ The AdvancedRecordProcessor class take cares the basic checkpoint logic, and the clients only required to implement the `process_record` method, for example:
169
+
170
+ ```ruby
171
+ require 'kcl'
172
+
173
+ class YourProcessor < Kcl::AdvancedRecordProcessor
174
+ def initialize
175
+ super sleep_seconds: 10, # default to 5
176
+ checkpoint_retries: 10, # default to 5
177
+ checkpoint_freq_seconds: 30 # default to 60
178
+ end
179
+
180
+ def process_record record
181
+ data = record[:data]
182
+ partition_key = record[:partition_key]
183
+ sequence_number = record[:sequence_number]
184
+
185
+ # do something with data
186
+ end
187
+ end
188
+
189
+ ```
190
+
191
+
192
+ Downloading
193
+ -----------
194
+ install stable releases with the following command:
195
+
196
+ ```bash
197
+ gem install amazon-kinesis-client-ruby
198
+ ```
199
+
200
+ The development version (hosted on Github) can be installed with:
201
+
202
+ ```bash
203
+ git clone git@github.com:everydayhero/amazon-kinesis-client-ruby.git
204
+ cd amazon-kinesis-client-ruby
205
+ rake install
206
+ ```
207
+
208
+ ###Run Tests
209
+ ```bash
210
+ rake spec
211
+ ```
212
+
213
+
214
+ Future Roadmap
215
+ --------------
216
+ - dependency management for the Amazon kinesis client jar files by utilising [ruby-maven](https://github.com/mkristian/ruby-maven) (potentially).
217
+
218
+
219
+ Contributing
220
+ ------------
221
+
222
+ 1. Fork it
223
+ 2. Create your feature branch (git checkout -b my-new-feature)
224
+ 3. Commit your changes (git commit -am 'Add some feature')
225
+ 4. Push to the branch (git push origin my-new-feature)
226
+ 5. Create new Pull Request
@@ -0,0 +1,9 @@
1
+ require 'bundler/gem_tasks'
2
+ require 'rspec/core/rake_task'
3
+ require 'rubocop/rake_task'
4
+
5
+ RSpec::Core::RakeTask.new :spec do |spec|
6
+ spec.rspec_opts = '--format documentation --color'
7
+ end
8
+
9
+ RuboCop::RakeTask.new
@@ -0,0 +1,29 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'kcl/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = 'amazon-kinesis-client-ruby'
8
+ spec.version = Kcl::VERSION
9
+ spec.authors = ['Soloman Weng']
10
+ spec.email = ['solomanw@everydayhero.com.au']
11
+ spec.summary = 'Amazon Kinesis Client Library for Ruby'
12
+ spec.description = 'Amazon Kinesis Client Library for Ruby'
13
+ spec.homepage = ''
14
+ spec.license = 'MIT'
15
+
16
+ spec.files = `git ls-files -z`.split("\x0")
17
+ spec.executables = spec.files.grep(/^bin\//) { |f| File.basename(f) }
18
+ spec.test_files = spec.files.grep(/^(test|spec|features)\//)
19
+ spec.require_paths = ['lib']
20
+
21
+ spec.add_runtime_dependency 'activesupport', '~> 4.1.8'
22
+
23
+ spec.add_development_dependency 'bundler', '~> 1.7'
24
+ spec.add_development_dependency 'rake', '~> 10.0'
25
+ spec.add_development_dependency 'rspec', '~> 3.1.0'
26
+ spec.add_development_dependency 'rubocop', '~> 0.27.1'
27
+
28
+ spec.required_ruby_version = '~> 2.0'
29
+ end
Binary file
@@ -0,0 +1,12 @@
1
+ require 'kcl/version'
2
+ require 'kcl/action_handler'
3
+ require 'kcl/record_processor'
4
+ require 'kcl/advanced_record_processor'
5
+ require 'kcl/checkpoint_error'
6
+ require 'kcl/checkpointer'
7
+ require 'kcl/executor_command_builder'
8
+ require 'kcl/io_handler'
9
+ require 'kcl/malformed_action_error'
10
+ require 'kcl/process'
11
+ require 'kcl/configuration'
12
+ require 'kcl/executor'
@@ -0,0 +1,34 @@
1
+ module Kcl
2
+ class ActionHandler
3
+ def initialize record_processor, checkpointer, io_handler
4
+ @record_processor = record_processor
5
+ @checkpointer = checkpointer
6
+ @io_handler = io_handler
7
+ end
8
+
9
+ # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
10
+ def handle action
11
+ case action.fetch('action')
12
+ when 'initialize'
13
+ record_processor.init action.fetch('shardId')
14
+ when 'processRecords'
15
+ record_processor.process_records action.fetch('records'), checkpointer
16
+ when 'shutdown'
17
+ record_processor.shutdown checkpointer, action.fetch('reason')
18
+ else
19
+ fail MalformedActionError,
20
+ "Received an action which couldn't be understood. Action was #{action}"
21
+ end
22
+ rescue KeyError => key_error
23
+ raise MalformedActionError,
24
+ "Action #{action} was expected to have key: #{key_error.message}"
25
+ rescue => error
26
+ io_handler.write_error error.backtrace.join "\n"
27
+ end
28
+ # rubocop:enable Metrics/MethodLength,Metrics/AbcSize
29
+
30
+ private
31
+
32
+ attr_reader :record_processor, :io_handler, :checkpointer
33
+ end
34
+ end
@@ -0,0 +1,105 @@
1
+ require 'base64'
2
+ require 'logger'
3
+
4
+ module Kcl
5
+ class AdvancedRecordProcessor
6
+ include RecordProcessor
7
+
8
+ LOG = Logger.new STDOUT
9
+ ERR_LOG = Logger.new STDERR
10
+
11
+ DEFAULT_SLEEP_SECONDS = 5
12
+ DEFAULT_CHECKPOINT_RETRIES = 5
13
+ DEFAULT_CHECKPOINT_FREQ_SECONDS = 60
14
+
15
+ def initialize sleep_seconds: DEFAULT_SLEEP_SECONDS,
16
+ checkpoint_retries: DEFAULT_CHECKPOINT_RETRIES,
17
+ checkpoint_freq_seconds: DEFAULT_CHECKPOINT_FREQ_SECONDS
18
+ @sleep_seconds = sleep_seconds
19
+ @checkpoint_retries = checkpoint_retries
20
+ @checkpoint_freq_seconds = checkpoint_freq_seconds
21
+ end
22
+
23
+ def process_record _record; end
24
+
25
+ def init _shared_id
26
+ self.largest_seq = nil
27
+ self.last_checkpoint_time = Time.now
28
+ end
29
+
30
+ def process_records records, checkpointer
31
+ records.each do |record|
32
+ handle_record record
33
+ end
34
+
35
+ if Time.now - last_checkpoint_time > checkpoint_freq_seconds
36
+ checkpoint checkpointer
37
+ self.last_checkpoint_time = Time.now
38
+ end
39
+ rescue => error
40
+ ERR_LOG.error "Encountered an exception while processing records. Exception was #{error}\n"
41
+ end
42
+
43
+ def shutdown checkpointer, reason
44
+ if reason == 'TERMINATE'
45
+ LOG.info 'Was told to terminate, will attempt to checkpoint.'
46
+ checkpoint checkpointer, sequence_number: nil
47
+ else
48
+ LOG.info 'Shutting down due to failover. Will not checkpoint.'
49
+ end
50
+ end
51
+
52
+ private
53
+
54
+ attr_accessor :largest_seq, :last_checkpoint_time
55
+
56
+ attr_reader :sleep_seconds, :checkpoint_retries, :checkpoint_freq_seconds
57
+
58
+ def handle_record record
59
+ data = Base64.decode64 record['data']
60
+ seq = record['sequenceNumber'].to_i
61
+ key = record['partitionKey']
62
+
63
+ process_record data: data, partition_key: key, sequence_number: seq
64
+
65
+ self.largest_seq = seq if largest_seq.nil? || seq > largest_seq
66
+ end
67
+
68
+ def checkpoint checkpointer, sequence_number: largest_seq
69
+ checkpoint_retries.times do |try_count|
70
+ break if try_checkpoint checkpointer, sequence_number, try_count
71
+
72
+ sleep sleep_seconds
73
+ end
74
+ end
75
+
76
+ # rubocop:disable Metrics/MethodLength,Metrics/CyclomaticComplexity
77
+ def try_checkpoint checkpointer, sequence_number, try_count
78
+ seq = sequence_number.to_s if sequence_number
79
+ checkpointer.checkpoint seq
80
+
81
+ true
82
+ rescue CheckpointError => checkpoint_error
83
+ case checkpoint_error.to_s
84
+ when 'ShutdownException'
85
+ LOG.info 'Encountered shutdown execption, skipping checkpoint'
86
+ true
87
+ when 'ThrottlingException'
88
+ if checkpoint_retries - 1 == try_count
89
+ ERR_LOG.error "Failed to checkpoint after #{try_count} attempts, giving up.\n"
90
+ true
91
+ else
92
+ LOG.info "Was throttled while checkpointing, will attempt again in #{sleep_seconds} seconds"
93
+ false
94
+ end
95
+ when 'InvalidStateException'
96
+ ERR_LOG.error "MultiLangDaemon reported an invalid state while checkpointing.\n"
97
+ false
98
+ else
99
+ ERR_LOG.error "Encountered an error while checkpointing, error was #{checkpoint_error}.\n"
100
+ false
101
+ end
102
+ end
103
+ # rubocop:enable Metrics/MethodLength,Metrics/CyclomaticComplexity
104
+ end
105
+ end
@@ -0,0 +1,16 @@
1
+ module Kcl
2
+ class CheckpointError < StandardError
3
+ def initialize error_name
4
+ super error_name
5
+ @error_name = error_name
6
+ end
7
+
8
+ def to_s
9
+ error_name.to_s
10
+ end
11
+
12
+ private
13
+
14
+ attr_reader :error_name
15
+ end
16
+ end
@@ -0,0 +1,37 @@
1
+ module Kcl
2
+ class Checkpointer
3
+ def initialize io_handler
4
+ @io_handler = io_handler
5
+ end
6
+
7
+ def checkpoint sequence_number = nil
8
+ io_handler.write_action action: 'checkpoint', checkpoint: sequence_number
9
+
10
+ action = fetch_action
11
+ if action['action'] == 'checkpoint'
12
+ fail CheckpointError, action['error'] unless action['error'].nil?
13
+ else
14
+ fail CheckpointError, 'InvalidStateException'
15
+ end
16
+ end
17
+
18
+ private
19
+
20
+ attr_reader :io_handler
21
+
22
+ def fetch_action
23
+ loop do
24
+ action = read_action
25
+
26
+ return action unless action.nil?
27
+ end
28
+ end
29
+
30
+ def read_action
31
+ io_handler.read_action
32
+ rescue IOHandler::ReadError => read_error
33
+ io_handler.write_error \
34
+ "Could not understand line read from input: #{read_error.line}"
35
+ end
36
+ end
37
+ end
@@ -0,0 +1,66 @@
1
+ require 'ostruct'
2
+ require 'active_support/core_ext/string'
3
+
4
+ module Kcl
5
+ class Configuration < OpenStruct
6
+ def initialize config = {}
7
+ super default_config.merge config
8
+ end
9
+
10
+ def to_properties
11
+ check_config
12
+
13
+ to_h.map do |key, value|
14
+ "#{make_prop_key key}=#{value}"
15
+ end.join "\n"
16
+ end
17
+
18
+ private
19
+
20
+ def check_config
21
+ required_propertie_keys.each do |required_key|
22
+ fail "#{required_key} is required" unless to_h[required_key].present?
23
+ end
24
+ end
25
+
26
+ def make_prop_key key
27
+ default_key_map.fetch key, key.to_s.camelize(:lower)
28
+ end
29
+
30
+ def application_name
31
+ ENV['APP_NAME']
32
+ end
33
+
34
+ def executable_name
35
+ caller.each do |trace|
36
+ matched = trace.match(/\A(?<file>.+)\:\d+\:in.*<main>.*\Z/)
37
+
38
+ return matched[:file] if matched
39
+ end
40
+ end
41
+
42
+ def processing_language
43
+ "ruby/#{RUBY_VERSION}"
44
+ end
45
+
46
+ def default_config
47
+ @default_config ||= {
48
+ executable_name: executable_name,
49
+ application_name: application_name,
50
+ processing_language: processing_language,
51
+ aws_credentials_provider: 'DefaultAWSCredentialsProviderChain',
52
+ initial_position_in_stream: 'TRIM_HORIZON'
53
+ }
54
+ end
55
+
56
+ def required_propertie_keys
57
+ default_config.keys.concat [:stream_name]
58
+ end
59
+
60
+ def default_key_map
61
+ {
62
+ aws_credentials_provider: 'AWSCredentialsProvider'
63
+ }
64
+ end
65
+ end
66
+ end
@@ -0,0 +1,75 @@
1
+ require 'tempfile'
2
+
3
+ module Kcl
4
+ class Executor
5
+ LOG = Logger.new STDOUT
6
+
7
+ def initialize
8
+ yield self
9
+
10
+ self.class.current_executor = self
11
+ end
12
+
13
+ def config configuration = nil
14
+ @configuration = Configuration.new configuration if configuration
15
+
16
+ @configuration
17
+ end
18
+
19
+ def record_processor record_processor = nil
20
+ @record_processor_callback =
21
+ if record_processor
22
+ proc { record_processor }
23
+ elsif block_given?
24
+ Proc.new
25
+ else
26
+ fail ArgumentError, 'RecordProcessor required'
27
+ end
28
+ end
29
+
30
+ def run argv
31
+ if argv[0] == 'exec'
32
+ run_exec
33
+ else
34
+ run_record_processor
35
+ end
36
+ end
37
+
38
+ private
39
+
40
+ attr_reader :record_processor_callback, :configuration
41
+
42
+ def run_exec
43
+ command = ExecutorCommandBuilder.new(config_properties_path).build
44
+ LOG.info "execute command:\n#{command.join ' '}"
45
+
46
+ system(*command)
47
+ end
48
+
49
+ def run_record_processor
50
+ processor_instance = record_processor_callback.call
51
+
52
+ processor_instance.run
53
+ end
54
+
55
+ def config_properties_path
56
+ config_properties_file = Tempfile.new ['config', '.properties']
57
+ config_properties_file.write configuration.to_properties
58
+ config_properties_file.close
59
+ LOG.info "properties path: #{config_properties_file.path}"
60
+ LOG.info "properties:\n#{File.read config_properties_file.path}"
61
+
62
+ config_properties_file.path
63
+ end
64
+
65
+ class << self
66
+ attr_accessor :current_executor
67
+
68
+ def run argv = ARGV
69
+ fail 'Executor not configured' unless current_executor
70
+
71
+ current_executor.run argv
72
+ end
73
+ end
74
+ end
75
+ end
@@ -0,0 +1,40 @@
1
+ module Kcl
2
+ class ExecutorCommandBuilder
3
+ def initialize properties_file_path
4
+ @properties_file_path = properties_file_path
5
+ end
6
+
7
+ def build
8
+ [java, '-cp', class_path, client_class, properties_file]
9
+ end
10
+
11
+ private
12
+
13
+ attr_reader :properties_file_path
14
+
15
+ def java
16
+ command = ENV.fetch('PATH_TO_JAVA', `which java`).strip
17
+ fail 'Missing JAVA PATH' if command.nil? || command.empty?
18
+
19
+ command
20
+ end
21
+
22
+ def client_class
23
+ 'com.amazonaws.services.kinesis.multilang.MultiLangDaemon'
24
+ end
25
+
26
+ def class_path
27
+ jar_dir = File.expand_path '../../jars', __FILE__
28
+
29
+ (Dir["#{jar_dir}/*.jar"] << properties_file_dir).join ':'
30
+ end
31
+
32
+ def properties_file
33
+ @properties_file ||= File.basename properties_file_path
34
+ end
35
+
36
+ def properties_file_dir
37
+ @properties_file_dir ||= File.dirname properties_file_path
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,48 @@
1
+ require 'json'
2
+
3
+ module Kcl
4
+ class IOHandler
5
+ def initialize input, output, error
6
+ @input = input
7
+ @output = output
8
+ @error = error
9
+ end
10
+
11
+ def write_action response
12
+ write_line response.to_json
13
+ end
14
+
15
+ def read_action
16
+ line = input.gets
17
+ JSON.parse line unless line.nil? || line.empty?
18
+ rescue => error
19
+ raise ReadError.new(error, line)
20
+ end
21
+
22
+ def write_error error_message
23
+ error << "#{error_message}\n"
24
+ ensure
25
+ error.flush
26
+ end
27
+
28
+ private
29
+
30
+ attr_reader :input, :output, :error
31
+
32
+ def write_line line
33
+ output << "\n#{line}\n"
34
+ ensure
35
+ output.flush
36
+ end
37
+
38
+ class ReadError < StandardError
39
+ attr_reader :base_error, :line
40
+
41
+ def initialize base_error, line
42
+ super base_error.message
43
+ @base_error = base_error
44
+ @line = line
45
+ end
46
+ end
47
+ end
48
+ end
@@ -0,0 +1,3 @@
1
+ module Kcl
2
+ class MalformedActionError < StandardError; end
3
+ end
@@ -0,0 +1,39 @@
1
+ module Kcl
2
+ class Process
3
+ def initialize record_processor,
4
+ input: $stdin,
5
+ output: $stdout,
6
+ error: $stderr
7
+ @record_processor = record_processor
8
+ @io_handler = IOHandler.new input, output, error
9
+ @checkpointer = Checkpointer.new @io_handler
10
+ end
11
+
12
+ def run
13
+ loop do
14
+ action = io_handler.read_action
15
+ perform action
16
+ report_done action
17
+
18
+ break if action.nil?
19
+ end
20
+ end
21
+
22
+ private
23
+
24
+ attr_reader :record_processor, :io_handler, :checkpointer
25
+
26
+ def perform action
27
+ action_handler.handle action
28
+ end
29
+
30
+ def action_handler
31
+ @action_handler ||=
32
+ ActionHandler.new record_processor, checkpointer, io_handler
33
+ end
34
+
35
+ def report_done action
36
+ io_handler.write_action action: 'status', responseFor: action['action']
37
+ end
38
+ end
39
+ end
@@ -0,0 +1,13 @@
1
+ module Kcl
2
+ module RecordProcessor
3
+ def init _shared_id; end
4
+
5
+ def process_records _records, _checkpointer; end
6
+
7
+ def shutdown _checkpointer, _reason; end
8
+
9
+ def run
10
+ Process.new(self).run
11
+ end
12
+ end
13
+ end
@@ -0,0 +1,3 @@
1
+ module Kcl
2
+ VERSION = '0.0.1'
3
+ end
@@ -0,0 +1,71 @@
1
+ require 'spec_helper'
2
+
3
+ describe Kcl::Configuration do
4
+ describe '#to_properties' do
5
+ let(:config) { Kcl::Configuration.new config_options }
6
+
7
+ let(:required_options) {
8
+ {
9
+ application_name: 'MyApp',
10
+ stream_name: 'MyStream'
11
+ }
12
+ }
13
+
14
+ let(:config_options) { required_options }
15
+
16
+ let(:default_properties) {
17
+ %w(
18
+ AWSCredentialsProvider=DefaultAWSCredentialsProviderChain
19
+ initialPositionInStream=TRIM_HORIZON
20
+ ).join "\n"
21
+ }
22
+
23
+ subject { config.to_properties }
24
+
25
+ it { is_expected.to match(%r{executableName=.+/exe/rspec}) }
26
+
27
+ it { is_expected.to include "processingLanguage=ruby/#{RUBY_VERSION}" }
28
+
29
+ it { is_expected.to include default_properties }
30
+
31
+ context 'When ruby style config key is set' do
32
+ let(:config_options) {
33
+ {
34
+ dummy_key: 1,
35
+ dummy_key_two: 'two'
36
+ }.merge required_options
37
+ }
38
+
39
+ it { is_expected.to include "dummyKey=1\ndummyKeyTwo=two" }
40
+ end
41
+
42
+ context 'When aws_credentials_provider is set' do
43
+ let(:config_options) {
44
+ {
45
+ aws_credentials_provider: 'Test'
46
+ }.merge required_options
47
+ }
48
+
49
+ it { is_expected.to include 'AWSCredentialsProvider=Test' }
50
+ end
51
+
52
+ context 'When APP_NAME environment vairlable is set' do
53
+ before { ENV['APP_NAME'] = 'Test App' }
54
+
55
+ let(:config_options) { { stream_name: 'MyStream' } }
56
+
57
+ it { is_expected.to include 'applicationName=Test App' }
58
+ end
59
+
60
+ %w(
61
+ executable_name application_name processing_language
62
+ aws_credentials_provider initial_position_in_stream stream_name
63
+ ).each do |key_prop|
64
+ context "When missing required property #{key_prop}" do
65
+ let(:config_options) { { key_prop => nil } }
66
+
67
+ it { expect { subject }.to raise_error "#{key_prop} is required" }
68
+ end
69
+ end
70
+ end
71
+ end
@@ -0,0 +1,62 @@
1
+ require 'spec_helper'
2
+
3
+ describe Kcl::ExecutorCommandBuilder do
4
+ let(:properties_file_path) { File.expand_path __FILE__ }
5
+ let(:builder) { Kcl::ExecutorCommandBuilder.new properties_file_path }
6
+
7
+ describe '#build' do
8
+ let(:command) { builder.build }
9
+
10
+ describe 'comamnd<java>' do
11
+ let(:java) { command[0] }
12
+
13
+ context 'With PATH_TO_JAVA environment variable' do
14
+ before { ENV['PATH_TO_JAVA'] = 'my_java' }
15
+
16
+ it { expect(java).to eq 'my_java' }
17
+ end
18
+
19
+ context 'Without PATH_TO_JAVA set' do
20
+ before { ENV['PATH_TO_JAVA'] = nil }
21
+
22
+ it { expect(java).to eq `which java`.strip }
23
+ end
24
+
25
+ context 'Without java executable available' do
26
+ before {
27
+ ENV['PATH_TO_JAVA'] = nil
28
+ allow(builder).to receive(:'`').with('which java').and_return ''
29
+ }
30
+
31
+ it { expect { java }.to raise_error 'Missing JAVA PATH' }
32
+ end
33
+ end
34
+
35
+ describe 'command<-cp>' do
36
+ it { expect(command[1]).to eq '-cp' }
37
+ end
38
+
39
+ describe 'command<classpath>' do
40
+ let(:classpath) { command[2] }
41
+
42
+ it { expect(classpath).to match(/\A(.+\.jar\:)+.+\z/) }
43
+
44
+ it { expect(classpath).to include File.dirname(properties_file_path) }
45
+ end
46
+
47
+ describe 'command<client_class>' do
48
+ let(:client_class) { command[3] }
49
+ let(:expected_client_class) {
50
+ 'com.amazonaws.services.kinesis.multilang.MultiLangDaemon'
51
+ }
52
+
53
+ it { expect(client_class).to eq expected_client_class }
54
+ end
55
+
56
+ describe 'command<properties_file>' do
57
+ let(:properties_file) { command[4] }
58
+
59
+ it { expect(properties_file).to eq File.basename(properties_file_path) }
60
+ end
61
+ end
62
+ end
@@ -0,0 +1,25 @@
1
+ require 'spec_helper'
2
+
3
+ describe Kcl::Executor do
4
+ describe '#initialize' do
5
+ context 'When no block provided' do
6
+ it { expect { Kcl::Executor.new }.to raise_error LocalJumpError }
7
+ end
8
+ end
9
+
10
+ describe '.run' do
11
+ context 'Without initialize with #new' do
12
+ it { expect { Kcl::Executor.run }.to raise_error 'Executor not configured' }
13
+ end
14
+
15
+ context 'With executor initialized' do
16
+ before {
17
+ expect_any_instance_of(Kcl::Executor).to receive(:run)
18
+
19
+ Kcl::Executor.new {}
20
+ }
21
+
22
+ it { expect Kcl::Executor.run }
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,56 @@
1
+ require 'spec_helper'
2
+ require 'stringio'
3
+
4
+ describe Kcl::IOHandler do
5
+ let(:input) { StringIO.new }
6
+ let(:output) { StringIO.new }
7
+ let(:error) { StringIO.new }
8
+
9
+ subject { Kcl::IOHandler.new input, output, error }
10
+
11
+ describe '#write_action' do
12
+ let(:action) { { action: 'test', value: 123 } }
13
+
14
+ before { subject.write_action action }
15
+
16
+ it { expect(output.string).to eq "\n#{action.to_json}\n" }
17
+
18
+ it { expect(input.string).to eq '' }
19
+
20
+ it { expect(error.string).to eq '' }
21
+ end
22
+
23
+ describe '#write_error' do
24
+ let(:message) { 'Some error' }
25
+
26
+ before { subject.write_error message }
27
+
28
+ it { expect(error.string).to eq "#{message}\n" }
29
+
30
+ it { expect(input.string).to eq '' }
31
+
32
+ it { expect(output.string).to eq '' }
33
+ end
34
+
35
+ describe '#read_action' do
36
+ context 'With empty line input' do
37
+ it { expect(subject.read_action).to be_nil }
38
+ end
39
+
40
+ context 'With valid action input' do
41
+ let(:action) { { 'action' => 'test', 'value' => 1 } }
42
+
43
+ before { input.string = action.to_json }
44
+
45
+ it { expect(subject.read_action).to eq action }
46
+ end
47
+
48
+ context 'With invalid action input' do
49
+ before { input.string = 'dummy' }
50
+
51
+ it {
52
+ expect { subject.read_action }.to raise_error Kcl::IOHandler::ReadError
53
+ }
54
+ end
55
+ end
56
+ end
@@ -0,0 +1 @@
1
+ require 'kcl'
metadata ADDED
@@ -0,0 +1,154 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: amazon-kinesis-client-ruby
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Soloman Weng
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2014-11-21 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: activesupport
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: 4.1.8
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: 4.1.8
27
+ - !ruby/object:Gem::Dependency
28
+ name: bundler
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '1.7'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '1.7'
41
+ - !ruby/object:Gem::Dependency
42
+ name: rake
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '10.0'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '10.0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: rspec
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: 3.1.0
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: 3.1.0
69
+ - !ruby/object:Gem::Dependency
70
+ name: rubocop
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - "~>"
74
+ - !ruby/object:Gem::Version
75
+ version: 0.27.1
76
+ type: :development
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - "~>"
81
+ - !ruby/object:Gem::Version
82
+ version: 0.27.1
83
+ description: Amazon Kinesis Client Library for Ruby
84
+ email:
85
+ - solomanw@everydayhero.com.au
86
+ executables: []
87
+ extensions: []
88
+ extra_rdoc_files: []
89
+ files:
90
+ - ".gitignore"
91
+ - ".rubocop.yml"
92
+ - Gemfile
93
+ - LICENSE.txt
94
+ - README.md
95
+ - Rakefile
96
+ - amazon-kinesis-client-ruby.gemspec
97
+ - lib/jars/amazon-kinesis-client-1.2.0.jar
98
+ - lib/jars/aws-java-sdk-1.7.13.jar
99
+ - lib/jars/commons-codec-1.3.jar
100
+ - lib/jars/commons-logging-1.1.1.jar
101
+ - lib/jars/httpclient-4.2.jar
102
+ - lib/jars/httpcore-4.2.jar
103
+ - lib/jars/jackson-annotations-2.1.1.jar
104
+ - lib/jars/jackson-core-2.1.1.jar
105
+ - lib/jars/jackson-databind-2.1.1.jar
106
+ - lib/jars/joda-time-2.4.jar
107
+ - lib/kcl.rb
108
+ - lib/kcl/action_handler.rb
109
+ - lib/kcl/advanced_record_processor.rb
110
+ - lib/kcl/checkpoint_error.rb
111
+ - lib/kcl/checkpointer.rb
112
+ - lib/kcl/configuration.rb
113
+ - lib/kcl/executor.rb
114
+ - lib/kcl/executor_command_builder.rb
115
+ - lib/kcl/io_handler.rb
116
+ - lib/kcl/malformed_action_error.rb
117
+ - lib/kcl/process.rb
118
+ - lib/kcl/record_processor.rb
119
+ - lib/kcl/version.rb
120
+ - spec/configuration_spec.rb
121
+ - spec/executor_command_builder_spec.rb
122
+ - spec/executor_spec.rb
123
+ - spec/io_handler_spec.rb
124
+ - spec/spec_helper.rb
125
+ homepage: ''
126
+ licenses:
127
+ - MIT
128
+ metadata: {}
129
+ post_install_message:
130
+ rdoc_options: []
131
+ require_paths:
132
+ - lib
133
+ required_ruby_version: !ruby/object:Gem::Requirement
134
+ requirements:
135
+ - - "~>"
136
+ - !ruby/object:Gem::Version
137
+ version: '2.0'
138
+ required_rubygems_version: !ruby/object:Gem::Requirement
139
+ requirements:
140
+ - - ">="
141
+ - !ruby/object:Gem::Version
142
+ version: '0'
143
+ requirements: []
144
+ rubyforge_project:
145
+ rubygems_version: 2.4.2
146
+ signing_key:
147
+ specification_version: 4
148
+ summary: Amazon Kinesis Client Library for Ruby
149
+ test_files:
150
+ - spec/configuration_spec.rb
151
+ - spec/executor_command_builder_spec.rb
152
+ - spec/executor_spec.rb
153
+ - spec/io_handler_spec.rb
154
+ - spec/spec_helper.rb