logstash-input-kinesis 2.0.8-java → 2.1.2-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6c30c3e57fd6684ef7828d46cae719101d821da13610d59b14943c4a98781968
4
- data.tar.gz: 98599b307cf033adbc3f295182e7ef48780f0d933e33d3c803b341f2581f083d
3
+ metadata.gz: 245ff794a871cf62045b673be84a549cda2ccba7ff84bb1c07f946e7a4d7ee81
4
+ data.tar.gz: ddf9cc491568333d6b69cd155f68d27ec22a27229dfcbd1b138c697cf496028d
5
5
  SHA512:
6
- metadata.gz: d40c14d095aaab40a03d0234cf40ace49426346833cd9937312e501b06adfa523e214594d67fd5e8891c785c3b2c545807c240062e0d4f7e541301e287dedf0a
7
- data.tar.gz: e0a3251ecdacaf66147313c7e518670804c8465db5036b787df9ec613a7ee4b3878d8210ff48546e487ec785f3a9191b5e6402b318372f171bd68fb9680ffe8f
6
+ metadata.gz: f264b7d24dbfaa1dde8a601f26bc54aed3654ff204fe1e74874599b8b013bfab29a9bf754d3f3a9cea8fe0b8807ad8095b1cbf964309ddb5f1646537dd2e465d
7
+ data.tar.gz: d60dd23223e523c4edde978fcb6e5a264bf09e4ad3c1e5a4aa6c882d869af07dc2cee7963f93fd4b3eb039385842441e774b82f251e9fc3ec9787d6874ab1804
@@ -1,3 +1,21 @@
1
+ ## 2.1.2
2
+ - Fixed logging level setup to use JUL and JCL loggers backend [#87](https://github.com/logstash-plugins/logstash-input-kinesis/issue/87)
3
+
4
+ ## 2.1.1
5
+ - Added helpful content from readme to doc file for publishing [#63](https://github.com/logstash-plugins/logstash-input-kinesis/pull/63)
6
+
7
+ ## 2.1.0
8
+ - Changed role assumption to also assume role for interactions with dynamodb and cloudwatch [#66](https://github.com/logstash-plugins/logstash-input-kinesis/pull/66)
9
+
10
+ ## 2.0.11
11
+ - Added the ability to assume a role [#40](https://github.com/logstash-plugins/logstash-input-kinesis/pull/40)
12
+
13
+ ## 2.0.10
14
+ - Added the ability to set additional settings exposed through KinesisClientLibConfiguration [#51](https://github.com/logstash-plugins/logstash-input-kinesis/pull/51)
15
+
16
+ ## 2.0.9
17
+ - Changed the 'workerid' to also include the host of the Logstash node [#48](https://github.com/logstash-plugins/logstash-input-kinesis/pull/48)
18
+
1
19
  ## 2.0.8
2
20
  - Changed plugin to use more recent versions of Kinesis Client library and AWS SDK[#45](https://github.com/logstash-plugins/logstash-input-kinesis/pull/45)
3
21
 
data/LICENSE ADDED
@@ -0,0 +1,202 @@
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright 2020 Elastic and contributors
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
data/README.md CHANGED
@@ -48,11 +48,22 @@ This are the properties you can configure and what are the default values:
48
48
  * **default value**: `nil`
49
49
  * `profile`: The AWS profile name for authentication. This ensures that the `~/.aws/credentials` AWS auth provider is used. By default this is empty and the default chain will be used.
50
50
  * **required**: false
51
- * **default value**: `""`
51
+ * `role_arn`: The AWS role to assume. This can be used, for example, to access a Kinesis stream in a different AWS
52
+ account. This role will be assumed after the default credentials or profile credentials are created. By default
53
+ this is empty and a role will not be assumed.
54
+ * **required**: false
55
+ * `role_session_name`: Session name to use when assuming an IAM role. This is recorded in CloudTrail logs for example.
56
+ * **required**: false
57
+ * **default value**: `"logstash"`
52
58
  * `initial_position_in_stream`: The value for initialPositionInStream. Accepts "TRIM_HORIZON" or "LATEST".
53
59
  * **required**: false
54
60
  * **default value**: `"TRIM_HORIZON"`
55
61
 
62
+ ### Additional KCL Settings
63
+ * `additional_settings`: The KCL provides several configuration options which can be set in [KinesisClientLibConfiguration](https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/coordinator/KinesisClientLibConfiguration.java). These options are configured via various function calls that all begin with `with`. Some of these functions take complex types, which are not supported. However, you may invoke any one of the `withX()` functions that take a primitive by providing key-value pairs in `snake_case`. For example, to set the dynamodb read and write capacity values, two functions exist, withInitialLeaseTableReadCapacity and withInitialLeaseTableWriteCapacity. To set a value for these, provide a hash of `additional_settings => {"initial_lease_table_read_capacity" => 25, "initial_lease_table_write_capacity" => 100}`
64
+ * **required**: false
65
+ * **default value**: `{}`
66
+
56
67
  ## Authentication
57
68
 
58
69
  This plugin uses the default AWS SDK auth chain, [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html), to determine which credentials the client will use, unless `profile` is set, in which case [ProfileCredentialsProvider](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/profile/ProfileCredentialsProvider.html) is used.
@@ -77,3 +88,12 @@ Look at the [documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javado
77
88
  3. Commit your changes (`git commit -am 'Add some feature'`)
78
89
  4. Push to the branch (`git push origin my-new-feature`)
79
90
  5. Create a new Pull Request
91
+
92
+
93
+ ## Development
94
+
95
+ To download all jars:
96
+ `bundler exec rake install_jars`
97
+
98
+ To run all specs:
99
+ `bundler exec rspec`
@@ -1,11 +1,12 @@
1
1
  # encoding: utf-8
2
+
3
+ require "socket"
2
4
  require "logstash/inputs/base"
3
5
  require "logstash/errors"
4
6
  require "logstash/environment"
5
7
  require "logstash/namespace"
6
8
 
7
9
  require 'logstash-input-kinesis_jars'
8
- require "logstash/inputs/kinesis/version"
9
10
 
10
11
 
11
12
  # Receive events through an AWS Kinesis stream.
@@ -53,23 +54,45 @@ class LogStash::Inputs::Kinesis < LogStash::Inputs::Base
53
54
  # Select AWS profile for input
54
55
  config :profile, :validate => :string
55
56
 
57
+ # The AWS IAM Role to assume, if any.
58
+ # This is used to generate temporary credentials typically for cross-account access.
59
+ # See https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html for more information.
60
+ config :role_arn, :validate => :string
61
+
62
+ # Session name to use when assuming an IAM role
63
+ config :role_session_name, :validate => :string, :default => "logstash"
64
+
56
65
  # Select initial_position_in_stream. Accepts TRIM_HORIZON or LATEST
57
66
  config :initial_position_in_stream, :validate => ["TRIM_HORIZON", "LATEST"], :default => "TRIM_HORIZON"
58
67
 
68
+ # Any additional arbitrary kcl options configurable in the KinesisClientLibConfiguration
69
+ config :additional_settings, :validate => :hash, :default => {}
70
+
59
71
  def initialize(params = {})
60
72
  super(params)
61
73
  end
62
74
 
63
75
  def register
64
76
  # the INFO log level is extremely noisy in KCL
65
- kinesis_logger = org.apache.commons.logging::LogFactory.getLog("com.amazonaws.services.kinesis").logger
66
- if kinesis_logger.java_kind_of?(java.util.logging::Logger)
67
- kinesis_logger.setLevel(java.util.logging::Level::WARNING)
77
+ lg = org.apache.commons.logging::LogFactory.getLog("com.amazonaws.services.kinesis")
78
+ if lg.java_kind_of?(org.apache.commons.logging.impl::Jdk14Logger)
79
+ kinesis_logger = lg.logger
80
+ if kinesis_logger.java_kind_of?(java.util.logging::Logger)
81
+ kinesis_logger.setLevel(java.util.logging::Level::WARNING)
82
+ else
83
+ kinesis_logger.setLevel(org.apache.log4j::Level::WARN)
84
+ end
85
+ elsif lg.java_kind_of?(org.apache.logging.log4jJcl::Log4jLog)
86
+ logContext = org.apache.logging.log4j::LogManager.getContext(false)
87
+ config = logContext.getConfiguration()
88
+ config.getLoggerConfig("com.amazonaws.services.kinesis").setLevel(org.apache.logging.log4j::Level::WARN)
68
89
  else
69
- kinesis_logger.setLevel(org.apache.log4j::Level::WARN)
90
+ raise "Can't configure WARN log level for logger wrapper class #{lg.class}"
70
91
  end
71
92
 
72
- worker_id = java.util::UUID.randomUUID.to_s
93
+ hostname = Socket.gethostname
94
+ uuid = java.util::UUID.randomUUID.to_s
95
+ worker_id = "#{hostname}:#{uuid}"
73
96
 
74
97
  # If the AWS profile is set, use the profile credentials provider.
75
98
  # Otherwise fall back to the default chain.
@@ -78,6 +101,14 @@ class LogStash::Inputs::Kinesis < LogStash::Inputs::Base
78
101
  else
79
102
  creds = com.amazonaws.auth::DefaultAWSCredentialsProviderChain.new
80
103
  end
104
+
105
+ # If a role ARN is set then assume the role as a new layer over the credentials already created
106
+ unless @role_arn.nil?
107
+ kinesis_creds = com.amazonaws.auth::STSAssumeRoleSessionCredentialsProvider.new(creds, @role_arn, @role_session_name)
108
+ else
109
+ kinesis_creds = creds
110
+ end
111
+
81
112
  initial_position_in_stream = if @initial_position_in_stream == "TRIM_HORIZON"
82
113
  KCL::InitialPositionInStream::TRIM_HORIZON
83
114
  else
@@ -87,10 +118,17 @@ class LogStash::Inputs::Kinesis < LogStash::Inputs::Base
87
118
  @kcl_config = KCL::KinesisClientLibConfiguration.new(
88
119
  @application_name,
89
120
  @kinesis_stream_name,
90
- creds,
121
+ kinesis_creds, # credential provider for Kinesis, DynamoDB and Cloudwatch access
91
122
  worker_id).
92
123
  withInitialPositionInStream(initial_position_in_stream).
93
124
  withRegionName(@region)
125
+
126
+ # Call arbitrary "withX()" functions
127
+ # snake_case => withCamelCase happens automatically
128
+ @additional_settings.each do |key, value|
129
+ fn = "with_#{key}"
130
+ @kcl_config.send(fn, value)
131
+ end
94
132
  end
95
133
 
96
134
  def run(output_queue)
@@ -1,11 +1,9 @@
1
1
  # coding: utf-8
2
- lib = File.expand_path('../lib', __FILE__)
3
- $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
- require 'logstash/inputs/kinesis/version'
2
+ version = File.read(File.expand_path(File.join(File.dirname(__FILE__), "VERSION"))).strip
5
3
 
6
4
  Gem::Specification.new do |spec|
7
5
  spec.name = "logstash-input-kinesis"
8
- spec.version = Logstash::Input::Kinesis::VERSION
6
+ spec.version = version
9
7
  spec.authors = ["Brian Palmer"]
10
8
  spec.email = ["brian@codekitchen.net"]
11
9
  spec.summary = "Receives events through an AWS Kinesis stream"
@@ -24,10 +22,11 @@ Gem::Specification.new do |spec|
24
22
 
25
23
  spec.requirements << "jar 'com.amazonaws:amazon-kinesis-client', '1.9.2'"
26
24
  spec.requirements << "jar 'com.amazonaws:aws-java-sdk-core', '1.11.414'"
25
+ spec.requirements << "jar 'com.amazonaws:aws-java-sdk-sts', '1.11.414'"
27
26
 
28
27
  spec.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
29
28
 
30
29
  spec.add_development_dependency 'logstash-devutils'
31
- spec.add_development_dependency 'jar-dependencies', '~> 0.3.4'
30
+ spec.add_development_dependency 'jar-dependencies', '~> 0.4'
32
31
  spec.add_development_dependency "logstash-codec-json"
33
32
  end
@@ -26,6 +26,17 @@ RSpec.describe "inputs/kinesis" do
26
26
  "profile" => "my-aws-profile"
27
27
  }}
28
28
 
29
+ # Config hash to test assume role provider if role_arn is specified
30
+ let(:config_with_role_arn) {{
31
+ "application_name" => "my-processor",
32
+ "kinesis_stream_name" => "run-specs",
33
+ "codec" => codec,
34
+ "metrics" => metrics,
35
+ "checkpoint_interval_seconds" => 120,
36
+ "region" => "ap-southeast-1",
37
+ "role_arn" => "arn:aws:iam::???????????:role/my-role"
38
+ }}
39
+
29
40
  # other config with LATEST as initial_position_in_stream
30
41
  let(:config_with_latest) {{
31
42
  "application_name" => "my-processor",
@@ -38,6 +49,50 @@ RSpec.describe "inputs/kinesis" do
38
49
  "initial_position_in_stream" => "LATEST"
39
50
  }}
40
51
 
52
+ # Config hash to test valid additional_settings
53
+ let(:config_with_valid_additional_settings) {{
54
+ "application_name" => "my-processor",
55
+ "kinesis_stream_name" => "run-specs",
56
+ "codec" => codec,
57
+ "metrics" => metrics,
58
+ "checkpoint_interval_seconds" => 120,
59
+ "region" => "ap-southeast-1",
60
+ "profile" => nil,
61
+ "additional_settings" => {
62
+ "initial_lease_table_read_capacity" => 25,
63
+ "initial_lease_table_write_capacity" => 100,
64
+ "kinesis_endpoint" => "http://localhost"
65
+ }
66
+ }}
67
+
68
+ # Config hash to test invalid additional_settings where the name is not found
69
+ let(:config_with_invalid_additional_settings_name_not_found) {{
70
+ "application_name" => "my-processor",
71
+ "kinesis_stream_name" => "run-specs",
72
+ "codec" => codec,
73
+ "metrics" => metrics,
74
+ "checkpoint_interval_seconds" => 120,
75
+ "region" => "ap-southeast-1",
76
+ "profile" => nil,
77
+ "additional_settings" => {
78
+ "foo" => "bar"
79
+ }
80
+ }}
81
+
82
+ # Config hash to test invalid additional_settings where the type is complex or wrong
83
+ let(:config_with_invalid_additional_settings_wrong_type) {{
84
+ "application_name" => "my-processor",
85
+ "kinesis_stream_name" => "run-specs",
86
+ "codec" => codec,
87
+ "metrics" => metrics,
88
+ "checkpoint_interval_seconds" => 120,
89
+ "region" => "ap-southeast-1",
90
+ "profile" => nil,
91
+ "additional_settings" => {
92
+ "metrics_level" => "invalid_metrics_level"
93
+ }
94
+ }}
95
+
41
96
  subject!(:kinesis) { LogStash::Inputs::Kinesis.new(config) }
42
97
  let(:kcl_worker) { double('kcl_worker') }
43
98
  let(:stub_builder) { double('kcl_builder', build: kcl_worker) }
@@ -66,6 +121,15 @@ RSpec.describe "inputs/kinesis" do
66
121
  expect(kinesis_with_profile.kcl_config.get_kinesis_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.profile.ProfileCredentialsProvider")
67
122
  end
68
123
 
124
+ subject!(:kinesis_with_role_arn) { LogStash::Inputs::Kinesis.new(config_with_role_arn) }
125
+
126
+ it "uses STS for accessing the kinesis stream if role_arn is specified" do
127
+ kinesis_with_role_arn.register
128
+ expect(kinesis_with_role_arn.kcl_config.get_kinesis_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider")
129
+ expect(kinesis_with_role_arn.kcl_config.get_dynamo_db_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider")
130
+ expect(kinesis_with_role_arn.kcl_config.get_cloud_watch_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider")
131
+ end
132
+
69
133
  subject!(:kinesis_with_latest) { LogStash::Inputs::Kinesis.new(config_with_latest) }
70
134
 
71
135
  it "configures the KCL" do
@@ -77,6 +141,33 @@ RSpec.describe "inputs/kinesis" do
77
141
  expect(kinesis_with_latest.kcl_config.get_kinesis_credentials_provider.getClass.to_s).to eq("com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
78
142
  end
79
143
 
144
+ subject!(:kinesis_with_valid_additional_settings) { LogStash::Inputs::Kinesis.new(config_with_valid_additional_settings) }
145
+
146
+ it "configures the KCL" do
147
+ kinesis_with_valid_additional_settings.register
148
+ expect(kinesis_with_valid_additional_settings.kcl_config.applicationName).to eq("my-processor")
149
+ expect(kinesis_with_valid_additional_settings.kcl_config.streamName).to eq("run-specs")
150
+ expect(kinesis_with_valid_additional_settings.kcl_config.regionName).to eq("ap-southeast-1")
151
+ expect(kinesis_with_valid_additional_settings.kcl_config.initialLeaseTableReadCapacity).to eq(25)
152
+ expect(kinesis_with_valid_additional_settings.kcl_config.initialLeaseTableWriteCapacity).to eq(100)
153
+ expect(kinesis_with_valid_additional_settings.kcl_config.kinesisEndpoint).to eq("http://localhost")
154
+ end
155
+
156
+
157
+ subject!(:kinesis_with_invalid_additional_settings_name_not_found) { LogStash::Inputs::Kinesis.new(config_with_invalid_additional_settings_name_not_found) }
158
+
159
+ it "raises NoMethodError for invalid configuration options" do
160
+ expect{ kinesis_with_invalid_additional_settings_name_not_found.register }.to raise_error(NoMethodError)
161
+ end
162
+
163
+
164
+ subject!(:kinesis_with_invalid_additional_settings_wrong_type) { LogStash::Inputs::Kinesis.new(config_with_invalid_additional_settings_wrong_type) }
165
+
166
+ it "raises an error for invalid configuration values such as the wrong type" do
167
+ expect{ kinesis_with_invalid_additional_settings_wrong_type.register }.to raise_error(Java::JavaLang::IllegalArgumentException)
168
+ end
169
+
170
+
80
171
  context "#run" do
81
172
  it "runs the KCL worker" do
82
173
  expect(kinesis).to receive(:kcl_builder).with(queue).and_return(stub_builder)
@@ -2,16 +2,17 @@
2
2
  begin
3
3
  require 'jar_dependencies'
4
4
  rescue LoadError
5
+ require 'com/amazonaws/aws-java-sdk-sts/1.11.414/aws-java-sdk-sts-1.11.414.jar'
5
6
  require 'com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar'
6
7
  require 'com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar'
7
8
  require 'com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar'
8
9
  require 'org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar'
9
- require 'com/amazonaws/jmespath-java/1.11.400/jmespath-java-1.11.400.jar'
10
10
  require 'com/amazonaws/aws-java-sdk-s3/1.11.400/aws-java-sdk-s3-1.11.400.jar'
11
11
  require 'com/google/guava/guava/18.0/guava-18.0.jar'
12
12
  require 'commons-lang/commons-lang/2.6/commons-lang-2.6.jar'
13
13
  require 'commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar'
14
14
  require 'joda-time/joda-time/2.8.1/joda-time-2.8.1.jar'
15
+ require 'com/amazonaws/jmespath-java/1.11.414/jmespath-java-1.11.414.jar'
15
16
  require 'software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar'
16
17
  require 'com/amazonaws/amazon-kinesis-client/1.9.2/amazon-kinesis-client-1.9.2.jar'
17
18
  require 'com/amazonaws/aws-java-sdk-kinesis/1.11.400/aws-java-sdk-kinesis-1.11.400.jar'
@@ -26,16 +27,17 @@ rescue LoadError
26
27
  end
27
28
 
28
29
  if defined? Jars
30
+ require_jar 'com.amazonaws', 'aws-java-sdk-sts', '1.11.414'
29
31
  require_jar 'com.fasterxml.jackson.core', 'jackson-databind', '2.6.7.1'
30
32
  require_jar 'com.fasterxml.jackson.core', 'jackson-core', '2.6.7'
31
33
  require_jar 'com.fasterxml.jackson.dataformat', 'jackson-dataformat-cbor', '2.6.7'
32
34
  require_jar 'org.apache.httpcomponents', 'httpclient', '4.5.5'
33
- require_jar 'com.amazonaws', 'jmespath-java', '1.11.400'
34
35
  require_jar 'com.amazonaws', 'aws-java-sdk-s3', '1.11.400'
35
36
  require_jar 'com.google.guava', 'guava', '18.0'
36
37
  require_jar 'commons-lang', 'commons-lang', '2.6'
37
38
  require_jar 'commons-logging', 'commons-logging', '1.1.3'
38
39
  require_jar 'joda-time', 'joda-time', '2.8.1'
40
+ require_jar 'com.amazonaws', 'jmespath-java', '1.11.414'
39
41
  require_jar 'software.amazon.ion', 'ion-java', '1.0.2'
40
42
  require_jar 'com.amazonaws', 'amazon-kinesis-client', '1.9.2'
41
43
  require_jar 'com.amazonaws', 'aws-java-sdk-kinesis', '1.11.400'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-kinesis
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.0.8
4
+ version: 2.1.2
5
5
  platform: java
6
6
  authors:
7
7
  - Brian Palmer
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2018-09-25 00:00:00.000000000 Z
11
+ date: 2020-09-01 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -49,7 +49,7 @@ dependencies:
49
49
  requirements:
50
50
  - - "~>"
51
51
  - !ruby/object:Gem::Version
52
- version: 0.3.4
52
+ version: '0.4'
53
53
  name: jar-dependencies
54
54
  prerelease: false
55
55
  type: :development
@@ -57,7 +57,7 @@ dependencies:
57
57
  requirements:
58
58
  - - "~>"
59
59
  - !ruby/object:Gem::Version
60
- version: 0.3.4
60
+ version: '0.4'
61
61
  - !ruby/object:Gem::Dependency
62
62
  requirement: !ruby/object:Gem::Requirement
63
63
  requirements:
@@ -84,9 +84,9 @@ files:
84
84
  - CHANGELOG.md
85
85
  - CONTRIBUTORS
86
86
  - Gemfile
87
+ - LICENSE
87
88
  - README.md
88
89
  - lib/logstash/inputs/kinesis.rb
89
- - lib/logstash/inputs/kinesis/version.rb
90
90
  - lib/logstash/inputs/kinesis/worker.rb
91
91
  - logstash-input-kinesis.gemspec
92
92
  - spec/inputs/kinesis/worker_spec.rb
@@ -99,7 +99,8 @@ files:
99
99
  - vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-kinesis/1.11.400/aws-java-sdk-kinesis-1.11.400.jar
100
100
  - vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-kms/1.11.400/aws-java-sdk-kms-1.11.400.jar
101
101
  - vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-s3/1.11.400/aws-java-sdk-s3-1.11.400.jar
102
- - vendor/jar-dependencies/runtime-jars/com/amazonaws/jmespath-java/1.11.400/jmespath-java-1.11.400.jar
102
+ - vendor/jar-dependencies/runtime-jars/com/amazonaws/aws-java-sdk-sts/1.11.414/aws-java-sdk-sts-1.11.414.jar
103
+ - vendor/jar-dependencies/runtime-jars/com/amazonaws/jmespath-java/1.11.414/jmespath-java-1.11.414.jar
103
104
  - vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar
104
105
  - vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar
105
106
  - vendor/jar-dependencies/runtime-jars/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar
@@ -138,6 +139,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
138
139
  requirements:
139
140
  - jar 'com.amazonaws:amazon-kinesis-client', '1.9.2'
140
141
  - jar 'com.amazonaws:aws-java-sdk-core', '1.11.414'
142
+ - jar 'com.amazonaws:aws-java-sdk-sts', '1.11.414'
141
143
  rubyforge_project:
142
144
  rubygems_version: 2.6.13
143
145
  signing_key:
@@ -1,8 +0,0 @@
1
- # encoding: utf-8
2
- module Logstash
3
- module Input
4
- module Kinesis
5
- VERSION = "2.0.8"
6
- end
7
- end
8
- end