logstash-input-kinesis 1.4.3-java → 1.5.0-java
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CONTRIBUTORS +14 -0
- data/README.md +5 -4
- data/lib/com/amazonaws/amazon-kinesis-client/1.6.1/amazon-kinesis-client-1.6.1.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-cloudwatch/1.10.20/aws-java-sdk-cloudwatch-1.10.20.jar +0 -0
- data/{vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.10.50.jar → lib/com/amazonaws/aws-java-sdk-core/1.10.20/aws-java-sdk-core-1.10.20.jar} +0 -0
- data/lib/com/amazonaws/aws-java-sdk-dynamodb/1.10.20/aws-java-sdk-dynamodb-1.10.20.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-kinesis/1.10.20/aws-java-sdk-kinesis-1.10.20.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-kms/1.10.20/aws-java-sdk-kms-1.10.20.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-s3/1.10.20/aws-java-sdk-s3-1.10.20.jar +0 -0
- data/lib/com/fasterxml/jackson/core/jackson-annotations/2.5.0/jackson-annotations-2.5.0.jar +0 -0
- data/lib/com/fasterxml/jackson/core/jackson-core/2.5.3/jackson-core-2.5.3.jar +0 -0
- data/lib/com/fasterxml/jackson/core/jackson-databind/2.5.3/jackson-databind-2.5.3.jar +0 -0
- data/lib/com/google/guava/guava/18.0/guava-18.0.jar +0 -0
- data/lib/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar +0 -0
- data/lib/commons-codec/commons-codec/1.9/commons-codec-1.9.jar +0 -0
- data/lib/commons-lang/commons-lang/2.6/commons-lang-2.6.jar +0 -0
- data/lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar +0 -0
- data/lib/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar +0 -0
- data/lib/logstash/inputs/kinesis.rb +1 -1
- data/lib/logstash/inputs/kinesis/version.rb +2 -1
- data/lib/logstash/inputs/kinesis/worker.rb +1 -0
- data/lib/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar +0 -0
- data/lib/org/apache/httpcomponents/httpcore/4.4.3/httpcore-4.4.3.jar +0 -0
- data/logstash-input-kinesis.gemspec +8 -3
- data/spec/inputs/kinesis/worker_spec.rb +1 -0
- data/spec/inputs/kinesis_spec.rb +12 -1
- metadata +50 -22
- data/.gitignore +0 -14
- data/.rspec +0 -2
- data/.travis.yml +0 -3
- data/LICENSE.txt +0 -14
- data/Rakefile +0 -15
- data/pom.xml +0 -32
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 03bbacd8aa1c1ead7454666582fa2ec84cd79d86
|
4
|
+
data.tar.gz: ae48dab34705083645faff5c7b7c3674ef658a8e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 04342e5d1a6e34138995d89cb894b732997d0d1b51378e46fd4db49823c7b0d8707fbe9bdbbb647b527240372a0ff59360f558866092cff849e91fca8a8b8fda
|
7
|
+
data.tar.gz: 86232000df97f16996c078caf62c6cd35f676c411ce1901966d4edb8da372267867a65739a75ab3a40267e4b6ac3c607702a8bdc69fa9e9e1288c4ca5a26b159
|
data/CONTRIBUTORS
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
The following is a list of people who have contributed ideas, code, bug
|
2
|
+
reports, or in general have helped logstash along its way.
|
3
|
+
|
4
|
+
Maintainers:
|
5
|
+
* Brian Palmer (codekitchen)
|
6
|
+
|
7
|
+
Contributors:
|
8
|
+
* Brian Palmer (codekitchen)
|
9
|
+
* Samuel García Martínez (samuelgmartinez)
|
10
|
+
|
11
|
+
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
12
|
+
Logstash, and you aren't on the list above and want to be, please let us know
|
13
|
+
and we'll make sure you're here. Contributions from folks like you are what make
|
14
|
+
open source awesome.
|
data/README.md
CHANGED
@@ -1,12 +1,12 @@
|
|
1
1
|
# Logstash AWS Kinesis Input Plugin
|
2
2
|
|
3
|
-
[](https://travis-ci.org/logstash-plugins/logstash-input-kinesis)
|
4
4
|
|
5
5
|
This is a [AWS Kinesis](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html) input plugin for [Logstash](https://github.com/elasticsearch/logstash). Under the hood uses the [Kinesis Client Library](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html).
|
6
6
|
|
7
7
|
## Installation
|
8
8
|
|
9
|
-
This plugin requires Logstash >=
|
9
|
+
This plugin requires Logstash >= 2.0, and can be installed by Logstash
|
10
10
|
itself.
|
11
11
|
|
12
12
|
```sh
|
@@ -39,7 +39,7 @@ This are the properties you can configure and what are the default values:
|
|
39
39
|
* `checkpoint_interval_seconds`: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
|
40
40
|
* **required**: false
|
41
41
|
* **default value**: `60`
|
42
|
-
* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
|
42
|
+
* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
|
43
43
|
* **required**: false
|
44
44
|
* **default value**: `nil`
|
45
45
|
|
@@ -61,7 +61,8 @@ Look at the [documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javado
|
|
61
61
|
|
62
62
|
## Contributing
|
63
63
|
|
64
|
-
|
64
|
+
0. https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md#contribution-steps
|
65
|
+
1. Fork it ( https://github.com/logstash-plugins/logstash-input-kinesis/fork )
|
65
66
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
66
67
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
67
68
|
4. Push to the branch (`git push origin my-new-feature`)
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
@@ -10,10 +10,10 @@ Gem::Specification.new do |spec|
|
|
10
10
|
spec.email = ["brian@codekitchen.net"]
|
11
11
|
spec.summary = %q{Logstash plugin for Kinesis input}
|
12
12
|
spec.description = %q{This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program}
|
13
|
-
spec.homepage = "https://github.com/
|
13
|
+
spec.homepage = "https://github.com/logstash-plugins/logstash-input-kinesis"
|
14
14
|
spec.licenses = ['Apache License (2.0)']
|
15
15
|
|
16
|
-
spec.files =
|
16
|
+
spec.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
|
17
17
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
18
18
|
spec.require_paths = ["lib"]
|
19
19
|
|
@@ -22,9 +22,14 @@ Gem::Specification.new do |spec|
|
|
22
22
|
|
23
23
|
spec.platform = 'java'
|
24
24
|
|
25
|
+
spec.requirements << "jar 'com.amazonaws:amazon-kinesis-client', '1.6.1'"
|
26
|
+
spec.requirements << "jar 'org.apache.httpcomponents:httpclient', '4.5.1'"
|
27
|
+
|
28
|
+
spec.add_development_dependency 'jar-dependencies', '0.3.2'
|
29
|
+
|
25
30
|
spec.add_development_dependency "bundler", "~> 1.7"
|
26
31
|
spec.add_development_dependency "rake", "~> 10.0"
|
27
32
|
spec.add_development_dependency "rspec", "~> 3.2.0"
|
28
|
-
spec.add_development_dependency "logstash-core", "
|
33
|
+
spec.add_development_dependency "logstash-core", "~> 2.4.0"
|
29
34
|
spec.add_development_dependency "logstash-codec-json"
|
30
35
|
end
|
@@ -48,6 +48,7 @@ RSpec.describe "LogStash::Inputs::Kinesis::Worker" do
|
|
48
48
|
describe "#processRecords" do
|
49
49
|
it "decodes and queues each record with decoration" do
|
50
50
|
worker.processRecords(process_input)
|
51
|
+
expect(queue.size).to eq(2)
|
51
52
|
m1 = queue.pop
|
52
53
|
m2 = queue.pop
|
53
54
|
expect(m1).to be_kind_of(LogStash::Event)
|
data/spec/inputs/kinesis_spec.rb
CHANGED
@@ -43,11 +43,22 @@ RSpec.describe "inputs/kinesis" do
|
|
43
43
|
end
|
44
44
|
end
|
45
45
|
|
46
|
+
context "#stop" do
|
47
|
+
it "stops the KCL worker" do
|
48
|
+
expect(kinesis).to receive(:kcl_builder).with(queue).and_return(stub_builder)
|
49
|
+
expect(kcl_worker).to receive(:run).with(no_args)
|
50
|
+
expect(kcl_worker).to receive(:shutdown).with(no_args)
|
51
|
+
kinesis.run(queue)
|
52
|
+
kinesis.do_stop # do_stop calls stop internally
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
46
56
|
context "#worker_factory" do
|
47
57
|
it "clones the codec for each worker" do
|
48
|
-
expect(codec).to receive(:clone).once
|
49
58
|
worker = kinesis.worker_factory(queue).call()
|
50
59
|
expect(worker).to be_kind_of(LogStash::Inputs::Kinesis::Worker)
|
60
|
+
expect(worker.codec).to_not eq(kinesis.codec)
|
61
|
+
expect(worker.codec).to be_kind_of(codec.class)
|
51
62
|
end
|
52
63
|
|
53
64
|
it "generates a valid worker" do
|
metadata
CHANGED
@@ -1,15 +1,29 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-input-kinesis
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.5.0
|
5
5
|
platform: java
|
6
6
|
authors:
|
7
7
|
- Brian Palmer
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-09-29 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
15
|
+
requirements:
|
16
|
+
- - '='
|
17
|
+
- !ruby/object:Gem::Version
|
18
|
+
version: 0.3.2
|
19
|
+
name: jar-dependencies
|
20
|
+
prerelease: false
|
21
|
+
type: :development
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - '='
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: 0.3.2
|
13
27
|
- !ruby/object:Gem::Dependency
|
14
28
|
requirement: !ruby/object:Gem::Requirement
|
15
29
|
requirements:
|
@@ -55,17 +69,17 @@ dependencies:
|
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
requirement: !ruby/object:Gem::Requirement
|
57
71
|
requirements:
|
58
|
-
- -
|
72
|
+
- - ~>
|
59
73
|
- !ruby/object:Gem::Version
|
60
|
-
version:
|
74
|
+
version: 2.4.0
|
61
75
|
name: logstash-core
|
62
76
|
prerelease: false
|
63
77
|
type: :development
|
64
78
|
version_requirements: !ruby/object:Gem::Requirement
|
65
79
|
requirements:
|
66
|
-
- -
|
80
|
+
- - ~>
|
67
81
|
- !ruby/object:Gem::Version
|
68
|
-
version:
|
82
|
+
version: 2.4.0
|
69
83
|
- !ruby/object:Gem::Dependency
|
70
84
|
requirement: !ruby/object:Gem::Requirement
|
71
85
|
requirements:
|
@@ -87,26 +101,34 @@ executables: []
|
|
87
101
|
extensions: []
|
88
102
|
extra_rdoc_files: []
|
89
103
|
files:
|
90
|
-
- .gitignore
|
91
|
-
- .rspec
|
92
|
-
- .travis.yml
|
93
|
-
- Gemfile
|
94
|
-
- LICENSE.txt
|
95
|
-
- README.md
|
96
|
-
- Rakefile
|
97
104
|
- lib/logstash-input-kinesis_jars.rb
|
105
|
+
- lib/com/amazonaws/amazon-kinesis-client/1.6.1/amazon-kinesis-client-1.6.1.jar
|
106
|
+
- lib/com/amazonaws/aws-java-sdk-cloudwatch/1.10.20/aws-java-sdk-cloudwatch-1.10.20.jar
|
107
|
+
- lib/com/amazonaws/aws-java-sdk-core/1.10.20/aws-java-sdk-core-1.10.20.jar
|
108
|
+
- lib/com/amazonaws/aws-java-sdk-dynamodb/1.10.20/aws-java-sdk-dynamodb-1.10.20.jar
|
109
|
+
- lib/com/amazonaws/aws-java-sdk-kinesis/1.10.20/aws-java-sdk-kinesis-1.10.20.jar
|
110
|
+
- lib/com/amazonaws/aws-java-sdk-kms/1.10.20/aws-java-sdk-kms-1.10.20.jar
|
111
|
+
- lib/com/amazonaws/aws-java-sdk-s3/1.10.20/aws-java-sdk-s3-1.10.20.jar
|
112
|
+
- lib/com/fasterxml/jackson/core/jackson-annotations/2.5.0/jackson-annotations-2.5.0.jar
|
113
|
+
- lib/com/fasterxml/jackson/core/jackson-core/2.5.3/jackson-core-2.5.3.jar
|
114
|
+
- lib/com/fasterxml/jackson/core/jackson-databind/2.5.3/jackson-databind-2.5.3.jar
|
115
|
+
- lib/com/google/guava/guava/18.0/guava-18.0.jar
|
116
|
+
- lib/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar
|
117
|
+
- lib/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
|
118
|
+
- lib/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
|
119
|
+
- lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
|
120
|
+
- lib/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar
|
98
121
|
- lib/logstash/inputs/kinesis.rb
|
99
122
|
- lib/logstash/inputs/kinesis/version.rb
|
100
123
|
- lib/logstash/inputs/kinesis/worker.rb
|
101
|
-
-
|
102
|
-
-
|
103
|
-
- spec/inputs/kinesis/worker_spec.rb
|
104
|
-
- spec/inputs/kinesis_spec.rb
|
124
|
+
- lib/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar
|
125
|
+
- lib/org/apache/httpcomponents/httpcore/4.4.3/httpcore-4.4.3.jar
|
105
126
|
- spec/spec_helper.rb
|
127
|
+
- spec/inputs/kinesis_spec.rb
|
128
|
+
- spec/inputs/kinesis/worker_spec.rb
|
106
129
|
- vendor/jar-dependencies/runtime-jars/amazon-kinesis-client-1.6.1.jar
|
107
130
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-cloudwatch-1.10.20.jar
|
108
131
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.10.20.jar
|
109
|
-
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.10.50.jar
|
110
132
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-dynamodb-1.10.20.jar
|
111
133
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-kinesis-1.10.20.jar
|
112
134
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-kms-1.10.20.jar
|
@@ -122,7 +144,11 @@ files:
|
|
122
144
|
- vendor/jar-dependencies/runtime-jars/jackson-databind-2.5.3.jar
|
123
145
|
- vendor/jar-dependencies/runtime-jars/joda-time-2.8.1.jar
|
124
146
|
- vendor/jar-dependencies/runtime-jars/protobuf-java-2.6.1.jar
|
125
|
-
|
147
|
+
- logstash-input-kinesis.gemspec
|
148
|
+
- README.md
|
149
|
+
- CONTRIBUTORS
|
150
|
+
- Gemfile
|
151
|
+
homepage: https://github.com/logstash-plugins/logstash-input-kinesis
|
126
152
|
licenses:
|
127
153
|
- Apache License (2.0)
|
128
154
|
metadata:
|
@@ -142,13 +168,15 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
142
168
|
- - '>='
|
143
169
|
- !ruby/object:Gem::Version
|
144
170
|
version: '0'
|
145
|
-
requirements:
|
171
|
+
requirements:
|
172
|
+
- jar 'com.amazonaws:amazon-kinesis-client', '1.6.1'
|
173
|
+
- jar 'org.apache.httpcomponents:httpclient', '4.5.1'
|
146
174
|
rubyforge_project:
|
147
175
|
rubygems_version: 2.1.9
|
148
176
|
signing_key:
|
149
177
|
specification_version: 4
|
150
178
|
summary: Logstash plugin for Kinesis input
|
151
179
|
test_files:
|
152
|
-
- spec/inputs/kinesis/worker_spec.rb
|
153
|
-
- spec/inputs/kinesis_spec.rb
|
154
180
|
- spec/spec_helper.rb
|
181
|
+
- spec/inputs/kinesis_spec.rb
|
182
|
+
- spec/inputs/kinesis/worker_spec.rb
|
data/.gitignore
DELETED
data/.rspec
DELETED
data/.travis.yml
DELETED
data/LICENSE.txt
DELETED
@@ -1,14 +0,0 @@
|
|
1
|
-
Copyright (c) 2015 Brian Palmer
|
2
|
-
|
3
|
-
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
-
you may not use this file except in compliance with the License.
|
5
|
-
You may obtain a copy of the License at
|
6
|
-
|
7
|
-
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
-
|
9
|
-
Unless required by applicable law or agreed to in writing, software
|
10
|
-
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
-
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
-
See the License for the specific language governing permissions and
|
13
|
-
limitations under the License.
|
14
|
-
|
data/Rakefile
DELETED
@@ -1,15 +0,0 @@
|
|
1
|
-
require "bundler/gem_tasks"
|
2
|
-
|
3
|
-
begin
|
4
|
-
require 'rspec/core/rake_task'
|
5
|
-
RSpec::Core::RakeTask.new(:spec)
|
6
|
-
rescue LoadError
|
7
|
-
end
|
8
|
-
|
9
|
-
task default: "spec"
|
10
|
-
|
11
|
-
task "vendor_jars" do
|
12
|
-
sh "mvn dependency:copy-dependencies -DoutputDirectory=vendor/jar-dependencies/runtime-jars/"
|
13
|
-
end
|
14
|
-
|
15
|
-
task build: "vendor_jars"
|
data/pom.xml
DELETED
@@ -1,32 +0,0 @@
|
|
1
|
-
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
2
|
-
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
3
|
-
<modelVersion>4.0.0</modelVersion>
|
4
|
-
<groupId>codekitchen</groupId>
|
5
|
-
<artifactId>logstash-input-kinesis</artifactId>
|
6
|
-
<version>dev</version>
|
7
|
-
<dependencies>
|
8
|
-
<dependency>
|
9
|
-
<groupId>org.apache.httpcomponents</groupId>
|
10
|
-
<artifactId>httpclient</artifactId>
|
11
|
-
<version>4.5.1</version>
|
12
|
-
</dependency>
|
13
|
-
<dependency>
|
14
|
-
<groupId>com.amazonaws</groupId>
|
15
|
-
<artifactId>amazon-kinesis-client</artifactId>
|
16
|
-
<version>1.6.1</version>
|
17
|
-
</dependency>
|
18
|
-
</dependencies>
|
19
|
-
<build>
|
20
|
-
<plugins>
|
21
|
-
<plugin>
|
22
|
-
<groupId>org.apache.maven.plugins</groupId>
|
23
|
-
<artifactId>maven-dependency-plugin</artifactId>
|
24
|
-
<executions>
|
25
|
-
<execution>
|
26
|
-
<id>copy-dependencies</id>
|
27
|
-
</execution>
|
28
|
-
</executions>
|
29
|
-
</plugin>
|
30
|
-
</plugins>
|
31
|
-
</build>
|
32
|
-
</project>
|