logstash-input-kinesis 1.4.3-java → 1.5.0-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CONTRIBUTORS +14 -0
- data/README.md +5 -4
- data/lib/com/amazonaws/amazon-kinesis-client/1.6.1/amazon-kinesis-client-1.6.1.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-cloudwatch/1.10.20/aws-java-sdk-cloudwatch-1.10.20.jar +0 -0
- data/{vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.10.50.jar → lib/com/amazonaws/aws-java-sdk-core/1.10.20/aws-java-sdk-core-1.10.20.jar} +0 -0
- data/lib/com/amazonaws/aws-java-sdk-dynamodb/1.10.20/aws-java-sdk-dynamodb-1.10.20.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-kinesis/1.10.20/aws-java-sdk-kinesis-1.10.20.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-kms/1.10.20/aws-java-sdk-kms-1.10.20.jar +0 -0
- data/lib/com/amazonaws/aws-java-sdk-s3/1.10.20/aws-java-sdk-s3-1.10.20.jar +0 -0
- data/lib/com/fasterxml/jackson/core/jackson-annotations/2.5.0/jackson-annotations-2.5.0.jar +0 -0
- data/lib/com/fasterxml/jackson/core/jackson-core/2.5.3/jackson-core-2.5.3.jar +0 -0
- data/lib/com/fasterxml/jackson/core/jackson-databind/2.5.3/jackson-databind-2.5.3.jar +0 -0
- data/lib/com/google/guava/guava/18.0/guava-18.0.jar +0 -0
- data/lib/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar +0 -0
- data/lib/commons-codec/commons-codec/1.9/commons-codec-1.9.jar +0 -0
- data/lib/commons-lang/commons-lang/2.6/commons-lang-2.6.jar +0 -0
- data/lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar +0 -0
- data/lib/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar +0 -0
- data/lib/logstash/inputs/kinesis.rb +1 -1
- data/lib/logstash/inputs/kinesis/version.rb +2 -1
- data/lib/logstash/inputs/kinesis/worker.rb +1 -0
- data/lib/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar +0 -0
- data/lib/org/apache/httpcomponents/httpcore/4.4.3/httpcore-4.4.3.jar +0 -0
- data/logstash-input-kinesis.gemspec +8 -3
- data/spec/inputs/kinesis/worker_spec.rb +1 -0
- data/spec/inputs/kinesis_spec.rb +12 -1
- metadata +50 -22
- data/.gitignore +0 -14
- data/.rspec +0 -2
- data/.travis.yml +0 -3
- data/LICENSE.txt +0 -14
- data/Rakefile +0 -15
- data/pom.xml +0 -32
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 03bbacd8aa1c1ead7454666582fa2ec84cd79d86
|
4
|
+
data.tar.gz: ae48dab34705083645faff5c7b7c3674ef658a8e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 04342e5d1a6e34138995d89cb894b732997d0d1b51378e46fd4db49823c7b0d8707fbe9bdbbb647b527240372a0ff59360f558866092cff849e91fca8a8b8fda
|
7
|
+
data.tar.gz: 86232000df97f16996c078caf62c6cd35f676c411ce1901966d4edb8da372267867a65739a75ab3a40267e4b6ac3c607702a8bdc69fa9e9e1288c4ca5a26b159
|
data/CONTRIBUTORS
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
The following is a list of people who have contributed ideas, code, bug
|
2
|
+
reports, or in general have helped logstash along its way.
|
3
|
+
|
4
|
+
Maintainers:
|
5
|
+
* Brian Palmer (codekitchen)
|
6
|
+
|
7
|
+
Contributors:
|
8
|
+
* Brian Palmer (codekitchen)
|
9
|
+
* Samuel García Martínez (samuelgmartinez)
|
10
|
+
|
11
|
+
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
12
|
+
Logstash, and you aren't on the list above and want to be, please let us know
|
13
|
+
and we'll make sure you're here. Contributions from folks like you are what make
|
14
|
+
open source awesome.
|
data/README.md
CHANGED
@@ -1,12 +1,12 @@
|
|
1
1
|
# Logstash AWS Kinesis Input Plugin
|
2
2
|
|
3
|
-
[](https://travis-ci.org/logstash-plugins/logstash-input-kinesis)
|
4
4
|
|
5
5
|
This is a [AWS Kinesis](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html) input plugin for [Logstash](https://github.com/elasticsearch/logstash). Under the hood uses the [Kinesis Client Library](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html).
|
6
6
|
|
7
7
|
## Installation
|
8
8
|
|
9
|
-
This plugin requires Logstash >=
|
9
|
+
This plugin requires Logstash >= 2.0, and can be installed by Logstash
|
10
10
|
itself.
|
11
11
|
|
12
12
|
```sh
|
@@ -39,7 +39,7 @@ This are the properties you can configure and what are the default values:
|
|
39
39
|
* `checkpoint_interval_seconds`: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
|
40
40
|
* **required**: false
|
41
41
|
* **default value**: `60`
|
42
|
-
* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
|
42
|
+
* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
|
43
43
|
* **required**: false
|
44
44
|
* **default value**: `nil`
|
45
45
|
|
@@ -61,7 +61,8 @@ Look at the [documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javado
|
|
61
61
|
|
62
62
|
## Contributing
|
63
63
|
|
64
|
-
|
64
|
+
0. https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md#contribution-steps
|
65
|
+
1. Fork it ( https://github.com/logstash-plugins/logstash-input-kinesis/fork )
|
65
66
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
66
67
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
67
68
|
4. Push to the branch (`git push origin my-new-feature`)
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
@@ -10,10 +10,10 @@ Gem::Specification.new do |spec|
|
|
10
10
|
spec.email = ["brian@codekitchen.net"]
|
11
11
|
spec.summary = %q{Logstash plugin for Kinesis input}
|
12
12
|
spec.description = %q{This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program}
|
13
|
-
spec.homepage = "https://github.com/
|
13
|
+
spec.homepage = "https://github.com/logstash-plugins/logstash-input-kinesis"
|
14
14
|
spec.licenses = ['Apache License (2.0)']
|
15
15
|
|
16
|
-
spec.files =
|
16
|
+
spec.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
|
17
17
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
18
18
|
spec.require_paths = ["lib"]
|
19
19
|
|
@@ -22,9 +22,14 @@ Gem::Specification.new do |spec|
|
|
22
22
|
|
23
23
|
spec.platform = 'java'
|
24
24
|
|
25
|
+
spec.requirements << "jar 'com.amazonaws:amazon-kinesis-client', '1.6.1'"
|
26
|
+
spec.requirements << "jar 'org.apache.httpcomponents:httpclient', '4.5.1'"
|
27
|
+
|
28
|
+
spec.add_development_dependency 'jar-dependencies', '0.3.2'
|
29
|
+
|
25
30
|
spec.add_development_dependency "bundler", "~> 1.7"
|
26
31
|
spec.add_development_dependency "rake", "~> 10.0"
|
27
32
|
spec.add_development_dependency "rspec", "~> 3.2.0"
|
28
|
-
spec.add_development_dependency "logstash-core", "
|
33
|
+
spec.add_development_dependency "logstash-core", "~> 2.4.0"
|
29
34
|
spec.add_development_dependency "logstash-codec-json"
|
30
35
|
end
|
@@ -48,6 +48,7 @@ RSpec.describe "LogStash::Inputs::Kinesis::Worker" do
|
|
48
48
|
describe "#processRecords" do
|
49
49
|
it "decodes and queues each record with decoration" do
|
50
50
|
worker.processRecords(process_input)
|
51
|
+
expect(queue.size).to eq(2)
|
51
52
|
m1 = queue.pop
|
52
53
|
m2 = queue.pop
|
53
54
|
expect(m1).to be_kind_of(LogStash::Event)
|
data/spec/inputs/kinesis_spec.rb
CHANGED
@@ -43,11 +43,22 @@ RSpec.describe "inputs/kinesis" do
|
|
43
43
|
end
|
44
44
|
end
|
45
45
|
|
46
|
+
context "#stop" do
|
47
|
+
it "stops the KCL worker" do
|
48
|
+
expect(kinesis).to receive(:kcl_builder).with(queue).and_return(stub_builder)
|
49
|
+
expect(kcl_worker).to receive(:run).with(no_args)
|
50
|
+
expect(kcl_worker).to receive(:shutdown).with(no_args)
|
51
|
+
kinesis.run(queue)
|
52
|
+
kinesis.do_stop # do_stop calls stop internally
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
46
56
|
context "#worker_factory" do
|
47
57
|
it "clones the codec for each worker" do
|
48
|
-
expect(codec).to receive(:clone).once
|
49
58
|
worker = kinesis.worker_factory(queue).call()
|
50
59
|
expect(worker).to be_kind_of(LogStash::Inputs::Kinesis::Worker)
|
60
|
+
expect(worker.codec).to_not eq(kinesis.codec)
|
61
|
+
expect(worker.codec).to be_kind_of(codec.class)
|
51
62
|
end
|
52
63
|
|
53
64
|
it "generates a valid worker" do
|
metadata
CHANGED
@@ -1,15 +1,29 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-input-kinesis
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.5.0
|
5
5
|
platform: java
|
6
6
|
authors:
|
7
7
|
- Brian Palmer
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-09-29 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
15
|
+
requirements:
|
16
|
+
- - '='
|
17
|
+
- !ruby/object:Gem::Version
|
18
|
+
version: 0.3.2
|
19
|
+
name: jar-dependencies
|
20
|
+
prerelease: false
|
21
|
+
type: :development
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - '='
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: 0.3.2
|
13
27
|
- !ruby/object:Gem::Dependency
|
14
28
|
requirement: !ruby/object:Gem::Requirement
|
15
29
|
requirements:
|
@@ -55,17 +69,17 @@ dependencies:
|
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
requirement: !ruby/object:Gem::Requirement
|
57
71
|
requirements:
|
58
|
-
- -
|
72
|
+
- - ~>
|
59
73
|
- !ruby/object:Gem::Version
|
60
|
-
version:
|
74
|
+
version: 2.4.0
|
61
75
|
name: logstash-core
|
62
76
|
prerelease: false
|
63
77
|
type: :development
|
64
78
|
version_requirements: !ruby/object:Gem::Requirement
|
65
79
|
requirements:
|
66
|
-
- -
|
80
|
+
- - ~>
|
67
81
|
- !ruby/object:Gem::Version
|
68
|
-
version:
|
82
|
+
version: 2.4.0
|
69
83
|
- !ruby/object:Gem::Dependency
|
70
84
|
requirement: !ruby/object:Gem::Requirement
|
71
85
|
requirements:
|
@@ -87,26 +101,34 @@ executables: []
|
|
87
101
|
extensions: []
|
88
102
|
extra_rdoc_files: []
|
89
103
|
files:
|
90
|
-
- .gitignore
|
91
|
-
- .rspec
|
92
|
-
- .travis.yml
|
93
|
-
- Gemfile
|
94
|
-
- LICENSE.txt
|
95
|
-
- README.md
|
96
|
-
- Rakefile
|
97
104
|
- lib/logstash-input-kinesis_jars.rb
|
105
|
+
- lib/com/amazonaws/amazon-kinesis-client/1.6.1/amazon-kinesis-client-1.6.1.jar
|
106
|
+
- lib/com/amazonaws/aws-java-sdk-cloudwatch/1.10.20/aws-java-sdk-cloudwatch-1.10.20.jar
|
107
|
+
- lib/com/amazonaws/aws-java-sdk-core/1.10.20/aws-java-sdk-core-1.10.20.jar
|
108
|
+
- lib/com/amazonaws/aws-java-sdk-dynamodb/1.10.20/aws-java-sdk-dynamodb-1.10.20.jar
|
109
|
+
- lib/com/amazonaws/aws-java-sdk-kinesis/1.10.20/aws-java-sdk-kinesis-1.10.20.jar
|
110
|
+
- lib/com/amazonaws/aws-java-sdk-kms/1.10.20/aws-java-sdk-kms-1.10.20.jar
|
111
|
+
- lib/com/amazonaws/aws-java-sdk-s3/1.10.20/aws-java-sdk-s3-1.10.20.jar
|
112
|
+
- lib/com/fasterxml/jackson/core/jackson-annotations/2.5.0/jackson-annotations-2.5.0.jar
|
113
|
+
- lib/com/fasterxml/jackson/core/jackson-core/2.5.3/jackson-core-2.5.3.jar
|
114
|
+
- lib/com/fasterxml/jackson/core/jackson-databind/2.5.3/jackson-databind-2.5.3.jar
|
115
|
+
- lib/com/google/guava/guava/18.0/guava-18.0.jar
|
116
|
+
- lib/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar
|
117
|
+
- lib/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
|
118
|
+
- lib/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
|
119
|
+
- lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
|
120
|
+
- lib/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar
|
98
121
|
- lib/logstash/inputs/kinesis.rb
|
99
122
|
- lib/logstash/inputs/kinesis/version.rb
|
100
123
|
- lib/logstash/inputs/kinesis/worker.rb
|
101
|
-
-
|
102
|
-
-
|
103
|
-
- spec/inputs/kinesis/worker_spec.rb
|
104
|
-
- spec/inputs/kinesis_spec.rb
|
124
|
+
- lib/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar
|
125
|
+
- lib/org/apache/httpcomponents/httpcore/4.4.3/httpcore-4.4.3.jar
|
105
126
|
- spec/spec_helper.rb
|
127
|
+
- spec/inputs/kinesis_spec.rb
|
128
|
+
- spec/inputs/kinesis/worker_spec.rb
|
106
129
|
- vendor/jar-dependencies/runtime-jars/amazon-kinesis-client-1.6.1.jar
|
107
130
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-cloudwatch-1.10.20.jar
|
108
131
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.10.20.jar
|
109
|
-
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-core-1.10.50.jar
|
110
132
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-dynamodb-1.10.20.jar
|
111
133
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-kinesis-1.10.20.jar
|
112
134
|
- vendor/jar-dependencies/runtime-jars/aws-java-sdk-kms-1.10.20.jar
|
@@ -122,7 +144,11 @@ files:
|
|
122
144
|
- vendor/jar-dependencies/runtime-jars/jackson-databind-2.5.3.jar
|
123
145
|
- vendor/jar-dependencies/runtime-jars/joda-time-2.8.1.jar
|
124
146
|
- vendor/jar-dependencies/runtime-jars/protobuf-java-2.6.1.jar
|
125
|
-
|
147
|
+
- logstash-input-kinesis.gemspec
|
148
|
+
- README.md
|
149
|
+
- CONTRIBUTORS
|
150
|
+
- Gemfile
|
151
|
+
homepage: https://github.com/logstash-plugins/logstash-input-kinesis
|
126
152
|
licenses:
|
127
153
|
- Apache License (2.0)
|
128
154
|
metadata:
|
@@ -142,13 +168,15 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
142
168
|
- - '>='
|
143
169
|
- !ruby/object:Gem::Version
|
144
170
|
version: '0'
|
145
|
-
requirements:
|
171
|
+
requirements:
|
172
|
+
- jar 'com.amazonaws:amazon-kinesis-client', '1.6.1'
|
173
|
+
- jar 'org.apache.httpcomponents:httpclient', '4.5.1'
|
146
174
|
rubyforge_project:
|
147
175
|
rubygems_version: 2.1.9
|
148
176
|
signing_key:
|
149
177
|
specification_version: 4
|
150
178
|
summary: Logstash plugin for Kinesis input
|
151
179
|
test_files:
|
152
|
-
- spec/inputs/kinesis/worker_spec.rb
|
153
|
-
- spec/inputs/kinesis_spec.rb
|
154
180
|
- spec/spec_helper.rb
|
181
|
+
- spec/inputs/kinesis_spec.rb
|
182
|
+
- spec/inputs/kinesis/worker_spec.rb
|
data/.gitignore
DELETED
data/.rspec
DELETED
data/.travis.yml
DELETED
data/LICENSE.txt
DELETED
@@ -1,14 +0,0 @@
|
|
1
|
-
Copyright (c) 2015 Brian Palmer
|
2
|
-
|
3
|
-
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
-
you may not use this file except in compliance with the License.
|
5
|
-
You may obtain a copy of the License at
|
6
|
-
|
7
|
-
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
-
|
9
|
-
Unless required by applicable law or agreed to in writing, software
|
10
|
-
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
-
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
-
See the License for the specific language governing permissions and
|
13
|
-
limitations under the License.
|
14
|
-
|
data/Rakefile
DELETED
@@ -1,15 +0,0 @@
|
|
1
|
-
require "bundler/gem_tasks"
|
2
|
-
|
3
|
-
begin
|
4
|
-
require 'rspec/core/rake_task'
|
5
|
-
RSpec::Core::RakeTask.new(:spec)
|
6
|
-
rescue LoadError
|
7
|
-
end
|
8
|
-
|
9
|
-
task default: "spec"
|
10
|
-
|
11
|
-
task "vendor_jars" do
|
12
|
-
sh "mvn dependency:copy-dependencies -DoutputDirectory=vendor/jar-dependencies/runtime-jars/"
|
13
|
-
end
|
14
|
-
|
15
|
-
task build: "vendor_jars"
|
data/pom.xml
DELETED
@@ -1,32 +0,0 @@
|
|
1
|
-
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
2
|
-
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
3
|
-
<modelVersion>4.0.0</modelVersion>
|
4
|
-
<groupId>codekitchen</groupId>
|
5
|
-
<artifactId>logstash-input-kinesis</artifactId>
|
6
|
-
<version>dev</version>
|
7
|
-
<dependencies>
|
8
|
-
<dependency>
|
9
|
-
<groupId>org.apache.httpcomponents</groupId>
|
10
|
-
<artifactId>httpclient</artifactId>
|
11
|
-
<version>4.5.1</version>
|
12
|
-
</dependency>
|
13
|
-
<dependency>
|
14
|
-
<groupId>com.amazonaws</groupId>
|
15
|
-
<artifactId>amazon-kinesis-client</artifactId>
|
16
|
-
<version>1.6.1</version>
|
17
|
-
</dependency>
|
18
|
-
</dependencies>
|
19
|
-
<build>
|
20
|
-
<plugins>
|
21
|
-
<plugin>
|
22
|
-
<groupId>org.apache.maven.plugins</groupId>
|
23
|
-
<artifactId>maven-dependency-plugin</artifactId>
|
24
|
-
<executions>
|
25
|
-
<execution>
|
26
|
-
<id>copy-dependencies</id>
|
27
|
-
</execution>
|
28
|
-
</executions>
|
29
|
-
</plugin>
|
30
|
-
</plugins>
|
31
|
-
</build>
|
32
|
-
</project>
|