logstash-output-kafka 1.0.0 → 2.0.0.beta.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +2 -0
- data/README.md +21 -9
- data/lib/logstash/outputs/kafka.rb +89 -106
- data/logstash-output-kafka.gemspec +5 -2
- data/spec/integration/outputs/kafka_spec.rb +103 -0
- data/spec/{outputs → unit/outputs}/kafka_spec.rb +10 -15
- metadata +38 -8
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a2297207e9c66650a481b7f8e3c88a6a6d66600c
|
4
|
+
data.tar.gz: ae56fa5de77a72b925188abd2ba503d4d76fd79a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f64bc25142b7a79159e62b794cd0fea3a2fba8c9c2b637c5b6ae56f0e74a426828c6c064a907cf2419cf62f3ffc9c35b53e0c9e852da253912edf08c8483e3d5
|
7
|
+
data.tar.gz: c8c0cc0ae90908783237fd53d188300fc757b92a43aa20f804a2962622991b768eda24a9d80a0062c4f31b48c235b0f6766c06db58568427cce87b42ab20f80d
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
# Logstash Plugin
|
2
2
|
|
3
|
-
This is a plugin for [Logstash](https://github.com/
|
3
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
4
4
|
|
5
5
|
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
|
6
6
|
|
7
7
|
## Documentation
|
8
8
|
|
9
|
-
Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.
|
9
|
+
Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
|
10
10
|
|
11
11
|
- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
|
12
|
-
- For more asciidoc formatting tips, see the excellent reference here https://github.com/
|
12
|
+
- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
|
13
13
|
|
14
14
|
## Need Help?
|
15
15
|
|
@@ -25,6 +25,7 @@ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/log
|
|
25
25
|
- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
|
26
26
|
|
27
27
|
- Install dependencies
|
28
|
+
|
28
29
|
```sh
|
29
30
|
bundle install
|
30
31
|
```
|
@@ -37,19 +38,30 @@ bundle install
|
|
37
38
|
bundle install
|
38
39
|
```
|
39
40
|
|
40
|
-
- Run tests
|
41
|
+
- Run unit tests
|
41
42
|
|
42
43
|
```sh
|
43
44
|
bundle exec rspec
|
44
45
|
```
|
45
46
|
|
47
|
+
- Run integration tests
|
48
|
+
|
49
|
+
you'll need to have docker available within your test environment before
|
50
|
+
running the integration tests. The tests depend on a specific Kafka image
|
51
|
+
found in Docker Hub called `spotify/kafka`. You will need internet connectivity
|
52
|
+
to pull in this image if it does not already exist locally.
|
53
|
+
|
54
|
+
```sh
|
55
|
+
bundle exec rspec --tag integration
|
56
|
+
```
|
57
|
+
|
46
58
|
### 2. Running your unpublished Plugin in Logstash
|
47
59
|
|
48
60
|
#### 2.1 Run in a local Logstash clone
|
49
61
|
|
50
62
|
- Edit Logstash `Gemfile` and add the local plugin path, for example:
|
51
63
|
```ruby
|
52
|
-
gem "logstash-
|
64
|
+
gem "logstash-output-kafka", :path => "/your/local/logstash-output-kafka"
|
53
65
|
```
|
54
66
|
- Install plugin
|
55
67
|
```sh
|
@@ -57,7 +69,7 @@ bin/plugin install --no-verify
|
|
57
69
|
```
|
58
70
|
- Run Logstash with your plugin
|
59
71
|
```sh
|
60
|
-
bin/logstash -e '
|
72
|
+
bin/logstash -e 'output { kafka { topic_id => "kafka_topic" }}'
|
61
73
|
```
|
62
74
|
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
|
63
75
|
|
@@ -67,11 +79,11 @@ You can use the same **2.1** method to run your plugin in an installed Logstash
|
|
67
79
|
|
68
80
|
- Build your plugin gem
|
69
81
|
```sh
|
70
|
-
gem build logstash-
|
82
|
+
gem build logstash-output-kafka.gemspec
|
71
83
|
```
|
72
84
|
- Install the plugin from the Logstash home
|
73
85
|
```sh
|
74
|
-
bin/plugin install /your/local/plugin/logstash-
|
86
|
+
bin/plugin install /your/local/plugin/logstash-output-kafka.gem
|
75
87
|
```
|
76
88
|
- Start Logstash and proceed to test the plugin
|
77
89
|
|
@@ -83,4 +95,4 @@ Programming is not a required skill. Whatever you've seen about open source and
|
|
83
95
|
|
84
96
|
It is more important to the community that you are able to contribute.
|
85
97
|
|
86
|
-
For more information about contributing, see the [CONTRIBUTING](https://github.com/
|
98
|
+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
|
@@ -20,132 +20,117 @@ require 'jruby-kafka'
|
|
20
20
|
# }
|
21
21
|
# For more information see http://kafka.apache.org/documentation.html#theproducer
|
22
22
|
#
|
23
|
-
# Kafka producer configuration: http://kafka.apache.org/documentation.html#
|
23
|
+
# Kafka producer configuration: http://kafka.apache.org/documentation.html#newproducerconfigs
|
24
24
|
class LogStash::Outputs::Kafka < LogStash::Outputs::Base
|
25
25
|
config_name 'kafka'
|
26
26
|
|
27
27
|
default :codec, 'json'
|
28
|
+
|
29
|
+
# The topic to produce messages to
|
30
|
+
config :topic_id, :validate => :string, :required => true
|
28
31
|
# This is for bootstrapping and the producer will only use it for getting metadata (topics,
|
29
32
|
# partitions and replicas). The socket connections for sending the actual data will be
|
30
33
|
# established based on the broker information returned in the metadata. The format is
|
31
34
|
# `host1:port1,host2:port2`, and the list can be a subset of brokers or a VIP pointing to a
|
32
35
|
# subset of brokers.
|
33
|
-
config :
|
34
|
-
#
|
35
|
-
config :
|
36
|
-
#
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
#
|
41
|
-
#
|
42
|
-
#
|
43
|
-
#
|
44
|
-
|
45
|
-
#
|
46
|
-
#
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
# The
|
52
|
-
|
53
|
-
#
|
54
|
-
#
|
55
|
-
|
56
|
-
#
|
57
|
-
|
58
|
-
#
|
59
|
-
|
60
|
-
|
61
|
-
#
|
62
|
-
#
|
63
|
-
|
64
|
-
#
|
65
|
-
|
66
|
-
#
|
67
|
-
|
68
|
-
# This
|
69
|
-
#
|
70
|
-
#
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
#
|
75
|
-
|
76
|
-
#
|
36
|
+
config :bootstrap_servers, :validate => :string, :default => 'localhost:9092'
|
37
|
+
# Serializer class for the key of the message
|
38
|
+
config :key_serializer, :validate => :string, :default => 'org.apache.kafka.common.serialization.StringSerializer'
|
39
|
+
# Serializer class for the value of the message
|
40
|
+
config :value_serializer, :validate => :string, :default => 'org.apache.kafka.common.serialization.StringSerializer'
|
41
|
+
# The key for the message
|
42
|
+
config :message_key, :validate => :string
|
43
|
+
# The number of acknowledgments the producer requires the leader to have received
|
44
|
+
# before considering a request complete.
|
45
|
+
#
|
46
|
+
# acks=0, the producer will not wait for any acknowledgment from the server at all.
|
47
|
+
# acks=1, This will mean the leader will write the record to its local log but
|
48
|
+
# will respond without awaiting full acknowledgement from all followers.
|
49
|
+
# acks=all, This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
|
50
|
+
config :acks, :validate => ["0", "1", "all"], :default => "1"
|
51
|
+
# The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
|
52
|
+
config :buffer_memory, :validate => :number, :default => 33554432
|
53
|
+
# The compression type for all data generated by the producer.
|
54
|
+
# The default is none (i.e. no compression). Valid values are none, gzip, or snappy.
|
55
|
+
config :compression_type, :validate => ["none", "gzip", "snappy"], :default => "none"
|
56
|
+
# Setting a value greater than zero will cause the client to
|
57
|
+
# resend any record whose send fails with a potentially transient error.
|
58
|
+
config :retries, :validate => :number, :default => 0
|
59
|
+
# The producer will attempt to batch records together into fewer requests whenever multiple
|
60
|
+
# records are being sent to the same partition. This helps performance on both the client
|
61
|
+
# and the server. This configuration controls the default batch size in bytes.
|
62
|
+
config :batch_size, :validate => :number, :default => 16384
|
63
|
+
# The id string to pass to the server when making requests.
|
64
|
+
# The purpose of this is to be able to track the source of requests beyond just
|
65
|
+
# ip/port by allowing a logical application name to be included with the request
|
66
|
+
config :client_id, :validate => :string
|
67
|
+
# The producer groups together any records that arrive in between request
|
68
|
+
# transmissions into a single batched request. Normally this occurs only under
|
69
|
+
# load when records arrive faster than they can be sent out. However in some circumstances
|
70
|
+
# the client may want to reduce the number of requests even under moderate load.
|
71
|
+
# This setting accomplishes this by adding a small amount of artificial delay—that is,
|
72
|
+
# rather than immediately sending out a record the producer will wait for up to the given delay
|
73
|
+
# to allow other records to be sent so that the sends can be batched together.
|
74
|
+
config :linger_ms, :validate => :number, :default => 0
|
75
|
+
# The maximum size of a request
|
76
|
+
config :max_request_size, :validate => :number, :default => 1048576
|
77
|
+
# The size of the TCP receive buffer to use when reading data
|
78
|
+
config :receive_buffer_bytes, :validate => :number, :default => 32768
|
79
|
+
# The size of the TCP send buffer to use when sending data.
|
80
|
+
config :send_buffer_bytes, :validate => :number, :default => 131072
|
81
|
+
# The configuration controls the maximum amount of time the server will wait for acknowledgments
|
82
|
+
# from followers to meet the acknowledgment requirements the producer has specified with the
|
83
|
+
# acks configuration. If the requested number of acknowledgments are not met when the timeout
|
84
|
+
# elapses an error will be returned. This timeout is measured on the server side and does not
|
85
|
+
# include the network latency of the request.
|
86
|
+
config :timeout_ms, :validate => :number, :default => 30000
|
87
|
+
# When our memory buffer is exhausted we must either stop accepting new
|
88
|
+
# records (block) or throw errors. By default this setting is true and we block,
|
89
|
+
# however in some scenarios blocking is not desirable and it is better to immediately give an error.
|
90
|
+
config :block_on_buffer_full, :validate => :boolean, :default => true
|
91
|
+
# the timeout setting for initial metadata request to fetch topic metadata.
|
92
|
+
config :metadata_fetch_timeout_ms, :validate => :number, :default => 60000
|
93
|
+
# the max time in milliseconds before a metadata refresh is forced.
|
94
|
+
config :metadata_max_age_ms, :validate => :number, :default => 300000
|
95
|
+
# The amount of time to wait before attempting to reconnect to a given host when a connection fails.
|
96
|
+
config :reconnect_backoff_ms, :validate => :number, :default => 10
|
97
|
+
# The amount of time to wait before attempting to retry a failed produce request to a given topic partition.
|
77
98
|
config :retry_backoff_ms, :validate => :number, :default => 100
|
78
|
-
# The producer generally refreshes the topic metadata from brokers when there is a failure
|
79
|
-
# (partition missing, leader not available...). It will also poll regularly (default: every
|
80
|
-
# 10min so 600000ms). If you set this to a negative value, metadata will only get refreshed on
|
81
|
-
# failure. If you set this to zero, the metadata will get refreshed after each message sent
|
82
|
-
# (not recommended). Important note: the refresh happen only AFTER the message is sent,
|
83
|
-
# so if the producer never sends a message the metadata is never refreshed
|
84
|
-
config :topic_metadata_refresh_interval_ms, :validate => :number, :default => 600 * 1000
|
85
|
-
# Maximum time to buffer data when using async mode. For example a setting of 100 will try to
|
86
|
-
# batch together 100ms of messages to send at once. This will improve throughput but adds
|
87
|
-
# message delivery latency due to the buffering.
|
88
|
-
config :queue_buffering_max_ms, :validate => :number, :default => 5000
|
89
|
-
# The maximum number of unsent messages that can be queued up the producer when using async
|
90
|
-
# mode before either the producer must be blocked or data must be dropped.
|
91
|
-
config :queue_buffering_max_messages, :validate => :number, :default => 10000
|
92
|
-
# The amount of time to block before dropping messages when running in async mode and the
|
93
|
-
# buffer has reached `queue.buffering.max.messages`. If set to 0 events will be enqueued
|
94
|
-
# immediately or dropped if the queue is full (the producer send call will never block). If set
|
95
|
-
# to -1 the producer will block indefinitely and never willingly drop a send.
|
96
|
-
config :queue_enqueue_timeout_ms, :validate => :number, :default => -1
|
97
|
-
# The number of messages to send in one batch when using async mode. The producer will wait
|
98
|
-
# until either this number of messages are ready to send or `queue.buffer.max.ms` is reached.
|
99
|
-
config :batch_num_messages, :validate => :number, :default => 200
|
100
|
-
# Socket write buffer size
|
101
|
-
config :send_buffer_bytes, :validate => :number, :default => 100 * 1024
|
102
|
-
# The client id is a user-specified string sent in each request to help trace calls. It should
|
103
|
-
# logically identify the application making the request.
|
104
|
-
config :client_id, :validate => :string, :default => ''
|
105
|
-
# Provides a way to specify a partition key as a string. To specify a partition key for
|
106
|
-
# Kafka, configure a format that will produce the key as a string. Defaults
|
107
|
-
# `key_serializer_class` to `kafka.serializer.StringEncoder` to match. For example, to partition
|
108
|
-
# by host:
|
109
|
-
# [source,ruby]
|
110
|
-
# output {
|
111
|
-
# kafka {
|
112
|
-
# partition_key_format => "%{host}"
|
113
|
-
# }
|
114
|
-
# }
|
115
|
-
config :partition_key_format, :validate => :string, :default => nil
|
116
99
|
|
117
100
|
public
|
118
101
|
def register
|
119
102
|
LogStash::Logger.setup_log4j(@logger)
|
120
103
|
|
121
104
|
options = {
|
122
|
-
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
105
|
+
:key_serializer => @key_serializer,
|
106
|
+
:value_serializer => @value_serializer,
|
107
|
+
:bootstrap_servers => @bootstrap_servers,
|
108
|
+
:acks => @acks,
|
109
|
+
:buffer_memory => @buffer_memory,
|
110
|
+
:compression_type => @compression_type,
|
111
|
+
:retries => @retries,
|
112
|
+
:batch_size => @batch_size,
|
113
|
+
:client_id => @client_id,
|
114
|
+
:linger_ms => @linger_ms,
|
115
|
+
:max_request_size => @max_request_size,
|
116
|
+
:receive_buffer_bytes => @receive_buffer_bytes,
|
117
|
+
:send_buffer_bytes => @send_buffer_bytes,
|
118
|
+
:timeout_ms => @timeout_ms,
|
119
|
+
:block_on_buffer_full => @block_on_buffer_full,
|
120
|
+
:metadata_fetch_timeout_ms => @metadata_fetch_timeout_ms,
|
121
|
+
:metadata_max_age_ms => @metadata_max_age_ms,
|
122
|
+
:reconnect_backoff_ms => @reconnect_backoff_ms,
|
123
|
+
:retry_backoff_ms => @retry_backoff_ms
|
140
124
|
}
|
141
|
-
@producer = Kafka::
|
125
|
+
@producer = Kafka::KafkaProducer.new(options)
|
142
126
|
@producer.connect
|
143
127
|
|
144
|
-
@logger.info('Registering kafka producer', :topic_id => @topic_id, :
|
128
|
+
@logger.info('Registering kafka producer', :topic_id => @topic_id, :bootstrap_servers => @bootstrap_servers)
|
145
129
|
|
146
130
|
@codec.on_event do |event, data|
|
147
131
|
begin
|
148
|
-
@
|
132
|
+
key = if @message_key.nil? then nil else event.sprintf(@message_key) end
|
133
|
+
@producer.send_msg(event.sprintf(@topic_id), nil, key, data)
|
149
134
|
rescue LogStash::ShutdownSignal
|
150
135
|
@logger.info('Kafka producer got shutdown signal')
|
151
136
|
rescue => e
|
@@ -161,9 +146,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
|
|
161
146
|
finished
|
162
147
|
return
|
163
148
|
end
|
164
|
-
@partition_key = if @partition_key_format.nil? then nil else event.sprintf(@partition_key_format) end
|
165
149
|
@codec.encode(event)
|
166
|
-
@partition_key = nil
|
167
150
|
end
|
168
151
|
|
169
152
|
def teardown
|
@@ -1,7 +1,7 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
2
|
|
3
3
|
s.name = 'logstash-output-kafka'
|
4
|
-
s.version = '
|
4
|
+
s.version = '2.0.0.beta.1'
|
5
5
|
s.licenses = ['Apache License (2.0)']
|
6
6
|
s.summary = 'Output events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on the broker'
|
7
7
|
s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
|
@@ -24,6 +24,9 @@ Gem::Specification.new do |s|
|
|
24
24
|
s.add_runtime_dependency 'logstash-codec-plain'
|
25
25
|
s.add_runtime_dependency 'logstash-codec-json'
|
26
26
|
|
27
|
-
s.add_runtime_dependency 'jruby-kafka',
|
27
|
+
s.add_runtime_dependency 'jruby-kafka', '>= 1.4.0', '< 2.0.0'
|
28
|
+
|
28
29
|
s.add_development_dependency 'logstash-devutils'
|
30
|
+
s.add_development_dependency 'poseidon'
|
31
|
+
s.add_development_dependency 'longshoreman'
|
29
32
|
end
|
@@ -0,0 +1,103 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
|
3
|
+
require "logstash/devutils/rspec/spec_helper"
|
4
|
+
require 'logstash/outputs/kafka'
|
5
|
+
require 'longshoreman'
|
6
|
+
require 'jruby-kafka'
|
7
|
+
require 'json'
|
8
|
+
require 'poseidon'
|
9
|
+
|
10
|
+
CONTAINER_NAME = "kafka-#{rand(999).to_s}"
|
11
|
+
IMAGE_NAME = 'spotify/kafka'
|
12
|
+
IMAGE_TAG = 'latest'
|
13
|
+
KAFKA_PORT = 9092
|
14
|
+
KAFKA_VERSION = "0.8.2.1"
|
15
|
+
KAFKA_SCALA_VERSION = "2.11"
|
16
|
+
|
17
|
+
RSpec.configure do |config|
|
18
|
+
config.before(:all, :integration => true) do
|
19
|
+
@ls = begin
|
20
|
+
ls = Longshoreman.new
|
21
|
+
ls.container.get(CONTAINER_NAME)
|
22
|
+
ls
|
23
|
+
rescue Docker::Error::NotFoundError
|
24
|
+
Longshoreman.pull_image(IMAGE_NAME, IMAGE_TAG)
|
25
|
+
Longshoreman.new("#{IMAGE_NAME}:#{IMAGE_TAG}", CONTAINER_NAME,
|
26
|
+
{ "ENV" => [ "ADVERTISED_HOST=#{Longshoreman.new.get_host_ip}", "ADVERTISED_PORT=#{KAFKA_PORT}"],
|
27
|
+
"PortBindings" => { "#{KAFKA_PORT}/tcp" => [{ "HostPort" => "#{KAFKA_PORT}" }]}})
|
28
|
+
end
|
29
|
+
@kafka_host = @ls.get_host_ip
|
30
|
+
@kafka_port = @ls.container.rport(9092)
|
31
|
+
@zk_port = @ls.container.rport(2181)
|
32
|
+
end
|
33
|
+
|
34
|
+
config.after(:suite) do
|
35
|
+
begin
|
36
|
+
ls = Longshoreman::new
|
37
|
+
ls.container.get(CONTAINER_NAME)
|
38
|
+
ls.cleanup
|
39
|
+
rescue Docker::Error::NotFoundError, Excon::Errors::SocketError
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
describe "outputs/kafka", :integration => true do
|
45
|
+
let(:test_topic) { 'test' }
|
46
|
+
let(:base_config) { {'client_id' => 'spectest', 'bootstrap_servers' => "#{@kafka_host}:#{@kafka_port}"} }
|
47
|
+
let(:event) { LogStash::Event.new({'message' => 'hello', '@timestamp' => LogStash::Timestamp.at(0) }) }
|
48
|
+
|
49
|
+
|
50
|
+
context 'when outputting messages' do
|
51
|
+
let(:num_events) { 3 }
|
52
|
+
let(:consumer) do
|
53
|
+
Poseidon::PartitionConsumer.new("my_test_consumer", @kafka_host, @kafka_port,
|
54
|
+
test_topic, 0, :earliest_offset)
|
55
|
+
end
|
56
|
+
subject do
|
57
|
+
consumer.fetch
|
58
|
+
end
|
59
|
+
|
60
|
+
before :each do
|
61
|
+
config = base_config.merge({"topic_id" => test_topic})
|
62
|
+
kafka = LogStash::Outputs::Kafka.new(config)
|
63
|
+
kafka.register
|
64
|
+
num_events.times do kafka.receive(event) end
|
65
|
+
kafka.teardown
|
66
|
+
end
|
67
|
+
|
68
|
+
it 'should have data integrity' do
|
69
|
+
expect(subject.size).to eq(num_events)
|
70
|
+
subject.each do |m|
|
71
|
+
expect(m.value).to eq(event.to_json)
|
72
|
+
end
|
73
|
+
end
|
74
|
+
end
|
75
|
+
|
76
|
+
context 'when setting message_key' do
|
77
|
+
let(:num_events) { 10 }
|
78
|
+
let(:test_topic) { 'test2' }
|
79
|
+
let!(:consumer0) do
|
80
|
+
Poseidon::PartitionConsumer.new("my_test_consumer", @kafka_host, @kafka_port,
|
81
|
+
test_topic, 0, :earliest_offset)
|
82
|
+
end
|
83
|
+
let!(:consumer1) do
|
84
|
+
Poseidon::PartitionConsumer.new("my_test_consumer", @kafka_host, @kafka_port,
|
85
|
+
test_topic, 1, :earliest_offset)
|
86
|
+
end
|
87
|
+
|
88
|
+
before :each do
|
89
|
+
command = ["/opt/kafka_#{KAFKA_SCALA_VERSION}-#{KAFKA_VERSION}/bin/kafka-topics.sh", "--create", "--topic", "#{test_topic}", "--partitions", "2", "--zookeeper", "#{@kafka_host}:#{@zk_port}", "--replication-factor", "1"]
|
90
|
+
@ls.container.raw.exec(command)
|
91
|
+
|
92
|
+
config = base_config.merge({"topic_id" => test_topic, "message_key" => "static_key"})
|
93
|
+
kafka = LogStash::Outputs::Kafka.new(config)
|
94
|
+
kafka.register
|
95
|
+
num_events.times do kafka.receive(event) end
|
96
|
+
kafka.teardown
|
97
|
+
end
|
98
|
+
|
99
|
+
it 'should send all events to one partition' do
|
100
|
+
expect(consumer0.fetch.size == num_events || consumer1.fetch.size == num_events).to be true
|
101
|
+
end
|
102
|
+
end
|
103
|
+
end
|
@@ -17,19 +17,16 @@ describe "outputs/kafka" do
|
|
17
17
|
|
18
18
|
it 'should populate kafka config with default values' do
|
19
19
|
kafka = LogStash::Outputs::Kafka.new(simple_kafka_config)
|
20
|
-
insist {kafka.
|
20
|
+
insist {kafka.bootstrap_servers} == 'localhost:9092'
|
21
21
|
insist {kafka.topic_id} == 'test'
|
22
|
-
insist {kafka.
|
23
|
-
insist {kafka.serializer_class} == 'kafka.serializer.StringEncoder'
|
24
|
-
insist {kafka.partitioner_class} == 'kafka.producer.DefaultPartitioner'
|
25
|
-
insist {kafka.producer_type} == 'sync'
|
22
|
+
insist {kafka.key_serializer} == 'org.apache.kafka.common.serialization.StringSerializer'
|
26
23
|
end
|
27
24
|
end
|
28
25
|
|
29
26
|
context 'when outputting messages' do
|
30
27
|
it 'should send logstash event to kafka broker' do
|
31
|
-
expect_any_instance_of(Kafka::
|
32
|
-
.with(simple_kafka_config['topic_id'], nil, event.to_json)
|
28
|
+
expect_any_instance_of(Kafka::KafkaProducer).to receive(:send_msg)
|
29
|
+
.with(simple_kafka_config['topic_id'], nil, nil, event.to_hash.to_json)
|
33
30
|
kafka = LogStash::Outputs::Kafka.new(simple_kafka_config)
|
34
31
|
kafka.register
|
35
32
|
kafka.receive(event)
|
@@ -37,19 +34,17 @@ describe "outputs/kafka" do
|
|
37
34
|
|
38
35
|
it 'should support Event#sprintf placeholders in topic_id' do
|
39
36
|
topic_field = 'topic_name'
|
40
|
-
expect_any_instance_of(Kafka::
|
41
|
-
.with(event[topic_field], nil, event.to_json)
|
37
|
+
expect_any_instance_of(Kafka::KafkaProducer).to receive(:send_msg)
|
38
|
+
.with(event[topic_field], nil, nil, event.to_hash.to_json)
|
42
39
|
kafka = LogStash::Outputs::Kafka.new({'topic_id' => "%{#{topic_field}}"})
|
43
40
|
kafka.register
|
44
41
|
kafka.receive(event)
|
45
42
|
end
|
46
43
|
|
47
|
-
it 'should support
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
kafka = LogStash::Outputs::Kafka.new({'topic_id' => simple_kafka_config['topic_id'],
|
52
|
-
'partition_key_format' => "%{#{partition_field}}"})
|
44
|
+
it 'should support field referenced message_keys' do
|
45
|
+
expect_any_instance_of(Kafka::KafkaProducer).to receive(:send_msg)
|
46
|
+
.with(simple_kafka_config['topic_id'], nil, event['host'], event.to_hash.to_json)
|
47
|
+
kafka = LogStash::Outputs::Kafka.new(simple_kafka_config.merge({"message_key" => "%{host}"}))
|
53
48
|
kafka.register
|
54
49
|
kafka.receive(event)
|
55
50
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-output-kafka
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0.beta.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Elasticsearch
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-09-01 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: logstash-core
|
@@ -64,7 +64,7 @@ dependencies:
|
|
64
64
|
requirements:
|
65
65
|
- - '>='
|
66
66
|
- !ruby/object:Gem::Version
|
67
|
-
version: 1.
|
67
|
+
version: 1.4.0
|
68
68
|
- - <
|
69
69
|
- !ruby/object:Gem::Version
|
70
70
|
version: 2.0.0
|
@@ -72,7 +72,7 @@ dependencies:
|
|
72
72
|
requirements:
|
73
73
|
- - '>='
|
74
74
|
- !ruby/object:Gem::Version
|
75
|
-
version: 1.
|
75
|
+
version: 1.4.0
|
76
76
|
- - <
|
77
77
|
- !ruby/object:Gem::Version
|
78
78
|
version: 2.0.0
|
@@ -92,6 +92,34 @@ dependencies:
|
|
92
92
|
version: '0'
|
93
93
|
prerelease: false
|
94
94
|
type: :development
|
95
|
+
- !ruby/object:Gem::Dependency
|
96
|
+
name: poseidon
|
97
|
+
version_requirements: !ruby/object:Gem::Requirement
|
98
|
+
requirements:
|
99
|
+
- - '>='
|
100
|
+
- !ruby/object:Gem::Version
|
101
|
+
version: '0'
|
102
|
+
requirement: !ruby/object:Gem::Requirement
|
103
|
+
requirements:
|
104
|
+
- - '>='
|
105
|
+
- !ruby/object:Gem::Version
|
106
|
+
version: '0'
|
107
|
+
prerelease: false
|
108
|
+
type: :development
|
109
|
+
- !ruby/object:Gem::Dependency
|
110
|
+
name: longshoreman
|
111
|
+
version_requirements: !ruby/object:Gem::Requirement
|
112
|
+
requirements:
|
113
|
+
- - '>='
|
114
|
+
- !ruby/object:Gem::Version
|
115
|
+
version: '0'
|
116
|
+
requirement: !ruby/object:Gem::Requirement
|
117
|
+
requirements:
|
118
|
+
- - '>='
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
version: '0'
|
121
|
+
prerelease: false
|
122
|
+
type: :development
|
95
123
|
description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
|
96
124
|
email: info@elastic.co
|
97
125
|
executables: []
|
@@ -109,7 +137,8 @@ files:
|
|
109
137
|
- Rakefile
|
110
138
|
- lib/logstash/outputs/kafka.rb
|
111
139
|
- logstash-output-kafka.gemspec
|
112
|
-
- spec/outputs/kafka_spec.rb
|
140
|
+
- spec/integration/outputs/kafka_spec.rb
|
141
|
+
- spec/unit/outputs/kafka_spec.rb
|
113
142
|
homepage: http://www.elastic.co/guide/en/logstash/current/index.html
|
114
143
|
licenses:
|
115
144
|
- Apache License (2.0)
|
@@ -127,9 +156,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
127
156
|
version: '0'
|
128
157
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
129
158
|
requirements:
|
130
|
-
- - '
|
159
|
+
- - '>'
|
131
160
|
- !ruby/object:Gem::Version
|
132
|
-
version:
|
161
|
+
version: 1.3.1
|
133
162
|
requirements: []
|
134
163
|
rubyforge_project:
|
135
164
|
rubygems_version: 2.2.2
|
@@ -137,4 +166,5 @@ signing_key:
|
|
137
166
|
specification_version: 4
|
138
167
|
summary: Output events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on the broker
|
139
168
|
test_files:
|
140
|
-
- spec/outputs/kafka_spec.rb
|
169
|
+
- spec/integration/outputs/kafka_spec.rb
|
170
|
+
- spec/unit/outputs/kafka_spec.rb
|