logstash-output-kafka 1.0.0 → 2.0.0.beta.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 244f5f82cabf8fdb25da72d0ad0e3cc28bed07ee
-  data.tar.gz: 9bab5e3e1b612a563a55d18be6874c6ac28a274f
+  metadata.gz: a2297207e9c66650a481b7f8e3c88a6a6d66600c
+  data.tar.gz: ae56fa5de77a72b925188abd2ba503d4d76fd79a
 SHA512:
-  metadata.gz: b2e48ecad097e027f68849edeadadd351f4e7fd3fcf84b31bf20e4d705c4ae9bc1535c4b33d80813f2a8ba36ee42e9f105f5b0b70a2ee32e17e84ec8ddea3575
-  data.tar.gz: 4fe7cb1ef2af86a5ff943cfaab249443dd7a6033fff6db7686023bc9a213dce9a3fc5b52b479b07fd79be430bcbe904e29dcf25fa63592d9ac07626d41b0630e
+  metadata.gz: f64bc25142b7a79159e62b794cd0fea3a2fba8c9c2b637c5b6ae56f0e74a426828c6c064a907cf2419cf62f3ffc9c35b53e0c9e852da253912edf08c8483e3d5
+  data.tar.gz: c8c0cc0ae90908783237fd53d188300fc757b92a43aa20f804a2962622991b768eda24a9d80a0062c4f31b48c235b0f6766c06db58568427cce87b42ab20f80d

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+# ## 2.0.0.beta.1
+ - Change to 0.8.2 version of Kafka producer. This will unfortunately break existing configuration, but has a lot of enhancements and fixes.

data/README.md

@@ -1,15 +1,15 @@
 # Logstash Plugin
 
-This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
 ## Documentation
 
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
 
 - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
 
 ## Need Help?
 
@@ -25,6 +25,7 @@ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/log
 - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
 
 - Install dependencies
+
 ```sh
 bundle install
 ```
@@ -37,19 +38,30 @@ bundle install
 bundle install
 ```
 
-- Run tests
+- Run unit tests
 
 ```sh
 bundle exec rspec
 ```
 
+- Run integration tests
+
+you'll need to have docker available within your test environment before 
+running the integration tests. The tests depend on a specific Kafka image 
+found in Docker Hub called `spotify/kafka`. You will need internet connectivity
+to pull in this image if it does not already exist locally.
+
+```sh
+bundle exec rspec --tag integration
+```
+
 ### 2. Running your unpublished Plugin in Logstash
 
 #### 2.1 Run in a local Logstash clone
 
 - Edit Logstash `Gemfile` and add the local plugin path, for example:
 ```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+gem "logstash-output-kafka", :path => "/your/local/logstash-output-kafka"
 ```
 - Install plugin
 ```sh
@@ -57,7 +69,7 @@ bin/plugin install --no-verify
 ```
 - Run Logstash with your plugin
 ```sh
-bin/logstash -e 'filter {awesome {}}'
+bin/logstash -e 'output { kafka { topic_id => "kafka_topic" }}'
 ```
 At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
 
@@ -67,11 +79,11 @@ You can use the same **2.1** method to run your plugin in an installed Logstash
 
 - Build your plugin gem
 ```sh
-gem build logstash-filter-awesome.gemspec
+gem build logstash-output-kafka.gemspec
 ```
 - Install the plugin from the Logstash home
 ```sh
-bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+bin/plugin install /your/local/plugin/logstash-output-kafka.gem
 ```
 - Start Logstash and proceed to test the plugin
 
@@ -83,4 +95,4 @@ Programming is not a required skill. Whatever you've seen about open source and
 
 It is more important to the community that you are able to contribute.
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/kafka.rb

@@ -20,132 +20,117 @@ require 'jruby-kafka'
 #     }
 # For more information see http://kafka.apache.org/documentation.html#theproducer
 #
-# Kafka producer configuration: http://kafka.apache.org/documentation.html#producerconfigs
+# Kafka producer configuration: http://kafka.apache.org/documentation.html#newproducerconfigs
 class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   config_name 'kafka'
 
   default :codec, 'json'
+
+  # The topic to produce messages to
+  config :topic_id, :validate => :string, :required => true
   # This is for bootstrapping and the producer will only use it for getting metadata (topics,
   # partitions and replicas). The socket connections for sending the actual data will be
   # established based on the broker information returned in the metadata. The format is
   # `host1:port1,host2:port2`, and the list can be a subset of brokers or a VIP pointing to a
   # subset of brokers.
-  config :broker_list, :validate => :string, :default => 'localhost:9092'
-  # The topic to produce the messages to
-  config :topic_id, :validate => :string, :required => true
-  # This parameter allows you to specify the compression codec for all data generated by this
-  # producer. Valid values are `none`, `gzip` and `snappy`.
-  config :compression_codec, :validate => %w( none gzip snappy ), :default => 'none'
-  # This parameter allows you to set whether compression should be turned on for particular
-  # topics. If the compression codec is anything other than `NoCompressionCodec`,
-  # enable compression only for specified topics if any. If the list of compressed topics is
-  # empty, then enable the specified compression codec for all topics. If the compression codec
-  # is `NoCompressionCodec`, compression is disabled for all topics
-  config :compressed_topics, :validate => :string, :default => ''
-  # This value controls when a produce request is considered completed. Specifically,
-  # how many other brokers must have committed the data to their log and acknowledged this to the
-  # leader. For more info, see -- http://kafka.apache.org/documentation.html#producerconfigs
-  config :request_required_acks, :validate => [-1,0,1], :default => 0
-  # The serializer class for messages. The default encoder takes a byte[] and returns the same byte[]
-  config :serializer_class, :validate => :string, :default => 'kafka.serializer.StringEncoder'
-  # The partitioner class for partitioning messages amongst partitions in the topic. The default
-  # partitioner is based on the hash of the key. If the key is null,
-  # the message is sent to a random partition in the broker.
-  # NOTE: `topic_metadata_refresh_interval_ms` controls how long the producer will distribute to a
-  # partition in the topic. This defaults to 10 mins, so the producer will continue to write to a
-  # single partition for 10 mins before it switches
-  config :partitioner_class, :validate => :string, :default => 'kafka.producer.DefaultPartitioner'
-  # The amount of time the broker will wait trying to meet the `request.required.acks` requirement
-  # before sending back an error to the client.
-  config :request_timeout_ms, :validate => :number, :default => 10000
-  # This parameter specifies whether the messages are sent asynchronously in a background thread.
-  # Valid values are (1) async for asynchronous send and (2) sync for synchronous send. By
-  # setting the producer to async we allow batching together of requests (which is great for
-  # throughput) but open the possibility of a failure of the client machine dropping unsent data.
-  config :producer_type, :validate => %w( sync async ), :default => 'sync'
-  # The serializer class for keys (defaults to the same as for messages if nothing is given)
-  config :key_serializer_class, :validate => :string, :default => 'kafka.serializer.StringEncoder'
-  # This property will cause the producer to automatically retry a failed send request. This
-  # property specifies the number of retries when such failures occur. Note that setting a
-  # non-zero value here can lead to duplicates in the case of network errors that cause a message
-  # to be sent but the acknowledgement to be lost.
-  config :message_send_max_retries, :validate => :number, :default => 3
-  # Before each retry, the producer refreshes the metadata of relevant topics to see if a new
-  # leader has been elected. Since leader election takes a bit of time,
-  # this property specifies the amount of time that the producer waits before refreshing the
-  # metadata.
+  config :bootstrap_servers, :validate => :string, :default => 'localhost:9092'
+  # Serializer class for the key of the message
+  config :key_serializer, :validate => :string, :default => 'org.apache.kafka.common.serialization.StringSerializer'
+  # Serializer class for the value of the message
+  config :value_serializer, :validate => :string, :default => 'org.apache.kafka.common.serialization.StringSerializer'
+  # The key for the message
+  config :message_key, :validate => :string
+  # The number of acknowledgments the producer requires the leader to have received
+  # before considering a request complete.
+  #
+  # acks=0,   the producer will not wait for any acknowledgment from the server at all.
+  # acks=1,   This will mean the leader will write the record to its local log but
+  #           will respond without awaiting full acknowledgement from all followers.
+  # acks=all, This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
+  config :acks, :validate => ["0", "1", "all"], :default => "1"
+  # The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
+  config :buffer_memory, :validate => :number, :default => 33554432
+  # The compression type for all data generated by the producer.
+  # The default is none (i.e. no compression). Valid values are none, gzip, or snappy.
+  config :compression_type, :validate => ["none", "gzip", "snappy"], :default => "none"
+  # Setting a value greater than zero will cause the client to
+  # resend any record whose send fails with a potentially transient error.
+  config :retries, :validate => :number, :default => 0
+  # The producer will attempt to batch records together into fewer requests whenever multiple
+  # records are being sent to the same partition. This helps performance on both the client
+  # and the server. This configuration controls the default batch size in bytes.
+  config :batch_size, :validate => :number, :default => 16384
+  # The id string to pass to the server when making requests.
+  # The purpose of this is to be able to track the source of requests beyond just
+  # ip/port by allowing a logical application name to be included with the request
+  config :client_id, :validate => :string
+  # The producer groups together any records that arrive in between request
+  # transmissions into a single batched request. Normally this occurs only under
+  # load when records arrive faster than they can be sent out. However in some circumstances
+  # the client may want to reduce the number of requests even under moderate load.
+  # This setting accomplishes this by adding a small amount of artificial delay—that is,
+  # rather than immediately sending out a record the producer will wait for up to the given delay
+  # to allow other records to be sent so that the sends can be batched together.
+  config :linger_ms, :validate => :number, :default => 0
+  # The maximum size of a request
+  config :max_request_size, :validate => :number, :default => 1048576
+  # The size of the TCP receive buffer to use when reading data
+  config :receive_buffer_bytes, :validate => :number, :default => 32768
+  # The size of the TCP send buffer to use when sending data.
+  config :send_buffer_bytes, :validate => :number, :default => 131072
+  # The configuration controls the maximum amount of time the server will wait for acknowledgments
+  # from followers to meet the acknowledgment requirements the producer has specified with the
+  # acks configuration. If the requested number of acknowledgments are not met when the timeout
+  # elapses an error will be returned. This timeout is measured on the server side and does not
+  # include the network latency of the request.
+  config :timeout_ms, :validate => :number, :default => 30000
+  # When our memory buffer is exhausted we must either stop accepting new
+  # records (block) or throw errors. By default this setting is true and we block,
+  # however in some scenarios blocking is not desirable and it is better to immediately give an error.
+  config :block_on_buffer_full, :validate => :boolean, :default => true
+  # the timeout setting for initial metadata request to fetch topic metadata.
+  config :metadata_fetch_timeout_ms, :validate => :number, :default => 60000
+  # the max time in milliseconds before a metadata refresh is forced.
+  config :metadata_max_age_ms, :validate => :number, :default => 300000
+  # The amount of time to wait before attempting to reconnect to a given host when a connection fails.
+  config :reconnect_backoff_ms, :validate => :number, :default => 10
+  # The amount of time to wait before attempting to retry a failed produce request to a given topic partition.
   config :retry_backoff_ms, :validate => :number, :default => 100
-  # The producer generally refreshes the topic metadata from brokers when there is a failure
-  # (partition missing, leader not available...). It will also poll regularly (default: every
-  # 10min so 600000ms). If you set this to a negative value, metadata will only get refreshed on
-  # failure. If you set this to zero, the metadata will get refreshed after each message sent
-  # (not recommended). Important note: the refresh happen only AFTER the message is sent,
-  # so if the producer never sends a message the metadata is never refreshed
-  config :topic_metadata_refresh_interval_ms, :validate => :number, :default => 600 * 1000
-  # Maximum time to buffer data when using async mode. For example a setting of 100 will try to
-  # batch together 100ms of messages to send at once. This will improve throughput but adds
-  # message delivery latency due to the buffering.
-  config :queue_buffering_max_ms, :validate => :number, :default => 5000
-  # The maximum number of unsent messages that can be queued up the producer when using async
-  # mode before either the producer must be blocked or data must be dropped.
-  config :queue_buffering_max_messages, :validate => :number, :default => 10000
-  # The amount of time to block before dropping messages when running in async mode and the
-  # buffer has reached `queue.buffering.max.messages`. If set to 0 events will be enqueued
-  # immediately or dropped if the queue is full (the producer send call will never block). If set
-  # to -1 the producer will block indefinitely and never willingly drop a send.
-  config :queue_enqueue_timeout_ms, :validate => :number, :default => -1
-  # The number of messages to send in one batch when using async mode. The producer will wait
-  # until either this number of messages are ready to send or `queue.buffer.max.ms` is reached.
-  config :batch_num_messages, :validate => :number, :default => 200
-  # Socket write buffer size
-  config :send_buffer_bytes, :validate => :number, :default => 100 * 1024
-  # The client id is a user-specified string sent in each request to help trace calls. It should
-  # logically identify the application making the request.
-  config :client_id, :validate => :string, :default => ''
-  # Provides a way to specify a partition key as a string. To specify a partition key for
-  # Kafka, configure a format that will produce the key as a string. Defaults
-  # `key_serializer_class` to `kafka.serializer.StringEncoder` to match. For example, to partition
-  # by host:
-  # [source,ruby]
-  #     output {
-  #       kafka {
-  #           partition_key_format => "%{host}"
-  #       }
-  #     }
-  config :partition_key_format, :validate => :string, :default => nil
 
   public
   def register
     LogStash::Logger.setup_log4j(@logger)
 
     options = {
-        :broker_list => @broker_list,
-        :compression_codec => @compression_codec,
-        :compressed_topics => @compressed_topics,
-        :request_required_acks => @request_required_acks,
-        :serializer_class => @serializer_class,
-        :partitioner_class => @partitioner_class,
-        :request_timeout_ms => @request_timeout_ms,
-        :producer_type => @producer_type,
-        :key_serializer_class => @key_serializer_class,
-        :message_send_max_retries => @message_send_max_retries,
-        :retry_backoff_ms => @retry_backoff_ms,
-        :topic_metadata_refresh_interval_ms => @topic_metadata_refresh_interval_ms,
-        :queue_buffering_max_ms => @queue_buffering_max_ms,
-        :queue_buffering_max_messages => @queue_buffering_max_messages,
-        :queue_enqueue_timeout_ms => @queue_enqueue_timeout_ms,
-        :batch_num_messages => @batch_num_messages,
-        :send_buffer_bytes => @send_buffer_bytes,
-        :client_id => @client_id
+      :key_serializer => @key_serializer,
+      :value_serializer => @value_serializer,
+      :bootstrap_servers => @bootstrap_servers,
+      :acks => @acks,
+      :buffer_memory => @buffer_memory,
+      :compression_type => @compression_type,
+      :retries => @retries,
+      :batch_size => @batch_size,
+      :client_id => @client_id,
+      :linger_ms => @linger_ms,
+      :max_request_size => @max_request_size,
+      :receive_buffer_bytes => @receive_buffer_bytes,
+      :send_buffer_bytes => @send_buffer_bytes,
+      :timeout_ms => @timeout_ms,
+      :block_on_buffer_full => @block_on_buffer_full,
+      :metadata_fetch_timeout_ms => @metadata_fetch_timeout_ms,
+      :metadata_max_age_ms => @metadata_max_age_ms,
+      :reconnect_backoff_ms => @reconnect_backoff_ms,
+      :retry_backoff_ms => @retry_backoff_ms
     }
-    @producer = Kafka::Producer.new(options)
+    @producer = Kafka::KafkaProducer.new(options)
     @producer.connect
 
-    @logger.info('Registering kafka producer', :topic_id => @topic_id, :broker_list => @broker_list)
+    @logger.info('Registering kafka producer', :topic_id => @topic_id, :bootstrap_servers => @bootstrap_servers)
 
     @codec.on_event do |event, data|
       begin
-        @producer.send_msg(event.sprintf(@topic_id),@partition_key,data)
+        key = if @message_key.nil? then nil else event.sprintf(@message_key) end
+        @producer.send_msg(event.sprintf(@topic_id), nil, key, data)
       rescue LogStash::ShutdownSignal
         @logger.info('Kafka producer got shutdown signal')
       rescue => e
@@ -161,9 +146,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
       finished
       return
     end
-    @partition_key = if @partition_key_format.nil? then nil else event.sprintf(@partition_key_format) end
     @codec.encode(event)
-    @partition_key = nil
   end
 
   def teardown

data/logstash-output-kafka.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-output-kafka'
-  s.version         = '1.0.0'
+  s.version         = '2.0.0.beta.1'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = 'Output events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on the broker'
   s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
@@ -24,6 +24,9 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'logstash-codec-plain'
   s.add_runtime_dependency 'logstash-codec-json'
 
-  s.add_runtime_dependency 'jruby-kafka', ['>= 1.1.0', '< 2.0.0']
+  s.add_runtime_dependency 'jruby-kafka', '>= 1.4.0', '< 2.0.0'
+
   s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'poseidon'
+  s.add_development_dependency 'longshoreman'
 end

data/spec/integration/outputs/kafka_spec.rb

@@ -0,0 +1,103 @@
+# encoding: utf-8
+
+require "logstash/devutils/rspec/spec_helper"
+require 'logstash/outputs/kafka'
+require 'longshoreman'
+require 'jruby-kafka'
+require 'json'
+require 'poseidon'
+
+CONTAINER_NAME = "kafka-#{rand(999).to_s}"
+IMAGE_NAME = 'spotify/kafka'
+IMAGE_TAG = 'latest'
+KAFKA_PORT = 9092
+KAFKA_VERSION = "0.8.2.1"
+KAFKA_SCALA_VERSION = "2.11"
+
+RSpec.configure do |config|
+  config.before(:all, :integration => true) do
+    @ls = begin
+            ls = Longshoreman.new
+            ls.container.get(CONTAINER_NAME)
+            ls
+          rescue Docker::Error::NotFoundError
+            Longshoreman.pull_image(IMAGE_NAME, IMAGE_TAG)
+            Longshoreman.new("#{IMAGE_NAME}:#{IMAGE_TAG}", CONTAINER_NAME,
+                             { "ENV" => [ "ADVERTISED_HOST=#{Longshoreman.new.get_host_ip}", "ADVERTISED_PORT=#{KAFKA_PORT}"],
+                               "PortBindings" => { "#{KAFKA_PORT}/tcp" => [{ "HostPort" => "#{KAFKA_PORT}" }]}})
+          end
+    @kafka_host = @ls.get_host_ip
+    @kafka_port = @ls.container.rport(9092)
+    @zk_port = @ls.container.rport(2181)
+  end
+
+  config.after(:suite) do
+    begin
+      ls = Longshoreman::new
+      ls.container.get(CONTAINER_NAME)
+      ls.cleanup
+    rescue Docker::Error::NotFoundError, Excon::Errors::SocketError
+    end
+  end
+end
+
+describe "outputs/kafka", :integration => true do
+  let(:test_topic) { 'test' }
+  let(:base_config) { {'client_id' => 'spectest', 'bootstrap_servers' => "#{@kafka_host}:#{@kafka_port}"} }
+  let(:event) { LogStash::Event.new({'message' => 'hello', '@timestamp' => LogStash::Timestamp.at(0) }) }
+
+
+  context 'when outputting messages' do
+    let(:num_events) { 3 }
+    let(:consumer) do
+      Poseidon::PartitionConsumer.new("my_test_consumer", @kafka_host, @kafka_port,
+                                      test_topic, 0, :earliest_offset)
+    end
+    subject do
+      consumer.fetch
+    end
+
+    before :each do
+      config = base_config.merge({"topic_id" => test_topic})
+      kafka = LogStash::Outputs::Kafka.new(config)
+      kafka.register
+      num_events.times do kafka.receive(event) end
+      kafka.teardown
+    end
+
+    it 'should have data integrity' do
+      expect(subject.size).to eq(num_events)
+      subject.each do |m|
+        expect(m.value).to eq(event.to_json)
+      end
+    end
+  end
+
+  context 'when setting message_key' do
+    let(:num_events) { 10 }
+    let(:test_topic) { 'test2' }
+    let!(:consumer0) do
+      Poseidon::PartitionConsumer.new("my_test_consumer", @kafka_host, @kafka_port,
+                                      test_topic, 0, :earliest_offset)
+    end
+    let!(:consumer1) do
+      Poseidon::PartitionConsumer.new("my_test_consumer", @kafka_host, @kafka_port,
+                                      test_topic, 1, :earliest_offset)
+    end
+
+    before :each do
+      command = ["/opt/kafka_#{KAFKA_SCALA_VERSION}-#{KAFKA_VERSION}/bin/kafka-topics.sh", "--create", "--topic", "#{test_topic}", "--partitions", "2", "--zookeeper", "#{@kafka_host}:#{@zk_port}", "--replication-factor", "1"]
+      @ls.container.raw.exec(command)
+
+      config = base_config.merge({"topic_id" => test_topic, "message_key" => "static_key"})
+      kafka = LogStash::Outputs::Kafka.new(config)
+      kafka.register
+      num_events.times do kafka.receive(event) end
+      kafka.teardown
+    end
+
+    it 'should send all events to one partition' do
+      expect(consumer0.fetch.size == num_events || consumer1.fetch.size == num_events).to be true
+    end
+  end
+end

data/spec/{outputs → unit/outputs}/kafka_spec.rb

@@ -17,19 +17,16 @@ describe "outputs/kafka" do
 
     it 'should populate kafka config with default values' do
       kafka = LogStash::Outputs::Kafka.new(simple_kafka_config)
-      insist {kafka.broker_list} == 'localhost:9092'
+      insist {kafka.bootstrap_servers} == 'localhost:9092'
       insist {kafka.topic_id} == 'test'
-      insist {kafka.compression_codec} == 'none'
-      insist {kafka.serializer_class} == 'kafka.serializer.StringEncoder'
-      insist {kafka.partitioner_class} == 'kafka.producer.DefaultPartitioner'
-      insist {kafka.producer_type} == 'sync'
+      insist {kafka.key_serializer} == 'org.apache.kafka.common.serialization.StringSerializer'
     end
   end
 
   context 'when outputting messages' do
     it 'should send logstash event to kafka broker' do
-      expect_any_instance_of(Kafka::Producer).to receive(:send_msg)
-        .with(simple_kafka_config['topic_id'], nil, event.to_json)
+      expect_any_instance_of(Kafka::KafkaProducer).to receive(:send_msg)
+        .with(simple_kafka_config['topic_id'], nil, nil, event.to_hash.to_json)
       kafka = LogStash::Outputs::Kafka.new(simple_kafka_config)
       kafka.register
       kafka.receive(event)
@@ -37,19 +34,17 @@ describe "outputs/kafka" do
 
     it 'should support Event#sprintf placeholders in topic_id' do
       topic_field = 'topic_name'
-      expect_any_instance_of(Kafka::Producer).to receive(:send_msg)
-        .with(event[topic_field], nil, event.to_json)
+      expect_any_instance_of(Kafka::KafkaProducer).to receive(:send_msg)
+        .with(event[topic_field], nil, nil, event.to_hash.to_json)
       kafka = LogStash::Outputs::Kafka.new({'topic_id' => "%{#{topic_field}}"})
       kafka.register
       kafka.receive(event)
     end
 
-    it 'should support Event#sprintf placeholders in partition_key_format' do
-      partition_field = 'host'
-      expect_any_instance_of(Kafka::Producer).to receive(:send_msg)
-        .with(simple_kafka_config['topic_id'], event[partition_field], event.to_json)
-      kafka = LogStash::Outputs::Kafka.new({'topic_id' => simple_kafka_config['topic_id'],
-                                            'partition_key_format' => "%{#{partition_field}}"})
+    it 'should support field referenced message_keys' do
+      expect_any_instance_of(Kafka::KafkaProducer).to receive(:send_msg)
+        .with(simple_kafka_config['topic_id'], nil, event['host'], event.to_hash.to_json)
+      kafka = LogStash::Outputs::Kafka.new(simple_kafka_config.merge({"message_key" => "%{host}"}))
       kafka.register
       kafka.receive(event)
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-kafka
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 2.0.0.beta.1
 platform: ruby
 authors:
 - Elasticsearch
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2015-06-24 00:00:00.000000000 Z
+date: 2015-09-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core
@@ -64,7 +64,7 @@ dependencies:
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 1.1.0
+        version: 1.4.0
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
@@ -72,7 +72,7 @@ dependencies:
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 1.1.0
+        version: 1.4.0
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
@@ -92,6 +92,34 @@ dependencies:
         version: '0'
   prerelease: false
   type: :development
+- !ruby/object:Gem::Dependency
+  name: poseidon
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
+- !ruby/object:Gem::Dependency
+  name: longshoreman
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
 description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
 email: info@elastic.co
 executables: []
@@ -109,7 +137,8 @@ files:
 - Rakefile
 - lib/logstash/outputs/kafka.rb
 - logstash-output-kafka.gemspec
-- spec/outputs/kafka_spec.rb
+- spec/integration/outputs/kafka_spec.rb
+- spec/unit/outputs/kafka_spec.rb
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)
@@ -127,9 +156,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - '>'
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project:
 rubygems_version: 2.2.2
@@ -137,4 +166,5 @@ signing_key:
 specification_version: 4
 summary: Output events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on the broker
 test_files:
-- spec/outputs/kafka_spec.rb
+- spec/integration/outputs/kafka_spec.rb
+- spec/unit/outputs/kafka_spec.rb