logstash-input-kafka 4.2.0 → 5.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 392af7e1646f486c6f81b5ed7dd2206335be8287
-  data.tar.gz: 75f322acc9c088a79a6e87c51349237d6bff91ed
+  metadata.gz: f496e94c2acee0bbb82357e17efe952416381c4c
+  data.tar.gz: 19279054c88de9ecc57dedab85c1e584afd9f160
 SHA512:
-  metadata.gz: 2541883711e5a2fdb1acf216fdf1f25a2af1ed266c84293a869805f3e7e418c2d6a162e5f57cece82dbadf63256d8edf8271be4b6e489e93381f7f8ca8ad3a93
-  data.tar.gz: 6b2fc56fbafe56be7c8a891f44b1c83dea8759e49856b8fc94da6e1378edcf69a7bd7e1b1fddbfaf2e8a333674d97a165fe9b151efd08b1168081cc35aef1be0
+  metadata.gz: 6b3ae71f305fe402aa9dbafdb815ace949445e90168b310f2b35f90cf68d83d5b4ec69389e61d3950e897b3b8c62f24126cc2995f98b6ada301b50641d495a00
+  data.tar.gz: 3eb69a86fe5377bc4d348b601db819b76d1ae296ec44fe761a98db2629eae194a34590612f939f88e270d6cf5fc66b227d5ba97cc2c2ad5e129eee6944f863ff

data/CHANGELOG.md

@@ -1,12 +1,8 @@
-## 4.2.0
-  - Add topic pattern config which allows consuming from multiple topics
-  - Restored event decorators which adds metadata.
+## 5.0.1
+  - Relax constraint on logstash-core-plugin-api to >= 1.60 <= 2.99
 
-## 4.1.1
-  - fix: vendor aliasing issue when publishing
-
-## 4.1.0
-  - fix: Really support Kafka 0.9 for LS 5.x, logging changes broke 4.0.0
+## 5.0.0
+  - Support for Kafka 0.10 which is not backward compatible with 0.9 broker.
 
 ## 4.0.0
   - Republish all the gems under jruby.

data/README.md

@@ -6,6 +6,18 @@ This is a plugin for [Logstash](https://github.com/elastic/logstash).
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
+## Kafka Compatibility
+
+Here's a table that describes the compatibility matrix for Kafka Broker support. Please remember that it is good advice to upgrade brokers before consumers/producers since brokers target backwards compatibility. The 0.9 broker will work with both the 0.8 consumer and 0.9 consumer APIs but not the other way around.
+
+| Kafka Broker Version | Logstash Version | Input Plugin | Output Plugin | Why? |
+|:---------------:|:------------------:|:--------------:|:---------------:|:------|
+| 0.8           | 2.0 - 2.x   | < 3.0.0 | <3.0.0 | Legacy, 0.8 is still popular |
+| 0.9           | 2.0 - 2.3.x   |   3.0.0 | 3.0.0  | Intermediate release before 0.10 that works with old Ruby Event API `[]`  |
+| 0.9          | 2.4, 5.0           |   4.0.0 | 4.0.0  | Intermediate release before 0.10 with new get/set API |
+| 0.10         | 2.4, 5.0           |   5.0.0 | 5.0.0  | Track latest Kafka release. Not compatible with 0.9 broker |
+
+
 ## Documentation
 
 Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
@@ -95,4 +107,4 @@ Programming is not a required skill. Whatever you've seen about open source and
 
 It is more important to the community that you are able to contribute.
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/kafka.rb

@@ -5,10 +5,22 @@ require 'java'
 require 'logstash-input-kafka_jars.rb'
 
 # This input will read events from a Kafka topic. It uses the the newly designed
-# 0.9 version of consumer API[https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design] 
-# provided by Kafka to read messages from the broker.
+# 0.10 version of consumer API provided by Kafka to read messages from the broker.
 #
-# NOTE: This consumer is not backward compatible with 0.8.x brokers and needs a 0.9 broker.
+# Here's a compatibility matrix that shows the Kafka broker and client versions that are compatible with each combination
+# of Logstash and the Kafka input plugin: 
+# 
+# [options="header"]
+# |==========================================================
+# |Kafka Broker Version |Kafka Client Version |Logstash Version |Plugin Version |Why?
+# |0.8       |0.8       |2.0.0 - 2.x.x   |<3.0.0 |Legacy, 0.8 is still popular 
+# |0.9       |0.9       |2.0.0 - 2.3.x   | 3.x.x |Works with the old Ruby Event API (`event['product']['price'] = 10`)  
+# |0.9       |0.9       |2.4.0 - 5.0.x   | 4.x.x |Works with the new getter/setter APIs (`event.set('[product][price]', 10)`)
+# |0.10      |0.10      |2.4.0 - 5.0.x   | 5.x.x |Not compatible with the 0.9 broker 
+# |==========================================================
+# 
+# NOTE: It's a good idea to upgrade brokers before consumers/producers because brokers target backwards compatibility.
+# For example, the 0.9 broker will work with both the 0.8 consumer and 0.9 consumer APIs, but not the other way around.
 #
 # The Logstash Kafka consumer handles group management and uses the default offset management
 # strategy using Kafka topics.
@@ -16,9 +28,9 @@ require 'logstash-input-kafka_jars.rb'
 # Logstash instances by default form a single logical group to subscribe to Kafka topics
 # Each Logstash Kafka consumer can run multiple threads to increase read throughput. Alternatively, 
 # you could run multiple Logstash instances with the same `group_id` to spread the load across
-# physical machines. Messages in a topic will be distributed to all Logstash instances with 
+# physical machines. Messages in a topic will be distributed to all Logstash instances with
 # the same `group_id`.
-# 
+#
 # Ideally you should have as many threads as the number of partitions for a perfect balance --
 # more threads than partitions means that some threads will be idle
 #
@@ -26,7 +38,7 @@ require 'logstash-input-kafka_jars.rb'
 #
 # Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
 #
-# This version also adds support for SSL/TLS security connection to Kafka. By default SSL is 
+# This version also adds support for SSL/TLS security connection to Kafka. By default SSL is
 # disabled but can be turned on as needed.
 #
 class LogStash::Inputs::Kafka < LogStash::Inputs::Base
@@ -35,7 +47,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   default :codec, 'plain'
 
   # The frequency in milliseconds that the consumer offsets are committed to Kafka.
-  config :auto_commit_interval_ms, :validate => :string, :default => "10"
+  config :auto_commit_interval_ms, :validate => :string, :default => "5000"
   # What to do when there is no initial offset in Kafka or if an offset is out of range:
   #
   # * earliest: automatically reset the offset to the earliest offset
@@ -43,86 +55,92 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # * none: throw exception to the consumer if no previous offset is found for the consumer's group
   # * anything else: throw exception to the consumer.
   config :auto_offset_reset, :validate => :string
-  # A list of URLs to use for establishing the initial connection to the cluster. 
-  # This list should be in the form of `host1:port1,host2:port2` These urls are just used 
-  # for the initial connection to discover the full cluster membership (which may change dynamically) 
-  # so this list need not contain the full set of servers (you may want more than one, though, in 
+  # A list of URLs to use for establishing the initial connection to the cluster.
+  # This list should be in the form of `host1:port1,host2:port2` These urls are just used
+  # for the initial connection to discover the full cluster membership (which may change dynamically)
+  # so this list need not contain the full set of servers (you may want more than one, though, in
   # case a server is down).
   config :bootstrap_servers, :validate => :string, :default => "localhost:9092"
-  # Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk 
-  # corruption to the messages occurred. This check adds some overhead, so it may be 
+  # Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk
+  # corruption to the messages occurred. This check adds some overhead, so it may be
   # disabled in cases seeking extreme performance.
   config :check_crcs, :validate => :string
-  # The id string to pass to the server when making requests. The purpose of this 
-  # is to be able to track the source of requests beyond just ip/port by allowing 
+  # The id string to pass to the server when making requests. The purpose of this
+  # is to be able to track the source of requests beyond just ip/port by allowing
   # a logical application name to be included.
   config :client_id, :validate => :string, :default => "logstash"
   # Close idle connections after the number of milliseconds specified by this config.
   config :connections_max_idle_ms, :validate => :string
-  # Ideally you should have as many threads as the number of partitions for a perfect 
+  # Ideally you should have as many threads as the number of partitions for a perfect
   # balance — more threads than partitions means that some threads will be idle
   config :consumer_threads, :validate => :number, :default => 1
   # If true, periodically commit to Kafka the offsets of messages already returned by the consumer. 
-  # This committed offset will be used when the process fails as the position from 
+  # This committed offset will be used when the process fails as the position from
   # which the consumption will begin.
   config :enable_auto_commit, :validate => :string, :default => "true"
-  # The maximum amount of time the server will block before answering the fetch request if 
-  # there isn't sufficient data to immediately satisfy `fetch_min_bytes`. This 
+  # Whether records from internal topics (such as offsets) should be exposed to the consumer.
+  # If set to true the only way to receive records from an internal topic is subscribing to it.
+  config :exclude_internal_topics, :validate => :string
+  # The maximum amount of time the server will block before answering the fetch request if
+  # there isn't sufficient data to immediately satisfy `fetch_min_bytes`. This
   # should be less than or equal to the timeout used in `poll_timeout_ms`
   config :fetch_max_wait_ms, :validate => :string
-  # The minimum amount of data the server should return for a fetch request. If insufficient 
+  # The minimum amount of data the server should return for a fetch request. If insufficient
   # data is available the request will wait for that much data to accumulate
   # before answering the request.
   config :fetch_min_bytes, :validate => :string
-  # The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber 
-  # that happens to be made up of multiple processors. Messages in a topic will be distributed to all 
+  # The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
+  # that happens to be made up of multiple processors. Messages in a topic will be distributed to all
   # Logstash instances with the same `group_id`
   config :group_id, :validate => :string, :default => "logstash"
   # The expected time between heartbeats to the consumer coordinator. Heartbeats are used to ensure 
   # that the consumer's session stays active and to facilitate rebalancing when new
-  # consumers join or leave the group. The value must be set lower than 
-  # `session.timeout.ms`, but typically should be set no higher than 1/3 of that value. 
+  # consumers join or leave the group. The value must be set lower than
+  # `session.timeout.ms`, but typically should be set no higher than 1/3 of that value.
   # It can be adjusted even lower to control the expected time for normal rebalances.
   config :heartbeat_interval_ms, :validate => :string
   # Java Class used to deserialize the record's key
   config :key_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
-  # The maximum amount of data per-partition the server will return. The maximum total memory used for a 
-  # request will be <code>#partitions * max.partition.fetch.bytes</code>. This size must be at least 
-  # as large as the maximum message size the server allows or else it is possible for the producer to 
-  # send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying 
+  # The maximum amount of data per-partition the server will return. The maximum total memory used for a
+  # request will be <code>#partitions * max.partition.fetch.bytes</code>. This size must be at least
+  # as large as the maximum message size the server allows or else it is possible for the producer to
+  # send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying
   # to fetch a large message on a certain partition.
   config :max_partition_fetch_bytes, :validate => :string
-  # The class name of the partition assignment strategy that the client will use to distribute 
+  # The maximum number of records returned in a single call to poll().
+  config :max_poll_records, :validate => :string
+  # The period of time in milliseconds after which we force a refresh of metadata even if
+  # we haven't seen any partition leadership changes to proactively discover any new brokers or partitions
+  config :metadata_max_age_ms, :validate => :string
+  # The class name of the partition assignment strategy that the client will use to distribute
   # partition ownership amongst consumer instances
   config :partition_assignment_strategy, :validate => :string
   # The size of the TCP receive buffer (SO_RCVBUF) to use when reading data.
   config :receive_buffer_bytes, :validate => :string
-  # The amount of time to wait before attempting to reconnect to a given host. 
-  # This avoids repeatedly connecting to a host in a tight loop. 
+  # The amount of time to wait before attempting to reconnect to a given host.
+  # This avoids repeatedly connecting to a host in a tight loop.
   # This backoff applies to all requests sent by the consumer to the broker.
   config :reconnect_backoff_ms, :validate => :string
-  # The configuration controls the maximum amount of time the client will wait 
-  # for the response of a request. If the response is not received before the timeout 
-  # elapses the client will resend the request if necessary or fail the request if 
+  # The configuration controls the maximum amount of time the client will wait
+  # for the response of a request. If the response is not received before the timeout
+  # elapses the client will resend the request if necessary or fail the request if
   # retries are exhausted.
   config :request_timeout_ms, :validate => :string
-  # The amount of time to wait before attempting to retry a failed fetch request 
+  # The amount of time to wait before attempting to retry a failed fetch request
   # to a given topic partition. This avoids repeated fetching-and-failing in a tight loop.
   config :retry_backoff_ms, :validate => :string
-  # The timeout after which, if the `poll_timeout_ms` is not invoked, the consumer is marked dead 
+  # The size of the TCP send buffer (SO_SNDBUF) to use when sending data
+  config :send_buffer_bytes, :validate => :string
+  # The timeout after which, if the `poll_timeout_ms` is not invoked, the consumer is marked dead
   # and a rebalance operation is triggered for the group identified by `group_id`
-  config :session_timeout_ms, :validate => :string, :default => "30000"
+  config :session_timeout_ms, :validate => :string
   # Java Class used to deserialize the record's value
   config :value_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
-  # A list of topics to subscribe to, defaults to ["logstash"].
-  config :topics, :validate => :array, :default => ["logstash"]
-  # A topic regex pattern to subscribe to. 
-  # The topics configuration will be ignored when using this configuration.
-  config :topics_pattern, :validate => :string
+  # A list of topics to subscribe to.
+  config :topics, :validate => :array, :required => true
   # Time kafka consumer will wait to receive new messages from topics
-  config :poll_timeout_ms, :validate => :number, :default => 100
-  # Enable SSL/TLS secured communication to Kafka broker. Note that secure communication 
-  # is only available with a broker running v0.9 of Kafka.
+  config :poll_timeout_ms, :validate => :number
+  # Enable SSL/TLS secured communication to Kafka broker.
   config :ssl, :validate => :boolean, :default => false
   # The JKS truststore path to validate the Kafka broker's certificate.
   config :ssl_truststore_location, :validate => :path
@@ -132,22 +150,11 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   config :ssl_keystore_location, :validate => :path
   # If client authentication is required, this setting stores the keystore password
   config :ssl_keystore_password, :validate => :password
-  # Option to add Kafka metadata like topic, message size to the event.
-  # This will add a field named `kafka` to the logstash event containing the following attributes:
-  #   `topic`: The topic this message is associated with
-  #   `consumer_group`: The consumer group used to read in this event
-  #   `partition`: The partition this message is associated with
-  #   `offset`: The offset from the partition this message is associated with
-  #   `key`: A ByteBuffer containing the message key
-  config :decorate_events, :validate => :boolean, :default => false
 
-  
+
   public
   def register
-    # Logstash 2.4
-    if defined?(LogStash::Logger) && LogStash::Logger.respond_to?(:setup_log4j)
-      LogStash::Logger.setup_log4j(@logger)
-    end
+    LogStash::Logger.setup_log4j(@logger)
     @runner_threads = []
   end # def register
 
@@ -167,25 +174,12 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   def thread_runner(logstash_queue, consumer)
     Thread.new do
       begin
-        unless @topics_pattern.nil?
-          nooplistener = org.apache.kafka.clients.consumer.internals.NoOpConsumerRebalanceListener.new
-          pattern = java.util.regex.Pattern.compile(@topics_pattern)
-          consumer.subscribe(pattern, nooplistener)
-        else
-          consumer.subscribe(topics);
-        end
+        consumer.subscribe(topics);
         while !stop?
           records = consumer.poll(poll_timeout_ms);
           for record in records do
             @codec.decode(record.value.to_s) do |event|
               decorate(event)
-              if @decorate_events
-                event.set("[kafka][topic]", record.topic)
-                event.set("[kafka][consumer_group]", @group_id)
-                event.set("[kafka][partition]", record.partition)
-                event.set("[kafka][offset]", record.offset)
-                event.set("[kafka][key]", record.key)
-              end
               logstash_queue << event
             end
           end
@@ -211,17 +205,21 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put(kafka::CLIENT_ID_CONFIG, client_id)
       props.put(kafka::CONNECTIONS_MAX_IDLE_MS_CONFIG, connections_max_idle_ms) unless connections_max_idle_ms.nil?
       props.put(kafka::ENABLE_AUTO_COMMIT_CONFIG, enable_auto_commit)
+      props.put(kafka::EXCLUDE_INTERNAL_TOPICS_CONFIG, exclude_internal_topics) unless exclude_internal_topics.nil?
       props.put(kafka::FETCH_MAX_WAIT_MS_CONFIG, fetch_max_wait_ms) unless fetch_max_wait_ms.nil?
       props.put(kafka::FETCH_MIN_BYTES_CONFIG, fetch_min_bytes) unless fetch_min_bytes.nil?
       props.put(kafka::GROUP_ID_CONFIG, group_id)
       props.put(kafka::HEARTBEAT_INTERVAL_MS_CONFIG, heartbeat_interval_ms) unless heartbeat_interval_ms.nil?
       props.put(kafka::KEY_DESERIALIZER_CLASS_CONFIG, key_deserializer_class)
       props.put(kafka::MAX_PARTITION_FETCH_BYTES_CONFIG, max_partition_fetch_bytes) unless max_partition_fetch_bytes.nil?
+      props.put(kafka::MAX_POLL_RECORDS_CONFIG, max_poll_records) unless max_poll_records.nil?
+      props.put(kafka::METADATA_MAX_AGE_MS_CONFIG, metadata_max_age_ms) unless metadata_max_age_ms.nil?
       props.put(kafka::PARTITION_ASSIGNMENT_STRATEGY_CONFIG, partition_assignment_strategy) unless partition_assignment_strategy.nil?
       props.put(kafka::RECEIVE_BUFFER_CONFIG, receive_buffer_bytes) unless receive_buffer_bytes.nil?
       props.put(kafka::RECONNECT_BACKOFF_MS_CONFIG, reconnect_backoff_ms) unless reconnect_backoff_ms.nil?
       props.put(kafka::REQUEST_TIMEOUT_MS_CONFIG, request_timeout_ms) unless request_timeout_ms.nil?
       props.put(kafka::RETRY_BACKOFF_MS_CONFIG, retry_backoff_ms) unless retry_backoff_ms.nil?
+      props.put(kafka::SEND_BUFFER_CONFIG, send_buffer_bytes) unless send_buffer_bytes.nil?
       props.put(kafka::SESSION_TIMEOUT_MS_CONFIG, session_timeout_ms) unless session_timeout_ms.nil?
       props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, value_deserializer_class)
 

data/logstash-input-kafka.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-input-kafka'
-  s.version         = '4.2.0'
+  s.version         = '5.0.1'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = 'This input will read events from a Kafka topic. It uses the high level consumer API provided by Kafka to read messages from the broker'
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -19,8 +19,8 @@ Gem::Specification.new do |s|
   # Special flag to let us know this is actually a logstash plugin
   s.metadata = { 'logstash_plugin' => 'true', 'group' => 'input'}
 
-  s.requirements << "jar 'org.apache.kafka:kafka-clients', '0.9.0.1'"
-  s.requirements << "jar 'org.slf4j:slf4j-log4j12', '1.7.13'"
+  s.requirements << "jar 'org.apache.kafka:kafka-clients', '0.10.0.0'"
+  s.requirements << "jar 'org.slf4j:slf4j-log4j12', '1.7.21'"
 
   s.add_development_dependency 'jar-dependencies', '~> 0.3.2'
 
@@ -31,5 +31,6 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'stud', '>= 0.0.22', '< 0.1.0'
 
   s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'rspec-wait'
 end
 

data/spec/integration/inputs/kafka_spec.rb

@@ -4,97 +4,47 @@ require "logstash/inputs/kafka"
 require "digest"
 require "rspec/wait"
 
-# Please run kafka_test_setup.sh prior to executing this integration test.
-describe "inputs/kafka", :integration => true do
-  # Group ids to make sure that the consumers get all the logs.
-  let(:group_id_1) {rand(36**8).to_s(36)}
-  let(:group_id_2) {rand(36**8).to_s(36)}
-  let(:group_id_3) {rand(36**8).to_s(36)}
-  let(:plain_config) { { 'topics' => ['logstash_topic_plain'], 'codec' => 'plain', 'group_id' => group_id_1, 'auto_offset_reset' => 'earliest'} }
-  let(:snappy_config) { { 'topics' => ['logstash_topic_snappy'], 'codec' => 'plain', 'group_id' => group_id_1, 'auto_offset_reset' => 'earliest'} }
-  let(:lz4_config) { { 'topics' => ['logstash_topic_lz4'], 'codec' => 'plain', 'group_id' => group_id_1, 'auto_offset_reset' => 'earliest'} }
-  let(:pattern_config) { { 'topics_pattern' => 'logstash_topic_.*', 'group_id' => group_id_2, 'codec' => 'plain', 'auto_offset_reset' => 'earliest'} }  
-  let(:decorate_config) { { 'topics' => ['logstash_topic_plain'], 'codec' => 'plain', 'group_id' => group_id_3, 'auto_offset_reset' => 'earliest', 'decorate_events' => true} }
-  let(:timeout_seconds) { 120 }
+describe "input/kafka", :integration => true do
+  let(:partition3_config) { { 'topics' => ['topic3'], 'codec' => 'plain', 'auto_offset_reset' => 'earliest'} }
+  let(:snappy_config) { { 'topics' => ['snappy_topic'], 'codec' => 'plain', 'auto_offset_reset' => 'earliest'} }
+  let(:lz4_config) { { 'topics' => ['lz4_topic'], 'codec' => 'plain', 'auto_offset_reset' => 'earliest'} }
+  
+  let(:timeout_seconds) { 3600 }
   let(:num_events) { 103 }
-
-  describe "#kafka-topics" do
-    def thread_it(kafka_input, queue)
-      Thread.new do
-        begin
-          kafka_input.run(queue)
-        end
+  
+  def thread_it(kafka_input, queue)
+    Thread.new do
+      begin
+        kafka_input.run(queue)
       end
     end
-
-    it "should consume all messages from plain 3-partition topic" do
-      kafka_input = LogStash::Inputs::Kafka.new(plain_config)
-      queue = Array.new
-      t = thread_it(kafka_input, queue)
-      t.run
-      wait(timeout_seconds).for { queue.length }.to eq(num_events)
-      expect(queue.length).to eq(num_events)
-    end
-
-    it "should consume all messages from snappy 3-partition topic" do
-      kafka_input = LogStash::Inputs::Kafka.new(snappy_config)
-      queue = Array.new
-      t = thread_it(kafka_input, queue)
-      t.run
-      wait(timeout_seconds).for { queue.length }.to eq(num_events)
-      expect(queue.length).to eq(num_events)
-    end
-
-    it "should consume all messages from lz4 3-partition topic" do
-      kafka_input = LogStash::Inputs::Kafka.new(lz4_config)
-      queue = Array.new
-      t = thread_it(kafka_input, queue)
-      t.run
-      wait(timeout_seconds).for { queue.length }.to eq(num_events)
-      expect(queue.length).to eq(num_events)
-    end
-    
   end
-
-  describe "#kafka-topics-pattern" do
     
-    def thread_it(kafka_input, queue)
-      Thread.new do
-        begin
-          kafka_input.run(queue)
-        end
-      end
-    end
-      
-    it "should consume all messages from all 3 topics" do
-      kafka_input = LogStash::Inputs::Kafka.new(pattern_config)
-      queue = Array.new
-      t = thread_it(kafka_input, queue)
-      t.run
-      wait(timeout_seconds).for { queue.length }.to eq(3*num_events)
-      expect(queue.length).to eq(3*num_events)
-    end    
+  it "should consume all messages from 3-partition topic" do
+    kafka_input = LogStash::Inputs::Kafka.new(partition3_config)
+    queue = Array.new
+    t = thread_it(kafka_input, queue)
+    t.run
+    wait(timeout_seconds).for { queue.length }.to eq(num_events)
+    expect(queue.length).to eq(num_events)
+  end
+  
+  it "should consume all messages from snappy 3-partition topic" do
+    kafka_input = LogStash::Inputs::Kafka.new(snappy_config)
+    queue = Array.new
+    t = thread_it(kafka_input, queue)
+    t.run
+    wait(timeout_seconds).for { queue.length }.to eq(num_events)
+    expect(queue.length).to eq(num_events)
   end
 
-  describe "#kafka-decorate" do
-    def thread_it(kafka_input, queue)
-      Thread.new do
-        begin
-          kafka_input.run(queue)
-        end
-      end
-    end
-      
-    it "should show the right topic and group name in decorated kafka section" do
-      kafka_input = LogStash::Inputs::Kafka.new(decorate_config)
-      queue = Queue.new
-      t = thread_it(kafka_input, queue)
-      t.run
-      wait(timeout_seconds).for { queue.length }.to eq(num_events)
-      expect(queue.length).to eq(num_events)
-      event = queue.shift
-      expect(event.get("kafka")["topic"]).to eq("logstash_topic_plain")
-      expect(event.get("kafka")["consumer_group"]).to eq(group_id_3)
-    end
+  it "should consume all messages from lz4 3-partition topic" do
+    kafka_input = LogStash::Inputs::Kafka.new(lz4_config)
+    queue = Array.new
+    t = thread_it(kafka_input, queue)
+    t.run
+    wait(timeout_seconds).for { queue.length }.to eq(num_events)
+    expect(queue.length).to eq(num_events)
   end
+  
 end

data/spec/unit/inputs/kafka_spec.rb

@@ -16,7 +16,7 @@ class MockConsumer
       raise org.apache.kafka.common.errors.WakeupException.new
     else
       10.times.map do
-        org.apache.kafka.clients.consumer.ConsumerRecord.new("logstash", 0, 0, "key", "value")
+        org.apache.kafka.clients.consumer.ConsumerRecord.new("test", 0, 0, "key", "value")
       end
     end
   end
@@ -30,7 +30,7 @@ class MockConsumer
 end
 
 describe LogStash::Inputs::Kafka do
-  let(:config) { { 'topics' => ['logstash'], 'consumer_threads' => 4 } }
+  let(:config) { { 'topics' => ['test'], 'consumer_threads' => 4 } }
   subject { LogStash::Inputs::Kafka.new(config) }
 
   it "should register" do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-kafka
 version: !ruby/object:Gem::Version
-  version: 4.2.0
+  version: 5.0.1
 platform: ruby
 authors:
 - Elasticsearch
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2017-02-18 00:00:00.000000000 Z
+date: 2016-07-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -106,6 +106,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: rspec-wait
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
 email: info@elastic.co
 executables: []
@@ -119,23 +133,11 @@ files:
 - LICENSE
 - NOTICE.TXT
 - README.md
-- lib/log4j/log4j/1.2.17/log4j-1.2.17.jar
 - lib/logstash-input-kafka_jars.rb
 - lib/logstash/inputs/kafka.rb
-- lib/net/jpountz/lz4/lz4/1.2.0/lz4-1.2.0.jar
-- lib/org/apache/kafka/kafka-clients/0.9.0.1/kafka-clients-0.9.0.1.jar
-- lib/org/slf4j/slf4j-api/1.7.6/slf4j-api-1.7.6.jar
-- lib/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar
-- lib/org/xerial/snappy/snappy-java/1.1.1.7/snappy-java-1.1.1.7.jar
 - logstash-input-kafka.gemspec
 - spec/integration/inputs/kafka_spec.rb
 - spec/unit/inputs/kafka_spec.rb
-- vendor/jar-dependencies/runtime-jars/kafka-clients-0.9.0.1.jar
-- vendor/jar-dependencies/runtime-jars/log4j-1.2.17.jar
-- vendor/jar-dependencies/runtime-jars/lz4-1.2.0.jar
-- vendor/jar-dependencies/runtime-jars/slf4j-api-1.7.6.jar
-- vendor/jar-dependencies/runtime-jars/slf4j-log4j12-1.7.13.jar
-- vendor/jar-dependencies/runtime-jars/snappy-java-1.1.1.7.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)
@@ -157,10 +159,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements:
-- jar 'org.apache.kafka:kafka-clients', '0.9.0.1'
-- jar 'org.slf4j:slf4j-log4j12', '1.7.13'
+- jar 'org.apache.kafka:kafka-clients', '0.10.0.0'
+- jar 'org.slf4j:slf4j-log4j12', '1.7.21'
 rubyforge_project:
-rubygems_version: 2.4.8
+rubygems_version: 2.6.3
 signing_key:
 specification_version: 4
 summary: This input will read events from a Kafka topic. It uses the high level consumer API provided by Kafka to read messages from the broker