logstash-input-kafka 5.1.11 → 6.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 16e4afb2f078c43e0d2c2cec5e626a39a05763ff
-  data.tar.gz: 548f91e907e2bc73a90ce449c87f8723652c1c78
+  metadata.gz: 2f20d67ea46283bd3aaf21569aab83e631c19921
+  data.tar.gz: e3da91a864987986ceef9acc22080f13a2845399
 SHA512:
-  metadata.gz: 4906c8d3c5ea405220c027d5c757283ee4c555cbe626f1459e25231da0f6350524fb395630545f092cbff25c3b83dc75b1e585febd55400c4c53af76817e215f
-  data.tar.gz: a4bda274415f4c3c6a42faf2568d8871d697686a16156deee49e4c50acd6efde90229b1155e45dc2c88b4e628377ad3147cf6c8aa26f941a63090c461613d9d6
+  metadata.gz: c8ddadc5b3151ee32602c2fed08698bf58459ab19590e6fe0c5a775ea74087649239870ed2aee591a6db1a938b675d5ab9cfd1f2a8bcf4cceeea48e612227f53
+  data.tar.gz: eb73a4f54c954c9b522a15c61ea6930ece83a3854a63c1cb3d82de7c08254a411795534a98ebc662ef0d0b73e21908c9d7adf48c0a95b25329b4b4eb18089de3

data/CHANGELOG.md

@@ -1,42 +1,5 @@
-## 5.1.11
-  Documentation fixes
-
-## 5.1.10
-  Documentation fixes
-
-## 5.1.9
-  - Docs: Fix topic title.
-
-## 5.1.8
-  - Docs: Fix asciidoc problems.
-
-## 5.1.7
-  - Fix NPE when SASL_SSL+PLAIN (no Kerberos) options are specified.
-
-## 5.1.6
-  - fix: Client ID is no longer reused across multiple Kafka consumer instances
-
-## 5.1.5
-  - Fix a bug where consumer was not correctly setup when `SASL_SSL` option was specified. 
-
-## 5.1.4
-  - Fix log4j bootstrapping for 2.4
-
-## 5.1.3
-  - Make error reporting clearer when connection fails
-  - set kerberos only when using GSSAPI
-
-## 5.1.2
-  - Docs: Move info about security features out of the compatibility matrix and into the main text.
- 
-## 5.1.1
-  - Docs: Clarify compatibility matrix and remove it from the changelog to avoid duplication.
-  
-## 5.1.0
-  - Add Kerberos authentication support.
-
-## 5.0.6
-  - default `poll_timeout_ms` to 100ms
+## 6.0.0
+  - Breaking: Support for Kafka 0.10.1.0. Only supports brokers 0.10.1.x or later.
 
 ## 5.0.5
   - place setup_log4j for logging registration behind version check

data/README.md

@@ -6,14 +6,16 @@ This is a plugin for [Logstash](https://github.com/elastic/logstash).
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
-## Logging
+## Kafka Compatibility
 
-Kafka logs do not respect the Log4J2 root logger level and defaults to INFO, for other levels, you must explicitly set the log level in your Logstash deployment's `log4j2.properties` file, e.g.:
-```
-logger.kafka.name=org.apache.kafka
-logger.kafka.appenderRef.console.ref=console
-logger.kafka.level=debug
-```
+Here's a table that describes the compatibility matrix for Kafka Broker support. Please remember that it is good advice to upgrade brokers before consumers/producers since brokers target backwards compatibility. The 0.9 broker will work with both the 0.8 consumer and 0.9 consumer APIs but not the other way around.
+
+| Kafka Broker Version | Logstash Version | Input Plugin | Output Plugin | Why? |
+|:---------------:|:------------------:|:--------------:|:---------------:|:------|
+| 0.8           | 2.0 - 2.x   | < 3.0.0 | <3.0.0 | Legacy, 0.8 is still popular |
+| 0.9           | 2.0 - 2.3.x   |   3.0.0 | 3.0.0  | Intermediate release before 0.10 that works with old Ruby Event API `[]`  |
+| 0.9          | 2.4, 5.0           |   4.0.0 | 4.0.0  | Intermediate release before 0.10 with new get/set API |
+| 0.10         | 2.4, 5.0           |   5.0.0 | 5.0.0  | Track latest Kafka release. Not compatible with 0.9 broker |
 
 ## Documentation
 

data/lib/logstash/inputs/kafka.rb

@@ -4,32 +4,25 @@ require 'stud/interval'
 require 'java'
 require 'logstash-input-kafka_jars.rb'
 
-# This input will read events from a Kafka topic. It uses the 0.10 version of the
-# consumer API provided by Kafka to read messages from the broker.
+# This input will read events from a Kafka topic. It uses the the newly designed
+# 0.10 version of consumer API provided by Kafka to read messages from the broker.
 #
 # Here's a compatibility matrix that shows the Kafka client versions that are compatible with each combination
 # of Logstash and the Kafka input plugin: 
 # 
 # [options="header"]
 # |==========================================================
-# |Kafka Client Version |Logstash Version |Plugin Version |Why?
-# |0.8       |2.0.0 - 2.x.x   |<3.0.0 |Legacy, 0.8 is still popular 
-# |0.9       |2.0.0 - 2.3.x   | 3.x.x |Works with the old Ruby Event API (`event['product']['price'] = 10`)  
-# |0.9       |2.4.x - 5.x.x   | 4.x.x |Works with the new getter/setter APIs (`event.set('[product][price]', 10)`)
-# |0.10.0.x  |2.4.x - 5.x.x   | 5.x.x |Not compatible with the <= 0.9 broker
+# |Kafka Client Version |Logstash Version |Plugin Version |Security Features |Why?
+# |0.8       |2.0.0 - 2.x.x   |<3.0.0 | |Legacy, 0.8 is still popular 
+# |0.9       |2.0.0 - 2.3.x   | 3.x.x |Basic Auth, SSL |Works with the old Ruby Event API (`event['product']['price'] = 10`)  
+# |0.9       |2.4.0 - 5.0.x   | 4.x.x |Basic Auth, SSL |Works with the new getter/setter APIs (`event.set('[product][price]', 10)`)
+# |0.10      |2.4.0 - 5.0.x   | 5.x.x |Basic Auth, SSL |Not compatible with the 0.9 broker 
 # |==========================================================
 # 
 # NOTE: We recommended that you use matching Kafka client and broker versions. During upgrades, you should
 # upgrade brokers before clients because brokers target backwards compatibility. For example, the 0.9 broker
 # is compatible with both the 0.8 consumer and 0.9 consumer APIs, but not the other way around.
 #
-# This input supports connecting to Kafka over:
-#
-# * SSL (requires plugin version 3.0.0 or later)
-# * Kerberos SASL (requires plugin version 5.1.0 or later) 
-#
-# By default security is disabled but can be turned on as needed.
-#
 # The Logstash Kafka consumer handles group management and uses the default offset management
 # strategy using Kafka topics.
 #
@@ -46,6 +39,9 @@ require 'logstash-input-kafka_jars.rb'
 #
 # Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
 #
+# This version also adds support for SSL/TLS security connection to Kafka. By default SSL is
+# disabled but can be turned on as needed.
+#
 class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   config_name 'kafka'
 
@@ -147,51 +143,17 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # The topics configuration will be ignored when using this configuration.
   config :topics_pattern, :validate => :string
   # Time kafka consumer will wait to receive new messages from topics
-  config :poll_timeout_ms, :validate => :number, :default => 100
+  config :poll_timeout_ms, :validate => :number
   # Enable SSL/TLS secured communication to Kafka broker.
-  config :ssl, :validate => :boolean, :default => false, :deprecated => "Use security_protocol => 'ssl'"
-  # The truststore type.
-  config :ssl_truststore_type, :validate => :string
+  config :ssl, :validate => :boolean, :default => false
   # The JKS truststore path to validate the Kafka broker's certificate.
   config :ssl_truststore_location, :validate => :path
   # The truststore password
   config :ssl_truststore_password, :validate => :password
-  # The keystore type.
-  config :ssl_keystore_type, :validate => :string
   # If client authentication is required, this setting stores the keystore path.
   config :ssl_keystore_location, :validate => :path
   # If client authentication is required, this setting stores the keystore password
   config :ssl_keystore_password, :validate => :password
-  # The password of the private key in the key store file.
-  config :ssl_key_password, :validate => :password
-  # Security protocol to use, which can be either of PLAINTEXT,SSL,SASL_PLAINTEXT,SASL_SSL
-  config :security_protocol, :validate => ["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"], :default => "PLAINTEXT"
-  # http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. 
-  # This may be any mechanism for which a security provider is available.
-  # GSSAPI is the default mechanism.
-  config :sasl_mechanism, :validate => :string, :default => "GSSAPI"
-  # The Kerberos principal name that Kafka broker runs as. 
-  # This can be defined either in Kafka's JAAS config or in Kafka's config.
-  config :sasl_kerberos_service_name, :validate => :string
-  # The Java Authentication and Authorization Service (JAAS) API supplies user authentication and authorization 
-  # services for Kafka. This setting provides the path to the JAAS file. Sample JAAS file for Kafka client:
-  # [source,java]
-  # ----------------------------------
-  # KafkaClient {
-  #   com.sun.security.auth.module.Krb5LoginModule required
-  #   useTicketCache=true
-  #   renewTicket=true
-  #   serviceName="kafka";
-  #   };
-  # ----------------------------------
-  #
-  # Please note that specifying `jaas_path` and `kerberos_config` in the config file will add these  
-  # to the global JVM system properties. This means if you have multiple Kafka inputs, all of them would be sharing the same 
-  # `jaas_path` and `kerberos_config`. If this is not desirable, you would have to run separate instances of Logstash on 
-  # different JVM instances.
-  config :jaas_path, :validate => :path
-  # Optional path to kerberos config file. This is krb5.conf style as detailed in https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html
-  config :kerberos_config, :validate => :path
   # Option to add Kafka metadata like topic, message size to the event.
   # This will add a field named `kafka` to the logstash event containing the following attributes:
   #   `topic`: The topic this message is associated with
@@ -204,12 +166,17 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
 
   public
   def register
+    # Logstash 2.4
+    if defined?(LogStash::Logger) && LogStash::Logger.respond_to?(:setup_log4j)
+      LogStash::Logger.setup_log4j(@logger)
+    end
+
     @runner_threads = []
   end # def register
 
   public
   def run(logstash_queue)
-    @runner_consumers = consumer_threads.times.map { |i| create_consumer("#{client_id}-#{i}") }
+    @runner_consumers = consumer_threads.times.map { || create_consumer }
     @runner_threads = @runner_consumers.map { |consumer| thread_runner(logstash_queue, consumer) }
     @runner_threads.each { |t| t.join }
   end # def run
@@ -219,11 +186,6 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
     @runner_consumers.each { |c| c.wakeup }
   end
 
-  public
-  def kafka_consumers
-    @runner_consumers
-  end
-
   private
   def thread_runner(logstash_queue, consumer)
     Thread.new do
@@ -260,7 +222,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   end
 
   private
-  def create_consumer(client_id)
+  def create_consumer
     begin
       props = java.util.Properties.new
       kafka = org.apache.kafka.clients.consumer.ConsumerConfig
@@ -290,47 +252,20 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put(kafka::SESSION_TIMEOUT_MS_CONFIG, session_timeout_ms) unless session_timeout_ms.nil?
       props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, value_deserializer_class)
 
-      props.put("security.protocol", security_protocol) unless security_protocol.nil?
+      if ssl
+        props.put("security.protocol", "SSL")
+        props.put("ssl.truststore.location", ssl_truststore_location)
+        props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
 
-      if security_protocol == "SSL"
-        set_trustore_keystore_config(props)
-      elsif security_protocol == "SASL_PLAINTEXT"
-        set_sasl_config(props)
-      elsif security_protocol == "SASL_SSL"
-        set_trustore_keystore_config(props)
-        set_sasl_config(props)
+        #Client auth stuff
+        props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
+        props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
       end
 
       org.apache.kafka.clients.consumer.KafkaConsumer.new(props)
     rescue => e
-      logger.error("Unable to create Kafka consumer from given configuration",
-                   :kafka_error_message => e,
-                   :cause => e.respond_to?(:getCause) ? e.getCause() : nil)
+      @logger.error("Unable to create Kafka consumer from given configuration", :kafka_error_message => e)
       throw e
     end
   end
-
-  def set_trustore_keystore_config(props)
-    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-    props.put("ssl.truststore.location", ssl_truststore_location)
-    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config",jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf",kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism",sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name",sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-  end
 end #class LogStash::Inputs::Kafka

data/logstash-input-kafka.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-input-kafka'
-  s.version         = '5.1.11'
+  s.version         = '6.0.0'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = 'This input will read events from a Kafka topic. It uses the high level consumer API provided by Kafka to read messages from the broker'
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -18,8 +18,8 @@ Gem::Specification.new do |s|
   # Special flag to let us know this is actually a logstash plugin
   s.metadata = { 'logstash_plugin' => 'true', 'group' => 'input'}
 
-  s.requirements << "jar 'org.apache.kafka:kafka-clients', '0.10.0.1'"
-  s.requirements << "jar 'org.apache.logging.log4j:log4j-slf4j-impl', '2.8.2'"
+  s.requirements << "jar 'org.apache.kafka:kafka-clients', '0.10.1.0'"
+  s.requirements << "jar 'org.slf4j:slf4j-log4j12', '1.7.21'"
 
   s.add_development_dependency 'jar-dependencies', '~> 0.3.2'
 

data/spec/integration/inputs/kafka_spec.rb

@@ -10,9 +10,7 @@ describe "inputs/kafka", :integration => true do
   let(:group_id_1) {rand(36**8).to_s(36)}
   let(:group_id_2) {rand(36**8).to_s(36)}
   let(:group_id_3) {rand(36**8).to_s(36)}
-  let(:group_id_4) {rand(36**8).to_s(36)}
   let(:plain_config) { { 'topics' => ['logstash_topic_plain'], 'codec' => 'plain', 'group_id' => group_id_1, 'auto_offset_reset' => 'earliest'} }
-  let(:multi_consumer_config) { plain_config.merge({"group_id" => group_id_4, "client_id" => "spec", "consumer_threads" => 3}) }
   let(:snappy_config) { { 'topics' => ['logstash_topic_snappy'], 'codec' => 'plain', 'group_id' => group_id_1, 'auto_offset_reset' => 'earliest'} }
   let(:lz4_config) { { 'topics' => ['logstash_topic_lz4'], 'codec' => 'plain', 'group_id' => group_id_1, 'auto_offset_reset' => 'earliest'} }
   let(:pattern_config) { { 'topics_pattern' => 'logstash_topic_.*', 'group_id' => group_id_2, 'codec' => 'plain', 'auto_offset_reset' => 'earliest'} }  
@@ -55,21 +53,11 @@ describe "inputs/kafka", :integration => true do
       wait(timeout_seconds).for { queue.length }.to eq(num_events)
       expect(queue.length).to eq(num_events)
     end
-
-    it "should consumer all messages with multiple consumers" do
-      kafka_input = LogStash::Inputs::Kafka.new(multi_consumer_config)
-      queue = Array.new
-      t = thread_it(kafka_input, queue)
-      t.run
-      wait(timeout_seconds).for { queue.length }.to eq(num_events)
-      expect(queue.length).to eq(num_events)
-      kafka_input.kafka_consumers.each_with_index do |consumer, i|
-        expect(consumer.metrics.keys.first.tags["client-id"]).to eq("spec-#{i}")
-      end
-    end
+    
   end
 
   describe "#kafka-topics-pattern" do
+    
     def thread_it(kafka_input, queue)
       Thread.new do
         begin
@@ -77,7 +65,7 @@ describe "inputs/kafka", :integration => true do
         end
       end
     end
-
+      
     it "should consume all messages from all 3 topics" do
       kafka_input = LogStash::Inputs::Kafka.new(pattern_config)
       queue = Array.new
@@ -85,7 +73,7 @@ describe "inputs/kafka", :integration => true do
       t.run
       wait(timeout_seconds).for { queue.length }.to eq(3*num_events)
       expect(queue.length).to eq(3*num_events)
-    end
+    end    
   end
 
   describe "#kafka-decorate" do
@@ -96,7 +84,7 @@ describe "inputs/kafka", :integration => true do
         end
       end
     end
-
+      
     it "should show the right topic and group name in decorated kafka section" do
       kafka_input = LogStash::Inputs::Kafka.new(decorate_config)
       queue = Queue.new

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-kafka
 version: !ruby/object:Gem::Version
-  version: 5.1.11
+  version: 6.0.0
 platform: ruby
 authors:
 - Elasticsearch
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2017-08-21 00:00:00.000000000 Z
+date: 2016-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -138,11 +138,11 @@ files:
 - logstash-input-kafka.gemspec
 - spec/integration/inputs/kafka_spec.rb
 - spec/unit/inputs/kafka_spec.rb
-- vendor/jar-dependencies/runtime-jars/kafka-clients-0.10.0.1.jar
-- vendor/jar-dependencies/runtime-jars/log4j-api-2.8.2.jar
-- vendor/jar-dependencies/runtime-jars/log4j-slf4j-impl-2.8.2.jar
+- vendor/jar-dependencies/runtime-jars/kafka-clients-0.10.1.0.jar
+- vendor/jar-dependencies/runtime-jars/log4j-1.2.17.jar
 - vendor/jar-dependencies/runtime-jars/lz4-1.3.0.jar
 - vendor/jar-dependencies/runtime-jars/slf4j-api-1.7.21.jar
+- vendor/jar-dependencies/runtime-jars/slf4j-log4j12-1.7.21.jar
 - vendor/jar-dependencies/runtime-jars/snappy-java-1.1.2.6.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
@@ -165,8 +165,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements:
-- jar 'org.apache.kafka:kafka-clients', '0.10.0.1'
-- jar 'org.apache.logging.log4j:log4j-slf4j-impl', '2.8.2'
+- jar 'org.apache.kafka:kafka-clients', '0.10.1.0'
+- jar 'org.slf4j:slf4j-log4j12', '1.7.21'
 rubyforge_project:
 rubygems_version: 2.4.8
 signing_key: