logstash-integration-kafka 10.5.1-java → 10.7.1-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 93e42aba43873cfb3c8165a31b184e8178f738787329a36bcde79b234d2f642a
-  data.tar.gz: 41cca0d58b15c3072d51d009399e01e6854dea4c1d0cc3791e52430365599d73
+  metadata.gz: d68bf1e5dada6ec2bea00b759962a1cf0db3b1555c38ffd2f5f4a84178a79d9e
+  data.tar.gz: d41de9544b85313e5f064d68304be2044a77837e3a55778836cb0f04da085b76
 SHA512:
-  metadata.gz: b40a479faa7568a326b53fb721f692bdb4f66e0c87e3553d39b29456b3e8b61f4849d0c325c21c3a8247989c16467d4c72268673b0c7e7ee41de7af1d939a523
-  data.tar.gz: 1df3fe42154e9b035145a159fbca4605f3c82d8d61765af3242ec6a2ff571b9b9d2bd536e484dfba766ca353722976c0c81b1e8f70676f083cec0ab3b6983a4e
+  metadata.gz: 7b25d38f7e6014889a03060e38f04bdfde157dcec9a7bb5567936cdae636c83448594cc53941980363b463521561ab27bc4d2bc1ad9c4d3f2a1f4f83cca11878
+  data.tar.gz: 88244076005fa307881cf3b5108fb0c2f274b8e03f9238fda2f629799ab4751c9d1c7ab09148d67894191045e676686d479c1612ce112d597654d576b5652b0f

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## 10.7.1
+  - Fix: dropped usage of SHUTDOWN event deprecated since Logstash 5.0 [#71](https://github.com/logstash-plugins/logstash-integration-kafka/issue/71)
+  
+## 10.7.0
+  - Switched use from Faraday to Manticore as HTTP client library to access Schema Registry service 
+    to fix issue [#63](https://github.com/logstash-plugins/logstash-integration-kafka/pull/63) 
+
+## 10.6.0
+  - Added functionality to Kafka input to use Avro deserializer in retrieving data from Kafka. The schema is retrieved
+    from an instance of Confluent's Schema Registry service [#51](https://github.com/logstash-plugins/logstash-integration-kafka/pull/51)
+     
+## 10.5.3
+  - Fix: set (optional) truststore when endpoint id check disabled [#60](https://github.com/logstash-plugins/logstash-integration-kafka/pull/60).
+    Since **10.1.0** disabling server host-name verification (`ssl_endpoint_identification_algorithm => ""`) did not allow 
+    the (output) plugin to set `ssl_truststore_location => "..."`.
+
+## 10.5.2
+  - Docs: explain group_id in case of multiple inputs [#59](https://github.com/logstash-plugins/logstash-integration-kafka/pull/59)
+
 ## 10.5.1
   - [DOC]Replaced plugin_header file with plugin_header-integration file. [#46](https://github.com/logstash-plugins/logstash-integration-kafka/pull/46)
   - [DOC]Update kafka client version across kafka integration docs [#47](https://github.com/logstash-plugins/logstash-integration-kafka/pull/47)

data/README.md

@@ -1,6 +1,6 @@
 # Logstash Plugin
 
-[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-integration-kafka.svg)](https://travis-ci.org/logstash-plugins/logstash-integration-kafka)
+[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-integration-kafka.svg)](https://travis-ci.com/logstash-plugins/logstash-integration-kafka)
 
 This is a plugin for [Logstash](https://github.com/elastic/logstash).
 

data/docs/input-kafka.asciidoc

@@ -124,6 +124,10 @@ See the https://kafka.apache.org/{kafka_client_doc}/documentation for more detai
 | <<plugins-{type}s-{plugin}-sasl_jaas_config>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_kerberos_service_name>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_mechanism>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_key>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_proxy>> |<<uri,uri>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_secret>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_url>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-security_protocol>> |<<string,string>>, one of `["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"]`|No
 | <<plugins-{type}s-{plugin}-send_buffer_bytes>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-session_timeout_ms>> |<<number,number>>|No
@@ -314,7 +318,11 @@ before answering the request.
 
 The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
 that happens to be made up of multiple processors. Messages in a topic will be distributed to all
-Logstash instances with the same `group_id`
+Logstash instances with the same `group_id`.
+
+NOTE: In cases when multiple inputs are being used in a single pipeline, reading from different topics,
+it's essential to set a different `group_id => ...` for each input. Setting a unique `client_id => ...`
+is also recommended.
 
 [id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
 ===== `heartbeat_interval_ms` 
@@ -524,6 +532,44 @@ http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used fo
 This may be any mechanism for which a security provider is available.
 GSSAPI is the default mechanism.
 
+[id="plugins-{type}s-{plugin}-schema_registry_key"]
+===== `schema_registry_key`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Set the username for basic authorization to access remote Schema Registry.
+
+[id="plugins-{type}s-{plugin}-schema_registry_proxy"]
+===== `schema_registry_proxy`
+
+* Value type is <<uri,uri>>
+* There is no default value for this setting.
+
+Set the address of a forward HTTP proxy. An empty string is treated as if proxy was not set.
+
+[id="plugins-{type}s-{plugin}-schema_registry_secret"]
+===== `schema_registry_secret`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Set the password for basic authorization to access remote Schema Registry.
+
+[id="plugins-{type}s-{plugin}-schema_registry_url"]
+===== `schema_registry_url`
+
+* Value type is <<uri,uri>>
+
+The URI that points to an instance of the
+https://docs.confluent.io/current/schema-registry/index.html[Schema Registry] service,
+used to manage Avro schemas. Be sure that the Avro schemas for deserializing the data from
+the specified topics have been uploaded to the Schema Registry service.
+The schemas must follow a naming convention with the pattern <topic name>-value.
+
+Use either the Schema Registry config option or the
+<<plugins-{type}s-{plugin}-value_deserializer_class>> config option, but not both.
+
 [id="plugins-{type}s-{plugin}-security_protocol"]
 ===== `security_protocol` 
 
@@ -637,7 +683,10 @@ The topics configuration will be ignored when using this configuration.
   * Value type is <<string,string>>
   * Default value is `"org.apache.kafka.common.serialization.StringDeserializer"`
 
-Java Class used to deserialize the record's value
+Java Class used to deserialize the record's value.
+A custom value deserializer can be used only if you are not using a Schema Registry.
+Use either the value_deserializer_class config option or the
+<<plugins-{type}s-{plugin}-schema_registry_url>> config option, but not both.
 
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]

data/lib/logstash-integration-kafka_jars.rb

@@ -1,8 +1,17 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('org.apache.kafka', 'kafka-clients', '2.4.1')
-require_jar('com.github.luben', 'zstd-jni', '1.4.3-1')
-require_jar('org.slf4j', 'slf4j-api', '1.7.28')
-require_jar('org.lz4', 'lz4-java', '1.6.0')
+require_jar('io.confluent', 'kafka-avro-serializer', '5.5.1')
+require_jar('io.confluent', 'kafka-schema-serializer', '5.5.1')
+require_jar('io.confluent', 'common-config', '5.5.1')
+require_jar('org.apache.avro', 'avro', '1.9.2')
+require_jar('io.confluent', 'kafka-schema-registry-client', '5.5.1')
+require_jar('org.apache.kafka', 'kafka_2.12', '2.5.1')
+require_jar('io.confluent', 'common-utils', '5.5.1')
+require_jar('javax.ws.rs', 'javax.ws.rs-api', '2.1.1')
+require_jar('org.glassfish.jersey.core', 'jersey-common', '2.30')
+require_jar('org.apache.kafka', 'kafka-clients', '2.5.1')
+require_jar('com.github.luben', 'zstd-jni', '1.4.4-7')
+require_jar('org.slf4j', 'slf4j-api', '1.7.30')
+require_jar('org.lz4', 'lz4-java', '1.7.1')
 require_jar('org.xerial.snappy', 'snappy-java', '1.1.7.3')

data/lib/logstash/inputs/kafka.rb

@@ -3,6 +3,11 @@ require 'logstash/inputs/base'
 require 'stud/interval'
 require 'java'
 require 'logstash-integration-kafka_jars.rb'
+require 'logstash/plugin_mixins/kafka_support'
+require 'manticore'
+require "json"
+require "logstash/json"
+require_relative '../plugin_mixins/common'
 
 # This input will read events from a Kafka topic. It uses the 0.10 version of
 # the consumer API provided by Kafka to read messages from the broker.
@@ -48,6 +53,12 @@ require 'logstash-integration-kafka_jars.rb'
 # Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
 #
 class LogStash::Inputs::Kafka < LogStash::Inputs::Base
+
+  DEFAULT_DESERIALIZER_CLASS = "org.apache.kafka.common.serialization.StringDeserializer"
+
+  include LogStash::PluginMixins::KafkaSupport
+  include ::LogStash::PluginMixins::KafkaAvroSchemaRegistry
+
   config_name 'kafka'
 
   default :codec, 'plain'
@@ -163,7 +174,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # and a rebalance operation is triggered for the group identified by `group_id`
   config :session_timeout_ms, :validate => :number, :default => 10_000 # (10s) Kafka default
   # Java Class used to deserialize the record's value
-  config :value_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
+  config :value_deserializer_class, :validate => :string, :default => DEFAULT_DESERIALIZER_CLASS
   # A list of topics to subscribe to, defaults to ["logstash"].
   config :topics, :validate => :array, :default => ["logstash"]
   # A topic regex pattern to subscribe to. 
@@ -232,11 +243,11 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   #   `timestamp`: The timestamp of this message
   config :decorate_events, :validate => :boolean, :default => false
 
-
   public
   def register
     @runner_threads = []
-  end # def register
+    check_schema_registry_parameters
+  end
 
   public
   def run(logstash_queue)
@@ -274,6 +285,13 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
           for record in records do
             codec_instance.decode(record.value.to_s) do |event|
               decorate(event)
+              if schema_registry_url
+                json = LogStash::Json.load(record.value.to_s)
+                json.each do |k, v|
+                  event.set(k, v)
+                end
+                event.remove("message")
+              end
               if @decorate_events
                 event.set("[@metadata][kafka][topic]", record.topic)
                 event.set("[@metadata][kafka][consumer_group]", @group_id)
@@ -333,7 +351,18 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put(kafka::CLIENT_RACK_CONFIG, client_rack) unless client_rack.nil? 
 
       props.put("security.protocol", security_protocol) unless security_protocol.nil?
-
+      if schema_registry_url
+        props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroDeserializer.java_class)
+        serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+        props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, schema_registry_url.to_s)
+        if schema_registry_proxy && !schema_registry_proxy.empty?
+          props.put(serdes_config::PROXY_HOST, @schema_registry_proxy_host)
+          props.put(serdes_config::PROXY_PORT, @schema_registry_proxy_port)
+        end
+        if schema_registry_key && !schema_registry_key.empty?
+          props.put(serdes_config::USER_INFO_CONFIG, schema_registry_key + ":" + schema_registry_secret.value)
+        end
+      end
       if security_protocol == "SSL"
         set_trustore_keystore_config(props)
       elsif security_protocol == "SASL_PLAINTEXT"
@@ -370,29 +399,4 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
     end
   end
 
-  def set_trustore_keystore_config(props)
-    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-    props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
-    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism", sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
-  end
 end #class LogStash::Inputs::Kafka

data/lib/logstash/outputs/kafka.rb

@@ -2,6 +2,7 @@ require 'logstash/namespace'
 require 'logstash/outputs/base'
 require 'java'
 require 'logstash-integration-kafka_jars.rb'
+require 'logstash/plugin_mixins/kafka_support'
 
 # Write events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on
 # the broker.
@@ -50,6 +51,8 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
 
   java_import org.apache.kafka.clients.producer.ProducerRecord
 
+  include LogStash::PluginMixins::KafkaSupport
+
   declare_threadsafe!
 
   config_name 'kafka'
@@ -221,7 +224,6 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
     end
 
     events.each do |event|
-      break if event == LogStash::SHUTDOWN
       @codec.encode(event)
     end
 
@@ -389,35 +391,4 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
     end
   end
 
-  def set_trustore_keystore_config(props)
-    unless ssl_endpoint_identification_algorithm.to_s.strip.empty?
-      if ssl_truststore_location.nil?
-        raise LogStash::ConfigurationError, "ssl_truststore_location must be set when SSL is enabled"
-      end
-      props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-      props.put("ssl.truststore.location", ssl_truststore_location)
-      props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-    end
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism",sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
-  end
-
 end #class LogStash::Outputs::Kafka

data/lib/logstash/plugin_mixins/common.rb

@@ -0,0 +1,93 @@
+module LogStash
+  module PluginMixins
+    module KafkaAvroSchemaRegistry
+
+      def self.included(base)
+        base.extend(self)
+        base.setup_schema_registry_config
+      end
+
+      def setup_schema_registry_config
+        # Option to set key to access Schema Registry.
+        config :schema_registry_key, :validate => :string
+
+        # Option to set secret to access Schema Registry.
+        config :schema_registry_secret, :validate => :password
+
+        # Option to set the endpoint of the Schema Registry.
+        # This option permit the usage of Avro Kafka deserializer which retrieve the schema of the Avro message from an
+        # instance of schema registry. If this option has value `value_deserializer_class` nor `topics_pattern` could be valued
+        config :schema_registry_url, :validate => :uri
+
+        # Option to set the proxy of the Schema Registry.
+        # This option permits to define a proxy to be used to reach the schema registry service instance.
+        config :schema_registry_proxy, :validate => :uri
+      end
+
+      def check_schema_registry_parameters
+        if @schema_registry_url
+          check_for_schema_registry_conflicts
+          @schema_registry_proxy_host, @schema_registry_proxy_port  = split_proxy_into_host_and_port(schema_registry_proxy)
+          check_for_key_and_secret
+          check_for_schema_registry_connectivity_and_subjects
+        end
+      end
+
+      private
+      def check_for_schema_registry_conflicts
+        if @value_deserializer_class != LogStash::Inputs::Kafka::DEFAULT_DESERIALIZER_CLASS
+          raise LogStash::ConfigurationError, 'Option schema_registry_url prohibit the customization of value_deserializer_class'
+        end
+        if @topics_pattern && !@topics_pattern.empty?
+         raise LogStash::ConfigurationError, 'Option schema_registry_url prohibit the customization of topics_pattern'
+        end
+      end
+
+      private
+      def check_for_schema_registry_connectivity_and_subjects
+        options = {}
+        if schema_registry_proxy && !schema_registry_proxy.empty?
+          options[:proxy] = schema_registry_proxy.to_s
+        end
+        if schema_registry_key and !schema_registry_key.empty?
+          options[:auth] = {:user => schema_registry_key, :password => schema_registry_secret.value}
+        end
+        client = Manticore::Client.new(options)
+
+        begin
+          response = client.get(@schema_registry_url.to_s + '/subjects').body
+        rescue Manticore::ManticoreException => e
+          raise LogStash::ConfigurationError.new("Schema registry service doesn't respond, error: #{e.message}")
+        end
+        registered_subjects = JSON.parse response
+        expected_subjects = @topics.map { |t| "#{t}-value"}
+        if (expected_subjects & registered_subjects).size != expected_subjects.size
+          undefined_topic_subjects = expected_subjects - registered_subjects
+          raise LogStash::ConfigurationError, "The schema registry does not contain definitions for required topic subjects: #{undefined_topic_subjects}"
+        end
+      end
+
+      def split_proxy_into_host_and_port(proxy_uri)
+        return nil unless proxy_uri && !proxy_uri.empty?
+
+        port = proxy_uri.port
+
+        host_spec = ""
+        host_spec << proxy_uri.scheme || "http"
+        host_spec << "://"
+        host_spec << "#{proxy_uri.userinfo}@" if proxy_uri.userinfo
+        host_spec << proxy_uri.host
+
+        [host_spec, port]
+      end
+
+      def check_for_key_and_secret
+        if schema_registry_key and !schema_registry_key.empty?
+          if !schema_registry_secret or schema_registry_secret.value.empty?
+            raise LogStash::ConfigurationError, "Setting `schema_registry_secret` is required when `schema_registry_key` is provided."
+          end
+        end
+      end
+    end
+  end
+end

data/lib/logstash/plugin_mixins/kafka_support.rb

@@ -0,0 +1,29 @@
+module LogStash module PluginMixins module KafkaSupport
+
+  def set_trustore_keystore_config(props)
+    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
+    props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
+    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
+
+    # Client auth stuff
+    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
+    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
+    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
+    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
+    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
+  end
+
+  def set_sasl_config(props)
+    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
+    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
+
+    props.put("sasl.mechanism", sasl_mechanism)
+    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
+      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
+    end
+
+    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
+    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
+  end
+
+end end end

data/logstash-integration-kafka.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-integration-kafka'
-  s.version         = '10.5.1'
+  s.version         = '10.7.1'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Integration with Kafka - input and output plugins"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+
@@ -46,6 +46,7 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'logstash-codec-json'
   s.add_runtime_dependency 'logstash-codec-plain'
   s.add_runtime_dependency 'stud', '>= 0.0.22', '< 0.1.0'
+  s.add_runtime_dependency "manticore", '>= 0.5.4', '< 1.0.0'
 
   s.add_development_dependency 'logstash-devutils'
   s.add_development_dependency 'rspec-wait'

data/spec/integration/inputs/kafka_spec.rb

@@ -2,6 +2,9 @@
 require "logstash/devutils/rspec/spec_helper"
 require "logstash/inputs/kafka"
 require "rspec/wait"
+require "stud/try"
+require "manticore"
+require "json"
 
 # Please run kafka_test_setup.sh prior to executing this integration test.
 describe "inputs/kafka", :integration => true do
@@ -120,20 +123,192 @@ describe "inputs/kafka", :integration => true do
       end
     end
   end
+end
+
+private
+
+def consume_messages(config, queue: Queue.new, timeout:, event_count:)
+  kafka_input = LogStash::Inputs::Kafka.new(config)
+  t = Thread.new { kafka_input.run(queue) }
+  begin
+    t.run
+    wait(timeout).for { queue.length }.to eq(event_count) unless timeout.eql?(false)
+    block_given? ? yield(queue, kafka_input) : queue
+  ensure
+    t.kill
+    t.join(30_000)
+  end
+end
+
+
+describe "schema registry connection options" do
+  context "remote endpoint validation" do
+    it "should fail if not reachable" do
+      config = {'schema_registry_url' => 'http://localnothost:8081'}
+      kafka_input = LogStash::Inputs::Kafka.new(config)
+      expect { kafka_input.register }.to raise_error LogStash::ConfigurationError, /Schema registry service doesn't respond.*/
+    end
+
+    it "should fail if any topic is not matched by a subject on the schema registry" do
+      config = {
+        'schema_registry_url' => 'http://localhost:8081',
+        'topics' => ['temperature_stream']
+      }
+
+      kafka_input = LogStash::Inputs::Kafka.new(config)
+      expect { kafka_input.register }.to raise_error LogStash::ConfigurationError, /The schema registry does not contain definitions for required topic subjects: \["temperature_stream-value"\]/
+    end
+
+    context "register with subject present" do
+      SUBJECT_NAME = "temperature_stream-value"
+
+      before(:each) do
+        response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), SUBJECT_NAME)
+        expect( response.code ).to be(200)
+      end
 
-  private
+      after(:each) do
+        schema_registry_client = Manticore::Client.new
+        delete_remote_schema(schema_registry_client, SUBJECT_NAME)
+      end
 
-  def consume_messages(config, queue: Queue.new, timeout:, event_count:)
-    kafka_input = LogStash::Inputs::Kafka.new(config)
-    t = Thread.new { kafka_input.run(queue) }
-    begin
-      t.run
-      wait(timeout).for { queue.length }.to eq(event_count) unless timeout.eql?(false)
-      block_given? ? yield(queue, kafka_input) : queue
-    ensure
-      t.kill
-      t.join(30_000)
+      it "should correctly complete registration phase" do
+        config = {
+          'schema_registry_url' => 'http://localhost:8081',
+          'topics' => ['temperature_stream']
+        }
+        kafka_input = LogStash::Inputs::Kafka.new(config)
+        kafka_input.register
+      end
     end
   end
+end
 
+def save_avro_schema_to_schema_registry(schema_file, subject_name)
+  raw_schema = File.readlines(schema_file).map(&:chomp).join
+  raw_schema_quoted = raw_schema.gsub('"', '\"')
+  response = Manticore.post("http://localhost:8081/subjects/#{subject_name}/versions",
+          body: '{"schema": "' + raw_schema_quoted + '"}',
+          headers: {"Content-Type" => "application/vnd.schemaregistry.v1+json"})
+  response
 end
+
+def delete_remote_schema(schema_registry_client, subject_name)
+  expect(schema_registry_client.delete("http://localhost:8081/subjects/#{subject_name}").code ).to be(200)
+  expect(schema_registry_client.delete("http://localhost:8081/subjects/#{subject_name}?permanent=true").code ).to be(200)
+end
+
+# AdminClientConfig = org.alpache.kafka.clients.admin.AdminClientConfig
+
+describe "Schema registry API", :integration => true do
+
+  let(:schema_registry) { Manticore::Client.new }
+
+  context 'listing subject on clean instance' do
+    it "should return an empty set" do
+      subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+      expect( subjects ).to be_empty
+    end
+  end
+
+  context 'send a schema definition' do
+    it "save the definition" do
+      response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), "schema_test_1")
+      expect( response.code ).to be(200)
+      delete_remote_schema(schema_registry, "schema_test_1")
+    end
+
+    it "delete the schema just added" do
+      response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), "schema_test_1")
+      expect( response.code ).to be(200)
+
+      expect( schema_registry.delete('http://localhost:8081/subjects/schema_test_1?permanent=false').code ).to be(200)
+      sleep(1)
+      subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+      expect( subjects ).to be_empty
+    end
+  end
+
+  context 'use the schema to serialize' do
+    after(:each) do
+      expect( schema_registry.delete('http://localhost:8081/subjects/topic_avro-value').code ).to be(200)
+      sleep 1
+      expect( schema_registry.delete('http://localhost:8081/subjects/topic_avro-value?permanent=true').code ).to be(200)
+
+      Stud.try(3.times, [StandardError, RSpec::Expectations::ExpectationNotMetError]) do
+        wait(10).for do
+          subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+          subjects.empty?
+        end.to be_truthy
+      end
+    end
+
+    let(:group_id_1) {rand(36**8).to_s(36)}
+
+    let(:avro_topic_name) { "topic_avro" }
+
+    let(:plain_config) do
+      { 'schema_registry_url' => 'http://localhost:8081',
+        'topics' => [avro_topic_name],
+        'codec' => 'plain',
+        'group_id' => group_id_1,
+        'auto_offset_reset' => 'earliest' }
+    end
+
+    def delete_topic_if_exists(topic_name)
+      props = java.util.Properties.new
+      props.put(Java::org.apache.kafka.clients.admin.AdminClientConfig::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
+
+      admin_client = org.apache.kafka.clients.admin.AdminClient.create(props)
+      topics_list = admin_client.listTopics().names().get()
+      if topics_list.contains(topic_name)
+        result = admin_client.deleteTopics([topic_name])
+        result.values.get(topic_name).get()
+      end
+    end
+
+    def write_some_data_to(topic_name)
+      props = java.util.Properties.new
+      config = org.apache.kafka.clients.producer.ProducerConfig
+
+      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+      props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081")
+
+      props.put(config::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
+      props.put(config::KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer.java_class)
+      props.put(config::VALUE_SERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroSerializer.java_class)
+
+      parser = org.apache.avro.Schema::Parser.new()
+      user_schema = '''{"type":"record",
+                        "name":"myrecord",
+                        "fields":[
+                            {"name":"str_field", "type": "string"},
+                            {"name":"map_field", "type": {"type": "map", "values": "string"}}
+                        ]}'''
+      schema = parser.parse(user_schema)
+      avro_record = org.apache.avro.generic.GenericData::Record.new(schema)
+      avro_record.put("str_field", "value1")
+      avro_record.put("map_field", {"inner_field" => "inner value"})
+
+      producer = org.apache.kafka.clients.producer.KafkaProducer.new(props)
+      record = org.apache.kafka.clients.producer.ProducerRecord.new(topic_name, "avro_key", avro_record)
+      producer.send(record)
+    end
+
+    it "stored a new schema using Avro Kafka serdes" do
+      delete_topic_if_exists avro_topic_name
+      write_some_data_to avro_topic_name
+
+      subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+      expect( subjects ).to contain_exactly("topic_avro-value")
+
+      num_events = 1
+      queue = consume_messages(plain_config, timeout: 30, event_count: num_events)
+      expect(queue.length).to eq(num_events)
+      elem = queue.pop
+      expect( elem.to_hash).not_to include("message")
+      expect( elem.get("str_field") ).to eq("value1")
+      expect( elem.get("map_field")["inner_field"] ).to eq("inner value")
+    end
+  end
+end

data/spec/unit/inputs/avro_schema_fixture_payment.asvc

@@ -0,0 +1,8 @@
+{"namespace": "io.confluent.examples.clients.basicavro",
+ "type": "record",
+ "name": "Payment",
+ "fields": [
+     {"name": "id", "type": "string"},
+     {"name": "amount", "type": "double"}
+ ]
+}

data/spec/unit/inputs/kafka_spec.rb

@@ -37,6 +37,22 @@ describe LogStash::Inputs::Kafka do
     expect { subject.register }.to_not raise_error
   end
 
+  context "register parameter verification" do
+    let(:config) do
+      { 'schema_registry_url' => 'http://localhost:8081', 'topics' => ['logstash'], 'consumer_threads' => 4 }
+    end
+
+    it "schema_registry_url conflict with value_deserializer_class should fail" do
+      config['value_deserializer_class'] = 'my.fantasy.Deserializer'
+      expect { subject.register }.to raise_error LogStash::ConfigurationError, /Option schema_registry_url prohibit the customization of value_deserializer_class/
+    end
+
+    it "schema_registry_url conflict with topics_pattern should fail" do
+      config['topics_pattern'] = 'topic_.*'
+      expect { subject.register }.to raise_error LogStash::ConfigurationError, /Option schema_registry_url prohibit the customization of topics_pattern/
+    end
+  end
+
   context 'with client_rack' do
     let(:config) { super.merge('client_rack' => 'EU-R1') }
 

data/spec/unit/outputs/kafka_spec.rb

@@ -50,9 +50,10 @@ describe "outputs/kafka" do
       kafka.multi_receive([event])
     end
 
-    it 'should raise config error when truststore location is not set and ssl is enabled' do
+    it 'should not raise config error when truststore location is not set and ssl is enabled' do
       kafka = LogStash::Outputs::Kafka.new(simple_kafka_config.merge("security_protocol" => "SSL"))
-      expect { kafka.register }.to raise_error(LogStash::ConfigurationError, /ssl_truststore_location must be set when SSL is enabled/)
+      expect(org.apache.kafka.clients.producer.KafkaProducer).to receive(:new)
+      expect { kafka.register }.to_not raise_error
     end
   end
 
@@ -225,21 +226,31 @@ describe "outputs/kafka" do
   context 'when ssl endpoint identification disabled' do
 
     let(:config) do
-      simple_kafka_config.merge('ssl_endpoint_identification_algorithm' => '', 'security_protocol' => 'SSL')
+      simple_kafka_config.merge(
+          'security_protocol' => 'SSL',
+          'ssl_endpoint_identification_algorithm' => '',
+          'ssl_truststore_location' => truststore_path,
+      )
+    end
+
+    let(:truststore_path) do
+      File.join(File.dirname(__FILE__), '../../fixtures/trust-store_stub.jks')
     end
 
     subject { LogStash::Outputs::Kafka.new(config) }
 
-    it 'does not configure truststore' do
+    it 'sets empty ssl.endpoint.identification.algorithm' do
       expect(org.apache.kafka.clients.producer.KafkaProducer).
-          to receive(:new).with(hash_excluding('ssl.truststore.location' => anything))
+          to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
       subject.register
     end
 
-    it 'sets empty ssl.endpoint.identification.algorithm' do
+    it 'configures truststore' do
       expect(org.apache.kafka.clients.producer.KafkaProducer).
-          to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
+          to receive(:new).with(hash_including('ssl.truststore.location' => truststore_path))
       subject.register
     end
+
   end
+
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-integration-kafka
 version: !ruby/object:Gem::Version
-  version: 10.5.1
+  version: 10.7.1
 platform: java
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-08-20 00:00:00.000000000 Z
+date: 2021-02-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -106,6 +106,26 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: 0.1.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.4
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+  name: manticore
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.4
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 1.0.0
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
@@ -183,15 +203,28 @@ files:
 - lib/logstash-integration-kafka_jars.rb
 - lib/logstash/inputs/kafka.rb
 - lib/logstash/outputs/kafka.rb
+- lib/logstash/plugin_mixins/common.rb
+- lib/logstash/plugin_mixins/kafka_support.rb
 - logstash-integration-kafka.gemspec
+- spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
 - spec/integration/outputs/kafka_spec.rb
+- spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb
-- vendor/jar-dependencies/com/github/luben/zstd-jni/1.4.3-1/zstd-jni-1.4.3-1.jar
-- vendor/jar-dependencies/org/apache/kafka/kafka-clients/2.4.1/kafka-clients-2.4.1.jar
-- vendor/jar-dependencies/org/lz4/lz4-java/1.6.0/lz4-java-1.6.0.jar
-- vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar
+- vendor/jar-dependencies/com/github/luben/zstd-jni/1.4.4-7/zstd-jni-1.4.4-7.jar
+- vendor/jar-dependencies/io/confluent/common-config/5.5.1/common-config-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/common-utils/5.5.1/common-utils-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-avro-serializer/5.5.1/kafka-avro-serializer-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-registry-client/5.5.1/kafka-schema-registry-client-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-serializer/5.5.1/kafka-schema-serializer-5.5.1.jar
+- vendor/jar-dependencies/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.jar
+- vendor/jar-dependencies/org/apache/avro/avro/1.9.2/avro-1.9.2.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka-clients/2.5.1/kafka-clients-2.5.1.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka_2.12/2.5.1/kafka_2.12-2.5.1.jar
+- vendor/jar-dependencies/org/glassfish/jersey/core/jersey-common/2.30/jersey-common-2.30.jar
+- vendor/jar-dependencies/org/lz4/lz4-java/1.7.1/lz4-java-1.7.1.jar
+- vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar
 - vendor/jar-dependencies/org/xerial/snappy/snappy-java/1.1.7.3/snappy-java-1.1.7.3.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
@@ -222,7 +255,9 @@ signing_key:
 specification_version: 4
 summary: Integration with Kafka - input and output plugins
 test_files:
+- spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
 - spec/integration/outputs/kafka_spec.rb
+- spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb