logstash-integration-kafka 10.5.0-java → 10.7.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c35575aafc1330e1fac14f137c818b3836f399ee1f4514a86a8a7387c4d8e8e9
-  data.tar.gz: 7fb89bca8ec2b25e07ab411b75f7de8fc4edc97f22fd2b0d5869452b10a529d9
+  metadata.gz: 166b8a4d60aaa51f4c22e6ba98f5ecf9c5991ce853c3dfa8ae15c85ad28ad4bd
+  data.tar.gz: 1fb417190daa76b08596d8c35df7004b0f1ea284eff347ab7bd5eaeabd004e0d
 SHA512:
-  metadata.gz: 7d0185e11c203175272ac52eb89a1fb3c1a09906832bc34dde3e2eca513575d9b47c79ca741be609c07f3d8f3d191f4371447d6987ad715516da786f7c1622a3
-  data.tar.gz: 93063bdf2cb603134865fc745c31c37a134d2fca09fac7bd9d68c798c792f471c1388fb29a600b45cc56d506c35a700a2c1236577d0fcbf1a65097cea91bf3f4
+  metadata.gz: 25150a143b4ccd2375572697df7bafadc87b35930f11e9916a7db1a25a69830ab1e3ae4a2d32119bc127c81af557af277188ef705fed24d1c8f1929e5bc1e73d
+  data.tar.gz: 845de0baed1db54a6ec2901b92665deca90a21f40b00400a5ae5490d6355f3771048547b158dbe19b8c7ac66587398640052c6295546793ee95251df276e35af

data/CHANGELOG.md

@@ -1,3 +1,24 @@
+## 10.7.0
+  - Switched use from Faraday to Manticore as HTTP client library to access Schema Registry service 
+    to fix issue [#63](https://github.com/logstash-plugins/logstash-integration-kafka/issue/63) 
+
+## 10.6.0
+  - Added functionality to Kafka input to use Avro deserializer in retrieving data from Kafka. The schema is retrieved
+    from an instance of Confluent's Schema Registry service [#51](https://github.com/logstash-plugins/logstash-integration-kafka/pull/51)
+     
+## 10.5.3
+  - Fix: set (optional) truststore when endpoint id check disabled [#60](https://github.com/logstash-plugins/logstash-integration-kafka/pull/60).
+    Since **10.1.0** disabling server host-name verification (`ssl_endpoint_identification_algorithm => ""`) did not allow 
+    the (output) plugin to set `ssl_truststore_location => "..."`.
+
+## 10.5.2
+  - Docs: explain group_id in case of multiple inputs [#59](https://github.com/logstash-plugins/logstash-integration-kafka/pull/59)
+
+## 10.5.1
+  - [DOC]Replaced plugin_header file with plugin_header-integration file. [#46](https://github.com/logstash-plugins/logstash-integration-kafka/pull/46)
+  - [DOC]Update kafka client version across kafka integration docs [#47](https://github.com/logstash-plugins/logstash-integration-kafka/pull/47)
+  - [DOC]Replace hard-coded kafka client and doc path version numbers with attributes to simplify doc maintenance [#48](https://github.com/logstash-plugins/logstash-integration-kafka/pull/48)  
+
 ## 10.5.0
   - Changed: retry sending messages only for retriable exceptions [#27](https://github.com/logstash-plugins/logstash-integration-kafka/pull/29)
 

data/docs/index.asciidoc

@@ -1,6 +1,7 @@
 :plugin: kafka
 :type: integration
 :no_codec:
+:kafka_client: 2.4
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -21,11 +22,15 @@ include::{include_path}/plugin_header.asciidoc[]
 
 ==== Description
 
-The Kafka Integration Plugin provides integrated plugins for working with the https://kafka.apache.org/[Kafka] distributed streaming platform.
+The Kafka Integration Plugin provides integrated plugins for working with the
+https://kafka.apache.org/[Kafka] distributed streaming platform.
 
  - {logstash-ref}/plugins-inputs-kafka.html[Kafka Input Plugin]
  - {logstash-ref}/plugins-outputs-kafka.html[Kafka Output Plugin]
  
-This plugin uses Kafka Client 2.4. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the official
+https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
+compatibility reference]. If the linked compatibility wiki is not up-to-date,
+please contact Kafka support/community to confirm compatibility.
 
 :no_codec!:

data/docs/input-kafka.asciidoc

@@ -1,6 +1,9 @@
+:integration: kafka
 :plugin: kafka
 :type: input
 :default_codec: plain
+:kafka_client: 2.4
+:kafka_client_doc: 24
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -17,15 +20,20 @@ END - GENERATED VARIABLES, DO NOT EDIT!
 
 === Kafka input plugin
 
-include::{include_path}/plugin_header.asciidoc[]
+include::{include_path}/plugin_header-integration.asciidoc[]
 
 ==== Description
 
 This input will read events from a Kafka topic.
 
-This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the
+official
+https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
+compatibility reference]. If the linked compatibility wiki is not up-to-date,
+please contact Kafka support/community to confirm compatibility.
 
-If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
+If you require features not yet available in this plugin (including client
+version upgrades), please file an issue with details about what you need.
 
 This input supports connecting to Kafka over:
 
@@ -46,9 +54,9 @@ the same `group_id`.
 Ideally you should have as many threads as the number of partitions for a perfect balance --
 more threads than partitions means that some threads will be idle
 
-For more information see https://kafka.apache.org/24/documentation.html#theconsumer
+For more information see https://kafka.apache.org/{kafka_client_doc}/documentation.html#theconsumer
 
-Kafka consumer configuration: https://kafka.apache.org/24/documentation.html#consumerconfigs
+Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/documentation.html#consumerconfigs
 
 ==== Metadata fields
 
@@ -59,7 +67,11 @@ The following metadata from Kafka broker are added under the `[@metadata]` field
 * `[@metadata][kafka][partition]`: Partition info for this message.
 * `[@metadata][kafka][offset]`: Original record offset for this message.
 * `[@metadata][kafka][key]`: Record key, if any.
-* `[@metadata][kafka][timestamp]`: Timestamp in the Record. Depending on your broker configuration, this can be either when the record was created (default) or when it was received by the broker. See more about property log.message.timestamp.type at https://kafka.apache.org/10/documentation.html#brokerconfigs
+* `[@metadata][kafka][timestamp]`: Timestamp in the Record. 
+Depending on your broker configuration, this can be
+either when the record was created (default) or when it was received by the
+broker. See more about property log.message.timestamp.type at
+https://kafka.apache.org/{kafka_client_doc}/documentation.html#brokerconfigs
 
 Metadata is only added to the event if the `decorate_events` option is set to true (it defaults to false).
 
@@ -73,7 +85,7 @@ This plugin supports these configuration options plus the <<plugins-{type}s-{plu
 
 NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
 and might change if Kafka's consumer defaults change.
-See the https://kafka.apache.org/24/documentation for more details.
+See the https://kafka.apache.org/{kafka_client_doc}/documentation for more details.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -112,6 +124,10 @@ See the https://kafka.apache.org/24/documentation for more details.
 | <<plugins-{type}s-{plugin}-sasl_jaas_config>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_kerberos_service_name>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_mechanism>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_key>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_proxy>> |<<uri,uri>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_secret>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_url>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-security_protocol>> |<<string,string>>, one of `["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"]`|No
 | <<plugins-{type}s-{plugin}-send_buffer_bytes>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-session_timeout_ms>> |<<number,number>>|No
@@ -302,7 +318,11 @@ before answering the request.
 
 The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
 that happens to be made up of multiple processors. Messages in a topic will be distributed to all
-Logstash instances with the same `group_id`
+Logstash instances with the same `group_id`.
+
+NOTE: In cases when multiple inputs are being used in a single pipeline, reading from different topics,
+it's essential to set a different `group_id => ...` for each input. Setting a unique `client_id => ...`
+is also recommended.
 
 [id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
 ===== `heartbeat_interval_ms` 
@@ -421,7 +441,7 @@ partition ownership amongst consumer instances, supported options are:
 * `sticky`
 * `cooperative_sticky`
 
-These map to Kafka's corresponding https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html[`ConsumerPartitionAssignor`]
+These map to Kafka's corresponding https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html[`ConsumerPartitionAssignor`]
 implementations.
 
 [id="plugins-{type}s-{plugin}-poll_timeout_ms"]
@@ -512,6 +532,44 @@ http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used fo
 This may be any mechanism for which a security provider is available.
 GSSAPI is the default mechanism.
 
+[id="plugins-{type}s-{plugin}-schema_registry_key"]
+===== `schema_registry_key`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Set the username for basic authorization to access remote Schema Registry.
+
+[id="plugins-{type}s-{plugin}-schema_registry_proxy"]
+===== `schema_registry_proxy`
+
+* Value type is <<uri,uri>>
+* There is no default value for this setting.
+
+Set the address of a forward HTTP proxy. An empty string is treated as if proxy was not set.
+
+[id="plugins-{type}s-{plugin}-schema_registry_secret"]
+===== `schema_registry_secret`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Set the password for basic authorization to access remote Schema Registry.
+
+[id="plugins-{type}s-{plugin}-schema_registry_url"]
+===== `schema_registry_url`
+
+* Value type is <<uri,uri>>
+
+The URI that points to an instance of the
+https://docs.confluent.io/current/schema-registry/index.html[Schema Registry] service,
+used to manage Avro schemas. Be sure that the Avro schemas for deserializing the data from
+the specified topics have been uploaded to the Schema Registry service.
+The schemas must follow a naming convention with the pattern <topic name>-value.
+
+Use either the Schema Registry config option or the
+<<plugins-{type}s-{plugin}-value_deserializer_class>> config option, but not both.
+
 [id="plugins-{type}s-{plugin}-security_protocol"]
 ===== `security_protocol` 
 
@@ -625,7 +683,10 @@ The topics configuration will be ignored when using this configuration.
   * Value type is <<string,string>>
   * Default value is `"org.apache.kafka.common.serialization.StringDeserializer"`
 
-Java Class used to deserialize the record's value
+Java Class used to deserialize the record's value.
+A custom value deserializer can be used only if you are not using a Schema Registry.
+Use either the value_deserializer_class config option or the
+<<plugins-{type}s-{plugin}-schema_registry_url>> config option, but not both.
 
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]

data/docs/output-kafka.asciidoc

@@ -1,6 +1,9 @@
+:integration: kafka
 :plugin: kafka
 :type: output
 :default_codec: plain
+:kafka_client: 2.4
+:kafka_client_doc: 24
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -17,15 +20,20 @@ END - GENERATED VARIABLES, DO NOT EDIT!
 
 === Kafka output plugin
 
-include::{include_path}/plugin_header.asciidoc[]
+include::{include_path}/plugin_header-integration.asciidoc[]
 
 ==== Description
 
 Write events to a Kafka topic. 
 
-This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the
+official
+https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
+compatibility reference]. If the linked compatibility wiki is not up-to-date,
+please contact Kafka support/community to confirm compatibility.
 
-If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
+If you require features not yet available in this plugin (including client
+version upgrades), please file an issue with details about what you need.
 
 This output supports connecting to Kafka over:
 
@@ -36,9 +44,12 @@ By default security is disabled but can be turned on as needed.
 
 The only required configuration is the topic_id. 
 
-The default codec is plain. Logstash will encode your events with not only the message field but also with a timestamp and hostname.
+The default codec is plain. Logstash will encode your events with not only the
+message field but also with a timestamp and hostname.
+
+If you want the full content of your events to be sent as json, you should set
+the codec in the output configuration like this:
 
-If you want the full content of your events to be sent as json, you should set the codec in the output configuration like this:
 [source,ruby]
     output {
       kafka {
@@ -47,9 +58,11 @@ If you want the full content of your events to be sent as json, you should set t
       }
     }
     
-For more information see https://kafka.apache.org/24/documentation.html#theproducer
+For more information see
+https://kafka.apache.org/{kafka_client_doc}/documentation.html#theproducer
 
-Kafka producer configuration: https://kafka.apache.org/24/documentation.html#producerconfigs
+Kafka producer configuration:
+https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== Kafka Output Configuration Options
@@ -58,7 +71,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 
 NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
 and might change if Kafka's producer defaults change.
-See the https://kafka.apache.org/24/documentation for more details.
+See the https://kafka.apache.org/{kafka_client_doc}/documentation for more details.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -328,9 +341,9 @@ Kafka down, etc).
 A value less than zero is a configuration error.
 
 Starting with version 10.5.0, this plugin will only retry exceptions that are a subclass of
-https://kafka.apache.org/25/javadoc/org/apache/kafka/common/errors/RetriableException.html[RetriableException]
+https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/common/errors/RetriableException.html[RetriableException]
 and
-https://kafka.apache.org/25/javadoc/org/apache/kafka/common/errors/InterruptException.html[InterruptException].
+https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/common/errors/InterruptException.html[InterruptException].
 If producing a message throws any other exception, an error is logged and the message is dropped without retrying.
 This prevents the Logstash pipeline from hanging indefinitely.
 

data/lib/logstash-integration-kafka_jars.rb

@@ -1,8 +1,17 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('org.apache.kafka', 'kafka-clients', '2.4.1')
-require_jar('com.github.luben', 'zstd-jni', '1.4.3-1')
-require_jar('org.slf4j', 'slf4j-api', '1.7.28')
-require_jar('org.lz4', 'lz4-java', '1.6.0')
+require_jar('io.confluent', 'kafka-avro-serializer', '5.5.1')
+require_jar('io.confluent', 'kafka-schema-serializer', '5.5.1')
+require_jar('io.confluent', 'common-config', '5.5.1')
+require_jar('org.apache.avro', 'avro', '1.9.2')
+require_jar('io.confluent', 'kafka-schema-registry-client', '5.5.1')
+require_jar('org.apache.kafka', 'kafka_2.12', '2.5.1')
+require_jar('io.confluent', 'common-utils', '5.5.1')
+require_jar('javax.ws.rs', 'javax.ws.rs-api', '2.1.1')
+require_jar('org.glassfish.jersey.core', 'jersey-common', '2.30')
+require_jar('org.apache.kafka', 'kafka-clients', '2.5.1')
+require_jar('com.github.luben', 'zstd-jni', '1.4.4-7')
+require_jar('org.slf4j', 'slf4j-api', '1.7.30')
+require_jar('org.lz4', 'lz4-java', '1.7.1')
 require_jar('org.xerial.snappy', 'snappy-java', '1.1.7.3')

data/lib/logstash/inputs/kafka.rb

@@ -3,6 +3,11 @@ require 'logstash/inputs/base'
 require 'stud/interval'
 require 'java'
 require 'logstash-integration-kafka_jars.rb'
+require 'logstash/plugin_mixins/kafka_support'
+require 'manticore'
+require "json"
+require "logstash/json"
+require_relative '../plugin_mixins/common'
 
 # This input will read events from a Kafka topic. It uses the 0.10 version of
 # the consumer API provided by Kafka to read messages from the broker.
@@ -48,6 +53,12 @@ require 'logstash-integration-kafka_jars.rb'
 # Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
 #
 class LogStash::Inputs::Kafka < LogStash::Inputs::Base
+
+  DEFAULT_DESERIALIZER_CLASS = "org.apache.kafka.common.serialization.StringDeserializer"
+
+  include LogStash::PluginMixins::KafkaSupport
+  include ::LogStash::PluginMixins::KafkaAvroSchemaRegistry
+
   config_name 'kafka'
 
   default :codec, 'plain'
@@ -163,7 +174,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # and a rebalance operation is triggered for the group identified by `group_id`
   config :session_timeout_ms, :validate => :number, :default => 10_000 # (10s) Kafka default
   # Java Class used to deserialize the record's value
-  config :value_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
+  config :value_deserializer_class, :validate => :string, :default => DEFAULT_DESERIALIZER_CLASS
   # A list of topics to subscribe to, defaults to ["logstash"].
   config :topics, :validate => :array, :default => ["logstash"]
   # A topic regex pattern to subscribe to. 
@@ -232,11 +243,11 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   #   `timestamp`: The timestamp of this message
   config :decorate_events, :validate => :boolean, :default => false
 
-
   public
   def register
     @runner_threads = []
-  end # def register
+    check_schema_registry_parameters
+  end
 
   public
   def run(logstash_queue)
@@ -274,6 +285,13 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
           for record in records do
             codec_instance.decode(record.value.to_s) do |event|
               decorate(event)
+              if schema_registry_url
+                json = LogStash::Json.load(record.value.to_s)
+                json.each do |k, v|
+                  event.set(k, v)
+                end
+                event.remove("message")
+              end
               if @decorate_events
                 event.set("[@metadata][kafka][topic]", record.topic)
                 event.set("[@metadata][kafka][consumer_group]", @group_id)
@@ -333,7 +351,18 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put(kafka::CLIENT_RACK_CONFIG, client_rack) unless client_rack.nil? 
 
       props.put("security.protocol", security_protocol) unless security_protocol.nil?
-
+      if schema_registry_url
+        props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroDeserializer.java_class)
+        serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+        props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, schema_registry_url.to_s)
+        if schema_registry_proxy && !schema_registry_proxy.empty?
+          props.put(serdes_config::PROXY_HOST, @schema_registry_proxy_host)
+          props.put(serdes_config::PROXY_PORT, @schema_registry_proxy_port)
+        end
+        if schema_registry_key && !schema_registry_key.empty?
+          props.put(serdes_config::USER_INFO_CONFIG, schema_registry_key + ":" + schema_registry_secret.value)
+        end
+      end
       if security_protocol == "SSL"
         set_trustore_keystore_config(props)
       elsif security_protocol == "SASL_PLAINTEXT"
@@ -370,29 +399,4 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
     end
   end
 
-  def set_trustore_keystore_config(props)
-    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-    props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
-    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism", sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
-  end
 end #class LogStash::Inputs::Kafka

data/lib/logstash/outputs/kafka.rb

@@ -2,6 +2,7 @@ require 'logstash/namespace'
 require 'logstash/outputs/base'
 require 'java'
 require 'logstash-integration-kafka_jars.rb'
+require 'logstash/plugin_mixins/kafka_support'
 
 # Write events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on
 # the broker.
@@ -50,6 +51,8 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
 
   java_import org.apache.kafka.clients.producer.ProducerRecord
 
+  include LogStash::PluginMixins::KafkaSupport
+
   declare_threadsafe!
 
   config_name 'kafka'
@@ -389,35 +392,4 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
     end
   end
 
-  def set_trustore_keystore_config(props)
-    unless ssl_endpoint_identification_algorithm.to_s.strip.empty?
-      if ssl_truststore_location.nil?
-        raise LogStash::ConfigurationError, "ssl_truststore_location must be set when SSL is enabled"
-      end
-      props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-      props.put("ssl.truststore.location", ssl_truststore_location)
-      props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-    end
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism",sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
-  end
-
 end #class LogStash::Outputs::Kafka

data/lib/logstash/plugin_mixins/common.rb

@@ -0,0 +1,93 @@
+module LogStash
+  module PluginMixins
+    module KafkaAvroSchemaRegistry
+
+      def self.included(base)
+        base.extend(self)
+        base.setup_schema_registry_config
+      end
+
+      def setup_schema_registry_config
+        # Option to set key to access Schema Registry.
+        config :schema_registry_key, :validate => :string
+
+        # Option to set secret to access Schema Registry.
+        config :schema_registry_secret, :validate => :password
+
+        # Option to set the endpoint of the Schema Registry.
+        # This option permit the usage of Avro Kafka deserializer which retrieve the schema of the Avro message from an
+        # instance of schema registry. If this option has value `value_deserializer_class` nor `topics_pattern` could be valued
+        config :schema_registry_url, :validate => :uri
+
+        # Option to set the proxy of the Schema Registry.
+        # This option permits to define a proxy to be used to reach the schema registry service instance.
+        config :schema_registry_proxy, :validate => :uri
+      end
+
+      def check_schema_registry_parameters
+        if @schema_registry_url
+          check_for_schema_registry_conflicts
+          @schema_registry_proxy_host, @schema_registry_proxy_port  = split_proxy_into_host_and_port(schema_registry_proxy)
+          check_for_key_and_secret
+          check_for_schema_registry_connectivity_and_subjects
+        end
+      end
+
+      private
+      def check_for_schema_registry_conflicts
+        if @value_deserializer_class != LogStash::Inputs::Kafka::DEFAULT_DESERIALIZER_CLASS
+          raise LogStash::ConfigurationError, 'Option schema_registry_url prohibit the customization of value_deserializer_class'
+        end
+        if @topics_pattern && !@topics_pattern.empty?
+         raise LogStash::ConfigurationError, 'Option schema_registry_url prohibit the customization of topics_pattern'
+        end
+      end
+
+      private
+      def check_for_schema_registry_connectivity_and_subjects
+        options = {}
+        if schema_registry_proxy && !schema_registry_proxy.empty?
+          options[:proxy] = schema_registry_proxy.to_s
+        end
+        if schema_registry_key and !schema_registry_key.empty?
+          options[:auth] = {:user => schema_registry_key, :password => schema_registry_secret.value}
+        end
+        client = Manticore::Client.new(options)
+
+        begin
+          response = client.get(@schema_registry_url.to_s + '/subjects').body
+        rescue Manticore::ManticoreException => e
+          raise LogStash::ConfigurationError.new("Schema registry service doesn't respond, error: #{e.message}")
+        end
+        registered_subjects = JSON.parse response
+        expected_subjects = @topics.map { |t| "#{t}-value"}
+        if (expected_subjects & registered_subjects).size != expected_subjects.size
+          undefined_topic_subjects = expected_subjects - registered_subjects
+          raise LogStash::ConfigurationError, "The schema registry does not contain definitions for required topic subjects: #{undefined_topic_subjects}"
+        end
+      end
+
+      def split_proxy_into_host_and_port(proxy_uri)
+        return nil unless proxy_uri && !proxy_uri.empty?
+
+        port = proxy_uri.port
+
+        host_spec = ""
+        host_spec << proxy_uri.scheme || "http"
+        host_spec << "://"
+        host_spec << "#{proxy_uri.userinfo}@" if proxy_uri.userinfo
+        host_spec << proxy_uri.host
+
+        [host_spec, port]
+      end
+
+      def check_for_key_and_secret
+        if schema_registry_key and !schema_registry_key.empty?
+          if !schema_registry_secret or schema_registry_secret.value.empty?
+            raise LogStash::ConfigurationError, "Setting `schema_registry_secret` is required when `schema_registry_key` is provided."
+          end
+        end
+      end
+    end
+  end
+end

data/lib/logstash/plugin_mixins/kafka_support.rb

@@ -0,0 +1,29 @@
+module LogStash module PluginMixins module KafkaSupport
+
+  def set_trustore_keystore_config(props)
+    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
+    props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
+    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
+
+    # Client auth stuff
+    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
+    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
+    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
+    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
+    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
+  end
+
+  def set_sasl_config(props)
+    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
+    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
+
+    props.put("sasl.mechanism", sasl_mechanism)
+    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
+      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
+    end
+
+    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
+    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
+  end
+
+end end end

data/logstash-integration-kafka.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-integration-kafka'
-  s.version         = '10.5.0'
+  s.version         = '10.7.0'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Integration with Kafka - input and output plugins"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+
@@ -46,6 +46,7 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'logstash-codec-json'
   s.add_runtime_dependency 'logstash-codec-plain'
   s.add_runtime_dependency 'stud', '>= 0.0.22', '< 0.1.0'
+  s.add_runtime_dependency "manticore", '>= 0.5.4', '< 1.0.0'
 
   s.add_development_dependency 'logstash-devutils'
   s.add_development_dependency 'rspec-wait'

data/spec/integration/inputs/kafka_spec.rb

@@ -2,6 +2,9 @@
 require "logstash/devutils/rspec/spec_helper"
 require "logstash/inputs/kafka"
 require "rspec/wait"
+require "stud/try"
+require "manticore"
+require "json"
 
 # Please run kafka_test_setup.sh prior to executing this integration test.
 describe "inputs/kafka", :integration => true do
@@ -120,20 +123,192 @@ describe "inputs/kafka", :integration => true do
       end
     end
   end
+end
+
+private
+
+def consume_messages(config, queue: Queue.new, timeout:, event_count:)
+  kafka_input = LogStash::Inputs::Kafka.new(config)
+  t = Thread.new { kafka_input.run(queue) }
+  begin
+    t.run
+    wait(timeout).for { queue.length }.to eq(event_count) unless timeout.eql?(false)
+    block_given? ? yield(queue, kafka_input) : queue
+  ensure
+    t.kill
+    t.join(30_000)
+  end
+end
+
+
+describe "schema registry connection options" do
+  context "remote endpoint validation" do
+    it "should fail if not reachable" do
+      config = {'schema_registry_url' => 'http://localnothost:8081'}
+      kafka_input = LogStash::Inputs::Kafka.new(config)
+      expect { kafka_input.register }.to raise_error LogStash::ConfigurationError, /Schema registry service doesn't respond.*/
+    end
+
+    it "should fail if any topic is not matched by a subject on the schema registry" do
+      config = {
+        'schema_registry_url' => 'http://localhost:8081',
+        'topics' => ['temperature_stream']
+      }
+
+      kafka_input = LogStash::Inputs::Kafka.new(config)
+      expect { kafka_input.register }.to raise_error LogStash::ConfigurationError, /The schema registry does not contain definitions for required topic subjects: \["temperature_stream-value"\]/
+    end
+
+    context "register with subject present" do
+      SUBJECT_NAME = "temperature_stream-value"
+
+      before(:each) do
+        response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), SUBJECT_NAME)
+        expect( response.code ).to be(200)
+      end
 
-  private
+      after(:each) do
+        schema_registry_client = Manticore::Client.new
+        delete_remote_schema(schema_registry_client, SUBJECT_NAME)
+      end
 
-  def consume_messages(config, queue: Queue.new, timeout:, event_count:)
-    kafka_input = LogStash::Inputs::Kafka.new(config)
-    t = Thread.new { kafka_input.run(queue) }
-    begin
-      t.run
-      wait(timeout).for { queue.length }.to eq(event_count) unless timeout.eql?(false)
-      block_given? ? yield(queue, kafka_input) : queue
-    ensure
-      t.kill
-      t.join(30_000)
+      it "should correctly complete registration phase" do
+        config = {
+          'schema_registry_url' => 'http://localhost:8081',
+          'topics' => ['temperature_stream']
+        }
+        kafka_input = LogStash::Inputs::Kafka.new(config)
+        kafka_input.register
+      end
     end
   end
+end
 
+def save_avro_schema_to_schema_registry(schema_file, subject_name)
+  raw_schema = File.readlines(schema_file).map(&:chomp).join
+  raw_schema_quoted = raw_schema.gsub('"', '\"')
+  response = Manticore.post("http://localhost:8081/subjects/#{subject_name}/versions",
+          body: '{"schema": "' + raw_schema_quoted + '"}',
+          headers: {"Content-Type" => "application/vnd.schemaregistry.v1+json"})
+  response
 end
+
+def delete_remote_schema(schema_registry_client, subject_name)
+  expect(schema_registry_client.delete("http://localhost:8081/subjects/#{subject_name}").code ).to be(200)
+  expect(schema_registry_client.delete("http://localhost:8081/subjects/#{subject_name}?permanent=true").code ).to be(200)
+end
+
+# AdminClientConfig = org.alpache.kafka.clients.admin.AdminClientConfig
+
+describe "Schema registry API", :integration => true do
+
+  let(:schema_registry) { Manticore::Client.new }
+
+  context 'listing subject on clean instance' do
+    it "should return an empty set" do
+      subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+      expect( subjects ).to be_empty
+    end
+  end
+
+  context 'send a schema definition' do
+    it "save the definition" do
+      response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), "schema_test_1")
+      expect( response.code ).to be(200)
+      delete_remote_schema(schema_registry, "schema_test_1")
+    end
+
+    it "delete the schema just added" do
+      response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), "schema_test_1")
+      expect( response.code ).to be(200)
+
+      expect( schema_registry.delete('http://localhost:8081/subjects/schema_test_1?permanent=false').code ).to be(200)
+      sleep(1)
+      subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+      expect( subjects ).to be_empty
+    end
+  end
+
+  context 'use the schema to serialize' do
+    after(:each) do
+      expect( schema_registry.delete('http://localhost:8081/subjects/topic_avro-value').code ).to be(200)
+      sleep 1
+      expect( schema_registry.delete('http://localhost:8081/subjects/topic_avro-value?permanent=true').code ).to be(200)
+
+      Stud.try(3.times, [StandardError, RSpec::Expectations::ExpectationNotMetError]) do
+        wait(10).for do
+          subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+          subjects.empty?
+        end.to be_truthy
+      end
+    end
+
+    let(:group_id_1) {rand(36**8).to_s(36)}
+
+    let(:avro_topic_name) { "topic_avro" }
+
+    let(:plain_config) do
+      { 'schema_registry_url' => 'http://localhost:8081',
+        'topics' => [avro_topic_name],
+        'codec' => 'plain',
+        'group_id' => group_id_1,
+        'auto_offset_reset' => 'earliest' }
+    end
+
+    def delete_topic_if_exists(topic_name)
+      props = java.util.Properties.new
+      props.put(Java::org.apache.kafka.clients.admin.AdminClientConfig::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
+
+      admin_client = org.apache.kafka.clients.admin.AdminClient.create(props)
+      topics_list = admin_client.listTopics().names().get()
+      if topics_list.contains(topic_name)
+        result = admin_client.deleteTopics([topic_name])
+        result.values.get(topic_name).get()
+      end
+    end
+
+    def write_some_data_to(topic_name)
+      props = java.util.Properties.new
+      config = org.apache.kafka.clients.producer.ProducerConfig
+
+      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+      props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081")
+
+      props.put(config::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
+      props.put(config::KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer.java_class)
+      props.put(config::VALUE_SERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroSerializer.java_class)
+
+      parser = org.apache.avro.Schema::Parser.new()
+      user_schema = '''{"type":"record",
+                        "name":"myrecord",
+                        "fields":[
+                            {"name":"str_field", "type": "string"},
+                            {"name":"map_field", "type": {"type": "map", "values": "string"}}
+                        ]}'''
+      schema = parser.parse(user_schema)
+      avro_record = org.apache.avro.generic.GenericData::Record.new(schema)
+      avro_record.put("str_field", "value1")
+      avro_record.put("map_field", {"inner_field" => "inner value"})
+
+      producer = org.apache.kafka.clients.producer.KafkaProducer.new(props)
+      record = org.apache.kafka.clients.producer.ProducerRecord.new(topic_name, "avro_key", avro_record)
+      producer.send(record)
+    end
+
+    it "stored a new schema using Avro Kafka serdes" do
+      delete_topic_if_exists avro_topic_name
+      write_some_data_to avro_topic_name
+
+      subjects = JSON.parse schema_registry.get('http://localhost:8081/subjects').body
+      expect( subjects ).to contain_exactly("topic_avro-value")
+
+      num_events = 1
+      queue = consume_messages(plain_config, timeout: 30, event_count: num_events)
+      expect(queue.length).to eq(num_events)
+      elem = queue.pop
+      expect( elem.to_hash).not_to include("message")
+      expect( elem.get("str_field") ).to eq("value1")
+      expect( elem.get("map_field")["inner_field"] ).to eq("inner value")
+    end
+  end
+end

data/spec/unit/inputs/avro_schema_fixture_payment.asvc

@@ -0,0 +1,8 @@
+{"namespace": "io.confluent.examples.clients.basicavro",
+ "type": "record",
+ "name": "Payment",
+ "fields": [
+     {"name": "id", "type": "string"},
+     {"name": "amount", "type": "double"}
+ ]
+}

data/spec/unit/inputs/kafka_spec.rb

@@ -37,6 +37,22 @@ describe LogStash::Inputs::Kafka do
     expect { subject.register }.to_not raise_error
   end
 
+  context "register parameter verification" do
+    let(:config) do
+      { 'schema_registry_url' => 'http://localhost:8081', 'topics' => ['logstash'], 'consumer_threads' => 4 }
+    end
+
+    it "schema_registry_url conflict with value_deserializer_class should fail" do
+      config['value_deserializer_class'] = 'my.fantasy.Deserializer'
+      expect { subject.register }.to raise_error LogStash::ConfigurationError, /Option schema_registry_url prohibit the customization of value_deserializer_class/
+    end
+
+    it "schema_registry_url conflict with topics_pattern should fail" do
+      config['topics_pattern'] = 'topic_.*'
+      expect { subject.register }.to raise_error LogStash::ConfigurationError, /Option schema_registry_url prohibit the customization of topics_pattern/
+    end
+  end
+
   context 'with client_rack' do
     let(:config) { super.merge('client_rack' => 'EU-R1') }
 

data/spec/unit/outputs/kafka_spec.rb

@@ -50,9 +50,10 @@ describe "outputs/kafka" do
       kafka.multi_receive([event])
     end
 
-    it 'should raise config error when truststore location is not set and ssl is enabled' do
+    it 'should not raise config error when truststore location is not set and ssl is enabled' do
       kafka = LogStash::Outputs::Kafka.new(simple_kafka_config.merge("security_protocol" => "SSL"))
-      expect { kafka.register }.to raise_error(LogStash::ConfigurationError, /ssl_truststore_location must be set when SSL is enabled/)
+      expect(org.apache.kafka.clients.producer.KafkaProducer).to receive(:new)
+      expect { kafka.register }.to_not raise_error
     end
   end
 
@@ -225,21 +226,31 @@ describe "outputs/kafka" do
   context 'when ssl endpoint identification disabled' do
 
     let(:config) do
-      simple_kafka_config.merge('ssl_endpoint_identification_algorithm' => '', 'security_protocol' => 'SSL')
+      simple_kafka_config.merge(
+          'security_protocol' => 'SSL',
+          'ssl_endpoint_identification_algorithm' => '',
+          'ssl_truststore_location' => truststore_path,
+      )
+    end
+
+    let(:truststore_path) do
+      File.join(File.dirname(__FILE__), '../../fixtures/trust-store_stub.jks')
     end
 
     subject { LogStash::Outputs::Kafka.new(config) }
 
-    it 'does not configure truststore' do
+    it 'sets empty ssl.endpoint.identification.algorithm' do
       expect(org.apache.kafka.clients.producer.KafkaProducer).
-          to receive(:new).with(hash_excluding('ssl.truststore.location' => anything))
+          to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
       subject.register
     end
 
-    it 'sets empty ssl.endpoint.identification.algorithm' do
+    it 'configures truststore' do
       expect(org.apache.kafka.clients.producer.KafkaProducer).
-          to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
+          to receive(:new).with(hash_including('ssl.truststore.location' => truststore_path))
       subject.register
     end
+
   end
+
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-integration-kafka
 version: !ruby/object:Gem::Version
-  version: 10.5.0
+  version: 10.7.0
 platform: java
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-07-20 00:00:00.000000000 Z
+date: 2020-12-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -106,6 +106,26 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: 0.1.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.4
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+  name: manticore
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.4
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 1.0.0
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
@@ -183,15 +203,28 @@ files:
 - lib/logstash-integration-kafka_jars.rb
 - lib/logstash/inputs/kafka.rb
 - lib/logstash/outputs/kafka.rb
+- lib/logstash/plugin_mixins/common.rb
+- lib/logstash/plugin_mixins/kafka_support.rb
 - logstash-integration-kafka.gemspec
+- spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
 - spec/integration/outputs/kafka_spec.rb
+- spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb
-- vendor/jar-dependencies/com/github/luben/zstd-jni/1.4.3-1/zstd-jni-1.4.3-1.jar
-- vendor/jar-dependencies/org/apache/kafka/kafka-clients/2.4.1/kafka-clients-2.4.1.jar
-- vendor/jar-dependencies/org/lz4/lz4-java/1.6.0/lz4-java-1.6.0.jar
-- vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar
+- vendor/jar-dependencies/com/github/luben/zstd-jni/1.4.4-7/zstd-jni-1.4.4-7.jar
+- vendor/jar-dependencies/io/confluent/common-config/5.5.1/common-config-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/common-utils/5.5.1/common-utils-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-avro-serializer/5.5.1/kafka-avro-serializer-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-registry-client/5.5.1/kafka-schema-registry-client-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-serializer/5.5.1/kafka-schema-serializer-5.5.1.jar
+- vendor/jar-dependencies/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.jar
+- vendor/jar-dependencies/org/apache/avro/avro/1.9.2/avro-1.9.2.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka-clients/2.5.1/kafka-clients-2.5.1.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka_2.12/2.5.1/kafka_2.12-2.5.1.jar
+- vendor/jar-dependencies/org/glassfish/jersey/core/jersey-common/2.30/jersey-common-2.30.jar
+- vendor/jar-dependencies/org/lz4/lz4-java/1.7.1/lz4-java-1.7.1.jar
+- vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar
 - vendor/jar-dependencies/org/xerial/snappy/snappy-java/1.1.7.3/snappy-java-1.1.7.3.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
@@ -222,7 +255,9 @@ signing_key:
 specification_version: 4
 summary: Integration with Kafka - input and output plugins
 test_files:
+- spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
 - spec/integration/outputs/kafka_spec.rb
+- spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb