logstash-integration-kafka 10.4.0-java → 10.6.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6ebbcd2d18d130e9fac997330c3c4b4bd9a959a982fe83215762b03638497ba4
-  data.tar.gz: 2b54ba231d9f74344a5ec321e0dcdec256ea5664001e7c3dc0323b2150761e30
+  metadata.gz: 1d8b40d779e91e9c05dece65249660ab5c272b6833658cbca51b977d92936f42
+  data.tar.gz: 58323e216be645aede9f0b49c27824958b6627485bdf79a6774cd5f87b818245
 SHA512:
-  metadata.gz: a8c2aa5c2123fa001f58fc3670bd90face614fed72cf24f17ad645ba4de3bd689923d51ba5b5dd3a9201507657f6ed54326ed48495d274bb7c2284525470bdf7
-  data.tar.gz: cebe4abeda34edd6d5d1872c96f1b119abfa7abb2e40c52fb061e2c0953789441223e4b5a93a0d2fd7e3de1918c592dce9f5fee91bb6713b0e16f167033c13ce
+  metadata.gz: 0f05eec028758745a2ab04b90d721128c088c46d0bb9a01923c389118c99a718a561d2ca2420fc2e206e4cf75e4e3695e1704a17d9eff9f4f18e66da3d3ccb85
+  data.tar.gz: 85e117a64d14d013674869ccadfb5487a04991ec83c6a9d6874496b862be200bf4e7203d3ac4fc779d67131ecbe82212fafd66884114337d17ed33dda7ad0963

data/CHANGELOG.md

@@ -1,8 +1,31 @@
+## 10.6.0
+  - Added functionality to Kafka input to use Avro deserializer in retrieving data from Kafka. The schema is retrieved
+    from an instance of Confluent's Schema Registry service [#51](https://github.com/logstash-plugins/logstash-integration-kafka/pull/51)
+     
+## 10.5.3
+  - Fix: set (optional) truststore when endpoint id check disabled [#60](https://github.com/logstash-plugins/logstash-integration-kafka/pull/60).
+    Since **10.1.0** disabling server host-name verification (`ssl_endpoint_identification_algorithm => ""`) did not allow 
+    the (output) plugin to set `ssl_truststore_location => "..."`.
+
+## 10.5.2
+  - Docs: explain group_id in case of multiple inputs [#59](https://github.com/logstash-plugins/logstash-integration-kafka/pull/59)
+
+## 10.5.1
+  - [DOC]Replaced plugin_header file with plugin_header-integration file. [#46](https://github.com/logstash-plugins/logstash-integration-kafka/pull/46)
+  - [DOC]Update kafka client version across kafka integration docs [#47](https://github.com/logstash-plugins/logstash-integration-kafka/pull/47)
+  - [DOC]Replace hard-coded kafka client and doc path version numbers with attributes to simplify doc maintenance [#48](https://github.com/logstash-plugins/logstash-integration-kafka/pull/48)  
+
+## 10.5.0
+  - Changed: retry sending messages only for retriable exceptions [#27](https://github.com/logstash-plugins/logstash-integration-kafka/pull/29)
+
+## 10.4.1
+  - [DOC] Fixed formatting issues and made minor content edits [#43](https://github.com/logstash-plugins/logstash-integration-kafka/pull/43)
+
 ## 10.4.0
  - added the input `isolation_level` to allow fine control of whether to return transactional messages [#44](https://github.com/logstash-plugins/logstash-integration-kafka/pull/44)
 
 ## 10.3.0
-  - added the input and output `client_dns_lookup` parameter to allow control of how DNS requests are made
+  - added the input and output `client_dns_lookup` parameter to allow control of how DNS requests are made [#28](https://github.com/logstash-plugins/logstash-integration-kafka/pull/28)
 
 ## 10.2.0
   - Changed: config defaults to be aligned with Kafka client defaults [#30](https://github.com/logstash-plugins/logstash-integration-kafka/pull/30)

data/docs/index.asciidoc

@@ -1,6 +1,7 @@
 :plugin: kafka
 :type: integration
 :no_codec:
+:kafka_client: 2.4
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -21,11 +22,15 @@ include::{include_path}/plugin_header.asciidoc[]
 
 ==== Description
 
-The Kafka Integration Plugin provides integrated plugins for working with the https://kafka.apache.org/[Kafka] distributed streaming platform.
+The Kafka Integration Plugin provides integrated plugins for working with the
+https://kafka.apache.org/[Kafka] distributed streaming platform.
 
  - {logstash-ref}/plugins-inputs-kafka.html[Kafka Input Plugin]
  - {logstash-ref}/plugins-outputs-kafka.html[Kafka Output Plugin]
  
-This plugin uses Kafka Client 2.4. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the official
+https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
+compatibility reference]. If the linked compatibility wiki is not up-to-date,
+please contact Kafka support/community to confirm compatibility.
 
 :no_codec!:

data/docs/input-kafka.asciidoc

@@ -1,6 +1,9 @@
+:integration: kafka
 :plugin: kafka
 :type: input
 :default_codec: plain
+:kafka_client: 2.4
+:kafka_client_doc: 24
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -17,15 +20,20 @@ END - GENERATED VARIABLES, DO NOT EDIT!
 
 === Kafka input plugin
 
-include::{include_path}/plugin_header.asciidoc[]
+include::{include_path}/plugin_header-integration.asciidoc[]
 
 ==== Description
 
 This input will read events from a Kafka topic.
 
-This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the
+official
+https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
+compatibility reference]. If the linked compatibility wiki is not up-to-date,
+please contact Kafka support/community to confirm compatibility.
 
-If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
+If you require features not yet available in this plugin (including client
+version upgrades), please file an issue with details about what you need.
 
 This input supports connecting to Kafka over:
 
@@ -46,9 +54,9 @@ the same `group_id`.
 Ideally you should have as many threads as the number of partitions for a perfect balance --
 more threads than partitions means that some threads will be idle
 
-For more information see https://kafka.apache.org/24/documentation.html#theconsumer
+For more information see https://kafka.apache.org/{kafka_client_doc}/documentation.html#theconsumer
 
-Kafka consumer configuration: https://kafka.apache.org/24/documentation.html#consumerconfigs
+Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/documentation.html#consumerconfigs
 
 ==== Metadata fields
 
@@ -59,7 +67,11 @@ The following metadata from Kafka broker are added under the `[@metadata]` field
 * `[@metadata][kafka][partition]`: Partition info for this message.
 * `[@metadata][kafka][offset]`: Original record offset for this message.
 * `[@metadata][kafka][key]`: Record key, if any.
-* `[@metadata][kafka][timestamp]`: Timestamp in the Record. Depending on your broker configuration, this can be either when the record was created (default) or when it was received by the broker. See more about property log.message.timestamp.type at https://kafka.apache.org/10/documentation.html#brokerconfigs
+* `[@metadata][kafka][timestamp]`: Timestamp in the Record. 
+Depending on your broker configuration, this can be
+either when the record was created (default) or when it was received by the
+broker. See more about property log.message.timestamp.type at
+https://kafka.apache.org/{kafka_client_doc}/documentation.html#brokerconfigs
 
 Metadata is only added to the event if the `decorate_events` option is set to true (it defaults to false).
 
@@ -73,7 +85,7 @@ This plugin supports these configuration options plus the <<plugins-{type}s-{plu
 
 NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
 and might change if Kafka's consumer defaults change.
-See the https://kafka.apache.org/24/documentation for more details.
+See the https://kafka.apache.org/{kafka_client_doc}/documentation for more details.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -112,6 +124,10 @@ See the https://kafka.apache.org/24/documentation for more details.
 | <<plugins-{type}s-{plugin}-sasl_jaas_config>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_kerberos_service_name>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_mechanism>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_key>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_proxy>> |<<uri,uri>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_secret>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-schema_registry_url>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-security_protocol>> |<<string,string>>, one of `["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"]`|No
 | <<plugins-{type}s-{plugin}-send_buffer_bytes>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-session_timeout_ms>> |<<number,number>>|No
@@ -302,7 +318,11 @@ before answering the request.
 
 The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
 that happens to be made up of multiple processors. Messages in a topic will be distributed to all
-Logstash instances with the same `group_id`
+Logstash instances with the same `group_id`.
+
+NOTE: In cases when multiple inputs are being used in a single pipeline, reading from different topics,
+it's essential to set a different `group_id => ...` for each input. Setting a unique `client_id => ...`
+is also recommended.
 
 [id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
 ===== `heartbeat_interval_ms` 
@@ -421,7 +441,7 @@ partition ownership amongst consumer instances, supported options are:
 * `sticky`
 * `cooperative_sticky`
 
-These map to Kafka's corresponding https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html[`ConsumerPartitionAssignor`]
+These map to Kafka's corresponding https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html[`ConsumerPartitionAssignor`]
 implementations.
 
 [id="plugins-{type}s-{plugin}-poll_timeout_ms"]
@@ -512,6 +532,44 @@ http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used fo
 This may be any mechanism for which a security provider is available.
 GSSAPI is the default mechanism.
 
+[id="plugins-{type}s-{plugin}-schema_registry_key"]
+===== `schema_registry_key`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Set the username for basic authorization to access remote Schema Registry.
+
+[id="plugins-{type}s-{plugin}-schema_registry_proxy"]
+===== `schema_registry_proxy`
+
+* Value type is <<uri,uri>>
+* There is no default value for this setting.
+
+Set the address of a forward HTTP proxy. An empty string is treated as if proxy was not set.
+
+[id="plugins-{type}s-{plugin}-schema_registry_secret"]
+===== `schema_registry_secret`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Set the password for basic authorization to access remote Schema Registry.
+
+[id="plugins-{type}s-{plugin}-schema_registry_url"]
+===== `schema_registry_url`
+
+* Value type is <<uri,uri>>
+
+The URI that points to an instance of the
+https://docs.confluent.io/current/schema-registry/index.html[Schema Registry] service,
+used to manage Avro schemas. Be sure that the Avro schemas for deserializing the data from
+the specified topics have been uploaded to the Schema Registry service.
+The schemas must follow a naming convention with the pattern <topic name>-value.
+
+Use either the Schema Registry config option or the
+<<plugins-{type}s-{plugin}-value_deserializer_class>> config option, but not both.
+
 [id="plugins-{type}s-{plugin}-security_protocol"]
 ===== `security_protocol` 
 
@@ -625,7 +683,10 @@ The topics configuration will be ignored when using this configuration.
   * Value type is <<string,string>>
   * Default value is `"org.apache.kafka.common.serialization.StringDeserializer"`
 
-Java Class used to deserialize the record's value
+Java Class used to deserialize the record's value.
+A custom value deserializer can be used only if you are not using a Schema Registry.
+Use either the value_deserializer_class config option or the
+<<plugins-{type}s-{plugin}-schema_registry_url>> config option, but not both.
 
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]

data/docs/output-kafka.asciidoc

@@ -1,6 +1,9 @@
+:integration: kafka
 :plugin: kafka
 :type: output
 :default_codec: plain
+:kafka_client: 2.4
+:kafka_client_doc: 24
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -17,15 +20,20 @@ END - GENERATED VARIABLES, DO NOT EDIT!
 
 === Kafka output plugin
 
-include::{include_path}/plugin_header.asciidoc[]
+include::{include_path}/plugin_header-integration.asciidoc[]
 
 ==== Description
 
 Write events to a Kafka topic. 
 
-This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the
+official
+https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
+compatibility reference]. If the linked compatibility wiki is not up-to-date,
+please contact Kafka support/community to confirm compatibility.
 
-If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
+If you require features not yet available in this plugin (including client
+version upgrades), please file an issue with details about what you need.
 
 This output supports connecting to Kafka over:
 
@@ -36,9 +44,12 @@ By default security is disabled but can be turned on as needed.
 
 The only required configuration is the topic_id. 
 
-The default codec is plain. Logstash will encode your events with not only the message field but also with a timestamp and hostname.
+The default codec is plain. Logstash will encode your events with not only the
+message field but also with a timestamp and hostname.
+
+If you want the full content of your events to be sent as json, you should set
+the codec in the output configuration like this:
 
-If you want the full content of your events to be sent as json, you should set the codec in the output configuration like this:
 [source,ruby]
     output {
       kafka {
@@ -47,9 +58,11 @@ If you want the full content of your events to be sent as json, you should set t
       }
     }
     
-For more information see https://kafka.apache.org/24/documentation.html#theproducer
+For more information see
+https://kafka.apache.org/{kafka_client_doc}/documentation.html#theproducer
 
-Kafka producer configuration: https://kafka.apache.org/24/documentation.html#producerconfigs
+Kafka producer configuration:
+https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== Kafka Output Configuration Options
@@ -58,7 +71,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 
 NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
 and might change if Kafka's producer defaults change.
-See the https://kafka.apache.org/24/documentation for more details.
+See the https://kafka.apache.org/{kafka_client_doc}/documentation for more details.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -115,10 +128,13 @@ output plugins.
 The number of acknowledgments the producer requires the leader to have received
 before considering a request complete.
 
-acks=0,   the producer will not wait for any acknowledgment from the server at all.
-acks=1,   This will mean the leader will write the record to its local log but
-          will respond without awaiting full acknowledgement from all followers.
-acks=all, This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
+`acks=0`. The producer will not wait for any acknowledgment from the server.
+
+`acks=1`. The leader will write the record to its local log, but will respond
+without waiting for full acknowledgement from all followers.
+          
+`acks=all`. The leader will wait for the full set of in-sync replicas before
+acknowledging the record.
 
 [id="plugins-{type}s-{plugin}-batch_size"]
 ===== `batch_size` 
@@ -154,11 +170,12 @@ The total bytes of memory the producer can use to buffer records waiting to be s
 ===== `client_dns_lookup` 
 
   * Value type is <<string,string>>
+  * Valid options are `use_all_dns_ips`, `resolve_canonical_bootstrap_servers_only`, `default`
   * Default value is `"default"`
 
-How DNS lookups should be done. If set to `use_all_dns_ips`, when the lookup returns multiple 
-IP addresses for a hostname, they will all be attempted to connect to before failing the 
-connection. If the value is `resolve_canonical_bootstrap_servers_only` each entry will be 
+Controls how DNS lookups are done. If set to `use_all_dns_ips`, Logstash tries
+all IP addresses returned for a hostname before failing the connection.
+If set to `resolve_canonical_bootstrap_servers_only`, each entry will be 
 resolved and expanded into a list of canonical names.
 
 [id="plugins-{type}s-{plugin}-client_id"]
@@ -178,7 +195,7 @@ ip/port by allowing a logical application name to be included with the request
   * Default value is `"none"`
 
 The compression type for all data generated by the producer.
-The default is none (i.e. no compression). Valid values are none, gzip, or snappy.
+The default is none (i.e. no compression). Valid values are none, gzip, snappy, or lz4.
 
 [id="plugins-{type}s-{plugin}-jaas_path"]
 ===== `jaas_path` 
@@ -323,6 +340,15 @@ Kafka down, etc).
 
 A value less than zero is a configuration error.
 
+Starting with version 10.5.0, this plugin will only retry exceptions that are a subclass of
+https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/common/errors/RetriableException.html[RetriableException]
+and
+https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/common/errors/InterruptException.html[InterruptException].
+If producing a message throws any other exception, an error is logged and the message is dropped without retrying.
+This prevents the Logstash pipeline from hanging indefinitely.
+
+In versions prior to 10.5.0, any exception is retried indefinitely unless the `retries` option is configured.
+
 [id="plugins-{type}s-{plugin}-retry_backoff_ms"]
 ===== `retry_backoff_ms` 
 

data/lib/logstash-integration-kafka_jars.rb

@@ -1,8 +1,17 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('org.apache.kafka', 'kafka-clients', '2.4.1')
-require_jar('com.github.luben', 'zstd-jni', '1.4.3-1')
-require_jar('org.slf4j', 'slf4j-api', '1.7.28')
-require_jar('org.lz4', 'lz4-java', '1.6.0')
+require_jar('io.confluent', 'kafka-avro-serializer', '5.5.1')
+require_jar('io.confluent', 'kafka-schema-serializer', '5.5.1')
+require_jar('io.confluent', 'common-config', '5.5.1')
+require_jar('org.apache.avro', 'avro', '1.9.2')
+require_jar('io.confluent', 'kafka-schema-registry-client', '5.5.1')
+require_jar('org.apache.kafka', 'kafka_2.12', '2.5.1')
+require_jar('io.confluent', 'common-utils', '5.5.1')
+require_jar('javax.ws.rs', 'javax.ws.rs-api', '2.1.1')
+require_jar('org.glassfish.jersey.core', 'jersey-common', '2.30')
+require_jar('org.apache.kafka', 'kafka-clients', '2.5.1')
+require_jar('com.github.luben', 'zstd-jni', '1.4.4-7')
+require_jar('org.slf4j', 'slf4j-api', '1.7.30')
+require_jar('org.lz4', 'lz4-java', '1.7.1')
 require_jar('org.xerial.snappy', 'snappy-java', '1.1.7.3')

data/lib/logstash/inputs/kafka.rb

@@ -3,6 +3,11 @@ require 'logstash/inputs/base'
 require 'stud/interval'
 require 'java'
 require 'logstash-integration-kafka_jars.rb'
+require 'logstash/plugin_mixins/kafka_support'
+require "faraday"
+require "json"
+require "logstash/json"
+require_relative '../plugin_mixins/common'
 
 # This input will read events from a Kafka topic. It uses the 0.10 version of
 # the consumer API provided by Kafka to read messages from the broker.
@@ -48,6 +53,12 @@ require 'logstash-integration-kafka_jars.rb'
 # Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
 #
 class LogStash::Inputs::Kafka < LogStash::Inputs::Base
+
+  DEFAULT_DESERIALIZER_CLASS = "org.apache.kafka.common.serialization.StringDeserializer"
+
+  include LogStash::PluginMixins::KafkaSupport
+  include ::LogStash::PluginMixins::KafkaAvroSchemaRegistry
+
   config_name 'kafka'
 
   default :codec, 'plain'
@@ -163,7 +174,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # and a rebalance operation is triggered for the group identified by `group_id`
   config :session_timeout_ms, :validate => :number, :default => 10_000 # (10s) Kafka default
   # Java Class used to deserialize the record's value
-  config :value_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
+  config :value_deserializer_class, :validate => :string, :default => DEFAULT_DESERIALIZER_CLASS
   # A list of topics to subscribe to, defaults to ["logstash"].
   config :topics, :validate => :array, :default => ["logstash"]
   # A topic regex pattern to subscribe to. 
@@ -232,11 +243,11 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   #   `timestamp`: The timestamp of this message
   config :decorate_events, :validate => :boolean, :default => false
 
-
   public
   def register
     @runner_threads = []
-  end # def register
+    check_schema_registry_parameters
+  end
 
   public
   def run(logstash_queue)
@@ -274,6 +285,13 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
           for record in records do
             codec_instance.decode(record.value.to_s) do |event|
               decorate(event)
+              if schema_registry_url
+                json = LogStash::Json.load(record.value.to_s)
+                json.each do |k, v|
+                  event.set(k, v)
+                end
+                event.remove("message")
+              end
               if @decorate_events
                 event.set("[@metadata][kafka][topic]", record.topic)
                 event.set("[@metadata][kafka][consumer_group]", @group_id)
@@ -333,7 +351,18 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put(kafka::CLIENT_RACK_CONFIG, client_rack) unless client_rack.nil? 
 
       props.put("security.protocol", security_protocol) unless security_protocol.nil?
-
+      if schema_registry_url
+        props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroDeserializer.java_class)
+        serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+        props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, schema_registry_url.to_s)
+        if schema_registry_proxy && !schema_registry_proxy.empty?
+          props.put(serdes_config::PROXY_HOST, @schema_registry_proxy_host)
+          props.put(serdes_config::PROXY_PORT, @schema_registry_proxy_port)
+        end
+        if schema_registry_key && !schema_registry_key.empty?
+          props.put(serdes_config::USER_INFO_CONFIG, schema_registry_key + ":" + schema_registry_secret.value)
+        end
+      end
       if security_protocol == "SSL"
         set_trustore_keystore_config(props)
       elsif security_protocol == "SASL_PLAINTEXT"
@@ -370,29 +399,4 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
     end
   end
 
-  def set_trustore_keystore_config(props)
-    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-    props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
-    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism", sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
-  end
 end #class LogStash::Inputs::Kafka

data/lib/logstash/outputs/kafka.rb

@@ -2,6 +2,7 @@ require 'logstash/namespace'
 require 'logstash/outputs/base'
 require 'java'
 require 'logstash-integration-kafka_jars.rb'
+require 'logstash/plugin_mixins/kafka_support'
 
 # Write events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on
 # the broker.
@@ -50,6 +51,8 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
 
   java_import org.apache.kafka.clients.producer.ProducerRecord
 
+  include LogStash::PluginMixins::KafkaSupport
+
   declare_threadsafe!
 
   config_name 'kafka'
@@ -236,7 +239,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
     remaining = @retries
 
     while batch.any?
-      if !remaining.nil?
+      unless remaining.nil?
         if remaining < 0
           # TODO(sissel): Offer to DLQ? Then again, if it's a transient fault,
           # DLQing would make things worse (you dlq data that would be successful
@@ -255,27 +258,39 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
         begin
           # send() can throw an exception even before the future is created.
           @producer.send(record)
-        rescue org.apache.kafka.common.errors.TimeoutException => e
-          failures << record
-          nil
-        rescue org.apache.kafka.common.errors.InterruptException => e
+        rescue org.apache.kafka.common.errors.InterruptException,
+               org.apache.kafka.common.errors.RetriableException => e
+          logger.info("producer send failed, will retry sending", :exception => e.class, :message => e.message)
           failures << record
           nil
-        rescue org.apache.kafka.common.errors.SerializationException => e
-          # TODO(sissel): Retrying will fail because the data itself has a problem serializing.
-          # TODO(sissel): Let's add DLQ here.
-          failures << record
+        rescue org.apache.kafka.common.KafkaException => e
+          # This error is not retriable, drop event
+          # TODO: add DLQ support
+          logger.warn("producer send failed, dropping record",:exception => e.class, :message => e.message,
+                      :record_value => record.value)
           nil
         end
-      end.compact
+      end
 
       futures.each_with_index do |future, i|
-        begin
-          result = future.get()
-        rescue => e
-          # TODO(sissel): Add metric to count failures, possibly by exception type.
-          logger.warn("producer send failed", :exception => e.class, :message => e.message)
-          failures << batch[i]
+        # We cannot skip nils using `futures.compact` because then our index `i` will not align with `batch`
+        unless future.nil?
+          begin
+            future.get
+          rescue java.util.concurrent.ExecutionException => e
+            # TODO(sissel): Add metric to count failures, possibly by exception type.
+            if e.get_cause.is_a? org.apache.kafka.common.errors.RetriableException or
+               e.get_cause.is_a? org.apache.kafka.common.errors.InterruptException
+              logger.info("producer send failed, will retry sending", :exception => e.cause.class,
+                          :message => e.cause.message)
+              failures << batch[i]
+            elsif e.get_cause.is_a? org.apache.kafka.common.KafkaException
+              # This error is not retriable, drop event
+              # TODO: add DLQ support
+              logger.warn("producer send failed, dropping record", :exception => e.cause.class,
+                          :message => e.cause.message, :record_value => batch[i].value)
+            end
+          end
         end
       end
 
@@ -377,35 +392,4 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
     end
   end
 
-  def set_trustore_keystore_config(props)
-    unless ssl_endpoint_identification_algorithm.to_s.strip.empty?
-      if ssl_truststore_location.nil?
-        raise LogStash::ConfigurationError, "ssl_truststore_location must be set when SSL is enabled"
-      end
-      props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
-      props.put("ssl.truststore.location", ssl_truststore_location)
-      props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
-    end
-
-    # Client auth stuff
-    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
-    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
-    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
-    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
-    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
-  end
-
-  def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-
-    props.put("sasl.mechanism",sasl_mechanism)
-    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
-      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
-    end
-
-    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
-  end
-
 end #class LogStash::Outputs::Kafka

data/lib/logstash/plugin_mixins/common.rb

@@ -0,0 +1,92 @@
+module LogStash
+  module PluginMixins
+    module KafkaAvroSchemaRegistry
+
+      def self.included(base)
+        base.extend(self)
+        base.setup_schema_registry_config
+      end
+
+      def setup_schema_registry_config
+        # Option to set key to access Schema Registry.
+        config :schema_registry_key, :validate => :string
+
+        # Option to set secret to access Schema Registry.
+        config :schema_registry_secret, :validate => :password
+
+        # Option to set the endpoint of the Schema Registry.
+        # This option permit the usage of Avro Kafka deserializer which retrieve the schema of the Avro message from an
+        # instance of schema registry. If this option has value `value_deserializer_class` nor `topics_pattern` could be valued
+        config :schema_registry_url, :validate => :uri
+
+        # Option to set the proxy of the Schema Registry.
+        # This option permits to define a proxy to be used to reach the schema registry service instance.
+        config :schema_registry_proxy, :validate => :uri
+      end
+
+      def check_schema_registry_parameters
+        if @schema_registry_url
+          check_for_schema_registry_conflicts
+          @schema_registry_proxy_host, @schema_registry_proxy_port  = split_proxy_into_host_and_port(schema_registry_proxy)
+          check_for_key_and_secret
+          check_for_schema_registry_connectivity_and_subjects
+        end
+      end
+
+      private
+      def check_for_schema_registry_conflicts
+        if @value_deserializer_class != LogStash::Inputs::Kafka::DEFAULT_DESERIALIZER_CLASS
+          raise LogStash::ConfigurationError, 'Option schema_registry_url prohibit the customization of value_deserializer_class'
+        end
+        if @topics_pattern && !@topics_pattern.empty?
+         raise LogStash::ConfigurationError, 'Option schema_registry_url prohibit the customization of topics_pattern'
+        end
+      end
+
+      private
+      def check_for_schema_registry_connectivity_and_subjects
+        client = Faraday.new(@schema_registry_url.to_s) do |conn|
+          if schema_registry_proxy && !schema_registry_proxy.empty?
+            conn.proxy = schema_registry_proxy.to_s
+          end
+          if schema_registry_key and !schema_registry_key.empty?
+            conn.basic_auth(schema_registry_key, schema_registry_secret.value)
+          end
+        end
+        begin
+          response = client.get('/subjects')
+        rescue Faraday::Error => e
+          raise LogStash::ConfigurationError.new("Schema registry service doesn't respond, error: #{e.message}")
+        end
+        registered_subjects = JSON.parse response.body
+        expected_subjects = @topics.map { |t| "#{t}-value"}
+        if (expected_subjects & registered_subjects).size != expected_subjects.size
+          undefined_topic_subjects = expected_subjects - registered_subjects
+          raise LogStash::ConfigurationError, "The schema registry does not contain definitions for required topic subjects: #{undefined_topic_subjects}"
+        end
+      end
+
+      def split_proxy_into_host_and_port(proxy_uri)
+        return nil unless proxy_uri && !proxy_uri.empty?
+
+        port = proxy_uri.port
+
+        host_spec = ""
+        host_spec << proxy_uri.scheme || "http"
+        host_spec << "://"
+        host_spec << "#{proxy_uri.userinfo}@" if proxy_uri.userinfo
+        host_spec << proxy_uri.host
+
+        [host_spec, port]
+      end
+
+      def check_for_key_and_secret
+        if schema_registry_key and !schema_registry_key.empty?
+          if !schema_registry_secret or schema_registry_secret.value.empty?
+            raise LogStash::ConfigurationError, "Setting `schema_registry_secret` is required when `schema_registry_key` is provided."
+          end
+        end
+      end
+    end
+  end
+end

data/lib/logstash/plugin_mixins/kafka_support.rb

@@ -0,0 +1,29 @@
+module LogStash module PluginMixins module KafkaSupport
+
+  def set_trustore_keystore_config(props)
+    props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
+    props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
+    props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
+
+    # Client auth stuff
+    props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
+    props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
+    props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
+    props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
+    props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
+  end
+
+  def set_sasl_config(props)
+    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
+    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
+
+    props.put("sasl.mechanism", sasl_mechanism)
+    if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
+      raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
+    end
+
+    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
+    props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
+  end
+
+end end end

data/logstash-integration-kafka.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-integration-kafka'
-  s.version         = '10.4.0'
+  s.version         = '10.6.0'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Integration with Kafka - input and output plugins"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+

data/spec/integration/inputs/kafka_spec.rb

@@ -2,6 +2,9 @@
 require "logstash/devutils/rspec/spec_helper"
 require "logstash/inputs/kafka"
 require "rspec/wait"
+require "stud/try"
+require "faraday"
+require "json"
 
 # Please run kafka_test_setup.sh prior to executing this integration test.
 describe "inputs/kafka", :integration => true do
@@ -120,20 +123,192 @@ describe "inputs/kafka", :integration => true do
       end
     end
   end
+end
+
+private
+
+def consume_messages(config, queue: Queue.new, timeout:, event_count:)
+  kafka_input = LogStash::Inputs::Kafka.new(config)
+  t = Thread.new { kafka_input.run(queue) }
+  begin
+    t.run
+    wait(timeout).for { queue.length }.to eq(event_count) unless timeout.eql?(false)
+    block_given? ? yield(queue, kafka_input) : queue
+  ensure
+    t.kill
+    t.join(30_000)
+  end
+end
+
+
+describe "schema registry connection options" do
+  context "remote endpoint validation" do
+    it "should fail if not reachable" do
+      config = {'schema_registry_url' => 'http://localnothost:8081'}
+      kafka_input = LogStash::Inputs::Kafka.new(config)
+      expect { kafka_input.register }.to raise_error LogStash::ConfigurationError, /Schema registry service doesn't respond.*/
+    end
+
+    it "should fail if any topic is not matched by a subject on the schema registry" do
+      config = {
+        'schema_registry_url' => 'http://localhost:8081',
+        'topics' => ['temperature_stream']
+      }
+
+      kafka_input = LogStash::Inputs::Kafka.new(config)
+      expect { kafka_input.register }.to raise_error LogStash::ConfigurationError, /The schema registry does not contain definitions for required topic subjects: \["temperature_stream-value"\]/
+    end
+
+    context "register with subject present" do
+      SUBJECT_NAME = "temperature_stream-value"
+
+      before(:each) do
+        response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), SUBJECT_NAME)
+        expect( response.status ).to be(200)
+      end
 
-  private
+      after(:each) do
+        schema_registry_client = Faraday.new('http://localhost:8081')
+        delete_remote_schema(schema_registry_client, SUBJECT_NAME)
+      end
 
-  def consume_messages(config, queue: Queue.new, timeout:, event_count:)
-    kafka_input = LogStash::Inputs::Kafka.new(config)
-    t = Thread.new { kafka_input.run(queue) }
-    begin
-      t.run
-      wait(timeout).for { queue.length }.to eq(event_count) unless timeout.eql?(false)
-      block_given? ? yield(queue, kafka_input) : queue
-    ensure
-      t.kill
-      t.join(30_000)
+      it "should correctly complete registration phase" do
+        config = {
+          'schema_registry_url' => 'http://localhost:8081',
+          'topics' => ['temperature_stream']
+        }
+        kafka_input = LogStash::Inputs::Kafka.new(config)
+        kafka_input.register
+      end
     end
   end
+end
 
+def save_avro_schema_to_schema_registry(schema_file, subject_name)
+  raw_schema = File.readlines(schema_file).map(&:chomp).join
+  raw_schema_quoted = raw_schema.gsub('"', '\"')
+  response = Faraday.post("http://localhost:8081/subjects/#{subject_name}/versions",
+          '{"schema": "' + raw_schema_quoted + '"}',
+          "Content-Type" => "application/vnd.schemaregistry.v1+json")
+  response
 end
+
+def delete_remote_schema(schema_registry_client, subject_name)
+  expect(schema_registry_client.delete("/subjects/#{subject_name}").status ).to be(200)
+  expect(schema_registry_client.delete("/subjects/#{subject_name}?permanent=true").status ).to be(200)
+end
+
+# AdminClientConfig = org.alpache.kafka.clients.admin.AdminClientConfig
+
+describe "Schema registry API", :integration => true do
+
+  let(:schema_registry) { Faraday.new('http://localhost:8081') }
+
+  context 'listing subject on clean instance' do
+    it "should return an empty set" do
+      subjects = JSON.parse schema_registry.get('/subjects').body
+      expect( subjects ).to be_empty
+    end
+  end
+
+  context 'send a schema definition' do
+    it "save the definition" do
+      response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), "schema_test_1")
+      expect( response.status ).to be(200)
+      delete_remote_schema(schema_registry, "schema_test_1")
+    end
+
+    it "delete the schema just added" do
+      response = save_avro_schema_to_schema_registry(File.join(Dir.pwd, "spec", "unit", "inputs", "avro_schema_fixture_payment.asvc"), "schema_test_1")
+      expect( response.status ).to be(200)
+
+      expect( schema_registry.delete('/subjects/schema_test_1?permanent=false').status ).to be(200)
+      sleep(1)
+      subjects = JSON.parse schema_registry.get('/subjects').body
+      expect( subjects ).to be_empty
+    end
+  end
+
+  context 'use the schema to serialize' do
+    after(:each) do
+      expect( schema_registry.delete('/subjects/topic_avro-value').status ).to be(200)
+      sleep 1
+      expect( schema_registry.delete('/subjects/topic_avro-value?permanent=true').status ).to be(200)
+
+      Stud.try(3.times, [StandardError, RSpec::Expectations::ExpectationNotMetError]) do
+        wait(10).for do
+          subjects = JSON.parse schema_registry.get('/subjects').body
+          subjects.empty?
+        end.to be_truthy
+      end
+    end
+
+    let(:group_id_1) {rand(36**8).to_s(36)}
+
+    let(:avro_topic_name) { "topic_avro" }
+
+    let(:plain_config) do
+      { 'schema_registry_url' => 'http://localhost:8081',
+        'topics' => [avro_topic_name],
+        'codec' => 'plain',
+        'group_id' => group_id_1,
+        'auto_offset_reset' => 'earliest' }
+    end
+
+    def delete_topic_if_exists(topic_name)
+      props = java.util.Properties.new
+      props.put(Java::org.apache.kafka.clients.admin.AdminClientConfig::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
+
+      admin_client = org.apache.kafka.clients.admin.AdminClient.create(props)
+      topics_list = admin_client.listTopics().names().get()
+      if topics_list.contains(topic_name)
+        result = admin_client.deleteTopics([topic_name])
+        result.values.get(topic_name).get()
+      end
+    end
+
+    def write_some_data_to(topic_name)
+      props = java.util.Properties.new
+      config = org.apache.kafka.clients.producer.ProducerConfig
+
+      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+      props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081")
+
+      props.put(config::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
+      props.put(config::KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer.java_class)
+      props.put(config::VALUE_SERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroSerializer.java_class)
+
+      parser = org.apache.avro.Schema::Parser.new()
+      user_schema = '''{"type":"record",
+                        "name":"myrecord",
+                        "fields":[
+                            {"name":"str_field", "type": "string"},
+                            {"name":"map_field", "type": {"type": "map", "values": "string"}}
+                        ]}'''
+      schema = parser.parse(user_schema)
+      avro_record = org.apache.avro.generic.GenericData::Record.new(schema)
+      avro_record.put("str_field", "value1")
+      avro_record.put("map_field", {"inner_field" => "inner value"})
+
+      producer = org.apache.kafka.clients.producer.KafkaProducer.new(props)
+      record = org.apache.kafka.clients.producer.ProducerRecord.new(topic_name, "avro_key", avro_record)
+      producer.send(record)
+    end
+
+    it "stored a new schema using Avro Kafka serdes" do
+      delete_topic_if_exists avro_topic_name
+      write_some_data_to avro_topic_name
+
+      subjects = JSON.parse schema_registry.get('/subjects').body
+      expect( subjects ).to contain_exactly("topic_avro-value")
+
+      num_events = 1
+      queue = consume_messages(plain_config, timeout: 30, event_count: num_events)
+      expect(queue.length).to eq(num_events)
+      elem = queue.pop
+      expect( elem.to_hash).not_to include("message")
+      expect( elem.get("str_field") ).to eq("value1")
+      expect( elem.get("map_field")["inner_field"] ).to eq("inner value")
+    end
+  end
+end

data/spec/unit/inputs/avro_schema_fixture_payment.asvc

@@ -0,0 +1,8 @@
+{"namespace": "io.confluent.examples.clients.basicavro",
+ "type": "record",
+ "name": "Payment",
+ "fields": [
+     {"name": "id", "type": "string"},
+     {"name": "amount", "type": "double"}
+ ]
+}

data/spec/unit/inputs/kafka_spec.rb

@@ -37,6 +37,22 @@ describe LogStash::Inputs::Kafka do
     expect { subject.register }.to_not raise_error
   end
 
+  context "register parameter verification" do
+    let(:config) do
+      { 'schema_registry_url' => 'http://localhost:8081', 'topics' => ['logstash'], 'consumer_threads' => 4 }
+    end
+
+    it "schema_registry_url conflict with value_deserializer_class should fail" do
+      config['value_deserializer_class'] = 'my.fantasy.Deserializer'
+      expect { subject.register }.to raise_error LogStash::ConfigurationError, /Option schema_registry_url prohibit the customization of value_deserializer_class/
+    end
+
+    it "schema_registry_url conflict with topics_pattern should fail" do
+      config['topics_pattern'] = 'topic_.*'
+      expect { subject.register }.to raise_error LogStash::ConfigurationError, /Option schema_registry_url prohibit the customization of topics_pattern/
+    end
+  end
+
   context 'with client_rack' do
     let(:config) { super.merge('client_rack' => 'EU-R1') }
 

data/spec/unit/outputs/kafka_spec.rb

@@ -50,20 +50,22 @@ describe "outputs/kafka" do
       kafka.multi_receive([event])
     end
 
-    it 'should raise config error when truststore location is not set and ssl is enabled' do
+    it 'should not raise config error when truststore location is not set and ssl is enabled' do
       kafka = LogStash::Outputs::Kafka.new(simple_kafka_config.merge("security_protocol" => "SSL"))
-      expect { kafka.register }.to raise_error(LogStash::ConfigurationError, /ssl_truststore_location must be set when SSL is enabled/)
+      expect(org.apache.kafka.clients.producer.KafkaProducer).to receive(:new)
+      expect { kafka.register }.to_not raise_error
     end
   end
 
-  context "when KafkaProducer#send() raises an exception" do
+  context "when KafkaProducer#send() raises a retriable exception" do
     let(:failcount) { (rand * 10).to_i }
     let(:sendcount) { failcount + 1 }
 
     let(:exception_classes) { [
       org.apache.kafka.common.errors.TimeoutException,
+      org.apache.kafka.common.errors.DisconnectException,
+      org.apache.kafka.common.errors.CoordinatorNotAvailableException,
       org.apache.kafka.common.errors.InterruptException,
-      org.apache.kafka.common.errors.SerializationException
     ] }
 
     before do
@@ -88,6 +90,37 @@ describe "outputs/kafka" do
     end
   end
 
+  context "when KafkaProducer#send() raises a non-retriable exception" do
+    let(:failcount) { (rand * 10).to_i }
+
+    let(:exception_classes) { [
+        org.apache.kafka.common.errors.SerializationException,
+        org.apache.kafka.common.errors.RecordTooLargeException,
+        org.apache.kafka.common.errors.InvalidTopicException
+    ] }
+
+    before do
+      count = 0
+      expect_any_instance_of(org.apache.kafka.clients.producer.KafkaProducer).to receive(:send)
+        .exactly(1).times
+        .and_wrap_original do |m, *args|
+        if count < failcount # fail 'failcount' times in a row.
+          count += 1
+          # Pick an exception at random
+          raise exception_classes.shuffle.first.new("injected exception for testing")
+        else
+          m.call(*args) # call original
+        end
+      end
+    end
+
+    it "should not retry" do
+      kafka = LogStash::Outputs::Kafka.new(simple_kafka_config)
+      kafka.register
+      kafka.multi_receive([event])
+    end
+  end
+
   context "when a send fails" do
     context "and the default retries behavior is used" do
       # Fail this many times and then finally succeed.
@@ -107,7 +140,7 @@ describe "outputs/kafka" do
             # inject some failures.
 
             # Return a custom Future that will raise an exception to simulate a Kafka send() problem.
-            future = java.util.concurrent.FutureTask.new { raise "Failed" }
+            future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
             future.run
             future
           else
@@ -129,7 +162,7 @@ describe "outputs/kafka" do
                                                                                        .once
                                                                                        .and_wrap_original do |m, *args|
           # Always fail.
-          future = java.util.concurrent.FutureTask.new { raise "Failed" }
+          future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
           future.run
           future
         end
@@ -143,7 +176,7 @@ describe "outputs/kafka" do
                                                                                        .once
                                                                                        .and_wrap_original do |m, *args|
           # Always fail.
-          future = java.util.concurrent.FutureTask.new { raise "Failed" }
+          future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
           future.run
           future
         end
@@ -164,7 +197,7 @@ describe "outputs/kafka" do
               .at_most(max_sends).times
               .and_wrap_original do |m, *args|
           # Always fail.
-          future = java.util.concurrent.FutureTask.new { raise "Failed" }
+          future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
           future.run
           future
         end
@@ -175,10 +208,10 @@ describe "outputs/kafka" do
 
       it 'should only sleep retries number of times' do
         expect_any_instance_of(org.apache.kafka.clients.producer.KafkaProducer).to receive(:send)
-                                                                                       .at_most(max_sends)
+                                                                                       .at_most(max_sends).times
                                                                                        .and_wrap_original do |m, *args|
           # Always fail.
-          future = java.util.concurrent.FutureTask.new { raise "Failed" }
+          future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
           future.run
           future
         end
@@ -193,21 +226,31 @@ describe "outputs/kafka" do
   context 'when ssl endpoint identification disabled' do
 
     let(:config) do
-      simple_kafka_config.merge('ssl_endpoint_identification_algorithm' => '', 'security_protocol' => 'SSL')
+      simple_kafka_config.merge(
+          'security_protocol' => 'SSL',
+          'ssl_endpoint_identification_algorithm' => '',
+          'ssl_truststore_location' => truststore_path,
+      )
+    end
+
+    let(:truststore_path) do
+      File.join(File.dirname(__FILE__), '../../fixtures/trust-store_stub.jks')
     end
 
     subject { LogStash::Outputs::Kafka.new(config) }
 
-    it 'does not configure truststore' do
+    it 'sets empty ssl.endpoint.identification.algorithm' do
       expect(org.apache.kafka.clients.producer.KafkaProducer).
-          to receive(:new).with(hash_excluding('ssl.truststore.location' => anything))
+          to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
       subject.register
     end
 
-    it 'sets empty ssl.endpoint.identification.algorithm' do
+    it 'configures truststore' do
       expect(org.apache.kafka.clients.producer.KafkaProducer).
-          to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
+          to receive(:new).with(hash_including('ssl.truststore.location' => truststore_path))
       subject.register
     end
+
   end
+
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-integration-kafka
 version: !ruby/object:Gem::Version
-  version: 10.4.0
+  version: 10.6.0
 platform: java
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-07-03 00:00:00.000000000 Z
+date: 2020-10-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -183,15 +183,28 @@ files:
 - lib/logstash-integration-kafka_jars.rb
 - lib/logstash/inputs/kafka.rb
 - lib/logstash/outputs/kafka.rb
+- lib/logstash/plugin_mixins/common.rb
+- lib/logstash/plugin_mixins/kafka_support.rb
 - logstash-integration-kafka.gemspec
+- spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
 - spec/integration/outputs/kafka_spec.rb
+- spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb
-- vendor/jar-dependencies/com/github/luben/zstd-jni/1.4.3-1/zstd-jni-1.4.3-1.jar
-- vendor/jar-dependencies/org/apache/kafka/kafka-clients/2.4.1/kafka-clients-2.4.1.jar
-- vendor/jar-dependencies/org/lz4/lz4-java/1.6.0/lz4-java-1.6.0.jar
-- vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar
+- vendor/jar-dependencies/com/github/luben/zstd-jni/1.4.4-7/zstd-jni-1.4.4-7.jar
+- vendor/jar-dependencies/io/confluent/common-config/5.5.1/common-config-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/common-utils/5.5.1/common-utils-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-avro-serializer/5.5.1/kafka-avro-serializer-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-registry-client/5.5.1/kafka-schema-registry-client-5.5.1.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-serializer/5.5.1/kafka-schema-serializer-5.5.1.jar
+- vendor/jar-dependencies/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.jar
+- vendor/jar-dependencies/org/apache/avro/avro/1.9.2/avro-1.9.2.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka-clients/2.5.1/kafka-clients-2.5.1.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka_2.12/2.5.1/kafka_2.12-2.5.1.jar
+- vendor/jar-dependencies/org/glassfish/jersey/core/jersey-common/2.30/jersey-common-2.30.jar
+- vendor/jar-dependencies/org/lz4/lz4-java/1.7.1/lz4-java-1.7.1.jar
+- vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar
 - vendor/jar-dependencies/org/xerial/snappy/snappy-java/1.1.7.3/snappy-java-1.1.7.3.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
@@ -222,7 +235,9 @@ signing_key:
 specification_version: 4
 summary: Integration with Kafka - input and output plugins
 test_files:
+- spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
 - spec/integration/outputs/kafka_spec.rb
+- spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb