logstash-integration-kafka 10.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d8bd0d1d4de277970d19245471ee43ab8832205e
+  data.tar.gz: 70c3fe4143e971f663d82fada824525097e65c37
+SHA512:
+  metadata.gz: fe2c06f811bb179846f4e606af383c2ff2dbead0684b274be12dbaaa021a0bbb5591bb7a903113f98844ed74badebe7550ace29f9fbe90b791a20c5a2265b5e4
+  data.tar.gz: c41661c9edaf931bee9eaa2af266f3aa8b2fe6c1a3fbcb6e9e76da6885d3ec854837a824ee2ac67e8da72b950bc238178a0c35161d1ac95947b3b87d7211e82e

data/CHANGELOG.md

@@ -0,0 +1,6 @@
+## 10.0.0
+  - Initial release of the Kafka Integration Plugin, which combines
+    previously-separate Kafka plugins and shared dependencies into a single
+    codebase; independent changelogs for previous versions can be found:
+     - [Kafka Input Plugin @9.1.0](https://github.com/logstash-plugins/logstash-input-rabbitmq/blob/v9.1.0/CHANGELOG.md)
+     - [Kafka Output Plugin @8.1.0](https://github.com/logstash-plugins/logstash-output-rabbitmq/blob/v8.1.0/CHANGELOG.md)

data/CONTRIBUTORS

@@ -0,0 +1,18 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Joseph Lawson (joekiller)
+* Pere Urbón (purbon)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Suyog Rao (suyograo)
+* Tal Levy (talevy)
+* João Duarte (jsvd)
+* Kurt Hurtado (kurtado)
+* Ry Biesemeyer (yaauie)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,97 @@
+# logsstash-integration-kafka
+
+Apache Kafka integration for Logstash, including Input and Output plugins.
+
+# Dependencies
+
+* Apache Kafka version 0.8.1.1
+* jruby-kafka library
+
+# Plugins
+
+
+## logstash-input-kafka
+
+Apache Kafka input for Logstash. This input will consume messages from a Kafka topic using the high level consumer API exposed by Kafka. 
+
+For more information about Kafka, refer to this [documentation](http://kafka.apache.org/documentation.html) 
+
+Information about high level consumer API can be found [here](http://kafka.apache.org/documentation.html#highlevelconsumerapi)
+
+### Logstash Configuration
+
+See http://kafka.apache.org/documentation.html#consumerconfigs for details about the Kafka consumer options.
+
+    input {
+        kafka {
+            topic_id => ... # string (optional), default: nil, The topic to consume messages from. Can be a java regular expression for whitelist of topics.
+            white_list => ... # string (optional), default: nil, Blacklist of topics to exclude from consumption.
+            black_list => ... # string (optional), default: nil, Whitelist of topics to include for consumption.
+            zk_connect => ... # string (optional), default: "localhost:2181", Specifies the ZooKeeper connection string in the form hostname:port
+            group_id => ... # string (optional), default: "logstash", A string that uniquely identifies the group of consumer processes
+            reset_beginning => ... # boolean (optional), default: false, Specify whether to jump to beginning of the queue when there is no initial offset in ZK
+            auto_offset_reset => ... # string (optional), one of [ "largest", "smallest"] default => 'largest', Where consumer should start if group does not already have an established offset or offset is invalid
+            consumer_threads => ... # number (optional), default: 1, Number of threads to read from the partitions
+            queue_size => ... # number (optional), default: 20, Internal Logstash queue size used to hold events in memory 
+            rebalance_max_retries => ... # number (optional), default: 4
+            rebalance_backoff_ms => ... # number (optional), default:  2000
+            consumer_timeout_ms => ... # number (optional), default: -1
+            consumer_restart_on_error => ... # boolean (optional), default: true
+            consumer_restart_sleep_ms => ... # number (optional), default: 0
+            decorate_events => ... # boolean (optional), default: false, Option to add Kafka metadata like topic, message size to the event
+            consumer_id => ... # string (optional), default: nil
+            fetch_message_max_bytes => ... # number (optional), default: 1048576
+        }
+    }
+
+The default codec is json
+
+## logstash-output-kafka
+
+Apache Kafka output for Logstash. This output will produce messages to a Kafka topic using the producer API exposed by Kafka. 
+
+For more information about Kafka, refer to this [documentation](http://kafka.apache.org/documentation.html) 
+
+Information about producer API can be found [here](http://kafka.apache.org/documentation.html#apidesign)
+
+### Logstash Configuration
+
+See http://kafka.apache.org/documentation.html#producerconfigs for details about the Kafka producer options.
+
+    output {
+        kafka {
+            topic_id => ... # string (required), The topic to produce the messages to
+            broker_list => ... # string (optional), default: "localhost:9092", This is for bootstrapping and the producer will only use it for getting metadata
+            compression_codec => ... # string (optional), one of ["none", "gzip", "snappy"], default: "none"
+            compressed_topics => ... # string (optional), default: "", This parameter allows you to set whether compression should be turned on for particular
+            request_required_acks => ... # number (optional), one of [-1, 0, 1], default: 0, This value controls when a produce request is considered completed
+            serializer_class => ... # string, (optional) default: "kafka.serializer.StringEncoder", The serializer class for messages. The default encoder takes a byte[] and returns the same byte[]
+            partitioner_class => ... # string (optional) default: "kafka.producer.DefaultPartitioner"
+            request_timeout_ms => ... # number (optional) default: 10000
+            producer_type => ... # string (optional), one of ["sync", "async"] default => 'sync'
+            key_serializer_class => ... # string (optional) default: kafka.serializer.StringEncoder
+            message_send_max_retries => ... # number (optional) default: 3
+            retry_backoff_ms => ... # number (optional) default: 100
+            topic_metadata_refresh_interval_ms => ... # number (optional) default: 600 * 1000
+            queue_buffering_max_ms => ... # number (optional) default: 5000
+            queue_buffering_max_messages => ... # number (optional) default: 10000
+            queue_enqueue_timeout_ms => ... # number (optional) default: -1
+            batch_num_messages => ... # number (optional) default: 200
+            send_buffer_bytes => ... # number (optional) default: 100 * 1024
+            client_id => ... # string (optional) default: ""
+            partition_key_format => ... # string (optional) default: nil, Provides a way to specify a partition key as a string
+        }
+    }
+
+The default codec is json for outputs.  If you select a codec of plain, logstash will encode your messages with not only the message
+but also with a timestamp and hostname.  If you do not want anything but your message passing through, you should make
+the output configuration something like:
+
+    output {
+        kafka {
+            codec => plain {
+                format => "%{message}"
+            }
+            topic_id => "my_topic_id"
+        }
+    }

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2019 Elastic NV
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,118 @@
+# Logstash Plugin
+
+[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-integration-kafka.svg)](https://travis-ci.org/logstash-plugins/logstash-integration-kafka)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Logging
+
+Kafka logs do not respect the Log4J2 root logger level and defaults to INFO, for other levels, you must explicitly set the log level in your Logstash deployment's `log4j2.properties` file, e.g.:
+```
+logger.kafka.name=org.apache.kafka
+logger.kafka.appenderRef.console.ref=console
+logger.kafka.level=debug
+```
+
+## Documentation
+
+https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+
+```sh
+bundle install
+rake install_jars
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+rake install_jars
+```
+
+- Run unit tests
+
+```sh
+bundle exec rspec
+```
+
+- Run integration tests
+
+you'll need to have docker available within your test environment before 
+running the integration tests. The tests depend on a specific Kafka image 
+found in Docker Hub called `spotify/kafka`. You will need internet connectivity
+to pull in this image if it does not already exist locally.
+
+```sh
+bundle exec rspec --tag integration
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-output-kafka", :path => "/your/local/logstash-output-kafka"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'output { kafka { topic_id => "kafka_topic" }}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-output-kafka.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-output-kafka.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/docs/index.asciidoc

@@ -0,0 +1,28 @@
+:plugin: kafka
+:type: integration
+:no_codec:
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Kafka Integration Plugin
+
+include::{include_path}/plugin_header.asciidoc[]
+
+==== Description
+
+The Kafka Integration Plugin provides integrated plugins for working with the https://kafka.apache.org/[Kafka] distributed streaming platform.
+
+This plugin uses Kafka Client 2.1.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+
+:no_codec!:

data/docs/input-kafka.asciidoc

@@ -0,0 +1,582 @@
+:plugin: kafka
+:type: input
+:default_codec: plain
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Kafka input plugin
+
+include::{include_path}/plugin_header.asciidoc[]
+
+==== Description
+
+This input will read events from a Kafka topic.
+
+This plugin uses Kafka Client 2.1.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+
+If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
+
+This input supports connecting to Kafka over:
+
+* SSL (requires plugin version 3.0.0 or later)
+* Kerberos SASL (requires plugin version 5.1.0 or later) 
+
+By default security is disabled but can be turned on as needed.
+
+The Logstash Kafka consumer handles group management and uses the default offset management
+strategy using Kafka topics.
+
+Logstash instances by default form a single logical group to subscribe to Kafka topics
+Each Logstash Kafka consumer can run multiple threads to increase read throughput. Alternatively, 
+you could run multiple Logstash instances with the same `group_id` to spread the load across
+physical machines. Messages in a topic will be distributed to all Logstash instances with
+the same `group_id`.
+
+Ideally you should have as many threads as the number of partitions for a perfect balance --
+more threads than partitions means that some threads will be idle
+
+For more information see http://kafka.apache.org/documentation.html#theconsumer
+
+Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
+
+==== Metadata fields
+
+The following metadata from Kafka broker are added under the `[@metadata]` field:
+
+* `[@metadata][kafka][topic]`: Original Kafka topic from where the message was consumed.
+* `[@metadata][kafka][consumer_group]`: Consumer group
+* `[@metadata][kafka][partition]`: Partition info for this message.
+* `[@metadata][kafka][offset]`: Original record offset for this message.
+* `[@metadata][kafka][key]`: Record key, if any.
+* `[@metadata][kafka][timestamp]`: Timestamp in the Record. Depending on your broker configuration, this can be either when the record was created (default) or when it was received by the broker. See more about property log.message.timestamp.type at https://kafka.apache.org/10/documentation.html#brokerconfigs
+
+Metadata is only added to the event if the `decorate_events` option is set to true (it defaults to false).
+
+Please note that `@metadata` fields are not part of any of your events at output time. If you need these information to be 
+inserted into your original event, you'll have to use the `mutate` filter to manually copy the required fields into your `event`.
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Kafka Input Configuration Options
+
+This plugin supports these configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+NOTE: Some of these options map to a Kafka option. See the
+https://kafka.apache.org/documentation for more details.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-auto_commit_interval_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-auto_offset_reset>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-bootstrap_servers>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-check_crcs>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-client_id>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-connections_max_idle_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-consumer_threads>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-decorate_events>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-enable_auto_commit>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-exclude_internal_topics>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-fetch_max_bytes>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-fetch_max_wait_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-fetch_min_bytes>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-group_id>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-heartbeat_interval_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-jaas_path>> |a valid filesystem path|No
+| <<plugins-{type}s-{plugin}-kerberos_config>> |a valid filesystem path|No
+| <<plugins-{type}s-{plugin}-key_deserializer_class>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-max_partition_fetch_bytes>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-max_poll_interval_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-max_poll_records>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-metadata_max_age_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-partition_assignment_strategy>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-poll_timeout_ms>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-receive_buffer_bytes>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-reconnect_backoff_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-request_timeout_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-retry_backoff_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-sasl_jaas_config>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-sasl_kerberos_service_name>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-sasl_mechanism>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-security_protocol>> |<<string,string>>, one of `["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"]`|No
+| <<plugins-{type}s-{plugin}-send_buffer_bytes>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-session_timeout_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ssl_endpoint_identification_algorithm>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ssl_key_password>> |<<password,password>>|No
+| <<plugins-{type}s-{plugin}-ssl_keystore_location>> |a valid filesystem path|No
+| <<plugins-{type}s-{plugin}-ssl_keystore_password>> |<<password,password>>|No
+| <<plugins-{type}s-{plugin}-ssl_keystore_type>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ssl_truststore_location>> |a valid filesystem path|No
+| <<plugins-{type}s-{plugin}-ssl_truststore_password>> |<<password,password>>|No
+| <<plugins-{type}s-{plugin}-ssl_truststore_type>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-topics>> |<<array,array>>|No
+| <<plugins-{type}s-{plugin}-topics_pattern>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-value_deserializer_class>> |<<string,string>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+input plugins.
+
+&nbsp;
+
+[id="plugins-{type}s-{plugin}-auto_commit_interval_ms"]
+===== `auto_commit_interval_ms` 
+
+  * Value type is <<string,string>>
+  * Default value is `"5000"`
+
+The frequency in milliseconds that the consumer offsets are committed to Kafka.
+
+[id="plugins-{type}s-{plugin}-auto_offset_reset"]
+===== `auto_offset_reset` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+What to do when there is no initial offset in Kafka or if an offset is out of range:
+
+* earliest: automatically reset the offset to the earliest offset
+* latest: automatically reset the offset to the latest offset
+* none: throw exception to the consumer if no previous offset is found for the consumer's group
+* anything else: throw exception to the consumer.
+
+[id="plugins-{type}s-{plugin}-bootstrap_servers"]
+===== `bootstrap_servers` 
+
+  * Value type is <<string,string>>
+  * Default value is `"localhost:9092"`
+
+A list of URLs of Kafka instances to use for establishing the initial connection to the cluster.
+This list should be in the form of `host1:port1,host2:port2` These urls are just used
+for the initial connection to discover the full cluster membership (which may change dynamically)
+so this list need not contain the full set of servers (you may want more than one, though, in
+case a server is down).
+
+[id="plugins-{type}s-{plugin}-check_crcs"]
+===== `check_crcs` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk
+corruption to the messages occurred. This check adds some overhead, so it may be
+disabled in cases seeking extreme performance.
+
+[id="plugins-{type}s-{plugin}-client_id"]
+===== `client_id` 
+
+  * Value type is <<string,string>>
+  * Default value is `"logstash"`
+
+The id string to pass to the server when making requests. The purpose of this
+is to be able to track the source of requests beyond just ip/port by allowing
+a logical application name to be included.
+
+[id="plugins-{type}s-{plugin}-connections_max_idle_ms"]
+===== `connections_max_idle_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Close idle connections after the number of milliseconds specified by this config.
+
+[id="plugins-{type}s-{plugin}-consumer_threads"]
+===== `consumer_threads` 
+
+  * Value type is <<number,number>>
+  * Default value is `1`
+
+Ideally you should have as many threads as the number of partitions for a perfect
+balance — more threads than partitions means that some threads will be idle
+
+[id="plugins-{type}s-{plugin}-decorate_events"]
+===== `decorate_events` 
+
+  * Value type is <<boolean,boolean>>
+  * Default value is `false`
+
+Option to add Kafka metadata like topic, message size to the event.
+This will add a field named `kafka` to the logstash event containing the following attributes:
+
+* `topic`: The topic this message is associated with
+* `consumer_group`: The consumer group used to read in this event
+* `partition`: The partition this message is associated with
+* `offset`: The offset from the partition this message is associated with
+* `key`: A ByteBuffer containing the message key
+
+[id="plugins-{type}s-{plugin}-enable_auto_commit"]
+===== `enable_auto_commit` 
+
+  * Value type is <<string,string>>
+  * Default value is `"true"`
+
+If true, periodically commit to Kafka the offsets of messages already returned by the consumer. 
+This committed offset will be used when the process fails as the position from
+which the consumption will begin.
+
+[id="plugins-{type}s-{plugin}-exclude_internal_topics"]
+===== `exclude_internal_topics` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Whether records from internal topics (such as offsets) should be exposed to the consumer.
+If set to true the only way to receive records from an internal topic is subscribing to it.
+
+[id="plugins-{type}s-{plugin}-fetch_max_bytes"]
+===== `fetch_max_bytes` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The maximum amount of data the server should return for a fetch request. This is not an 
+absolute maximum, if the first message in the first non-empty partition of the fetch is larger 
+than this value, the message will still be returned to ensure that the consumer can make progress.
+
+[id="plugins-{type}s-{plugin}-fetch_max_wait_ms"]
+===== `fetch_max_wait_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The maximum amount of time the server will block before answering the fetch request if
+there isn't sufficient data to immediately satisfy `fetch_min_bytes`. This
+should be less than or equal to the timeout used in `poll_timeout_ms`
+
+[id="plugins-{type}s-{plugin}-fetch_min_bytes"]
+===== `fetch_min_bytes` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The minimum amount of data the server should return for a fetch request. If insufficient
+data is available the request will wait for that much data to accumulate
+before answering the request.
+
+[id="plugins-{type}s-{plugin}-group_id"]
+===== `group_id` 
+
+  * Value type is <<string,string>>
+  * Default value is `"logstash"`
+
+The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
+that happens to be made up of multiple processors. Messages in a topic will be distributed to all
+Logstash instances with the same `group_id`
+
+[id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
+===== `heartbeat_interval_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The expected time between heartbeats to the consumer coordinator. Heartbeats are used to ensure 
+that the consumer's session stays active and to facilitate rebalancing when new
+consumers join or leave the group. The value must be set lower than
+`session.timeout.ms`, but typically should be set no higher than 1/3 of that value.
+It can be adjusted even lower to control the expected time for normal rebalances.
+
+[id="plugins-{type}s-{plugin}-jaas_path"]
+===== `jaas_path` 
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+The Java Authentication and Authorization Service (JAAS) API supplies user authentication and authorization 
+services for Kafka. This setting provides the path to the JAAS file. Sample JAAS file for Kafka client:
+[source,java]
+----------------------------------
+KafkaClient {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useTicketCache=true
+  renewTicket=true
+  serviceName="kafka";
+  };
+----------------------------------
+
+Please note that specifying `jaas_path` and `kerberos_config` in the config file will add these  
+to the global JVM system properties. This means if you have multiple Kafka inputs, all of them would be sharing the same 
+`jaas_path` and `kerberos_config`. If this is not desirable, you would have to run separate instances of Logstash on 
+different JVM instances.
+
+[id="plugins-{type}s-{plugin}-kerberos_config"]
+===== `kerberos_config` 
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+Optional path to kerberos config file. This is krb5.conf style as detailed in https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html
+
+[id="plugins-{type}s-{plugin}-key_deserializer_class"]
+===== `key_deserializer_class` 
+
+  * Value type is <<string,string>>
+  * Default value is `"org.apache.kafka.common.serialization.StringDeserializer"`
+
+Java Class used to deserialize the record's key
+
+[id="plugins-{type}s-{plugin}-max_partition_fetch_bytes"]
+===== `max_partition_fetch_bytes` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The maximum amount of data per-partition the server will return. The maximum total memory used for a
+request will be `#partitions * max.partition.fetch.bytes`. This size must be at least
+as large as the maximum message size the server allows or else it is possible for the producer to
+send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying
+to fetch a large message on a certain partition.
+
+[id="plugins-{type}s-{plugin}-max_poll_interval_ms"]
+===== `max_poll_interval_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The maximum delay between invocations of poll() when using consumer group management. This places 
+an upper bound on the amount of time that the consumer can be idle before fetching more records. 
+If poll() is not called before expiration of this timeout, then the consumer is considered failed and 
+the group will rebalance in order to reassign the partitions to another member.
+The value of the configuration `request_timeout_ms` must always be larger than max_poll_interval_ms
+
+[id="plugins-{type}s-{plugin}-max_poll_records"]
+===== `max_poll_records` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The maximum number of records returned in a single call to poll().
+
+[id="plugins-{type}s-{plugin}-metadata_max_age_ms"]
+===== `metadata_max_age_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The period of time in milliseconds after which we force a refresh of metadata even if
+we haven't seen any partition leadership changes to proactively discover any new brokers or partitions
+
+[id="plugins-{type}s-{plugin}-partition_assignment_strategy"]
+===== `partition_assignment_strategy` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The class name of the partition assignment strategy that the client uses to
+distribute partition ownership amongst consumer instances. Maps to
+the Kafka `partition.assignment.strategy` setting, which defaults to
+`org.apache.kafka.clients.consumer.RangeAssignor`.
+
+[id="plugins-{type}s-{plugin}-poll_timeout_ms"]
+===== `poll_timeout_ms` 
+
+  * Value type is <<number,number>>
+  * Default value is `100`
+
+Time kafka consumer will wait to receive new messages from topics
+
+[id="plugins-{type}s-{plugin}-receive_buffer_bytes"]
+===== `receive_buffer_bytes` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The size of the TCP receive buffer (SO_RCVBUF) to use when reading data.
+
+[id="plugins-{type}s-{plugin}-reconnect_backoff_ms"]
+===== `reconnect_backoff_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The amount of time to wait before attempting to reconnect to a given host.
+This avoids repeatedly connecting to a host in a tight loop.
+This backoff applies to all requests sent by the consumer to the broker.
+
+[id="plugins-{type}s-{plugin}-request_timeout_ms"]
+===== `request_timeout_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The configuration controls the maximum amount of time the client will wait
+for the response of a request. If the response is not received before the timeout
+elapses the client will resend the request if necessary or fail the request if
+retries are exhausted.
+
+[id="plugins-{type}s-{plugin}-retry_backoff_ms"]
+===== `retry_backoff_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The amount of time to wait before attempting to retry a failed fetch request
+to a given topic partition. This avoids repeated fetching-and-failing in a tight loop.
+
+[id="plugins-{type}s-{plugin}-sasl_jaas_config"]
+===== `sasl_jaas_config` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+JAAS configuration setting local to this plugin instance, as opposed to settings using config file configured using `jaas_path`, which are shared across the JVM. This allows each plugin instance to have its own configuration. 
+
+If both `sasl_jaas_config` and `jaas_path` configurations are set, the setting here takes precedence.
+
+Example (setting for Azure Event Hub):
+[source,ruby]
+    input {
+      kafka {
+        sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='auser'  password='apassword';"
+      }
+    }
+
+[id="plugins-{type}s-{plugin}-sasl_kerberos_service_name"]
+===== `sasl_kerberos_service_name` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The Kerberos principal name that Kafka broker runs as. 
+This can be defined either in Kafka's JAAS config or in Kafka's config.
+
+[id="plugins-{type}s-{plugin}-sasl_mechanism"]
+===== `sasl_mechanism` 
+
+  * Value type is <<string,string>>
+  * Default value is `"GSSAPI"`
+
+http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. 
+This may be any mechanism for which a security provider is available.
+GSSAPI is the default mechanism.
+
+[id="plugins-{type}s-{plugin}-security_protocol"]
+===== `security_protocol` 
+
+  * Value can be any of: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL`
+  * Default value is `"PLAINTEXT"`
+
+Security protocol to use, which can be either of PLAINTEXT,SSL,SASL_PLAINTEXT,SASL_SSL
+
+[id="plugins-{type}s-{plugin}-send_buffer_bytes"]
+===== `send_buffer_bytes` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The size of the TCP send buffer (SO_SNDBUF) to use when sending data
+
+[id="plugins-{type}s-{plugin}-session_timeout_ms"]
+===== `session_timeout_ms` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The timeout after which, if the `poll_timeout_ms` is not invoked, the consumer is marked dead
+and a rebalance operation is triggered for the group identified by `group_id`
+
+[id="plugins-{type}s-{plugin}-ssl_endpoint_identification_algorithm"]
+===== `ssl_endpoint_identification_algorithm`
+
+  * Value type is <<string,string>>
+  * Default value is `"https"`
+
+The endpoint identification algorithm, defaults to `"https"`. Set to empty string `""` to disable endpoint verification
+
+
+[id="plugins-{type}s-{plugin}-ssl_key_password"]
+===== `ssl_key_password` 
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+The password of the private key in the key store file.
+
+[id="plugins-{type}s-{plugin}-ssl_keystore_location"]
+===== `ssl_keystore_location` 
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+If client authentication is required, this setting stores the keystore path.
+
+[id="plugins-{type}s-{plugin}-ssl_keystore_password"]
+===== `ssl_keystore_password` 
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+If client authentication is required, this setting stores the keystore password
+
+[id="plugins-{type}s-{plugin}-ssl_keystore_type"]
+===== `ssl_keystore_type` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The keystore type.
+
+[id="plugins-{type}s-{plugin}-ssl_truststore_location"]
+===== `ssl_truststore_location` 
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+The JKS truststore path to validate the Kafka broker's certificate.
+
+[id="plugins-{type}s-{plugin}-ssl_truststore_password"]
+===== `ssl_truststore_password` 
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+The truststore password
+
+[id="plugins-{type}s-{plugin}-ssl_truststore_type"]
+===== `ssl_truststore_type` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The truststore type.
+
+[id="plugins-{type}s-{plugin}-topics"]
+===== `topics` 
+
+  * Value type is <<array,array>>
+  * Default value is `["logstash"]`
+
+A list of topics to subscribe to, defaults to ["logstash"].
+
+[id="plugins-{type}s-{plugin}-topics_pattern"]
+===== `topics_pattern` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+A topic regex pattern to subscribe to. 
+The topics configuration will be ignored when using this configuration.
+
+[id="plugins-{type}s-{plugin}-value_deserializer_class"]
+===== `value_deserializer_class` 
+
+  * Value type is <<string,string>>
+  * Default value is `"org.apache.kafka.common.serialization.StringDeserializer"`
+
+Java Class used to deserialize the record's value
+
+
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
+
+:default_codec!: