RubyGems - logstash-integration-kafka - Versions diffs - 10.1.0-java → 10.2.0-java - Mend

logstash-integration-kafka 10.1.0-java → 10.2.0-java

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (10) hide show

checksums.yaml +4 -4
data/CHANGELOG.md +3 -0
data/docs/index.asciidoc +1 -1
data/docs/input-kafka.asciidoc +84 -77
data/docs/output-kafka.asciidoc +21 -17
data/lib/logstash/inputs/kafka.rb +49 -51
data/lib/logstash/outputs/kafka.rb +14 -15
data/logstash-integration-kafka.gemspec +1 -1
data/spec/unit/inputs/kafka_spec.rb +50 -0
metadata +2 -2

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 254abccf066d63d45cf0660dafa06b603c97fb5557c1f978ecc41b41078c6ead
-  data.tar.gz: a6bcb799f703db46d80a4412f35809b7a7d13bcdf8eaf91e12ef06befc700a93
+  metadata.gz: 88d87b3c5b9e443c0bc4ba8064b6316243f24bb0c0d6ce150ca0a19dd25ecb2a
+  data.tar.gz: 5fc09468e8cdc7094bc6099c24d70b1c19bc339c3ec3dfa3562e0c389fa85753
 SHA512:
-  metadata.gz: 9551a410f21e1015e56ebd2d2881d75c1eb3d13e5a3aff609e98ac4111376764bcc1978612bb9b239b2335757e129897a94be69bac02d996dfbf31d50ffc9614
-  data.tar.gz: 106b3fa2816631035f132a4771f4b5776fd9a79811305200d37a1b539a3ba1600079e8b9d9c1ec718ff9a2cecdea9115fb31b2ccd425edfe9097114da7ca752f
+  metadata.gz: 4e0fc4586ca91967e9f8afa4a2eef95b7a5f8cd13f3b6689b4ace12b64d3ea628bc5112735d54edcb20c0fe1c6debac1e4d32983e1555eb9a84d380ab5d68a4e
+  data.tar.gz: 10ae655222832b9b5f6b9d8aa155d0907da91e38f04c7ad74c949d7bf4c8ff5d222bde6b2e1779eca5bd4bebb99aaf54701c8e2fddd4cb5a3cff47fabe9831fe

data/CHANGELOG.md CHANGED Viewed

@@ -1,3 +1,6 @@
+## 10.2.0
+  - Changed: config defaults to be aligned with Kafka client defaults [#30](https://github.com/logstash-plugins/logstash-integration-kafka/pull/30)
 ## 10.1.0
   - updated kafka client (and its dependencies) to version 2.4.1 ([#16](https://github.com/logstash-plugins/logstash-integration-kafka/pull/16))
   - added the input `client_rack` parameter to enable support for follower fetching

data/docs/index.asciidoc CHANGED Viewed

@@ -26,6 +26,6 @@ The Kafka Integration Plugin provides integrated plugins for working with the ht
  - {logstash-ref}/plugins-inputs-kafka.html[Kafka Input Plugin]
  - {logstash-ref}/plugins-outputs-kafka.html[Kafka Output Plugin]
-This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
+This plugin uses Kafka Client 2.4. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
 :no_codec!:

data/docs/input-kafka.asciidoc CHANGED Viewed

@@ -46,9 +46,9 @@ the same `group_id`.
 Ideally you should have as many threads as the number of partitions for a perfect balance --
 more threads than partitions means that some threads will be idle
-For more information see http://kafka.apache.org/documentation.html#theconsumer
+For more information see https://kafka.apache.org/24/documentation.html#theconsumer
-Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
+Kafka consumer configuration: https://kafka.apache.org/24/documentation.html#consumerconfigs
 ==== Metadata fields
@@ -71,45 +71,48 @@ inserted into your original event, you'll have to use the `mutate` filter to man
 This plugin supports these configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
-NOTE: Some of these options map to a Kafka option. See the https://kafka.apache.org/documentation for more details.
+NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
+and might change if Kafka's consumer defaults change.
+See the https://kafka.apache.org/24/documentation for more details.
 [cols="<,<,<",options="header",]
 |=======================================================================
 |Setting |Input type|Required
-| <<plugins-{type}s-{plugin}-auto_commit_interval_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-auto_commit_interval_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-auto_offset_reset>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-bootstrap_servers>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-check_crcs>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-check_crcs>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-client_id>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-connections_max_idle_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-client_rack>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-connections_max_idle_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-consumer_threads>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-decorate_events>> |<<boolean,boolean>>|No
-| <<plugins-{type}s-{plugin}-enable_auto_commit>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-enable_auto_commit>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-exclude_internal_topics>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-fetch_max_bytes>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-fetch_max_wait_ms>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-fetch_min_bytes>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-fetch_max_bytes>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-fetch_max_wait_ms>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-fetch_min_bytes>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-group_id>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-heartbeat_interval_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-heartbeat_interval_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-jaas_path>> |a valid filesystem path|No
 | <<plugins-{type}s-{plugin}-kerberos_config>> |a valid filesystem path|No
 | <<plugins-{type}s-{plugin}-key_deserializer_class>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-max_partition_fetch_bytes>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-max_poll_interval_ms>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-max_poll_records>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-metadata_max_age_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-max_partition_fetch_bytes>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-max_poll_interval_ms>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-max_poll_records>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-metadata_max_age_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-partition_assignment_strategy>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-poll_timeout_ms>> |<<number,number>>|No
-| <<plugins-{type}s-{plugin}-receive_buffer_bytes>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-reconnect_backoff_ms>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-request_timeout_ms>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-retry_backoff_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-receive_buffer_bytes>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-reconnect_backoff_ms>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-request_timeout_ms>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-retry_backoff_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-sasl_jaas_config>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_kerberos_service_name>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sasl_mechanism>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-security_protocol>> |<<string,string>>, one of `["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"]`|No
-| <<plugins-{type}s-{plugin}-send_buffer_bytes>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-session_timeout_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-send_buffer_bytes>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-session_timeout_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-ssl_endpoint_identification_algorithm>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-ssl_key_password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-ssl_keystore_location>> |a valid filesystem path|No
@@ -121,7 +124,6 @@ NOTE: Some of these options map to a Kafka option. See the https://kafka.apache.
 | <<plugins-{type}s-{plugin}-topics>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-topics_pattern>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-value_deserializer_class>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-client_rack>> |<<string,string>>|No
 |=======================================================================
 Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
@@ -132,8 +134,8 @@ input plugins.
 [id="plugins-{type}s-{plugin}-auto_commit_interval_ms"]
 ===== `auto_commit_interval_ms`
-  * Value type is <<string,string>>
-  * Default value is `"5000"`
+  * Value type is <<number,number>>
+  * Default value is `5000`.
 The frequency in milliseconds that the consumer offsets are committed to Kafka.
@@ -165,12 +167,12 @@ case a server is down).
 [id="plugins-{type}s-{plugin}-check_crcs"]
 ===== `check_crcs`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<boolean,boolean>>
+  * Default value is `true`
-Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk
-corruption to the messages occurred. This check adds some overhead, so it may be
-disabled in cases seeking extreme performance.
+Automatically check the CRC32 of the records consumed.
+This ensures no on-the-wire or on-disk corruption to the messages occurred.
+This check adds some overhead, so it may be disabled in cases seeking extreme performance.
 [id="plugins-{type}s-{plugin}-client_id"]
 ===== `client_id`
@@ -182,12 +184,25 @@ The id string to pass to the server when making requests. The purpose of this
 is to be able to track the source of requests beyond just ip/port by allowing
 a logical application name to be included.
-[id="plugins-{type}s-{plugin}-connections_max_idle_ms"]
-===== `connections_max_idle_ms`
+[id="plugins-{type}s-{plugin}-client_rack"]
+===== `client_rack`
   * Value type is <<string,string>>
   * There is no default value for this setting.
+A rack identifier for the Kafka consumer.
+Used to select the physically closest rack for the consumer to read from.
+The setting corresponds with Kafka's `broker.rack` configuration.
+NOTE: Available only for Kafka 2.4.0 and higher. See
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica[KIP-392].
+[id="plugins-{type}s-{plugin}-connections_max_idle_ms"]
+===== `connections_max_idle_ms`
+  * Value type is <<number,number>>
+  * Default value is `540000` milliseconds (9 minutes).
 Close idle connections after the number of milliseconds specified by this config.
 [id="plugins-{type}s-{plugin}-consumer_threads"]
@@ -217,8 +232,8 @@ This will add a field named `kafka` to the logstash event containing the followi
 [id="plugins-{type}s-{plugin}-enable_auto_commit"]
 ===== `enable_auto_commit`
-  * Value type is <<string,string>>
-  * Default value is `"true"`
+  * Value type is <<boolean,boolean>>
+  * Default value is `true`
 This committed offset will be used when the process fails as the position from
 which the consumption will begin.
@@ -239,8 +254,8 @@ If set to true the only way to receive records from an internal topic is subscri
 [id="plugins-{type}s-{plugin}-fetch_max_bytes"]
 ===== `fetch_max_bytes`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `52428800` (50MB)
 The maximum amount of data the server should return for a fetch request. This is not an
 absolute maximum, if the first message in the first non-empty partition of the fetch is larger
@@ -249,8 +264,8 @@ than this value, the message will still be returned to ensure that the consumer
 [id="plugins-{type}s-{plugin}-fetch_max_wait_ms"]
 ===== `fetch_max_wait_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `500` milliseconds.
 The maximum amount of time the server will block before answering the fetch request if
 there isn't sufficient data to immediately satisfy `fetch_min_bytes`. This
@@ -259,7 +274,7 @@ should be less than or equal to the timeout used in `poll_timeout_ms`
 [id="plugins-{type}s-{plugin}-fetch_min_bytes"]
 ===== `fetch_min_bytes`
-  * Value type is <<string,string>>
+  * Value type is <<number,number>>
   * There is no default value for this setting.
 The minimum amount of data the server should return for a fetch request. If insufficient
@@ -279,8 +294,8 @@ Logstash instances with the same `group_id`
 [id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
 ===== `heartbeat_interval_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `3000` milliseconds (3 seconds).
 The expected time between heartbeats to the consumer coordinator. Heartbeats are used to ensure
 that the consumer's session stays active and to facilitate rebalancing when new
@@ -330,8 +345,8 @@ Java Class used to deserialize the record's key
 [id="plugins-{type}s-{plugin}-max_partition_fetch_bytes"]
 ===== `max_partition_fetch_bytes`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `1048576` (1MB).
 The maximum amount of data per-partition the server will return. The maximum total memory used for a
 request will be `#partitions * max.partition.fetch.bytes`. This size must be at least
@@ -342,28 +357,28 @@ to fetch a large message on a certain partition.
 [id="plugins-{type}s-{plugin}-max_poll_interval_ms"]
 ===== `max_poll_interval_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `300000` milliseconds (5 minutes).
 The maximum delay between invocations of poll() when using consumer group management. This places
 an upper bound on the amount of time that the consumer can be idle before fetching more records.
 If poll() is not called before expiration of this timeout, then the consumer is considered failed and
 the group will rebalance in order to reassign the partitions to another member.
-The value of the configuration `request_timeout_ms` must always be larger than max_poll_interval_ms
+The value of the configuration `request_timeout_ms` must always be larger than `max_poll_interval_ms`. ???
 [id="plugins-{type}s-{plugin}-max_poll_records"]
 ===== `max_poll_records`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `500`.
 The maximum number of records returned in a single call to poll().
 [id="plugins-{type}s-{plugin}-metadata_max_age_ms"]
 ===== `metadata_max_age_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `300000` milliseconds (5 minutes).
 The period of time in milliseconds after which we force a refresh of metadata even if
 we haven't seen any partition leadership changes to proactively discover any new brokers or partitions
@@ -389,23 +404,28 @@ implementations.
 ===== `poll_timeout_ms`
   * Value type is <<number,number>>
-  * Default value is `100`
+  * Default value is `100` milliseconds.
+Time Kafka consumer will wait to receive new messages from topics.
-Time kafka consumer will wait to receive new messages from topics
+After subscribing to a set of topics, the Kafka consumer automatically joins the group when polling.
+The plugin poll-ing in a loop ensures consumer liveness.
+Underneath the covers, Kafka client sends periodic heartbeats to the server.
+The timeout specified the time to block waiting for input on each poll.
 [id="plugins-{type}s-{plugin}-receive_buffer_bytes"]
 ===== `receive_buffer_bytes`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `32768` (32KB).
 The size of the TCP receive buffer (SO_RCVBUF) to use when reading data.
 [id="plugins-{type}s-{plugin}-reconnect_backoff_ms"]
 ===== `reconnect_backoff_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `50` milliseconds.
 The amount of time to wait before attempting to reconnect to a given host.
 This avoids repeatedly connecting to a host in a tight loop.
@@ -414,8 +434,8 @@ This backoff applies to all requests sent by the consumer to the broker.
 [id="plugins-{type}s-{plugin}-request_timeout_ms"]
 ===== `request_timeout_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `40000` milliseconds (40 seconds).
 The configuration controls the maximum amount of time the client will wait
 for the response of a request. If the response is not received before the timeout
@@ -425,8 +445,8 @@ retries are exhausted.
 [id="plugins-{type}s-{plugin}-retry_backoff_ms"]
 ===== `retry_backoff_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `100` milliseconds.
 The amount of time to wait before attempting to retry a failed fetch request
 to a given topic partition. This avoids repeated fetching-and-failing in a tight loop.
@@ -479,16 +499,16 @@ Security protocol to use, which can be either of PLAINTEXT,SSL,SASL_PLAINTEXT,SA
 [id="plugins-{type}s-{plugin}-send_buffer_bytes"]
 ===== `send_buffer_bytes`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `131072` (128KB).
 The size of the TCP send buffer (SO_SNDBUF) to use when sending data
 [id="plugins-{type}s-{plugin}-session_timeout_ms"]
 ===== `session_timeout_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `10000` milliseconds (10 seconds).
 The timeout after which, if the `poll_timeout_ms` is not invoked, the consumer is marked dead
 and a rebalance operation is triggered for the group identified by `group_id`
@@ -548,7 +568,7 @@ The JKS truststore path to validate the Kafka broker's certificate.
   * Value type is <<password,password>>
   * There is no default value for this setting.
-The truststore password
+The truststore password.
 [id="plugins-{type}s-{plugin}-ssl_truststore_type"]
 ===== `ssl_truststore_type`
@@ -583,19 +603,6 @@ The topics configuration will be ignored when using this configuration.
 Java Class used to deserialize the record's value
-[id="plugins-{type}s-{plugin}-client_rack"]
-===== `client_rack`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
-A rack identifier for the Kafka consumer.
-Used to select the physically closest rack for the consumer to read from.
-The setting corresponds with Kafka's `broker.rack` configuration.
-NOTE: Only available for Kafka 2.4.0 and higher; see https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica[KIP-392].
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]

data/docs/output-kafka.asciidoc CHANGED Viewed

@@ -47,15 +47,19 @@ If you want the full content of your events to be sent as json, you should set t
       }
     }
-For more information see http://kafka.apache.org/documentation.html#theproducer
+For more information see https://kafka.apache.org/24/documentation.html#theproducer
-Kafka producer configuration: http://kafka.apache.org/documentation.html#newproducerconfigs
+Kafka producer configuration: https://kafka.apache.org/24/documentation.html#producerconfigs
 [id="plugins-{type}s-{plugin}-options"]
 ==== Kafka Output Configuration Options
 This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
+and might change if Kafka's producer defaults change.
+See the https://kafka.apache.org/24/documentation for more details.
 [cols="<,<,<",options="header",]
 |=======================================================================
 |Setting |Input type|Required
@@ -76,7 +80,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-partitioner>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-receive_buffer_bytes>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-reconnect_backoff_ms>> |<<number,number>>|No
-| <<plugins-{type}s-{plugin}-request_timeout_ms>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-request_timeout_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retries>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retry_backoff_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-sasl_jaas_config>> |<<string,string>>|No
@@ -119,7 +123,7 @@ acks=all, This means the leader will wait for the full set of in-sync replicas t
 ===== `batch_size`
   * Value type is <<number,number>>
-  * Default value is `16384`
+  * Default value is `16384`.
 The producer will attempt to batch records together into fewer requests whenever multiple
 records are being sent to the same partition. This helps performance on both the client
@@ -141,7 +145,7 @@ subset of brokers.
 ===== `buffer_memory`
   * Value type is <<number,number>>
-  * Default value is `33554432`
+  * Default value is `33554432` (32MB).
 The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
@@ -221,7 +225,7 @@ to allow other records to be sent so that the sends can be batched together.
 ===== `max_request_size`
   * Value type is <<number,number>>
-  * Default value is `1048576`
+  * Default value is `1048576` (1MB).
 The maximum size of a request
@@ -231,23 +235,23 @@ The maximum size of a request
   * Value type is <<string,string>>
   * There is no default value for this setting.
-The key for the message
+The key for the message.
 [id="plugins-{type}s-{plugin}-metadata_fetch_timeout_ms"]
 ===== `metadata_fetch_timeout_ms`
   * Value type is <<number,number>>
-  * Default value is `60000`
+  * Default value is `60000` milliseconds (60 seconds).
-the timeout setting for initial metadata request to fetch topic metadata.
+The timeout setting for initial metadata request to fetch topic metadata.
 [id="plugins-{type}s-{plugin}-metadata_max_age_ms"]
 ===== `metadata_max_age_ms`
   * Value type is <<number,number>>
-  * Default value is `300000`
+  * Default value is `300000` milliseconds (5 minutes).
-the max time in milliseconds before a metadata refresh is forced.
+The max time in milliseconds before a metadata refresh is forced.
 [id="plugins-{type}s-{plugin}-partitioner"]
 ===== `partitioner`
@@ -268,7 +272,7 @@ Available options for choosing a partitioning strategy are as follows:
 ===== `receive_buffer_bytes`
   * Value type is <<number,number>>
-  * Default value is `32768`
+  * Default value is `32768` (32KB).
 The size of the TCP receive buffer to use when reading data
@@ -276,15 +280,15 @@ The size of the TCP receive buffer to use when reading data
 ===== `reconnect_backoff_ms`
   * Value type is <<number,number>>
-  * Default value is `10`
+  * Default value is `50`.
 The amount of time to wait before attempting to reconnect to a given host when a connection fails.
 [id="plugins-{type}s-{plugin}-request_timeout_ms"]
 ===== `request_timeout_ms`
-  * Value type is <<string,string>>
-  * There is no default value for this setting.
+  * Value type is <<number,number>>
+  * Default value is `40000` milliseconds (40 seconds).
 The configuration controls the maximum amount of time the client will wait
 for the response of a request. If the response is not received before the timeout
@@ -311,7 +315,7 @@ A value less than zero is a configuration error.
 ===== `retry_backoff_ms`
   * Value type is <<number,number>>
-  * Default value is `100`
+  * Default value is `100` milliseconds.
 The amount of time to wait before attempting to retry a failed produce request to a given topic partition.
@@ -364,7 +368,7 @@ Security protocol to use, which can be either of PLAINTEXT,SSL,SASL_PLAINTEXT,SA
 ===== `send_buffer_bytes`
   * Value type is <<number,number>>
-  * Default value is `131072`
+  * Default value is `131072` (128KB).
 The size of the TCP send buffer to use when sending data.

data/lib/logstash/inputs/kafka.rb CHANGED Viewed

@@ -53,7 +53,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   default :codec, 'plain'
   # The frequency in milliseconds that the consumer offsets are committed to Kafka.
-  config :auto_commit_interval_ms, :validate => :string, :default => "5000"
+  config :auto_commit_interval_ms, :validate => :number, :default => 5000 # Kafka default
   # What to do when there is no initial offset in Kafka or if an offset is out of range:
   #
   # * earliest: automatically reset the offset to the earliest offset
@@ -70,35 +70,35 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk
   # corruption to the messages occurred. This check adds some overhead, so it may be
   # disabled in cases seeking extreme performance.
-  config :check_crcs, :validate => :string
+  config :check_crcs, :validate => :boolean, :default => true
   # The id string to pass to the server when making requests. The purpose of this
   # is to be able to track the source of requests beyond just ip/port by allowing
   # a logical application name to be included.
   config :client_id, :validate => :string, :default => "logstash"
   # Close idle connections after the number of milliseconds specified by this config.
-  config :connections_max_idle_ms, :validate => :string
+  config :connections_max_idle_ms, :validate => :number, :default => 540_000 # (9m) Kafka default
   # Ideally you should have as many threads as the number of partitions for a perfect
   # balance — more threads than partitions means that some threads will be idle
   config :consumer_threads, :validate => :number, :default => 1
   # If true, periodically commit to Kafka the offsets of messages already returned by the consumer.
   # This committed offset will be used when the process fails as the position from
   # which the consumption will begin.
-  config :enable_auto_commit, :validate => :string, :default => "true"
+  config :enable_auto_commit, :validate => :boolean, :default => true
   # Whether records from internal topics (such as offsets) should be exposed to the consumer.
   # If set to true the only way to receive records from an internal topic is subscribing to it.
   config :exclude_internal_topics, :validate => :string
   # The maximum amount of data the server should return for a fetch request. This is not an
   # absolute maximum, if the first message in the first non-empty partition of the fetch is larger
   # than this value, the message will still be returned to ensure that the consumer can make progress.
-  config :fetch_max_bytes, :validate => :string
+  config :fetch_max_bytes, :validate => :number, :default => 52_428_800 # (50MB) Kafka default
   # The maximum amount of time the server will block before answering the fetch request if
   # there isn't sufficient data to immediately satisfy `fetch_min_bytes`. This
   # should be less than or equal to the timeout used in `poll_timeout_ms`
-  config :fetch_max_wait_ms, :validate => :string
+  config :fetch_max_wait_ms, :validate => :number, :default => 500 # Kafka default
   # The minimum amount of data the server should return for a fetch request. If insufficient
   # data is available the request will wait for that much data to accumulate
   # before answering the request.
-  config :fetch_min_bytes, :validate => :string
+  config :fetch_min_bytes, :validate => :number
   # The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
   # that happens to be made up of multiple processors. Messages in a topic will be distributed to all
   # Logstash instances with the same `group_id`
@@ -108,50 +108,50 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # consumers join or leave the group. The value must be set lower than
   # `session.timeout.ms`, but typically should be set no higher than 1/3 of that value.
   # It can be adjusted even lower to control the expected time for normal rebalances.
-  config :heartbeat_interval_ms, :validate => :string
+  config :heartbeat_interval_ms, :validate => :number, :default => 3000 # Kafka default
   # Java Class used to deserialize the record's key
   config :key_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
   # The maximum delay between invocations of poll() when using consumer group management. This places
   # an upper bound on the amount of time that the consumer can be idle before fetching more records.
   # If poll() is not called before expiration of this timeout, then the consumer is considered failed and
   # the group will rebalance in order to reassign the partitions to another member.
-  # The value of the configuration `request_timeout_ms` must always be larger than max_poll_interval_ms
-  config :max_poll_interval_ms, :validate => :string
+  config :max_poll_interval_ms, :validate => :number, :default => 300_000 # (5m) Kafka default
   # The maximum amount of data per-partition the server will return. The maximum total memory used for a
   # request will be <code>#partitions * max.partition.fetch.bytes</code>. This size must be at least
   # as large as the maximum message size the server allows or else it is possible for the producer to
   # send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying
   # to fetch a large message on a certain partition.
-  config :max_partition_fetch_bytes, :validate => :string
+  config :max_partition_fetch_bytes, :validate => :number, :default => 1_048_576 # (1MB) Kafka default
   # The maximum number of records returned in a single call to poll().
-  config :max_poll_records, :validate => :string
+  config :max_poll_records, :validate => :number, :default => 500 # Kafka default
   # The period of time in milliseconds after which we force a refresh of metadata even if
   # we haven't seen any partition leadership changes to proactively discover any new brokers or partitions
-  config :metadata_max_age_ms, :validate => :string
+  config :metadata_max_age_ms, :validate => :number, :default => 300_000 # (5m) Kafka default
   # The name of the partition assignment strategy that the client uses to distribute
   # partition ownership amongst consumer instances, supported options are `range`,
   # `round_robin`, `sticky` and `cooperative_sticky`
   # (for backwards compatibility setting the class name directly is supported).
   config :partition_assignment_strategy, :validate => :string
   # The size of the TCP receive buffer (SO_RCVBUF) to use when reading data.
-  config :receive_buffer_bytes, :validate => :string
-  # The amount of time to wait before attempting to reconnect to a given host.
+  # If the value is `-1`, the OS default will be used.
+  config :receive_buffer_bytes, :validate => :number, :default => 32_768 # (32KB) Kafka default
+  # The base amount of time to wait before attempting to reconnect to a given host.
   # This avoids repeatedly connecting to a host in a tight loop.
-  # This backoff applies to all requests sent by the consumer to the broker.
-  config :reconnect_backoff_ms, :validate => :string
-  # The configuration controls the maximum amount of time the client will wait
-  # for the response of a request. If the response is not received before the timeout
-  # elapses the client will resend the request if necessary or fail the request if
-  # retries are exhausted.
-  config :request_timeout_ms, :validate => :string
+  # This backoff applies to all connection attempts by the client to a broker.
+  config :reconnect_backoff_ms, :validate => :number, :default => 50 # Kafka default
+  # The configuration controls the maximum amount of time the client will wait for the response of a request.
+  # If the response is not received before the timeout elapses the client will resend the request if necessary
+  # or fail the request if retries are exhausted.
+  config :request_timeout_ms, :validate => :number, :default => 40_000 # Kafka default
   # The amount of time to wait before attempting to retry a failed fetch request
   # to a given topic partition. This avoids repeated fetching-and-failing in a tight loop.
-  config :retry_backoff_ms, :validate => :string
-  # The size of the TCP send buffer (SO_SNDBUF) to use when sending data
-  config :send_buffer_bytes, :validate => :string
+  config :retry_backoff_ms, :validate => :number, :default => 100 # Kafka default
+  # The size of the TCP send buffer (SO_SNDBUF) to use when sending data.
+  # If the value is -1, the OS default will be used.
+  config :send_buffer_bytes, :validate => :number, :default => 131_072 # (128KB) Kafka default
   # The timeout after which, if the `poll_timeout_ms` is not invoked, the consumer is marked dead
   # and a rebalance operation is triggered for the group identified by `group_id`
-  config :session_timeout_ms, :validate => :string
+  config :session_timeout_ms, :validate => :number, :default => 10_000 # (10s) Kafka default
   # Java Class used to deserialize the record's value
   config :value_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
   # A list of topics to subscribe to, defaults to ["logstash"].
@@ -276,9 +276,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
             end
           end
           # Manual offset commit
-          if @enable_auto_commit == "false"
-            consumer.commitSync
-          end
+          consumer.commitSync if @enable_auto_commit.eql?(false)
         end
       rescue org.apache.kafka.common.errors.WakeupException => e
         raise e if !stop?
@@ -294,31 +292,31 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props = java.util.Properties.new
       kafka = org.apache.kafka.clients.consumer.ConsumerConfig
-      props.put(kafka::AUTO_COMMIT_INTERVAL_MS_CONFIG, auto_commit_interval_ms)
+      props.put(kafka::AUTO_COMMIT_INTERVAL_MS_CONFIG, auto_commit_interval_ms.to_s) unless auto_commit_interval_ms.nil?
       props.put(kafka::AUTO_OFFSET_RESET_CONFIG, auto_offset_reset) unless auto_offset_reset.nil?
       props.put(kafka::BOOTSTRAP_SERVERS_CONFIG, bootstrap_servers)
-      props.put(kafka::CHECK_CRCS_CONFIG, check_crcs) unless check_crcs.nil?
+      props.put(kafka::CHECK_CRCS_CONFIG, check_crcs.to_s) unless check_crcs.nil?
       props.put(kafka::CLIENT_ID_CONFIG, client_id)
-      props.put(kafka::CONNECTIONS_MAX_IDLE_MS_CONFIG, connections_max_idle_ms) unless connections_max_idle_ms.nil?
-      props.put(kafka::ENABLE_AUTO_COMMIT_CONFIG, enable_auto_commit)
+      props.put(kafka::CONNECTIONS_MAX_IDLE_MS_CONFIG, connections_max_idle_ms.to_s) unless connections_max_idle_ms.nil?
+      props.put(kafka::ENABLE_AUTO_COMMIT_CONFIG, enable_auto_commit.to_s)
       props.put(kafka::EXCLUDE_INTERNAL_TOPICS_CONFIG, exclude_internal_topics) unless exclude_internal_topics.nil?
-      props.put(kafka::FETCH_MAX_BYTES_CONFIG, fetch_max_bytes) unless fetch_max_bytes.nil?
-      props.put(kafka::FETCH_MAX_WAIT_MS_CONFIG, fetch_max_wait_ms) unless fetch_max_wait_ms.nil?
-      props.put(kafka::FETCH_MIN_BYTES_CONFIG, fetch_min_bytes) unless fetch_min_bytes.nil?
+      props.put(kafka::FETCH_MAX_BYTES_CONFIG, fetch_max_bytes.to_s) unless fetch_max_bytes.nil?
+      props.put(kafka::FETCH_MAX_WAIT_MS_CONFIG, fetch_max_wait_ms.to_s) unless fetch_max_wait_ms.nil?
+      props.put(kafka::FETCH_MIN_BYTES_CONFIG, fetch_min_bytes.to_s) unless fetch_min_bytes.nil?
       props.put(kafka::GROUP_ID_CONFIG, group_id)
-      props.put(kafka::HEARTBEAT_INTERVAL_MS_CONFIG, heartbeat_interval_ms) unless heartbeat_interval_ms.nil?
+      props.put(kafka::HEARTBEAT_INTERVAL_MS_CONFIG, heartbeat_interval_ms.to_s) unless heartbeat_interval_ms.nil?
       props.put(kafka::KEY_DESERIALIZER_CLASS_CONFIG, key_deserializer_class)
-      props.put(kafka::MAX_PARTITION_FETCH_BYTES_CONFIG, max_partition_fetch_bytes) unless max_partition_fetch_bytes.nil?
-      props.put(kafka::MAX_POLL_RECORDS_CONFIG, max_poll_records) unless max_poll_records.nil?
-      props.put(kafka::MAX_POLL_INTERVAL_MS_CONFIG, max_poll_interval_ms) unless max_poll_interval_ms.nil?
-      props.put(kafka::METADATA_MAX_AGE_CONFIG, metadata_max_age_ms) unless metadata_max_age_ms.nil?
+      props.put(kafka::MAX_PARTITION_FETCH_BYTES_CONFIG, max_partition_fetch_bytes.to_s) unless max_partition_fetch_bytes.nil?
+      props.put(kafka::MAX_POLL_RECORDS_CONFIG, max_poll_records.to_s) unless max_poll_records.nil?
+      props.put(kafka::MAX_POLL_INTERVAL_MS_CONFIG, max_poll_interval_ms.to_s) unless max_poll_interval_ms.nil?
+      props.put(kafka::METADATA_MAX_AGE_CONFIG, metadata_max_age_ms.to_s) unless metadata_max_age_ms.nil?
       props.put(kafka::PARTITION_ASSIGNMENT_STRATEGY_CONFIG, partition_assignment_strategy_class) unless partition_assignment_strategy.nil?
-      props.put(kafka::RECEIVE_BUFFER_CONFIG, receive_buffer_bytes) unless receive_buffer_bytes.nil?
-      props.put(kafka::RECONNECT_BACKOFF_MS_CONFIG, reconnect_backoff_ms) unless reconnect_backoff_ms.nil?
-      props.put(kafka::REQUEST_TIMEOUT_MS_CONFIG, request_timeout_ms) unless request_timeout_ms.nil?
-      props.put(kafka::RETRY_BACKOFF_MS_CONFIG, retry_backoff_ms) unless retry_backoff_ms.nil?
-      props.put(kafka::SEND_BUFFER_CONFIG, send_buffer_bytes) unless send_buffer_bytes.nil?
-      props.put(kafka::SESSION_TIMEOUT_MS_CONFIG, session_timeout_ms) unless session_timeout_ms.nil?
+      props.put(kafka::RECEIVE_BUFFER_CONFIG, receive_buffer_bytes.to_s) unless receive_buffer_bytes.nil?
+      props.put(kafka::RECONNECT_BACKOFF_MS_CONFIG, reconnect_backoff_ms.to_s) unless reconnect_backoff_ms.nil?
+      props.put(kafka::REQUEST_TIMEOUT_MS_CONFIG, request_timeout_ms.to_s) unless request_timeout_ms.nil?
+      props.put(kafka::RETRY_BACKOFF_MS_CONFIG, retry_backoff_ms.to_s) unless retry_backoff_ms.nil?
+      props.put(kafka::SEND_BUFFER_CONFIG, send_buffer_bytes.to_s) unless send_buffer_bytes.nil?
+      props.put(kafka::SESSION_TIMEOUT_MS_CONFIG, session_timeout_ms.to_s) unless session_timeout_ms.nil?
       props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, value_deserializer_class)
       props.put(kafka::CLIENT_RACK_CONFIG, client_rack) unless client_rack.nil?
@@ -374,15 +372,15 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   end
   def set_sasl_config(props)
-    java.lang.System.setProperty("java.security.auth.login.config",jaas_path) unless jaas_path.nil?
-    java.lang.System.setProperty("java.security.krb5.conf",kerberos_config) unless kerberos_config.nil?
+    java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
+    java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
-    props.put("sasl.mechanism",sasl_mechanism)
+    props.put("sasl.mechanism", sasl_mechanism)
     if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
       raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
     end
-    props.put("sasl.kerberos.service.name",sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
+    props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
     props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
   end
 end #class LogStash::Inputs::Kafka

data/lib/logstash/outputs/kafka.rb CHANGED Viewed

@@ -67,7 +67,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   # The producer will attempt to batch records together into fewer requests whenever multiple
   # records are being sent to the same partition. This helps performance on both the client
   # and the server. This configuration controls the default batch size in bytes.
-  config :batch_size, :validate => :number, :default => 16384
+  config :batch_size, :validate => :number, :default => 16_384 # Kafka default
   # This is for bootstrapping and the producer will only use it for getting metadata (topics,
   # partitions and replicas). The socket connections for sending the actual data will be
   # established based on the broker information returned in the metadata. The format is
@@ -75,7 +75,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   # subset of brokers.
   config :bootstrap_servers, :validate => :string, :default => 'localhost:9092'
   # The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
-  config :buffer_memory, :validate => :number, :default => 33554432
+  config :buffer_memory, :validate => :number, :default => 33_554_432 # (32M) Kafka default
   # The compression type for all data generated by the producer.
   # The default is none (i.e. no compression). Valid values are none, gzip, or snappy.
   config :compression_type, :validate => ["none", "gzip", "snappy", "lz4"], :default => "none"
@@ -92,26 +92,26 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   # This setting accomplishes this by adding a small amount of artificial delay—that is,
   # rather than immediately sending out a record the producer will wait for up to the given delay
   # to allow other records to be sent so that the sends can be batched together.
-  config :linger_ms, :validate => :number, :default => 0
+  config :linger_ms, :validate => :number, :default => 0 # Kafka default
   # The maximum size of a request
-  config :max_request_size, :validate => :number, :default => 1048576
+  config :max_request_size, :validate => :number, :default => 1_048_576 # (1MB) Kafka default
   # The key for the message
   config :message_key, :validate => :string
   # the timeout setting for initial metadata request to fetch topic metadata.
-  config :metadata_fetch_timeout_ms, :validate => :number, :default => 60000
+  config :metadata_fetch_timeout_ms, :validate => :number, :default => 60_000
   # the max time in milliseconds before a metadata refresh is forced.
-  config :metadata_max_age_ms, :validate => :number, :default => 300000
+  config :metadata_max_age_ms, :validate => :number, :default => 300_000 # (5m) Kafka default
   # Partitioner to use - can be `default`, `uniform_sticky`, `round_robin` or a fully qualified class name of a custom partitioner.
   config :partitioner, :validate => :string
   # The size of the TCP receive buffer to use when reading data
-  config :receive_buffer_bytes, :validate => :number, :default => 32768
+  config :receive_buffer_bytes, :validate => :number, :default => 32_768 # (32KB) Kafka default
   # The amount of time to wait before attempting to reconnect to a given host when a connection fails.
-  config :reconnect_backoff_ms, :validate => :number, :default => 10
+  config :reconnect_backoff_ms, :validate => :number, :default => 50 # Kafka default
   # The configuration controls the maximum amount of time the client will wait
   # for the response of a request. If the response is not received before the timeout
   # elapses the client will resend the request if necessary or fail the request if
   # retries are exhausted.
-  config :request_timeout_ms, :validate => :string
+  config :request_timeout_ms, :validate => :number, :default => 40_000 # (40s) Kafka default
   # The default retry behavior is to retry until successful. To prevent data loss,
   # the use of this setting is discouraged.
   #
@@ -122,9 +122,9 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   # A value less than zero is a configuration error.
   config :retries, :validate => :number
   # The amount of time to wait before attempting to retry a failed produce request to a given topic partition.
-  config :retry_backoff_ms, :validate => :number, :default => 100
+  config :retry_backoff_ms, :validate => :number, :default => 100 # Kafka default
   # The size of the TCP send buffer to use when sending data.
-  config :send_buffer_bytes, :validate => :number, :default => 131072
+  config :send_buffer_bytes, :validate => :number, :default => 131_072 # (128KB) Kafka default
   # The truststore type.
   config :ssl_truststore_type, :validate => :string
   # The JKS truststore path to validate the Kafka broker's certificate.
@@ -322,14 +322,14 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
       props.put(kafka::KEY_SERIALIZER_CLASS_CONFIG, key_serializer)
       props.put(kafka::LINGER_MS_CONFIG, linger_ms.to_s)
       props.put(kafka::MAX_REQUEST_SIZE_CONFIG, max_request_size.to_s)
-      props.put(kafka::METADATA_MAX_AGE_CONFIG, metadata_max_age_ms) unless metadata_max_age_ms.nil?
+      props.put(kafka::METADATA_MAX_AGE_CONFIG, metadata_max_age_ms.to_s) unless metadata_max_age_ms.nil?
       unless partitioner.nil?
         props.put(kafka::PARTITIONER_CLASS_CONFIG, partitioner = partitioner_class)
         logger.debug('producer configured using partitioner', :partitioner_class => partitioner)
       end
       props.put(kafka::RECEIVE_BUFFER_CONFIG, receive_buffer_bytes.to_s) unless receive_buffer_bytes.nil?
-      props.put(kafka::RECONNECT_BACKOFF_MS_CONFIG, reconnect_backoff_ms) unless reconnect_backoff_ms.nil?
-      props.put(kafka::REQUEST_TIMEOUT_MS_CONFIG, request_timeout_ms) unless request_timeout_ms.nil?
+      props.put(kafka::RECONNECT_BACKOFF_MS_CONFIG, reconnect_backoff_ms.to_s) unless reconnect_backoff_ms.nil?
+      props.put(kafka::REQUEST_TIMEOUT_MS_CONFIG, request_timeout_ms.to_s) unless request_timeout_ms.nil?
       props.put(kafka::RETRIES_CONFIG, retries.to_s) unless retries.nil?
       props.put(kafka::RETRY_BACKOFF_MS_CONFIG, retry_backoff_ms.to_s)
       props.put(kafka::SEND_BUFFER_CONFIG, send_buffer_bytes.to_s)
@@ -346,7 +346,6 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
         set_sasl_config(props)
       end
       org.apache.kafka.clients.producer.KafkaProducer.new(props)
     rescue => e
       logger.error("Unable to create Kafka producer from given configuration",

data/logstash-integration-kafka.gemspec CHANGED Viewed

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-integration-kafka'
-  s.version         = '10.1.0'
+  s.version         = '10.2.0'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Integration with Kafka - input and output plugins"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+

data/spec/unit/inputs/kafka_spec.rb CHANGED Viewed

@@ -48,4 +48,54 @@ describe LogStash::Inputs::Kafka do
       expect( subject.send(:create_consumer, 'sample_client-0') ).to be kafka_client
     end
   end
+  context 'string integer config' do
+    let(:config) { super.merge('session_timeout_ms' => '25000', 'max_poll_interval_ms' => '345000') }
+    it "sets integer values" do
+      expect(org.apache.kafka.clients.consumer.KafkaConsumer).
+          to receive(:new).with(hash_including('session.timeout.ms' => '25000', 'max.poll.interval.ms' => '345000')).
+              and_return kafka_client = double('kafka-consumer')
+      expect( subject.send(:create_consumer, 'sample_client-1') ).to be kafka_client
+    end
+  end
+  context 'integer config' do
+    let(:config) { super.merge('session_timeout_ms' => 25200, 'max_poll_interval_ms' => 123_000) }
+    it "sets integer values" do
+      expect(org.apache.kafka.clients.consumer.KafkaConsumer).
+          to receive(:new).with(hash_including('session.timeout.ms' => '25200', 'max.poll.interval.ms' => '123000')).
+              and_return kafka_client = double('kafka-consumer')
+      expect( subject.send(:create_consumer, 'sample_client-2') ).to be kafka_client
+    end
+  end
+  context 'string boolean config' do
+    let(:config) { super.merge('enable_auto_commit' => 'false', 'check_crcs' => 'true') }
+    it "sets parameters" do
+      expect(org.apache.kafka.clients.consumer.KafkaConsumer).
+          to receive(:new).with(hash_including('enable.auto.commit' => 'false', 'check.crcs' => 'true')).
+              and_return kafka_client = double('kafka-consumer')
+      expect( subject.send(:create_consumer, 'sample_client-3') ).to be kafka_client
+      expect( subject.enable_auto_commit ).to be false
+    end
+  end
+  context 'boolean config' do
+    let(:config) { super.merge('enable_auto_commit' => true, 'check_crcs' => false) }
+    it "sets parameters" do
+      expect(org.apache.kafka.clients.consumer.KafkaConsumer).
+          to receive(:new).with(hash_including('enable.auto.commit' => 'true', 'check.crcs' => 'false')).
+              and_return kafka_client = double('kafka-consumer')
+      expect( subject.send(:create_consumer, 'sample_client-4') ).to be kafka_client
+      expect( subject.enable_auto_commit ).to be true
+    end
+  end
 end

metadata CHANGED Viewed

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-integration-kafka
 version: !ruby/object:Gem::Version
-  version: 10.1.0
+  version: 10.2.0
 platform: java
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-03-26 00:00:00.000000000 Z
+date: 2020-04-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement