logstash-integration-kafka 10.3.0-java → 10.5.3-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: cbbffe559ff38eaa26c9922c62a3722b2b1404446602238bcc7de309ca428da6
4
- data.tar.gz: 0ce7e58400e06ae5e1c7f689a14220600efe7ad9ba46f285a52323a05568f69f
3
+ metadata.gz: 4383db6ec7c8fa26ef358d104c490f51620f615afb2f68359b6f6e98d4e58f8b
4
+ data.tar.gz: 040637202d15cb1e5784104ff505f10a6610e91187a57f104a8f81cd2b24475a
5
5
  SHA512:
6
- metadata.gz: 7be900607b24146853783d4e3a2e5a6d351853ab4dda0b30d10769adf8b13df21540cea2699d94660bd9e56580c9bf88e9684d5762ff7684f5d3a6804637b602
7
- data.tar.gz: cd1f72db84b5af7de286d683f0f312e4b826c8d75b9b8fc475f693d8bf62b517c61950dd5d4887767f79f69df17b2c9bf1223e03880b2724333aa0b695b81525
6
+ metadata.gz: 98da085bceebd241a6d45f9166aa4ff1a132551cd2cda8825ceab3c22ebbb5f78579f0d1a0b596aeaf3d504147d656cf2ef784570ef3fd9ab98c792ae6e15be4
7
+ data.tar.gz: f552e5ec8d84f3ae7d85b3d4bc4ba4f7309321ea81efaa9bfb69a4a493141868b2576a626f0ee061bb0ebe6cbc8071993dd8592bc53030478baad2b5173e1086
@@ -1,5 +1,27 @@
1
+ ## 10.5.3
2
+ - Fix: set (optional) truststore when endpoint id check disabled [#60](https://github.com/logstash-plugins/logstash-integration-kafka/pull/60).
3
+ Since **10.1.0** disabling server host-name verification (`ssl_endpoint_identification_algorithm => ""`) did not allow
4
+ the (output) plugin to set `ssl_truststore_location => "..."`.
5
+
6
+ ## 10.5.2
7
+ - Docs: explain group_id in case of multiple inputs [#59](https://github.com/logstash-plugins/logstash-integration-kafka/pull/59)
8
+
9
+ ## 10.5.1
10
+ - [DOC]Replaced plugin_header file with plugin_header-integration file. [#46](https://github.com/logstash-plugins/logstash-integration-kafka/pull/46)
11
+ - [DOC]Update kafka client version across kafka integration docs [#47](https://github.com/logstash-plugins/logstash-integration-kafka/pull/47)
12
+ - [DOC]Replace hard-coded kafka client and doc path version numbers with attributes to simplify doc maintenance [#48](https://github.com/logstash-plugins/logstash-integration-kafka/pull/48)
13
+
14
+ ## 10.5.0
15
+ - Changed: retry sending messages only for retriable exceptions [#27](https://github.com/logstash-plugins/logstash-integration-kafka/pull/29)
16
+
17
+ ## 10.4.1
18
+ - [DOC] Fixed formatting issues and made minor content edits [#43](https://github.com/logstash-plugins/logstash-integration-kafka/pull/43)
19
+
20
+ ## 10.4.0
21
+ - added the input `isolation_level` to allow fine control of whether to return transactional messages [#44](https://github.com/logstash-plugins/logstash-integration-kafka/pull/44)
22
+
1
23
  ## 10.3.0
2
- - added the input and output `client_dns_lookup` parameter to allow control of how DNS requests are made
24
+ - added the input and output `client_dns_lookup` parameter to allow control of how DNS requests are made [#28](https://github.com/logstash-plugins/logstash-integration-kafka/pull/28)
3
25
 
4
26
  ## 10.2.0
5
27
  - Changed: config defaults to be aligned with Kafka client defaults [#30](https://github.com/logstash-plugins/logstash-integration-kafka/pull/30)
@@ -12,6 +12,7 @@ Contributors:
12
12
  * Kurt Hurtado (kurtado)
13
13
  * Ry Biesemeyer (yaauie)
14
14
  * Rob Cowart (robcowart)
15
+ * Tim te Beek (timtebeek)
15
16
 
16
17
  Note: If you've sent us patches, bug reports, or otherwise contributed to
17
18
  Logstash, and you aren't on the list above and want to be, please let us know
@@ -1,6 +1,7 @@
1
1
  :plugin: kafka
2
2
  :type: integration
3
3
  :no_codec:
4
+ :kafka_client: 2.4
4
5
 
5
6
  ///////////////////////////////////////////
6
7
  START - GENERATED VARIABLES, DO NOT EDIT!
@@ -21,11 +22,15 @@ include::{include_path}/plugin_header.asciidoc[]
21
22
 
22
23
  ==== Description
23
24
 
24
- The Kafka Integration Plugin provides integrated plugins for working with the https://kafka.apache.org/[Kafka] distributed streaming platform.
25
+ The Kafka Integration Plugin provides integrated plugins for working with the
26
+ https://kafka.apache.org/[Kafka] distributed streaming platform.
25
27
 
26
28
  - {logstash-ref}/plugins-inputs-kafka.html[Kafka Input Plugin]
27
29
  - {logstash-ref}/plugins-outputs-kafka.html[Kafka Output Plugin]
28
30
 
29
- This plugin uses Kafka Client 2.4. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
31
+ This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the official
32
+ https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
33
+ compatibility reference]. If the linked compatibility wiki is not up-to-date,
34
+ please contact Kafka support/community to confirm compatibility.
30
35
 
31
36
  :no_codec!:
@@ -1,6 +1,9 @@
1
+ :integration: kafka
1
2
  :plugin: kafka
2
3
  :type: input
3
4
  :default_codec: plain
5
+ :kafka_client: 2.4
6
+ :kafka_client_doc: 24
4
7
 
5
8
  ///////////////////////////////////////////
6
9
  START - GENERATED VARIABLES, DO NOT EDIT!
@@ -17,15 +20,20 @@ END - GENERATED VARIABLES, DO NOT EDIT!
17
20
 
18
21
  === Kafka input plugin
19
22
 
20
- include::{include_path}/plugin_header.asciidoc[]
23
+ include::{include_path}/plugin_header-integration.asciidoc[]
21
24
 
22
25
  ==== Description
23
26
 
24
27
  This input will read events from a Kafka topic.
25
28
 
26
- This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
29
+ This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the
30
+ official
31
+ https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
32
+ compatibility reference]. If the linked compatibility wiki is not up-to-date,
33
+ please contact Kafka support/community to confirm compatibility.
27
34
 
28
- If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
35
+ If you require features not yet available in this plugin (including client
36
+ version upgrades), please file an issue with details about what you need.
29
37
 
30
38
  This input supports connecting to Kafka over:
31
39
 
@@ -46,9 +54,9 @@ the same `group_id`.
46
54
  Ideally you should have as many threads as the number of partitions for a perfect balance --
47
55
  more threads than partitions means that some threads will be idle
48
56
 
49
- For more information see https://kafka.apache.org/24/documentation.html#theconsumer
57
+ For more information see https://kafka.apache.org/{kafka_client_doc}/documentation.html#theconsumer
50
58
 
51
- Kafka consumer configuration: https://kafka.apache.org/24/documentation.html#consumerconfigs
59
+ Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/documentation.html#consumerconfigs
52
60
 
53
61
  ==== Metadata fields
54
62
 
@@ -59,7 +67,11 @@ The following metadata from Kafka broker are added under the `[@metadata]` field
59
67
  * `[@metadata][kafka][partition]`: Partition info for this message.
60
68
  * `[@metadata][kafka][offset]`: Original record offset for this message.
61
69
  * `[@metadata][kafka][key]`: Record key, if any.
62
- * `[@metadata][kafka][timestamp]`: Timestamp in the Record. Depending on your broker configuration, this can be either when the record was created (default) or when it was received by the broker. See more about property log.message.timestamp.type at https://kafka.apache.org/10/documentation.html#brokerconfigs
70
+ * `[@metadata][kafka][timestamp]`: Timestamp in the Record.
71
+ Depending on your broker configuration, this can be
72
+ either when the record was created (default) or when it was received by the
73
+ broker. See more about property log.message.timestamp.type at
74
+ https://kafka.apache.org/{kafka_client_doc}/documentation.html#brokerconfigs
63
75
 
64
76
  Metadata is only added to the event if the `decorate_events` option is set to true (it defaults to false).
65
77
 
@@ -73,7 +85,7 @@ This plugin supports these configuration options plus the <<plugins-{type}s-{plu
73
85
 
74
86
  NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
75
87
  and might change if Kafka's consumer defaults change.
76
- See the https://kafka.apache.org/24/documentation for more details.
88
+ See the https://kafka.apache.org/{kafka_client_doc}/documentation for more details.
77
89
 
78
90
  [cols="<,<,<",options="header",]
79
91
  |=======================================================================
@@ -95,6 +107,7 @@ See the https://kafka.apache.org/24/documentation for more details.
95
107
  | <<plugins-{type}s-{plugin}-fetch_min_bytes>> |<<number,number>>|No
96
108
  | <<plugins-{type}s-{plugin}-group_id>> |<<string,string>>|No
97
109
  | <<plugins-{type}s-{plugin}-heartbeat_interval_ms>> |<<number,number>>|No
110
+ | <<plugins-{type}s-{plugin}-isolation_level>> |<<string,string>>|No
98
111
  | <<plugins-{type}s-{plugin}-jaas_path>> |a valid filesystem path|No
99
112
  | <<plugins-{type}s-{plugin}-kerberos_config>> |a valid filesystem path|No
100
113
  | <<plugins-{type}s-{plugin}-key_deserializer_class>> |<<string,string>>|No
@@ -301,7 +314,11 @@ before answering the request.
301
314
 
302
315
  The identifier of the group this consumer belongs to. Consumer group is a single logical subscriber
303
316
  that happens to be made up of multiple processors. Messages in a topic will be distributed to all
304
- Logstash instances with the same `group_id`
317
+ Logstash instances with the same `group_id`.
318
+
319
+ NOTE: In cases when multiple inputs are being used in a single pipeline, reading from different topics,
320
+ it's essential to set a different `group_id => ...` for each input. Setting a unique `client_id => ...`
321
+ is also recommended.
305
322
 
306
323
  [id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
307
324
  ===== `heartbeat_interval_ms`
@@ -315,6 +332,17 @@ consumers join or leave the group. The value must be set lower than
315
332
  `session.timeout.ms`, but typically should be set no higher than 1/3 of that value.
316
333
  It can be adjusted even lower to control the expected time for normal rebalances.
317
334
 
335
+ [id="plugins-{type}s-{plugin}-isolation_level"]
336
+ ===== `isolation_level`
337
+
338
+ * Value type is <<string,string>>
339
+ * Default value is `"read_uncommitted"`
340
+
341
+ Controls how to read messages written transactionally. If set to `read_committed`, polling messages will only return
342
+ transactional messages which have been committed. If set to `read_uncommitted` (the default), polling messages will
343
+ return all messages, even transactional messages which have been aborted. Non-transactional messages will be returned
344
+ unconditionally in either mode.
345
+
318
346
  [id="plugins-{type}s-{plugin}-jaas_path"]
319
347
  ===== `jaas_path`
320
348
 
@@ -409,7 +437,7 @@ partition ownership amongst consumer instances, supported options are:
409
437
  * `sticky`
410
438
  * `cooperative_sticky`
411
439
 
412
- These map to Kafka's corresponding https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html[`ConsumerPartitionAssignor`]
440
+ These map to Kafka's corresponding https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.html[`ConsumerPartitionAssignor`]
413
441
  implementations.
414
442
 
415
443
  [id="plugins-{type}s-{plugin}-poll_timeout_ms"]
@@ -1,6 +1,9 @@
1
+ :integration: kafka
1
2
  :plugin: kafka
2
3
  :type: output
3
4
  :default_codec: plain
5
+ :kafka_client: 2.4
6
+ :kafka_client_doc: 24
4
7
 
5
8
  ///////////////////////////////////////////
6
9
  START - GENERATED VARIABLES, DO NOT EDIT!
@@ -17,15 +20,20 @@ END - GENERATED VARIABLES, DO NOT EDIT!
17
20
 
18
21
  === Kafka output plugin
19
22
 
20
- include::{include_path}/plugin_header.asciidoc[]
23
+ include::{include_path}/plugin_header-integration.asciidoc[]
21
24
 
22
25
  ==== Description
23
26
 
24
27
  Write events to a Kafka topic.
25
28
 
26
- This plugin uses Kafka Client 2.3.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka compatibility reference]. If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
29
+ This plugin uses Kafka Client {kafka_client}. For broker compatibility, see the
30
+ official
31
+ https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix[Kafka
32
+ compatibility reference]. If the linked compatibility wiki is not up-to-date,
33
+ please contact Kafka support/community to confirm compatibility.
27
34
 
28
- If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
35
+ If you require features not yet available in this plugin (including client
36
+ version upgrades), please file an issue with details about what you need.
29
37
 
30
38
  This output supports connecting to Kafka over:
31
39
 
@@ -36,9 +44,12 @@ By default security is disabled but can be turned on as needed.
36
44
 
37
45
  The only required configuration is the topic_id.
38
46
 
39
- The default codec is plain. Logstash will encode your events with not only the message field but also with a timestamp and hostname.
47
+ The default codec is plain. Logstash will encode your events with not only the
48
+ message field but also with a timestamp and hostname.
49
+
50
+ If you want the full content of your events to be sent as json, you should set
51
+ the codec in the output configuration like this:
40
52
 
41
- If you want the full content of your events to be sent as json, you should set the codec in the output configuration like this:
42
53
  [source,ruby]
43
54
  output {
44
55
  kafka {
@@ -47,9 +58,11 @@ If you want the full content of your events to be sent as json, you should set t
47
58
  }
48
59
  }
49
60
 
50
- For more information see https://kafka.apache.org/24/documentation.html#theproducer
61
+ For more information see
62
+ https://kafka.apache.org/{kafka_client_doc}/documentation.html#theproducer
51
63
 
52
- Kafka producer configuration: https://kafka.apache.org/24/documentation.html#producerconfigs
64
+ Kafka producer configuration:
65
+ https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs
53
66
 
54
67
  [id="plugins-{type}s-{plugin}-options"]
55
68
  ==== Kafka Output Configuration Options
@@ -58,7 +71,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
58
71
 
59
72
  NOTE: Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting,
60
73
  and might change if Kafka's producer defaults change.
61
- See the https://kafka.apache.org/24/documentation for more details.
74
+ See the https://kafka.apache.org/{kafka_client_doc}/documentation for more details.
62
75
 
63
76
  [cols="<,<,<",options="header",]
64
77
  |=======================================================================
@@ -115,10 +128,13 @@ output plugins.
115
128
  The number of acknowledgments the producer requires the leader to have received
116
129
  before considering a request complete.
117
130
 
118
- acks=0, the producer will not wait for any acknowledgment from the server at all.
119
- acks=1, This will mean the leader will write the record to its local log but
120
- will respond without awaiting full acknowledgement from all followers.
121
- acks=all, This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
131
+ `acks=0`. The producer will not wait for any acknowledgment from the server.
132
+
133
+ `acks=1`. The leader will write the record to its local log, but will respond
134
+ without waiting for full acknowledgement from all followers.
135
+
136
+ `acks=all`. The leader will wait for the full set of in-sync replicas before
137
+ acknowledging the record.
122
138
 
123
139
  [id="plugins-{type}s-{plugin}-batch_size"]
124
140
  ===== `batch_size`
@@ -154,11 +170,12 @@ The total bytes of memory the producer can use to buffer records waiting to be s
154
170
  ===== `client_dns_lookup`
155
171
 
156
172
  * Value type is <<string,string>>
173
+ * Valid options are `use_all_dns_ips`, `resolve_canonical_bootstrap_servers_only`, `default`
157
174
  * Default value is `"default"`
158
175
 
159
- How DNS lookups should be done. If set to `use_all_dns_ips`, when the lookup returns multiple
160
- IP addresses for a hostname, they will all be attempted to connect to before failing the
161
- connection. If the value is `resolve_canonical_bootstrap_servers_only` each entry will be
176
+ Controls how DNS lookups are done. If set to `use_all_dns_ips`, Logstash tries
177
+ all IP addresses returned for a hostname before failing the connection.
178
+ If set to `resolve_canonical_bootstrap_servers_only`, each entry will be
162
179
  resolved and expanded into a list of canonical names.
163
180
 
164
181
  [id="plugins-{type}s-{plugin}-client_id"]
@@ -178,7 +195,7 @@ ip/port by allowing a logical application name to be included with the request
178
195
  * Default value is `"none"`
179
196
 
180
197
  The compression type for all data generated by the producer.
181
- The default is none (i.e. no compression). Valid values are none, gzip, or snappy.
198
+ The default is none (i.e. no compression). Valid values are none, gzip, snappy, or lz4.
182
199
 
183
200
  [id="plugins-{type}s-{plugin}-jaas_path"]
184
201
  ===== `jaas_path`
@@ -323,6 +340,15 @@ Kafka down, etc).
323
340
 
324
341
  A value less than zero is a configuration error.
325
342
 
343
+ Starting with version 10.5.0, this plugin will only retry exceptions that are a subclass of
344
+ https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/common/errors/RetriableException.html[RetriableException]
345
+ and
346
+ https://kafka.apache.org/{kafka_client_doc}/javadoc/org/apache/kafka/common/errors/InterruptException.html[InterruptException].
347
+ If producing a message throws any other exception, an error is logged and the message is dropped without retrying.
348
+ This prevents the Logstash pipeline from hanging indefinitely.
349
+
350
+ In versions prior to 10.5.0, any exception is retried indefinitely unless the `retries` option is configured.
351
+
326
352
  [id="plugins-{type}s-{plugin}-retry_backoff_ms"]
327
353
  ===== `retry_backoff_ms`
328
354
 
@@ -3,6 +3,7 @@ require 'logstash/inputs/base'
3
3
  require 'stud/interval'
4
4
  require 'java'
5
5
  require 'logstash-integration-kafka_jars.rb'
6
+ require 'logstash/plugin_mixins/kafka_support'
6
7
 
7
8
  # This input will read events from a Kafka topic. It uses the 0.10 version of
8
9
  # the consumer API provided by Kafka to read messages from the broker.
@@ -48,6 +49,9 @@ require 'logstash-integration-kafka_jars.rb'
48
49
  # Kafka consumer configuration: http://kafka.apache.org/documentation.html#consumerconfigs
49
50
  #
50
51
  class LogStash::Inputs::Kafka < LogStash::Inputs::Base
52
+
53
+ include LogStash::PluginMixins::KafkaSupport
54
+
51
55
  config_name 'kafka'
52
56
 
53
57
  default :codec, 'plain'
@@ -114,6 +118,11 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
114
118
  # `session.timeout.ms`, but typically should be set no higher than 1/3 of that value.
115
119
  # It can be adjusted even lower to control the expected time for normal rebalances.
116
120
  config :heartbeat_interval_ms, :validate => :number, :default => 3000 # Kafka default
121
+ # Controls how to read messages written transactionally. If set to read_committed, consumer.poll()
122
+ # will only return transactional messages which have been committed. If set to read_uncommitted'
123
+ # (the default), consumer.poll() will return all messages, even transactional messages which have
124
+ # been aborted. Non-transactional messages will be returned unconditionally in either mode.
125
+ config :isolation_level, :validate => ["read_uncommitted", "read_committed"], :default => "read_uncommitted" # Kafka default
117
126
  # Java Class used to deserialize the record's key
118
127
  config :key_deserializer_class, :validate => :string, :default => "org.apache.kafka.common.serialization.StringDeserializer"
119
128
  # The maximum delay between invocations of poll() when using consumer group management. This places
@@ -311,6 +320,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
311
320
  props.put(kafka::FETCH_MIN_BYTES_CONFIG, fetch_min_bytes.to_s) unless fetch_min_bytes.nil?
312
321
  props.put(kafka::GROUP_ID_CONFIG, group_id)
313
322
  props.put(kafka::HEARTBEAT_INTERVAL_MS_CONFIG, heartbeat_interval_ms.to_s) unless heartbeat_interval_ms.nil?
323
+ props.put(kafka::ISOLATION_LEVEL_CONFIG, isolation_level)
314
324
  props.put(kafka::KEY_DESERIALIZER_CLASS_CONFIG, key_deserializer_class)
315
325
  props.put(kafka::MAX_PARTITION_FETCH_BYTES_CONFIG, max_partition_fetch_bytes.to_s) unless max_partition_fetch_bytes.nil?
316
326
  props.put(kafka::MAX_POLL_RECORDS_CONFIG, max_poll_records.to_s) unless max_poll_records.nil?
@@ -364,29 +374,4 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
364
374
  end
365
375
  end
366
376
 
367
- def set_trustore_keystore_config(props)
368
- props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
369
- props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
370
- props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
371
-
372
- # Client auth stuff
373
- props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
374
- props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
375
- props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
376
- props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
377
- props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
378
- end
379
-
380
- def set_sasl_config(props)
381
- java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
382
- java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
383
-
384
- props.put("sasl.mechanism", sasl_mechanism)
385
- if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
386
- raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
387
- end
388
-
389
- props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
390
- props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
391
- end
392
377
  end #class LogStash::Inputs::Kafka
@@ -2,6 +2,7 @@ require 'logstash/namespace'
2
2
  require 'logstash/outputs/base'
3
3
  require 'java'
4
4
  require 'logstash-integration-kafka_jars.rb'
5
+ require 'logstash/plugin_mixins/kafka_support'
5
6
 
6
7
  # Write events to a Kafka topic. This uses the Kafka Producer API to write messages to a topic on
7
8
  # the broker.
@@ -50,6 +51,8 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
50
51
 
51
52
  java_import org.apache.kafka.clients.producer.ProducerRecord
52
53
 
54
+ include LogStash::PluginMixins::KafkaSupport
55
+
53
56
  declare_threadsafe!
54
57
 
55
58
  config_name 'kafka'
@@ -236,7 +239,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
236
239
  remaining = @retries
237
240
 
238
241
  while batch.any?
239
- if !remaining.nil?
242
+ unless remaining.nil?
240
243
  if remaining < 0
241
244
  # TODO(sissel): Offer to DLQ? Then again, if it's a transient fault,
242
245
  # DLQing would make things worse (you dlq data that would be successful
@@ -255,27 +258,39 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
255
258
  begin
256
259
  # send() can throw an exception even before the future is created.
257
260
  @producer.send(record)
258
- rescue org.apache.kafka.common.errors.TimeoutException => e
259
- failures << record
260
- nil
261
- rescue org.apache.kafka.common.errors.InterruptException => e
261
+ rescue org.apache.kafka.common.errors.InterruptException,
262
+ org.apache.kafka.common.errors.RetriableException => e
263
+ logger.info("producer send failed, will retry sending", :exception => e.class, :message => e.message)
262
264
  failures << record
263
265
  nil
264
- rescue org.apache.kafka.common.errors.SerializationException => e
265
- # TODO(sissel): Retrying will fail because the data itself has a problem serializing.
266
- # TODO(sissel): Let's add DLQ here.
267
- failures << record
266
+ rescue org.apache.kafka.common.KafkaException => e
267
+ # This error is not retriable, drop event
268
+ # TODO: add DLQ support
269
+ logger.warn("producer send failed, dropping record",:exception => e.class, :message => e.message,
270
+ :record_value => record.value)
268
271
  nil
269
272
  end
270
- end.compact
273
+ end
271
274
 
272
275
  futures.each_with_index do |future, i|
273
- begin
274
- result = future.get()
275
- rescue => e
276
- # TODO(sissel): Add metric to count failures, possibly by exception type.
277
- logger.warn("producer send failed", :exception => e.class, :message => e.message)
278
- failures << batch[i]
276
+ # We cannot skip nils using `futures.compact` because then our index `i` will not align with `batch`
277
+ unless future.nil?
278
+ begin
279
+ future.get
280
+ rescue java.util.concurrent.ExecutionException => e
281
+ # TODO(sissel): Add metric to count failures, possibly by exception type.
282
+ if e.get_cause.is_a? org.apache.kafka.common.errors.RetriableException or
283
+ e.get_cause.is_a? org.apache.kafka.common.errors.InterruptException
284
+ logger.info("producer send failed, will retry sending", :exception => e.cause.class,
285
+ :message => e.cause.message)
286
+ failures << batch[i]
287
+ elsif e.get_cause.is_a? org.apache.kafka.common.KafkaException
288
+ # This error is not retriable, drop event
289
+ # TODO: add DLQ support
290
+ logger.warn("producer send failed, dropping record", :exception => e.cause.class,
291
+ :message => e.cause.message, :record_value => batch[i].value)
292
+ end
293
+ end
279
294
  end
280
295
  end
281
296
 
@@ -377,35 +392,4 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
377
392
  end
378
393
  end
379
394
 
380
- def set_trustore_keystore_config(props)
381
- unless ssl_endpoint_identification_algorithm.to_s.strip.empty?
382
- if ssl_truststore_location.nil?
383
- raise LogStash::ConfigurationError, "ssl_truststore_location must be set when SSL is enabled"
384
- end
385
- props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
386
- props.put("ssl.truststore.location", ssl_truststore_location)
387
- props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
388
- end
389
-
390
- # Client auth stuff
391
- props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
392
- props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
393
- props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
394
- props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
395
- props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
396
- end
397
-
398
- def set_sasl_config(props)
399
- java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
400
- java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
401
-
402
- props.put("sasl.mechanism",sasl_mechanism)
403
- if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
404
- raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
405
- end
406
-
407
- props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
408
- props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
409
- end
410
-
411
395
  end #class LogStash::Outputs::Kafka
@@ -0,0 +1,29 @@
1
+ module LogStash module PluginMixins module KafkaSupport
2
+
3
+ def set_trustore_keystore_config(props)
4
+ props.put("ssl.truststore.type", ssl_truststore_type) unless ssl_truststore_type.nil?
5
+ props.put("ssl.truststore.location", ssl_truststore_location) unless ssl_truststore_location.nil?
6
+ props.put("ssl.truststore.password", ssl_truststore_password.value) unless ssl_truststore_password.nil?
7
+
8
+ # Client auth stuff
9
+ props.put("ssl.keystore.type", ssl_keystore_type) unless ssl_keystore_type.nil?
10
+ props.put("ssl.key.password", ssl_key_password.value) unless ssl_key_password.nil?
11
+ props.put("ssl.keystore.location", ssl_keystore_location) unless ssl_keystore_location.nil?
12
+ props.put("ssl.keystore.password", ssl_keystore_password.value) unless ssl_keystore_password.nil?
13
+ props.put("ssl.endpoint.identification.algorithm", ssl_endpoint_identification_algorithm) unless ssl_endpoint_identification_algorithm.nil?
14
+ end
15
+
16
+ def set_sasl_config(props)
17
+ java.lang.System.setProperty("java.security.auth.login.config", jaas_path) unless jaas_path.nil?
18
+ java.lang.System.setProperty("java.security.krb5.conf", kerberos_config) unless kerberos_config.nil?
19
+
20
+ props.put("sasl.mechanism", sasl_mechanism)
21
+ if sasl_mechanism == "GSSAPI" && sasl_kerberos_service_name.nil?
22
+ raise LogStash::ConfigurationError, "sasl_kerberos_service_name must be specified when SASL mechanism is GSSAPI"
23
+ end
24
+
25
+ props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
26
+ props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
27
+ end
28
+
29
+ end end end
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-integration-kafka'
3
- s.version = '10.3.0'
3
+ s.version = '10.5.3'
4
4
  s.licenses = ['Apache-2.0']
5
5
  s.summary = "Integration with Kafka - input and output plugins"
6
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+
File without changes
@@ -50,20 +50,22 @@ describe "outputs/kafka" do
50
50
  kafka.multi_receive([event])
51
51
  end
52
52
 
53
- it 'should raise config error when truststore location is not set and ssl is enabled' do
53
+ it 'should not raise config error when truststore location is not set and ssl is enabled' do
54
54
  kafka = LogStash::Outputs::Kafka.new(simple_kafka_config.merge("security_protocol" => "SSL"))
55
- expect { kafka.register }.to raise_error(LogStash::ConfigurationError, /ssl_truststore_location must be set when SSL is enabled/)
55
+ expect(org.apache.kafka.clients.producer.KafkaProducer).to receive(:new)
56
+ expect { kafka.register }.to_not raise_error
56
57
  end
57
58
  end
58
59
 
59
- context "when KafkaProducer#send() raises an exception" do
60
+ context "when KafkaProducer#send() raises a retriable exception" do
60
61
  let(:failcount) { (rand * 10).to_i }
61
62
  let(:sendcount) { failcount + 1 }
62
63
 
63
64
  let(:exception_classes) { [
64
65
  org.apache.kafka.common.errors.TimeoutException,
66
+ org.apache.kafka.common.errors.DisconnectException,
67
+ org.apache.kafka.common.errors.CoordinatorNotAvailableException,
65
68
  org.apache.kafka.common.errors.InterruptException,
66
- org.apache.kafka.common.errors.SerializationException
67
69
  ] }
68
70
 
69
71
  before do
@@ -88,6 +90,37 @@ describe "outputs/kafka" do
88
90
  end
89
91
  end
90
92
 
93
+ context "when KafkaProducer#send() raises a non-retriable exception" do
94
+ let(:failcount) { (rand * 10).to_i }
95
+
96
+ let(:exception_classes) { [
97
+ org.apache.kafka.common.errors.SerializationException,
98
+ org.apache.kafka.common.errors.RecordTooLargeException,
99
+ org.apache.kafka.common.errors.InvalidTopicException
100
+ ] }
101
+
102
+ before do
103
+ count = 0
104
+ expect_any_instance_of(org.apache.kafka.clients.producer.KafkaProducer).to receive(:send)
105
+ .exactly(1).times
106
+ .and_wrap_original do |m, *args|
107
+ if count < failcount # fail 'failcount' times in a row.
108
+ count += 1
109
+ # Pick an exception at random
110
+ raise exception_classes.shuffle.first.new("injected exception for testing")
111
+ else
112
+ m.call(*args) # call original
113
+ end
114
+ end
115
+ end
116
+
117
+ it "should not retry" do
118
+ kafka = LogStash::Outputs::Kafka.new(simple_kafka_config)
119
+ kafka.register
120
+ kafka.multi_receive([event])
121
+ end
122
+ end
123
+
91
124
  context "when a send fails" do
92
125
  context "and the default retries behavior is used" do
93
126
  # Fail this many times and then finally succeed.
@@ -107,7 +140,7 @@ describe "outputs/kafka" do
107
140
  # inject some failures.
108
141
 
109
142
  # Return a custom Future that will raise an exception to simulate a Kafka send() problem.
110
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
143
+ future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
111
144
  future.run
112
145
  future
113
146
  else
@@ -129,7 +162,7 @@ describe "outputs/kafka" do
129
162
  .once
130
163
  .and_wrap_original do |m, *args|
131
164
  # Always fail.
132
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
165
+ future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
133
166
  future.run
134
167
  future
135
168
  end
@@ -143,7 +176,7 @@ describe "outputs/kafka" do
143
176
  .once
144
177
  .and_wrap_original do |m, *args|
145
178
  # Always fail.
146
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
179
+ future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
147
180
  future.run
148
181
  future
149
182
  end
@@ -164,7 +197,7 @@ describe "outputs/kafka" do
164
197
  .at_most(max_sends).times
165
198
  .and_wrap_original do |m, *args|
166
199
  # Always fail.
167
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
200
+ future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
168
201
  future.run
169
202
  future
170
203
  end
@@ -175,10 +208,10 @@ describe "outputs/kafka" do
175
208
 
176
209
  it 'should only sleep retries number of times' do
177
210
  expect_any_instance_of(org.apache.kafka.clients.producer.KafkaProducer).to receive(:send)
178
- .at_most(max_sends)
211
+ .at_most(max_sends).times
179
212
  .and_wrap_original do |m, *args|
180
213
  # Always fail.
181
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
214
+ future = java.util.concurrent.FutureTask.new { raise org.apache.kafka.common.errors.TimeoutException.new("Failed") }
182
215
  future.run
183
216
  future
184
217
  end
@@ -193,21 +226,31 @@ describe "outputs/kafka" do
193
226
  context 'when ssl endpoint identification disabled' do
194
227
 
195
228
  let(:config) do
196
- simple_kafka_config.merge('ssl_endpoint_identification_algorithm' => '', 'security_protocol' => 'SSL')
229
+ simple_kafka_config.merge(
230
+ 'security_protocol' => 'SSL',
231
+ 'ssl_endpoint_identification_algorithm' => '',
232
+ 'ssl_truststore_location' => truststore_path,
233
+ )
234
+ end
235
+
236
+ let(:truststore_path) do
237
+ File.join(File.dirname(__FILE__), '../../fixtures/trust-store_stub.jks')
197
238
  end
198
239
 
199
240
  subject { LogStash::Outputs::Kafka.new(config) }
200
241
 
201
- it 'does not configure truststore' do
242
+ it 'sets empty ssl.endpoint.identification.algorithm' do
202
243
  expect(org.apache.kafka.clients.producer.KafkaProducer).
203
- to receive(:new).with(hash_excluding('ssl.truststore.location' => anything))
244
+ to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
204
245
  subject.register
205
246
  end
206
247
 
207
- it 'sets empty ssl.endpoint.identification.algorithm' do
248
+ it 'configures truststore' do
208
249
  expect(org.apache.kafka.clients.producer.KafkaProducer).
209
- to receive(:new).with(hash_including('ssl.endpoint.identification.algorithm' => ''))
250
+ to receive(:new).with(hash_including('ssl.truststore.location' => truststore_path))
210
251
  subject.register
211
252
  end
253
+
212
254
  end
255
+
213
256
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-integration-kafka
3
3
  version: !ruby/object:Gem::Version
4
- version: 10.3.0
4
+ version: 10.5.3
5
5
  platform: java
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-06-18 00:00:00.000000000 Z
11
+ date: 2020-10-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -183,7 +183,9 @@ files:
183
183
  - lib/logstash-integration-kafka_jars.rb
184
184
  - lib/logstash/inputs/kafka.rb
185
185
  - lib/logstash/outputs/kafka.rb
186
+ - lib/logstash/plugin_mixins/kafka_support.rb
186
187
  - logstash-integration-kafka.gemspec
188
+ - spec/fixtures/trust-store_stub.jks
187
189
  - spec/integration/inputs/kafka_spec.rb
188
190
  - spec/integration/outputs/kafka_spec.rb
189
191
  - spec/unit/inputs/kafka_spec.rb
@@ -222,6 +224,7 @@ signing_key:
222
224
  specification_version: 4
223
225
  summary: Integration with Kafka - input and output plugins
224
226
  test_files:
227
+ - spec/fixtures/trust-store_stub.jks
225
228
  - spec/integration/inputs/kafka_spec.rb
226
229
  - spec/integration/outputs/kafka_spec.rb
227
230
  - spec/unit/inputs/kafka_spec.rb