logstash-integration-kafka 11.8.6-java → 12.0.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 01affd92375e003285e4b4bd03a4c14fe35ec6712c215b5f06a98ff2c56780f9
-  data.tar.gz: 2da5a10ef322b92e9fe1ba17e5d1df00244a07067746a723ecc7e5751ef79862
+  metadata.gz: 744abb784463992c54a691cf16cbab0eca25f9b5cc73a54d9808f15bca223021
+  data.tar.gz: d35f68e589ee17adb865b7edcb1c9627b1345e9de93c5037828c15e8e83defdb
 SHA512:
-  metadata.gz: 830fd54d12d41b4eefbb510107c031d73faf3db1ec240b2f7a9eed3a1cf74ca9f53e23f1960f028cba722d6a1ef203a8a22179093ae7fea3e3ce33e84f810654
-  data.tar.gz: 3d8563cd037862c2635223c6e1e40752a3de3228ff375a1f5d065f7498733d1c40c533d3dd8efce9c7a67356a5e45e6e428580edd5004d838fc4bdfbe36cc286
+  metadata.gz: b85e870af8ae8dc030320ecd0c9ad312d72cceacb64b2e0ac313ca3629b3a1c24a3029f1512485b79ccd68e272eccddfee1ac32a314dc562af7cc1762b82aa53
+  data.tar.gz: 89fe6266982b250eb000650c91d8fdfc653cfc02d9cdf570c743edd5d7290ffd1735b0fbc3b68b018fb65215b67a909cdc5bb1dfb05e025256b35de0cca6b1af

data/CHANGELOG.md

@@ -1,24 +1,9 @@
-## 11.8.6
-  - [DOC] Add info about Kafka timestamp behavior  [#241](https://github.com/logstash-plugins/logstash-integration-kafka/pull/241)
-
-## 11.8.5
-  - Redact `sasl_jaas_config` to prevent credentials from appearing in debug logs. [#237](https://github.com/logstash-plugins/logstash-integration-kafka/pull/237)
-
-## 11.8.4
-  - Upgrade kafka-avro-serializer dependency [#225](https://github.com/logstash-plugins/logstash-integration-kafka/pull/225)
-
-## 11.8.3
-  - Re-packaging the plugin [#223](https://github.com/logstash-plugins/logstash-integration-kafka/pull/223)
-
-## 11.8.2
-  - Upgrade transitive `org.apache.commons:commons-lang3` dependency [#217](https://github.com/logstash-plugins/logstash-integration-kafka/pull/217)
-
-## 11.8.1
-  - Upgrade lz4 dependency [#213](https://github.com/logstash-plugins/logstash-integration-kafka/pull/213)
-
-## 11.8.0
-  - Deprecate partitioner `default` and `uniform_sticky` options [#206](https://github.com/logstash-plugins/logstash-integration-kafka/pull/206)
-      Both options are deprecated in Kafka client 3 and will be removed in the plugin 12.0.0.
+## 12.0.0
+  - Update kafka client to 4.1.0 and transitive dependencies [#205](https://github.com/logstash-plugins/logstash-integration-kafka/pull/205)
+    - Breaking Change: partitioner options `default` and `uniform_sticky` are removed
+    - `linger_ms` default value changed from 0 to 5
+  - Add `group_protocols` options for configuring Kafka consumer rebalance protocol
+    - Setting `group_protocols => consumer` opts in to the new consumer group protocol
 
 ## 11.7.0
   - Add `reconnect_backoff_max_ms` option for configuring kafka client [#204](https://github.com/logstash-plugins/logstash-integration-kafka/pull/204)

data/docs/index.asciidoc

@@ -1,7 +1,7 @@
 :plugin: kafka
 :type: integration
 :no_codec:
-:kafka_client: 3.9.2
+:kafka_client: 4.1.0
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!

data/docs/input-kafka.asciidoc

@@ -2,8 +2,8 @@
 :plugin: kafka
 :type: input
 :default_codec: plain
-:kafka_client: 3.9.2
-:kafka_client_doc: 39
+:kafka_client: 4.1.0
+:kafka_client_doc: 41
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -132,6 +132,7 @@ See the https://kafka.apache.org/{kafka_client_doc}/documentation for more detai
 | <<plugins-{type}s-{plugin}-fetch_min_bytes>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-group_id>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-group_instance_id>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-group_protocol>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-heartbeat_interval_ms>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-isolation_level>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-jaas_path>> |a valid filesystem path|No
@@ -412,6 +413,21 @@ You can set this value to use information such as a hostname, an IP, or anything
 NOTE: In cases when multiple threads are configured and `consumer_threads` is greater than one, a suffix is appended to
 the `group_instance_id` to avoid collisions.
 
+[id="plugins-{type}s-{plugin}-group_protocol"]
+===== `group_protocol`
+
+* Value can be either of: `classic`, `consumer`
+* Default value is `classic`.
+
+Specifies the consumer group rebalance protocol used by the Kafka client.
+
+`classic` is the default protocol. During a rebalance, all consumer instances pause message processing until partition assignments are complete.
+
+`consumer` is an incremental rebalance protocol introduced in Kafka 4. It avoids global synchronization barriers by only pausing partitions that are reassigned.
+When using `consumer`, the following settings **cannot be configured**:
+`partition_assignment_strategy`, `heartbeat_interval_ms`, and `session_timeout_ms`.
+
+
 [id="plugins-{type}s-{plugin}-heartbeat_interval_ms"]
 ===== `heartbeat_interval_ms` 
 

data/docs/output-kafka.asciidoc

@@ -2,8 +2,8 @@
 :plugin: kafka
 :type: output
 :default_codec: plain
-:kafka_client: 3.9.2
-:kafka_client_doc: 39
+:kafka_client: 4.1.0
+:kafka_client_doc: 41
 
 ///////////////////////////////////////////
 START - GENERATED VARIABLES, DO NOT EDIT!
@@ -66,24 +66,6 @@ https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs
 
 NOTE:  This plugin does not support using a proxy when communicating to the Kafka broker.
 
-.Kafka timestamps and Logstash
-****
-* Kafka 3.6+ introduces stricter timestamp validation with the introduction of two new broker/topic-level properties: https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html#message-timestamp-before-max-ms[log.message.timestamp.before.max.ms] and 
-https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html#message-timestamp-after-max-ms[log.message.timestamp.after.max.ms].
-+
-These properties limit the time difference between the message timestamp (from Logstash) and the Kafka broker receive time. 
-Messages can be rejected if the values are exceeded and `log.message.timestamp.type=CreateTime` is set. 
-+
-These checks are ignored if `log.message.timestamp.type=LogAppendTime` is set. 
-
-* For Kafka version 0.10.0.0+ the message creation timestamp is set by Logstash and equals the initial timestamp of the event. 
-This behavior affects Kafka’s retention policy. 
-For example, if a Logstash event was created two weeks ago and Kafka retention is set to seven days, then when the message arrives in Kafka today it may be discarded immediately, because its timestamp is older than seven days.
-+
-You can change this behavior by setting timestamps on message arrival instead.
-  The message is not discarded but kept for 7 more days. Set `log.message.timestamp.type` to `LogAppendTime` (default `CreateTime`) in your Kafka configuration.
-****
-
 [id="plugins-{type}s-{plugin}-aws_msk_iam_auth"]
 ==== AWS MSK IAM authentication
 If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM.
@@ -303,7 +285,7 @@ Serializer class for the key of the message
 ===== `linger_ms` 
 
   * Value type is <<number,number>>
-  * Default value is `0`
+  * Default value is `5`
 
 The producer groups together any records that arrive in between request
 transmissions into a single batched request. Normally this occurs only under
@@ -369,13 +351,9 @@ The max time in milliseconds before a metadata refresh is forced.
 
 By not setting this value, the plugin uses the built-in partitioning strategy provided by the Kafka client. Read more about the "partitioner.class" on the Kafka documentation.
 
-Available options are as follows:
+Available option is as follows:
 
-* `default` hashes the `message_key` of an event to get the partition. When no message key is present, the plugin picks a partition in a round-robin fashion. Please note that this is a different strategy than the one used when `partitioner` is left unset.
 * `round_robin` distributes writes to all partitions equally, regardless of `message_key`
-* `uniform_sticky` hashes the `message_key` of an event to get the partition. When no message key is present, the plugin sticks to a partition for the duration of a batch than randomly picks a new one.
-
-NOTE: `default` and `uniform_sticky` are deprecated and will be removed in `12.0.0`.
 
 [id="plugins-{type}s-{plugin}-receive_buffer_bytes"]
 ===== `receive_buffer_bytes` 

data/lib/logstash/inputs/kafka.rb

@@ -129,6 +129,12 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # consumer crated by each thread an artificial suffix is appended to the user provided `group_instance_id`
   # to avoid clashing.
   config :group_instance_id, :validate => :string
+  # `classic` is the "stop-the-world" rebalances.
+  #   Any consumer restart or failure triggers a full-group rebalance, pausing processing for all consumers.
+  # `consumer` is an incremental rebalance protocol that avoids global sync barriers,
+  #   pausing only the partitions that are reassigned.
+  #   It cannot set along with `partition_assignment_strategy`, `heartbeat_interval_ms` and `session_timeout_ms`
+  config :group_protocol, :validate => ["classic", "consumer"], :default => "classic"
   # The expected time between heartbeats to the consumer coordinator. Heartbeats are used to ensure 
   # that the consumer's session stays active and to facilitate rebalancing when new
   # consumers join or leave the group. The value must be set lower than
@@ -254,7 +260,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
   # different JVM instances.
   config :jaas_path, :validate => :path
   # JAAS configuration settings. This allows JAAS config to be a part of the plugin configuration and allows for different JAAS configuration per each plugin config.
-  config :sasl_jaas_config, :validate => :password
+  config :sasl_jaas_config, :validate => :string
   # Optional path to kerberos config file. This is krb5.conf style as detailed in https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html
   config :kerberos_config, :validate => :path
   # Option to add Kafka metadata like topic, message size and header key values to the event.
@@ -293,6 +299,8 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
     reassign_dns_lookup
     @pattern ||= java.util.regex.Pattern.compile(@topics_pattern) unless @topics_pattern.nil?
     check_schema_registry_parameters
+
+    set_group_protocol!
   end
 
   METADATA_NONE     = Set[].freeze
@@ -450,6 +458,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put(kafka::FETCH_MIN_BYTES_CONFIG, fetch_min_bytes.to_s) unless fetch_min_bytes.nil?
       props.put(kafka::GROUP_ID_CONFIG, group_id)
       props.put(kafka::GROUP_INSTANCE_ID_CONFIG, group_instance_id) unless group_instance_id.nil?
+      props.put(kafka::GROUP_PROTOCOL_CONFIG, group_protocol)
       props.put(kafka::HEARTBEAT_INTERVAL_MS_CONFIG, heartbeat_interval_ms.to_s) unless heartbeat_interval_ms.nil?
       props.put(kafka::ISOLATION_LEVEL_CONFIG, isolation_level)
       props.put(kafka::KEY_DESERIALIZER_CLASS_CONFIG, key_deserializer_class)
@@ -471,7 +480,7 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
       props.put("security.protocol", security_protocol) unless security_protocol.nil?
       if schema_registry_url
         props.put(kafka::VALUE_DESERIALIZER_CLASS_CONFIG, Java::io.confluent.kafka.serializers.KafkaAvroDeserializer.java_class)
-        serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+        serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig
         props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, schema_registry_url.uri.to_s)
         if schema_registry_proxy && !schema_registry_proxy.empty?
           props.put(serdes_config::PROXY_HOST, @schema_registry_proxy_host)
@@ -513,6 +522,25 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base
     end
   end
 
+  # In order to use group_protocol => consumer, heartbeat_interval_ms, session_timeout_ms and partition_assignment_strategy need to be unset
+  # If any of these are not using the default value of the plugin, we raise a configuration error
+  def set_group_protocol!
+    return unless group_protocol == "consumer"
+
+    heartbeat_overridden = heartbeat_interval_ms != self.class.get_config.dig("heartbeat_interval_ms", :default)
+    session_overridden   = session_timeout_ms != self.class.get_config.dig("session_timeout_ms", :default)
+    strategy_defined     = !partition_assignment_strategy.nil?
+
+    if strategy_defined || heartbeat_overridden || session_overridden
+      raise LogStash::ConfigurationError, "group_protocol cannot be set to 'consumer' "\
+        "when any of partition_assignment_strategy, heartbeat_interval_ms or session_timeout_ms is set"
+    end
+
+    @heartbeat_interval_ms = nil
+    @session_timeout_ms = nil
+    logger.debug("Settings 'heartbeat_interval_ms' and 'session_timeout_ms' have been reset since 'group_protocol' is configured as 'consumer'")
+  end
+
   def partition_assignment_strategy_class
     case partition_assignment_strategy
     when 'range'

data/lib/logstash/outputs/kafka.rb

@@ -101,7 +101,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   # This setting accomplishes this by adding a small amount of artificial delay—that is,
   # rather than immediately sending out a record the producer will wait for up to the given delay
   # to allow other records to be sent so that the sends can be batched together.
-  config :linger_ms, :validate => :number, :default => 0 # Kafka default
+  config :linger_ms, :validate => :number, :default => 5 # Kafka default
   # The maximum size of a request
   config :max_request_size, :validate => :number, :default => 1_048_576 # (1MB) Kafka default
   # The key for the message
@@ -110,7 +110,7 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   config :message_headers, :validate => :hash, :default => {}
   # the timeout setting for initial metadata request to fetch topic metadata.
   config :metadata_fetch_timeout_ms, :validate => :number, :default => 60_000
-  # Partitioner to use - can be `default`, `uniform_sticky`, `round_robin` or a fully qualified class name of a custom partitioner.
+  # Partitioner to use - can be `round_robin` or a fully qualified class name of a custom partitioner.
   config :partitioner, :validate => :string
   # The size of the TCP receive buffer to use when reading data
   config :receive_buffer_bytes, :validate => :number, :default => 32_768 # (32KB) Kafka default
@@ -369,8 +369,8 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
       props.put(kafka::LINGER_MS_CONFIG, linger_ms.to_s)
       props.put(kafka::MAX_REQUEST_SIZE_CONFIG, max_request_size.to_s)
       props.put(kafka::METADATA_MAX_AGE_CONFIG, metadata_max_age_ms.to_s) unless metadata_max_age_ms.nil?
-      unless partitioner.nil?
-        props.put(kafka::PARTITIONER_CLASS_CONFIG, partitioner = partitioner_class)
+      partitioner_class&.tap do |partitioner|
+        props.put(kafka::PARTITIONER_CLASS_CONFIG, partitioner)
         logger.debug('producer configured using partitioner', :partitioner_class => partitioner)
       end
       props.put(kafka::RECEIVE_BUFFER_CONFIG, receive_buffer_bytes.to_s) unless receive_buffer_bytes.nil?
@@ -405,27 +405,12 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base
   end
 
   def partitioner_class
-    case partitioner
-    when 'round_robin'
-      'org.apache.kafka.clients.producer.RoundRobinPartitioner'
-    when 'uniform_sticky'
-      log_partitioner_warning(partitioner, 'UniformStickyPartitioner')
-      'org.apache.kafka.clients.producer.UniformStickyPartitioner'
-    when 'default'
-      log_partitioner_warning(partitioner, 'DefaultPartitioner')
-      'org.apache.kafka.clients.producer.internals.DefaultPartitioner'
-    else
-      unless partitioner.index('.')
-        raise LogStash::ConfigurationError, "unsupported partitioner: #{partitioner.inspect}"
-      end
-      partitioner # assume a fully qualified class-name
-    end
-  end
+    return nil if partitioner.nil?
+    return 'org.apache.kafka.clients.producer.RoundRobinPartitioner' if partitioner == 'round_robin'
+
+    raise LogStash::ConfigurationError, "unsupported partitioner: #{partitioner.inspect}" unless partitioner.include?('.')
 
-  def log_partitioner_warning(partitioner, class_name)
-    deprecation_logger.deprecated("Producer `partitioner` is configured with the deprecated option `#{partitioner}`. " \
-                  "#{class_name} is removed in kafka-client 4.0 and the `#{partitioner}` option will be removed in the plugin 12.0.0. "\
-                  'Please update your configuration to use `round_robin` or unset the option to use the build-in partitioning strategy. ')
+    partitioner
   end
 
 end #class LogStash::Outputs::Kafka

data/lib/logstash/plugin_mixins/kafka/common.rb

@@ -40,7 +40,7 @@ module LogStash module PluginMixins module Kafka
       end
 
       props.put("sasl.kerberos.service.name", sasl_kerberos_service_name) unless sasl_kerberos_service_name.nil?
-      props.put("sasl.jaas.config", sasl_jaas_config.value) unless sasl_jaas_config.nil?
+      props.put("sasl.jaas.config", sasl_jaas_config) unless sasl_jaas_config.nil?
       props.put("sasl.client.callback.handler.class", sasl_client_callback_handler_class) unless sasl_client_callback_handler_class.nil?
       props.put("sasl.oauthbearer.token.endpoint.url", sasl_oauthbearer_token_endpoint_url) unless sasl_oauthbearer_token_endpoint_url.nil?
       props.put("sasl.oauthbearer.scope.claim.name", sasl_oauthbearer_scope_claim_name) unless sasl_oauthbearer_scope_claim_name.nil?

data/lib/logstash-integration-kafka_jars.rb

@@ -1,35 +1,13 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('io.confluent', 'kafka-avro-serializer', '7.9.5')
-require_jar('org.apache.kafka', 'kafka-clients', '3.9.2')
-require_jar('at.yawk.lz4', 'lz4-java', '1.10.1')
-require_jar('io.confluent', 'kafka-schema-serializer', '7.9.5')
-require_jar('io.confluent', 'kafka-schema-registry-client', '7.9.5')
+require_jar('io.confluent', 'kafka-avro-serializer', '8.0.0')
+require_jar('io.confluent', 'kafka-schema-serializer', '8.0.0')
 require_jar('org.apache.avro', 'avro', '1.11.4')
-require_jar('org.apache.commons', 'commons-compress', '1.28.0')
-require_jar('com.google.guava', 'guava', '32.0.1-jre')
-require_jar('io.confluent', 'logredactor', '1.0.12')
-require_jar('io.confluent', 'common-utils', '7.9.5')
-require_jar('com.github.luben', 'zstd-jni', '1.5.6-4')
-require_jar('org.xerial.snappy', 'snappy-java', '1.1.10.5')
-require_jar('org.slf4j', 'slf4j-api', '1.7.36')
+require_jar('io.confluent', 'kafka-schema-registry-client', '8.0.0')
 require_jar('com.fasterxml.jackson.datatype', 'jackson-datatype-jdk8', '2.16.0')
-require_jar('com.fasterxml.jackson.core', 'jackson-databind', '2.16.0')
-require_jar('com.fasterxml.jackson.core', 'jackson-annotations', '2.16.0')
-require_jar('com.fasterxml.jackson.dataformat', 'jackson-dataformat-csv', '2.16.0')
-require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.16.0')
-require_jar('org.yaml', 'snakeyaml', '2.0')
-require_jar('io.swagger.core.v3', 'swagger-annotations', '2.1.10')
-require_jar('com.google.guava', 'failureaccess', '1.0.1')
-require_jar('com.google.guava', 'listenablefuture', '9999.0-empty-to-avoid-conflict-with-guava')
-require_jar('com.google.code.findbugs', 'jsr305', '3.0.2')
-require_jar('org.checkerframework', 'checker-qual', '3.33.0')
-require_jar('com.google.errorprone', 'error_prone_annotations', '2.18.0')
-require_jar('com.google.j2objc', 'j2objc-annotations', '2.8')
-require_jar('com.google.re2j', 're2j', '1.6')
-require_jar('io.confluent', 'logredactor-metrics', '1.0.12')
-require_jar('com.eclipsesource.minimal-json', 'minimal-json', '0.9.5')
-require_jar('commons-codec', 'commons-codec', '1.19.0')
-require_jar('commons-io', 'commons-io', '2.20.0')
-require_jar('org.apache.commons', 'commons-lang3', '3.18.0')
+require_jar('org.apache.kafka', 'kafka-clients', '4.1.0')
+require_jar('org.slf4j', 'slf4j-api', '1.7.36')
+require_jar('com.github.luben', 'zstd-jni', '1.5.6-10')
+require_jar('org.lz4', 'lz4-java', '1.8.0')
+require_jar('org.xerial.snappy', 'snappy-java', '1.1.10.7')

data/logstash-integration-kafka.gemspec

@@ -1,14 +1,14 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-integration-kafka'
-  s.version         = '11.8.6'
+  s.version         = '12.0.0'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Integration with Kafka - input and output plugins"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+
                       "using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program."
   s.authors         = ["Elastic"]
   s.email           = 'info@elastic.co'
-  s.homepage        = "https://www.elastic.co/logstash"
-  s.require_paths   = %w[lib vendor/jar-dependencies]
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths   = ['lib', 'vendor/jar-dependencies']
 
   # Files
   s.files = Dir.glob(%w(

data/spec/fixtures/jaas.config

@@ -1,5 +1,5 @@
 SchemaRegistry-Props {
-  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
+  org.eclipse.jetty.security.jaas.spi.PropertyFileLoginModule required
   file="build/confluent_platform/etc/schema-registry/pwd"
   debug="true";
 };

data/spec/fixtures/jaas3.config

@@ -0,0 +1,5 @@
+SchemaRegistry-Props {
+  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
+  file="build/confluent_platform/etc/schema-registry/pwd"
+  debug="true";
+};

data/spec/integration/inputs/kafka_spec.rb

@@ -187,53 +187,83 @@ describe "inputs/kafka", :integration => true do
     end
   end
 
-  # ToDo: add tests for other sasl config options as well (https://github.com/logstash-plugins/logstash-integration-kafka/issues/234)
-  context 'setting sasl_jaas_config' do
-    let(:base_config) do
-      {
-        'topics' => ['logstash_integration_topic_plain'],
-        'group_id' => rand(36**8).to_s(36),
-      }
+  context 'setting group_protocol' do
+    let(:test_topic) { 'logstash_integration_partitioner_topic' }
+    let(:consumer_config) do
+      plain_config.merge(
+        "topics" => [test_topic],
+        'group_protocol' => group_protocol,
+        "partition_assignment_strategy" => partition_assignment_strategy,
+        "heartbeat_interval_ms" => heartbeat_interval_ms,
+        "session_timeout_ms" => session_timeout_ms
+      )
     end
+    let(:group_protocol) { nil }
+    let(:partition_assignment_strategy) { nil }
+    let(:heartbeat_interval_ms) { LogStash::Inputs::Kafka.get_config().dig("heartbeat_interval_ms", :default) }
+    let(:session_timeout_ms) { LogStash::Inputs::Kafka.get_config().dig("session_timeout_ms", :default) }
 
-    shared_examples 'sasl_jaas_config password handling' do
-      it 'stores sasl_jaas_config as password type' do
-        kafka_input = LogStash::Inputs::Kafka.new(consumer_config)
-        expect(kafka_input.sasl_jaas_config).to be_a(LogStash::Util::Password)
-        expect(kafka_input.sasl_jaas_config.value).to eq(jaas_config_value)
-      end
+    describe "group_protocol = classic" do
+      let(:group_protocol) { 'classic' }
 
-      it 'does not expose password in inspect output' do
+      it 'passes register check' do
         kafka_input = LogStash::Inputs::Kafka.new(consumer_config)
-        expect(kafka_input.sasl_jaas_config.inspect).to eq('<password>')
-        expect(kafka_input.sasl_jaas_config.inspect).not_to include('admin-secret')
+        expect {
+          kafka_input.register
+        }.to_not raise_error
+
+        expect( kafka_input.instance_variable_get(:@heartbeat_interval_ms)).eql?(heartbeat_interval_ms)
+        expect( kafka_input.instance_variable_get(:@session_timeout_ms)).eql?(session_timeout_ms)
       end
     end
 
-    context 'with single-line config' do
-      let(:jaas_config_value) { 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";' }
-      let(:consumer_config) { base_config.merge('sasl_jaas_config' => jaas_config_value) }
+    describe "group_protocol = consumer" do
+      let(:group_protocol) { 'consumer' }
 
-      include_examples 'sasl_jaas_config password handling'
-    end
+      describe "passes register check with supported config" do
+        it 'reset unsupported config to nil' do
+          kafka_input = LogStash::Inputs::Kafka.new(consumer_config)
+          expect {
+            kafka_input.register
+          }.to_not raise_error
 
-    context 'with multiline config' do
-      let(:jaas_config_value) do
-        <<~JAAS
-          org.apache.kafka.common.security.plain.PlainLoginModule required
-            username="admin"
-            password="admin-secret"
-            user_admin="admin-secret"
-            user_alice="alice-secret";
-        JAAS
+          expect( kafka_input.instance_variable_get(:@heartbeat_interval_ms)).to be_nil
+          expect( kafka_input.instance_variable_get(:@session_timeout_ms)).to be_nil
+        end
+      end
+
+      {
+        partition_assignment_strategy: 'range',
+        heartbeat_interval_ms: 2000,
+        session_timeout_ms: 6000
+      }.each do |config_key, config_value|
+        context "with unsupported config #{config_key}" do
+          let(config_key) { config_value }
+
+          it 'raises LogStash::ConfigurationError' do
+            kafka_input = LogStash::Inputs::Kafka.new(consumer_config)
+            expect {
+              kafka_input.register
+            }.to raise_error(LogStash::ConfigurationError, /group_protocol cannot be set to.*consumer.*/)
+          end
+        end
       end
-      let(:consumer_config) { base_config.merge('sasl_jaas_config' => jaas_config_value) }
 
-      include_examples 'sasl_jaas_config password handling'
+      context "with valid config" do
+        let(:test_topic) { 'logstash_integration_topic_plain' }
+        let(:manual_commit_config) do
+          consumer_config.merge(
+            'enable_auto_commit' => 'false'
+          )
+        end
+        it 'consume data' do
+          queue = consume_messages(manual_commit_config, timeout: timeout_seconds, event_count: num_events)
+          expect(queue.length).to eq(num_events)
+        end
+      end
     end
   end
 
-
   context "static membership 'group.instance.id' setting" do
     let(:base_config) do
       {
@@ -533,7 +563,7 @@ describe "Deserializing with the schema registry", :integration => true do
     def delete_topic_if_exists(topic_name, user = nil, password = nil)
       props = java.util.Properties.new
       props.put(Java::org.apache.kafka.clients.admin.AdminClientConfig::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
-      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig
       unless user.nil?
         props.put(serdes_config::BASIC_AUTH_CREDENTIALS_SOURCE, 'USER_INFO')
         props.put(serdes_config::USER_INFO_CONFIG,  "#{user}:#{password}")
@@ -542,7 +572,7 @@ describe "Deserializing with the schema registry", :integration => true do
       topics_list = admin_client.listTopics().names().get()
       if topics_list.contains(topic_name)
         result = admin_client.deleteTopics([topic_name])
-        result.values.get(topic_name).get()
+        result.topicNameValues().get(topic_name).get()
       end
     end
 
@@ -550,7 +580,7 @@ describe "Deserializing with the schema registry", :integration => true do
       props = java.util.Properties.new
       config = org.apache.kafka.clients.producer.ProducerConfig
 
-      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
+      serdes_config = Java::io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig
       props.put(serdes_config::SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081")
 
       props.put(config::BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")

data/spec/integration/outputs/kafka_spec.rb

@@ -164,7 +164,7 @@ describe "outputs/kafka", :integration => true do
     let(:test_topic) { 'logstash_integration_topic3' }
 
     before :each do
-      config = base_config.merge("topic_id" => test_topic, "partitioner" => 'org.apache.kafka.clients.producer.UniformStickyPartitioner')
+      config = base_config.merge("topic_id" => test_topic, "partitioner" => 'org.apache.kafka.clients.producer.RoundRobinPartitioner')
       load_kafka_data(config) do # let's have a bit more (diverse) dataset
         num_events.times.collect do
           LogStash::Event.new.tap do |e|
@@ -212,7 +212,7 @@ describe "outputs/kafka", :integration => true do
 
   context 'setting partitioner' do
     let(:test_topic) { 'logstash_integration_partitioner_topic' }
-    let(:partitioner) { nil }
+    let(:partitioner) { 'round_robin' }
 
     before :each do
       @messages_offset = fetch_messages_from_all_partitions
@@ -221,13 +221,8 @@ describe "outputs/kafka", :integration => true do
       load_kafka_data(config)
     end
 
-    [ 'default', 'round_robin', 'uniform_sticky' ].each do |partitioner|
-      describe partitioner do
-        let(:partitioner) { partitioner }
-        it 'loads data' do
-          expect(fetch_messages_from_all_partitions - @messages_offset).to eql num_events
-        end
-      end
+    it 'loads data' do
+      expect(fetch_messages_from_all_partitions - @messages_offset).to eql num_events
     end
 
     def fetch_messages_from_all_partitions

data/spec/unit/inputs/kafka_spec.rb

@@ -264,43 +264,6 @@ describe LogStash::Inputs::Kafka do
 
       expect(subject.send(:create_consumer, 'test-client-2', 'group_instance_id')).to be kafka_client
     end
-
-    context 'with sasl_jaas_config' do
-      shared_examples 'sasl_jaas_config password handling' do
-        it "sasl_jaas_config.value returns the original string" do
-          subject.register
-          expect(subject.sasl_jaas_config.value).to eq(jaas_config_value)
-        end
-
-        it "sasl_jaas_config.inspect does not expose the password" do
-          subject.register
-          expect(subject.sasl_jaas_config.inspect).not_to include('admin-secret')
-          expect(subject.sasl_jaas_config.inspect).to eq('<password>')
-        end
-      end
-
-      context 'with single-line config' do
-        let(:jaas_config_value) { 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";' }
-        let(:config) { super().merge('sasl_jaas_config' => jaas_config_value) }
-
-        include_examples 'sasl_jaas_config password handling'
-      end
-
-      context 'with multiline config' do
-        let(:jaas_config_value) {
-          <<~JAAS
-            org.apache.kafka.common.security.plain.PlainLoginModule required
-              username="admin"
-              password="admin-secret"
-              user_admin="admin-secret"
-              user_alice="alice-secret";
-          JAAS
-        }
-        let(:config) { super().merge('sasl_jaas_config' => jaas_config_value) }
-
-        include_examples 'sasl_jaas_config password handling'
-      end
-    end
   end
 
   describe "schema registry" do

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: logstash-integration-kafka
 version: !ruby/object:Gem::Version
-  version: 11.8.6
+  version: 12.0.0
 platform: java
 authors:
 - Elastic
 bindir: bin
 cert_chain: []
-date: 2026-03-16 00:00:00.000000000 Z
+date: 2025-10-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core-plugin-api
@@ -235,6 +235,7 @@ files:
 - logstash-integration-kafka.gemspec
 - spec/check_docs_spec.rb
 - spec/fixtures/jaas.config
+- spec/fixtures/jaas3.config
 - spec/fixtures/pwd
 - spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb
@@ -242,39 +243,17 @@ files:
 - spec/unit/inputs/avro_schema_fixture_payment.asvc
 - spec/unit/inputs/kafka_spec.rb
 - spec/unit/outputs/kafka_spec.rb
-- vendor/jar-dependencies/at/yawk/lz4/lz4-java/1.10.1/lz4-java-1.10.1.jar
-- vendor/jar-dependencies/com/eclipsesource/minimal-json/minimal-json/0.9.5/minimal-json-0.9.5.jar
-- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-annotations/2.16.0/jackson-annotations-2.16.0.jar
-- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.16.0/jackson-core-2.16.0.jar
-- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.16.0/jackson-databind-2.16.0.jar
-- vendor/jar-dependencies/com/fasterxml/jackson/dataformat/jackson-dataformat-csv/2.16.0/jackson-dataformat-csv-2.16.0.jar
 - vendor/jar-dependencies/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.16.0/jackson-datatype-jdk8-2.16.0.jar
-- vendor/jar-dependencies/com/github/luben/zstd-jni/1.5.6-4/zstd-jni-1.5.6-4.jar
-- vendor/jar-dependencies/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar
-- vendor/jar-dependencies/com/google/errorprone/error_prone_annotations/2.18.0/error_prone_annotations-2.18.0.jar
-- vendor/jar-dependencies/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar
-- vendor/jar-dependencies/com/google/guava/guava/32.0.1-jre/guava-32.0.1-jre.jar
-- vendor/jar-dependencies/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
-- vendor/jar-dependencies/com/google/j2objc/j2objc-annotations/2.8/j2objc-annotations-2.8.jar
-- vendor/jar-dependencies/com/google/re2j/re2j/1.6/re2j-1.6.jar
-- vendor/jar-dependencies/commons-codec/commons-codec/1.19.0/commons-codec-1.19.0.jar
-- vendor/jar-dependencies/commons-io/commons-io/2.20.0/commons-io-2.20.0.jar
-- vendor/jar-dependencies/io/confluent/common-utils/7.9.5/common-utils-7.9.5.jar
-- vendor/jar-dependencies/io/confluent/kafka-avro-serializer/7.9.5/kafka-avro-serializer-7.9.5.jar
-- vendor/jar-dependencies/io/confluent/kafka-schema-registry-client/7.9.5/kafka-schema-registry-client-7.9.5.jar
-- vendor/jar-dependencies/io/confluent/kafka-schema-serializer/7.9.5/kafka-schema-serializer-7.9.5.jar
-- vendor/jar-dependencies/io/confluent/logredactor-metrics/1.0.12/logredactor-metrics-1.0.12.jar
-- vendor/jar-dependencies/io/confluent/logredactor/1.0.12/logredactor-1.0.12.jar
-- vendor/jar-dependencies/io/swagger/core/v3/swagger-annotations/2.1.10/swagger-annotations-2.1.10.jar
+- vendor/jar-dependencies/com/github/luben/zstd-jni/1.5.6-10/zstd-jni-1.5.6-10.jar
+- vendor/jar-dependencies/io/confluent/kafka-avro-serializer/8.0.0/kafka-avro-serializer-8.0.0.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-registry-client/8.0.0/kafka-schema-registry-client-8.0.0.jar
+- vendor/jar-dependencies/io/confluent/kafka-schema-serializer/8.0.0/kafka-schema-serializer-8.0.0.jar
 - vendor/jar-dependencies/org/apache/avro/avro/1.11.4/avro-1.11.4.jar
-- vendor/jar-dependencies/org/apache/commons/commons-compress/1.28.0/commons-compress-1.28.0.jar
-- vendor/jar-dependencies/org/apache/commons/commons-lang3/3.18.0/commons-lang3-3.18.0.jar
-- vendor/jar-dependencies/org/apache/kafka/kafka-clients/3.9.2/kafka-clients-3.9.2.jar
-- vendor/jar-dependencies/org/checkerframework/checker-qual/3.33.0/checker-qual-3.33.0.jar
+- vendor/jar-dependencies/org/apache/kafka/kafka-clients/4.1.0/kafka-clients-4.1.0.jar
+- vendor/jar-dependencies/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar
 - vendor/jar-dependencies/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
-- vendor/jar-dependencies/org/xerial/snappy/snappy-java/1.1.10.5/snappy-java-1.1.10.5.jar
-- vendor/jar-dependencies/org/yaml/snakeyaml/2.0/snakeyaml-2.0.jar
-homepage: https://www.elastic.co/logstash
+- vendor/jar-dependencies/org/xerial/snappy/snappy-java/1.1.10.7/snappy-java-1.1.10.7.jar
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache-2.0
 metadata:
@@ -302,6 +281,7 @@ summary: Integration with Kafka - input and output plugins
 test_files:
 - spec/check_docs_spec.rb
 - spec/fixtures/jaas.config
+- spec/fixtures/jaas3.config
 - spec/fixtures/pwd
 - spec/fixtures/trust-store_stub.jks
 - spec/integration/inputs/kafka_spec.rb