logstash-input-beats 6.0.10-java → 6.1.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e0a57668d8737ad18f7f4399fd1d99a1d125d3ad3d358194d175d5ca0e9b9546
-  data.tar.gz: 3d4e65c0798a8d66a8b803a7fa399166c8ac1f8569319de682faf92d178a6d3e
+  metadata.gz: 5924244575544fab0560730b5e773b170a04e6c25d1a72c683ea27c2cc47a597
+  data.tar.gz: 7e316315f18e4f33f821dc4c5655d78452e66d605b6db7edc1bc23b55ae3e1d4
 SHA512:
-  metadata.gz: a885d465c14af1bef52498ab496e36c3f83932745f91f8e52980c25f1386844ad791db7bcd9ebe17cceaff6d5518805e30adadebb96719853ef3d0fb7f3fd89e
-  data.tar.gz: e1837acabdd774915634ff99409a853a2863c46f216d250f20d8c16fc4b234587f7a7020e665bcc064e647531180bda4488c6e9afbd524028b0330e6d5d4eb84
+  metadata.gz: c17ac9a8663f465fd38282753f4cda55e6bf5d6ac7659ecafc4524193742dfc7f22ab325b6cdff36cb20daaf18448ff0a1f0345471d0467e023c68b924b70dc3
+  data.tar.gz: 422714a92f313e7f2d504c3b4927955759188a4246bb6417bbc4d77f24f1a5b4ce8ad403c7e059bc9c025ba97c9f3175bb5e186ff526dd34c850e7c97eb4befe

data/CHANGELOG.md

@@ -1,3 +1,24 @@
+## 6.1.0
+ - ECS compatibility enablement, now an `ecs_compatibility` setting is used to declare the level of ECS compatibility 
+   (`disabled` or `v1`) at plugin level. `disabled` let the plugin behave like before while `v1` does a rename of
+   `host` and `@metadata.ip_address` event fields. [404](https://github.com/logstash-plugins/logstash-input-beats/pull/404)
+
+## 6.0.14
+- Feat: log + unwrap generic SSL context exceptions [#405](https://github.com/logstash-plugins/logstash-input-beats/pull/405)
+
+## 6.0.13
+ - [DOC] Update links to use shared attributes
+
+## 6.0.12
+ - Fix: log error when SSL context building fails [#402](https://github.com/logstash-plugins/logstash-input-beats/pull/402).
+   We've also made sure to log messages on configuration errors as LS 7.8/7.9 only prints details when level set to debug.
+
+## 6.0.11
+ - Updated jackson databind and Netty dependencies. Additionally, this release removes the dependency on `tcnative` +
+      `boringssl`, using JVM supplied ciphers instead. This may result in fewer ciphers being available if the JCE
+      unlimited strength jurisdiction policy is not installed. (This policy is installed by default on versions of the
+      JDK from u161 onwards)[#393](https://github.com/logstash-plugins/logstash-input-beats/pull/393)
+
 ## 6.0.10
  - Added error handling to detect if ssl certificate or key files can't be read [#394](https://github.com/logstash-plugins/logstash-input-beats/pull/394)
 

data/README.md

@@ -1,6 +1,6 @@
 # Logstash Plugin
 
-[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-beats.svg)](https://travis-ci.org/logstash-plugins/logstash-input-beats)
+[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-input-beats.svg)](https://travis-ci.com/logstash-plugins/logstash-input-beats)
 
 This is a plugin for [Logstash](https://github.com/elastic/logstash).
 

data/VERSION

@@ -1 +1 @@
-6.0.10
+6.1.0

data/docs/index.asciidoc

@@ -56,11 +56,12 @@ NOTE: If ILM is not being used, set `index` to
 Logstash creates an index per day, based on the `@timestamp` value of the events
 coming from Beats.
 
-IMPORTANT: If you are shipping events that span multiple lines, you need to
-use the https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html[configuration options available in Filebeat] to handle multiline events
-before sending the event data to Logstash. You cannot use the
-{logstash-ref}/plugins-codecs-multiline.html[Multiline codec plugin] to handle multiline events. Doing so will
-result in the failure to start Logstash.
+IMPORTANT: If you are shipping events that span multiple lines, you need to use
+the {filebeat-ref}/multiline-examples.html[configuration options available in
+Filebeat] to handle multiline events before sending the event data to Logstash.
+You cannot use the {logstash-ref}/plugins-codecs-multiline.html[Multiline codec
+plugin] to handle multiline events. Doing so will result in the failure to start
+Logstash.
 
 [id="plugins-{type}s-{plugin}-versioned-indexes"]
 ==== Versioned Beats Indices
@@ -85,6 +86,14 @@ Logstash `@timestamp` field.
 This configuration results in daily index names like
 +filebeat-{logstash_version}-{localdate}+.
 
+
+[id="plugins-{type}s-{plugin}-ecs_metadata"]
+==== Event Metadata and the Elastic Common Schema (ECS)
+When decoding `beats` events, this plugin adds two fields related to the event: the deprecated `host`
+which contains the `hostname` provided by beats and the `ip_address` containing the remote address
+of the client's connection. When <<plugins-{type}s-{plugin}-ecs_compatibility,ECS compatibility mode>> is
+enabled these are now moved in ECS compatible namespace.
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Beats Input Configuration Options
 
@@ -96,6 +105,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-add_hostname>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-cipher_suites>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-client_inactivity_timeout>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
 | <<plugins-{type}s-{plugin}-host>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-include_codec_tag>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-port>> |<<number,number>>|Yes
@@ -143,6 +153,33 @@ The list of ciphers suite to use, listed by priorities.
 
 Close Idle clients after X seconds of inactivity.
 
+[id="plugins-{type}s-{plugin}-ecs_compatibility"]
+===== `ecs_compatibility`
+
+* Value type is <<string,string>>
+* Supported values are:
+** `disabled`: unstructured connection metadata added at root level
+** `v1`: structured connection metadata added under ECS compliant namespaces
+* Default value depends on which version of Logstash is running:
+** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
+** Otherwise, the default value is `disabled`.
+
+Controls this plugin's compatibility with the {ecs-ref}[Elastic Common Schema (ECS)].
+The value of this setting affects the keys for the Beats connection's metadata on the event:
+
+.Metadata Location by `ecs_compatibility` value
+[cols="<l,<l,e,<e"]
+|=======================================================================
+|`disabled`              |`v1`                      |Availability |Description
+
+|[host]                  |[@metadata][input][beats][host][name]   |Always       |Name or address of the beat host
+|[@metadata][ip_address] |[@metadata][input][beats][host][ip]     |Always       |IP address of the Beats client
+|[@metadata][tls_peer][status] | [@metadata][tls_peer][status] | When SSL related fields are populated | Contains "verified"/"unverified" labels in `disabled`, `true`/`false` in `v1`
+|[@metadata][tls_peer][protocol] | [@metadata][input][beats][tls][version_protocol] | When SSL status is "verified" | Contains the TLS version used (e.g. `TLSv1.2`)
+|[@metadata][tls_peer][subject] | [@metadata][input][beats][tls][client][subject] | When SSL status is "verified" | Contains the identity name of the remote end (e.g. `CN=artifacts-no-kpi.elastic.co`)
+|[@metadata][tls_peer][cipher_suite] | [@metadata][input][beats][tls][cipher] | When SSL status is "verified" | Contains the name of cipher suite used (e.g. `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`)
+|=======================================================================
+
 [id="plugins-{type}s-{plugin}-host"]
 ===== `host`
 

data/lib/logstash-input-beats_jars.rb

@@ -1,12 +1,11 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('io.netty', 'netty-all', '4.1.30.Final')
-require_jar('io.netty', 'netty-tcnative-boringssl-static', '2.0.12.Final')
+require_jar('io.netty', 'netty-all', '4.1.49.Final')
 require_jar('org.javassist', 'javassist', '3.24.0-GA')
 require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.9.10')
 require_jar('com.fasterxml.jackson.core', 'jackson-annotations', '2.9.10')
-require_jar('com.fasterxml.jackson.core', 'jackson-databind', '2.9.10.1')
+require_jar('com.fasterxml.jackson.core', 'jackson-databind', '2.9.10.4')
 require_jar('com.fasterxml.jackson.module', 'jackson-module-afterburner', '2.9.10')
 require_jar('org.apache.logging.log4j', 'log4j-api', '2.11.1')
-require_jar('org.logstash.beats', 'logstash-input-beats', '6.0.10')
+require_jar('org.logstash.beats', 'logstash-input-beats', '6.1.0')

data/lib/logstash/inputs/beats.rb

@@ -5,6 +5,7 @@ require "logstash/timestamp"
 require "logstash/codecs/multiline"
 require "logstash/util"
 require "logstash-input-beats_jars"
+require "logstash/plugin_mixins/ecs_compatibility_support"
 require_relative "beats/patch"
 
 # This input plugin enables Logstash to receive events from the
@@ -49,6 +50,9 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
   require "logstash/inputs/beats/message_listener"
   require "logstash/inputs/beats/tls"
 
+  # adds ecs_compatibility config which could be :disabled or :v1
+  include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled,:v1)
+
   config_name "beats"
 
   default :codec, "plain"
@@ -114,14 +118,15 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
   config :tls_max_version, :validate => :number, :default => TLS.max.version
 
   # The list of ciphers suite to use, listed by priorities.
-  config :cipher_suites, :validate => :array, :default => org.logstash.netty.SslContextBuilder::DEFAULT_CIPHERS
-
+  config :cipher_suites, :validate => :array, :default => org.logstash.netty.SslContextBuilder.getDefaultCiphers
   # Close Idle clients after X seconds of inactivity.
   config :client_inactivity_timeout, :validate => :number, :default => 60
 
   # Beats handler executor thread
   config :executor_threads, :validate => :number, :default => LogStash::Config::CpuCoreStrategy.maximum
 
+  attr_reader :field_hostname, :field_hostip
+
   def register
     # For Logstash 2.4 we need to make sure that the logger is correctly set for the
     # java classes before actually loading them.
@@ -132,27 +137,39 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
       LogStash::Logger.setup_log4j(@logger)
     end
 
-    if !@ssl
-      @logger.warn("Beats input: SSL Certificate will not be used") unless @ssl_certificate.nil?
-      @logger.warn("Beats input: SSL Key will not be used") unless @ssl_key.nil?
-    elsif !ssl_configured?
-      raise LogStash::ConfigurationError, "Certificate or Certificate Key not configured"
-    end
+    if @ssl
+      if @ssl_key.nil? || @ssl_key.empty?
+        configuration_error "ssl_key => is a required setting when ssl => true is configured"
+      end
+      if @ssl_certificate.nil? || @ssl_certificate.empty?
+        configuration_error "ssl_certificate => is a required setting when ssl => true is configured"
+      end
 
-    if @ssl && require_certificate_authorities? && !client_authentification?
-      raise LogStash::ConfigurationError, "Using `verify_mode` set to PEER or FORCE_PEER, requires the configuration of `certificate_authorities`"
-    end
+      if require_certificate_authorities? && !client_authentification?
+        configuration_error "ssl_certificate_authorities => is a required setting when ssl_verify_mode => '#{@ssl_verify_mode}' is configured"
+      end
 
-    if client_authentication_metadata? && !require_certificate_authorities?
-      raise LogStash::ConfigurationError, "Enabling `peer_metadata` requires using `verify_mode` set to PEER or FORCE_PEER"
+      if client_authentication_metadata? && !require_certificate_authorities?
+        configuration_error "Configuring ssl_peer_metadata => true requires ssl_verify_mode => to be configured with 'peer' or 'force_peer'"
+      end
+    else
+      @logger.warn("configured ssl_certificate => #{@ssl_certificate.inspect} will not be used") if @ssl_certificate
+      @logger.warn("configured ssl_key => #{@ssl_key.inspect} will not be used") if @ssl_key
     end
 
     # Logstash 6.x breaking change (introduced with 4.0.0 of this gem)
     if @codec.kind_of? LogStash::Codecs::Multiline
-      raise LogStash::ConfigurationError, "Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html"
+      configuration_error "Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html"
     end
 
-    @logger.info("Beats inputs: Starting input listener", :address => "#{@host}:#{@port}")
+    # define ecs name mapping
+    @field_hostname = ecs_select[disabled: "host", v1: "[@metadata][input][beats][host][name]"]
+    @field_hostip   = ecs_select[disabled: "[@metadata][ip_address]", v1: "[@metadata][input][beats][host][ip]"]
+    @field_tls_protocol_version   = ecs_select[disabled: "[@metadata][tls_peer][protocol]", v1: "[@metadata][input][beats][tls][version_protocol]"]
+    @field_tls_peer_subject   = ecs_select[disabled: "[@metadata][tls_peer][subject]", v1: "[@metadata][input][beats][tls][client][subject]"]
+    @field_tls_cipher   = ecs_select[disabled: "[@metadata][tls_peer][cipher_suite]", v1: "[@metadata][input][beats][tls][cipher]"]
+
+    @logger.info("Starting input listener", :address => "#{@host}:#{@port}")
 
     @server = create_server
   end # def register
@@ -160,37 +177,20 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
   def create_server
     server = org.logstash.beats.Server.new(@host, @port, @client_inactivity_timeout, @executor_threads)
     if @ssl
-
-      begin
-        ssl_context_builder = org.logstash.netty.SslContextBuilder.new(@ssl_certificate, @ssl_key, @ssl_key_passphrase.nil? ? nil : @ssl_key_passphrase.value)
-                          .setProtocols(convert_protocols)
-                          .setCipherSuites(normalized_ciphers)
-      rescue java.lang.IllegalArgumentException => e
-        raise LogStash::ConfigurationError, e
-      end
-
-
+      ssl_context_builder = new_ssl_context_builder
       if client_authentification?
-        if @ssl_verify_mode.upcase == "FORCE_PEER"
+        if @ssl_verify_mode == "force_peer"
           ssl_context_builder.setVerifyMode(org.logstash.netty.SslContextBuilder::SslClientVerifyMode::FORCE_PEER)
-        elsif @ssl_verify_mode.upcase == "PEER"
+        elsif @ssl_verify_mode == "peer"
           ssl_context_builder.setVerifyMode(org.logstash.netty.SslContextBuilder::SslClientVerifyMode::VERIFY_PEER)
         end
         ssl_context_builder.setCertificateAuthorities(@ssl_certificate_authorities)
       end
-      server.setSslHandlerProvider(org.logstash.netty.SslHandlerProvider.new(ssl_context_builder.build_context, @ssl_handshake_timeout))
+      server.setSslHandlerProvider(new_ssl_handshake_provider(ssl_context_builder))
     end
     server
   end
 
-  def ssl_configured?
-    !(@ssl_certificate.nil? || @ssl_key.nil?)
-  end
-
-  def target_codec_on_field?
-    !@target_codec_on_field.empty?
-  end
-
   def run(output_queue)
     message_listener = MessageListener.new(output_queue, self)
     @server.setMessageListener(message_listener)
@@ -201,6 +201,14 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
     @server.stop unless @server.nil?
   end
 
+  def ssl_configured?
+    !(@ssl_certificate.nil? || @ssl_key.nil?)
+  end
+
+  def target_codec_on_field?
+    !@target_codec_on_field.empty?
+  end
+
   def client_authentification?
     @ssl_certificate_authorities && @ssl_certificate_authorities.size > 0
   end
@@ -217,6 +225,32 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
     @ssl_verify_mode == "force_peer" || @ssl_verify_mode == "peer"
   end
 
+  private
+
+  def new_ssl_handshake_provider(ssl_context_builder)
+    begin
+      org.logstash.netty.SslHandlerProvider.new(ssl_context_builder.build_context, @ssl_handshake_timeout)
+    rescue java.lang.IllegalArgumentException => e
+      @logger.error("SSL configuration invalid", error_details(e))
+      raise LogStash::ConfigurationError, e
+    rescue java.lang.Exception => e # java.security.GeneralSecurityException
+      @logger.error("SSL configuration failed", error_details(e, true))
+      raise e
+    end
+  end
+
+  def new_ssl_context_builder
+    passphrase = @ssl_key_passphrase.nil? ? nil : @ssl_key_passphrase.value
+    begin
+      org.logstash.netty.SslContextBuilder.new(@ssl_certificate, @ssl_key, passphrase)
+          .setProtocols(convert_protocols)
+          .setCipherSuites(normalized_ciphers)
+    rescue java.lang.IllegalArgumentException => e
+      @logger.error("SSL configuration invalid", error_details(e))
+      raise LogStash::ConfigurationError, e
+    end
+  end
+
   def normalized_ciphers
     @cipher_suites.map(&:upcase)
   end
@@ -224,4 +258,21 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
   def convert_protocols
     TLS.get_supported(@tls_min_version..@tls_max_version).map(&:name)
   end
+
+  def configuration_error(message)
+    @logger.error message
+    raise LogStash::ConfigurationError, message
+  end
+
+  def error_details(e, trace = false)
+    error_details = { :exception => e.class, :message => e.message }
+    error_details[:backtrace] = e.backtrace if trace || @logger.debug?
+    cause = e.cause
+    if cause && e != cause
+      error_details[:cause] = { :exception => cause.class, :message => cause.message }
+      error_details[:cause][:backtrace] = cause.backtrace if trace || @logger.debug?
+    end
+    error_details
+  end
+
 end

data/lib/logstash/inputs/beats/decoded_event_transform.rb

@@ -9,7 +9,16 @@ module LogStash module Inputs class Beats
       ts = coerce_ts(hash.delete("@timestamp"))
 
       event.set("@timestamp", ts) unless ts.nil?
-      hash.each { |k, v| event.set(k, v) }
+      hash.each do |k, v|
+        #could be a nested map, so we need to merge and not overwrite
+        existing_value = event.get(k)
+        if existing_value.is_a?(Hash)
+          existing_value = existing_value.merge(v)
+        else
+          existing_value = v
+        end
+        event.set(k, existing_value)
+      end
       super(event)
       event.tag("beats_input_codec_#{codec_name}_applied") if include_codec_tag?
       event

data/lib/logstash/inputs/beats/event_transform_common.rb

@@ -15,8 +15,8 @@ module LogStash module Inputs class Beats
       return unless @input.add_hostname
       host = event.get("[beat][hostname]")
 
-      if host && event.get("host").nil?
-        event.set("host", host)
+      if host && event.get(@input.field_hostname).nil?
+        event.set(@input.field_hostname, host)
       end
     end
 

data/lib/logstash/inputs/beats/message_listener.rb

@@ -31,7 +31,9 @@ module LogStash module Inputs class Beats
       hash = message.getData
       ip_address = ip_address(ctx)
 
-      hash['@metadata']['ip_address'] = ip_address unless ip_address.nil? || hash['@metadata'].nil?
+      unless ip_address.nil? || hash['@metadata'].nil?
+        set_nested(hash, @input.field_hostip, ip_address)
+      end
       target_field = extract_target_field(hash)
 
       extract_tls_peer(hash, ctx)
@@ -140,11 +142,12 @@ module LogStash module Inputs class Beats
         end
 
         if tls_verified
+          set_nested(hash, @field_tls_protocol_version, tls_session.getProtocol())
+          set_nested(hash, @field_tls_peer_subject, tls_session.getPeerPrincipal().getName())
+          set_nested(hash, @field_tls_cipher, tls_session.getCipherSuite())
+
           hash['@metadata']['tls_peer'] = {
-            :status       => "verified",
-            :protocol     => tls_session.getProtocol(),
-            :subject      => tls_session.getPeerPrincipal().getName(),
-            :cipher_suite => tls_session.getCipherSuite()
+            :status       => "verified"
           }
         else
           hash['@metadata']['tls_peer'] = {
@@ -154,6 +157,28 @@ module LogStash module Inputs class Beats
       end
     end
 
+    # set the value for field_name into the hash, nesting into sub-hashes and creating hashes where necessary
+    public #only to make it testable
+    def set_nested(hash, field_name, value)
+      field_ref = Java::OrgLogstash::FieldReference.from(field_name)
+      # create @metadata sub-hash if needed
+      if field_ref.type == Java::OrgLogstash::FieldReference::META_CHILD
+        unless hash.key?("@metadata")
+          hash["@metadata"] = {}
+        end
+        nesting_hash = hash["@metadata"]
+      else
+        nesting_hash = hash
+      end
+
+      field_ref.path.each do |token|
+        nesting_hash[token] = {} unless nesting_hash.key?(token)
+        nesting_hash = nesting_hash[token]
+      end
+      nesting_hash[field_ref.key] = value
+    end
+
+    private
     def extract_target_field(hash)
       if from_filebeat?(hash)
         hash.delete(FILEBEAT_LOG_LINE_FIELD).to_s

data/logstash-input-beats.gemspec

@@ -27,6 +27,7 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency "thread_safe", "~> 0.3.5"
   s.add_runtime_dependency "logstash-codec-multiline", ">= 2.0.5"
   s.add_runtime_dependency 'jar-dependencies', '~> 0.3', '>= 0.3.4'
+  s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.1'
 
   s.add_development_dependency "flores", "~>0.0.6"
   s.add_development_dependency "rspec"

data/spec/inputs/beats/decoded_event_transform_spec.rb

@@ -29,7 +29,8 @@ describe LogStash::Inputs::Beats::DecodedEventTransform do
 
   subject { described_class.new(input).transform(event, map) }
 
-  include_examples "Common Event Transformation"
+  include_examples "Common Event Transformation", :disabled, "host"
+  include_examples "Common Event Transformation", :v1, "[@metadata][input][beats][host][name]"
 
   it "tags the event" do
     expect(subject.get("tags")).to include("beats_input_codec_plain_applied")

data/spec/inputs/beats/event_transform_common_spec.rb

@@ -7,5 +7,6 @@ require "spec_helper"
 describe LogStash::Inputs::Beats::EventTransformCommon do
   subject { described_class.new(input).transform(event) }
 
-  include_examples "Common Event Transformation"
+  include_examples "Common Event Transformation", :disabled, "host"
+  include_examples "Common Event Transformation", :v1, "[@metadata][input][beats][host][name]"
 end

data/spec/inputs/beats/message_listener_spec.rb

@@ -52,10 +52,73 @@ class DummyCodec < LogStash::Codecs::Base
   end
 end
 
+shared_examples "when the message is from any libbeat" do |ecs_compatibility, host_field_name|
+  let(:input) do
+    input = LogStash::Inputs::Beats.new({ "port" => 5555, "codec" => codec, "ecs_compatibility" => "#{ecs_compatibility}" })
+    input.register
+    input
+  end
+
+  #Requires data modeled as Java, not Ruby since the actual code pulls from Java backed (Netty) object
+  let(:data) do
+    d = HashMap.new
+    d.put('@metadata', HashMap.new)
+    d.put('metric',  1)
+    d.put('name', "super-stats")
+    d
+  end
+
+  let(:message) { MockMessage.new("abc",  data)}
+
+  it "extract the event" do
+    subject.onNewMessage(ctx, message)
+    event = queue.pop
+    expect(event.get("message")).to be_nil
+    expect(event.get("metric")).to eq(1)
+    expect(event.get("name")).to eq("super-stats")
+    expect(event.get(host_field_name)).to eq(ip_address)
+  end
+
+  context 'when the remote address is nil' do
+    let(:ctx) { OngoingMethodMock.new("remoteAddress", nil)}
+
+    it 'extracts the event' do
+      subject.onNewMessage(ctx, message)
+      event = queue.pop
+      expect(event.get("message")).to be_nil
+      expect(event.get("metric")).to eq(1)
+      expect(event.get("name")).to eq("super-stats")
+      expect(event.get(host_field_name)).to eq(nil)
+    end
+  end
+
+  context 'when getting the remote address raises' do
+    let(:raising_ctx) { double("context")}
+
+    before  do
+      allow(raising_ctx).to receive(:channel).and_raise("nope")
+      subject.onNewConnection(raising_ctx)
+    end
+
+    it 'extracts the event' do
+      subject.onNewMessage(raising_ctx, message)
+      event = queue.pop
+      expect(event.get("message")).to be_nil
+      expect(event.get("metric")).to eq(1)
+      expect(event.get("name")).to eq("super-stats")
+      expect(event.get(host_field_name)).to eq(nil)
+    end
+  end
+end
+
 describe LogStash::Inputs::Beats::MessageListener do
   let(:queue)  { Queue.new }
   let(:codec) { DummyCodec.new }
-  let(:input) { LogStash::Inputs::Beats.new({ "port" => 5555, "codec" => codec }) }
+  let(:input) do
+    input = LogStash::Inputs::Beats.new({ "port" => 5555, "codec" => codec })
+    input.register
+    input
+  end
 
   let(:ip_address) { "10.0.0.1" }
   let(:remote_address) { OngoingMethodMock.new("getHostAddress", ip_address) }
@@ -146,59 +209,8 @@ describe LogStash::Inputs::Beats::MessageListener do
       end
     end
 
-    context "when the message is from any libbeat" do
-      #Requires data modeled as Java, not Ruby since the actual code pulls from Java backed (Netty) object
-      let(:data) do
-        d = HashMap.new
-        d.put('@metadata', HashMap.new)
-        d.put('metric',  1)
-        d.put('name', "super-stats")
-        d
-      end
-
-      let(:message) { MockMessage.new("abc",  data)}
-
-      it "extract the event" do
-        subject.onNewMessage(ctx, message)
-        event = queue.pop
-        expect(event.get("message")).to be_nil
-        expect(event.get("metric")).to eq(1)
-        expect(event.get("name")).to eq("super-stats")
-        expect(event.get("[@metadata][ip_address]")).to eq(ip_address)
-      end
-
-      context 'when the remote address is nil' do
-        let(:ctx) { OngoingMethodMock.new("remoteAddress", nil)}
-
-        it 'extracts the event' do
-          subject.onNewMessage(ctx, message)
-          event = queue.pop
-          expect(event.get("message")).to be_nil
-          expect(event.get("metric")).to eq(1)
-          expect(event.get("name")).to eq("super-stats")
-          expect(event.get("[@metadata][ip_address]")).to eq(nil)
-        end
-      end
-
-      context 'when getting the remote address raises' do
-        let(:raising_ctx) { double("context")}
-
-        before  do
-          allow(raising_ctx).to receive(:channel).and_raise("nope")
-          subject.onNewConnection(raising_ctx)
-        end
-
-        it 'extracts the event' do
-          subject.onNewMessage(raising_ctx, message)
-          event = queue.pop
-          expect(event.get("message")).to be_nil
-          expect(event.get("metric")).to eq(1)
-          expect(event.get("name")).to eq("super-stats")
-          expect(event.get("[@metadata][ip_address]")).to eq(nil)
-        end
-      end
-
-    end
+    it_behaves_like "when the message is from any libbeat", :disabled, "[@metadata][ip_address]"
+    it_behaves_like "when the message is from any libbeat", :v1, "[@metadata][input][beats][host][ip]"
   end
 
   context "onException" do
@@ -222,4 +234,20 @@ describe LogStash::Inputs::Beats::MessageListener do
       expect(queue).not_to be_empty
     end
   end
+
+  context "set_nested" do
+    let(:hash) {{}}
+
+    it "creates correctly the nested maps" do
+      subject.set_nested(hash, "[root][inner][leaf]", 5)
+      expect(hash["root"]["inner"]["leaf"]).to eq(5)
+    end
+
+    it "doesn't overwrite existing the nested maps" do
+      hash = {"root" => {"foo" => {"bar" => "Hello"}}}
+      subject.set_nested(hash, "[root][inner][leaf]", 5)
+      expect(hash["root"]["inner"]["leaf"]).to eq(5)
+      expect(hash["root"]["foo"]["bar"]).to eq("Hello")
+    end
+  end
 end

data/spec/inputs/beats/raw_event_transform_spec.rb

@@ -18,7 +18,8 @@ describe LogStash::Inputs::Beats::RawEventTransform do
 
   subject { described_class.new(input).transform(event) }
 
-  include_examples "Common Event Transformation"
+  include_examples "Common Event Transformation", :disabled, "host"
+  include_examples "Common Event Transformation", :v1, "[@metadata][input][beats][host][name]"
 
   it "tags the event" do
     expect(subject.get("tags")).to include("beats_input_raw_event")

data/spec/inputs/beats_spec.rb

@@ -13,11 +13,19 @@ describe LogStash::Inputs::Beats do
   let(:certificate) { BeatsInputTest.certificate }
   let(:port) { BeatsInputTest.random_port }
   let(:queue)  { Queue.new }
-  let(:config)   { { "port" => 0, "ssl_certificate" => certificate.ssl_cert, "ssl_key" => certificate.ssl_key, "type" => "example", "tags" => "beats"} }
+  let(:config)   do
+    {
+        "port" => 0,
+        "ssl_certificate" => certificate.ssl_cert,
+        "ssl_key" => certificate.ssl_key,
+        "type" => "example",
+        "tags" => "beats"
+    }
+  end
 
   context "#register" do
     context "host related configuration" do
-      let(:config) { super.merge!({ "host" => host, "port" => port, "client_inactivity_timeout" => client_inactivity_timeout, "executor_threads" => threads }) }
+      let(:config) { super.merge("host" => host, "port" => port, "client_inactivity_timeout" => client_inactivity_timeout, "executor_threads" => threads) }
       let(:host) { "192.168.1.20" }
       let(:port) { 9000 }
       let(:client_inactivity_timeout) { 400 }
@@ -38,38 +46,55 @@ describe LogStash::Inputs::Beats do
 
     context "with ssl enabled" do
       context "without certificate configuration" do
-        let(:config) {{ "port" => 0, "ssl" => true, "ssl_key" => certificate.ssl_key, "type" => "example", "tags" => "beats" }}
+        let(:config) { { "port" => 0, "ssl" => true, "ssl_key" => certificate.ssl_key, "type" => "example" } }
 
         it "should fail to register the plugin with ConfigurationError" do
           plugin = LogStash::Inputs::Beats.new(config)
-          expect {plugin.register}.to raise_error(LogStash::ConfigurationError)
+          expect { plugin.register }.to raise_error(LogStash::ConfigurationError)
         end
       end
 
       context "without key configuration" do
-        let(:config)   { { "port" => 0, "ssl" => true, "ssl_certificate" => certificate.ssl_cert, "type" => "example", "tags" => "Beats"} }
+        let(:config) { { "port" => 0, "ssl" => true, "ssl_certificate" => certificate.ssl_cert, "type" => "example" } }
         it "should fail to register the plugin with ConfigurationError" do
           plugin = LogStash::Inputs::Beats.new(config)
-          expect {plugin.register}.to raise_error(LogStash::ConfigurationError)
+          expect { plugin.register }.to raise_error(LogStash::ConfigurationError)
+        end
+      end
+
+      context "with invalid key configuration" do
+        let(:p12_key) { certificate.p12_key }
+        let(:config) { { "port" => 0, "ssl" => true, "ssl_certificate" => certificate.ssl_cert, "ssl_key" => p12_key } }
+        it "should fail to register the plugin" do
+          plugin = LogStash::Inputs::Beats.new(config)
+          expect( plugin.logger ).to receive(:error) do |msg, opts|
+            expect( msg ).to match /.*?configuration invalid/
+            expect( opts[:message] ).to match /does not contain valid private key/
+          end
+          expect { plugin.register }.to raise_error(LogStash::ConfigurationError)
         end
       end
 
       context "with invalid ciphers" do
-        let(:config)   { { "port" => 0, "ssl" => true, "ssl_certificate" => certificate.ssl_cert, "type" => "example", "tags" => "Beats", "cipher_suites" => "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA38"} }
+        let(:config) { super.merge("ssl" => true, "cipher_suites" => "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA38") }
 
         it "should raise a configuration error" do
           plugin = LogStash::Inputs::Beats.new(config)
+          expect( plugin.logger ).to receive(:error) do |msg, opts|
+            expect( msg ).to match /.*?configuration invalid/
+            expect( opts[:message] ).to match /TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA38.*? not available/
+          end
           expect { plugin.register }.to raise_error(LogStash::ConfigurationError)
         end
       end
 
       context "verify_mode" do
         context "verify_mode configured to PEER" do
-          let(:config)   { { "port" => 0, "ssl" => true, "ssl_verify_mode" => "peer", "ssl_certificate" => certificate.ssl_cert, "ssl_key" => certificate.ssl_key, "type" => "example", "tags" => "Beats"} }
+          let(:config) { super.merge("ssl" => true, "ssl_verify_mode" => "peer") }
 
           it "raise a ConfigurationError when certificate_authorities is not set" do
             plugin = LogStash::Inputs::Beats.new(config)
-            expect {plugin.register}.to raise_error(LogStash::ConfigurationError, "Using `verify_mode` set to PEER or FORCE_PEER, requires the configuration of `certificate_authorities`")
+            expect {plugin.register}.to raise_error(LogStash::ConfigurationError, "ssl_certificate_authorities => is a required setting when ssl_verify_mode => 'peer' is configured")
           end
 
           it "doesn't raise a configuration error when certificate_authorities is set" do
@@ -80,11 +105,11 @@ describe LogStash::Inputs::Beats do
         end
 
         context "verify_mode configured to FORCE_PEER" do
-          let(:config)   { { "port" => 0, "ssl" => true, "ssl_verify_mode" => "force_peer", "ssl_certificate" => certificate.ssl_cert, "ssl_key" => certificate.ssl_key, "type" => "example", "tags" => "Beats"} }
+          let(:config) { super.merge("ssl" => true, "ssl_verify_mode" => "force_peer") }
 
           it "raise a ConfigurationError when certificate_authorities is not set" do
             plugin = LogStash::Inputs::Beats.new(config)
-            expect {plugin.register}.to raise_error(LogStash::ConfigurationError, "Using `verify_mode` set to PEER or FORCE_PEER, requires the configuration of `certificate_authorities`")
+            expect {plugin.register}.to raise_error(LogStash::ConfigurationError, "ssl_certificate_authorities => is a required setting when ssl_verify_mode => 'force_peer' is configured")
           end
 
           it "doesn't raise a configuration error when certificate_authorities is set" do
@@ -98,7 +123,7 @@ describe LogStash::Inputs::Beats do
 
     context "with ssl disabled" do
       context "and certificate configuration" do
-        let(:config)   { { "port" => 0, "ssl" => false, "ssl_certificate" => certificate.ssl_cert, "type" => "example", "tags" => "Beats" } }
+        let(:config) { { "port" => 0, "ssl" => false, "ssl_certificate" => certificate.ssl_cert, "type" => "example", "tags" => "Beats" } }
 
         it "should not fail" do
           plugin = LogStash::Inputs::Beats.new(config)

data/spec/integration/filebeat_spec.rb

@@ -166,8 +166,8 @@ describe "Filebeat", :integration => true do
             end
 
             context "when the cipher is not supported" do
-              let(:beats_cipher) { "ECDHE-RSA-AES-128-GCM-SHA256" }
-              let(:logstash_cipher) { "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"}
+              let(:beats_cipher) { "ECDHE-RSA-AES-256-GCM-SHA384" }
+              let(:logstash_cipher) { "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"}
 
               include_examples "doesn't send events"
             end

data/spec/support/logstash_test.rb

@@ -13,6 +13,13 @@ module BeatsInputTest
 
       system("openssl req -x509  -batch -nodes -newkey rsa:2048 -keyout #{ssl_key} -out #{ssl_cert} -subj /CN=localhost > /dev/null 2>&1")
     end
+
+    def p12_key
+      p12_key = Stud::Temporary.pathname("p12_key")
+      system "openssl pkcs12 -export -passout pass:123 -inkey #{ssl_key} -in #{ssl_cert} -out #{p12_key}"
+      p12_key
+    end
+
   end
 
   class << self

data/spec/support/shared_examples.rb

@@ -1,5 +1,5 @@
 # encoding: utf-8
-shared_examples "Common Event Transformation" do
+shared_examples "Common Event Transformation" do |ecs_compatibility, host_field_name|
   let(:tag) { "140-rpm-beats" }
   let(:config) do
     {
@@ -21,12 +21,18 @@ shared_examples "Common Event Transformation" do
     }
   end
 
+  def key_as_nested_maps(key, value)
+    evt = LogStash::Event.new
+    evt.set(key, value)
+    evt.to_hash_with_metadata
+  end
+
   it "adds configured tags to the event" do
     expect(subject.get("tags")).to include(tag)
   end
 
   context 'when add_hostname is true' do
-    let(:config) { super.merge({'add_hostname' => true})}
+    let(:config) { super.merge({'add_hostname' => true, 'ecs_compatibility' => ecs_compatibility})}
 
     context 'when a host is provided in beat.host.name' do
       let(:already_exist) { "already_exist" }
@@ -53,35 +59,35 @@ shared_examples "Common Event Transformation" do
       let(:producer_host) { "newhost01" }
       let(:event_map) { super.merge({ "beat" => { "hostname" => producer_host }}) }
 
-      context "when no `host` key already exists on the event" do
-        it "copies the value in `beat.hostname` to `host`" do
-          expect(subject.get("host")).to eq(producer_host)
+      context "when no `#{host_field_name}` key already exists on the event" do
+        it "copies the value in `beat.hostname` to `#{host_field_name}`" do
+          expect(subject.get(host_field_name)).to eq(producer_host)
         end
       end
 
-      context "when `host` key exists on the event" do
+      context "when `#{host_field_name}` key exists on the event" do
         let(:already_exist) { "already_exist" }
-        let(:event_map) { super.merge({ "host" => already_exist }) }
+        let(:event_map) { super.merge(key_as_nested_maps(host_field_name, already_exist)) }
 
         it "doesn't override it" do
-          expect(subject.get("host")).to eq(already_exist)
+          expect(subject.get(host_field_name)).to eq(already_exist)
         end
       end
     end
 
     context "when no host is provided in beat" do
-      context "when no `host` key already exists on the event" do
+      context "when no `#{host_field_name}` key already exists on the event" do
         it "does not set the host" do
-          expect(subject.get("host")).to be_nil
+          expect(subject.get(host_field_name)).to be_nil
         end
       end
 
-      context "when `host` key already exists on the event" do
+      context "when `#{host_field_name}` key already exists on the event" do
         let(:already_exist) { "already_exist" }
-        let(:event_map) { super.merge({ "host" => already_exist }) }
+        let(:event_map) { super.merge(key_as_nested_maps(host_field_name, already_exist)) }
 
         it "doesn't override it" do
-          expect(subject.get("host")).to eq(already_exist)
+          expect(subject.get(host_field_name)).to eq(already_exist)
         end
       end
     end
@@ -95,18 +101,18 @@ shared_examples "Common Event Transformation" do
       let(:producer_host) { "newhost01" }
       let(:event_map) { super.merge({ "beat" => { "host" => {"name" => producer_host }}}) }
 
-      context "when no `host` key already exists on the event" do
+      context "when no `#{host_field_name}` key already exists on the event" do
         it "does not set the host" do
-          expect(subject.get("host")).to be_nil
+          expect(subject.get(host_field_name)).to be_nil
         end
       end
 
-      context "when `host` key already exists on the event" do
+      context "when `#{host_field_name}` key already exists on the event" do
         let(:already_exist) { "already_exist" }
-        let(:event_map) { super.merge({ "host" => already_exist }) }
+        let(:event_map) { super.merge(key_as_nested_maps(host_field_name, already_exist)) }
 
         it "doesn't override it" do
-          expect(subject.get("host")).to eq(already_exist)
+          expect(subject.get(host_field_name)).to eq(already_exist)
         end
       end
     end
@@ -115,35 +121,35 @@ shared_examples "Common Event Transformation" do
       let(:producer_host) { "newhost01" }
       let(:event_map) { super.merge({ "beat" => { "hostname" => producer_host }}) }
 
-      context "when no `host` key already exists on the event" do
+      context "when no `#{host_field_name}` key already exists on the event" do
         it "does not set the host" do
-          expect(subject.get("host")).to be_nil
+          expect(subject.get(host_field_name)).to be_nil
         end
       end
 
       context "when `host` key already exists on the event" do
         let(:already_exist) { "already_exist" }
-        let(:event_map) { super.merge({ "host" => already_exist }) }
+        let(:event_map) { super.merge(key_as_nested_maps(host_field_name, already_exist)) }
 
         it "doesn't override it" do
-          expect(subject.get("host")).to eq(already_exist)
+          expect(subject.get(host_field_name)).to eq(already_exist)
         end
       end
     end
 
     context "when no host is provided in beat" do
-      context "when no `host` key already exists on the event" do
+      context "when no `#{host_field_name}` key already exists on the event" do
         it "does not set the host" do
-          expect(subject.get("host")).to be_nil
+          expect(subject.get(host_field_name)).to be_nil
         end
       end
 
-      context "when `host` key already exists on the event" do
+      context "when `#{host_field_name}` key already exists on the event" do
         let(:already_exist) { "already_exist" }
-        let(:event_map) { super.merge({ "host" => already_exist }) }
+        let(:event_map) { super.merge(key_as_nested_maps(host_field_name, already_exist)) }
 
         it "doesn't override it" do
-          expect(subject.get("host")).to eq(already_exist)
+          expect(subject.get(host_field_name)).to eq(already_exist)
         end
       end
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-beats
 version: !ruby/object:Gem::Version
-  version: 6.0.10
+  version: 6.1.0
 platform: java
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-05-12 00:00:00.000000000 Z
+date: 2021-02-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -106,6 +106,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.3.4
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+  name: logstash-mixin-ecs_compatibility_support
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
@@ -266,13 +280,12 @@ files:
 - spec/support/shared_examples.rb
 - vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-annotations/2.9.10/jackson-annotations-2.9.10.jar
 - vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.9.10/jackson-core-2.9.10.jar
-- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar
+- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.10.4/jackson-databind-2.9.10.4.jar
 - vendor/jar-dependencies/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.10/jackson-module-afterburner-2.9.10.jar
-- vendor/jar-dependencies/io/netty/netty-all/4.1.30.Final/netty-all-4.1.30.Final.jar
-- vendor/jar-dependencies/io/netty/netty-tcnative-boringssl-static/2.0.12.Final/netty-tcnative-boringssl-static-2.0.12.Final.jar
+- vendor/jar-dependencies/io/netty/netty-all/4.1.49.Final/netty-all-4.1.49.Final.jar
 - vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1.jar
 - vendor/jar-dependencies/org/javassist/javassist/3.24.0-GA/javassist-3.24.0-GA.jar
-- vendor/jar-dependencies/org/logstash/beats/logstash-input-beats/6.0.10/logstash-input-beats-6.0.10.jar
+- vendor/jar-dependencies/org/logstash/beats/logstash-input-beats/6.1.0/logstash-input-beats-6.1.0.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)