logstash-output-elasticsearch 10.4.1-java → 10.6.1-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5e5b6da5658007e7a4a0a9c1b9e087cd9d91ea37b9d13909370c3982e0d353f7
-  data.tar.gz: d20c029af20c6ea1fed2a4287b81d8d5b723e8dda2d078abe7d686ea14075286
+  metadata.gz: 712190af3d33e9433c9f6b6dd8efa85406b28e1dddaf344bcaa1c5769708bd44
+  data.tar.gz: 860e97c8f49c09bb0659939211843cc2799e69feccfb58d4584144a067e6b43f
 SHA512:
-  metadata.gz: 9c7facb3b428ebac3ddb3e2a55c1ceb410ddf64429197872565e1a7069174d1410be4044ca099666bbe3c5b65afcaf555358043881d3654a5bdd1fa81488ebea
-  data.tar.gz: bfc16cff30b59df4e66d3adb306ecf1da02e4036f3e1146db2004bc4576a46e1463d8a63f6806f53651380ca4f9df892730e65387a9e0cb43484cef2388f27be
+  metadata.gz: 2aa2d08a6bdf2248091afb58fe46d79f5144a07ff2512aa6ba3511242037da6577cefae0e0da7278bab13e1382abda215dee081154eec79d75604f8f4a054386
+  data.tar.gz: ebeb646959ce0baf384036dfd34c035a13f88deded066d9557c3d13770dfeb37a241385ab4612f702a4e850fb835903cddc81ed28445e799a2f9d8ce81b50e0c

data/CHANGELOG.md

@@ -1,3 +1,15 @@
+## 10.6.1
+ - Fixed an issue introduced in 10.6.0 that broke Logstash Core's monitoring feature when this plugin is run in Logstash 7.7-7.8. [#953](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/953)
+
+## 10.6.0
+ - Added `ecs_compatiblity` mode, for managing ECS-compatable templates [#952](https://github.com/logstash-plugins/logstash-output-elasticsearch/issue/952)
+
+## 10.5.1
+  - [DOC] Removed outdated compatibility notices, reworked cloud notice, and fixed formatting for `hosts` examples [#938](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/938)
+
+## 10.5.0
+  - Added api_key support [#934](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/934)
+
 ## 10.4.1
  - [DOC] Added note about `_type` setting change from `doc` to `_doc` [#884](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/884)
 

data/docs/index.asciidoc

@@ -21,6 +21,21 @@ include::{include_path}/plugin_header.asciidoc[]
 
 ==== Description
 
+If you plan to use the Kibana web interface to analyze data transformed by
+Logstash, use the Elasticsearch output plugin to get your data into
+Elasticsearch. 
+
+This output only speaks the HTTP protocol as it is the preferred protocol for
+interacting with Elasticsearch. In previous versions it was possible to
+communicate with Elasticsearch through the transport protocol, which is now
+reserved for internal cluster communication between nodes
+https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html[communication between nodes].
+Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol] 
+to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and
+will be removed in 8.0.0
+
+You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
+
 .Compatibility Note
 [NOTE]
 ================================================================================
@@ -36,47 +51,26 @@ ensure your template uses the `_doc` document-type before
 connecting to Elasticsearch 7.x.
 ================================================================================
 
-.Compatibility Note
-[NOTE]
-================================================================================
-Starting with Elasticsearch 5.3, there's an {ref}/modules-http.html[HTTP setting]
-called `http.content_type.required`. If this option is set to `true`, and you
-are using Logstash 2.4 through 5.2, you need to update the Elasticsearch output
-plugin to version 6.2.5 or higher.
+===== Hosted {es} Service on Elastic Cloud
 
-================================================================================
-
-If you plan to use the Kibana web
-interface, use the Elasticsearch output plugin to get your log data into
-Elasticsearch. 
-
-TIP: You can run Elasticsearch on your own hardware, or use our
+You can run Elasticsearch on your own hardware, or use our
 https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on
-Elastic Cloud. The Elasticsearch Service is available on both AWS and GCP.
-{ess-trial}[Try the {es} Service for free].
-
-This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0.
-We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower,
-yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having
-to upgrade Logstash in lock-step. 
-
-You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
-
-==== Template management for Elasticsearch 5.x
-
-Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch's mapping changes in version 5.0.
-Most importantly, the subfield for string multi-fields has changed from `.raw` to `.keyword` to match ES default
-behavior.
+Elastic Cloud. The Elasticsearch Service is available on AWS, Google Cloud
+Platform, and Microsoft Azure. {ess-trial}[Try the {es} Service for free].
 
-**Users installing ES 5.x and LS 5.x**
+==== Compatibility with the Elastic Common Schema (ECS)
 
-This change will not affect you and you will continue to use the ES defaults.
+This plugin will persist events to Elasticsearch in the shape produced by
+your pipeline, and _cannot_ be used to re-shape the event structure into a
+shape that complies with ECS. To produce events that fully comply with ECS,
+you will need to populate ECS-defined fields throughout your pipeline
+definition.
 
-**Users upgrading from LS 2.x to LS 5.x with ES 5.x**
-
-LS will not force upgrade the template, if `logstash` template already exists. This means you will still use
-`.raw` for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data after
-the new template is installed.
+However, the Elasticsearch Index Templates it manages can be configured to
+be ECS-compatible by setting <<plugins-{type}s-{plugin}-ecs_compatibility>>.
+By having an ECS-compatible template in place, we can ensure that Elasticsearch
+is prepared to create and index fields in a way that is compatible with ECS,
+and will correctly reject events with fields that conflict and cannot be coerced.
 
 ==== Writing to different indices: best practices
 
@@ -87,7 +81,7 @@ when using `ilm_rollover_alias`.
 
 ================================================================================
 
-If you're sending events to the same Elasticsearch cluster but you're targeting different indices you can:
+If you're sending events to the same Elasticsearch cluster, but you're targeting different indices you can:
 
  * use different Elasticsearch outputs, each one with a different value for the `index` parameter
  * use one Elasticsearch output and use the dynamic variable substitution for the `index` parameter
@@ -232,6 +226,9 @@ Elasticsearch] to take advantage of response compression when using this plugin
 For requests compression, regardless of the Elasticsearch version, users have to enable `http_compression` 
 setting in their Logstash config file.
 
+==== Authentication
+
+Authentication to a secure Elasticsearch cluster is possible using one of the `user`/`password`, `cloud_auth` or `api_key` options.
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Output Configuration Options
@@ -242,6 +239,7 @@ This plugin supports the following configuration options plus the <
 |=======================================================================
 |Setting |Input type|Required
 | <<plugins-{type}s-{plugin}-action>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-api_key>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-bulk_path>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|No
 | <<plugins-{type}s-{plugin}-cloud_auth>> |<<password,password>>|No
@@ -250,6 +248,7 @@ This plugin supports the following configuration options plus the <
 | <<plugins-{type}s-{plugin}-doc_as_upsert>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-document_id>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-document_type>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
 | <<plugins-{type}s-{plugin}-failure_type_logging_whitelist>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-healthcheck_path>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-hosts>> |<<uri,uri>>|No
@@ -324,6 +323,16 @@ The Elasticsearch action to perform. Valid actions are:
 
 For more details on actions, check out the http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
 
+[id="plugins-{type}s-{plugin}-api_key"]
+===== `api_key`
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+Authenticate using Elasticsearch API key. Note that this option also requires enabling the `ssl` option.
+
+Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key API].
+
 [id="plugins-{type}s-{plugin}-bulk_path"]
 ===== `bulk_path` 
 
@@ -399,6 +408,25 @@ If you don't set a value for this option:
 - for elasticsearch clusters 6.x: the value of 'doc' will be used;
 - for elasticsearch clusters 5.x and below: the event's 'type' field will be used, if the field is not present the value of 'doc' will be used.
 
+[id="plugins-{type}s-{plugin}-ecs_compatibility"]
+===== `ecs_compatibility`
+
+* Value type is <<string,string>>
+* Supported values are:
+** `disabled`: does not provide ECS-compatible templates
+** `v1`: provides defaults that are compatible with v1 of the Elastic Common Schema
+* Default value depends on which version of Logstash is running:
+** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
+** Otherwise, the default value is `disabled`.
+
+Controls this plugin's compatibility with the https://www.elastic.co/guide/en/ecs/current/index.html[Elastic Common Schema (ECS)],
+including the installation of ECS-compatible index templates.
+The value of this setting affects the _default_ values of:
+
+* <<plugins-{type}s-{plugin}-index>>
+* <<plugins-{type}s-{plugin}-template_name>>
+* <<plugins-{type}s-{plugin}-ilm_rollover_alias>>
+
 [id="plugins-{type}s-{plugin}-failure_type_logging_whitelist"]
 ===== `failure_type_logging_whitelist` 
 
@@ -439,11 +467,15 @@ If you have custom firewall rules you may need to change this
 
 Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the `hosts` parameter.
 Remember the `http` protocol uses the http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[http] address (eg. 9200, not 9300).
+
+Examples:
+
     `"127.0.0.1"`
     `["127.0.0.1:9200","127.0.0.2:9200"]`
     `["http://127.0.0.1"]`
     `["https://127.0.0.1:9200"]`
     `["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)
+
 It is important to exclude http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated master nodes] from the `hosts` list
 to prevent LS from sending bulk requests to the master nodes.  So this parameter should only reference either data or client nodes in Elasticsearch.
 
@@ -502,7 +534,9 @@ NOTE: If this setting is specified, the policy must already exist in Elasticsear
 ===== `ilm_rollover_alias`
 
   * Value type is <<string,string>>
-  * Default value is `logstash`
+  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
+  ** ECS Compatibility disabled: `logstash`
+  ** ECS Compatibility enabled: `ecs-logstash`
 
 The rollover alias is the alias where indices managed using Index Lifecycle Management will be written to.
 
@@ -516,7 +550,9 @@ NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as `in
 ===== `index` 
 
   * Value type is <<string,string>>
-  * Default value is `"logstash-%{+yyyy.MM.dd}"`
+  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
+  ** ECS Compatibility disabled: `"logstash-%{+yyyy.MM.dd}"`
+  ** ECS Compatibility enabled: `"ecs-logstash-%{+yyyy.MM.dd}"`
 
 The index to write events to. This can be dynamic using the `%{foo}` syntax.
 The default value will partition your indices by day so you can more easily
@@ -550,7 +586,8 @@ Set the keystore password
   * Default value is `true`
 
 From Logstash 1.3 onwards, a template is applied to Elasticsearch during
-Logstash's startup if one with the name `template_name` does not already exist.
+Logstash's startup if one with the name <<plugins-{type}s-{plugin}-template_name>>
+does not already exist.
 By default, the contents of this template is the default template for
 `logstash-%{+YYYY.MM.dd}` which always matches indices based on the pattern
 `logstash-*`.  Should you require support for other index names, or would like
@@ -801,7 +838,10 @@ If not set, the included template will be used.
 ===== `template_name` 
 
   * Value type is <<string,string>>
-  * Default value is `"logstash"`
+  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
+  ** ECS Compatibility disabled: `logstash`
+  ** ECS Compatibility enabled: `ecs-logstash`
+
 
 This configuration option defines how the template is named inside Elasticsearch.
 Note that if you have used the template management features and subsequently

data/lib/logstash/outputs/elasticsearch.rb

@@ -92,6 +92,8 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   require "logstash/outputs/elasticsearch/common"
   require "logstash/outputs/elasticsearch/ilm"
 
+  require 'logstash/plugin_mixins/ecs_compatibility_support'
+
   # Protocol agnostic (i.e. non-http, non-java specific) configs go here
   include(LogStash::Outputs::ElasticSearch::CommonConfigs)
 
@@ -101,6 +103,9 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # Methods for ILM support
   include(LogStash::Outputs::ElasticSearch::Ilm)
 
+  # ecs_compatibility option, provided by Logstash core or the support adapter.
+  include(LogStash::PluginMixins::ECSCompatibilitySupport)
+
   config_name "elasticsearch"
 
   # The Elasticsearch action to perform. Valid actions are:
@@ -122,6 +127,10 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # Password to authenticate to a secure Elasticsearch cluster
   config :password, :validate => :password
 
+  # Authenticate using Elasticsearch API key.
+  # format is id:api_key (as returned by https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key])
+  config :api_key, :validate => :password
+
   # Cloud authentication string ("<username>:<password>" format) is an alternative for the `user`/`password` configuration.
   #
   # For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_auth[cloud documentation]
@@ -238,6 +247,34 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # Custom Headers to send on each request to elasticsearch nodes
   config :custom_headers, :validate => :hash, :default => {}
 
+  def initialize(*params)
+    super
+    setup_ecs_compatibility_related_defaults
+  end
+
+  def setup_ecs_compatibility_related_defaults
+    case ecs_compatibility
+    when :disabled
+      @default_index = "logstash-%{+yyyy.MM.dd}"
+      @default_ilm_rollover_alias = "logstash"
+      @default_template_name = 'logstash'
+    when :v1
+      @default_index = "ecs-logstash-%{+yyyy.MM.dd}"
+      @default_ilm_rollover_alias = "ecs-logstash"
+      @default_template_name = 'ecs-logstash'
+    else
+      fail("unsupported ECS Compatibility `#{ecs_compatibility}`")
+    end
+
+    @index ||= default_index
+    @ilm_rollover_alias ||= default_ilm_rollover_alias
+    @template_name ||= default_template_name
+  end
+
+  attr_reader :default_index
+  attr_reader :default_ilm_rollover_alias
+  attr_reader :default_template_name
+
   # @override to handle proxy => '' as if none was set
   def config_init(params)
     proxy = params['proxy']
@@ -255,6 +292,14 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   end
 
   def build_client
+    # the following 3 options validation & setup methods are called inside build_client
+    # because they must be executed prior to building the client and logstash
+    # monitoring and management rely on directly calling build_client
+    # see https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/934#pullrequestreview-396203307
+    validate_authentication
+    fill_hosts_from_cloud_id
+    setup_hosts
+
     params["metric"] = metric
     if @proxy.eql?('')
       @logger.warn "Supplied proxy setting (proxy => '') has no effect"

data/lib/logstash/outputs/elasticsearch/common.rb

@@ -20,10 +20,6 @@ module LogStash; module Outputs; class ElasticSearch;
       @stopping = Concurrent::AtomicBoolean.new(false)
       # To support BWC, we check if DLQ exists in core (< 5.4). If it doesn't, we use nil to resort to previous behavior.
       @dlq_writer = dlq_enabled? ? execution_context.dlq_writer : nil
-
-      fill_hosts_from_cloud_id
-      fill_user_password_from_cloud_auth
-      setup_hosts # properly sets @hosts
       build_client
       setup_after_successful_connection
       check_action_validity
@@ -64,8 +60,15 @@ module LogStash; module Outputs; class ElasticSearch;
       !!maximum_seen_major_version
     end
 
-    def use_event_type?(client)
-      client.maximum_seen_major_version < 8
+    ##
+    # WARNING: This method is overridden in a subclass in Logstash Core 7.7-7.8's monitoring,
+    #          where a `client` argument is both required and ignored. In later versions of
+    #          Logstash Core it is optional and ignored, but to make it optional here would
+    #          allow us to accidentally break compatibility with Logstashes where it was required.
+    # @param noop_required_client [nil]: required `nil` for legacy reasons.
+    # @return [Boolean]
+    def use_event_type?(noop_required_client)
+      maximum_seen_major_version < 8
     end
 
     # Convert the event into a 3-tuple of action, params, and event
@@ -78,7 +81,7 @@ module LogStash; module Outputs; class ElasticSearch;
         routing_field_name => @routing ? event.sprintf(@routing) : nil
       }
 
-      params[:_type] = get_event_type(event) if use_event_type?(client)
+      params[:_type] = get_event_type(event) if use_event_type?(nil)
 
       if @pipeline
         params[:pipeline] = event.sprintf(@pipeline)
@@ -112,6 +115,28 @@ module LogStash; module Outputs; class ElasticSearch;
       [action, params, event]
     end
 
+    def validate_authentication
+      authn_options = 0
+      authn_options += 1 if @cloud_auth
+      authn_options += 1 if (@api_key && @api_key.value)
+      authn_options += 1 if (@user || (@password && @password.value))
+
+      if authn_options > 1
+        raise LogStash::ConfigurationError, 'Multiple authentication options are specified, please only use one of user/password, cloud_auth or api_key'
+      end
+
+      if @api_key && @api_key.value && @ssl   != true
+        raise(LogStash::ConfigurationError, "Using api_key authentication requires SSL/TLS secured communication using the `ssl => true` option")
+      end
+
+      if @cloud_auth
+        @user, @password = parse_user_password_from_cloud_auth(@cloud_auth)
+        # params is the plugin global params hash which will be passed to HttpClientBuilder.build
+        params['user'], params['password'] = @user, @password
+      end
+    end
+    private :validate_authentication
+
     def setup_hosts
       @hosts = Array(@hosts)
       if @hosts.empty?
@@ -135,16 +160,6 @@ module LogStash; module Outputs; class ElasticSearch;
       @hosts = parse_host_uri_from_cloud_id(@cloud_id)
     end
 
-    def fill_user_password_from_cloud_auth
-      return unless @cloud_auth
-
-      if @user || @password
-        raise LogStash::ConfigurationError, 'Both cloud_auth and user/password specified, please only use one.'
-      end
-      @user, @password = parse_user_password_from_cloud_auth(@cloud_auth)
-      params['user'], params['password'] = @user, @password
-    end
-
     def parse_host_uri_from_cloud_id(cloud_id)
       begin # might not be available on older LS
         require 'logstash/util/cloud_setting_id'
@@ -339,11 +354,11 @@ module LogStash; module Outputs; class ElasticSearch;
       type = if @document_type
                event.sprintf(@document_type)
              else
-               if client.maximum_seen_major_version < 6
+               if maximum_seen_major_version < 6
                  event.get("type") || DEFAULT_EVENT_TYPE_ES6
-               elsif client.maximum_seen_major_version == 6
+               elsif maximum_seen_major_version == 6
                  DEFAULT_EVENT_TYPE_ES6
-               elsif client.maximum_seen_major_version == 7
+               elsif maximum_seen_major_version == 7
                  DEFAULT_EVENT_TYPE_ES7
                else
                  nil
@@ -428,7 +443,7 @@ module LogStash; module Outputs; class ElasticSearch;
     end
 
     def default_index?(index)
-      @index == LogStash::Outputs::ElasticSearch::CommonConfigs::DEFAULT_INDEX_NAME
+      @index == @default_index
     end
 
     def dlq_enabled?

data/lib/logstash/outputs/elasticsearch/common_configs.rb

@@ -17,7 +17,7 @@ module LogStash; module Outputs; class ElasticSearch
       # For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}.
       # LS uses Joda to format the index pattern from event timestamp.
       # Joda formats are defined http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[here].
-      mod.config :index, :validate => :string, :default => DEFAULT_INDEX_NAME
+      mod.config :index, :validate => :string
 
       mod.config :document_type, 
         :validate => :string, 
@@ -44,7 +44,7 @@ module LogStash; module Outputs; class ElasticSearch
       # `curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>`
       #
       # where `OldTemplateName` is whatever the former setting was.
-      mod.config :template_name, :validate => :string, :default => "logstash"
+      mod.config :template_name, :validate => :string
 
       # You can set the path to your own template here, if you so desire.
       # If not set, the included template will be used.
@@ -153,7 +153,7 @@ module LogStash; module Outputs; class ElasticSearch
       mod.config :ilm_enabled, :validate => [true, false, 'true', 'false', 'auto'], :default => 'auto'
 
       # Rollover alias used for indexing data. If rollover alias doesn't exist, Logstash will create it and map it to the relevant index
-      mod.config :ilm_rollover_alias, :validate => :string, :default => DEFAULT_ROLLOVER_ALIAS
+      mod.config :ilm_rollover_alias, :validate => :string
 
       # appends “{now/d}-000001” by default for new index creation, subsequent rollover indices will increment based on this pattern i.e. “000002”
       # {now/d} is date math, and will insert the appropriate value automatically.