logstash-output-elasticsearch 11.12.4-java → 11.15.9-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a912267d383b51082d4c1b68e21d83dcb506433abc4f3f12bce0465eef84f439
-  data.tar.gz: dc69623c94ad47986ed07f25818eeabfbfdd49532dfd9970f8c638dfe950f615
+  metadata.gz: c9e537b9f31644ce80834b295b99d22566863f666ab319efc34f641c15018d74
+  data.tar.gz: a99f63dd55f4b0a12e597e812db124dd9a7fe82ce1a5e7af4057992f903eac65
 SHA512:
-  metadata.gz: 99568149dc8d313460046ca2e247c43e2e505e6649441a22a365e0d5484ba493d5d91c4eedfb5700fc447f64122555db3b92d6b59513a6b42e07dcc43dda3ada
-  data.tar.gz: 56d2ce9c942f2dffbfc850f8feec0368dc2456ee08f7d7f46b83de94f9439fdd9047a1bae9231cc41caf45105a1abda0d1e60429a37059709f337e5ea8f8b6e4
+  metadata.gz: 12fa3b203130210b5d274364ff97e31bfb01aaedd23ac22fc53fea1626cad628d3f33e952dcf12555fc4860d7577235684e255550dfc7668d9dc93d7e6bf55ff
+  data.tar.gz: 50ca989af2afc85f439995c6dde9c7eeda56924c9d9729ef91426b34dc99146fadecf3e290dc7e122113bb2cbf50bdc1eeac22f8448d2d4373b0d251660fb6a7

data/CHANGELOG.md

@@ -1,3 +1,61 @@
+## 11.15.9
+  - allow dlq_ settings when using data streams [#1144](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1144)
+
+## 11.15.8
+  - Fixes a regression introduced in 11.14.0 which could prevent Logstash 8.8 from establishing a connection to Elasticsearch for Central Management and Monitoring core features [#1141](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1141)
+
+## 11.15.7
+  - Fixes a regression introduced in 11.14.0 which could prevent a connection from being established to Elasticsearch in some SSL configurations [#1138](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1138)
+
+## 11.15.6
+  - Fix: avoid to reject a batch when the Elasticsearch connection is alive and the processing should continue [#1132](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1132).
+
+## 11.15.5
+  - Fixes `undefined 'shutdown_requested' method` error when plugin checks if shutdown request is received [#1134](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1134)
+
+## 11.15.4
+  - Improved connection handling under several partial-failure scenarios [#1130](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1130)
+   - Ensures an HTTP connection can be established before adding the connection to the pool
+   - Ensures that the version of the connected Elasticsearch is retrieved _successfully_ before the connection is added to the pool.
+   - Fixes a crash that could occur when the plugin is configured to connect to a live HTTP resource that is _not_ Elasticsearch
+
+## 11.15.3
+  -  Removes the ECS v8 unreleased preview warning [#1131](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1131)
+
+## 11.15.2
+ - Restores DLQ logging behavior from 11.8.x to include the action-tuple as structured [#1105](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1105)
+
+## 11.15.1
+ - Move async finish_register to bottom of register to avoid race condition [#1125](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1125)
+
+## 11.15.0
+ - Added the ability to negatively acknowledge the batch under processing if the plugin is blocked in a retry-error-loop and a shutdown is requested. [#1119](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1119)
+
+## 11.14.1
+ - [DOC] Fixed incorrect pull request link on the CHANGELOG `11.14.0` entry [#1122](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1122)
+
+## 11.14.0
+ - Added SSL settings for: [#1118](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1118)
+   - `ssl_truststore_type`: The format of the truststore file
+   - `ssl_keystore_type`: The format of the keystore file
+   - `ssl_certificate`: OpenSSL-style X.509 certificate file to authenticate the client
+   - `ssl_key`: OpenSSL-style RSA private key that corresponds to the `ssl_certificate`
+   - `ssl_cipher_suites`: The list of cipher suites
+ - Reviewed and deprecated SSL settings to comply with Logstash's naming convention
+   - Deprecated `ssl` in favor of `ssl_enabled`
+   - Deprecated `cacert` in favor of `ssl_certificate_authorities`
+   - Deprecated `keystore` in favor of `ssl_keystore_path`
+   - Deprecated `keystore_password` in favor of `ssl_keystore_password`
+   - Deprecated `truststore` in favor of `ssl_truststore_path`
+   - Deprecated `truststore_password` in favor of `ssl_truststore_password`
+   - Deprecated `ssl_certificate_verification` in favor of `ssl_verification_mode`
+
+## 11.13.1
+ - Avoid crash by ensuring ILM settings are injected in the correct location depending on the default (or custom) template format, template_api setting and ES version [#1102](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1102)
+
+## 11.13.0
+ - add technology preview support for allowing events to individually encode a default pipeline with `[@metadata][target_ingest_pipeline]` (as part of a technology preview, this feature may change without notice) [#1113](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1113)
+
 ## 11.12.4
  - Changed the `manage_template` default value to `false` when data streams is enabled [#1111](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1111)
    - Added the `manage_template => false` as a valid data stream option

data/docs/index.asciidoc

@@ -299,7 +299,7 @@ checks.
 ==== Elasticsearch Output Configuration Options
 
 This plugin supports the following configuration options plus the
-<<plugins-{type}s-{plugin}-common-options>> described later.
+<<plugins-{type}s-{plugin}-common-options>> and the <<plugins-{type}s-{plugin}-deprecated-options>> described later.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -307,7 +307,6 @@ This plugin supports the following configuration options plus the
 | <<plugins-{type}s-{plugin}-action>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-api_key>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-bulk_path>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|No
 | <<plugins-{type}s-{plugin}-ca_trusted_fingerprint>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-cloud_auth>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-cloud_id>> |<<string,string>>|No
@@ -333,8 +332,6 @@ This plugin supports the following configuration options plus the
 | <<plugins-{type}s-{plugin}-ilm_policy>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-ilm_rollover_alias>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-index>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-keystore>> |a valid filesystem path|No
-| <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-silence_errors_in_log>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-manage_template>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-parameters>> |<<hash,hash>>|No
@@ -358,16 +355,24 @@ This plugin supports the following configuration options plus the
 | <<plugins-{type}s-{plugin}-sniffing>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-sniffing_delay>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-sniffing_path>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-ssl>> |<<boolean,boolean>>|No
-| <<plugins-{type}s-{plugin}-ssl_certificate_verification>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-ssl_certificate>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-ssl_certificate_authorities>> |list of <<path,path>>|No
+| <<plugins-{type}s-{plugin}-ssl_cipher_suites>> |list of <<string,string>>|No
+| <<plugins-{type}s-{plugin}-ssl_enabled>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-ssl_key>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-ssl_keystore_password>> |<<password,password>>|No
+| <<plugins-{type}s-{plugin}-ssl_keystore_path>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-ssl_keystore_type>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-ssl_supported_protocols>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ssl_truststore_password>> |<<password,password>>|No
+| <<plugins-{type}s-{plugin}-ssl_truststore_path>> |<<path,path>>|No
+| <<plugins-{type}s-{plugin}-ssl_truststore_type>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ssl_verification_mode>> |<<string,string>>, one of `["full", "none"]`|No
 | <<plugins-{type}s-{plugin}-template>> |a valid filesystem path|No
 | <<plugins-{type}s-{plugin}-template_api>> |<<string,string>>, one of `["auto", "legacy", "composable"]`|No
 | <<plugins-{type}s-{plugin}-template_name>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-template_overwrite>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-timeout>> |<<number,number>>|No
-| <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|No
-| <<plugins-{type}s-{plugin}-truststore_password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-upsert>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-user>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-validate_after_inactivity>> |<<number,number>>|No
@@ -408,7 +413,7 @@ For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch bu
   * There is no default value for this setting.
 
 Authenticate using Elasticsearch API key.
-Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl,`ssl => true`>>.
+Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl,`ssl_enabled => true`>>.
 
 Format is `id:api_key` where `id` and `api_key` are as returned by the
 Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
@@ -422,14 +427,6 @@ Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
 HTTP Path to perform the _bulk requests to
 this defaults to a concatenation of the path parameter and "_bulk"
 
-[id="plugins-{type}s-{plugin}-cacert"]
-===== `cacert`
-
-  * Value type is <<path,path>>
-  * There is no default value for this setting.
-
-The .cer or .pem file to validate the server's certificate.
-
 [id="plugins-{type}s-{plugin}-ca_trusted_fingerprint"]
 ===== `ca_trusted_fingerprint`
 
@@ -769,23 +766,6 @@ Logstash uses
 http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[Joda
 formats] and the `@timestamp` field of each event is being used as source for the date.
 
-[id="plugins-{type}s-{plugin}-keystore"]
-===== `keystore`
-
-  * Value type is <<path,path>>
-  * There is no default value for this setting.
-
-The keystore used to present a certificate to the server.
-It can be either .jks or .p12
-
-[id="plugins-{type}s-{plugin}-keystore_password"]
-===== `keystore_password`
-
-  * Value type is <<password,password>>
-  * There is no default value for this setting.
-
-Set the keystore password
-
 [id="plugins-{type}s-{plugin}-manage_template"]
 ===== `manage_template`
 
@@ -849,12 +829,11 @@ not also set this field. That will raise an error at startup
 ===== `pipeline`
 
   * Value type is <<string,string>>
-  * Default value is `nil`
+  * There is no default value.
 
 Set which ingest pipeline you wish to execute for an event. You can also use
-event dependent configuration here like `pipeline =>
-"%{[@metadata][pipeline]}"`. The pipeline parameter won't be set if the value
-resolves to empty string ("").
+event dependent configuration here like `pipeline => "%{[@metadata][pipeline]}"`.
+The pipeline parameter won't be set if the value resolves to empty string ("").
 
 [id="plugins-{type}s-{plugin}-pool_max"]
 ===== `pool_max`
@@ -1035,8 +1014,35 @@ the default value is computed by concatenating the path value and "_nodes/http"
 if sniffing_path is set it will be used as an absolute path
 do not use full URL here, only paths, e.g. "/sniff/_nodes/http"
 
-[id="plugins-{type}s-{plugin}-ssl"]
-===== `ssl`
+[id="plugins-{type}s-{plugin}-ssl_certificate"]
+===== `ssl_certificate`
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+SSL certificate to use to authenticate the client. This certificate should be an OpenSSL-style X.509 certificate file.
+
+NOTE: This setting can be used only if <<plugins-{type}s-{plugin}-ssl_key>> is set.
+
+[id="plugins-{type}s-{plugin}-ssl_certificate_authorities"]
+===== `ssl_certificate_authorities`
+
+  * Value type is a list of <<path,path>>
+  * There is no default value for this setting
+
+The .cer or .pem files to validate the server's certificate.
+
+NOTE: You cannot use this setting and <<plugins-{type}s-{plugin}-ssl_truststore_path>> at the same time.
+
+[id="plugins-{type}s-{plugin}-ssl_cipher_suites"]
+===== `ssl_cipher_suites`
+  * Value type is a list of <<string,string>>
+  * There is no default value for this setting
+
+The list of cipher suites to use, listed by priorities.
+Supported cipher suites vary depending on the Java and protocol versions.
+
+[id="plugins-{type}s-{plugin}-ssl_enabled"]
+===== `ssl_enabled`
 
   * Value type is <<boolean,boolean>>
   * There is no default value for this setting.
@@ -1045,15 +1051,41 @@ Enable SSL/TLS secured communication to Elasticsearch cluster.
 Leaving this unspecified will use whatever scheme is specified in the URLs listed in <<plugins-{type}s-{plugin}-hosts>> or extracted from the <<plugins-{type}s-{plugin}-cloud_id>>.
 If no explicit protocol is specified plain HTTP will be used.
 
-[id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
-===== `ssl_certificate_verification`
+[id="plugins-{type}s-{plugin}-ssl_key"]
+===== `ssl_key`
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
 
-  * Value type is <<boolean,boolean>>
-  * Default value is `true`
+OpenSSL-style RSA private key that corresponds to the <<plugins-{type}s-{plugin}-ssl_certificate>>.
 
-Option to validate the server's certificate. Disabling this severely compromises security.
-For more information on disabling certificate verification please read
-https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
+NOTE: This setting can be used only if <<plugins-{type}s-{plugin}-ssl_certificate>> is set.
+
+[id="plugins-{type}s-{plugin}-ssl_keystore_password"]
+===== `ssl_keystore_password`
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+Set the keystore password
+
+[id="plugins-{type}s-{plugin}-ssl_keystore_path"]
+===== `ssl_keystore_path`
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+The keystore used to present a certificate to the server.
+It can be either `.jks` or `.p12`
+
+NOTE: You cannot use this setting and <<plugins-{type}s-{plugin}-ssl_certificate>> at the same time.
+
+[id="plugins-{type}s-{plugin}-ssl_keystore_type"]
+===== `ssl_keystore_type`
+
+  * Value can be any of: `jks`, `pkcs12`
+  * If not provided, the value will be inferred from the keystore filename.
+
+The format of the keystore file. It must be either `jks` or `pkcs12`.
 
 [id="plugins-{type}s-{plugin}-ssl_supported_protocols"]
 ===== `ssl_supported_protocols`
@@ -1065,13 +1097,56 @@ https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
 
 List of allowed SSL/TLS versions to use when establishing a connection to the Elasticsearch cluster.
 
-For Java 8 `'TLSv1.3'` is supported  only since **8u262** (AdoptOpenJDK), but requires that you set the
+For Java 8 `'TLSv1.3'` is supported only since **8u262** (AdoptOpenJDK), but requires that you set the
 `LS_JAVA_OPTS="-Djdk.tls.client.protocols=TLSv1.3"` system property in Logstash.
 
 NOTE: If you configure the plugin to use `'TLSv1.1'` on any recent JVM, such as the one packaged with Logstash,
 the protocol is disabled by default and needs to be enabled manually by changing `jdk.tls.disabledAlgorithms` in
 the *$JDK_HOME/conf/security/java.security* configuration file. That is, `TLSv1.1` needs to be removed from the list.
 
+[id="plugins-{type}s-{plugin}-ssl_truststore_password"]
+===== `ssl_truststore_password`
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+Set the truststore password
+
+[id="plugins-{type}s-{plugin}-ssl_truststore_path"]
+===== `ssl_truststore_path`
+
+  * Value type is <<path,path>>
+  * There is no default value for this setting.
+
+The truststore to validate the server's certificate.
+It can be either `.jks` or `.p12`.
+
+NOTE: You cannot use this setting and <<plugins-{type}s-{plugin}-ssl_certificate_authorities>> at the same time.
+
+[id="plugins-{type}s-{plugin}-ssl_truststore_type"]
+===== `ssl_truststore_type`
+
+* Value can be any of: `jks`, `pkcs12`
+* If not provided, the value will be inferred from the truststore filename.
+
+The format of the truststore file. It must be either `jks` or `pkcs12`.
+
+[id="plugins-{type}s-{plugin}-ssl_verification_mode"]
+===== `ssl_verification_mode`
+
+  * Value can be any of: `full`, `none`
+  * Default value is `full`
+
+Defines how to verify the certificates presented by another party in the TLS connection:
+
+`full` validates that the server certificate has an issue date that’s within
+the not_before and not_after dates; chains to a trusted Certificate Authority (CA), and
+has a hostname or IP address that matches the names within the certificate.
+
+`none` performs no certificate validation.
+
+WARNING: Setting certificate verification to `none` disables many security benefits of SSL/TLS, which is very dangerous. For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
+
 [id="plugins-{type}s-{plugin}-template"]
 ===== `template`
 
@@ -1140,24 +1215,6 @@ the "logstash" template (i.e. removing all customized settings)
 Set the timeout, in seconds, for network operations and requests sent Elasticsearch. If
 a timeout occurs, the request will be retried.
 
-[id="plugins-{type}s-{plugin}-truststore"]
-===== `truststore`
-
-  * Value type is <<path,path>>
-  * There is no default value for this setting.
-
-The truststore to validate the server's certificate.
-It can be either .jks or .p12.
-Use either `:truststore` or `:cacert`.
-
-[id="plugins-{type}s-{plugin}-truststore_password"]
-===== `truststore_password`
-
-  * Value type is <<password,password>>
-  * There is no default value for this setting.
-
-Set the truststore password
-
 [id="plugins-{type}s-{plugin}-upsert"]
 ===== `upsert`
 
@@ -1214,6 +1271,97 @@ https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
 blog] and {ref}/docs-index_.html#_version_types[Version types] in the
 Elasticsearch documentation.
 
+[id="plugins-{type}s-{plugin}-deprecated-options"]
+==== Elasticsearch Output Deprecated Configuration Options
+
+This plugin supports the following deprecated configurations.
+
+WARNING: Deprecated options are subject to removal in future releases.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting|Input type|Replaced by
+| <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
+| <<plugins-{type}s-{plugin}-keystore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_keystore_path>>
+| <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_keystore_password>>
+| <<plugins-{type}s-{plugin}-ssl>> |<<boolean,boolean>>|<<plugins-{type}s-{plugin}-ssl_enabled>>
+| <<plugins-{type}s-{plugin}-ssl_certificate_verification>> |<<boolean,boolean>>|<<plugins-{type}s-{plugin}-ssl_verification_mode>>
+| <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_truststore_path>>
+| <<plugins-{type}s-{plugin}-truststore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_truststore_password>>
+|=======================================================================
+
+
+[id="plugins-{type}s-{plugin}-cacert"]
+===== `cacert`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_certificate_authorities>>]
+
+* Value type is a list of <<path,path>>
+* There is no default value for this setting.
+
+The .cer or .pem file to validate the server's certificate.
+
+[id="plugins-{type}s-{plugin}-keystore"]
+===== `keystore`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_path>>]
+
+* Value type is <<path,path>>
+* There is no default value for this setting.
+
+The keystore used to present a certificate to the server.
+It can be either .jks or .p12
+
+NOTE: You cannot use this setting and <<plugins-{type}s-{plugin}-ssl_certificate>> at the same time.
+
+[id="plugins-{type}s-{plugin}-keystore_password"]
+===== `keystore_password`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_password>>]
+
+* Value type is <<password,password>>
+* There is no default value for this setting.
+
+Set the keystore password
+
+[id="plugins-{type}s-{plugin}-ssl"]
+===== `ssl`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_enabled>>]
+
+* Value type is <<boolean,boolean>>
+* There is no default value for this setting.
+
+Enable SSL/TLS secured communication to Elasticsearch cluster.
+Leaving this unspecified will use whatever scheme is specified in the URLs listed in <<plugins-{type}s-{plugin}-hosts>> or extracted from the <<plugins-{type}s-{plugin}-cloud_id>>.
+If no explicit protocol is specified plain HTTP will be used.
+
+[id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
+===== `ssl_certificate_verification`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_verification_mode>>]
+
+* Value type is <<boolean,boolean>>
+* Default value is `true`
+
+Option to validate the server's certificate. Disabling this severely compromises security.
+For more information on disabling certificate verification please read
+https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
+
+[id="plugins-{type}s-{plugin}-truststore"]
+===== `truststore`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_truststore_path>>]
+
+* Value type is <<path,path>>
+* There is no default value for this setting.
+
+The truststore to validate the server's certificate.
+It can be either `.jks` or `.p12`.
+Use either `:truststore` or `:cacert`.
+
+[id="plugins-{type}s-{plugin}-truststore_password"]
+===== `truststore_password`
+deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_truststore_password>>]
+
+* Value type is <<password,password>>
+* There is no default value for this setting.
+
+Set the truststore password
 
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]

data/lib/logstash/outputs/elasticsearch/http_client/pool.rb

@@ -229,14 +229,16 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
     end
 
     def health_check_request(url)
-      logger.debug("Running health check to see if an ES connection is working", url: url.sanitized.to_s, path: @healthcheck_path)
-      perform_request_to_url(url, :head, @healthcheck_path)
+      response = perform_request_to_url(url, :head, @healthcheck_path)
+      raise BadResponseCodeError.new(response.code, url, nil, response.body) unless (200..299).cover?(response.code)
     end
 
     def healthcheck!(register_phase = true)
       # Try to keep locking granularity low such that we don't affect IO...
       @state_mutex.synchronize { @url_info.select {|url,meta| meta[:state] != :alive } }.each do |url,meta|
         begin
+          logger.debug("Running health check to see if an Elasticsearch connection is working",
+                        :healthcheck_url => url.sanitized.to_s, :path => @healthcheck_path)
           health_check_request(url)
 
           # when called from resurrectionist skip the product check done during register phase
@@ -249,6 +251,10 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
           logger.warn("Restored connection to ES instance", url: url.sanitized.to_s)
           # We reconnected to this node, check its ES version
           es_version = get_es_version(url)
+          if es_version.nil?
+            logger.warn("Failed to retrieve Elasticsearch version data from connected endpoint, connection aborted", :url => url.sanitized.to_s)
+            next
+          end
           @state_mutex.synchronize do
             meta[:version] = es_version
             set_last_es_version(es_version, url)
@@ -464,8 +470,12 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
     end
 
     def get_es_version(url)
-      request = perform_request_to_url(url, :get, ROOT_URI_PATH)
-      LogStash::Json.load(request.body)["version"]["number"] # e.g. "7.10.0"
+      response = perform_request_to_url(url, :get, ROOT_URI_PATH)
+      return nil unless (200..299).cover?(response.code)
+
+      response = LogStash::Json.load(response.body)
+
+      response.fetch('version', {}).fetch('number', nil)
     end
 
     def last_es_version

data/lib/logstash/outputs/elasticsearch/http_client_builder.rb

@@ -107,38 +107,55 @@ module LogStash; module Outputs; class ElasticSearch;
     end
 
     def self.setup_ssl(logger, params)
-      params["ssl"] = true if params["hosts"].any? {|h| h.scheme == "https" }
-      return {} if params["ssl"].nil?
+      params["ssl_enabled"] = true if params["hosts"].any? {|h| h.scheme == "https" }
+      return {} if params["ssl_enabled"].nil?
 
-      return {:ssl => {:enabled => false}} if params["ssl"] == false
+      return {:ssl => {:enabled => false}} if params["ssl_enabled"] == false
 
-      cacert, truststore, truststore_password, keystore, keystore_password =
-        params.values_at('cacert', 'truststore', 'truststore_password', 'keystore', 'keystore_password')
+      ssl_certificate_authorities, ssl_truststore_path, ssl_certificate, ssl_keystore_path = params.values_at('ssl_certificate_authorities', 'ssl_truststore_path', 'ssl_certificate', 'ssl_keystore_path')
 
-      if cacert && truststore
-        raise(LogStash::ConfigurationError, "Use either \"cacert\" or \"truststore\" when configuring the CA certificate") if truststore
+      if ssl_certificate_authorities && ssl_truststore_path
+        raise LogStash::ConfigurationError, 'Use either "ssl_certificate_authorities/cacert" or "ssl_truststore_path/truststore" when configuring the CA certificate'
+      end
+
+      if ssl_certificate && ssl_keystore_path
+        raise LogStash::ConfigurationError, 'Use either "ssl_certificate" or "ssl_keystore_path/keystore" when configuring client certificates'
       end
 
       ssl_options = {:enabled => true}
 
-      if cacert
-        ssl_options[:ca_file] = cacert
-      elsif truststore
-        ssl_options[:truststore_password] = truststore_password.value if truststore_password
+      if ssl_certificate_authorities&.any?
+        raise LogStash::ConfigurationError, 'Multiple values on "ssl_certificate_authorities" are not supported by this plugin' if ssl_certificate_authorities.size > 1
+        ssl_options[:ca_file] = ssl_certificate_authorities.first
       end
 
-      ssl_options[:truststore] = truststore if truststore
-      if keystore
-        ssl_options[:keystore] = keystore
-        ssl_options[:keystore_password] = keystore_password.value if keystore_password
+      setup_ssl_store(ssl_options, 'truststore', params)
+      setup_ssl_store(ssl_options, 'keystore', params)
+
+      ssl_key = params["ssl_key"]
+      if ssl_certificate
+        raise LogStash::ConfigurationError, 'Using an "ssl_certificate" requires an "ssl_key"' unless ssl_key
+        ssl_options[:client_cert] = ssl_certificate
+        ssl_options[:client_key] = ssl_key
+      elsif !ssl_key.nil?
+        raise LogStash::ConfigurationError, 'An "ssl_certificate" is required when using an "ssl_key"'
       end
 
-      if !params["ssl_certificate_verification"]
-        logger.warn "You have enabled encryption but DISABLED certificate verification, " +
-                    "to make sure your data is secure remove `ssl_certificate_verification => false`"
-        ssl_options[:verify] = :disable # false accepts self-signed but still validates hostname
+      ssl_verification_mode = params["ssl_verification_mode"]
+      unless ssl_verification_mode.nil?
+        case ssl_verification_mode
+        when 'none'
+          logger.warn "You have enabled encryption but DISABLED certificate verification, " +
+                        "to make sure your data is secure set `ssl_verification_mode => full`"
+          ssl_options[:verify] = :disable
+        else
+          # Manticore's :default maps to Apache HTTP Client's DefaultHostnameVerifier,
+          # which is the modern STRICT verifier that replaces the deprecated StrictHostnameVerifier
+          ssl_options[:verify] = :default
+        end
       end
 
+      ssl_options[:cipher_suites] = params["ssl_cipher_suites"] if params.include?("ssl_cipher_suites")
       ssl_options[:trust_strategy] = params["ssl_trust_strategy"] if params.include?("ssl_trust_strategy")
 
       protocols = params['ssl_supported_protocols']
@@ -147,6 +164,16 @@ module LogStash; module Outputs; class ElasticSearch;
       { ssl: ssl_options }
     end
 
+    # @param kind is a string [truststore|keystore]
+    def self.setup_ssl_store(ssl_options, kind, params)
+      store_path = params["ssl_#{kind}_path"]
+      if store_path
+        ssl_options[kind.to_sym] = store_path
+        ssl_options["#{kind}_type".to_sym] = params["ssl_#{kind}_type"] if params.include?("ssl_#{kind}_type")
+        ssl_options["#{kind}_password".to_sym] = params["ssl_#{kind}_password"].value if params.include?("ssl_#{kind}_password")
+      end
+    end
+
     def self.setup_basic_auth(logger, params)
       user, password = params["user"], params["password"]
       

data/lib/logstash/outputs/elasticsearch/template_manager.rb

@@ -46,15 +46,38 @@ module LogStash; module Outputs; class ElasticSearch
       # definition - remove any existing definition of 'template'
       template.delete('template') if template.include?('template') if plugin.maximum_seen_major_version < 8
       template['index_patterns'] = "#{plugin.ilm_rollover_alias}-*"
-      settings = template_settings(plugin, template)
+      settings = resolve_template_settings(plugin, template)
       if settings && (settings['index.lifecycle.name'] || settings['index.lifecycle.rollover_alias'])
         plugin.logger.info("Overwriting index lifecycle name and rollover alias as ILM is enabled")
       end
       settings.update({ 'index.lifecycle.name' => plugin.ilm_policy, 'index.lifecycle.rollover_alias' => plugin.ilm_rollover_alias})
     end
 
-    def self.template_settings(plugin, template)
-      plugin.maximum_seen_major_version < 8 ? template['settings']: template['template']['settings']
+    def self.resolve_template_settings(plugin, template)
+      if template.key?('template')
+        plugin.logger.trace("Resolving ILM template settings: under 'template' key", :template => template, :template_api => plugin.template_api, :es_version => plugin.maximum_seen_major_version)
+        composable_index_template_settings(template)
+      elsif template.key?('settings')
+        plugin.logger.trace("Resolving ILM template settings: under 'settings' key", :template => template, :template_api => plugin.template_api, :es_version => plugin.maximum_seen_major_version)
+        legacy_index_template_settings(template)
+      else
+        template_endpoint = template_endpoint(plugin)
+        plugin.logger.trace("Resolving ILM template settings: template doesn't have 'settings' or 'template' fields, falling back to auto detection", :template => template, :template_api => plugin.template_api, :es_version => plugin.maximum_seen_major_version, :template_endpoint => template_endpoint)
+        template_endpoint == INDEX_TEMPLATE_ENDPOINT ?
+          composable_index_template_settings(template) :
+          legacy_index_template_settings(template)
+      end
+    end
+
+    # Sets ['settings'] field to be compatible with _template API structure
+    def self.legacy_index_template_settings(template)
+      template['settings'] ||= {}
+    end
+
+    # Sets the ['template']['settings'] fields if not exist to be compatible with _index_template API structure
+    def self.composable_index_template_settings(template)
+      template['template'] ||= {}
+      template['template']['settings'] ||= {}
     end
 
     # Template name - if template_name set, use it