logstash-output-elasticsearch 10.6.2-java → 10.8.2-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c2b97b57f05641257d4d82abf2ffc924252385a2376e80ed760bef849d5d2699
-  data.tar.gz: e4fd909cb0fbe096ff54abc0f77c013e601c7a4c01e3ba3bc72a7e121739a80e
+  metadata.gz: d6c80e3faba2acade7f891f8e55f4371859337b80457051529264bed69ef0151
+  data.tar.gz: 00d8cfa6da32835ba9151f5f379c416bd0d4b5ea4d26d05e179da922ede8d8ed
 SHA512:
-  metadata.gz: a17060e61a967293beda8046c0bc03d8583049509f0434c07c6024873b166a6cc2a011a6fa6293daee18f7fef6b6ca5f562e37c61f3150684ac88e58a397384b
-  data.tar.gz: f9f660a1b5d5eecb4a80dcafbd52872b34b4491a3d3b62e57d3710961c771488368960ec1662c5c2503de1ce2da45fc2b39f0900f407ed8f304b28539c291c26
+  metadata.gz: 311b6c538024338a609f5d101e91157487fe5ddb1c876f1cfef7426cd1f721dd46ade457b47052d7387238c76c11f88288f334caf0549506564f6d11bfadc3e3
+  data.tar.gz: f6af92b3f7438531b77f81dce0da90e0a0d495bf1a3bcb49929112d267388a68f6897596c3fca580d81360279b41fa0ae378d822022c146890491eb2c92e5810

data/CHANGELOG.md

@@ -1,3 +1,25 @@
+## 10.8.2
+ - [DOC] Update links to use shared attributes [#985](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/985)
+
+## 10.8.1
+ - Fixed an issue when assigning the no-op license checker [#984](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/984)
+
+## 10.8.0
+ - Refactored configuration options into specific and shared in PluginMixins namespace [#973](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/973)
+ - Refactored common methods into specific and shared in PluginMixins namespace [#976](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/976)
+
+## 10.7.3
+ - Added composable index template support for elasticsearch version 8 [#980](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/980)
+
+## 10.7.2
+ - [DOC] Fixed links to restructured Logstash-to-cloud docs [#975](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/975)
+
+## 10.7.1
+ - [DOC] Document the permissions required in secured clusters [#969](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/969)
+  
+## 10.7.0
+ - Changed: don't set the pipeline parameter if the value resolves to an empty string [#962](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/962)
+
 ## 10.6.2
  - [DOC] Added clarifying info on http compression settings and behaviors [#943](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/943)
  - [DOC] Fixed entry for ilm_policy default value[#956](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/956)

data/CONTRIBUTORS

@@ -26,6 +26,7 @@ Contributors:
 * Tom Hodder (tolland)
 * jimmyjones2
 * Gabriel Moskovicz (gmoskovicz)
+* Luca Belluccini (lucabelluccini)
 
 Note: If you've sent us patches, bug reports, or otherwise contributed to
 Logstash, and you aren't on the list above and want to be, please let us know

data/README.md

@@ -1,6 +1,6 @@
 # Logstash Plugin
 
-[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-output-elasticsearch.svg)](https://travis-ci.org/logstash-plugins/logstash-output-elasticsearch)
+[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-output-elasticsearch.svg)](https://travis-ci.com/logstash-plugins/logstash-output-elasticsearch)
 
 This is a plugin for [Logstash](https://github.com/elastic/logstash).
 

data/docs/index.asciidoc

@@ -23,18 +23,18 @@ include::{include_path}/plugin_header.asciidoc[]
 
 If you plan to use the Kibana web interface to analyze data transformed by
 Logstash, use the Elasticsearch output plugin to get your data into
-Elasticsearch. 
+Elasticsearch.
 
 This output only speaks the HTTP protocol as it is the preferred protocol for
 interacting with Elasticsearch. In previous versions it was possible to
 communicate with Elasticsearch through the transport protocol, which is now
 reserved for internal cluster communication between nodes
-https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html[communication between nodes].
-Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol] 
-to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and
-will be removed in 8.0.0
+{ref}/modules-transport.html[communication between nodes].
+Using the transport protocol to communicate with the cluster has been deprecated
+in Elasticsearch 7.0.0 and will be removed in 8.0.0
 
-You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
+You can https://www.elastic.co/elasticsearch/[learn more about Elasticsearch] on
+the website landing page or in the {ref}[Elasticsearch documentation].
 
 .Compatibility Note
 [NOTE]
@@ -53,10 +53,7 @@ connecting to Elasticsearch 7.x.
 
 ===== Hosted {es} Service on Elastic Cloud
 
-You can run Elasticsearch on your own hardware, or use our
-https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on
-Elastic Cloud. The Elasticsearch Service is available on AWS, Google Cloud
-Platform, and Microsoft Azure. {ess-trial}[Try the {es} Service for free].
+{ess-leadin}
 
 ==== Compatibility with the Elastic Common Schema (ECS)
 
@@ -76,7 +73,7 @@ and will correctly reject events with fields that conflict and cannot be coerced
 
 [NOTE]
 ================================================================================
-You cannot use dynamic variable substitution when `ilm_enabled` is `true` and 
+You cannot use dynamic variable substitution when `ilm_enabled` is `true` and
 when using `ilm_rollover_alias`.
 
 ================================================================================
@@ -91,7 +88,10 @@ Each Elasticsearch output is a new client connected to the cluster:
  * it has to initialize the client and connect to Elasticsearch (restart time is longer if you have more clients)
  * it has an associated connection pool
 
-In order to minimize the number of open connections to Elasticsearch, maximize the bulk size and reduce the number of "small" bulk requests (which could easily fill up the queue), it is usually more efficient to have a single Elasticsearch output.
+In order to minimize the number of open connections to Elasticsearch, maximize
+the bulk size and reduce the number of "small" bulk requests (which could easily
+fill up the queue), it is usually more efficient to have a single Elasticsearch
+output.
 
 Example:
 [source,ruby]
@@ -100,11 +100,13 @@ Example:
         index => "%{[some_field][sub_field]}-%{+YYYY.MM.dd}"
       }
     }
- 
+
 **What to do in case there is no field in the event containing the destination index prefix?**
 
-You can use the `mutate` filter and conditionals to add a `[@metadata]` field (see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata) to set
-the destination index for each event. The `[@metadata]` fields will not be sent to Elasticsearch.
+You can use the `mutate` filter and conditionals to add a
+{logstash-ref}/event-dependent-configuration.html#metadata[`[@metadata]` field]
+to set the destination index for each event. The `[@metadata]` fields will not
+be sent to Elasticsearch.
 
 Example:
 [source,ruby]
@@ -136,7 +138,7 @@ HTTP requests to the bulk API are expected to return a 200 response code. All ot
 The following document errors are handled as follows:
 
   * 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See <<plugins-{type}s-{plugin}-dlq-policy>> for more info.
-  * 409 errors (conflict) are logged as a warning and dropped. 
+  * 409 errors (conflict) are logged as a warning and dropped.
 
 Note that 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
 It is more performant for Elasticsearch to retry these exceptions than this plugin.
@@ -156,30 +158,37 @@ happens, the problem is logged as a warning, and the event is dropped. See
 [id="plugins-{type}s-{plugin}-ilm"]
 ==== Index Lifecycle Management
 
-
 [NOTE]
 The Index Lifecycle Management feature requires plugin version `9.3.1` or higher.
 
 [NOTE]
 This feature requires an Elasticsearch instance of 6.6.0 or higher with at least a Basic license
 
-Logstash can use {ref}/index-lifecycle-management.html[Index Lifecycle Management] to automate the management of indices over time.
+Logstash can use {ref}/index-lifecycle-management.html[Index Lifecycle
+Management] to automate the management of indices over time.
 
 The use of Index Lifecycle Management is controlled by the `ilm_enabled`
 setting. By default, this setting detects whether the Elasticsearch instance
 supports ILM, and uses it if it is available. `ilm_enabled` can also be set to
 `true` or `false` to override the automatic detection, or disable ILM.
 
-This will overwrite the index settings and adjust the Logstash template to write the necessary settings for the template
-to support index lifecycle management, including the index policy and rollover alias to be used.
+This will overwrite the index settings and adjust the Logstash template to write
+the necessary settings for the template to support index lifecycle management,
+including the index policy and rollover alias to be used.
 
-Logstash will create a rollover alias for the indices to be written to, including a pattern for how the actual indices will be named, and unless an ILM policy that already exists has been specified,
-a default policy will also be created. The default policy is configured to rollover an index when it reaches either 50 gigabytes in size, or is 30 days old, whichever happens first.
+Logstash will create a rollover alias for the indices to be written to,
+including a pattern for how the actual indices will be named, and unless an ILM
+policy that already exists has been specified, a default policy will also be
+created. The default policy is configured to rollover an index when it reaches
+either 50 gigabytes in size, or is 30 days old, whichever happens first.
 
-The default rollover alias is called `logstash`, with a default pattern for the rollover index of `{now/d}-00001`,
-which will name indices on the date that the index is rolled over, followed by an incrementing number. Note that the pattern must end with a dash and a number that will be incremented.
+The default rollover alias is called `logstash`, with a default pattern for the
+rollover index of `{now/d}-00001`, which will name indices on the date that the
+index is rolled over, followed by an incrementing number. Note that the pattern
+must end with a dash and a number that will be incremented.
 
-See the {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover API documentation] for more details on naming.
+See the {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
+API documentation] for more details on naming.
 
 The rollover alias, ilm pattern and policy can be modified.
 
@@ -195,19 +204,24 @@ See config below for an example:
 
 NOTE: Custom ILM policies must already exist on the Elasticsearch cluster before they can be used.
 
-NOTE: If the rollover alias or pattern is modified, the index template will need to be overwritten as the settings `index.lifecycle.name` and `index.lifecycle.rollover_alias` are automatically written to the template
+NOTE: If the rollover alias or pattern is modified, the index template will need to be
+overwritten as the settings `index.lifecycle.name` and
+`index.lifecycle.rollover_alias` are automatically written to the template
 
 NOTE: If the index property is supplied in the output definition, it will be overwritten by the rollover alias.
 
 
 ==== Batch Sizes
 
-This plugin attempts to send batches of events as a single request. However, if
-a request exceeds 20MB we will break it up into multiple batch requests. If a single document exceeds 20MB it will be sent as a single request.
+This plugin attempts to send batches of events to the {ref}/docs-bulk.html[{es}
+Bulk API] as a single request. However, if a batch exceeds 20MB we break it up
+into multiple bulk requests. If a single document exceeds 20MB it is sent as a
+single request.
 
 ==== DNS Caching
 
-This plugin uses the JVM to lookup DNS entries and is subject to the value of https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl],
+This plugin uses the JVM to lookup DNS entries and is subject to the value of
+https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl],
 a global setting for the JVM.
 
 As an example, to set your DNS TTL to 1 second you would set
@@ -219,12 +233,11 @@ not reevaluate its DNS value while the keepalive is in effect.
 ==== HTTP Compression
 
 This plugin supports request and response compression. Response compression is
-enabled by default for HTTP and for Elasticsearch versions 5.0 and later. 
+enabled by default for HTTP and for Elasticsearch versions 5.0 and later.
 
 You don't have to set any configs in Elasticsearch for it to send back a
 compressed response. For versions before 5.0, or if HTTPS is enabled,
-`http.compression` must be set to `true`
-https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[in
+`http.compression` must be set to `true` {ref}/modules-http.html#modules-http[in
 Elasticsearch] to take advantage of response compression when using this plugin.
 
 For requests compression, regardless of the Elasticsearch version, enable the
@@ -232,12 +245,23 @@ For requests compression, regardless of the Elasticsearch version, enable the
 
 ==== Authentication
 
-Authentication to a secure Elasticsearch cluster is possible using one of the `user`/`password`, `cloud_auth` or `api_key` options.
+Authentication to a secure Elasticsearch cluster is possible using one of the
+`user`/`password`, `cloud_auth` or `api_key` options.
+
+[id="plugins-{type}s-{plugin}-autz"]
+==== Authorization
+
+Authorization to a secure Elasticsearch cluster requires `read` permission at
+index level and `monitoring` permissions at cluster level. The `monitoring`
+permission at cluster level is necessary to perform periodic connectivity
+checks.
+
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Output Configuration Options
 
-This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+This plugin supports the following configuration options plus the
+<<plugins-{type}s-{plugin}-common-options>> described later.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -307,7 +331,7 @@ output plugins.
 &nbsp;
 
 [id="plugins-{type}s-{plugin}-action"]
-===== `action` 
+===== `action`
 
   * Value type is <<string,string>>
   * Default value is `"index"`
@@ -325,7 +349,8 @@ The Elasticsearch action to perform. Valid actions are:
 - A sprintf style string to change the action based on the content of the event. The value `%{[foo]}`
   would use the foo field for the action
 
-For more details on actions, check out the http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
+For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch
+bulk API documentation].
 
 [id="plugins-{type}s-{plugin}-api_key"]
 ===== `api_key`
@@ -333,12 +358,14 @@ For more details on actions, check out the http://www.elastic.co/guide/en/elasti
   * Value type is <<password,password>>
   * There is no default value for this setting.
 
-Authenticate using Elasticsearch API key. Note that this option also requires enabling the `ssl` option.
+Authenticate using Elasticsearch API key. Note that this option also requires
+enabling the `ssl` option.
 
-Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key API].
+Format is `id:api_key` where `id` and `api_key` are as returned by the
+Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
 
 [id="plugins-{type}s-{plugin}-bulk_path"]
-===== `bulk_path` 
+===== `bulk_path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -347,12 +374,12 @@ HTTP Path to perform the _bulk requests to
 this defaults to a concatenation of the path parameter and "_bulk"
 
 [id="plugins-{type}s-{plugin}-cacert"]
-===== `cacert` 
+===== `cacert`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
 
-The .cer or .pem file to validate the server's certificate
+The .cer or .pem file to validate the server's certificate.
 
 [id="plugins-{type}s-{plugin}-cloud_auth"]
 ===== `cloud_auth`
@@ -360,9 +387,11 @@ The .cer or .pem file to validate the server's certificate
   * Value type is <<password,password>>
   * There is no default value for this setting.
 
-Cloud authentication string ("<username>:<password>" format) is an alternative for the `user`/`password` pair.
+Cloud authentication string ("<username>:<password>" format) is an alternative
+for the `user`/`password` pair.
 
-For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_auth[Logstash-to-Cloud documentation]
+For more details, check out the
+{logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].
 
 [id="plugins-{type}s-{plugin}-cloud_id"]
 ===== `cloud_id`
@@ -372,34 +401,38 @@ For more details, check out the https://www.elastic.co/guide/en/logstash/current
 
 Cloud ID, from the Elastic Cloud web console. If set `hosts` should not be used.
 
-For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_id[Logstash-to-Cloud documentation]
+For more details, check out the
+{logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].
 
 [id="plugins-{type}s-{plugin}-doc_as_upsert"]
-===== `doc_as_upsert` 
+===== `doc_as_upsert`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
 
 Enable `doc_as_upsert` for update mode.
-Create a new document with source if `document_id` doesn't exist in Elasticsearch
+Create a new document with source if `document_id` doesn't exist in Elasticsearch.
 
 [id="plugins-{type}s-{plugin}-document_id"]
-===== `document_id` 
+===== `document_id`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.
+The document ID for the index. Useful for overwriting existing entries in
+Elasticsearch with the same ID.
 
 [id="plugins-{type}s-{plugin}-document_type"]
-===== `document_type` 
+===== `document_type`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
   * This option is deprecated
 
-NOTE: This option is deprecated due to the https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal of types in Elasticsearch 6.0].
-It will be removed in the next major version of Logstash.
+NOTE: This option is deprecated due to the
+https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal
+of types in Elasticsearch 6.0]. It will be removed in the next major version of
+Logstash.
 
 NOTE: This value is ignored and has no effect for Elasticsearch clusters `8.x`.
 
@@ -423,16 +456,17 @@ If you don't set a value for this option:
 ** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
 ** Otherwise, the default value is `disabled`.
 
-Controls this plugin's compatibility with the https://www.elastic.co/guide/en/ecs/current/index.html[Elastic Common Schema (ECS)],
-including the installation of ECS-compatible index templates.
-The value of this setting affects the _default_ values of:
+Controls this plugin's compatibility with the
+https://www.elastic.co/guide/en/ecs/current/index.html[Elastic Common Schema
+(ECS)], including the installation of ECS-compatible index templates. The value
+of this setting affects the _default_ values of:
 
 * <<plugins-{type}s-{plugin}-index>>
 * <<plugins-{type}s-{plugin}-template_name>>
 * <<plugins-{type}s-{plugin}-ilm_rollover_alias>>
 
 [id="plugins-{type}s-{plugin}-failure_type_logging_whitelist"]
-===== `failure_type_logging_whitelist` 
+===== `failure_type_logging_whitelist`
 
   * Value type is <<array,array>>
   * Default value is `[]`
@@ -453,7 +487,7 @@ an elasticsearch node. The headers will be used for any kind of request
 These custom headers will be overidden by settings like `http_compression`.
 
 [id="plugins-{type}s-{plugin}-healthcheck_path"]
-===== `healthcheck_path` 
+===== `healthcheck_path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -464,13 +498,15 @@ before it is once again eligible to service requests.
 If you have custom firewall rules you may need to change this
 
 [id="plugins-{type}s-{plugin}-hosts"]
-===== `hosts` 
+===== `hosts`
 
   * Value type is <<uri,uri>>
   * Default value is `[//127.0.0.1]`
 
-Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the `hosts` parameter.
-Remember the `http` protocol uses the http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[http] address (eg. 9200, not 9300).
+Sets the host(s) of the remote instance. If given an array it will load balance
+requests across the hosts specified in the `hosts` parameter. Remember the
+`http` protocol uses the {ref}/modules-http.html#modules-http[http] address (eg.
+9200, not 9300).
 
 Examples:
 
@@ -480,18 +516,21 @@ Examples:
     `["https://127.0.0.1:9200"]`
     `["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)
 
-It is important to exclude http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated master nodes] from the `hosts` list
-to prevent LS from sending bulk requests to the master nodes.  So this parameter should only reference either data or client nodes in Elasticsearch.
+Exclude {ref}/modules-node.html[dedicated master nodes] from the `hosts` list to
+prevent Logstash from sending bulk requests to the master nodes. This parameter
+should reference only data or client nodes in Elasticsearch.
 
-Any special characters present in the URLs here MUST be URL escaped! This means `#` should be put in as `%23` for instance.
+Any special characters present in the URLs here MUST be URL escaped! This means
+`#` should be put in as `%23` for instance.
 
 [id="plugins-{type}s-{plugin}-http_compression"]
-===== `http_compression` 
+===== `http_compression`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
 
-Enable gzip compression on requests. Note that response compression is on by default for Elasticsearch v5.0 and beyond
+Enable gzip compression on requests. Note that response compression is on by
+default for Elasticsearch v5.0 and beyond
 
 [id="plugins-{type}s-{plugin}-ilm_enabled"]
 ===== `ilm_enabled`
@@ -499,14 +538,19 @@ Enable gzip compression on requests. Note that response compression is on by def
   * Value can be any of: `true`, `false`, `auto`
   * Default value is `auto`
 
-The default setting of `auto` will automatically enable the Index Lifecycle Management feature, if the Elasticsearch
- cluster is running Elasticsearch version `7.0.0` or higher with the ILM feature enabled, and disable it otherwise.
+The default setting of `auto` will automatically enable
+{ref}/index-lifecycle-management.html[Index Lifecycle Management], if the
+Elasticsearch cluster is running Elasticsearch version `7.0.0` or higher with
+the ILM feature enabled, and disable it otherwise.
 
-Setting this flag to `false` will disable the Index Lifecycle Management feature, even if the Elasticsearch cluster supports ILM.
-Setting this flag to `true` will enable Index Lifecycle Management feature, if the Elasticsearch cluster supports it. This is required
-to enable Index Lifecycle Management on a version of Elasticsearch earlier than version `7.0.0`.
+Setting this flag to `false` will disable the Index Lifecycle Management
+feature, even if the Elasticsearch cluster supports ILM.
+Setting this flag to `true` will enable Index Lifecycle Management feature, if
+the Elasticsearch cluster supports it. This is required to enable Index
+Lifecycle Management on a version of Elasticsearch earlier than version `7.0.0`.
 
-NOTE: This feature requires a Basic License or above to be installed on an Elasticsearch cluster version 6.6.0 or later
+NOTE: This feature requires a Basic License or above to be installed on an
+Elasticsearch cluster version 6.6.0 or later.
 
 [id="plugins-{type}s-{plugin}-ilm_pattern"]
 ===== `ilm_pattern`
@@ -514,14 +558,24 @@ NOTE: This feature requires a Basic License or above to be installed on an Elast
   * Value type is <<string,string>>
   * Default value is `{now/d}-000001`
 
-Pattern used for generating indices managed by Index Lifecycle Management. The value specified in the pattern will be appended to
-the write alias, and incremented automatically when a new index is created by ILM.
+Pattern used for generating indices managed by
+{ref}/index-lifecycle-management.html[Index Lifecycle Management]. The value
+specified in the pattern will be appended to the write alias, and incremented
+automatically when a new index is created by ILM.
 
-Date Math can be used when specifying an ilm pattern, see {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover API docs] for details
+Date Math can be used when specifying an ilm pattern, see
+{ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
+API docs] for details.
 
-NOTE: Updating the pattern will require the index template to be rewritten
+NOTE: Updating the pattern will require the index template to be rewritten.
 
-NOTE: The pattern must finish with a dash and a number that will be automatically incremented when indices rollover.
+NOTE: The pattern must finish with a dash and a number that will be automatically
+incremented when indices rollover.
+
+NOTE: The pattern is a 6-digit string padded by zeros, regardless of prior index name.
+Example: 000001. See
+{ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path
+parameters API docs] for details.
 
 [id="plugins-{type}s-{plugin}-ilm_policy"]
 ===== `ilm_policy`
@@ -529,10 +583,12 @@ NOTE: The pattern must finish with a dash and a number that will be automaticall
   * Value type is <<string,string>>
   * Default value is `logstash-policy`
 
-Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy will
-be automatically installed into Elasticsearch
+Modify this setting to use a custom Index Lifecycle Management policy, rather
+than the default. If this value is not set, the default policy will be
+automatically installed into Elasticsearch
 
-NOTE: If this setting is specified, the policy must already exist in Elasticsearch cluster.
+NOTE: If this setting is specified, the policy must already exist in Elasticsearch
+cluster.
 
 [id="plugins-{type}s-{plugin}-ilm_rollover_alias"]
 ===== `ilm_rollover_alias`
@@ -542,16 +598,20 @@ NOTE: If this setting is specified, the policy must already exist in Elasticsear
   ** ECS Compatibility disabled: `logstash`
   ** ECS Compatibility enabled: `ecs-logstash`
 
-The rollover alias is the alias where indices managed using Index Lifecycle Management will be written to.
+The rollover alias is the alias where indices managed using Index Lifecycle
+Management will be written to.
 
-NOTE: If both `index` and `ilm_rollover_alias` are specified, `ilm_rollover_alias` takes precedence.
+NOTE: If both `index` and `ilm_rollover_alias` are specified,
+`ilm_rollover_alias` takes precedence.
 
-NOTE: Updating the rollover alias will require the index template to be rewritten
+NOTE: Updating the rollover alias will require the index template to be
+rewritten.
 
-NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as `index` does.
+NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as
+`index` does.
 
 [id="plugins-{type}s-{plugin}-index"]
-===== `index` 
+===== `index`
 
   * Value type is <<string,string>>
   * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
@@ -563,11 +623,13 @@ The default value will partition your indices by day so you can more easily
 delete old data or only search specific date ranges.
 Indexes may not contain uppercase characters.
 For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}.
-LS uses Joda to format the index pattern from event timestamp.
-Joda formats are defined http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[here].
+Logstash uses
+http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[Joda
+formats] for the index pattern from event timestamp.
+
 
 [id="plugins-{type}s-{plugin}-keystore"]
-===== `keystore` 
+===== `keystore`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -576,7 +638,7 @@ The keystore used to present a certificate to the server.
 It can be either .jks or .p12
 
 [id="plugins-{type}s-{plugin}-keystore_password"]
-===== `keystore_password` 
+===== `keystore_password`
 
   * Value type is <<password,password>>
   * There is no default value for this setting.
@@ -584,7 +646,7 @@ It can be either .jks or .p12
 Set the keystore password
 
 [id="plugins-{type}s-{plugin}-manage_template"]
-===== `manage_template` 
+===== `manage_template`
 
   * Value type is <<boolean,boolean>>
   * Default value is `true`
@@ -604,7 +666,7 @@ field names) you should set `manage_template` to false and use the REST
 API to apply your templates manually.
 
 [id="plugins-{type}s-{plugin}-parameters"]
-===== `parameters` 
+===== `parameters`
 
   * Value type is <<hash,hash>>
   * There is no default value for this setting.
@@ -614,7 +676,7 @@ to every host listed in the 'hosts' configuration. If the 'hosts' list contains
 urls that already have query strings, the one specified here will be appended.
 
 [id="plugins-{type}s-{plugin}-parent"]
-===== `parent` 
+===== `parent`
 
   * Value type is <<string,string>>
   * Default value is `nil`
@@ -623,7 +685,7 @@ For child documents, ID of the associated parent.
 This can be dynamic using the `%{foo}` syntax.
 
 [id="plugins-{type}s-{plugin}-password"]
-===== `password` 
+===== `password`
 
   * Value type is <<password,password>>
   * There is no default value for this setting.
@@ -631,27 +693,30 @@ This can be dynamic using the `%{foo}` syntax.
 Password to authenticate to a secure Elasticsearch cluster
 
 [id="plugins-{type}s-{plugin}-path"]
-===== `path` 
+===== `path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps
-the root path for the Elasticsearch HTTP API lives.
+HTTP Path at which the Elasticsearch server lives. Use this if you must run
+Elasticsearch behind a proxy that remaps the root path for the Elasticsearch
+HTTP API lives.
 Note that if you use paths as components of URLs in the 'hosts' field you may
 not also set this field. That will raise an error at startup
 
 [id="plugins-{type}s-{plugin}-pipeline"]
-===== `pipeline` 
+===== `pipeline`
 
   * Value type is <<string,string>>
   * Default value is `nil`
 
-Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration
-here like `pipeline => "%{INGEST_PIPELINE}"`
+Set which ingest pipeline you wish to execute for an event. You can also use
+event dependent configuration here like `pipeline =>
+"%{[@metadata][pipeline]}"`. The pipeline parameter won't be set if the value
+resolves to empty string ("").
 
 [id="plugins-{type}s-{plugin}-pool_max"]
-===== `pool_max` 
+===== `pool_max`
 
   * Value type is <<number,number>>
   * Default value is `1000`
@@ -662,7 +727,7 @@ Setting this too low may mean frequently closing / opening connections
 which is bad.
 
 [id="plugins-{type}s-{plugin}-pool_max_per_route"]
-===== `pool_max_per_route` 
+===== `pool_max_per_route`
 
   * Value type is <<number,number>>
   * Default value is `100`
@@ -673,7 +738,7 @@ Setting this too low may mean frequently closing / opening connections
 which is bad.
 
 [id="plugins-{type}s-{plugin}-proxy"]
-===== `proxy` 
+===== `proxy`
 
   * Value type is <<uri,uri>>
   * There is no default value for this setting.
@@ -684,7 +749,7 @@ An empty string is treated as if proxy was not set. This is useful when using
 environment variables e.g. `proxy => '${LS_PROXY:}'`.
 
 [id="plugins-{type}s-{plugin}-resurrect_delay"]
-===== `resurrect_delay` 
+===== `resurrect_delay`
 
   * Value type is <<number,number>>
   * Default value is `5`
@@ -694,15 +759,16 @@ Resurrection is the process by which backend endpoints marked 'down' are checked
 to see if they have come back to life
 
 [id="plugins-{type}s-{plugin}-retry_initial_interval"]
-===== `retry_initial_interval` 
+===== `retry_initial_interval`
 
   * Value type is <<number,number>>
   * Default value is `2`
 
-Set initial interval in seconds between bulk retries. Doubled on each retry up to `retry_max_interval`
+Set initial interval in seconds between bulk retries. Doubled on each retry up
+to `retry_max_interval`
 
 [id="plugins-{type}s-{plugin}-retry_max_interval"]
-===== `retry_max_interval` 
+===== `retry_max_interval`
 
   * Value type is <<number,number>>
   * Default value is `64`
@@ -710,17 +776,15 @@ Set initial interval in seconds between bulk retries. Doubled on each retry up t
 Set max interval in seconds between bulk retries.
 
 [id="plugins-{type}s-{plugin}-retry_on_conflict"]
-===== `retry_on_conflict` 
+===== `retry_on_conflict`
 
   * Value type is <<number,number>>
   * Default value is `1`
 
-The number of times Elasticsearch should internally retry an update/upserted document
-See the https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html[partial updates]
-for more info
+The number of times Elasticsearch should internally retry an update/upserted document.
 
 [id="plugins-{type}s-{plugin}-routing"]
-===== `routing` 
+===== `routing`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -729,7 +793,7 @@ A routing override to be applied to all processed events.
 This can be dynamic using the `%{foo}` syntax.
 
 [id="plugins-{type}s-{plugin}-script"]
-===== `script` 
+===== `script`
 
   * Value type is <<string,string>>
   * Default value is `""`
@@ -745,16 +809,17 @@ Example:
     }
 
 [id="plugins-{type}s-{plugin}-script_lang"]
-===== `script_lang` 
+===== `script_lang`
 
   * Value type is <<string,string>>
   * Default value is `"painless"`
 
-Set the language of the used script. If not set, this defaults to painless in ES 5.0.
-When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to `""` (empty string).
+Set the language of the used script.
+When using indexed (stored) scripts on Elasticsearch 6.0 and higher, you must set
+this parameter to `""` (empty string).
 
 [id="plugins-{type}s-{plugin}-script_type"]
-===== `script_type` 
+===== `script_type`
 
   * Value can be any of: `inline`, `indexed`, `file`
   * Default value is `["inline"]`
@@ -765,7 +830,7 @@ Define the type of script referenced by "script" variable
  file    : "script" contains the name of script stored in elasticsearch's config directory
 
 [id="plugins-{type}s-{plugin}-script_var_name"]
-===== `script_var_name` 
+===== `script_var_name`
 
   * Value type is <<string,string>>
   * Default value is `"event"`
@@ -773,7 +838,7 @@ Define the type of script referenced by "script" variable
 Set variable name passed to script (scripted update)
 
 [id="plugins-{type}s-{plugin}-scripted_upsert"]
-===== `scripted_upsert` 
+===== `scripted_upsert`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -781,17 +846,18 @@ Set variable name passed to script (scripted update)
 if enabled, script is in charge of creating non-existent document (scripted update)
 
 [id="plugins-{type}s-{plugin}-sniffing"]
-===== `sniffing` 
+===== `sniffing`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
 
-This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list.
-For Elasticsearch 1.x and 2.x any nodes with `http.enabled` (on by default) will be added to the hosts list, including master-only nodes!
-For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will be added to the hosts list, excluding master-only nodes.
+This setting asks Elasticsearch for the list of all cluster nodes and adds them
+to the hosts list.
+For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
+be added to the hosts list, excluding master-only nodes.
 
 [id="plugins-{type}s-{plugin}-sniffing_delay"]
-===== `sniffing_delay` 
+===== `sniffing_delay`
 
   * Value type is <<number,number>>
   * Default value is `5`
@@ -799,7 +865,7 @@ For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
 How long to wait, in seconds, between sniffing attempts
 
 [id="plugins-{type}s-{plugin}-sniffing_path"]
-===== `sniffing_path` 
+===== `sniffing_path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -810,17 +876,19 @@ if sniffing_path is set it will be used as an absolute path
 do not use full URL here, only paths, e.g. "/sniff/_nodes/http"
 
 [id="plugins-{type}s-{plugin}-ssl"]
-===== `ssl` 
+===== `ssl`
 
   * Value type is <<boolean,boolean>>
   * There is no default value for this setting.
 
-Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme
-is specified in the URLs listed in 'hosts'. If no explicit protocol is specified plain HTTP will be used.
-If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts'
+Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this
+unspecified will use whatever scheme is specified in the URLs listed in 'hosts'.
+If no explicit protocol is specified plain HTTP will be used. If SSL is
+explicitly disabled here the plugin will refuse to start if an HTTPS URL is
+given in 'hosts'
 
 [id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
-===== `ssl_certificate_verification` 
+===== `ssl_certificate_verification`
 
   * Value type is <<boolean,boolean>>
   * Default value is `true`
@@ -830,7 +898,7 @@ For more information on disabling certificate verification please read
 https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
 
 [id="plugins-{type}s-{plugin}-template"]
-===== `template` 
+===== `template`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -839,7 +907,7 @@ You can set the path to your own template here, if you so desire.
 If not set, the included template will be used.
 
 [id="plugins-{type}s-{plugin}-template_name"]
-===== `template_name` 
+===== `template_name`
 
   * Value type is <<string,string>>
   * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
@@ -856,7 +924,7 @@ change this, you will need to prune the old template manually, e.g.
 where `OldTemplateName` is whatever the former setting was.
 
 [id="plugins-{type}s-{plugin}-template_overwrite"]
-===== `template_overwrite` 
+===== `template_overwrite`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -873,7 +941,7 @@ template (logstash), setting this to true will make Logstash to overwrite
 the "logstash" template (i.e. removing all customized settings)
 
 [id="plugins-{type}s-{plugin}-timeout"]
-===== `timeout` 
+===== `timeout`
 
   * Value type is <<number,number>>
   * Default value is `60`
@@ -882,7 +950,7 @@ Set the timeout, in seconds, for network operations and requests sent Elasticsea
 a timeout occurs, the request will be retried.
 
 [id="plugins-{type}s-{plugin}-truststore"]
-===== `truststore` 
+===== `truststore`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -892,7 +960,7 @@ It can be either .jks or .p12.
 Use either `:truststore` or `:cacert`.
 
 [id="plugins-{type}s-{plugin}-truststore_password"]
-===== `truststore_password` 
+===== `truststore_password`
 
   * Value type is <<password,password>>
   * There is no default value for this setting.
@@ -900,7 +968,7 @@ Use either `:truststore` or `:cacert`.
 Set the truststore password
 
 [id="plugins-{type}s-{plugin}-upsert"]
-===== `upsert` 
+===== `upsert`
 
   * Value type is <<string,string>>
   * Default value is `""`
@@ -909,7 +977,7 @@ Set upsert content for update mode.
 Create a new document with this parameter as json string if `document_id` doesn't exists
 
 [id="plugins-{type}s-{plugin}-user"]
-===== `user` 
+===== `user`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -917,39 +985,43 @@ Create a new document with this parameter as json string if `document_id` doesn'
 Username to authenticate to a secure Elasticsearch cluster
 
 [id="plugins-{type}s-{plugin}-validate_after_inactivity"]
-===== `validate_after_inactivity` 
+===== `validate_after_inactivity`
 
   * Value type is <<number,number>>
   * Default value is `10000`
 
-How long to wait before checking if the connection is stale before executing a request on a connection using keepalive.
-You may want to set this lower, if you get connection errors regularly
-Quoting the Apache commons docs (this client is based Apache Commmons):
-'Defines period of inactivity in milliseconds after which persistent connections must
-be re-validated prior to being leased to the consumer. Non-positive value passed to
-this method disables connection validation. This check helps detect connections that
-have become stale (half-closed) while kept inactive in the pool.'
-See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[these docs for more info]
+How long to wait before checking for a stale connection to determine if a keepalive request is needed.
+Consider setting this value lower than the default, possibly to 0, if you get connection errors regularly.
+
+This client is based on Apache Commons. Here's how the
+https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[Apache
+Commons documentation] describes this option: "Defines period of inactivity in
+milliseconds after which persistent connections must be re-validated prior to
+being leased to the consumer. Non-positive value passed to this method disables
+connection validation. This check helps detect connections that have become
+stale (half-closed) while kept inactive in the pool."
 
 [id="plugins-{type}s-{plugin}-version"]
-===== `version` 
+===== `version`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-The version to use for indexing. Use sprintf syntax like `%{my_version}` to use a field value here.
-See https://www.elastic.co/blog/elasticsearch-versioning-support.
+The version to use for indexing. Use sprintf syntax like `%{my_version}` to use
+a field value here. See the
+https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
+blog] for more information.
 
 [id="plugins-{type}s-{plugin}-version_type"]
-===== `version_type` 
+===== `version_type`
 
   * Value can be any of: `internal`, `external`, `external_gt`, `external_gte`, `force`
   * There is no default value for this setting.
 
-The version_type to use for indexing.
-See https://www.elastic.co/blog/elasticsearch-versioning-support.
-See also https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#_version_types
-
+The version_type to use for indexing. See the
+https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
+blog] and {ref}/docs-index_.html#_version_types[Version types] in the
+Elasticsearch documentation.
 
 
 [id="plugins-{type}s-{plugin}-common-options"]