logstash-output-elasticsearch 10.7.0-java → 10.7.3-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 311303e9bfc0369329af5ec4da290d92e428bddee2e022fb1f22967fb1f1d6d9
-  data.tar.gz: c88b0816b49bd5f5ab232156340f946b674e70a62b86f8a1c4c25ed985840c6e
+  metadata.gz: 800beb4697a44f1c41a41490b627b5b6dd192c695a818b33118e778a0fde7902
+  data.tar.gz: 380591f1e759722fac05fbc8893113ad0821cb9878e5257a9da8dc90f9a4fd09
 SHA512:
-  metadata.gz: 7d9910d4f28a864ac45450cd2be179268d8b83c91f8dab57328e6c772b06a39d7d8aa3fa7c34cddcd45a185bc994da2f2b46ee3b29bf2dc219a3189b6e995b30
-  data.tar.gz: de41059508b3f06466446d3a183d4959cdaaacffb00492d917acc57824b4864a694c5834c8e210d8d277ae186c146d6bf3c6b9ab0c0cd1ab73bc7f95a6f4e871
+  metadata.gz: 895bdaa0f45d48396b594df34545174a7788eb7647df0fc6cad58ab162950c399319d0c462d97a49133368a88731e3548e4ef508dc7115fd448ebd75b7da6d47
+  data.tar.gz: 96feefdd53810337a249cb4412296515ff049a6a07c0a81c81fc23b4152bd0061f3d7fd3cec7956eb0abb07636c3b78c1dbb16daf1040a77e359dc771f79d67d

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 10.7.3
+ - Added composable index template support for elasticsearch version 8 [#980](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/980)
+
+## 10.7.2
+ - [DOC] Fixed links to restructured Logstash-to-cloud docs [#975](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/975)
+
+## 10.7.1
+ - [DOC] Document the permissions required in secured clusters [#969](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/969)
+  
 ## 10.7.0
  - Changed: don't set the pipeline parameter if the value resolves to an empty string [#962](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/962)
 

data/CONTRIBUTORS

@@ -26,6 +26,7 @@ Contributors:
 * Tom Hodder (tolland)
 * jimmyjones2
 * Gabriel Moskovicz (gmoskovicz)
+* Luca Belluccini (lucabelluccini)
 
 Note: If you've sent us patches, bug reports, or otherwise contributed to
 Logstash, and you aren't on the list above and want to be, please let us know

data/docs/index.asciidoc

@@ -23,14 +23,14 @@ include::{include_path}/plugin_header.asciidoc[]
 
 If you plan to use the Kibana web interface to analyze data transformed by
 Logstash, use the Elasticsearch output plugin to get your data into
-Elasticsearch. 
+Elasticsearch.
 
 This output only speaks the HTTP protocol as it is the preferred protocol for
 interacting with Elasticsearch. In previous versions it was possible to
 communicate with Elasticsearch through the transport protocol, which is now
 reserved for internal cluster communication between nodes
 https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html[communication between nodes].
-Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol] 
+Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol]
 to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and
 will be removed in 8.0.0
 
@@ -53,10 +53,7 @@ connecting to Elasticsearch 7.x.
 
 ===== Hosted {es} Service on Elastic Cloud
 
-You can run Elasticsearch on your own hardware, or use our
-https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on
-Elastic Cloud. The Elasticsearch Service is available on AWS, Google Cloud
-Platform, and Microsoft Azure. {ess-trial}[Try the {es} Service for free].
+{ess-leadin}
 
 ==== Compatibility with the Elastic Common Schema (ECS)
 
@@ -76,7 +73,7 @@ and will correctly reject events with fields that conflict and cannot be coerced
 
 [NOTE]
 ================================================================================
-You cannot use dynamic variable substitution when `ilm_enabled` is `true` and 
+You cannot use dynamic variable substitution when `ilm_enabled` is `true` and
 when using `ilm_rollover_alias`.
 
 ================================================================================
@@ -91,7 +88,10 @@ Each Elasticsearch output is a new client connected to the cluster:
  * it has to initialize the client and connect to Elasticsearch (restart time is longer if you have more clients)
  * it has an associated connection pool
 
-In order to minimize the number of open connections to Elasticsearch, maximize the bulk size and reduce the number of "small" bulk requests (which could easily fill up the queue), it is usually more efficient to have a single Elasticsearch output.
+In order to minimize the number of open connections to Elasticsearch, maximize
+the bulk size and reduce the number of "small" bulk requests (which could easily
+fill up the queue), it is usually more efficient to have a single Elasticsearch
+output.
 
 Example:
 [source,ruby]
@@ -100,11 +100,14 @@ Example:
         index => "%{[some_field][sub_field]}-%{+YYYY.MM.dd}"
       }
     }
- 
+
 **What to do in case there is no field in the event containing the destination index prefix?**
 
-You can use the `mutate` filter and conditionals to add a `[@metadata]` field (see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata) to set
-the destination index for each event. The `[@metadata]` fields will not be sent to Elasticsearch.
+You can use the `mutate` filter and conditionals to add a `[@metadata]` field
+(see
+https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata)
+to set the destination index for each event. The `[@metadata]` fields will not
+be sent to Elasticsearch.
 
 Example:
 [source,ruby]
@@ -136,7 +139,7 @@ HTTP requests to the bulk API are expected to return a 200 response code. All ot
 The following document errors are handled as follows:
 
   * 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See <<plugins-{type}s-{plugin}-dlq-policy>> for more info.
-  * 409 errors (conflict) are logged as a warning and dropped. 
+  * 409 errors (conflict) are logged as a warning and dropped.
 
 Note that 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
 It is more performant for Elasticsearch to retry these exceptions than this plugin.
@@ -219,7 +222,7 @@ not reevaluate its DNS value while the keepalive is in effect.
 ==== HTTP Compression
 
 This plugin supports request and response compression. Response compression is
-enabled by default for HTTP and for Elasticsearch versions 5.0 and later. 
+enabled by default for HTTP and for Elasticsearch versions 5.0 and later.
 
 You don't have to set any configs in Elasticsearch for it to send back a
 compressed response. For versions before 5.0, or if HTTPS is enabled,
@@ -234,6 +237,13 @@ For requests compression, regardless of the Elasticsearch version, enable the
 
 Authentication to a secure Elasticsearch cluster is possible using one of the `user`/`password`, `cloud_auth` or `api_key` options.
 
+[id="plugins-{type}s-{plugin}-autz"]
+==== Authorization
+
+Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
+The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.
+
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Output Configuration Options
 
@@ -307,7 +317,7 @@ output plugins.
 &nbsp;
 
 [id="plugins-{type}s-{plugin}-action"]
-===== `action` 
+===== `action`
 
   * Value type is <<string,string>>
   * Default value is `"index"`
@@ -338,7 +348,7 @@ Authenticate using Elasticsearch API key. Note that this option also requires en
 Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key API].
 
 [id="plugins-{type}s-{plugin}-bulk_path"]
-===== `bulk_path` 
+===== `bulk_path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -347,7 +357,7 @@ HTTP Path to perform the _bulk requests to
 this defaults to a concatenation of the path parameter and "_bulk"
 
 [id="plugins-{type}s-{plugin}-cacert"]
-===== `cacert` 
+===== `cacert`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -362,7 +372,7 @@ The .cer or .pem file to validate the server's certificate
 
 Cloud authentication string ("<username>:<password>" format) is an alternative for the `user`/`password` pair.
 
-For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_auth[Logstash-to-Cloud documentation]
+For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html[Logstash-to-Cloud documentation]
 
 [id="plugins-{type}s-{plugin}-cloud_id"]
 ===== `cloud_id`
@@ -372,10 +382,10 @@ For more details, check out the https://www.elastic.co/guide/en/logstash/current
 
 Cloud ID, from the Elastic Cloud web console. If set `hosts` should not be used.
 
-For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_id[Logstash-to-Cloud documentation]
+For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html[Logstash-to-Cloud documentation]
 
 [id="plugins-{type}s-{plugin}-doc_as_upsert"]
-===== `doc_as_upsert` 
+===== `doc_as_upsert`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -384,7 +394,7 @@ Enable `doc_as_upsert` for update mode.
 Create a new document with source if `document_id` doesn't exist in Elasticsearch
 
 [id="plugins-{type}s-{plugin}-document_id"]
-===== `document_id` 
+===== `document_id`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -392,7 +402,7 @@ Create a new document with source if `document_id` doesn't exist in Elasticsearc
 The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.
 
 [id="plugins-{type}s-{plugin}-document_type"]
-===== `document_type` 
+===== `document_type`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -432,7 +442,7 @@ The value of this setting affects the _default_ values of:
 * <<plugins-{type}s-{plugin}-ilm_rollover_alias>>
 
 [id="plugins-{type}s-{plugin}-failure_type_logging_whitelist"]
-===== `failure_type_logging_whitelist` 
+===== `failure_type_logging_whitelist`
 
   * Value type is <<array,array>>
   * Default value is `[]`
@@ -453,7 +463,7 @@ an elasticsearch node. The headers will be used for any kind of request
 These custom headers will be overidden by settings like `http_compression`.
 
 [id="plugins-{type}s-{plugin}-healthcheck_path"]
-===== `healthcheck_path` 
+===== `healthcheck_path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -464,7 +474,7 @@ before it is once again eligible to service requests.
 If you have custom firewall rules you may need to change this
 
 [id="plugins-{type}s-{plugin}-hosts"]
-===== `hosts` 
+===== `hosts`
 
   * Value type is <<uri,uri>>
   * Default value is `[//127.0.0.1]`
@@ -480,13 +490,17 @@ Examples:
     `["https://127.0.0.1:9200"]`
     `["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)
 
-It is important to exclude http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated master nodes] from the `hosts` list
-to prevent LS from sending bulk requests to the master nodes.  So this parameter should only reference either data or client nodes in Elasticsearch.
+Exclude
+http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated
+master nodes] from the `hosts` list to prevent Logstash from sending bulk
+requests to the master nodes. This parameter should reference only data or
+client nodes in Elasticsearch.
 
-Any special characters present in the URLs here MUST be URL escaped! This means `#` should be put in as `%23` for instance.
+Any special characters present in the URLs here MUST be URL escaped! This means
+`#` should be put in as `%23` for instance.
 
 [id="plugins-{type}s-{plugin}-http_compression"]
-===== `http_compression` 
+===== `http_compression`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -523,6 +537,9 @@ NOTE: Updating the pattern will require the index template to be rewritten
 
 NOTE: The pattern must finish with a dash and a number that will be automatically incremented when indices rollover.
 
+NOTE: The pattern is a 6-digit string padded by zeros, regardless of prior index name. Example: 000001.
+See {ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path parameters API docs] for details.
+
 [id="plugins-{type}s-{plugin}-ilm_policy"]
 ===== `ilm_policy`
 
@@ -551,7 +568,7 @@ NOTE: Updating the rollover alias will require the index template to be rewritte
 NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as `index` does.
 
 [id="plugins-{type}s-{plugin}-index"]
-===== `index` 
+===== `index`
 
   * Value type is <<string,string>>
   * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
@@ -567,7 +584,7 @@ LS uses Joda to format the index pattern from event timestamp.
 Joda formats are defined http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[here].
 
 [id="plugins-{type}s-{plugin}-keystore"]
-===== `keystore` 
+===== `keystore`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -576,7 +593,7 @@ The keystore used to present a certificate to the server.
 It can be either .jks or .p12
 
 [id="plugins-{type}s-{plugin}-keystore_password"]
-===== `keystore_password` 
+===== `keystore_password`
 
   * Value type is <<password,password>>
   * There is no default value for this setting.
@@ -584,7 +601,7 @@ It can be either .jks or .p12
 Set the keystore password
 
 [id="plugins-{type}s-{plugin}-manage_template"]
-===== `manage_template` 
+===== `manage_template`
 
   * Value type is <<boolean,boolean>>
   * Default value is `true`
@@ -604,7 +621,7 @@ field names) you should set `manage_template` to false and use the REST
 API to apply your templates manually.
 
 [id="plugins-{type}s-{plugin}-parameters"]
-===== `parameters` 
+===== `parameters`
 
   * Value type is <<hash,hash>>
   * There is no default value for this setting.
@@ -614,7 +631,7 @@ to every host listed in the 'hosts' configuration. If the 'hosts' list contains
 urls that already have query strings, the one specified here will be appended.
 
 [id="plugins-{type}s-{plugin}-parent"]
-===== `parent` 
+===== `parent`
 
   * Value type is <<string,string>>
   * Default value is `nil`
@@ -623,7 +640,7 @@ For child documents, ID of the associated parent.
 This can be dynamic using the `%{foo}` syntax.
 
 [id="plugins-{type}s-{plugin}-password"]
-===== `password` 
+===== `password`
 
   * Value type is <<password,password>>
   * There is no default value for this setting.
@@ -631,7 +648,7 @@ This can be dynamic using the `%{foo}` syntax.
 Password to authenticate to a secure Elasticsearch cluster
 
 [id="plugins-{type}s-{plugin}-path"]
-===== `path` 
+===== `path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -642,7 +659,7 @@ Note that if you use paths as components of URLs in the 'hosts' field you may
 not also set this field. That will raise an error at startup
 
 [id="plugins-{type}s-{plugin}-pipeline"]
-===== `pipeline` 
+===== `pipeline`
 
   * Value type is <<string,string>>
   * Default value is `nil`
@@ -651,7 +668,7 @@ Set which ingest pipeline you wish to execute for an event. You can also use eve
 like `pipeline => "%{[@metadata][pipeline]}"`. The pipeline parameter won't be set if the value resolves to empty string ("").
 
 [id="plugins-{type}s-{plugin}-pool_max"]
-===== `pool_max` 
+===== `pool_max`
 
   * Value type is <<number,number>>
   * Default value is `1000`
@@ -662,7 +679,7 @@ Setting this too low may mean frequently closing / opening connections
 which is bad.
 
 [id="plugins-{type}s-{plugin}-pool_max_per_route"]
-===== `pool_max_per_route` 
+===== `pool_max_per_route`
 
   * Value type is <<number,number>>
   * Default value is `100`
@@ -673,7 +690,7 @@ Setting this too low may mean frequently closing / opening connections
 which is bad.
 
 [id="plugins-{type}s-{plugin}-proxy"]
-===== `proxy` 
+===== `proxy`
 
   * Value type is <<uri,uri>>
   * There is no default value for this setting.
@@ -684,7 +701,7 @@ An empty string is treated as if proxy was not set. This is useful when using
 environment variables e.g. `proxy => '${LS_PROXY:}'`.
 
 [id="plugins-{type}s-{plugin}-resurrect_delay"]
-===== `resurrect_delay` 
+===== `resurrect_delay`
 
   * Value type is <<number,number>>
   * Default value is `5`
@@ -694,7 +711,7 @@ Resurrection is the process by which backend endpoints marked 'down' are checked
 to see if they have come back to life
 
 [id="plugins-{type}s-{plugin}-retry_initial_interval"]
-===== `retry_initial_interval` 
+===== `retry_initial_interval`
 
   * Value type is <<number,number>>
   * Default value is `2`
@@ -702,7 +719,7 @@ to see if they have come back to life
 Set initial interval in seconds between bulk retries. Doubled on each retry up to `retry_max_interval`
 
 [id="plugins-{type}s-{plugin}-retry_max_interval"]
-===== `retry_max_interval` 
+===== `retry_max_interval`
 
   * Value type is <<number,number>>
   * Default value is `64`
@@ -710,17 +727,15 @@ Set initial interval in seconds between bulk retries. Doubled on each retry up t
 Set max interval in seconds between bulk retries.
 
 [id="plugins-{type}s-{plugin}-retry_on_conflict"]
-===== `retry_on_conflict` 
+===== `retry_on_conflict`
 
   * Value type is <<number,number>>
   * Default value is `1`
 
-The number of times Elasticsearch should internally retry an update/upserted document
-See the https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html[partial updates]
-for more info
+The number of times Elasticsearch should internally retry an update/upserted document.
 
 [id="plugins-{type}s-{plugin}-routing"]
-===== `routing` 
+===== `routing`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -729,7 +744,7 @@ A routing override to be applied to all processed events.
 This can be dynamic using the `%{foo}` syntax.
 
 [id="plugins-{type}s-{plugin}-script"]
-===== `script` 
+===== `script`
 
   * Value type is <<string,string>>
   * Default value is `""`
@@ -745,7 +760,7 @@ Example:
     }
 
 [id="plugins-{type}s-{plugin}-script_lang"]
-===== `script_lang` 
+===== `script_lang`
 
   * Value type is <<string,string>>
   * Default value is `"painless"`
@@ -754,7 +769,7 @@ Set the language of the used script. If not set, this defaults to painless in ES
 When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to `""` (empty string).
 
 [id="plugins-{type}s-{plugin}-script_type"]
-===== `script_type` 
+===== `script_type`
 
   * Value can be any of: `inline`, `indexed`, `file`
   * Default value is `["inline"]`
@@ -765,7 +780,7 @@ Define the type of script referenced by "script" variable
  file    : "script" contains the name of script stored in elasticsearch's config directory
 
 [id="plugins-{type}s-{plugin}-script_var_name"]
-===== `script_var_name` 
+===== `script_var_name`
 
   * Value type is <<string,string>>
   * Default value is `"event"`
@@ -773,7 +788,7 @@ Define the type of script referenced by "script" variable
 Set variable name passed to script (scripted update)
 
 [id="plugins-{type}s-{plugin}-scripted_upsert"]
-===== `scripted_upsert` 
+===== `scripted_upsert`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -781,7 +796,7 @@ Set variable name passed to script (scripted update)
 if enabled, script is in charge of creating non-existent document (scripted update)
 
 [id="plugins-{type}s-{plugin}-sniffing"]
-===== `sniffing` 
+===== `sniffing`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -791,7 +806,7 @@ For Elasticsearch 1.x and 2.x any nodes with `http.enabled` (on by default) will
 For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will be added to the hosts list, excluding master-only nodes.
 
 [id="plugins-{type}s-{plugin}-sniffing_delay"]
-===== `sniffing_delay` 
+===== `sniffing_delay`
 
   * Value type is <<number,number>>
   * Default value is `5`
@@ -799,7 +814,7 @@ For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
 How long to wait, in seconds, between sniffing attempts
 
 [id="plugins-{type}s-{plugin}-sniffing_path"]
-===== `sniffing_path` 
+===== `sniffing_path`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -810,7 +825,7 @@ if sniffing_path is set it will be used as an absolute path
 do not use full URL here, only paths, e.g. "/sniff/_nodes/http"
 
 [id="plugins-{type}s-{plugin}-ssl"]
-===== `ssl` 
+===== `ssl`
 
   * Value type is <<boolean,boolean>>
   * There is no default value for this setting.
@@ -820,7 +835,7 @@ is specified in the URLs listed in 'hosts'. If no explicit protocol is specified
 If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts'
 
 [id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
-===== `ssl_certificate_verification` 
+===== `ssl_certificate_verification`
 
   * Value type is <<boolean,boolean>>
   * Default value is `true`
@@ -830,7 +845,7 @@ For more information on disabling certificate verification please read
 https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
 
 [id="plugins-{type}s-{plugin}-template"]
-===== `template` 
+===== `template`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -839,7 +854,7 @@ You can set the path to your own template here, if you so desire.
 If not set, the included template will be used.
 
 [id="plugins-{type}s-{plugin}-template_name"]
-===== `template_name` 
+===== `template_name`
 
   * Value type is <<string,string>>
   * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
@@ -856,7 +871,7 @@ change this, you will need to prune the old template manually, e.g.
 where `OldTemplateName` is whatever the former setting was.
 
 [id="plugins-{type}s-{plugin}-template_overwrite"]
-===== `template_overwrite` 
+===== `template_overwrite`
 
   * Value type is <<boolean,boolean>>
   * Default value is `false`
@@ -873,7 +888,7 @@ template (logstash), setting this to true will make Logstash to overwrite
 the "logstash" template (i.e. removing all customized settings)
 
 [id="plugins-{type}s-{plugin}-timeout"]
-===== `timeout` 
+===== `timeout`
 
   * Value type is <<number,number>>
   * Default value is `60`
@@ -882,7 +897,7 @@ Set the timeout, in seconds, for network operations and requests sent Elasticsea
 a timeout occurs, the request will be retried.
 
 [id="plugins-{type}s-{plugin}-truststore"]
-===== `truststore` 
+===== `truststore`
 
   * Value type is <<path,path>>
   * There is no default value for this setting.
@@ -892,7 +907,7 @@ It can be either .jks or .p12.
 Use either `:truststore` or `:cacert`.
 
 [id="plugins-{type}s-{plugin}-truststore_password"]
-===== `truststore_password` 
+===== `truststore_password`
 
   * Value type is <<password,password>>
   * There is no default value for this setting.
@@ -900,7 +915,7 @@ Use either `:truststore` or `:cacert`.
 Set the truststore password
 
 [id="plugins-{type}s-{plugin}-upsert"]
-===== `upsert` 
+===== `upsert`
 
   * Value type is <<string,string>>
   * Default value is `""`
@@ -909,7 +924,7 @@ Set upsert content for update mode.
 Create a new document with this parameter as json string if `document_id` doesn't exists
 
 [id="plugins-{type}s-{plugin}-user"]
-===== `user` 
+===== `user`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -917,7 +932,7 @@ Create a new document with this parameter as json string if `document_id` doesn'
 Username to authenticate to a secure Elasticsearch cluster
 
 [id="plugins-{type}s-{plugin}-validate_after_inactivity"]
-===== `validate_after_inactivity` 
+===== `validate_after_inactivity`
 
   * Value type is <<number,number>>
   * Default value is `10000`
@@ -932,7 +947,7 @@ have become stale (half-closed) while kept inactive in the pool.'
 See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[these docs for more info]
 
 [id="plugins-{type}s-{plugin}-version"]
-===== `version` 
+===== `version`
 
   * Value type is <<string,string>>
   * There is no default value for this setting.
@@ -941,7 +956,7 @@ The version to use for indexing. Use sprintf syntax like `%{my_version}` to use
 See https://www.elastic.co/blog/elasticsearch-versioning-support.
 
 [id="plugins-{type}s-{plugin}-version_type"]
-===== `version_type` 
+===== `version_type`
 
   * Value can be any of: `internal`, `external`, `external_gt`, `external_gte`, `force`
   * There is no default value for this setting.

data/lib/logstash/outputs/elasticsearch/http_client.rb

@@ -343,15 +343,19 @@ module LogStash; module Outputs; class ElasticSearch;
     end
 
     def template_exists?(name)
-      exists?("/_template/#{name}")
+      exists?("/#{template_endpoint}/#{name}")
     end
 
     def template_put(name, template)
-      path = "_template/#{name}"
+      path = "#{template_endpoint}/#{name}"
       logger.info("Installing elasticsearch template to #{path}")
       @pool.put(path, nil, LogStash::Json.dump(template))
     end
 
+    def template_endpoint
+      maximum_seen_major_version < 8 ? '_template' : '_index_template'
+    end
+
     # ILM methods
 
     # check whether rollover alias already exists

data/lib/logstash/outputs/elasticsearch/template_manager.rb

@@ -34,12 +34,17 @@ module LogStash; module Outputs; class ElasticSearch
     def self.add_ilm_settings_to_template(plugin, template)
       # Overwrite any index patterns, and use the rollover alias. Use 'index_patterns' rather than 'template' for pattern
       # definition - remove any existing definition of 'template'
-      template.delete('template') if template.include?('template')
+      template.delete('template') if template.include?('template') if plugin.maximum_seen_major_version < 8
       template['index_patterns'] = "#{plugin.ilm_rollover_alias}-*"
-      if template['settings'] && (template['settings']['index.lifecycle.name'] || template['settings']['index.lifecycle.rollover_alias'])
+      settings = template_settings(plugin, template)
+      if settings && (settings['index.lifecycle.name'] || settings['index.lifecycle.rollover_alias'])
         plugin.logger.info("Overwriting index lifecycle name and rollover alias as ILM is enabled.")
       end
-      template['settings'].update({ 'index.lifecycle.name' => plugin.ilm_policy, 'index.lifecycle.rollover_alias' => plugin.ilm_rollover_alias})
+      settings.update({ 'index.lifecycle.name' => plugin.ilm_policy, 'index.lifecycle.rollover_alias' => plugin.ilm_rollover_alias})
+    end
+
+    def self.template_settings(plugin, template)
+      plugin.maximum_seen_major_version < 8 ? template['settings']: template['template']['settings']
     end
 
     # Template name - if template_name set, use it

data/lib/logstash/outputs/elasticsearch/templates/ecs-disabled/elasticsearch-8x.json

@@ -1,44 +1,50 @@
 {
   "index_patterns" : "logstash-*",
   "version" : 80001,
-  "settings" : {
-    "index.refresh_interval" : "5s",
-    "number_of_shards": 1
-  },
-  "mappings" : {
-    "dynamic_templates" : [ {
-      "message_field" : {
-        "path_match" : "message",
-        "match_mapping_type" : "string",
-        "mapping" : {
-          "type" : "text",
-          "norms" : false
+  "template" : {
+    "settings" : {
+      "index.refresh_interval" : "5s",
+      "number_of_shards": 1
+    },
+    "mappings" : {
+      "dynamic_templates" : [ {
+        "message_field" : {
+          "path_match" : "message",
+          "match_mapping_type" : "string",
+          "mapping" : {
+            "type" : "text",
+            "norms" : false
+          }
         }
-      }
-    }, {
-      "string_fields" : {
-        "match" : "*",
-        "match_mapping_type" : "string",
-        "mapping" : {
-          "type" : "text", "norms" : false,
-          "fields" : {
-            "keyword" : { "type": "keyword", "ignore_above": 256 }
+      }, {
+        "string_fields" : {
+          "match" : "*",
+          "match_mapping_type" : "string",
+          "mapping" : {
+            "type" : "text", "norms" : false,
+            "fields" : {
+              "keyword" : { "type": "keyword", "ignore_above": 256 }
+            }
           }
         }
-      }
-    } ],
-    "properties" : {
-      "@timestamp": { "type": "date"},
-      "@version": { "type": "keyword"},
-      "geoip"  : {
-        "dynamic": true,
-        "properties" : {
-          "ip": { "type": "ip" },
-          "location" : { "type" : "geo_point" },
-          "latitude" : { "type" : "half_float" },
-          "longitude" : { "type" : "half_float" }
+      } ],
+      "properties" : {
+        "@timestamp": { "type": "date" },
+        "@version": { "type": "keyword" },
+        "geoip"  : {
+          "dynamic": true,
+          "properties" : {
+            "ip": { "type": "ip" },
+            "location" : { "type" : "geo_point" },
+            "latitude" : { "type" : "half_float" },
+            "longitude" : { "type" : "half_float" }
+          }
         }
       }
     }
+  },
+  "priority": 200,
+  "_meta" : {
+    "description": "index template for logstash-output-elasticsearch"
   }
 }

data/logstash-output-elasticsearch.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-output-elasticsearch'
-  s.version         = '10.7.0'
+  s.version         = '10.7.3'
 
   s.licenses        = ['apache-2.0']
   s.summary         = "Stores logs in Elasticsearch"

data/spec/es_spec_helper.rb

@@ -49,19 +49,10 @@ module ESHelper
     Time.now.strftime("%Y.%m.%d")
   end
 
-
-  def default_mapping_from_mappings(mappings)
-    if ESHelper.es_version_satisfies?(">=7")
-      mappings
-    else
-      mappings["_default_"]
-    end
-  end
-
   def field_properties_from_template(template_name, field)
-    mappings = @es.indices.get_template(name: template_name)[template_name]["mappings"]
-    mapping = default_mapping_from_mappings(mappings)
-    mapping["properties"][field]["properties"]
+    template = get_template(@es, template_name)
+    mappings = get_template_mappings(template)
+    mappings["properties"][field]["properties"]
   end
 
   def routing_field_name
@@ -105,6 +96,7 @@ module ESHelper
 
   def clean(client)
     client.indices.delete_template(:name => "*")
+    client.indices.delete_index_template(:name => "logstash*") rescue nil
     # This can fail if there are no indexes, ignore failure.
     client.indices.delete(:index => "*") rescue nil
     clean_ilm(client) if supports_ilm?(client)
@@ -182,6 +174,34 @@ module ESHelper
     }
   }
   end
+
+  def get_template(client, name)
+    if ESHelper.es_version_satisfies?(">=8")
+      t = client.indices.get_index_template(name: name)
+      t['index_templates'][0]['index_template']
+    else
+      t = client.indices.get_template(name: name)
+      t[name]
+    end
+  end
+
+  def get_template_settings(template)
+    if ESHelper.es_version_satisfies?(">=8")
+      template['template']['settings']
+    else
+      template['settings']
+    end
+  end
+
+  def get_template_mappings(template)
+    if ESHelper.es_version_satisfies?(">=8")
+      template['template']['mappings']
+    elsif ESHelper.es_version_satisfies?(">=7")
+      template['mappings']
+    else
+      template['mappings']["_default_"]
+    end
+  end
 end
 
 RSpec.configure do |config|

data/spec/fixtures/template-with-policy-es8x.json

@@ -0,0 +1,50 @@
+{
+  "index_patterns" : "overwrite-*",
+  "version" : 80001,
+  "template" : {
+    "settings" : {
+      "index.refresh_interval" : "1s",
+      "number_of_shards": 1
+    },
+    "mappings" : {
+      "dynamic_templates" : [ {
+        "message_field" : {
+          "path_match" : "message",
+          "match_mapping_type" : "string",
+          "mapping" : {
+            "type" : "text",
+            "norms" : false
+          }
+        }
+      }, {
+        "string_fields" : {
+          "match" : "*",
+          "match_mapping_type" : "string",
+          "mapping" : {
+            "type" : "text", "norms" : false,
+            "fields" : {
+              "keyword" : { "type": "keyword", "ignore_above": 256 }
+            }
+          }
+        }
+      } ],
+      "properties" : {
+        "@timestamp": { "type": "date" },
+        "@version": { "type": "keyword" },
+        "geoip"  : {
+          "dynamic": true,
+          "properties" : {
+            "ip": { "type": "ip" },
+            "location" : { "type" : "geo_point" },
+            "latitude" : { "type" : "half_float" },
+            "longitude" : { "type" : "half_float" }
+          }
+        }
+      }
+    }
+  },
+  "priority": 200,
+  "_meta" : {
+    "description": "index template for logstash-output-elasticsearch"
+  }
+}

data/spec/integration/outputs/ilm_spec.rb

@@ -8,7 +8,7 @@ shared_examples_for 'an ILM enabled Logstash' do
     let (:settings) { super.merge("ilm_policy" => ilm_policy_name)}
 
     it 'should rollover when the policy max docs is reached' do
-      put_policy(@es,ilm_policy_name, policy)
+      put_policy(@es, ilm_policy_name, policy)
       subject.register
 
       subject.multi_receive([
@@ -108,9 +108,11 @@ shared_examples_for 'an ILM disabled Logstash' do
   it 'should not write the ILM settings into the template' do
     subject.register
     sleep(1)
-    expect(@es.indices.get_template(name: "logstash")["logstash"]).to have_index_pattern("logstash-*")
+
+    template = get_template(@es, "logstash")
+    expect(template).to have_index_pattern("logstash-*")
     if ESHelper.es_version_satisfies?(">= 2")
-      expect(@es.indices.get_template(name: "logstash")["logstash"]["settings"]['index']['lifecycle']).to be_nil
+      expect(get_template_settings(template)['index']['lifecycle']).to be_nil
     end
   end
 
@@ -152,16 +154,17 @@ shared_examples_for 'an ILM disabled Logstash' do
   end
 
   context 'with a custom template name' do
-    let (:template_name) { "custom_template_name" }
+    let (:template_name) { "logstash_custom_template_name" }
     let (:settings) { super.merge('template_name' => template_name)}
 
     it 'should not write the ILM settings into the template' do
       subject.register
       sleep(1)
 
-      expect(@es.indices.get_template(name: template_name)[template_name]).to have_index_pattern("logstash-*")
+      template = get_template(@es, template_name)
+      expect(template).to have_index_pattern("logstash-*")
       if ESHelper.es_version_satisfies?(">= 2")
-        expect(@es.indices.get_template(name: template_name)[template_name]["settings"]['index']['lifecycle']).to be_nil
+        expect(get_template_settings(template)['index']['lifecycle']).to be_nil
       end
     end
   end
@@ -387,16 +390,20 @@ if ESHelper.es_version_satisfies?(">= 6.6")
         it 'should write the ILM settings into the template' do
           subject.register
           sleep(1)
-          expect(@es.indices.get_template(name: "logstash")["logstash"]).to have_index_pattern("logstash-*")
-          expect(@es.indices.get_template(name: "logstash")["logstash"]["settings"]['index']['lifecycle']['name']).to eq("logstash-policy")
-          expect(@es.indices.get_template(name: "logstash")["logstash"]["settings"]['index']['lifecycle']['rollover_alias']).to eq("logstash")
+
+          template = get_template(@es, "logstash")
+          expect(template).to have_index_pattern("logstash-*")
+          expect(get_template_settings(template)['index']['lifecycle']['name']).to eq("logstash-policy")
+          expect(get_template_settings(template)['index']['lifecycle']['rollover_alias']).to eq("logstash")
         end
 
         it_behaves_like 'an ILM enabled Logstash'
       end
 
       context 'with a set index and a custom index pattern' do
-        if ESHelper.es_version_satisfies?(">= 7.0")
+        if ESHelper.es_version_satisfies?(">= 8.0")
+          let (:template) { "spec/fixtures/template-with-policy-es8x.json" }
+        elsif ESHelper.es_version_satisfies?(">= 7.0")
           let (:template) { "spec/fixtures/template-with-policy-es7x.json" }
         else
           let (:template) { "spec/fixtures/template-with-policy-es6x.json" }
@@ -408,13 +415,15 @@ if ESHelper.es_version_satisfies?(">= 6.6")
         it 'should not overwrite the index patterns' do
           subject.register
           sleep(1)
-          expect(@es.indices.get_template(name: "logstash")["logstash"]).to have_index_pattern("overwrite-*")
+
+          template = get_template(@es, "logstash")
+          expect(template).to have_index_pattern("overwrite-*")
         end
       end
 
 
       context 'with a custom template' do
-        let (:ilm_rollover_alias) { "the_cat_in_the_hat" }
+        let (:ilm_rollover_alias) { "logstash_the_cat_in_the_hat" }
         let (:index) { ilm_rollover_alias }
         let(:expected_index) { index }
         let (:settings) { super.merge("ilm_policy" => ilm_policy_name,
@@ -422,7 +431,9 @@ if ESHelper.es_version_satisfies?(">= 6.6")
                                       "ilm_rollover_alias" => ilm_rollover_alias)}
 
 
-        if ESHelper.es_version_satisfies?(">= 7.0")
+        if ESHelper.es_version_satisfies?(">= 8.0")
+          let (:template) { "spec/fixtures/template-with-policy-es8x.json" }
+        elsif ESHelper.es_version_satisfies?(">= 7.0")
           let (:template) { "spec/fixtures/template-with-policy-es7x.json" }
         else
           let (:template) { "spec/fixtures/template-with-policy-es6x.json" }
@@ -460,13 +471,15 @@ if ESHelper.es_version_satisfies?(">= 6.6")
         it 'should write the ILM settings into the template' do
           subject.register
           sleep(1)
-          expect(@es.indices.get_template(name: ilm_rollover_alias)[ilm_rollover_alias]).to have_index_pattern("#{ilm_rollover_alias}-*")
-          expect(@es.indices.get_template(name: ilm_rollover_alias)[ilm_rollover_alias]["settings"]['index']['lifecycle']['name']).to eq(ilm_policy_name)
-          expect(@es.indices.get_template(name: ilm_rollover_alias)[ilm_rollover_alias]["settings"]['index']['lifecycle']['rollover_alias']).to eq(ilm_rollover_alias)
+
+          template = get_template(@es, ilm_rollover_alias)
+          expect(template).to have_index_pattern("#{ilm_rollover_alias}-*")
+          expect(get_template_settings(template)['index']['lifecycle']['name']).to eq(ilm_policy_name)
+          expect(get_template_settings(template)['index']['lifecycle']['rollover_alias']).to eq(ilm_rollover_alias)
         end
 
         context 'with a different template_name' do
-          let (:template_name) { "custom_template_name" }
+          let (:template_name) { "logstash_custom_template_name" }
           let (:settings) { super.merge('template_name' => template_name)}
 
           it_behaves_like 'an ILM enabled Logstash'
@@ -474,9 +487,10 @@ if ESHelper.es_version_satisfies?(">= 6.6")
           it 'should write the ILM settings into the template' do
             subject.register
             sleep(1)
-            expect(@es.indices.get_template(name: template_name)[template_name]).to have_index_pattern("#{ilm_rollover_alias}-*")
-            expect(@es.indices.get_template(name: template_name)[template_name]["settings"]['index']['lifecycle']['name']).to eq(ilm_policy_name)
-            expect(@es.indices.get_template(name: template_name)[template_name]["settings"]['index']['lifecycle']['rollover_alias']).to eq(ilm_rollover_alias)
+            template = get_template(@es, template_name)
+            expect(template).to have_index_pattern("#{ilm_rollover_alias}-*")
+            expect(get_template_settings(template)['index']['lifecycle']['name']).to eq(ilm_policy_name)
+            expect(get_template_settings(template)['index']['lifecycle']['rollover_alias']).to eq(ilm_rollover_alias)
           end
         end
 

data/spec/integration/outputs/metrics_spec.rb

@@ -19,11 +19,7 @@ describe "metrics", :integration => true do
 
     # Clean ES of data before we start.
     @es = get_client
-    @es.indices.delete_template(:name => "*")
-
-    # This can fail if there are no indexes, ignore failure.
-    @es.indices.delete(:index => "*") rescue nil
-    #@es.indices.refresh
+    clean(@es)
     subject.register
   end
 

data/spec/unit/outputs/elasticsearch/http_client_spec.rb

@@ -1,3 +1,4 @@
+require_relative "../../../../spec/es_spec_helper"
 require "logstash/devutils/rspec/spec_helper"
 require "logstash/outputs/elasticsearch/http_client"
 require "java"
@@ -138,6 +139,27 @@ describe LogStash::Outputs::ElasticSearch::HttpClient do
     end
   end
 
+  describe "index template" do
+    subject { described_class.new(base_options) }
+    let(:template_name) { "logstash" }
+    let(:template) { {} }
+    let(:get_response) {
+      double("response", :body => {})
+    }
+
+    it "should call composable index template in version 8+" do
+      expect(subject).to receive(:maximum_seen_major_version).and_return(8)
+      expect(subject.pool).to receive(:put).with("_index_template/#{template_name}", nil, anything).and_return(get_response)
+      subject.template_put(template_name, template)
+    end
+
+    it "should call index template in version < 8" do
+      expect(subject).to receive(:maximum_seen_major_version).and_return(7)
+      expect(subject.pool).to receive(:put).with("_template/#{template_name}", nil, anything).and_return(get_response)
+      subject.template_put(template_name, template)
+    end
+  end
+
   describe "join_bulk_responses" do
     subject { described_class.new(base_options) }
 

data/spec/unit/outputs/elasticsearch/template_manager_spec.rb

@@ -28,4 +28,35 @@ describe LogStash::Outputs::ElasticSearch::TemplateManager do
       expect(described_class.default_template_path(7, :v1)).to end_with("/templates/ecs-v1/elasticsearch-7x.json")
     end
   end
+
+  describe "index template with ilm settings" do
+    let(:plugin_settings) { {"manage_template" => true, "template_overwrite" => true} }
+    let(:plugin) { LogStash::Outputs::ElasticSearch.new(plugin_settings) }
+
+    describe "in version 8+" do
+      let(:file_path) { described_class.default_template_path(8) }
+      let(:template) { described_class.read_template_file(file_path)}
+
+      it "should update settings" do
+        expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(8)
+        described_class.add_ilm_settings_to_template(plugin, template)
+        expect(template['template']['settings']['index.lifecycle.name']).not_to eq(nil)
+        expect(template['template']['settings']['index.lifecycle.rollover_alias']).not_to eq(nil)
+        expect(template.include?('settings')).to be_falsey
+      end
+    end
+
+    describe "in version < 8" do
+      let(:file_path) { described_class.default_template_path(7) }
+      let(:template) { described_class.read_template_file(file_path)}
+
+      it "should update settings" do
+        expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(7)
+        described_class.add_ilm_settings_to_template(plugin, template)
+        expect(template['settings']['index.lifecycle.name']).not_to eq(nil)
+        expect(template['settings']['index.lifecycle.rollover_alias']).not_to eq(nil)
+        expect(template.include?('template')).to be_falsey
+      end
+    end
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 10.7.0
+  version: 10.7.3
 platform: java
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-08-13 00:00:00.000000000 Z
+date: 2020-11-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -201,6 +201,7 @@ files:
 - spec/fixtures/scripts/painless/scripted_upsert.painless
 - spec/fixtures/template-with-policy-es6x.json
 - spec/fixtures/template-with-policy-es7x.json
+- spec/fixtures/template-with-policy-es8x.json
 - spec/fixtures/test_certs/ca/ca.crt
 - spec/fixtures/test_certs/ca/ca.key
 - spec/fixtures/test_certs/test.crt
@@ -278,6 +279,7 @@ test_files:
 - spec/fixtures/scripts/painless/scripted_upsert.painless
 - spec/fixtures/template-with-policy-es6x.json
 - spec/fixtures/template-with-policy-es7x.json
+- spec/fixtures/template-with-policy-es8x.json
 - spec/fixtures/test_certs/ca/ca.crt
 - spec/fixtures/test_certs/ca/ca.key
 - spec/fixtures/test_certs/test.crt