logstash-output-elasticsearch 10.7.3-java → 10.8.4-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +16 -0
- data/README.md +1 -1
- data/docs/index.asciidoc +157 -99
- data/lib/logstash/outputs/elasticsearch.rb +302 -165
- data/lib/logstash/outputs/elasticsearch/http_client.rb +2 -0
- data/lib/logstash/outputs/elasticsearch/http_client/pool.rb +13 -28
- data/lib/logstash/outputs/elasticsearch/http_client_builder.rb +1 -0
- data/lib/logstash/outputs/elasticsearch/ilm.rb +9 -5
- data/lib/logstash/outputs/elasticsearch/license_checker.rb +47 -0
- data/lib/logstash/plugin_mixins/elasticsearch/api_configs.rb +163 -0
- data/lib/logstash/{outputs → plugin_mixins}/elasticsearch/common.rb +40 -167
- data/lib/logstash/plugin_mixins/elasticsearch/noop_license_checker.rb +9 -0
- data/logstash-output-elasticsearch.gemspec +1 -1
- data/spec/integration/outputs/retry_spec.rb +14 -2
- data/spec/unit/outputs/elasticsearch/http_client/pool_spec.rb +45 -5
- data/spec/unit/outputs/elasticsearch_spec.rb +2 -2
- data/spec/unit/outputs/license_check_spec.rb +41 -0
- metadata +8 -4
- data/lib/logstash/outputs/elasticsearch/common_configs.rb +0 -167
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 7bbaed62e36b6543bf68420603a64e84f39a61e3ffaaf3456f28fad73295430c
|
4
|
+
data.tar.gz: '0682c7ac69384d7db244e33186237461c148683cc205b72660251e5d3041765c'
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8a546bff1b7623d8e37fd11b9551691ff87072831f81b00af1c4bfd8086e8f7ee45fad2b78682f68927e97d883dfc923a90f76bc53f648885fcd3d5d3c437028
|
7
|
+
data.tar.gz: 4ea4dc460aa38935cd4e72d2c35c069fb372d02c99fa5cf47e48dbce13994d3615296a3601c8a6c6641f6db587ec0776bc04fcd7b7902c32f31c8c6e95eb4340
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,19 @@
|
|
1
|
+
## 10.8.4
|
2
|
+
- Fixed an issue where a retried request would drop "update" parameters [#800](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/800)
|
3
|
+
|
4
|
+
## 10.8.3
|
5
|
+
- Avoid to implicitly set deprecated type to `_doc` when connects to Elasticsearch version 7.x [#994](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/994)
|
6
|
+
|
7
|
+
## 10.8.2
|
8
|
+
- [DOC] Update links to use shared attributes [#985](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/985)
|
9
|
+
|
10
|
+
## 10.8.1
|
11
|
+
- Fixed an issue when assigning the no-op license checker [#984](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/984)
|
12
|
+
|
13
|
+
## 10.8.0
|
14
|
+
- Refactored configuration options into specific and shared in PluginMixins namespace [#973](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/973)
|
15
|
+
- Refactored common methods into specific and shared in PluginMixins namespace [#976](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/976)
|
16
|
+
|
1
17
|
## 10.7.3
|
2
18
|
- Added composable index template support for elasticsearch version 8 [#980](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/980)
|
3
19
|
|
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# Logstash Plugin
|
2
2
|
|
3
|
-
[![Travis Build Status](https://travis-ci.
|
3
|
+
[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-output-elasticsearch.svg)](https://travis-ci.com/logstash-plugins/logstash-output-elasticsearch)
|
4
4
|
|
5
5
|
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
6
6
|
|
data/docs/index.asciidoc
CHANGED
@@ -29,18 +29,19 @@ This output only speaks the HTTP protocol as it is the preferred protocol for
|
|
29
29
|
interacting with Elasticsearch. In previous versions it was possible to
|
30
30
|
communicate with Elasticsearch through the transport protocol, which is now
|
31
31
|
reserved for internal cluster communication between nodes
|
32
|
-
|
33
|
-
Using the
|
34
|
-
|
35
|
-
will be removed in 8.0.0
|
32
|
+
{ref}/modules-transport.html[communication between nodes].
|
33
|
+
Using the transport protocol to communicate with the cluster has been deprecated
|
34
|
+
in Elasticsearch 7.0.0 and will be removed in 8.0.0
|
36
35
|
|
37
|
-
You can
|
36
|
+
You can https://www.elastic.co/elasticsearch/[learn more about Elasticsearch] on
|
37
|
+
the website landing page or in the {ref}[Elasticsearch documentation].
|
38
38
|
|
39
39
|
.Compatibility Note
|
40
40
|
[NOTE]
|
41
41
|
================================================================================
|
42
42
|
When connected to Elasticsearch 7.x, modern versions of this plugin
|
43
|
-
use the
|
43
|
+
don't use the document-type when inserting documents, unless the user
|
44
|
+
explicitly sets <<plugins-{type}s-{plugin}-document_type>>.
|
44
45
|
|
45
46
|
If you are using an earlier version of Logstash and wish to connect to
|
46
47
|
Elasticsearch 7.x, first upgrade Logstash to version 6.8 to ensure it
|
@@ -103,9 +104,8 @@ Example:
|
|
103
104
|
|
104
105
|
**What to do in case there is no field in the event containing the destination index prefix?**
|
105
106
|
|
106
|
-
You can use the `mutate` filter and conditionals to add a
|
107
|
-
|
108
|
-
https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata)
|
107
|
+
You can use the `mutate` filter and conditionals to add a
|
108
|
+
{logstash-ref}/event-dependent-configuration.html#metadata[`[@metadata]` field]
|
109
109
|
to set the destination index for each event. The `[@metadata]` fields will not
|
110
110
|
be sent to Elasticsearch.
|
111
111
|
|
@@ -159,30 +159,37 @@ happens, the problem is logged as a warning, and the event is dropped. See
|
|
159
159
|
[id="plugins-{type}s-{plugin}-ilm"]
|
160
160
|
==== Index Lifecycle Management
|
161
161
|
|
162
|
-
|
163
162
|
[NOTE]
|
164
163
|
The Index Lifecycle Management feature requires plugin version `9.3.1` or higher.
|
165
164
|
|
166
165
|
[NOTE]
|
167
166
|
This feature requires an Elasticsearch instance of 6.6.0 or higher with at least a Basic license
|
168
167
|
|
169
|
-
Logstash can use {ref}/index-lifecycle-management.html[Index Lifecycle
|
168
|
+
Logstash can use {ref}/index-lifecycle-management.html[Index Lifecycle
|
169
|
+
Management] to automate the management of indices over time.
|
170
170
|
|
171
171
|
The use of Index Lifecycle Management is controlled by the `ilm_enabled`
|
172
172
|
setting. By default, this setting detects whether the Elasticsearch instance
|
173
173
|
supports ILM, and uses it if it is available. `ilm_enabled` can also be set to
|
174
174
|
`true` or `false` to override the automatic detection, or disable ILM.
|
175
175
|
|
176
|
-
This will overwrite the index settings and adjust the Logstash template to write
|
177
|
-
|
176
|
+
This will overwrite the index settings and adjust the Logstash template to write
|
177
|
+
the necessary settings for the template to support index lifecycle management,
|
178
|
+
including the index policy and rollover alias to be used.
|
178
179
|
|
179
|
-
Logstash will create a rollover alias for the indices to be written to,
|
180
|
-
a
|
180
|
+
Logstash will create a rollover alias for the indices to be written to,
|
181
|
+
including a pattern for how the actual indices will be named, and unless an ILM
|
182
|
+
policy that already exists has been specified, a default policy will also be
|
183
|
+
created. The default policy is configured to rollover an index when it reaches
|
184
|
+
either 50 gigabytes in size, or is 30 days old, whichever happens first.
|
181
185
|
|
182
|
-
The default rollover alias is called `logstash`, with a default pattern for the
|
183
|
-
which will name indices on the date that the
|
186
|
+
The default rollover alias is called `logstash`, with a default pattern for the
|
187
|
+
rollover index of `{now/d}-00001`, which will name indices on the date that the
|
188
|
+
index is rolled over, followed by an incrementing number. Note that the pattern
|
189
|
+
must end with a dash and a number that will be incremented.
|
184
190
|
|
185
|
-
See the {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
|
191
|
+
See the {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
|
192
|
+
API documentation] for more details on naming.
|
186
193
|
|
187
194
|
The rollover alias, ilm pattern and policy can be modified.
|
188
195
|
|
@@ -198,19 +205,24 @@ See config below for an example:
|
|
198
205
|
|
199
206
|
NOTE: Custom ILM policies must already exist on the Elasticsearch cluster before they can be used.
|
200
207
|
|
201
|
-
NOTE: If the rollover alias or pattern is modified, the index template will need to be
|
208
|
+
NOTE: If the rollover alias or pattern is modified, the index template will need to be
|
209
|
+
overwritten as the settings `index.lifecycle.name` and
|
210
|
+
`index.lifecycle.rollover_alias` are automatically written to the template
|
202
211
|
|
203
212
|
NOTE: If the index property is supplied in the output definition, it will be overwritten by the rollover alias.
|
204
213
|
|
205
214
|
|
206
215
|
==== Batch Sizes
|
207
216
|
|
208
|
-
This plugin attempts to send batches of events
|
209
|
-
|
217
|
+
This plugin attempts to send batches of events to the {ref}/docs-bulk.html[{es}
|
218
|
+
Bulk API] as a single request. However, if a batch exceeds 20MB we break it up
|
219
|
+
into multiple bulk requests. If a single document exceeds 20MB it is sent as a
|
220
|
+
single request.
|
210
221
|
|
211
222
|
==== DNS Caching
|
212
223
|
|
213
|
-
This plugin uses the JVM to lookup DNS entries and is subject to the value of
|
224
|
+
This plugin uses the JVM to lookup DNS entries and is subject to the value of
|
225
|
+
https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl],
|
214
226
|
a global setting for the JVM.
|
215
227
|
|
216
228
|
As an example, to set your DNS TTL to 1 second you would set
|
@@ -226,8 +238,7 @@ enabled by default for HTTP and for Elasticsearch versions 5.0 and later.
|
|
226
238
|
|
227
239
|
You don't have to set any configs in Elasticsearch for it to send back a
|
228
240
|
compressed response. For versions before 5.0, or if HTTPS is enabled,
|
229
|
-
`http.compression` must be set to `true`
|
230
|
-
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[in
|
241
|
+
`http.compression` must be set to `true` {ref}/modules-http.html#modules-http[in
|
231
242
|
Elasticsearch] to take advantage of response compression when using this plugin.
|
232
243
|
|
233
244
|
For requests compression, regardless of the Elasticsearch version, enable the
|
@@ -235,19 +246,23 @@ For requests compression, regardless of the Elasticsearch version, enable the
|
|
235
246
|
|
236
247
|
==== Authentication
|
237
248
|
|
238
|
-
Authentication to a secure Elasticsearch cluster is possible using one of the
|
249
|
+
Authentication to a secure Elasticsearch cluster is possible using one of the
|
250
|
+
`user`/`password`, `cloud_auth` or `api_key` options.
|
239
251
|
|
240
252
|
[id="plugins-{type}s-{plugin}-autz"]
|
241
253
|
==== Authorization
|
242
254
|
|
243
|
-
Authorization to a secure Elasticsearch cluster requires `read` permission at
|
244
|
-
|
255
|
+
Authorization to a secure Elasticsearch cluster requires `read` permission at
|
256
|
+
index level and `monitoring` permissions at cluster level. The `monitoring`
|
257
|
+
permission at cluster level is necessary to perform periodic connectivity
|
258
|
+
checks.
|
245
259
|
|
246
260
|
|
247
261
|
[id="plugins-{type}s-{plugin}-options"]
|
248
262
|
==== Elasticsearch Output Configuration Options
|
249
263
|
|
250
|
-
This plugin supports the following configuration options plus the
|
264
|
+
This plugin supports the following configuration options plus the
|
265
|
+
<<plugins-{type}s-{plugin}-common-options>> described later.
|
251
266
|
|
252
267
|
[cols="<,<,<",options="header",]
|
253
268
|
|=======================================================================
|
@@ -335,7 +350,8 @@ The Elasticsearch action to perform. Valid actions are:
|
|
335
350
|
- A sprintf style string to change the action based on the content of the event. The value `%{[foo]}`
|
336
351
|
would use the foo field for the action
|
337
352
|
|
338
|
-
For more details on actions, check out the
|
353
|
+
For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch
|
354
|
+
bulk API documentation].
|
339
355
|
|
340
356
|
[id="plugins-{type}s-{plugin}-api_key"]
|
341
357
|
===== `api_key`
|
@@ -343,9 +359,11 @@ For more details on actions, check out the http://www.elastic.co/guide/en/elasti
|
|
343
359
|
* Value type is <<password,password>>
|
344
360
|
* There is no default value for this setting.
|
345
361
|
|
346
|
-
Authenticate using Elasticsearch API key. Note that this option also requires
|
362
|
+
Authenticate using Elasticsearch API key. Note that this option also requires
|
363
|
+
enabling the `ssl` option.
|
347
364
|
|
348
|
-
Format is `id:api_key` where `id` and `api_key` are as returned by the
|
365
|
+
Format is `id:api_key` where `id` and `api_key` are as returned by the
|
366
|
+
Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
|
349
367
|
|
350
368
|
[id="plugins-{type}s-{plugin}-bulk_path"]
|
351
369
|
===== `bulk_path`
|
@@ -362,7 +380,7 @@ this defaults to a concatenation of the path parameter and "_bulk"
|
|
362
380
|
* Value type is <<path,path>>
|
363
381
|
* There is no default value for this setting.
|
364
382
|
|
365
|
-
The .cer or .pem file to validate the server's certificate
|
383
|
+
The .cer or .pem file to validate the server's certificate.
|
366
384
|
|
367
385
|
[id="plugins-{type}s-{plugin}-cloud_auth"]
|
368
386
|
===== `cloud_auth`
|
@@ -370,9 +388,11 @@ The .cer or .pem file to validate the server's certificate
|
|
370
388
|
* Value type is <<password,password>>
|
371
389
|
* There is no default value for this setting.
|
372
390
|
|
373
|
-
Cloud authentication string ("<username>:<password>" format) is an alternative
|
391
|
+
Cloud authentication string ("<username>:<password>" format) is an alternative
|
392
|
+
for the `user`/`password` pair.
|
374
393
|
|
375
|
-
For more details, check out the
|
394
|
+
For more details, check out the
|
395
|
+
{logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].
|
376
396
|
|
377
397
|
[id="plugins-{type}s-{plugin}-cloud_id"]
|
378
398
|
===== `cloud_id`
|
@@ -382,7 +402,8 @@ For more details, check out the https://www.elastic.co/guide/en/logstash/current
|
|
382
402
|
|
383
403
|
Cloud ID, from the Elastic Cloud web console. If set `hosts` should not be used.
|
384
404
|
|
385
|
-
For more details, check out the
|
405
|
+
For more details, check out the
|
406
|
+
{logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].
|
386
407
|
|
387
408
|
[id="plugins-{type}s-{plugin}-doc_as_upsert"]
|
388
409
|
===== `doc_as_upsert`
|
@@ -391,7 +412,7 @@ For more details, check out the https://www.elastic.co/guide/en/logstash/current
|
|
391
412
|
* Default value is `false`
|
392
413
|
|
393
414
|
Enable `doc_as_upsert` for update mode.
|
394
|
-
Create a new document with source if `document_id` doesn't exist in Elasticsearch
|
415
|
+
Create a new document with source if `document_id` doesn't exist in Elasticsearch.
|
395
416
|
|
396
417
|
[id="plugins-{type}s-{plugin}-document_id"]
|
397
418
|
===== `document_id`
|
@@ -399,7 +420,8 @@ Create a new document with source if `document_id` doesn't exist in Elasticsearc
|
|
399
420
|
* Value type is <<string,string>>
|
400
421
|
* There is no default value for this setting.
|
401
422
|
|
402
|
-
The document ID for the index. Useful for overwriting existing entries in
|
423
|
+
The document ID for the index. Useful for overwriting existing entries in
|
424
|
+
Elasticsearch with the same ID.
|
403
425
|
|
404
426
|
[id="plugins-{type}s-{plugin}-document_type"]
|
405
427
|
===== `document_type`
|
@@ -408,8 +430,10 @@ The document ID for the index. Useful for overwriting existing entries in Elasti
|
|
408
430
|
* There is no default value for this setting.
|
409
431
|
* This option is deprecated
|
410
432
|
|
411
|
-
NOTE: This option is deprecated due to the
|
412
|
-
|
433
|
+
NOTE: This option is deprecated due to the
|
434
|
+
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal
|
435
|
+
of types in Elasticsearch 6.0]. It will be removed in the next major version of
|
436
|
+
Logstash.
|
413
437
|
|
414
438
|
NOTE: This value is ignored and has no effect for Elasticsearch clusters `8.x`.
|
415
439
|
|
@@ -433,9 +457,10 @@ If you don't set a value for this option:
|
|
433
457
|
** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
|
434
458
|
** Otherwise, the default value is `disabled`.
|
435
459
|
|
436
|
-
Controls this plugin's compatibility with the
|
437
|
-
|
438
|
-
|
460
|
+
Controls this plugin's compatibility with the
|
461
|
+
https://www.elastic.co/guide/en/ecs/current/index.html[Elastic Common Schema
|
462
|
+
(ECS)], including the installation of ECS-compatible index templates. The value
|
463
|
+
of this setting affects the _default_ values of:
|
439
464
|
|
440
465
|
* <<plugins-{type}s-{plugin}-index>>
|
441
466
|
* <<plugins-{type}s-{plugin}-template_name>>
|
@@ -479,8 +504,10 @@ If you have custom firewall rules you may need to change this
|
|
479
504
|
* Value type is <<uri,uri>>
|
480
505
|
* Default value is `[//127.0.0.1]`
|
481
506
|
|
482
|
-
Sets the host(s) of the remote instance. If given an array it will load balance
|
483
|
-
|
507
|
+
Sets the host(s) of the remote instance. If given an array it will load balance
|
508
|
+
requests across the hosts specified in the `hosts` parameter. Remember the
|
509
|
+
`http` protocol uses the {ref}/modules-http.html#modules-http[http] address (eg.
|
510
|
+
9200, not 9300).
|
484
511
|
|
485
512
|
Examples:
|
486
513
|
|
@@ -490,11 +517,9 @@ Examples:
|
|
490
517
|
`["https://127.0.0.1:9200"]`
|
491
518
|
`["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)
|
492
519
|
|
493
|
-
Exclude
|
494
|
-
|
495
|
-
|
496
|
-
requests to the master nodes. This parameter should reference only data or
|
497
|
-
client nodes in Elasticsearch.
|
520
|
+
Exclude {ref}/modules-node.html[dedicated master nodes] from the `hosts` list to
|
521
|
+
prevent Logstash from sending bulk requests to the master nodes. This parameter
|
522
|
+
should reference only data or client nodes in Elasticsearch.
|
498
523
|
|
499
524
|
Any special characters present in the URLs here MUST be URL escaped! This means
|
500
525
|
`#` should be put in as `%23` for instance.
|
@@ -505,7 +530,8 @@ Any special characters present in the URLs here MUST be URL escaped! This means
|
|
505
530
|
* Value type is <<boolean,boolean>>
|
506
531
|
* Default value is `false`
|
507
532
|
|
508
|
-
Enable gzip compression on requests. Note that response compression is on by
|
533
|
+
Enable gzip compression on requests. Note that response compression is on by
|
534
|
+
default for Elasticsearch v5.0 and beyond
|
509
535
|
|
510
536
|
[id="plugins-{type}s-{plugin}-ilm_enabled"]
|
511
537
|
===== `ilm_enabled`
|
@@ -513,14 +539,19 @@ Enable gzip compression on requests. Note that response compression is on by def
|
|
513
539
|
* Value can be any of: `true`, `false`, `auto`
|
514
540
|
* Default value is `auto`
|
515
541
|
|
516
|
-
The default setting of `auto` will automatically enable
|
517
|
-
|
542
|
+
The default setting of `auto` will automatically enable
|
543
|
+
{ref}/index-lifecycle-management.html[Index Lifecycle Management], if the
|
544
|
+
Elasticsearch cluster is running Elasticsearch version `7.0.0` or higher with
|
545
|
+
the ILM feature enabled, and disable it otherwise.
|
518
546
|
|
519
|
-
Setting this flag to `false` will disable the Index Lifecycle Management
|
520
|
-
|
521
|
-
to enable Index Lifecycle Management
|
547
|
+
Setting this flag to `false` will disable the Index Lifecycle Management
|
548
|
+
feature, even if the Elasticsearch cluster supports ILM.
|
549
|
+
Setting this flag to `true` will enable Index Lifecycle Management feature, if
|
550
|
+
the Elasticsearch cluster supports it. This is required to enable Index
|
551
|
+
Lifecycle Management on a version of Elasticsearch earlier than version `7.0.0`.
|
522
552
|
|
523
|
-
NOTE: This feature requires a Basic License or above to be installed on an
|
553
|
+
NOTE: This feature requires a Basic License or above to be installed on an
|
554
|
+
Elasticsearch cluster version 6.6.0 or later.
|
524
555
|
|
525
556
|
[id="plugins-{type}s-{plugin}-ilm_pattern"]
|
526
557
|
===== `ilm_pattern`
|
@@ -528,17 +559,24 @@ NOTE: This feature requires a Basic License or above to be installed on an Elast
|
|
528
559
|
* Value type is <<string,string>>
|
529
560
|
* Default value is `{now/d}-000001`
|
530
561
|
|
531
|
-
Pattern used for generating indices managed by
|
532
|
-
|
562
|
+
Pattern used for generating indices managed by
|
563
|
+
{ref}/index-lifecycle-management.html[Index Lifecycle Management]. The value
|
564
|
+
specified in the pattern will be appended to the write alias, and incremented
|
565
|
+
automatically when a new index is created by ILM.
|
533
566
|
|
534
|
-
Date Math can be used when specifying an ilm pattern, see
|
567
|
+
Date Math can be used when specifying an ilm pattern, see
|
568
|
+
{ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
|
569
|
+
API docs] for details.
|
535
570
|
|
536
|
-
NOTE: Updating the pattern will require the index template to be rewritten
|
571
|
+
NOTE: Updating the pattern will require the index template to be rewritten.
|
537
572
|
|
538
|
-
NOTE: The pattern must finish with a dash and a number that will be automatically
|
573
|
+
NOTE: The pattern must finish with a dash and a number that will be automatically
|
574
|
+
incremented when indices rollover.
|
539
575
|
|
540
|
-
NOTE: The pattern is a 6-digit string padded by zeros, regardless of prior index name.
|
541
|
-
|
576
|
+
NOTE: The pattern is a 6-digit string padded by zeros, regardless of prior index name.
|
577
|
+
Example: 000001. See
|
578
|
+
{ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path
|
579
|
+
parameters API docs] for details.
|
542
580
|
|
543
581
|
[id="plugins-{type}s-{plugin}-ilm_policy"]
|
544
582
|
===== `ilm_policy`
|
@@ -546,10 +584,12 @@ See {ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover pa
|
|
546
584
|
* Value type is <<string,string>>
|
547
585
|
* Default value is `logstash-policy`
|
548
586
|
|
549
|
-
Modify this setting to use a custom Index Lifecycle Management policy, rather
|
550
|
-
|
587
|
+
Modify this setting to use a custom Index Lifecycle Management policy, rather
|
588
|
+
than the default. If this value is not set, the default policy will be
|
589
|
+
automatically installed into Elasticsearch
|
551
590
|
|
552
|
-
NOTE: If this setting is specified, the policy must already exist in Elasticsearch
|
591
|
+
NOTE: If this setting is specified, the policy must already exist in Elasticsearch
|
592
|
+
cluster.
|
553
593
|
|
554
594
|
[id="plugins-{type}s-{plugin}-ilm_rollover_alias"]
|
555
595
|
===== `ilm_rollover_alias`
|
@@ -559,13 +599,17 @@ NOTE: If this setting is specified, the policy must already exist in Elasticsear
|
|
559
599
|
** ECS Compatibility disabled: `logstash`
|
560
600
|
** ECS Compatibility enabled: `ecs-logstash`
|
561
601
|
|
562
|
-
The rollover alias is the alias where indices managed using Index Lifecycle
|
602
|
+
The rollover alias is the alias where indices managed using Index Lifecycle
|
603
|
+
Management will be written to.
|
563
604
|
|
564
|
-
NOTE: If both `index` and `ilm_rollover_alias` are specified,
|
605
|
+
NOTE: If both `index` and `ilm_rollover_alias` are specified,
|
606
|
+
`ilm_rollover_alias` takes precedence.
|
565
607
|
|
566
|
-
NOTE: Updating the rollover alias will require the index template to be
|
608
|
+
NOTE: Updating the rollover alias will require the index template to be
|
609
|
+
rewritten.
|
567
610
|
|
568
|
-
NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as
|
611
|
+
NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as
|
612
|
+
`index` does.
|
569
613
|
|
570
614
|
[id="plugins-{type}s-{plugin}-index"]
|
571
615
|
===== `index`
|
@@ -580,8 +624,10 @@ The default value will partition your indices by day so you can more easily
|
|
580
624
|
delete old data or only search specific date ranges.
|
581
625
|
Indexes may not contain uppercase characters.
|
582
626
|
For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}.
|
583
|
-
|
584
|
-
|
627
|
+
Logstash uses
|
628
|
+
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[Joda
|
629
|
+
formats] for the index pattern from event timestamp.
|
630
|
+
|
585
631
|
|
586
632
|
[id="plugins-{type}s-{plugin}-keystore"]
|
587
633
|
===== `keystore`
|
@@ -653,8 +699,9 @@ Password to authenticate to a secure Elasticsearch cluster
|
|
653
699
|
* Value type is <<string,string>>
|
654
700
|
* There is no default value for this setting.
|
655
701
|
|
656
|
-
HTTP Path at which the Elasticsearch server lives. Use this if you must run
|
657
|
-
the root path for the Elasticsearch
|
702
|
+
HTTP Path at which the Elasticsearch server lives. Use this if you must run
|
703
|
+
Elasticsearch behind a proxy that remaps the root path for the Elasticsearch
|
704
|
+
HTTP API lives.
|
658
705
|
Note that if you use paths as components of URLs in the 'hosts' field you may
|
659
706
|
not also set this field. That will raise an error at startup
|
660
707
|
|
@@ -664,8 +711,10 @@ not also set this field. That will raise an error at startup
|
|
664
711
|
* Value type is <<string,string>>
|
665
712
|
* Default value is `nil`
|
666
713
|
|
667
|
-
Set which ingest pipeline you wish to execute for an event. You can also use
|
668
|
-
|
714
|
+
Set which ingest pipeline you wish to execute for an event. You can also use
|
715
|
+
event dependent configuration here like `pipeline =>
|
716
|
+
"%{[@metadata][pipeline]}"`. The pipeline parameter won't be set if the value
|
717
|
+
resolves to empty string ("").
|
669
718
|
|
670
719
|
[id="plugins-{type}s-{plugin}-pool_max"]
|
671
720
|
===== `pool_max`
|
@@ -716,7 +765,8 @@ to see if they have come back to life
|
|
716
765
|
* Value type is <<number,number>>
|
717
766
|
* Default value is `2`
|
718
767
|
|
719
|
-
Set initial interval in seconds between bulk retries. Doubled on each retry up
|
768
|
+
Set initial interval in seconds between bulk retries. Doubled on each retry up
|
769
|
+
to `retry_max_interval`
|
720
770
|
|
721
771
|
[id="plugins-{type}s-{plugin}-retry_max_interval"]
|
722
772
|
===== `retry_max_interval`
|
@@ -765,8 +815,9 @@ Example:
|
|
765
815
|
* Value type is <<string,string>>
|
766
816
|
* Default value is `"painless"`
|
767
817
|
|
768
|
-
Set the language of the used script.
|
769
|
-
When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set
|
818
|
+
Set the language of the used script.
|
819
|
+
When using indexed (stored) scripts on Elasticsearch 6.0 and higher, you must set
|
820
|
+
this parameter to `""` (empty string).
|
770
821
|
|
771
822
|
[id="plugins-{type}s-{plugin}-script_type"]
|
772
823
|
===== `script_type`
|
@@ -801,9 +852,10 @@ if enabled, script is in charge of creating non-existent document (scripted upda
|
|
801
852
|
* Value type is <<boolean,boolean>>
|
802
853
|
* Default value is `false`
|
803
854
|
|
804
|
-
This setting asks Elasticsearch for the list of all cluster nodes and adds them
|
805
|
-
|
806
|
-
For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
|
855
|
+
This setting asks Elasticsearch for the list of all cluster nodes and adds them
|
856
|
+
to the hosts list.
|
857
|
+
For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
|
858
|
+
be added to the hosts list, excluding master-only nodes.
|
807
859
|
|
808
860
|
[id="plugins-{type}s-{plugin}-sniffing_delay"]
|
809
861
|
===== `sniffing_delay`
|
@@ -830,9 +882,11 @@ do not use full URL here, only paths, e.g. "/sniff/_nodes/http"
|
|
830
882
|
* Value type is <<boolean,boolean>>
|
831
883
|
* There is no default value for this setting.
|
832
884
|
|
833
|
-
Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this
|
834
|
-
is specified in the URLs listed in 'hosts'.
|
835
|
-
If
|
885
|
+
Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this
|
886
|
+
unspecified will use whatever scheme is specified in the URLs listed in 'hosts'.
|
887
|
+
If no explicit protocol is specified plain HTTP will be used. If SSL is
|
888
|
+
explicitly disabled here the plugin will refuse to start if an HTTPS URL is
|
889
|
+
given in 'hosts'
|
836
890
|
|
837
891
|
[id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
|
838
892
|
===== `ssl_certificate_verification`
|
@@ -937,14 +991,16 @@ Username to authenticate to a secure Elasticsearch cluster
|
|
937
991
|
* Value type is <<number,number>>
|
938
992
|
* Default value is `10000`
|
939
993
|
|
940
|
-
How long to wait before checking
|
941
|
-
|
942
|
-
|
943
|
-
|
944
|
-
|
945
|
-
|
946
|
-
|
947
|
-
|
994
|
+
How long to wait before checking for a stale connection to determine if a keepalive request is needed.
|
995
|
+
Consider setting this value lower than the default, possibly to 0, if you get connection errors regularly.
|
996
|
+
|
997
|
+
This client is based on Apache Commons. Here's how the
|
998
|
+
https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[Apache
|
999
|
+
Commons documentation] describes this option: "Defines period of inactivity in
|
1000
|
+
milliseconds after which persistent connections must be re-validated prior to
|
1001
|
+
being leased to the consumer. Non-positive value passed to this method disables
|
1002
|
+
connection validation. This check helps detect connections that have become
|
1003
|
+
stale (half-closed) while kept inactive in the pool."
|
948
1004
|
|
949
1005
|
[id="plugins-{type}s-{plugin}-version"]
|
950
1006
|
===== `version`
|
@@ -952,8 +1008,10 @@ See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache
|
|
952
1008
|
* Value type is <<string,string>>
|
953
1009
|
* There is no default value for this setting.
|
954
1010
|
|
955
|
-
The version to use for indexing. Use sprintf syntax like `%{my_version}` to use
|
956
|
-
See
|
1011
|
+
The version to use for indexing. Use sprintf syntax like `%{my_version}` to use
|
1012
|
+
a field value here. See the
|
1013
|
+
https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
|
1014
|
+
blog] for more information.
|
957
1015
|
|
958
1016
|
[id="plugins-{type}s-{plugin}-version_type"]
|
959
1017
|
===== `version_type`
|
@@ -961,10 +1019,10 @@ See https://www.elastic.co/blog/elasticsearch-versioning-support.
|
|
961
1019
|
* Value can be any of: `internal`, `external`, `external_gt`, `external_gte`, `force`
|
962
1020
|
* There is no default value for this setting.
|
963
1021
|
|
964
|
-
The version_type to use for indexing.
|
965
|
-
|
966
|
-
|
967
|
-
|
1022
|
+
The version_type to use for indexing. See the
|
1023
|
+
https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
|
1024
|
+
blog] and {ref}/docs-index_.html#_version_types[Version types] in the
|
1025
|
+
Elasticsearch documentation.
|
968
1026
|
|
969
1027
|
|
970
1028
|
[id="plugins-{type}s-{plugin}-common-options"]
|