logstash-output-elasticsearch 10.6.0-java → 10.8.0-java

Sign up to get free protection for your applications and to get access to all the features.
Files changed (27) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +24 -1
  3. data/CONTRIBUTORS +1 -0
  4. data/docs/index.asciidoc +97 -78
  5. data/lib/logstash/outputs/elasticsearch.rb +300 -165
  6. data/lib/logstash/outputs/elasticsearch/http_client.rb +7 -2
  7. data/lib/logstash/outputs/elasticsearch/http_client/pool.rb +13 -28
  8. data/lib/logstash/outputs/elasticsearch/http_client_builder.rb +1 -0
  9. data/lib/logstash/outputs/elasticsearch/ilm.rb +9 -5
  10. data/lib/logstash/outputs/elasticsearch/license_checker.rb +47 -0
  11. data/lib/logstash/outputs/elasticsearch/template_manager.rb +8 -3
  12. data/lib/logstash/outputs/elasticsearch/templates/ecs-disabled/elasticsearch-8x.json +39 -33
  13. data/lib/logstash/plugin_mixins/elasticsearch/api_configs.rb +163 -0
  14. data/lib/logstash/{outputs → plugin_mixins}/elasticsearch/common.rb +40 -154
  15. data/lib/logstash/plugin_mixins/elasticsearch/noop_license_checker.rb +9 -0
  16. data/logstash-output-elasticsearch.gemspec +1 -1
  17. data/spec/es_spec_helper.rb +32 -12
  18. data/spec/fixtures/template-with-policy-es8x.json +50 -0
  19. data/spec/integration/outputs/ilm_spec.rb +34 -20
  20. data/spec/integration/outputs/metrics_spec.rb +1 -5
  21. data/spec/unit/outputs/elasticsearch/http_client/pool_spec.rb +45 -5
  22. data/spec/unit/outputs/elasticsearch/http_client_spec.rb +22 -0
  23. data/spec/unit/outputs/elasticsearch/template_manager_spec.rb +31 -0
  24. data/spec/unit/outputs/elasticsearch_spec.rb +22 -0
  25. data/spec/unit/outputs/license_check_spec.rb +41 -0
  26. metadata +10 -4
  27. data/lib/logstash/outputs/elasticsearch/common_configs.rb +0 -167
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 69cd4a7aaa5a5b66bc94d633f1e482f472a557a464f0077e1382fe3b8d8bf6a4
4
- data.tar.gz: 3b08e4236f1ba8d580a4f35b528400a3c3a3e0b33123ca9e5d0b983cdb6a256d
3
+ metadata.gz: 29d2b5e316c2ba3dd1d67af673733bd90dd0b5b82c436b1f107cc11d681bd8b5
4
+ data.tar.gz: 102c70fde50016c59245287f09eeed0d51d4fb411a617b7d397ab306688f022d
5
5
  SHA512:
6
- metadata.gz: 5acda5b8b7a8654f0a8ffd9c99483f25eb7076184d018a76615e7d88861a45c898a6f1d20a80396791f8e6ec53746d716705ebde4db7d60b258b17941bb073b9
7
- data.tar.gz: 9927d71a573cf5aee344d8d8e0f8c7808b4e98769a1a35ae83ac25186439326f9ffa9910c676aec567607da3b9cb8bb26570fb5dd662c1ca62f648288230acb5
6
+ metadata.gz: 6bd3b56c08debf7408ac1cd31980235d3224b5705a42ecc78d28e8da94825cd310ee1b8d0f47b16400327bcc6b0c1e777e2a937a09b1a5a483584d6ca523317d
7
+ data.tar.gz: c9ba096fb1a39064616ef10be681d1d3b64ab344a7b3b51560b68391c525cf61c6c98a289e068958e43806e7042d9a7ced140ec29407f80680e0323ba44ce165
@@ -1,5 +1,28 @@
1
+ ## 10.8.0
2
+ - Refactored configuration options into specific and shared in PluginMixins namespace [#973](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/973)
3
+ - Refactored common methods into specific and shared in PluginMixins namespace [#976](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/976)
4
+
5
+ ## 10.7.3
6
+ - Added composable index template support for elasticsearch version 8 [#980](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/980)
7
+
8
+ ## 10.7.2
9
+ - [DOC] Fixed links to restructured Logstash-to-cloud docs [#975](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/975)
10
+
11
+ ## 10.7.1
12
+ - [DOC] Document the permissions required in secured clusters [#969](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/969)
13
+
14
+ ## 10.7.0
15
+ - Changed: don't set the pipeline parameter if the value resolves to an empty string [#962](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/962)
16
+
17
+ ## 10.6.2
18
+ - [DOC] Added clarifying info on http compression settings and behaviors [#943](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/943)
19
+ - [DOC] Fixed entry for ilm_policy default value[#956](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/956)
20
+
21
+ ## 10.6.1
22
+ - Fixed an issue introduced in 10.6.0 that broke Logstash Core's monitoring feature when this plugin is run in Logstash 7.7-7.8. [#953](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/953)
23
+
1
24
  ## 10.6.0
2
- - Added `ecs_compatiblity` mode, for managing ECS-compatable templates [#952](https://github.com/logstash-plugins/logstash-output-elasticsearch/issue/952)
25
+ - Added `ecs_compatiblity` mode, for managing ECS-compatable templates [#952](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/952)
3
26
 
4
27
  ## 10.5.1
5
28
  - [DOC] Removed outdated compatibility notices, reworked cloud notice, and fixed formatting for `hosts` examples [#938](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/938)
@@ -26,6 +26,7 @@ Contributors:
26
26
  * Tom Hodder (tolland)
27
27
  * jimmyjones2
28
28
  * Gabriel Moskovicz (gmoskovicz)
29
+ * Luca Belluccini (lucabelluccini)
29
30
 
30
31
  Note: If you've sent us patches, bug reports, or otherwise contributed to
31
32
  Logstash, and you aren't on the list above and want to be, please let us know
@@ -23,14 +23,14 @@ include::{include_path}/plugin_header.asciidoc[]
23
23
 
24
24
  If you plan to use the Kibana web interface to analyze data transformed by
25
25
  Logstash, use the Elasticsearch output plugin to get your data into
26
- Elasticsearch.
26
+ Elasticsearch.
27
27
 
28
28
  This output only speaks the HTTP protocol as it is the preferred protocol for
29
29
  interacting with Elasticsearch. In previous versions it was possible to
30
30
  communicate with Elasticsearch through the transport protocol, which is now
31
31
  reserved for internal cluster communication between nodes
32
32
  https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html[communication between nodes].
33
- Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol]
33
+ Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol]
34
34
  to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and
35
35
  will be removed in 8.0.0
36
36
 
@@ -53,10 +53,7 @@ connecting to Elasticsearch 7.x.
53
53
 
54
54
  ===== Hosted {es} Service on Elastic Cloud
55
55
 
56
- You can run Elasticsearch on your own hardware, or use our
57
- https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on
58
- Elastic Cloud. The Elasticsearch Service is available on AWS, Google Cloud
59
- Platform, and Microsoft Azure. {ess-trial}[Try the {es} Service for free].
56
+ {ess-leadin}
60
57
 
61
58
  ==== Compatibility with the Elastic Common Schema (ECS)
62
59
 
@@ -76,7 +73,7 @@ and will correctly reject events with fields that conflict and cannot be coerced
76
73
 
77
74
  [NOTE]
78
75
  ================================================================================
79
- You cannot use dynamic variable substitution when `ilm_enabled` is `true` and
76
+ You cannot use dynamic variable substitution when `ilm_enabled` is `true` and
80
77
  when using `ilm_rollover_alias`.
81
78
 
82
79
  ================================================================================
@@ -91,7 +88,10 @@ Each Elasticsearch output is a new client connected to the cluster:
91
88
  * it has to initialize the client and connect to Elasticsearch (restart time is longer if you have more clients)
92
89
  * it has an associated connection pool
93
90
 
94
- In order to minimize the number of open connections to Elasticsearch, maximize the bulk size and reduce the number of "small" bulk requests (which could easily fill up the queue), it is usually more efficient to have a single Elasticsearch output.
91
+ In order to minimize the number of open connections to Elasticsearch, maximize
92
+ the bulk size and reduce the number of "small" bulk requests (which could easily
93
+ fill up the queue), it is usually more efficient to have a single Elasticsearch
94
+ output.
95
95
 
96
96
  Example:
97
97
  [source,ruby]
@@ -100,11 +100,14 @@ Example:
100
100
  index => "%{[some_field][sub_field]}-%{+YYYY.MM.dd}"
101
101
  }
102
102
  }
103
-
103
+
104
104
  **What to do in case there is no field in the event containing the destination index prefix?**
105
105
 
106
- You can use the `mutate` filter and conditionals to add a `[@metadata]` field (see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata) to set
107
- the destination index for each event. The `[@metadata]` fields will not be sent to Elasticsearch.
106
+ You can use the `mutate` filter and conditionals to add a `[@metadata]` field
107
+ (see
108
+ https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata)
109
+ to set the destination index for each event. The `[@metadata]` fields will not
110
+ be sent to Elasticsearch.
108
111
 
109
112
  Example:
110
113
  [source,ruby]
@@ -136,7 +139,7 @@ HTTP requests to the bulk API are expected to return a 200 response code. All ot
136
139
  The following document errors are handled as follows:
137
140
 
138
141
  * 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See <<plugins-{type}s-{plugin}-dlq-policy>> for more info.
139
- * 409 errors (conflict) are logged as a warning and dropped.
142
+ * 409 errors (conflict) are logged as a warning and dropped.
140
143
 
141
144
  Note that 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
142
145
  It is more performant for Elasticsearch to retry these exceptions than this plugin.
@@ -218,18 +221,29 @@ not reevaluate its DNS value while the keepalive is in effect.
218
221
 
219
222
  ==== HTTP Compression
220
223
 
221
- This plugin supports request and response compression. Response compression is enabled by default and
222
- for Elasticsearch versions 5.0 and later, the user doesn't have to set any configs in Elasticsearch for
223
- it to send back compressed response. For versions before 5.0, `http.compression` must be set to `true` https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[in
224
- Elasticsearch] to take advantage of response compression when using this plugin
224
+ This plugin supports request and response compression. Response compression is
225
+ enabled by default for HTTP and for Elasticsearch versions 5.0 and later.
226
+
227
+ You don't have to set any configs in Elasticsearch for it to send back a
228
+ compressed response. For versions before 5.0, or if HTTPS is enabled,
229
+ `http.compression` must be set to `true`
230
+ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[in
231
+ Elasticsearch] to take advantage of response compression when using this plugin.
225
232
 
226
- For requests compression, regardless of the Elasticsearch version, users have to enable `http_compression`
227
- setting in their Logstash config file.
233
+ For requests compression, regardless of the Elasticsearch version, enable the
234
+ `http_compression` setting in the Logstash config file.
228
235
 
229
236
  ==== Authentication
230
237
 
231
238
  Authentication to a secure Elasticsearch cluster is possible using one of the `user`/`password`, `cloud_auth` or `api_key` options.
232
239
 
240
+ [id="plugins-{type}s-{plugin}-autz"]
241
+ ==== Authorization
242
+
243
+ Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
244
+ The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.
245
+
246
+
233
247
  [id="plugins-{type}s-{plugin}-options"]
234
248
  ==== Elasticsearch Output Configuration Options
235
249
 
@@ -303,7 +317,7 @@ output plugins.
303
317
  &nbsp;
304
318
 
305
319
  [id="plugins-{type}s-{plugin}-action"]
306
- ===== `action`
320
+ ===== `action`
307
321
 
308
322
  * Value type is <<string,string>>
309
323
  * Default value is `"index"`
@@ -334,7 +348,7 @@ Authenticate using Elasticsearch API key. Note that this option also requires en
334
348
  Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key API].
335
349
 
336
350
  [id="plugins-{type}s-{plugin}-bulk_path"]
337
- ===== `bulk_path`
351
+ ===== `bulk_path`
338
352
 
339
353
  * Value type is <<string,string>>
340
354
  * There is no default value for this setting.
@@ -343,7 +357,7 @@ HTTP Path to perform the _bulk requests to
343
357
  this defaults to a concatenation of the path parameter and "_bulk"
344
358
 
345
359
  [id="plugins-{type}s-{plugin}-cacert"]
346
- ===== `cacert`
360
+ ===== `cacert`
347
361
 
348
362
  * Value type is <<path,path>>
349
363
  * There is no default value for this setting.
@@ -358,7 +372,7 @@ The .cer or .pem file to validate the server's certificate
358
372
 
359
373
  Cloud authentication string ("<username>:<password>" format) is an alternative for the `user`/`password` pair.
360
374
 
361
- For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_auth[Logstash-to-Cloud documentation]
375
+ For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html[Logstash-to-Cloud documentation]
362
376
 
363
377
  [id="plugins-{type}s-{plugin}-cloud_id"]
364
378
  ===== `cloud_id`
@@ -368,10 +382,10 @@ For more details, check out the https://www.elastic.co/guide/en/logstash/current
368
382
 
369
383
  Cloud ID, from the Elastic Cloud web console. If set `hosts` should not be used.
370
384
 
371
- For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_id[Logstash-to-Cloud documentation]
385
+ For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html[Logstash-to-Cloud documentation]
372
386
 
373
387
  [id="plugins-{type}s-{plugin}-doc_as_upsert"]
374
- ===== `doc_as_upsert`
388
+ ===== `doc_as_upsert`
375
389
 
376
390
  * Value type is <<boolean,boolean>>
377
391
  * Default value is `false`
@@ -380,7 +394,7 @@ Enable `doc_as_upsert` for update mode.
380
394
  Create a new document with source if `document_id` doesn't exist in Elasticsearch
381
395
 
382
396
  [id="plugins-{type}s-{plugin}-document_id"]
383
- ===== `document_id`
397
+ ===== `document_id`
384
398
 
385
399
  * Value type is <<string,string>>
386
400
  * There is no default value for this setting.
@@ -388,7 +402,7 @@ Create a new document with source if `document_id` doesn't exist in Elasticsearc
388
402
  The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.
389
403
 
390
404
  [id="plugins-{type}s-{plugin}-document_type"]
391
- ===== `document_type`
405
+ ===== `document_type`
392
406
 
393
407
  * Value type is <<string,string>>
394
408
  * There is no default value for this setting.
@@ -419,7 +433,7 @@ If you don't set a value for this option:
419
433
  ** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
420
434
  ** Otherwise, the default value is `disabled`.
421
435
 
422
- Controls this plugin's compatibility with the {ecs-ref}}[Elastic Common Schema (ECS)],
436
+ Controls this plugin's compatibility with the https://www.elastic.co/guide/en/ecs/current/index.html[Elastic Common Schema (ECS)],
423
437
  including the installation of ECS-compatible index templates.
424
438
  The value of this setting affects the _default_ values of:
425
439
 
@@ -428,7 +442,7 @@ The value of this setting affects the _default_ values of:
428
442
  * <<plugins-{type}s-{plugin}-ilm_rollover_alias>>
429
443
 
430
444
  [id="plugins-{type}s-{plugin}-failure_type_logging_whitelist"]
431
- ===== `failure_type_logging_whitelist`
445
+ ===== `failure_type_logging_whitelist`
432
446
 
433
447
  * Value type is <<array,array>>
434
448
  * Default value is `[]`
@@ -449,7 +463,7 @@ an elasticsearch node. The headers will be used for any kind of request
449
463
  These custom headers will be overidden by settings like `http_compression`.
450
464
 
451
465
  [id="plugins-{type}s-{plugin}-healthcheck_path"]
452
- ===== `healthcheck_path`
466
+ ===== `healthcheck_path`
453
467
 
454
468
  * Value type is <<string,string>>
455
469
  * There is no default value for this setting.
@@ -460,7 +474,7 @@ before it is once again eligible to service requests.
460
474
  If you have custom firewall rules you may need to change this
461
475
 
462
476
  [id="plugins-{type}s-{plugin}-hosts"]
463
- ===== `hosts`
477
+ ===== `hosts`
464
478
 
465
479
  * Value type is <<uri,uri>>
466
480
  * Default value is `[//127.0.0.1]`
@@ -476,13 +490,17 @@ Examples:
476
490
  `["https://127.0.0.1:9200"]`
477
491
  `["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)
478
492
 
479
- It is important to exclude http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated master nodes] from the `hosts` list
480
- to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.
493
+ Exclude
494
+ http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated
495
+ master nodes] from the `hosts` list to prevent Logstash from sending bulk
496
+ requests to the master nodes. This parameter should reference only data or
497
+ client nodes in Elasticsearch.
481
498
 
482
- Any special characters present in the URLs here MUST be URL escaped! This means `#` should be put in as `%23` for instance.
499
+ Any special characters present in the URLs here MUST be URL escaped! This means
500
+ `#` should be put in as `%23` for instance.
483
501
 
484
502
  [id="plugins-{type}s-{plugin}-http_compression"]
485
- ===== `http_compression`
503
+ ===== `http_compression`
486
504
 
487
505
  * Value type is <<boolean,boolean>>
488
506
  * Default value is `false`
@@ -519,11 +537,14 @@ NOTE: Updating the pattern will require the index template to be rewritten
519
537
 
520
538
  NOTE: The pattern must finish with a dash and a number that will be automatically incremented when indices rollover.
521
539
 
540
+ NOTE: The pattern is a 6-digit string padded by zeros, regardless of prior index name. Example: 000001.
541
+ See {ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path parameters API docs] for details.
542
+
522
543
  [id="plugins-{type}s-{plugin}-ilm_policy"]
523
544
  ===== `ilm_policy`
524
545
 
525
546
  * Value type is <<string,string>>
526
- * Default value is `logstash`
547
+ * Default value is `logstash-policy`
527
548
 
528
549
  Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy will
529
550
  be automatically installed into Elasticsearch
@@ -547,7 +568,7 @@ NOTE: Updating the rollover alias will require the index template to be rewritte
547
568
  NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as `index` does.
548
569
 
549
570
  [id="plugins-{type}s-{plugin}-index"]
550
- ===== `index`
571
+ ===== `index`
551
572
 
552
573
  * Value type is <<string,string>>
553
574
  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
@@ -563,7 +584,7 @@ LS uses Joda to format the index pattern from event timestamp.
563
584
  Joda formats are defined http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[here].
564
585
 
565
586
  [id="plugins-{type}s-{plugin}-keystore"]
566
- ===== `keystore`
587
+ ===== `keystore`
567
588
 
568
589
  * Value type is <<path,path>>
569
590
  * There is no default value for this setting.
@@ -572,7 +593,7 @@ The keystore used to present a certificate to the server.
572
593
  It can be either .jks or .p12
573
594
 
574
595
  [id="plugins-{type}s-{plugin}-keystore_password"]
575
- ===== `keystore_password`
596
+ ===== `keystore_password`
576
597
 
577
598
  * Value type is <<password,password>>
578
599
  * There is no default value for this setting.
@@ -580,7 +601,7 @@ It can be either .jks or .p12
580
601
  Set the keystore password
581
602
 
582
603
  [id="plugins-{type}s-{plugin}-manage_template"]
583
- ===== `manage_template`
604
+ ===== `manage_template`
584
605
 
585
606
  * Value type is <<boolean,boolean>>
586
607
  * Default value is `true`
@@ -600,7 +621,7 @@ field names) you should set `manage_template` to false and use the REST
600
621
  API to apply your templates manually.
601
622
 
602
623
  [id="plugins-{type}s-{plugin}-parameters"]
603
- ===== `parameters`
624
+ ===== `parameters`
604
625
 
605
626
  * Value type is <<hash,hash>>
606
627
  * There is no default value for this setting.
@@ -610,7 +631,7 @@ to every host listed in the 'hosts' configuration. If the 'hosts' list contains
610
631
  urls that already have query strings, the one specified here will be appended.
611
632
 
612
633
  [id="plugins-{type}s-{plugin}-parent"]
613
- ===== `parent`
634
+ ===== `parent`
614
635
 
615
636
  * Value type is <<string,string>>
616
637
  * Default value is `nil`
@@ -619,7 +640,7 @@ For child documents, ID of the associated parent.
619
640
  This can be dynamic using the `%{foo}` syntax.
620
641
 
621
642
  [id="plugins-{type}s-{plugin}-password"]
622
- ===== `password`
643
+ ===== `password`
623
644
 
624
645
  * Value type is <<password,password>>
625
646
  * There is no default value for this setting.
@@ -627,7 +648,7 @@ This can be dynamic using the `%{foo}` syntax.
627
648
  Password to authenticate to a secure Elasticsearch cluster
628
649
 
629
650
  [id="plugins-{type}s-{plugin}-path"]
630
- ===== `path`
651
+ ===== `path`
631
652
 
632
653
  * Value type is <<string,string>>
633
654
  * There is no default value for this setting.
@@ -638,16 +659,16 @@ Note that if you use paths as components of URLs in the 'hosts' field you may
638
659
  not also set this field. That will raise an error at startup
639
660
 
640
661
  [id="plugins-{type}s-{plugin}-pipeline"]
641
- ===== `pipeline`
662
+ ===== `pipeline`
642
663
 
643
664
  * Value type is <<string,string>>
644
665
  * Default value is `nil`
645
666
 
646
- Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration
647
- here like `pipeline => "%{INGEST_PIPELINE}"`
667
+ Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration here
668
+ like `pipeline => "%{[@metadata][pipeline]}"`. The pipeline parameter won't be set if the value resolves to empty string ("").
648
669
 
649
670
  [id="plugins-{type}s-{plugin}-pool_max"]
650
- ===== `pool_max`
671
+ ===== `pool_max`
651
672
 
652
673
  * Value type is <<number,number>>
653
674
  * Default value is `1000`
@@ -658,7 +679,7 @@ Setting this too low may mean frequently closing / opening connections
658
679
  which is bad.
659
680
 
660
681
  [id="plugins-{type}s-{plugin}-pool_max_per_route"]
661
- ===== `pool_max_per_route`
682
+ ===== `pool_max_per_route`
662
683
 
663
684
  * Value type is <<number,number>>
664
685
  * Default value is `100`
@@ -669,7 +690,7 @@ Setting this too low may mean frequently closing / opening connections
669
690
  which is bad.
670
691
 
671
692
  [id="plugins-{type}s-{plugin}-proxy"]
672
- ===== `proxy`
693
+ ===== `proxy`
673
694
 
674
695
  * Value type is <<uri,uri>>
675
696
  * There is no default value for this setting.
@@ -680,7 +701,7 @@ An empty string is treated as if proxy was not set. This is useful when using
680
701
  environment variables e.g. `proxy => '${LS_PROXY:}'`.
681
702
 
682
703
  [id="plugins-{type}s-{plugin}-resurrect_delay"]
683
- ===== `resurrect_delay`
704
+ ===== `resurrect_delay`
684
705
 
685
706
  * Value type is <<number,number>>
686
707
  * Default value is `5`
@@ -690,7 +711,7 @@ Resurrection is the process by which backend endpoints marked 'down' are checked
690
711
  to see if they have come back to life
691
712
 
692
713
  [id="plugins-{type}s-{plugin}-retry_initial_interval"]
693
- ===== `retry_initial_interval`
714
+ ===== `retry_initial_interval`
694
715
 
695
716
  * Value type is <<number,number>>
696
717
  * Default value is `2`
@@ -698,7 +719,7 @@ to see if they have come back to life
698
719
  Set initial interval in seconds between bulk retries. Doubled on each retry up to `retry_max_interval`
699
720
 
700
721
  [id="plugins-{type}s-{plugin}-retry_max_interval"]
701
- ===== `retry_max_interval`
722
+ ===== `retry_max_interval`
702
723
 
703
724
  * Value type is <<number,number>>
704
725
  * Default value is `64`
@@ -706,17 +727,15 @@ Set initial interval in seconds between bulk retries. Doubled on each retry up t
706
727
  Set max interval in seconds between bulk retries.
707
728
 
708
729
  [id="plugins-{type}s-{plugin}-retry_on_conflict"]
709
- ===== `retry_on_conflict`
730
+ ===== `retry_on_conflict`
710
731
 
711
732
  * Value type is <<number,number>>
712
733
  * Default value is `1`
713
734
 
714
- The number of times Elasticsearch should internally retry an update/upserted document
715
- See the https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html[partial updates]
716
- for more info
735
+ The number of times Elasticsearch should internally retry an update/upserted document.
717
736
 
718
737
  [id="plugins-{type}s-{plugin}-routing"]
719
- ===== `routing`
738
+ ===== `routing`
720
739
 
721
740
  * Value type is <<string,string>>
722
741
  * There is no default value for this setting.
@@ -725,7 +744,7 @@ A routing override to be applied to all processed events.
725
744
  This can be dynamic using the `%{foo}` syntax.
726
745
 
727
746
  [id="plugins-{type}s-{plugin}-script"]
728
- ===== `script`
747
+ ===== `script`
729
748
 
730
749
  * Value type is <<string,string>>
731
750
  * Default value is `""`
@@ -741,7 +760,7 @@ Example:
741
760
  }
742
761
 
743
762
  [id="plugins-{type}s-{plugin}-script_lang"]
744
- ===== `script_lang`
763
+ ===== `script_lang`
745
764
 
746
765
  * Value type is <<string,string>>
747
766
  * Default value is `"painless"`
@@ -750,7 +769,7 @@ Set the language of the used script. If not set, this defaults to painless in ES
750
769
  When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to `""` (empty string).
751
770
 
752
771
  [id="plugins-{type}s-{plugin}-script_type"]
753
- ===== `script_type`
772
+ ===== `script_type`
754
773
 
755
774
  * Value can be any of: `inline`, `indexed`, `file`
756
775
  * Default value is `["inline"]`
@@ -761,7 +780,7 @@ Define the type of script referenced by "script" variable
761
780
  file : "script" contains the name of script stored in elasticsearch's config directory
762
781
 
763
782
  [id="plugins-{type}s-{plugin}-script_var_name"]
764
- ===== `script_var_name`
783
+ ===== `script_var_name`
765
784
 
766
785
  * Value type is <<string,string>>
767
786
  * Default value is `"event"`
@@ -769,7 +788,7 @@ Define the type of script referenced by "script" variable
769
788
  Set variable name passed to script (scripted update)
770
789
 
771
790
  [id="plugins-{type}s-{plugin}-scripted_upsert"]
772
- ===== `scripted_upsert`
791
+ ===== `scripted_upsert`
773
792
 
774
793
  * Value type is <<boolean,boolean>>
775
794
  * Default value is `false`
@@ -777,7 +796,7 @@ Set variable name passed to script (scripted update)
777
796
  if enabled, script is in charge of creating non-existent document (scripted update)
778
797
 
779
798
  [id="plugins-{type}s-{plugin}-sniffing"]
780
- ===== `sniffing`
799
+ ===== `sniffing`
781
800
 
782
801
  * Value type is <<boolean,boolean>>
783
802
  * Default value is `false`
@@ -787,7 +806,7 @@ For Elasticsearch 1.x and 2.x any nodes with `http.enabled` (on by default) will
787
806
  For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will be added to the hosts list, excluding master-only nodes.
788
807
 
789
808
  [id="plugins-{type}s-{plugin}-sniffing_delay"]
790
- ===== `sniffing_delay`
809
+ ===== `sniffing_delay`
791
810
 
792
811
  * Value type is <<number,number>>
793
812
  * Default value is `5`
@@ -795,7 +814,7 @@ For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
795
814
  How long to wait, in seconds, between sniffing attempts
796
815
 
797
816
  [id="plugins-{type}s-{plugin}-sniffing_path"]
798
- ===== `sniffing_path`
817
+ ===== `sniffing_path`
799
818
 
800
819
  * Value type is <<string,string>>
801
820
  * There is no default value for this setting.
@@ -806,7 +825,7 @@ if sniffing_path is set it will be used as an absolute path
806
825
  do not use full URL here, only paths, e.g. "/sniff/_nodes/http"
807
826
 
808
827
  [id="plugins-{type}s-{plugin}-ssl"]
809
- ===== `ssl`
828
+ ===== `ssl`
810
829
 
811
830
  * Value type is <<boolean,boolean>>
812
831
  * There is no default value for this setting.
@@ -816,7 +835,7 @@ is specified in the URLs listed in 'hosts'. If no explicit protocol is specified
816
835
  If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts'
817
836
 
818
837
  [id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
819
- ===== `ssl_certificate_verification`
838
+ ===== `ssl_certificate_verification`
820
839
 
821
840
  * Value type is <<boolean,boolean>>
822
841
  * Default value is `true`
@@ -826,7 +845,7 @@ For more information on disabling certificate verification please read
826
845
  https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
827
846
 
828
847
  [id="plugins-{type}s-{plugin}-template"]
829
- ===== `template`
848
+ ===== `template`
830
849
 
831
850
  * Value type is <<path,path>>
832
851
  * There is no default value for this setting.
@@ -835,7 +854,7 @@ You can set the path to your own template here, if you so desire.
835
854
  If not set, the included template will be used.
836
855
 
837
856
  [id="plugins-{type}s-{plugin}-template_name"]
838
- ===== `template_name`
857
+ ===== `template_name`
839
858
 
840
859
  * Value type is <<string,string>>
841
860
  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
@@ -852,7 +871,7 @@ change this, you will need to prune the old template manually, e.g.
852
871
  where `OldTemplateName` is whatever the former setting was.
853
872
 
854
873
  [id="plugins-{type}s-{plugin}-template_overwrite"]
855
- ===== `template_overwrite`
874
+ ===== `template_overwrite`
856
875
 
857
876
  * Value type is <<boolean,boolean>>
858
877
  * Default value is `false`
@@ -869,7 +888,7 @@ template (logstash), setting this to true will make Logstash to overwrite
869
888
  the "logstash" template (i.e. removing all customized settings)
870
889
 
871
890
  [id="plugins-{type}s-{plugin}-timeout"]
872
- ===== `timeout`
891
+ ===== `timeout`
873
892
 
874
893
  * Value type is <<number,number>>
875
894
  * Default value is `60`
@@ -878,7 +897,7 @@ Set the timeout, in seconds, for network operations and requests sent Elasticsea
878
897
  a timeout occurs, the request will be retried.
879
898
 
880
899
  [id="plugins-{type}s-{plugin}-truststore"]
881
- ===== `truststore`
900
+ ===== `truststore`
882
901
 
883
902
  * Value type is <<path,path>>
884
903
  * There is no default value for this setting.
@@ -888,7 +907,7 @@ It can be either .jks or .p12.
888
907
  Use either `:truststore` or `:cacert`.
889
908
 
890
909
  [id="plugins-{type}s-{plugin}-truststore_password"]
891
- ===== `truststore_password`
910
+ ===== `truststore_password`
892
911
 
893
912
  * Value type is <<password,password>>
894
913
  * There is no default value for this setting.
@@ -896,7 +915,7 @@ Use either `:truststore` or `:cacert`.
896
915
  Set the truststore password
897
916
 
898
917
  [id="plugins-{type}s-{plugin}-upsert"]
899
- ===== `upsert`
918
+ ===== `upsert`
900
919
 
901
920
  * Value type is <<string,string>>
902
921
  * Default value is `""`
@@ -905,7 +924,7 @@ Set upsert content for update mode.
905
924
  Create a new document with this parameter as json string if `document_id` doesn't exists
906
925
 
907
926
  [id="plugins-{type}s-{plugin}-user"]
908
- ===== `user`
927
+ ===== `user`
909
928
 
910
929
  * Value type is <<string,string>>
911
930
  * There is no default value for this setting.
@@ -913,7 +932,7 @@ Create a new document with this parameter as json string if `document_id` doesn'
913
932
  Username to authenticate to a secure Elasticsearch cluster
914
933
 
915
934
  [id="plugins-{type}s-{plugin}-validate_after_inactivity"]
916
- ===== `validate_after_inactivity`
935
+ ===== `validate_after_inactivity`
917
936
 
918
937
  * Value type is <<number,number>>
919
938
  * Default value is `10000`
@@ -928,7 +947,7 @@ have become stale (half-closed) while kept inactive in the pool.'
928
947
  See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[these docs for more info]
929
948
 
930
949
  [id="plugins-{type}s-{plugin}-version"]
931
- ===== `version`
950
+ ===== `version`
932
951
 
933
952
  * Value type is <<string,string>>
934
953
  * There is no default value for this setting.
@@ -937,7 +956,7 @@ The version to use for indexing. Use sprintf syntax like `%{my_version}` to use
937
956
  See https://www.elastic.co/blog/elasticsearch-versioning-support.
938
957
 
939
958
  [id="plugins-{type}s-{plugin}-version_type"]
940
- ===== `version_type`
959
+ ===== `version_type`
941
960
 
942
961
  * Value can be any of: `internal`, `external`, `external_gt`, `external_gte`, `force`
943
962
  * There is no default value for this setting.