logstash-output-elasticsearch 8.2.0-java → 8.2.2-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 71b2fb654d86950996c5fae7c523295a1bc9a9cd
-  data.tar.gz: 2dd7cc43347238f9bbfa19d27b49368f6c162dc4
+  metadata.gz: 100a0be14e669078212f2b1a5f11513814819bfd
+  data.tar.gz: 4be944e73c0e885b740f29e211b657dcca3e05f9
 SHA512:
-  metadata.gz: a5d577e3f5bc1b3fb4be51eb896f7b5eaa4ecd66a0b4ebb0ff1f37e5d92b555becad7b1ffcf88356deb0a2ec9162ae45aa4526078925057afb22e26ac3760d5d
-  data.tar.gz: 055311a5f0f991216851381a8f0c0da0480467b84942b64420ecc58d146351a51ab39151dd28a7e89ab12348bdcfde0fa488dfea4769a6cc6ab7c7e015b47d06
+  metadata.gz: 73cd4330d52da5f8a7b30843d554ea95b79c242de425fa8ffae55a5e33b541c6841d70b18d418aeb24f4fbb258e82aa6828d161a16a422345d30b95fe206fdcf
+  data.tar.gz: d82311c0b9dc47484b50846b5bf05adc0950909b38b5bedfea2b21bf0e464655ff68794efc956864dabb153824f0a1121680fd9de924e63513b43a6fd92dbf7b

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+### 8.2.2
+  - Use `#response_body` instead of `#body` when debugging response from the server #679
+
+## 8.2.1
+  - Docs: Add DLQ policy section
+
 ## 8.2.0
   - Improved Elasticsearch version handling
   - Improved event error logging when DLQ is disabled in Logstash

data/docs/index.asciidoc

@@ -23,7 +23,7 @@ include::{include_path}/plugin_header.asciidoc[]
 .Compatibility Note
 [NOTE]
 ================================================================================
-Starting with Elasticsearch 5.3, there's an {ref}modules-http.html[HTTP setting]
+Starting with Elasticsearch 5.3, there's an {ref}/modules-http.html[HTTP setting]
 called `http.content_type.required`. If this option is set to `true`, and you
 are using Logstash 2.4 through 5.2, you need to update the Elasticsearch output
 plugin to version 6.2.5 or higher.
@@ -41,14 +41,17 @@ to upgrade Logstash in lock-step.
 You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
 
 ==== Template management for Elasticsearch 5.x
+
 Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch's mapping changes in version 5.0.
 Most importantly, the subfield for string multi-fields has changed from `.raw` to `.keyword` to match ES default
 behavior.
 
-** Users installing ES 5.x and LS 5.x **
+**Users installing ES 5.x and LS 5.x**
+
 This change will not affect you and you will continue to use the ES defaults.
 
-** Users upgrading from LS 2.x to LS 5.x with ES 5.x **
+**Users upgrading from LS 2.x to LS 5.x with ES 5.x**
+
 LS will not force upgrade the template, if `logstash` template already exists. This means you will still use
 `.raw` for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data after
 the new template is installed.
@@ -63,13 +66,27 @@ request are handled differently than error codes for individual documents.
 HTTP requests to the bulk API are expected to return a 200 response code. All other response codes are retried indefinitely.
 
 The following document errors are handled as follows:
-- 400 and 404 errors are sent to the DLQ if enabled. If a DLQ is not enabled a log message will be emitted and the event will be dropped.
-- 409 errors (conflict) are logged as a warning and dropped. 
+
+* 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See <<dlq-policy>> for more info.
+* 409 errors (conflict) are logged as a warning and dropped. 
 
 Note that 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
 It is more performant for Elasticsearch to retry these exceptions than this plugin.
 
-==== Batch Sizes ====
+[[dlq-policy]]
+==== DLQ Policy
+
+Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunately
+mapping errors cannot be handled without human intervention and without looking
+at the field that caused the mapping mismatch. If the DLQ is enabled, the
+original events causing the mapping errors are stored in a file that can be
+processed at a later time. Often times, the offending field can be removed and
+re-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping error
+happens, the problem is logged as a warning, and the event is dropped. See
+<<dead-letter-queues>> for more information about processing events in the DLQ.
+
+==== Batch Sizes
+
 This plugin attempts to send batches of events as a single request. However, if
 a request exceeds 20MB we will break it up until multiple batch requests. If a single document exceeds 20MB it will be sent as a single request.
 

data/lib/logstash/outputs/elasticsearch/common.rb

@@ -248,7 +248,7 @@ module LogStash; module Outputs; class ElasticSearch;
         retry unless @stopping.true?
       rescue ::LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError => e
         log_hash = {:code => e.response_code, :url => e.url.sanitized.to_s}
-        log_hash[:body] = e.body if @logger.debug? # Generally this is too verbose
+        log_hash[:body] = e.response_body if @logger.debug? # Generally this is too verbose
         message = "Encountered a retryable error. Will Retry with exponential backoff "
 
         # We treat 429s as a special case because these really aren't errors, but

data/logstash-output-elasticsearch.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-output-elasticsearch'
-  s.version         = '8.2.0'
+  s.version         = '8.2.2'
   s.licenses        = ['apache-2.0']
   s.summary         = "Logstash Output to Elasticsearch"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 8.2.0
+  version: 8.2.2
 platform: java
 authors:
 - Elastic