logstash-output-elasticsearch 7.4.3-java → 8.0.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: 138d01e9314f05b8eee7f9bd1115f3c7aeb33117544e840caa83b3b3e2d1ac54
-  data.tar.gz: 7516db608590f0261e1d66b957648fae9878433bec8e4b67b3325207ceb8604a
+SHA1:
+  metadata.gz: 54e5ec0cdf1cd6e3331fa31ed9794128debfc581
+  data.tar.gz: 37e763dae42fb2b01fdbd601356847fd5515bf4a
 SHA512:
-  metadata.gz: 5f3e2c6287de60def0469b17d4fac31336cd986c35f04f9fd20e7d0f710d8dfdd75163383994da0e627d3f7e78aa61e1fb5d4613e92f0c9f2b2a38e6a147b84a
-  data.tar.gz: 401784c66a69b13c905586f3b5643babc78c37d5996d448c4f7b16d0b3534999236f0efe7717cd56fedd61bc226c12ab0add7b13ec50913e5ea36906a6369ced
+  metadata.gz: 43493de2f061e43a2d94cb38b0047307fa8d6713f8bef930f8754fd4e312a1959124b8f3ba9a5946bfa1774ba619bc1e91de1c0022a054c297e669b3fd703794
+  data.tar.gz: 32ce21a37acfefdcfa46230143ab4c84fee33d7866d1e47eaca0f955be77b2a256c22e29dbbffdb68b816e6f82c2e63d55c206e3ac1be2d9c733ac086ba11c5c

data/CHANGELOG.md

@@ -1,21 +1,8 @@
-## 7.4.3
-  - Avoid infinite loop by checking for zero body size #737
-  - Add support for join based parent indexing (backport of #632) #686
-
-## 7.4.2
-  - Use `#response_body` instead of `#body` when debugging response from the server #679
-  - Docs: Add DLQ policy section
-  - Fix passing of String instead of SafeURI to BadURIError
-
-## 7.4.1
-  - Properly detect if DLQ is supported and enabled
-
-## 7.4.0
-  - Retry all non-200 responses of the bulk API indefinitely
-  - Improve documentation on retry codes
-
-##  7.3.8
- - Fix bug where java class names were logged rather than actual host names in various scenarios
+## 8.0.0
+ - Breaking: make deprecated options :flush_size and :idle_flush_time obsolete
+ - Remove obsolete options :max_retries and :retry_max_items
+ - Fix: handling of initial single big event
+ - Fix: typo was enabling http compression by default this returns it back to false
 
 ## 7.3.7
  - Properly support characters needing escaping in users / passwords across multiple SafeURI implementions (pre/post LS 5.5.1)

data/docs/index.asciidoc

@@ -23,7 +23,7 @@ include::{include_path}/plugin_header.asciidoc[]
 .Compatibility Note
 [NOTE]
 ================================================================================
-Starting with Elasticsearch 5.3, there's an {ref}/modules-http.html[HTTP setting]
+Starting with Elasticsearch 5.3, there's an {ref}modules-http.html[HTTP setting]
 called `http.content_type.required`. If this option is set to `true`, and you
 are using Logstash 2.4 through 5.2, you need to update the Elasticsearch output
 plugin to version 6.2.5 or higher.
@@ -41,52 +41,34 @@ to upgrade Logstash in lock-step.
 You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
 
 ==== Template management for Elasticsearch 5.x
-
 Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch's mapping changes in version 5.0.
 Most importantly, the subfield for string multi-fields has changed from `.raw` to `.keyword` to match ES default
 behavior.
 
-**Users installing ES 5.x and LS 5.x**
-
+** Users installing ES 5.x and LS 5.x **
 This change will not affect you and you will continue to use the ES defaults.
 
-**Users upgrading from LS 2.x to LS 5.x with ES 5.x**
-
+** Users upgrading from LS 2.x to LS 5.x with ES 5.x **
 LS will not force upgrade the template, if `logstash` template already exists. This means you will still use
 `.raw` for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data after
 the new template is installed.
 
 ==== Retry Policy
 
-The retry policy has changed significantly in the 7.4.0 release.
+The retry policy has changed significantly in the 2.2.0 release.
 This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience
-either partial or total failures. The bulk API sends batches of requests to an HTTP endpoint. Error codes for the HTTP
-request are handled differently than error codes for individual documents.
+either partial or total failures.
 
-HTTP requests to the bulk API are expected to return a 200 response code. All other response codes are retried indefinitely.
+The following errors are retried infinitely:
 
-The following document errors are handled as follows:
+- Network errors (inability to connect)
+- 429 (Too many requests) and
+- 503 (Service unavailable) errors
 
-  * 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See <<dlq-policy>> for more info.
-  * 409 errors (conflict) are logged as a warning and dropped. 
-
-Note that 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
+NOTE: 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
 It is more performant for Elasticsearch to retry these exceptions than this plugin.
 
-[[dlq-policy]]
-==== DLQ Policy
-
-Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunately
-mapping errors cannot be handled without human intervention and without looking
-at the field that caused the mapping mismatch. If the DLQ is enabled, the
-original events causing the mapping errors are stored in a file that can be
-processed at a later time. Often times, the offending field can be removed and
-re-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping error
-happens, the problem is logged as a warning, and the event is dropped. See
-<<dead-letter-queues>> for more information about processing events in the DLQ.
-
-==== Batch Sizes
-
+==== Batch Sizes ====
 This plugin attempts to send batches of events as a single request. However, if
 a request exceeds 20MB we will break it up until multiple batch requests. If a single document exceeds 20MB it will be sent as a single request.
 
@@ -252,15 +234,6 @@ Set the Elasticsearch errors in the whitelist that you don't want to log.
 A useful example is when you want to skip all 409 errors
 which are `document_already_exists_exception`.
 
-[id="plugins-{type}s-{plugin}-flush_size"]
-===== `flush_size`  (DEPRECATED)
-
-  * DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
-  * Value type is <<number,number>>
-  * There is no default value for this setting.
-
-
-
 [id="plugins-{type}s-{plugin}-healthcheck_path"]
 ===== `healthcheck_path` 
 
@@ -298,15 +271,6 @@ Any special characters present in the URLs here MUST be URL escaped! This means
 
 Enable gzip compression on requests. Note that response compression is on by default for Elasticsearch v5.0 and beyond
 
-[id="plugins-{type}s-{plugin}-idle_flush_time"]
-===== `idle_flush_time`  (DEPRECATED)
-
-  * DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
-  * Value type is <<number,number>>
-  * Default value is `1`
-
-
-
 [id="plugins-{type}s-{plugin}-index"]
 ===== `index` 
 
@@ -495,8 +459,7 @@ Set script name for scripted update mode
   * Value type is <<string,string>>
   * Default value is `"painless"`
 
-Set the language of the used script. If not set, this defaults to painless in ES 5.0.
-When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to `""` (empty string).
+Set the language of the used script. If not set, this defaults to painless in ES 5.0
 
 [id="plugins-{type}s-{plugin}-script_type"]
 ===== `script_type` 
@@ -696,4 +659,4 @@ See also https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-in
 
 
 [id="plugins-{type}s-{plugin}-common-options"]
-include::{include_path}/{type}.asciidoc[]
+include::{include_path}/{type}.asciidoc[]

data/lib/logstash/outputs/elasticsearch/common.rb

@@ -4,10 +4,14 @@ module LogStash; module Outputs; class ElasticSearch;
   module Common
     attr_reader :client, :hosts
 
-    # These codes apply to documents, not at the request level
-    DOC_DLQ_CODES = [400, 404]
-    DOC_SUCCESS_CODES = [200, 201]
-    DOC_CONFLICT_CODE = 409
+    # These are codes for temporary recoverable conditions
+    # 429 just means that ES has too much traffic ATM
+    # 503 means it , or a proxy is temporarily unavailable
+    RETRYABLE_CODES = [429, 503]
+
+    DLQ_CODES = [400, 404]
+    SUCCESS_CODES = [200, 201]
+    CONFLICT_CODE = 409
 
     # When you use external versioning, you are communicating that you want
     # to ignore conflicts. More obviously, since an external version is a 
@@ -18,7 +22,7 @@ module LogStash; module Outputs; class ElasticSearch;
     def register
       @stopping = Concurrent::AtomicBoolean.new(false)
       # To support BWC, we check if DLQ exists in core (< 5.4). If it doesn't, we use nil to resort to previous behavior.
-      @dlq_writer = dlq_enabled? ? execution_context.dlq_writer : nil
+      @dlq_writer = supports_dlq? ? execution_context.dlq_writer : nil
 
       setup_hosts # properly sets @hosts
       build_client
@@ -30,13 +34,7 @@ module LogStash; module Outputs; class ElasticSearch;
 
     # Receive an array of events and immediately attempt to index them (no buffering)
     def multi_receive(events)
-      if @flush_size
-        events.each_slice(@flush_size) do |slice|
-          retrying_submit(slice.map {|e| event_action_tuple(e) })
-        end
-      else
-        retrying_submit(events.map {|e| event_action_tuple(e)})
-      end
+      retrying_submit(events.map {|e| event_action_tuple(e)})
     end
 
     # Convert the event into a 3-tuple of action, params, and event
@@ -136,12 +134,12 @@ module LogStash; module Outputs; class ElasticSearch;
         # - For 409, we log and drop. there is nothing we can do
         # - For a mapping error, we send to dead letter queue for a human to intervene at a later point.
         # - For everything else there's mastercard. Yep, and we retry indefinitely. This should fix #572 and other transient network issues
-        if DOC_SUCCESS_CODES.include?(status)
+        if SUCCESS_CODES.include?(status)
           next
-        elsif DOC_CONFLICT_CODE == status
+        elsif CONFLICT_CODE == status
           @logger.warn "Failed action.", status: status, action: action, response: response if !failure_type_logging_whitelist.include?(failure["type"])
           next
-        elsif DOC_DLQ_CODES.include?(status)
+        elsif DLQ_CODES.include?(status)
           action_event = action[2]
           # To support bwc, we check if DLQ exists. otherwise we log and drop event (previous behavior)
           if @dlq_writer
@@ -176,15 +174,8 @@ module LogStash; module Outputs; class ElasticSearch;
         params[:pipeline] = event.sprintf(@pipeline)
       end
 
-      if @parent
-        if @join_field
-          join_value = event.get(@join_field)
-          parent_value = event.sprintf(@parent)
-          event.set(@join_field, { "name" => join_value, "parent" => parent_value })
-          params[:_routing] = event.sprintf(@parent)
-        else
-          params[:parent] = event.sprintf(@parent)
-        end
+     if @parent
+        params[:parent] = event.sprintf(@parent)
       end
 
       if @action == 'update'
@@ -253,22 +244,29 @@ module LogStash; module Outputs; class ElasticSearch;
         sleep_interval = next_sleep_interval(sleep_interval)
         retry unless @stopping.true?
       rescue ::LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError => e
-        log_hash = {:code => e.response_code, :url => e.url.sanitized.to_s}
-        log_hash[:body] = e.response_body if @logger.debug? # Generally this is too verbose
-        message = "Encountered a retryable error. Will Retry with exponential backoff "
-
-        # We treat 429s as a special case because these really aren't errors, but
-        # rather just ES telling us to back off a bit, which we do.
-        # The other retryable code is 503, which are true errors
-        # Even though we retry the user should be made aware of these
-        if e.response_code == 429
-          logger.debug(message, log_hash)
+        if RETRYABLE_CODES.include?(e.response_code)
+          log_hash = {:code => e.response_code, :url => e.url.sanitized.to_s}
+          log_hash[:body] = e.body if @logger.debug? # Generally this is too verbose
+          message = "Encountered a retryable error. Will Retry with exponential backoff "
+
+          # We treat 429s as a special case because these really aren't errors, but
+          # rather just ES telling us to back off a bit, which we do.
+          # The other retryable code is 503, which are true errors
+          # Even though we retry the user should be made aware of these
+          if e.response_code == 429
+            logger.debug(message, log_hash)
+          else
+            logger.error(message, log_hash)
+          end
+
+          sleep_interval = sleep_for_interval(sleep_interval)
+          retry
         else
-          logger.error(message, log_hash)
+          log_hash = {:code => e.response_code, 
+                      :response_body => e.response_body}
+          log_hash[:request_body] = e.request_body if @logger.debug?
+          @logger.error("Got a bad response code from server, but this code is not considered retryable. Request will be dropped", log_hash)
         end
-
-        sleep_interval = sleep_for_interval(sleep_interval)
-        retry
       rescue => e
         # Stuff that should never happen
         # For all other errors print out full connection issues
@@ -281,16 +279,14 @@ module LogStash; module Outputs; class ElasticSearch;
 
         @logger.debug("Failed actions for last bad bulk request!", :actions => actions)
 
+        # We retry until there are no errors! Errors should all go to the retry queue
         sleep_interval = sleep_for_interval(sleep_interval)
         retry unless @stopping.true?
       end
     end
 
-    def dlq_enabled?
-      # TODO there should be a better way to query if DLQ is enabled
-      # See more in: https://github.com/elastic/logstash/issues/8064
-      respond_to?(:execution_context) && execution_context.respond_to?(:dlq_writer) &&
-        !execution_context.dlq_writer.inner_writer.is_a?(::LogStash::Util::DummyDeadLetterQueueWriter)
+    def supports_dlq?
+      respond_to?(:execution_context) && execution_context.respond_to?(:dlq_writer)
     end
   end
 end; end; end

data/lib/logstash/outputs/elasticsearch/common_configs.rb

@@ -78,9 +78,6 @@ module LogStash; module Outputs; class ElasticSearch
       # This can be dynamic using the `%{foo}` syntax.
       mod.config :parent, :validate => :string, :default => nil
 
-      # For child documents, name of the join field
-      mod.config :join_field, :validate => :string, :default => nil
-
       # Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the `hosts` parameter.
       # Remember the `http` protocol uses the http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[http] address (eg. 9200, not 9300).
       #     `"127.0.0.1"`
@@ -94,9 +91,9 @@ module LogStash; module Outputs; class ElasticSearch
       # Any special characters present in the URLs here MUST be URL escaped! This means `#` should be put in as `%23` for instance.
       mod.config :hosts, :validate => :uri, :default => [::LogStash::Util::SafeURI.new("//127.0.0.1")], :list => true
 
-      mod.config :flush_size, :validate => :number, :deprecated => "This setting is no longer necessary as we now try to restrict bulk requests to sane sizes. See the 'Batch Sizes' section of the docs. If you think you still need to restrict payloads based on the number, not size, of events, please open a ticket."
+      mod.config :flush_size, :validate => :number, :obsolete => "This setting is no longer available as we now try to restrict bulk requests to sane sizes. See the 'Batch Sizes' section of the docs. If you think you still need to restrict payloads based on the number, not size, of events, please open a ticket."
 
-      mod.config :idle_flush_time, :validate => :number, :default => 1, :deprecated => "This is a no-op now as every pipeline batch is flushed synchronously obviating the need for this option."
+      mod.config :idle_flush_time, :validate => :number, :obsolete => "This settings is no longer valid. This was a no-op now as every pipeline batch is flushed synchronously obviating the need for this option."
 
       # Set upsert content for update mode.s
       # Create a new document with this parameter as json string if `document_id` doesn't exists
@@ -106,9 +103,6 @@ module LogStash; module Outputs; class ElasticSearch
       # Create a new document with source if `document_id` doesn't exist in Elasticsearch
       mod.config :doc_as_upsert, :validate => :boolean, :default => false
 
-      #Obsolete since 4.1.0
-      mod.config :max_retries, :obsolete => "This setting no longer does anything. Please remove it from your config"
-
       # Set script name for scripted update mode
       mod.config :script, :validate => :string, :default => ""
 
@@ -133,9 +127,6 @@ module LogStash; module Outputs; class ElasticSearch
       # Set max interval in seconds between bulk retries.
       mod.config :retry_max_interval, :validate => :number, :default => 64
 
-      #Obsolete since 4.1.0
-      mod.config :retry_max_items, :obsolete => "This setting no longer does anything. Please remove it from your config"
-
       # The number of times Elasticsearch should internally retry an update/upserted document
       # See the https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html[partial updates]
       # for more info

data/lib/logstash/outputs/elasticsearch/http_client.rb

@@ -50,11 +50,11 @@ module LogStash; module Outputs; class ElasticSearch;
     #   through a special http path, such as using mod_rewrite.
     def initialize(options={})
       @logger = options[:logger]
-
+      
       # Again, in case we use DEFAULT_OPTIONS in the future, uncomment this.
       # @options = DEFAULT_OPTIONS.merge(options)
       @options = options
-
+      
       @url_template = build_url_template
 
       @pool = build_pool(@options)
@@ -62,7 +62,7 @@ module LogStash; module Outputs; class ElasticSearch;
       # connection pool at the same time
       @bulk_path = @options[:bulk_path]
     end
-
+    
     def build_url_template
       {
         :scheme => self.scheme,
@@ -82,8 +82,9 @@ module LogStash; module Outputs; class ElasticSearch;
       template_put(name, template)
     end
 
-    def connected_es_versions
-      @pool.connected_es_versions
+    def get_version
+      url, response = @pool.get("")
+      LogStash::Json.load(response.body)["version"]
     end
 
     def bulk(actions)
@@ -106,7 +107,7 @@ module LogStash; module Outputs; class ElasticSearch;
       if http_compression
         body_stream.set_encoding "BINARY"
         stream_writer = Zlib::GzipWriter.new(body_stream, Zlib::DEFAULT_COMPRESSION, Zlib::DEFAULT_STRATEGY)
-      else
+      else 
         stream_writer = body_stream
       end
       bulk_responses = []
@@ -115,8 +116,8 @@ module LogStash; module Outputs; class ElasticSearch;
                     action.map {|line| LogStash::Json.dump(line)}.join("\n") :
                     LogStash::Json.dump(action)
         as_json << "\n"
-        if body_stream.size > 0 && (body_stream.size + as_json.bytesize) > TARGET_BULK_BYTES
-          bulk_responses << bulk_send(body_stream)
+        if (body_stream.size + as_json.bytesize) > TARGET_BULK_BYTES
+          bulk_responses << bulk_send(body_stream) unless body_stream.size == 0
         end
         stream_writer.write(as_json)
       end
@@ -135,29 +136,22 @@ module LogStash; module Outputs; class ElasticSearch;
 
     def bulk_send(body_stream)
       params = http_compression ? {:headers => {"Content-Encoding" => "gzip"}} : {}
-      # Discard the URL
-      response = @pool.post(@bulk_path, params, body_stream.string)
+      # Discard the URL 
+      _, response = @pool.post(@bulk_path, params, body_stream.string)
       if !body_stream.closed?
         body_stream.truncate(0)
         body_stream.seek(0)
       end
-
-      if response.code != 200
-        raise ::LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError.new(
-          response.code, @bulk_path, body_stream.to_s, response.body
-        )
-      end
-
       LogStash::Json.load(response.body)
     end
 
     def get(path)
-      response = @pool.get(path, nil)
+      url, response = @pool.get(path, nil)
       LogStash::Json.load(response.body)
     end
 
     def post(path, params = {}, body_string)
-      response = @pool.post(path, params, body_string)
+      url, response = @pool.post(path, params, body_string)
       LogStash::Json.load(response.body)
     end
 
@@ -208,7 +202,7 @@ module LogStash; module Outputs; class ElasticSearch;
                         else
                           nil
                         end
-
+      
       calculated_scheme = calculate_property(uris, :scheme, explicit_scheme, sniffing)
 
       if calculated_scheme && calculated_scheme !~ /https?/
@@ -228,7 +222,7 @@ module LogStash; module Outputs; class ElasticSearch;
       # Enter things like foo:123, bar and wind up with foo:123, bar:9200
       calculate_property(uris, :port, nil, sniffing) || 9200
     end
-
+    
     def uris
       @options[:hosts]
     end
@@ -247,7 +241,7 @@ module LogStash; module Outputs; class ElasticSearch;
 
     def build_adapter(options)
       timeout = options[:timeout] || 0
-
+      
       adapter_options = {
         :socket_timeout => timeout,
         :request_timeout => timeout,
@@ -268,11 +262,11 @@ module LogStash; module Outputs; class ElasticSearch;
       end
 
       adapter_options[:ssl] = ssl_options if self.scheme == 'https'
-
+      
       adapter_class = ::LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter
       adapter = adapter_class.new(@logger, adapter_options)
     end
-
+    
     def build_pool(options)
       adapter = build_adapter(options)
 
@@ -321,21 +315,22 @@ module LogStash; module Outputs; class ElasticSearch;
                     h.query
                   end
       prefixed_raw_query = raw_query && !raw_query.empty? ? "?#{raw_query}" : nil
-
+      
       raw_url = "#{raw_scheme}://#{postfixed_userinfo}#{raw_host}:#{raw_port}#{prefixed_raw_path}#{prefixed_raw_query}"
 
       ::LogStash::Util::SafeURI.new(raw_url)
     end
 
     def template_exists?(name)
-      response = @pool.head("/_template/#{name}")
+      url, response = @pool.head("/_template/#{name}")
       response.code >= 200 && response.code <= 299
     end
 
     def template_put(name, template)
       path = "_template/#{name}"
       logger.info("Installing elasticsearch template to #{path}")
-      @pool.put(path, nil, LogStash::Json.dump(template))
+      url, response = @pool.put(path, nil, LogStash::Json.dump(template))
+      response
     end
 
     # Build a bulk item for an elasticsearch update action