logstash-output-elasticsearch 2.0.0.pre.beta-java → 2.1.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ff0491347a2d439e0bf928efb828a6fd9e9d652b
-  data.tar.gz: c7e9d5ad2322c92ac571894937a84888306555d2
+  metadata.gz: 3b46615e7a20412351d62189c561aff137efed71
+  data.tar.gz: ff02b2887b46616fadf1ae075a6512cd4012d745
 SHA512:
-  metadata.gz: 461aea49e5262b4fc469c6c2652793e5f1acdee720e268c68efebc36d0f718fbd14f4d9ba7a6277061bdaa465df6d1515b72c4f8d7dcf60d9c5773a577d80a6a
-  data.tar.gz: 0f634800a20a4e437accd28c7b7694352a603fac90cf707d9b9214304adf5cc3582c6eced384a17eb5d0ff3ca5a2f6bf5cc8551a3ed83f6358905cca26ae7e83
+  metadata.gz: 0b1a582068d0f6fe453b48566c1b97a829d32501c15c4ba8eb241f9365b63b6f5cf9fd7be305698c4ba0dbe7b9b72aa1cbb517225809ce341475947d924018a9
+  data.tar.gz: 123a5e7173cd0984aeb750419021da26bd55644ed7f5bf607be20f697069eca492a669836a1663c9d7c24cbea7957a6e9f68d48abc99fa9ad5813896ed10c4d9

data/CHANGELOG.md

@@ -1,3 +1,23 @@
+## 2.1.0
+ - New setting: timeout. This lets you control the behavior of a slow/stuck
+   request to Elasticsearch that could be, for example, caused by network,
+   firewall, or load balancer issues.
+
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+
+## 2.0.0-beta2
+ - Massive internal refactor of client handling
+ - Background HTTP sniffing support
+ - Reduced bulk request size to 500 from 5000 (better memory utilization)
+ - Removed 'host' config option. Now use 'hosts'
+
+## 2.0.0-beta
+ - Only support HTTP Protocol
+ - Removed support for node and transport protocols (now in logstash-output-elasticsearch_java)
+ 
 ## 1.0.7
  - Add update API support
 

data/lib/logstash/outputs/elasticsearch.rb

@@ -10,44 +10,33 @@ require "thread" # for safe queueing
 require "uri" # for escaping user input
 require "logstash/outputs/elasticsearch/http_client"
 
-# This output lets you store logs in Elasticsearch and is the most recommended
-# output for Logstash. If you plan on using the Kibana web interface, you'll
-# want to use this output.
+# This plugin is the recommended method of storing logs in Elasticsearch.
+# If you plan on using the Kibana web interface, you'll want to use this output.
 #
-# This output only speaks the HTTP, which is the preferred protocol for interacting with Elasticsearch. By default
-# Elasticsearch exposes HTTP on port 9200.
-#
-# We strongly encourage the use of HTTP over the node protocol. It is just as
-# fast and far easier to administer. For those wishing to use the java protocol please see the 'elasticsearch_java' gem.
+# This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0.
+# We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower,
+# yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having
+# to upgrade Logstash in lock-step. For those still wishing to use the node or transport protocols please see
+# the https://www.elastic.co/guide/en/logstash/2.0/plugins-outputs-elasticsearch_java.html[logstash-output-elasticsearch_java] plugin.
 #
 # You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
 #
 # ==== Retry Policy
 #
-# By default all bulk requests to ES are synchronous. Not all events in the bulk requests
-# always make it successfully. For example, there could be events which are not formatted
-# correctly for the index they are targeting (type mismatch in mapping). So that we minimize loss of 
-# events, we have a specific retry policy in place. We retry all events which fail to be reached by 
-# Elasticsearch for network related issues. We retry specific events which exhibit errors under a separate 
-# policy described below. Events of this nature are ones which experience ES error codes described as 
-# retryable errors.
-#
-# *Retryable Errors:*
+# This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience
+# either partial or total failures. Events are retried if they fail due to either a network error or the status codes
+# 429 (the server is busy), 409 (Version Conflict), or 503 (temporary overloading/maintenance).
 #
-# - 429, Too Many Requests (RFC6585)
-# - 503, The server is currently unable to handle the request due to a temporary overloading or maintenance of the server.
-# 
-# Here are the rules of what is retried when:
+# The retry policy's logic can be described as follows:
 #
-# - Block and retry all events in bulk response that experiences transient network exceptions until
+# - Block and retry all events in the bulk response that experience transient network exceptions until
 #   a successful submission is received by Elasticsearch.
-# - Retry subset of sent events which resulted in ES errors of a retryable nature which can be found 
-#   in RETRYABLE_CODES
-# - For events which returned retryable error codes, they will be pushed onto a separate queue for 
-#   retrying events. events in this queue will be retried a maximum of 5 times by default (configurable through :max_retries). The size of 
-#   this queue is capped by the value set in :retry_max_items.
-# - Events from the retry queue are submitted again either when the queue reaches its max size or when
-#   the max interval time is reached, which is set in :retry_max_interval.
+# - Retry the subset of sent events which resulted in ES errors of a retryable nature.
+# - Events which returned retryable error codes will be pushed onto a separate queue for
+#   retrying events. Events in this queue will be retried a maximum of 5 times by default (configurable through :max_retries).
+#   The size of this queue is capped by the value set in :retry_max_items.
+# - Events from the retry queue are submitted again when the queue reaches its max size or when
+#   the max interval time is reached. The max interval time is configurable via :retry_max_interval.
 # - Events which are not retryable or have reached their max retry count are logged to stderr.
 class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   attr_reader :client
@@ -67,13 +56,13 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
 
   # The index type to write events to. Generally you should try to write only
   # similar events to the same 'type'. String expansion `%{foo}` works here.
-  # 
+  #
   # Deprecated in favor of `document_type` field.
   config :index_type, :validate => :string, :deprecated => "Please use the 'document_type' setting instead. It has the same effect, but is more appropriately named."
 
   # The document type to write events to. Generally you should try to write only
   # similar events to the same 'type'. String expansion `%{foo}` works here.
-  # Unless you set 'document_type', the event 'type' will be used if it exists 
+  # Unless you set 'document_type', the event 'type' will be used if it exists
   # otherwise the document type will be assigned the value of 'logs'
   config :document_type, :validate => :string
 
@@ -113,24 +102,27 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # This can be dynamic using the `%{foo}` syntax.
   config :routing, :validate => :string
 
-  # Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the `host` parameter.
+  # Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the `hosts` parameter.
   # Remember the `http` protocol uses the http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http[http] address (eg. 9200, not 9300).
   #     `"127.0.0.1"`
   #     `["127.0.0.1:9200","127.0.0.2:9200"]`
-  # It is important to exclude http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated master nodes] from the `host` list
-  # to prevent LS from sending bulk requests to the master nodes.  So this parameter should only reference either data or client nodes.
+  # It is important to exclude http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html[dedicated master nodes] from the `hosts` list
+  # to prevent LS from sending bulk requests to the master nodes.  So this parameter should only reference either data or client nodes in Elasticsearch.
 
   config :hosts, :validate => :array
 
-  # You can set the remote port as part of the host, or explicitly here as well
-  config :port, :validate => :string, :default => 9200
+  # The port setting is obsolete.  Please use the 'hosts' setting instead.
+  # Hosts entries can be in "host:port" format.
+  config :port, :obsolete => "Please use the 'hosts' setting instead. Hosts entries can be in 'host:port' format."
 
-  # This plugin uses the bulk index api for improved indexing performance.
-  # To make efficient bulk api calls, we will buffer a certain number of
+  # This plugin uses the bulk index API for improved indexing performance.
+  # To make efficient bulk API calls, we will buffer a certain number of
   # events before flushing that out to Elasticsearch. This setting
   # controls how many events will be buffered before sending a batch
-  # of events.
-  config :flush_size, :validate => :number, :default => 5000
+  # of events. Increasing the `flush_size` has an effect on Logstash's heap size.
+  # Remember to also increase the heap size using `LS_HEAP_SIZE` if you are sending big documents
+  # or have increased the `flush_size` to a higher value.
+  config :flush_size, :validate => :number, :default => 500
 
   # The amount of time since last flush before a flush is forced.
   #
@@ -143,64 +135,61 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # near-real-time.
   config :idle_flush_time, :validate => :number, :default => 1
 
-  # The Elasticsearch action to perform. Valid actions are: `index`, `delete`.
-  #
-  # Use of this setting *REQUIRES* you also configure the `document_id` setting
-  # because `delete` actions all require a document id.
-  #
-  # What does each action do?
+  # The Elasticsearch action to perform. Valid actions are:
   #
   # - index: indexes a document (an event from Logstash).
-  # - delete: deletes a document by id
+  # - delete: deletes a document by id (An id is required for this action)
   # - create: indexes a document, fails if a document by that id already exists in the index.
-  # - update: updates a document by id
-  # following action is not supported by HTTP protocol
+  # - update: updates a document by id. Update has a special case where you can upsert -- update a
+  #   document if not already present. See the `upsert` option
   #
-  # For more details on actions, check out the http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
+  # For more details on actions, check out the http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
   config :action, :validate => %w(index delete create update), :default => "index"
 
-  # Username and password (only valid when protocol is HTTP; this setting works with HTTP or HTTPS auth)
+  # Username to authenticate to a secure Elasticsearch cluster
   config :user, :validate => :string
+  # Password to authenticate to a secure Elasticsearch cluster
   config :password, :validate => :password
 
-  # HTTP Path at which the Elasticsearch server lives. Use this if you must run ES behind a proxy that remaps
-  # the root path for the Elasticsearch HTTP API lives. This option is ignored for non-HTTP transports.
+  # HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps
+  # the root path for the Elasticsearch HTTP API lives.
   config :path, :validate => :string, :default => "/"
 
-  # SSL Configurations (only valid when protocol is HTTP)
-  #
-  # Enable SSL
+  # Enable SSL/TLS secured communication to Elasticsearch cluster
   config :ssl, :validate => :boolean, :default => false
 
-  # Validate the server's certificate
-  # Disabling this severely compromises security
-  # For more information read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
+  # Option to validate the server's certificate. Disabling this severely compromises security.
+  # For more information on disabling certificate verification please read
+  # https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
   config :ssl_certificate_verification, :validate => :boolean, :default => true
 
   # The .cer or .pem file to validate the server's certificate
   config :cacert, :validate => :path
 
-  # The JKS truststore to validate the server's certificate
+  # The JKS truststore to validate the server's certificate.
   # Use either `:truststore` or `:cacert`
   config :truststore, :validate => :path
 
   # Set the truststore password
   config :truststore_password, :validate => :password
 
-  # The keystore used to present a certificate to the server
+  # The keystore used to present a certificate to the server.
   # It can be either .jks or .p12
   config :keystore, :validate => :path
 
   # Set the truststore password
   config :keystore_password, :validate => :password
 
-  # Enable cluster sniffing
-  # Asks host for the list of all cluster nodes and adds them to the hosts list
-  # Will return ALL nodes with HTTP enabled (including master nodes!). If you use
+  # This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list.
+  # Note: This will return ALL nodes with HTTP enabled (including master nodes!). If you use
   # this with master nodes, you probably want to disable HTTP on them by setting
-  # `http.enabled` to false in their elasticsearch.yml.
+  # `http.enabled` to false in their elasticsearch.yml. You can either use the `sniffing` option or
+  # manually enter multiple Elasticsearch hosts using the `hosts` paramater.
   config :sniffing, :validate => :boolean, :default => false
 
+  # How long to wait, in seconds, between sniffing attempts
+  config :sniffing_delay, :validate => :number, :default => 5
+
   # Set max retry for each event
   config :max_retries, :validate => :number, :default => 3
 
@@ -210,45 +199,54 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # Set max interval between bulk retries
   config :retry_max_interval, :validate => :number, :default => 5
 
-  # Set the address of a forward HTTP proxy. Must be used with the 'http' protocol
-  # Can be either a string, such as 'http://localhost:123' or a hash in the form
-  # {host: 'proxy.org' port: 80 scheme: 'http'}
+  # Set the address of a forward HTTP proxy.
+  # Can be either a string, such as `http://localhost:123` or a hash in the form
+  # of `{host: 'proxy.org' port: 80 scheme: 'http'}`.
   # Note, this is NOT a SOCKS proxy, but a plain HTTP proxy
   config :proxy
 
-  # Enable doc_as_upsert for update mode
-  # create a new document with source if document_id doesn't exists
+  # Enable `doc_as_upsert` for update mode.
+  # Create a new document with source if `document_id` doesn't exist in Elasticsearch
   config :doc_as_upsert, :validate => :boolean, :default => false
 
-  # Set upsert content for update mode
-  # create a new document with this parameter as json string if document_id doesn't exists
+  # Set upsert content for update mode.
+  # Create a new document with this parameter as json string if `document_id` doesn't exists
   config :upsert, :validate => :string, :default => ""
 
+  # Set the timeout for network operations and requests sent Elasticsearch. If
+  # a timeout occurs, the request will be retried.
+  config :timeout, :validate => :number
+
   public
   def register
     @hosts = Array(@hosts)
     # retry-specific variables
     @retry_flush_mutex = Mutex.new
-    @retry_teardown_requested = Concurrent::AtomicBoolean.new(false)
+    @retry_close_requested = Concurrent::AtomicBoolean.new(false)
     # needs flushing when interval
     @retry_queue_needs_flushing = ConditionVariable.new
     @retry_queue_not_full = ConditionVariable.new
     @retry_queue = Queue.new
+    @submit_mutex = Mutex.new
 
     client_settings = {}
-    common_options = {:client_settings => client_settings}
+    common_options = {
+      :client_settings => client_settings,
+      :sniffing => @sniffing,
+      :sniffing_delay => @sniffing_delay
+    }
 
+    common_options[:timeout] = @timeout if @timeout
     client_settings[:path] = "/#{@path}/".gsub(/\/+/, "/") # Normalize slashes
     @logger.debug? && @logger.debug("Normalizing http path", :path => @path, :normalized => client_settings[:path])
 
-    if @host.nil?
+    if @hosts.nil? || @hosts.empty?
       @logger.info("No 'host' set in elasticsearch output. Defaulting to localhost")
-      @host = ["localhost"]
+      @hosts = ["localhost"]
     end
 
     client_settings.merge! setup_ssl()
     client_settings.merge! setup_proxy()
-    client_settings.merge! setup_sniffing()
     common_options.merge! setup_basic_auth()
 
     # Update API setup
@@ -258,13 +256,8 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
     }
     common_options.merge! update_options if @action == 'update'
 
-    option_hosts = @hosts.map do |host|
-      host_name, port = host.split(":")
-      { :host => host_name, :port => (port || @port).to_i }
-    end
-
     @client = LogStash::Outputs::Elasticsearch::HttpClient.new(
-      common_options.merge(:hosts => @hosts)
+      common_options.merge(:hosts => @hosts, :logger => @logger)
     )
 
     if @manage_template
@@ -276,7 +269,7 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
       end
     end
 
-    @logger.info("New Elasticsearch output", :hosts => @hosts, :port => @port)
+    @logger.info("New Elasticsearch output", :hosts => @hosts)
 
     @client_idx = 0
 
@@ -294,7 +287,7 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
     end
 
     @retry_thread = Thread.new do
-      while @retry_teardown_requested.false?
+      while @retry_close_requested.false?
         @retry_flush_mutex.synchronize { @retry_queue_needs_flushing.wait(@retry_flush_mutex) }
         retry_flush
       end
@@ -317,9 +310,8 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
 
   public
   def receive(event)
-    return unless output?(event)
 
-    # block until we have not maxed out our 
+    # block until we have not maxed out our
     # retry queue. This is applying back-pressure
     # to slow down the receive-rate
     @retry_flush_mutex.synchronize {
@@ -343,7 +335,7 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
       :_type => type,
       :_routing => @routing ? event.sprintf(@routing) : nil
     }
-    
+
     params[:_upsert] = LogStash::Json.load(event.sprintf(@upsert)) if @action == 'update' && @upsert != ""
 
     buffer_receive([event.sprintf(@action), params, event])
@@ -353,21 +345,29 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # The submit method can be called from both the
   # Stud::Buffer flush thread and from our own retry thread.
   def submit(actions)
-    es_actions = actions.map { |a, doc, event| [a, doc, event.to_hash] }
+    @submit_mutex.synchronize do
+      es_actions = actions.map { |a, doc, event| [a, doc, event.to_hash] }
+
+      bulk_response = @client.bulk(es_actions)
 
-    bulk_response = @client.bulk(es_actions)
+      next unless bulk_response["errors"]
 
-    if bulk_response["errors"]
-      actions_with_responses = actions.zip(bulk_response['statuses'])
       actions_to_retry = []
-      actions_with_responses.each do |action, resp_code|
-        if RETRYABLE_CODES.include?(resp_code)
-          @logger.warn "retrying failed action with response code: #{resp_code}"
+
+      bulk_response["items"].each_with_index do |resp,idx|
+        action_type, action_props = resp.first
+
+        status = action_props["status"]
+        action = actions[idx]
+
+        if RETRYABLE_CODES.include?(status)
+          @logger.warn "retrying failed action with response code: #{status}"
           actions_to_retry << action
-        elsif not SUCCESS_CODES.include?(resp_code)
-          @logger.warn "failed action with response of #{resp_code}, dropping action: #{action}"
+        elsif not SUCCESS_CODES.include?(status)
+          @logger.warn "Failed action. ", status: status, action: action, response: resp
         end
       end
+
       retry_push(actions_to_retry) unless actions_to_retry.empty?
     end
   end
@@ -375,7 +375,7 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # When there are exceptions raised upon submission, we raise an exception so that
   # Stud::Buffer will retry to flush
   public
-  def flush(actions, teardown = false)
+  def flush(actions, close = false)
     begin
       submit(actions)
     rescue Manticore::SocketException => e
@@ -407,26 +407,24 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   end # def flush
 
   public
-  def teardown
-    if @cacert # remove temporary jks store created from the cacert
-      File.delete(@truststore)
-    end
+  def close
+    @client.stop_sniffing!
 
-    @retry_teardown_requested.make_true
+    @retry_close_requested.make_true
     # First, make sure retry_timer_thread is stopped
-    # to ensure we do not signal a retry based on 
+    # to ensure we do not signal a retry based on
     # the retry interval.
     Thread.kill(@retry_timer_thread)
     @retry_timer_thread.join
-    # Signal flushing in the case that #retry_flush is in 
+    # Signal flushing in the case that #retry_flush is in
     # the process of waiting for a signal.
     @retry_flush_mutex.synchronize { @retry_queue_needs_flushing.signal }
-    # Now, #retry_flush is ensured to not be in a state of 
+    # Now, #retry_flush is ensured to not be in a state of
     # waiting and can be safely joined into the main thread
     # for further final execution of an in-process remaining call.
     @retry_thread.join
 
-    # execute any final actions along with a proceeding retry for any 
+    # execute any final actions along with a proceeding retry for any
     # final actions that did not succeed.
     buffer_flush(:final => true)
     retry_flush
@@ -448,11 +446,6 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
     return {:proxy => proxy}
   end
 
-  private
-  def setup_sniffing
-    { :reload_connections => @reload_connections }
-  end
-
   private
   def setup_ssl
     return {} unless @ssl
@@ -460,12 +453,15 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
     if @cacert && @truststore
       raise(LogStash::ConfigurationError, "Use either \"cacert\" or \"truststore\" when configuring the CA certificate") if @truststore
     end
+
     ssl_options = {}
-    if @cacert then
-      @truststore, ssl_options[:truststore_password] = generate_jks @cacert
+
+    if @cacert
+      ssl_options[:ca_file] = @cacert
     elsif @truststore
       ssl_options[:truststore_password] = @truststore_password.value if @truststore_password
     end
+
     ssl_options[:truststore] = @truststore if @truststore
     if @keystore
       ssl_options[:keystore] = @keystore
@@ -493,35 +489,12 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   end
 
   private
-  def generate_jks cert_path
-
-    require 'securerandom'
-    require 'tempfile'
-    require 'java'
-    import java.io.FileInputStream
-    import java.io.FileOutputStream
-    import java.security.KeyStore
-    import java.security.cert.CertificateFactory
-
-    jks = java.io.File.createTempFile("cert", ".jks")
-
-    ks = KeyStore.getInstance "JKS"
-    ks.load nil, nil
-    cf = CertificateFactory.getInstance "X.509"
-    cert = cf.generateCertificate FileInputStream.new(cert_path)
-    ks.setCertificateEntry "cacert", cert
-    pwd = SecureRandom.urlsafe_base64(9)
-    ks.store FileOutputStream.new(jks), pwd.to_java.toCharArray
-    [jks.path, pwd]
-  end
-
-  private
-  # in charge of submitting any actions in @retry_queue that need to be 
+  # in charge of submitting any actions in @retry_queue that need to be
   # retried
   #
   # This method is not called concurrently. It is only called by @retry_thread
-  # and once that thread is ended during the teardown process, a final call 
-  # to this method is done upon teardown in the main thread.
+  # and once that thread is ended during the close process, a final call
+  # to this method is done upon close in the main thread.
   def retry_flush()
     unless @retry_queue.empty?
       buffer = @retry_queue.size.times.map do