logstash-output-elasticsearch 0.1.6 → 3.0.0

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    N2UwMzczMTE0ZWNmZjFhOGU4Mjk0YjUwZDE5YmVmNDQyMWUzMzg0Mw==
-  data.tar.gz: !binary |-
-    ODJkYzg1NWU5YWEzYmZmOWRiMDFkNjFlYmM0Y2Q2NmZiZGEwNjYyMQ==
+SHA1:
+  metadata.gz: 51dba928b726d91c06b243e7cb8283d6acde3096
+  data.tar.gz: a877ceda5efa85042dbd832bb86d13e9f1ce98d7
 SHA512:
-  metadata.gz: !binary |-
-    Y2M3ZDliMjhjYmY4OGM0NjI5ODA2OTYyZWE1MzM2MzE0MmVlZDI1NmE0OTNl
-    YWY0MmEzNzQ2OWUwODc0ZWRkNmI0OWY3MDViOTU3ZjlhMGY5NzFjZDJhNGQz
-    NjdlZWQ4YWM3Nzk1NTI3OWQ1MDEyZmY0MzBkODM0Nzc3YTgzNWM=
-  data.tar.gz: !binary |-
-    MjY2Mzc0ZDRiOGEyOGIzMDg2NWMzNGRiMDA2MWYxMDM4YzhiZmZiN2ViZDhk
-    ZDI2ZGE2MzRhMTA1ZDMxZGM2NTQ3YzU1ZjE0NjYzMTU0NTk4OTJhMWFiNTMy
-    MzQzYzc0OGM2NTNkMzJmNmIxY2I4YWY5ODZhYTgzOTAwODUxYWM=
+  metadata.gz: 03367771f9c23e04d3bd88cd191022e90236361a45ede08dd904915ba2b1df890384cdf7e979553dd5defa5e1e705a6af4898732be4a235ed5bd315769ac8aaa
+  data.tar.gz: 0694642bc5d96be78358761f60724d2b68adf05c2d5c05c670b929a8696751baaa691a851b488a8551ac404bdb31cf21bf0089ec863f98fb7f0487dc81bf194d

data/CHANGELOG.md

@@ -0,0 +1,117 @@
+## 3.0.0
+  - Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141
+## 2.7.0
+ - Add `pipeline` configuration option for setting an ingest pipeline to run upon indexing
+
+## 2.6.2
+ - Fix bug where update index actions would not work with events with 'data' field
+
+## 2.6.1
+ - Add 'retry_on_conflict' configuration option which should have been here from the beginning
+ 
+## 2.5.2
+ - Fix bug with update document with doc_as_upsert and scripting (#364, #359)
+ - Make error messages more verbose and easier to parse by humans
+ - Retryable failures are now logged at the info level instead of warning. (issue #372)
+ 
+## 2.5.1
+ - Fix bug where SSL would sometimes not be enabled
+
+## 2.5.0
+ - Host settings now are more robust to bad input
+ - Host settings can now take full URLs
+
+## 2.4.2
+ - Make flush_size actually cap the batch size in LS 2.2+
+
+## 2.4.1
+ - Used debug level instead of info when emitting flush log message
+ - Updated docs about template
+
+## 2.4.0
+ - Scripted update support courtesy of @Da-Wei
+
+## 2.3.2
+ - Fix bug where max_retry_interval was not respected for HTTP error codes
+
+## 2.3.1
+ - Bump manticore dependenvy to 0.5.2
+
+## 2.3.0
+ - Now retry too busy and service unavailable errors infinitely.
+ - Never retry conflict errors
+ - Fix broken delete verb that would fail due to sending body with verb
+
+## 2.2.0
+ - Serialize access to the connection pool in es-ruby client
+ - Add support for parent relationship
+
+## 2.1.5
+ - Sprintf style 'action' parameters no longer raise a LogStash::ConfigurationError
+
+## 2.1.4
+ - Improved the default template to disable fielddata on analyzed string fields. #309
+ - Dependend on logstash-core 2.0.0 released version, rather than RC1
+
+## 2.1.3
+ - Improved the default template to use doc_values wherever possible.
+ - Template contains example mappings for every numeric type. You must map your
+   own fields to make use of anything other than long and double.
+
+## 2.1.2
+ - Fixed dependencies (#280)
+ - Fixed an RSpec test (#281)
+
+## 2.1.1
+ - Made host config obsolete.
+
+## 2.1.0
+ - New setting: timeout. This lets you control the behavior of a slow/stuck
+   request to Elasticsearch that could be, for example, caused by network,
+   firewall, or load balancer issues.
+
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+
+## 2.0.0-beta2
+ - Massive internal refactor of client handling
+ - Background HTTP sniffing support
+ - Reduced bulk request size to 500 from 5000 (better memory utilization)
+ - Removed 'host' config option. Now use 'hosts'
+
+## 2.0.0-beta
+ - Only support HTTP Protocol
+ - Removed support for node and transport protocols (now in logstash-output-elasticsearch_java)
+
+## 1.0.7
+ - Add update API support
+
+## 1.0.6
+ - Fix warning about Concurrent lib deprecation
+
+## 1.0.4
+ - Update to Elasticsearch 1.7
+
+## 1.0.3
+ - Add HTTP proxy support
+
+## 1.0.2
+ - Upgrade Manticore HTTP Client
+
+## 1.0.1
+ - Allow client certificates
+
+## 0.2.9
+ - Add 'path' parameter for ES HTTP hosts behind a proxy on a subpath
+
+## 0.2.8 (June 12, 2015)
+ - Add option to enable and disable SSL certificate verification during handshake (#160)
+ - Doc improvements for clarifying round robin behavior using hosts config
+
+## 0.2.7 (May 28, 2015)
+ - Bump es-ruby version to 1.0.10
+
+## 0.2.6 (May 28, 2015)
+ - Disable timeouts when using http protocol which would cause bulk requests to fail (#103)

data/CONTRIBUTORS

@@ -0,0 +1,32 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Bob Corsaro (dokipen)
+* Colin Surprenant (colinsurprenant)
+* Dmitry Koprov (dkoprov)
+* Graham Bleach (bleach)
+* Hao Chen (haoch)
+* Ivan Babrou (bobrik)
+* James Turnbull (jamtur01)
+* John E. Vincent (lusis)
+* Jordan Sissel (jordansissel)
+* João Duarte (jsvd)
+* Kurt Hurtado (kurtado)
+* Miah Johnson (miah)
+* Pere Urbón (purbon)
+* Pete Fritchman (fetep)
+* Pier-Hugues Pellerin (ph)
+* Raymond Feng (raymondfeng)
+* Richard Pijnenburg (electrical)
+* Spenser Jones (SpenserJ)
+* Suyog Rao (suyograo)
+* Tal Levy (talevy)
+* Tom Hodder (tolland)
+* jimmyjones2
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -1,4 +1,4 @@
-source 'http://rubygems.org'
-gem 'rake'
-gem 'gem_publisher'
-gem 'archive-tar-minitar'
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in logstash-mass_effect.gemspec
+gemspec

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012-2014 Elasticsearch <http://www.elasticsearch.org>
+Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,110 @@
+# Logstash Plugin
+
+[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-output-elasticsearch.svg)](https://travis-ci.org/logstash-plugins/logstash-output-elasticsearch)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run unit tests
+
+```sh
+bundle exec rspec
+```
+
+- Run integration tests
+
+Dependencies: [Docker](http://docker.com)
+
+Before the test suite is run, we will load and run an
+Elasticsearch instance within a docker container. This container 
+will be cleaned up when suite has finished.
+
+```sh
+bundle exec rspec --tag integration
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/elasticsearch.rb

@@ -3,467 +3,139 @@ require "logstash/namespace"
 require "logstash/environment"
 require "logstash/outputs/base"
 require "logstash/json"
+require "concurrent"
 require "stud/buffer"
 require "socket" # for Socket.gethostname
+require "thread" # for safe queueing
 require "uri" # for escaping user input
-require 'logstash-output-elasticsearch_jars.rb'
 
-# This output lets you store logs in Elasticsearch and is the most recommended
-# output for Logstash. If you plan on using the Kibana web interface, you'll
-# need to use this output.
+# This plugin is the recommended method of storing logs in Elasticsearch.
+# If you plan on using the Kibana web interface, you'll want to use this output.
 #
-#   *VERSION NOTE*: Your Elasticsearch cluster must be running Elasticsearch
-#   1.0.0 or later.
+# This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0.
+# We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower,
+# yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having
+# to upgrade Logstash in lock-step. For those still wishing to use the node or transport protocols please see
+# the <<plugins-outputs-elasticsearch_java,elasticsearch_java output plugin>>.
 #
-# If you want to set other Elasticsearch options that are not exposed directly
-# as configuration options, there are two methods:
+# You can learn more about Elasticsearch at <https://www.elastic.co/products/elasticsearch>
 #
-# * Create an `elasticsearch.yml` file in the $PWD of the Logstash process
-# * Pass in es.* java properties (`java -Des.node.foo=` or `ruby -J-Des.node.foo=`)
+# ==== Retry Policy
 #
-# With the default `protocol` setting ("node"), this plugin will join your
-# Elasticsearch cluster as a client node, so it will show up in Elasticsearch's
-# cluster status.
+# The retry policy has changed significantly in the 2.2.0 release.
+# This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience
+# either partial or total failures.
 #
-# You can learn more about Elasticsearch at <http://www.elasticsearch.org>
+# The following errors are retried infinitely:
 #
-# ## Operational Notes
+# - Network errors (inability to connect)
+# - 429 (Too many requests) and
+# - 503 (Service unavailable) errors
 #
-# If using the default `protocol` setting ("node"), your firewalls might need
-# to permit port 9300 in *both* directions (from Logstash to Elasticsearch, and
-# Elasticsearch to Logstash)
+# NOTE: 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions.
+# It is more performant for Elasticsearch to retry these exceptions than this plugin.
+#
+# ==== DNS Caching
+#
+# This plugin uses the JVM to lookup DNS entries and is subject to the value of https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl],
+# a global setting for the JVM.
+#
+# As an example, to set your DNS TTL to 1 second you would set
+# the `LS_JAVA_OPTS` environment variable to `-Dnetwordaddress.cache.ttl=1`.
+#
+# Keep in mind that a connection with keepalive enabled will
+# not reevaluate its DNS value while the keepalive is in effect.
 class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
-  include Stud::Buffer
-
-  config_name "elasticsearch"
-  milestone 3
-
-  # The index to write events to. This can be dynamic using the `%{foo}` syntax.
-  # The default value will partition your indices by day so you can more easily
-  # delete old data or only search specific date ranges.
-  # Indexes may not contain uppercase characters.
-  # For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}
-  config :index, :validate => :string, :default => "logstash-%{+YYYY.MM.dd}"
-
-  # The index type to write events to. Generally you should try to write only
-  # similar events to the same 'type'. String expansion `%{foo}` works here.
-  config :index_type, :validate => :string
-
-  # Starting in Logstash 1.3 (unless you set option `manage_template` to false)
-  # a default mapping template for Elasticsearch will be applied, if you do not
-  # already have one set to match the index pattern defined (default of
-  # `logstash-%{+YYYY.MM.dd}`), minus any variables.  For example, in this case
-  # the template will be applied to all indices starting with `logstash-*`
-  #
-  # If you have dynamic templating (e.g. creating indices based on field names)
-  # then you should set `manage_template` to false and use the REST API to upload
-  # your templates manually.
-  config :manage_template, :validate => :boolean, :default => true
-
-  # This configuration option defines how the template is named inside Elasticsearch.
-  # Note that if you have used the template management features and subsequently
-  # change this, you will need to prune the old template manually, e.g.
-  #
-  # `curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>`
-  #
-  # where `OldTemplateName` is whatever the former setting was.
-  config :template_name, :validate => :string, :default => "logstash"
-
-  # You can set the path to your own template here, if you so desire.
-  # If not set, the included template will be used.
-  config :template, :validate => :path
-
-  # Overwrite the current template with whatever is configured
-  # in the `template` and `template_name` directives.
-  config :template_overwrite, :validate => :boolean, :default => false
-
-  # The document ID for the index. Useful for overwriting existing entries in
-  # Elasticsearch with the same ID.
-  config :document_id, :validate => :string, :default => nil
-
-  # The name of your cluster if you set it on the Elasticsearch side. Useful
-  # for discovery.
-  config :cluster, :validate => :string
-
-  # The hostname or IP address of the host to use for Elasticsearch unicast discovery
-  # This is only required if the normal multicast/cluster discovery stuff won't
-  # work in your environment.
-  #
-  #     `"127.0.0.1"`
-  #     `["127.0.0.1:9300","127.0.0.2:9300"]`
-  config :host, :validate => :array
-
-  # The port for Elasticsearch transport to use.
-  #
-  # If you do not set this, the following defaults are used:
-  # * `protocol => http` - port 9200
-  # * `protocol => transport` - port 9300-9305
-  # * `protocol => node` - port 9300-9305
-  config :port, :validate => :string
+  require "logstash/outputs/elasticsearch/http_client"
+  require "logstash/outputs/elasticsearch/http_client_builder"
+  require "logstash/outputs/elasticsearch/common_configs"
+  require "logstash/outputs/elasticsearch/common"
 
-  # The name/address of the host to bind to for Elasticsearch clustering
-  config :bind_host, :validate => :string
+  # Protocol agnostic (i.e. non-http, non-java specific) configs go here
+  include(LogStash::Outputs::ElasticSearch::CommonConfigs)
 
-  # This is only valid for the 'node' protocol.
-  #
-  # The port for the node to listen on.
-  config :bind_port, :validate => :number
-
-  # Run the Elasticsearch server embedded in this process.
-  # This option is useful if you want to run a single Logstash process that
-  # handles log processing and indexing; it saves you from needing to run
-  # a separate Elasticsearch process.
-  config :embedded, :validate => :boolean, :default => false
-
-  # If you are running the embedded Elasticsearch server, you can set the http
-  # port it listens on here; it is not common to need this setting changed from
-  # default.
-  config :embedded_http_port, :validate => :string, :default => "9200-9300"
-
-  # This setting no longer does anything. It exists to keep config validation
-  # from failing. It will be removed in future versions.
-  config :max_inflight_requests, :validate => :number, :default => 50, :deprecated => true
-
-  # The node name Elasticsearch will use when joining a cluster.
-  #
-  # By default, this is generated internally by the ES client.
-  config :node_name, :validate => :string
+  # Protocol agnostic methods
+  include(LogStash::Outputs::ElasticSearch::Common)
 
-  # This plugin uses the bulk index api for improved indexing performance.
-  # To make efficient bulk api calls, we will buffer a certain number of
-  # events before flushing that out to Elasticsearch. This setting
-  # controls how many events will be buffered before sending a batch
-  # of events.
-  config :flush_size, :validate => :number, :default => 5000
-
-  # The amount of time since last flush before a flush is forced.
-  #
-  # This setting helps ensure slow event rates don't get stuck in Logstash.
-  # For example, if your `flush_size` is 100, and you have received 10 events,
-  # and it has been more than `idle_flush_time` seconds since the last flush,
-  # Logstash will flush those 10 events automatically.
-  #
-  # This helps keep both fast and slow log streams moving along in
-  # near-real-time.
-  config :idle_flush_time, :validate => :number, :default => 1
-
-  # Choose the protocol used to talk to Elasticsearch.
-  #
-  # The 'node' protocol will connect to the cluster as a normal Elasticsearch
-  # node (but will not store data). This allows you to use things like
-  # multicast discovery. If you use the `node` protocol, you must permit
-  # bidirectional communication on the port 9300 (or whichever port you have
-  # configured).
-  #
-  # The 'transport' protocol will connect to the host you specify and will
-  # not show up as a 'node' in the Elasticsearch cluster. This is useful
-  # in situations where you cannot permit connections outbound from the
-  # Elasticsearch cluster to this Logstash server.
-  #
-  # The 'http' protocol will use the Elasticsearch REST/HTTP interface to talk
-  # to elasticsearch.
-  #
-  # All protocols will use bulk requests when talking to Elasticsearch.
-  #
-  # The default `protocol` setting under java/jruby is "node". The default
-  # `protocol` on non-java rubies is "http"
-  config :protocol, :validate => [ "node", "transport", "http" ]
+  config_name "elasticsearch"
 
-  # The Elasticsearch action to perform. Valid actions are: `index`, `delete`.
-  #
-  # Use of this setting *REQUIRES* you also configure the `document_id` setting
-  # because `delete` actions all require a document id.
-  #
-  # What does each action do?
+  # The Elasticsearch action to perform. Valid actions are:
   #
   # - index: indexes a document (an event from Logstash).
-  # - delete: deletes a document by id
-  #
-  # For more details on actions, check out the http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
+  # - delete: deletes a document by id (An id is required for this action)
+  # - create: indexes a document, fails if a document by that id already exists in the index.
+  # - update: updates a document by id. Update has a special case where you can upsert -- update a
+  #   document if not already present. See the `upsert` option
+  # - A sprintf style string to change the action based on the content of the event. The value `%{[foo]}`
+  #   would use the foo field for the action
+  #
+  # For more details on actions, check out the http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
   config :action, :validate => :string, :default => "index"
 
-  # Username and password (HTTP only)
+  # Username to authenticate to a secure Elasticsearch cluster
   config :user, :validate => :string
+  # Password to authenticate to a secure Elasticsearch cluster
   config :password, :validate => :password
 
-  # SSL Configurations (HTTP only)
-  #
-  # Enable SSL
-  config :ssl, :validate => :boolean, :default => false
+  # HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps
+  # the root path for the Elasticsearch HTTP API lives.
+  config :path, :validate => :string, :default => "/"
+
+  # Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme
+  # is specified in the URLs listed in 'hosts'. If no explicit protocol is specified plain HTTP will be used.
+  # If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts'
+  config :ssl, :validate => :boolean
+
+  # Option to validate the server's certificate. Disabling this severely compromises security.
+  # For more information on disabling certificate verification please read
+  # https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
+  config :ssl_certificate_verification, :validate => :boolean, :default => true
 
   # The .cer or .pem file to validate the server's certificate
   config :cacert, :validate => :path
 
-  # The JKS truststore to validate the server's certificate
+  # The JKS truststore to validate the server's certificate.
   # Use either `:truststore` or `:cacert`
   config :truststore, :validate => :path
 
   # Set the truststore password
   config :truststore_password, :validate => :password
 
-  # helper function to replace placeholders
-  # in index names to wildcards
-  # example:
-  #    "logs-%{YYYY}" -> "logs-*"
-  def wildcard_substitute(name)
-    name.gsub(/%\{[^}]+\}/, "*")
-  end
-
-  public
-  def register
-    client_settings = {}
-
-    if @protocol.nil?
-      @protocol = LogStash::Environment.jruby? ? "node" : "http"
-    end
-
-    if ["node", "transport"].include?(@protocol)
-      # Node or TransportClient; requires JRuby
-      raise(LogStash::PluginLoadingError, "This configuration requires JRuby. If you are not using JRuby, you must set 'protocol' to 'http'. For example: output { elasticsearch { protocol => \"http\" } }") unless LogStash::Environment.jruby?
-
-      client_settings["cluster.name"] = @cluster if @cluster
-      client_settings["network.host"] = @bind_host if @bind_host
-      client_settings["transport.tcp.port"] = @bind_port if @bind_port
- 
-      if @node_name
-        client_settings["node.name"] = @node_name
-      else
-        client_settings["node.name"] = "logstash-#{Socket.gethostname}-#{$$}-#{object_id}"
-      end
-
-      @@plugins.each do |plugin|
-        name = plugin.name.split('-')[-1]
-        client_settings.merge!(LogStash::Outputs::ElasticSearch.const_get(name.capitalize).create_client_config(self))
-      end
-    end
-
-    require "logstash/outputs/elasticsearch/protocol"
-
-    if @port.nil?
-      @port = case @protocol
-        when "http"; "9200"
-        when "transport", "node"; "9300-9305"
-      end
-    end
-
-    if @host.nil? && @protocol == "http"
-      @logger.info("No 'host' set in elasticsearch output. Defaulting to localhost")
-      @host = ["localhost"]
-    end
-
-    client_settings.merge! setup_ssl()
-
-    common_options = {
-      :protocol => @protocol,
-      :client_settings => client_settings
-    }
-
-    common_options.merge! setup_basic_auth()
-
-    client_class = case @protocol
-      when "transport"
-        LogStash::Outputs::Elasticsearch::Protocols::TransportClient
-      when "node"
-        LogStash::Outputs::Elasticsearch::Protocols::NodeClient
-      when /http/
-        LogStash::Outputs::Elasticsearch::Protocols::HTTPClient
-    end
-
-    if @embedded
-      raise(LogStash::ConfigurationError, "The 'embedded => true' setting is only valid for the elasticsearch output under JRuby. You are running #{RUBY_DESCRIPTION}") unless LogStash::Environment.jruby?
-#      LogStash::Environment.load_elasticsearch_jars!
-
-      # Default @host with embedded to localhost. This should help avoid
-      # newbies tripping on ubuntu and other distros that have a default
-      # firewall that blocks multicast.
-      @host ||= ["localhost"]
-
-      # Start Elasticsearch local.
-      start_local_elasticsearch
-    end
-
-    @client = Array.new
-
-    if protocol == "node" or @host.nil? # if @protocol is "node" or @host is not set
-      options = {
-          :host => @host,
-          :port => @port,
-      }.merge(common_options)
-      @client << client_class.new(options)
-    else # if @protocol in ["transport","http"]
-      @host.each do |host|
-          (_host,_port) = host.split ":"
-          options = {
-            :host => _host,
-            :port => _port || @port,
-          }.merge(common_options)
-          @logger.info "Create client to elasticsearch server on #{_host}:#{_port}"
-          @client << client_class.new(options)
-      end # @host.each
-    end
-
-    if @manage_template
-      for client in @client
-          begin
-            @logger.info("Automatic template management enabled", :manage_template => @manage_template.to_s)
-            client.template_install(@template_name, get_template, @template_overwrite)
-            break
-          rescue => e
-            @logger.error("Failed to install template: #{e.message}")
-          end
-      end # for @client loop
-    end # if @manage_templates
-
-    @logger.info("New Elasticsearch output", :cluster => @cluster,
-                 :host => @host, :port => @port, :embedded => @embedded,
-                 :protocol => @protocol)
+  # The keystore used to present a certificate to the server.
+  # It can be either .jks or .p12
+  config :keystore, :validate => :path
 
-    @client_idx = 0
-    @current_client = @client[@client_idx]
-
-    buffer_initialize(
-      :max_items => @flush_size,
-      :max_interval => @idle_flush_time,
-      :logger => @logger
-    )
-  end # def register
-
-  protected
-  def shift_client
-    @client_idx = (@client_idx+1) % @client.length
-    @current_client = @client[@client_idx]
-    @logger.debug? and @logger.debug("Switched current elasticsearch client to ##{@client_idx} at #{@host[@client_idx]}")
-  end
-
-  private
-  def setup_ssl
-    return {} unless @ssl
-    if @protocol != "http"
-      raise(LogStash::ConfigurationError, "SSL is not supported for '#{@protocol}'. Change the protocol to 'http' if you need SSL.")
-    end
-    @protocol = "https"
-    if @cacert && @truststore
-      raise(LogStash::ConfigurationError, "Use either \"cacert\" or \"truststore\" when configuring the CA certificate") if @truststore
-    end
-    ssl_options = {}
-    if @cacert then
-      @truststore, ssl_options[:truststore_password] = generate_jks @cacert
-    elsif @truststore
-      ssl_options[:truststore_password] = @truststore_password.value if @truststore_password
-    end
-    ssl_options[:truststore] = @truststore
-    { ssl: ssl_options }
-  end
-
-  private
-  def setup_basic_auth
-    return {} unless @user && @password
-
-    if @protocol =~ /http/
-      {
-        :user => ::URI.escape(@user, "@:"),
-        :password => ::URI.escape(@password.value, "@:")
-      }
-    else
-      raise(LogStash::ConfigurationError, "User and password parameters are not supported for '#{@protocol}'. Change the protocol to 'http' if you need them.")
-    end
-  end
-
-  public
-  def get_template
-    if @template.nil?
-      @template = ::File.expand_path('elasticsearch/elasticsearch-template.json', ::File.dirname(__FILE__))
-      if !File.exists?(@template)
-        raise "You must specify 'template => ...' in your elasticsearch output (I looked for '#{@template}')"
-      end
-    end
-    template_json = IO.read(@template).gsub(/\n/,'')
-    template = LogStash::Json.load(template_json)
-    template['template'] = wildcard_substitute(@index)
-    @logger.info("Using mapping template", :template => template)
-    return template
-  end # def get_template
-
-  protected
-  def start_local_elasticsearch
-    @logger.info("Starting embedded Elasticsearch local node.")
-    builder = org.elasticsearch.node.NodeBuilder.nodeBuilder
-    # Disable 'local only' - LOGSTASH-277
-    #builder.local(true)
-    builder.settings.put("cluster.name", @cluster) if @cluster
-    builder.settings.put("node.name", @node_name) if @node_name
-    builder.settings.put("network.host", @bind_host) if @bind_host
-    builder.settings.put("http.port", @embedded_http_port)
-
-    @embedded_elasticsearch = builder.node
-    @embedded_elasticsearch.start
-  end # def start_local_elasticsearch
-
-  private
-  def generate_jks cert_path
-
-    require 'securerandom'
-    require 'tempfile'
-    require 'java'
-    import java.io.FileInputStream
-    import java.io.FileOutputStream
-    import java.security.KeyStore
-    import java.security.cert.CertificateFactory
-
-    jks = java.io.File.createTempFile("cert", ".jks")
-
-    ks = KeyStore.getInstance "JKS"
-    ks.load nil, nil
-    cf = CertificateFactory.getInstance "X.509"
-    cert = cf.generateCertificate FileInputStream.new(cert_path)
-    ks.setCertificateEntry "cacert", cert
-    pwd = SecureRandom.urlsafe_base64(9)
-    ks.store FileOutputStream.new(jks), pwd.to_java.toCharArray
-    [jks.path, pwd]
+  # Set the truststore password
+  config :keystore_password, :validate => :password
+
+  # This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list.
+  # Note: This will return ALL nodes with HTTP enabled (including master nodes!). If you use
+  # this with master nodes, you probably want to disable HTTP on them by setting
+  # `http.enabled` to false in their elasticsearch.yml. You can either use the `sniffing` option or
+  # manually enter multiple Elasticsearch hosts using the `hosts` parameter.
+  config :sniffing, :validate => :boolean, :default => false
+
+  # How long to wait, in seconds, between sniffing attempts
+  config :sniffing_delay, :validate => :number, :default => 5
+
+  # Set the address of a forward HTTP proxy.
+  # Can be either a string, such as `http://localhost:123` or a hash in the form
+  # of `{host: 'proxy.org' port: 80 scheme: 'http'}`.
+  # Note, this is NOT a SOCKS proxy, but a plain HTTP proxy
+  config :proxy
+
+  # Set the timeout for network operations and requests sent Elasticsearch. If
+  # a timeout occurs, the request will be retried.
+  config :timeout, :validate => :number
+
+  def build_client
+    @client = ::LogStash::Outputs::ElasticSearch::HttpClientBuilder.build(@logger, @hosts, params)
   end
 
-  public
-  def receive(event)
-    return unless output?(event)
-
-    # Set the 'type' value for the index.
-    if @index_type
-      type = event.sprintf(@index_type)
-    else
-      type = event["type"] || "logs"
-    end
-
-    index = event.sprintf(@index)
-
-    document_id = @document_id ? event.sprintf(@document_id) : nil
-    buffer_receive([event.sprintf(@action), { :_id => document_id, :_index => index, :_type => type }, event.to_hash])
-  end # def receive
-
-  def flush(actions, teardown=false)
-    begin
-      @logger.debug? and @logger.debug "Sending bulk of actions to client[#{@client_idx}]: #{@host[@client_idx]}"
-      @current_client.bulk(actions)
-    rescue => e
-      @logger.error "Got error to send bulk of actions to elasticsearch server at #{@host[@client_idx]} : #{e.message}"
-      raise e
-    ensure
-      unless @protocol == "node"
-          @logger.debug? and @logger.debug "Shifting current elasticsearch client"
-          shift_client
-      end
-    end
-    # TODO(sissel): Handle errors. Since bulk requests could mostly succeed
-    # (aka partially fail), we need to figure out what documents need to be
-    # retried.
-    #
-    # In the worst case, a failing flush (exception) will incur a retry from Stud::Buffer.
-  end # def flush
-
-  def teardown
-    if @cacert # remove temporary jks store created from the cacert
-      File.delete(@truststore)
-    end
-    buffer_flush(:final => true)
+  def close
+    @stopping.make_true
+    @client.stop_sniffing!
+    @buffer.stop
   end
 
   @@plugins = Gem::Specification.find_all{|spec| spec.name =~ /logstash-output-elasticsearch-/ }