logstash-output-elasticsearch 0.1.9-java → 0.1.10-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 65e766f91594739db3a82e79d4a1d34d5f9a8018
-  data.tar.gz: d723f3674448b3fa5417911bd62602b0550a2c19
+  metadata.gz: 90b616b39de31794c9b9047c5eccf544908752a0
+  data.tar.gz: b28c3292d533055b0c540d16bf0390e90fba212f
 SHA512:
-  metadata.gz: b633a076b3332505f3d9578539ccb47c9e06343dd0c3616f65fdd4e591e16e159a283616b9fede371474278e8e10ffad77e9381d06650b6dad3bbf21d627cf3b
-  data.tar.gz: edd6a187ada5d1270eb01d57c3ae6cd5b79b1cf0e5ec0638ef42282042bed349832fa739c3a471bb6ca2026696c8431b416a210980abab553cf315215d71a6ce
+  metadata.gz: e8a86c367834828819f246adc6e6a19d896bef9d6f2570c68aaeb578830357725639c27561e73dd5f85410740dc6cfd21aaf1622daabe2e0ed7469690338ffc5
+  data.tar.gz: edb1fd8e7d447a68a9ac4261398771a641b7dc9c8b8022fddf486bb2328764b1415de58c0c6ce81368d1703c80d593ea56e51bcd5febf098b51a253b38c5425c

data/CONTRIBUTORS

@@ -0,0 +1,31 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Bob Corsaro (dokipen)
+* Colin Surprenant (colinsurprenant)
+* Dmitry Koprov (dkoprov)
+* Graham Bleach (bleach)
+* Hao Chen (haoch)
+* James Turnbull (jamtur01)
+* John E. Vincent (lusis)
+* Jordan Sissel (jordansissel)
+* João Duarte (jsvd)
+* Kurt Hurtado (kurtado)
+* Miah Johnson (miah)
+* Pere Urbón (purbon)
+* Pete Fritchman (fetep)
+* Pier-Hugues Pellerin (ph)
+* Raymond Feng (raymondfeng)
+* Richard Pijnenburg (electrical)
+* Spenser Jones (SpenserJ)
+* Suyog Rao (suyograo)
+* Tal Levy (talevy)
+* Tom Hodder (tolland)
+* jimmyjones2
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012-2014 Elasticsearch <http://www.elasticsearch.org>
+Copyright (c) 2012-2015 Elasticsearch <http://www.elasticsearch.org>
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

data/README.md

@@ -0,0 +1,95 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the logstash-users@googlegroups.com mailing list.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization.
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+```sh
+bundle exec rspec
+```
+
+The Logstash code required to run the tests/specs is specified in the `Gemfile` by the line similar to:
+```ruby
+gem "logstash", :github => "elasticsearch/logstash", :branch => "1.5"
+```
+To test against another version or a local Logstash, edit the `Gemfile` to specify an alternative location, for example:
+```ruby
+gem "logstash", :github => "elasticsearch/logstash", :ref => "master"
+```
+```ruby
+gem "logstash", :path => "/your/local/logstash"
+```
+
+Then update your dependencies and run your tests:
+
+```sh
+bundle install
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `tools/Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Update Logstash dependencies
+```sh
+rake vendor:gems
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to me that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/elasticsearch.rb

@@ -3,8 +3,10 @@ require "logstash/namespace"
 require "logstash/environment"
 require "logstash/outputs/base"
 require "logstash/json"
+require "concurrent_ruby"
 require "stud/buffer"
 require "socket" # for Socket.gethostname
+require "thread" # for safe queueing
 require "uri" # for escaping user input
 require 'logstash-output-elasticsearch_jars.rb'
 
@@ -35,7 +37,6 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   include Stud::Buffer
 
   config_name "elasticsearch"
-  milestone 3
 
   # The index to write events to. This can be dynamic using the `%{foo}` syntax.
   # The default value will partition your indices by day so you can more easily
@@ -88,6 +89,11 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # This is only required if the normal multicast/cluster discovery stuff won't
   # work in your environment.
   #
+  # The plugin will use multicast discovery to connect to Elasticsearch
+  # when using `protocol => node` without setting a host.
+  #
+  # http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#multicast[Multicast Discovery Docs]
+  #
   #     `"127.0.0.1"`
   #     `["127.0.0.1:9300","127.0.0.2:9300"]`
   config :host, :validate => :array
@@ -177,6 +183,9 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   #
   # - index: indexes a document (an event from Logstash).
   # - delete: deletes a document by id
+  # - create: indexes a document, fails if a document by that id already exists in the index.
+  # following action is not supported by HTTP protocol
+  # - create_unless_exists: creates a document, fails if no id is provided
   #
   # For more details on actions, check out the http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
   config :action, :validate => :string, :default => "index"
@@ -200,18 +209,33 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   # Set the truststore password
   config :truststore_password, :validate => :password
 
-  # helper function to replace placeholders
-  # in index names to wildcards
-  # example:
-  #    "logs-%{YYYY}" -> "logs-*"
-  def wildcard_substitute(name)
-    name.gsub(/%\{[^}]+\}/, "*")
-  end
+  # Enable cluster sniffing (transport only)
+  # Asks host for the list of all cluster nodes and adds them to the hosts list
+  config :sniffing, :validate => :boolean, :default => false
+
+  # Set max retry for each event
+  config :max_retries, :validate => :number, :default => 3
+
+  # Set retry policy for events that failed to send
+  config :retry_max_items, :validate => :number, :default => 5000
+
+  # Set max interval between bulk retries
+  config :retry_max_interval, :validate => :number, :default => 5
 
   public
   def register
+    @submit_mutex = Mutex.new
+    # retry-specific variables
+    @retry_flush_mutex = Mutex.new
+    @retry_teardown_requested = Concurrent::AtomicBoolean.new(false)
+    # needs flushing when interval
+    @retry_queue_needs_flushing = ConditionVariable.new
+    @retry_queue_not_full = ConditionVariable.new
+    @retry_queue = Queue.new
+
     client_settings = {}
 
+
     if @protocol.nil?
       @protocol = LogStash::Environment.jruby? ? "node" : "http"
     end
@@ -223,6 +247,7 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
       client_settings["cluster.name"] = @cluster if @cluster
       client_settings["network.host"] = @bind_host if @bind_host
       client_settings["transport.tcp.port"] = @bind_port if @bind_port
+      client_settings["client.transport.sniff"] = @sniffing
  
       if @node_name
         client_settings["node.name"] = @node_name
@@ -283,33 +308,27 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
 
     @client = Array.new
 
-    if protocol == "node" or @host.nil? # if @protocol is "node" or @host is not set
-      options = {
-          :host => @host,
-          :port => @port,
-      }.merge(common_options)
-      @client << client_class.new(options)
+    if protocol == "node" || @host.nil? # if @protocol is "node" or @host is not set
+      options = { :host => @host, :port => @port }.merge(common_options)
+      @client = [client_class.new(options)]
     else # if @protocol in ["transport","http"]
-      @host.each do |host|
-          (_host,_port) = host.split ":"
-          options = {
-            :host => _host,
-            :port => _port || @port,
-          }.merge(common_options)
-          @logger.info "Create client to elasticsearch server on #{_host}:#{_port}"
-          @client << client_class.new(options)
-      end # @host.each
+      @client = @host.map do |host|
+        (_host,_port) = host.split ":"
+        options = { :host => _host, :port => _port || @port }.merge(common_options)
+        @logger.info "Create client to elasticsearch server on #{_host}:#{_port}"
+        client_class.new(options)
+      end # @host.map
     end
 
     if @manage_template
       for client in @client
-          begin
-            @logger.info("Automatic template management enabled", :manage_template => @manage_template.to_s)
-            client.template_install(@template_name, get_template, @template_overwrite)
-            break
-          rescue => e
-            @logger.error("Failed to install template: #{e.message}")
-          end
+        begin
+          @logger.info("Automatic template management enabled", :manage_template => @manage_template.to_s)
+          client.template_install(@template_name, get_template, @template_overwrite)
+          break
+        rescue => e
+          @logger.error("Failed to install template: #{e.message}")
+        end
       end # for @client loop
     end # if @manage_templates
 
@@ -325,8 +344,147 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
       :max_interval => @idle_flush_time,
       :logger => @logger
     )
+
+    @retry_timer_thread = Thread.new do
+      loop do
+        sleep(@retry_max_interval)
+        @retry_flush_mutex.synchronize { @retry_queue_needs_flushing.signal }
+      end
+    end
+
+    @retry_thread = Thread.new do
+      while @retry_teardown_requested.false?
+        @retry_flush_mutex.synchronize { @retry_queue_needs_flushing.wait(@retry_flush_mutex) }
+        retry_flush
+      end
+    end
   end # def register
 
+
+  public
+  def get_template
+    if @template.nil?
+      @template = ::File.expand_path('elasticsearch/elasticsearch-template.json', ::File.dirname(__FILE__))
+      if !File.exists?(@template)
+        raise "You must specify 'template => ...' in your elasticsearch output (I looked for '#{@template}')"
+      end
+    end
+    template_json = IO.read(@template).gsub(/\n/,'')
+    template = LogStash::Json.load(template_json)
+    template['template'] = wildcard_substitute(@index)
+    @logger.info("Using mapping template", :template => template)
+    return template
+  end # def get_template
+
+  public
+  def receive(event)
+    return unless output?(event)
+
+    # block until we have not maxed out our 
+    # retry queue. This is applying back-pressure
+    # to slow down the receive-rate
+    @retry_flush_mutex.synchronize {
+      @retry_queue_not_full.wait(@retry_flush_mutex) while @retry_queue.size > @retry_max_items
+    }
+
+    event['@metadata']['retry_count'] = 0
+
+    # Set the 'type' value for the index.
+    type = @index_type ? event.sprintf(@index_type) : (event["type"] || "logs")
+
+    index = event.sprintf(@index)
+
+    document_id = @document_id ? event.sprintf(@document_id) : nil
+    buffer_receive([event.sprintf(@action), { :_id => document_id, :_index => index, :_type => type }, event])
+  end # def receive
+
+  public
+  # synchronize the @current_client.bulk call to avoid concurrency/thread safety issues with the
+  # # client libraries which might not be thread safe. the submit method can be called from both the
+  # # Stud::Buffer flush thread and from our own retry thread.
+  def submit(actions)
+    es_actions = actions.map { |a, doc, event| [a, doc, event.to_hash] }
+    @submit_mutex.lock
+    begin
+      bulk_response = @current_client.bulk(es_actions)
+    ensure
+      @submit_mutex.unlock
+    end
+    if bulk_response["errors"]
+      failed_actions = actions.select.with_index {|_,i| [429, 503].include?(bulk_response['statuses'][i]) }
+      unless failed_actions.empty?
+        @logger.debug "#{failed_actions.size}/#{actions.size} events were unsuccessful in sending"
+        retry_push(failed_actions)
+      end
+    end
+  end
+
+  # Does not raise an exception to prevent Stud::Buffer from 
+  # attempting to resubmit successful actions within the bulk 
+  # request.
+  public
+  def flush(actions, teardown = false)
+    begin
+      @logger.debug? and @logger.debug "Sending bulk of actions to client[#{@client_idx}]: #{@host[@client_idx]}"
+      submit(actions)
+    rescue => e
+      @logger.error "Got error to send bulk of actions to elasticsearch server at #{@host[@client_idx]} : #{e.message}"
+    ensure
+      unless @protocol == "node"
+        @logger.debug? and @logger.debug "Shifting current elasticsearch client"
+        shift_client
+      end
+    end
+  end # def flush
+
+  public
+  def teardown
+    if @cacert # remove temporary jks store created from the cacert
+      File.delete(@truststore)
+    end
+
+    @retry_teardown_requested.make_true
+    # First, make sure retry_timer_thread is stopped
+    # to ensure we do not signal a retry based on 
+    # the retry interval.
+    Thread.kill(@retry_timer_thread)
+    @retry_timer_thread.join
+    # Signal flushing in the case that #retry_flush is in 
+    # the process of waiting for a signal.
+    @retry_flush_mutex.synchronize { @retry_queue_needs_flushing.signal }
+    # Now, #retry_flush is ensured to not be in a state of 
+    # waiting and can be safely joined into the main thread
+    # for further final execution of an in-process remaining call.
+    @retry_thread.join
+
+    # execute any final actions along with a proceeding retry for any 
+    # final actions that did not succeed.
+    buffer_flush(:final => true)
+    retry_flush
+  end
+
+  @@plugins = Gem::Specification.find_all{|spec| spec.name =~ /logstash-output-elasticsearch-/ }
+
+  @@plugins.each do |plugin|
+    name = plugin.name.split('-')[-1]
+    require "logstash/outputs/elasticsearch/#{name}"
+  end
+
+  protected
+  def start_local_elasticsearch
+    @logger.info("Starting embedded Elasticsearch local node.")
+    builder = org.elasticsearch.node.NodeBuilder.nodeBuilder
+    # Disable 'local only' - LOGSTASH-277
+    #builder.local(true)
+    builder.settings.put("cluster.name", @cluster) if @cluster
+    builder.settings.put("node.name", @node_name) if @node_name
+    builder.settings.put("network.host", @bind_host) if @bind_host
+    builder.settings.put("http.port", @embedded_http_port)
+
+    @embedded_elasticsearch = builder.node
+    @embedded_elasticsearch.start
+  end # def start_local_elasticsearch
+
   protected
   def shift_client
     @client_idx = (@client_idx+1) % @client.length
@@ -368,36 +526,6 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
     end
   end
 
-  public
-  def get_template
-    if @template.nil?
-      @template = ::File.expand_path('elasticsearch/elasticsearch-template.json', ::File.dirname(__FILE__))
-      if !File.exists?(@template)
-        raise "You must specify 'template => ...' in your elasticsearch output (I looked for '#{@template}')"
-      end
-    end
-    template_json = IO.read(@template).gsub(/\n/,'')
-    template = LogStash::Json.load(template_json)
-    template['template'] = wildcard_substitute(@index)
-    @logger.info("Using mapping template", :template => template)
-    return template
-  end # def get_template
-
-  protected
-  def start_local_elasticsearch
-    @logger.info("Starting embedded Elasticsearch local node.")
-    builder = org.elasticsearch.node.NodeBuilder.nodeBuilder
-    # Disable 'local only' - LOGSTASH-277
-    #builder.local(true)
-    builder.settings.put("cluster.name", @cluster) if @cluster
-    builder.settings.put("node.name", @node_name) if @node_name
-    builder.settings.put("network.host", @bind_host) if @bind_host
-    builder.settings.put("http.port", @embedded_http_port)
-
-    @embedded_elasticsearch = builder.node
-    @embedded_elasticsearch.start
-  end # def start_local_elasticsearch
-
   private
   def generate_jks cert_path
 
@@ -421,55 +549,49 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
     [jks.path, pwd]
   end
 
-  public
-  def receive(event)
-    return unless output?(event)
-
-    # Set the 'type' value for the index.
-    if @index_type
-      type = event.sprintf(@index_type)
-    else
-      type = event["type"] || "logs"
-    end
-
-    index = event.sprintf(@index)
-
-    document_id = @document_id ? event.sprintf(@document_id) : nil
-    buffer_receive([event.sprintf(@action), { :_id => document_id, :_index => index, :_type => type }, event.to_hash])
-  end # def receive
-
-  def flush(actions, teardown=false)
-    begin
-      @logger.debug? and @logger.debug "Sending bulk of actions to client[#{@client_idx}]: #{@host[@client_idx]}"
-      @current_client.bulk(actions)
-    rescue => e
-      @logger.error "Got error to send bulk of actions to elasticsearch server at #{@host[@client_idx]} : #{e.message}"
-      raise e
-    ensure
-      unless @protocol == "node"
-          @logger.debug? and @logger.debug "Shifting current elasticsearch client"
-          shift_client
-      end
+  private
+  # in charge of submitting any actions in @retry_queue that need to be 
+  # retried
+  #
+  # This method is not called concurrently. It is only called by @retry_thread
+  # and once that thread is ended during the teardown process, a final call 
+  # to this method is done upon teardown in the main thread.
+  def retry_flush()
+    unless @retry_queue.empty?
+      buffer = @retry_queue.size.times.map do
+        next_action, next_doc, next_event = @retry_queue.pop
+        next_event['@metadata']['retry_count'] += 1
+
+        if next_event['@metadata']['retry_count'] > @max_retries
+          @logger.error "too many attempts at sending event. dropping: #{next_event}"
+          nil
+        else
+          [next_action, next_doc, next_event]
+        end
+      end.compact
+
+      submit(buffer) unless buffer.empty?
     end
-    # TODO(sissel): Handle errors. Since bulk requests could mostly succeed
-    # (aka partially fail), we need to figure out what documents need to be
-    # retried.
-    #
-    # In the worst case, a failing flush (exception) will incur a retry from Stud::Buffer.
-  end # def flush
 
-  def teardown
-    if @cacert # remove temporary jks store created from the cacert
-      File.delete(@truststore)
-    end
-    buffer_flush(:final => true)
+    @retry_flush_mutex.synchronize {
+      @retry_queue_not_full.signal if @retry_queue.size < @retry_max_items
+    }
   end
 
-  @@plugins = Gem::Specification.find_all{|spec| spec.name =~ /logstash-output-elasticsearch-/ }
-
-  @@plugins.each do |plugin|
-    name = plugin.name.split('-')[-1]
-    require "logstash/outputs/elasticsearch/#{name}"
+  private
+  def retry_push(actions)
+    Array(actions).each{|action| @retry_queue << action}
+    @retry_flush_mutex.synchronize {
+      @retry_queue_needs_flushing.signal if @retry_queue.size >= @retry_max_items
+    }
   end
 
+  # helper function to replace placeholders
+  # in index names to wildcards
+  # example:
+  #    "logs-%{YYYY}" -> "logs-*"
+  private
+  def wildcard_substitute(name)
+    name.gsub(/%\{[^}]+\}/, "*")
+  end
 end # class LogStash::Outputs::Elasticsearch

data/lib/logstash/outputs/elasticsearch/protocol.rb

@@ -79,13 +79,18 @@ module LogStash::Outputs::Elasticsearch
       end
 
       def bulk(actions)
-        @client.bulk(:body => actions.collect do |action, args, source|
+        bulk_response = @client.bulk(:body => actions.collect do |action, args, source|
           if source
             next [ { action => args }, source ]
           else
             next { action => args }
           end
         end.flatten)
+        if bulk_response["errors"]
+          return {"errors" => true, "statuses" => bulk_response["items"].map { |i| i["status"] }}
+        else
+          return {"errors" => false}
+        end
       end # def bulk
 
       def template_exists?(name)
@@ -190,7 +195,14 @@ module LogStash::Outputs::Elasticsearch
         end
         response = prep.execute.actionGet()
 
-        # TODO(sissel): What format should the response be in?
+        if response.has_failures()
+          return {"errors" => true,
+                  "statuses" => response.map { |i| (i.is_failed && i.get_failure.get_status.get_status) || 200 }}
+        else
+          return {"errors" => false}
+        end
+        # returns 200 for all successful actions, represents 201 & 200
+        # TODO(talevy): parse item response objects to retrieve correct 200 (OK) or 201(created) status codes
       end # def bulk
 
       def build_request(action, args, source)
@@ -202,8 +214,23 @@ module LogStash::Outputs::Elasticsearch
           when "delete"
             request = org.elasticsearch.action.delete.DeleteRequest.new(args[:_index])
             request.id(args[:_id])
+          when "create"
+            request = org.elasticsearch.action.index.IndexRequest.new(args[:_index])
+            request.id(args[:_id]) if args[:_id]
+            request.source(source)
+            request.opType("create")
+          when "create_unless_exists"
+            unless args[:_id].nil?
+              request = org.elasticsearch.action.index.IndexRequest.new(args[:_index])
+              request.id(args[:_id])
+              request.source(source)
+              request.opType("create")
+            else
+              raise(LogStash::ConfigurationError, "Specifying action => 'create_unless_exists' without a document '_id' is not supported.")
+            end
+          else
+            raise(LogStash::ConfigurationError, "action => '#{action_name}' is not currently supported.")
           #when "update"
-          #when "create"
         end # case action
 
         request.type(args[:_type]) if args[:_type]

data/logstash-output-elasticsearch.gemspec

@@ -1,8 +1,8 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-output-elasticsearch'
-  s.version         = '0.1.9'
-  s.licenses        = ['Apache License (2.0)']
+  s.version         = '0.1.10'
+  s.licenses        = ['apache-2.0']
   s.summary         = "Logstash Output to Elasticsearch"
   s.description     = "Output events to elasticsearch"
   s.authors         = ["Elasticsearch"]
@@ -23,6 +23,7 @@ Gem::Specification.new do |s|
   s.requirements << "jar 'org.elasticsearch:elasticsearch', '1.4.0'"
 
   # Gem dependencies
+  s.add_runtime_dependency 'concurrent-ruby'
   s.add_runtime_dependency 'elasticsearch', ['>= 1.0.6', '~> 1.0']
   s.add_runtime_dependency 'stud', ['>= 0.0.17', '~> 0.0']
   s.add_runtime_dependency 'cabin', ['~> 0.6']

data/spec/outputs/elasticsearch_spec.rb

@@ -2,6 +2,7 @@ require "logstash/devutils/rspec/spec_helper"
 require "ftw"
 require "logstash/plugin"
 require "logstash/json"
+require "stud/try"
 
 describe "outputs/elasticsearch" do
 
@@ -72,6 +73,103 @@ describe "outputs/elasticsearch" do
     end
   end
 
+  describe "node client create actions", :elasticsearch => true do
+    require "logstash/outputs/elasticsearch"
+    require "elasticsearch"
+    let(:es) { Elasticsearch::Client.new }
+
+    def get_es_output(action, id = nil)
+      settings = {
+        "manage_template" => true,
+        "index" => "logstash-create",
+        "template_overwrite" => true,
+        "protocol" => "node",
+        "host" => "localhost",
+        "action" => action
+      }
+      settings['document_id'] = id unless id.nil?
+      LogStash::Outputs::ElasticSearch.new(settings)
+    end
+
+    before :each do
+      # Delete all templates first.
+      # Clean ES of data before we start.
+      es.indices.delete_template(:name => "*")
+      # This can fail if there are no indexes, ignore failure.
+      es.indices.delete(:index => "*") rescue nil
+    end
+
+    context "when action => create" do
+      it "should create new documents with or without id" do
+        subject = get_es_output("create", "id123")
+        subject.register
+        subject.receive(LogStash::Event.new("message" => "sample message here"))
+        subject.buffer_flush(:final => true)
+        es.indices.refresh
+        # Wait or fail until everything's indexed.
+        Stud::try(3.times) do
+          r = es.search
+          insist { r["hits"]["total"] } == 1
+        end
+      end
+
+      it "should creat new documents without id" do
+        subject = get_es_output("create")
+        subject.register
+        subject.receive(LogStash::Event.new("message" => "sample message here"))
+        subject.buffer_flush(:final => true)
+        es.indices.refresh
+        # Wait or fail until everything's indexed.
+        Stud::try(3.times) do
+          r = es.search
+          insist { r["hits"]["total"] } == 1
+        end
+      end
+    end
+
+    context "when action => create_unless_exists" do
+      it "should create new documents when specific id is specified" do
+        subject = get_es_output("create_unless_exists", "id123")
+        subject.register
+        subject.receive(LogStash::Event.new("message" => "sample message here"))
+        subject.buffer_flush(:final => true)
+        es.indices.refresh
+        # Wait or fail until everything's indexed.
+        Stud::try(3.times) do
+          r = es.search
+          insist { r["hits"]["total"] } == 1
+        end
+      end
+
+      it "should fail to create a document when no id is specified" do
+        subject = get_es_output("create_unless_exists")
+        subject.register
+        subject.receive(LogStash::Event.new("message" => "sample message here"))
+        subject.buffer_flush(:final => true)
+        es.indices.refresh
+        # Wait or fail until everything's indexed.
+        Stud::try(3.times) do
+          r = es.search
+          insist { r["hits"]["total"] } == 0
+        end
+      end
+
+      it "should unsuccesfully submit two records with the same document id" do
+        subject = get_es_output("create_unless_exists", "id123")
+        subject.register
+        subject.receive(LogStash::Event.new("message" => "sample message here"))
+        subject.receive(LogStash::Event.new("message" => "sample message here")) # 400 status failure (same id)
+        subject.buffer_flush(:final => true)
+        es.indices.refresh
+        # Wait or fail until everything's indexed.
+        Stud::try(3.times) do
+          r = es.search
+          insist { r["hits"]["total"] } == 1
+        end
+      end
+    end
+  end
+
   describe "testing index_type", :elasticsearch => true do
     describe "no type value" do
       # Generate a random index name
@@ -257,7 +355,7 @@ describe "outputs/elasticsearch" do
     end
   end
 
-  describe "wildcard substitution in index templates", :todo => true do
+  describe "wildcard substitution in index templates", :elasticsearch => true do
     require "logstash/outputs/elasticsearch"
 
     let(:template) { '{"template" : "not important, will be updated by :index"}' }
@@ -345,13 +443,18 @@ describe "outputs/elasticsearch" do
           reject { values }.include?(1)
         end
 
-        it "creates .raw field fro any string field which is not_analyzed" do
+        it "does not create .raw field for the message field" do
           results = @es.search(:q => "message.raw:\"sample message here\"")
+          insist { results["hits"]["total"] } == 0
+        end
+
+        it "creates .raw field from any string field which is not_analyzed" do
+          results = @es.search(:q => "country.raw:\"us\"")
           insist { results["hits"]["total"] } == 1
-          insist { results["hits"]["hits"][0]["_source"]["message"] } == "sample message here"
+          insist { results["hits"]["hits"][0]["_source"]["country"] } == "us"
 
           # partial or terms should not work.
-          results = @es.search(:q => "message.raw:\"sample\"")
+          results = @es.search(:q => "country.raw:\"u\"")
           insist { results["hits"]["total"] } == 0
         end
 
@@ -374,6 +477,105 @@ describe "outputs/elasticsearch" do
     end
   end
 
+  describe "failures in bulk class expected behavior", :elasticsearch => true do
+    let(:template) { '{"template" : "not important, will be updated by :index"}' }
+    let(:event1) { LogStash::Event.new("somevalue" => 100, "@timestamp" => "2014-11-17T20:37:17.223Z", "@metadata" => {"retry_count" => 0}) }
+    let(:action1) { ["index", {:_id=>nil, :_index=>"logstash-2014.11.17", :_type=>"logs"}, event1] }
+    let(:event2) { LogStash::Event.new("geoip" => { "location" => [ 0.0, 0.0] }, "@timestamp" => "2014-11-17T20:37:17.223Z", "@metadata" => {"retry_count" => 0}) }
+    let(:action2) { ["index", {:_id=>nil, :_index=>"logstash-2014.11.17", :_type=>"logs"}, event2] }
+    let(:max_retries) { 3 }
+
+    def mock_actions_with_response(*resp)
+      LogStash::Outputs::Elasticsearch::Protocols::HTTPClient
+        .any_instance.stub(:bulk).and_return(*resp)
+      LogStash::Outputs::Elasticsearch::Protocols::NodeClient
+        .any_instance.stub(:bulk).and_return(*resp)
+      LogStash::Outputs::Elasticsearch::Protocols::TransportClient
+        .any_instance.stub(:bulk).and_return(*resp)
+    end
+
+    ["node", "transport", "http"].each do |protocol|
+      context "with protocol => #{protocol}" do
+        subject do
+          require "logstash/outputs/elasticsearch"
+          settings = {
+            "manage_template" => true,
+            "index" => "logstash-2014.11.17",
+            "template_overwrite" => true,
+            "protocol" => protocol,
+            "host" => "localhost",
+            "retry_max_items" => 10,
+            "retry_max_interval" => 1,
+            "max_retries" => max_retries
+          }
+          next LogStash::Outputs::ElasticSearch.new(settings)
+        end
+
+        before :each do
+          # Delete all templates first.
+          require "elasticsearch"
+
+          # Clean ES of data before we start.
+          @es = Elasticsearch::Client.new
+          @es.indices.delete_template(:name => "*")
+          @es.indices.delete(:index => "*")
+          @es.indices.refresh
+        end
+
+        it "should return no errors if all bulk actions are successful" do
+          mock_actions_with_response({"errors" => false})
+          expect(subject).to receive(:submit).with([action1, action2]).once.and_call_original
+          subject.register
+          subject.receive(event1)
+          subject.receive(event2)
+          subject.buffer_flush(:final => true)
+          sleep(2)
+        end
+
+        it "should retry actions with response status of 503" do
+          mock_actions_with_response({"errors" => true, "statuses" => [200, 200, 503, 503]},
+                                     {"errors" => true, "statuses" => [200, 503]},
+                                     {"errors" => false})
+          expect(subject).to receive(:submit).with([action1, action1, action1, action2]).ordered.once.and_call_original
+          expect(subject).to receive(:submit).with([action1, action2]).ordered.once.and_call_original
+          expect(subject).to receive(:submit).with([action2]).ordered.once.and_call_original
+
+          subject.register
+          subject.receive(event1)
+          subject.receive(event1)
+          subject.receive(event1)
+          subject.receive(event2)
+          subject.buffer_flush(:final => true)
+          sleep(3)
+        end
+        
+        it "should retry actions with response status of 429" do
+          mock_actions_with_response({"errors" => true, "statuses" => [429]},
+                                     {"errors" => false})
+          expect(subject).to receive(:submit).with([action1]).twice.and_call_original
+          subject.register
+          subject.receive(event1)
+          subject.buffer_flush(:final => true)
+          sleep(3)
+        end
+
+        it "should retry an event until max_retries reached" do
+          mock_actions_with_response({"errors" => true, "statuses" => [429]},
+                                     {"errors" => true, "statuses" => [429]},
+                                     {"errors" => true, "statuses" => [429]},
+                                     {"errors" => true, "statuses" => [429]},
+                                     {"errors" => true, "statuses" => [429]},
+                                     {"errors" => true, "statuses" => [429]})
+          expect(subject).to receive(:submit).with([action1]).exactly(max_retries).times.and_call_original
+          subject.register
+          subject.receive(event1)
+          subject.buffer_flush(:final => true)
+          sleep(3)
+        end
+      end
+    end
+  end
+
   describe "elasticsearch protocol", :elasticsearch => true do
     # ElasticSearch related jars
 #LogStash::Environment.load_elasticsearch_jars!
@@ -515,3 +717,31 @@ describe "outputs/elasticsearch" do
     end
   end
 end
+
+describe "outputs/elasticsearch", :elasticsearch => true do
+  require 'elasticsearch'
+
+  it "set sniffing in transport mode" do
+
+    config = %q[
+      output {
+        elasticsearch {
+          host => "localhost"
+          protocol => "transport"
+          sniffing => true
+        }
+      }
+    ]
+
+
+    settings_class = org.elasticsearch.common.settings.ImmutableSettings
+    settings = settings_class.settingsBuilder
+
+    expect(settings_class).to receive(:settingsBuilder).and_return(settings)
+
+    pipeline = LogStash::Pipeline.new(config)
+    pipeline.run
+
+    expect(settings.build.getAsMap["client.transport.sniff"]).to eq("true")
+  end
+end

metadata

@@ -1,25 +1,30 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 0.1.9
+  version: 0.1.10
 platform: java
 authors:
 - Elasticsearch
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2014-11-21 00:00:00.000000000 Z
+date: 2015-01-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: elasticsearch
-  version_requirements: !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 1.0.6
-    - - ~>
+        version: '0'
+  name: concurrent-ruby
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '0'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -28,18 +33,18 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.0'
+  name: elasticsearch
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: stud
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 0.0.17
+        version: 1.0.6
     - - ~>
       - !ruby/object:Gem::Version
-        version: '0.0'
+        version: '1.0'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -48,32 +53,32 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.0'
+  name: stud
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: cabin
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 0.0.17
     - - ~>
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0.0'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.6'
+  name: cabin
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: logstash
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
-      - !ruby/object:Gem::Version
-        version: 1.4.0
-    - - <
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: '0.6'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -82,32 +87,32 @@ dependencies:
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
+  name: logstash
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: jar-dependencies
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 1.4.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+  name: jar-dependencies
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: ftw
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
-      - !ruby/object:Gem::Version
-        version: 0.0.40
-    - - ~>
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -116,50 +121,59 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0'
+  name: ftw
   prerelease: false
   type: :development
-- !ruby/object:Gem::Dependency
-  name: logstash-input-generator
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
+      - !ruby/object:Gem::Version
+        version: 0.0.40
+    - - ~>
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+  name: logstash-input-generator
   prerelease: false
   type: :development
-- !ruby/object:Gem::Dependency
-  name: manticore
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - '>='
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '0'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.3'
+  name: manticore
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: logstash-devutils
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.3'
+- !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+  name: logstash-devutils
   prerelease: false
   type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Output events to elasticsearch
 email: info@elasticsearch.com
 executables: []
@@ -167,8 +181,10 @@ extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
+- CONTRIBUTORS
 - Gemfile
 - LICENSE
+- README.md
 - Rakefile
 - lib/logstash/outputs/elasticsearch.rb
 - lib/logstash/outputs/elasticsearch/elasticsearch-template.json
@@ -177,7 +193,7 @@ files:
 - spec/outputs/elasticsearch_spec.rb
 homepage: http://logstash.net/
 licenses:
-- Apache License (2.0)
+- apache-2.0
 metadata:
   logstash_plugin: 'true'
   logstash_group: output
@@ -198,7 +214,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements:
 - jar 'org.elasticsearch:elasticsearch', '1.4.0'
 rubyforge_project:
-rubygems_version: 2.2.2
+rubygems_version: 2.1.9
 signing_key:
 specification_version: 4
 summary: Logstash Output to Elasticsearch