fluent-plugin-elasticsearch 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8544959ad2a4289ea64e3057372eafee047d707c
-  data.tar.gz: 4434e04393705d4d696ef0d7bc4e9ac32e1755db
+  metadata.gz: 421e2c4bb196e72e87a6c6e3c8444844b755fe07
+  data.tar.gz: 8568c9999361d7d0ab68f5916721c91d0312ef16
 SHA512:
-  metadata.gz: c46a10b6978857b09d9741226a8d8238b13ccfaccd63967debc1654dcf1acc6fb215e3934e9b42c27a4b73e29b07d7b6120427ea57ef5ab591c8b06596f00616
-  data.tar.gz: aaf4966ea945f5966b2abece6836180ddf78f6ca991de090f23427dd6ae983f0cd8e70e9e28d9d7587e7dde2fce0aee8e94d3812c8f43572c5c2965cb9c33fe8
+  metadata.gz: 9878a3743fb7ff425713cf822e7d16ec877eb9697a3e702b428f2aa62474bc8c3c1ee8e8438770e8b5ec5e5d2e37e15b6a81cce1eafb013ffce9970ee5d98098
+  data.tar.gz: c9cbebe13d2c81e4aa36dd99c666f02d4b0f1d0301ed39ad14dd93cef088a03e949113c6003d635b0ebe603eb6d58521bed47234c7b650603cd87abd4899eb9e

data/.travis.yml

@@ -6,4 +6,4 @@ rvm:
   - 2.1
   - 2.2
 
-script: bundle exec ruby -S -Itest test/plugin/test_out_elasticsearch.rb
+script: bundle exec rake test

data/History.md

@@ -2,6 +2,10 @@
 
 ### Future
 
+### 1.1.0
+- Support SSL client verification and custom CA file (#123)
+- Release experimental `type elasticsearch_dynamic` (#127)
+
 ### 1.0.0
 - password config param is now marked as secret and won't be displayed in logs.
 
@@ -9,7 +13,7 @@
 - Add `ssl_verify` option (#108)
 
 ### 0.8.0
-- Replace Patron with Excon HTTP client
+- Replace Patron with Excon HTTP client (#93)
 
 ### 0.7.0
 - Add new option `time_key` (#85)

data/README.md

@@ -8,46 +8,69 @@
 
 I wrote this so you can search logs routed through Fluentd.
 
-## Installation
+* [Installation](#installation)
+* [Usage](#usage)
+  + [Index templates](#index-templates)
+* [Configuration](#configuration)
+  + [hosts](#hosts)
+  + [user, password, path, scheme, ssl_verify](#user-password-path-scheme-ssl_verify)
+  + [logstash_format](#logstash_format)
+  + [logstash_prefix](#logstash_prefix)
+  + [logstash_dateformat](#logstash_dateformat)
+  + [time_key](#time_key)
+  + [utc_index](#utc_index)
+  + [request_timeout](#request_timeout)
+  + [reload_connections](#reload_connections)
+  + [reload_on_failure](#reload_on_failure)
+  + [include_tag_key, tag_key](#include_tag_key-tag_key)
+  + [id_key](#id_key)
+  + [Client/host certificate options](#clienthost-certificate-options)
+  + [Buffered output options](#buffered-output-options)
+  + [Not seeing a config you need?](#not-seeing-a-config-you-need)
+  + [Dynamic configuration](#dynamic-configuration)
+* [Contact](#contact)
+* [Contributing](#contributing)
+* [Running tests](#running-tests)
 
-    $ gem install fluent-plugin-elasticsearch
+## Installation
 
-* prerequisite : You need to install [libcurl (libcurl-devel)](http://curl.haxx.se/libcurl/) to work with.
+```sh
+$ gem install fluent-plugin-elasticsearch
+```
 
 ## Usage
 
-In your fluentd configration, use `type elasticsearch`. Additional configuration is optional, default values would look like this:
+In your Fluentd configuration, use `type elasticsearch`. Additional configuration is optional, default values would look like this:
 
 ```
-host localhost
-port 9200
-index_name fluentd
-type_name fluentd
+<match my.logs>
+  type elasticsearch
+  host localhost
+  port 9200
+  index_name fluentd
+  type_name fluentd
+</match>
 ```
 
-**Index templates**
+### Index templates
 
-This plugin creates ElasticSearch indices by merely writing to them. Consider using [Index Templates](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html) to gain control of what get indexed and how. See [this example](https://github.com/uken/fluent-plugin-elasticsearch/issues/33#issuecomment-38693282) for a good starting point.
+This plugin creates ElasticSearch indices by merely writing to them. Consider using [Index Templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) to gain control of what get indexed and how. See [this example](https://github.com/uken/fluent-plugin-elasticsearch/issues/33#issuecomment-38693282) for a good starting point.
 
-**More options:**
+## Configuration
 
-**hosts**
+### hosts
 
 ```
 hosts host1:port1,host2:port2,host3:port3
-```
-
-or
-
-```
+# or
 hosts https://customhost.com:443/path,https://username:password@host-failover.com:443
 ```
 
-You can specify multiple elasticsearch hosts with separator ",".
+You can specify multiple ElasticSearch hosts with separator ",".
 
-If you specify multiple hosts, this plugin will load balance updates to elasticsearch. This is an [elasticsearch-ruby](https://github.com/elasticsearch/elasticsearch-ruby) feature, the default strategy is round-robin.
+If you specify multiple hosts, this plugin will load balance updates to ElasticSearch. This is an [elasticsearch-ruby](https://github.com/elasticsearch/elasticsearch-ruby) feature, the default strategy is round-robin.
 
-**user, password, path, scheme, ssl_verify**
+### user, password, path, scheme, ssl_verify
 
 If you specify this option, host and port options are ignored.
 
@@ -62,21 +85,21 @@ You can specify user and password for HTTP basic auth. If used in conjunction wi
 
 Specify `ssl_verify false` to skip ssl verification (defaults to true)
 
-**logstash_format**
+### logstash_format
 
 ```
 logstash_format true # defaults to false
 ```
 
-This is meant to make writing data into elasticsearch compatible to what logstash writes. By doing this, one could take advantade of [kibana](http://kibana.org/).
+This is meant to make writing data into ElasticSearch compatible to what [Logstash](https://www.elastic.co/products/logstash) writes. By doing this, one could take advantage of [Kibana](https://www.elastic.co/products/kibana).
 
-**logstash_prefix**
+### logstash_prefix
 
 ```
 logstash_prefix mylogs # defaults to "logstash"
 ```
 
-**logstash_dateformat**
+### logstash_dateformat
 
 By default, the records inserted into index `logstash-YYMMDD`. This option allows to insert into specified index like `mylogs-YYYYMM` for a monthly index.
 
@@ -84,15 +107,15 @@ By default, the records inserted into index `logstash-YYMMDD`. This option allow
 logstash_dateformat %Y.%m. # defaults to "%Y.%m.%d"
 ```
 
-**time_key**
+### time_key
 
-By default, when inserting records in logstash format, @timestamp is dynamically created with the time at log ingestion. If you'd like to use a custom time. Include an @timestamp with your record.
+By default, when inserting records in [Logstash](https://www.elastic.co/products/logstash) format, `@timestamp` is dynamically created with the time at log ingestion. If you'd like to use a custom time, include an `@timestamp` with your record.
 
 ```
 {"@timestamp":"2014-04-07T000:00:00-00:00"}
 ```
 
-You can specify an option `time_key` (like the option described in [tail Input Plugin](http://docs.fluentd.org/articles/in_tail)) if you don't like `@timestamp`.
+You can specify an option `time_key` (like the option described in [tail Input Plugin](http://docs.fluentd.org/articles/in_tail)) to replace `@timestamp` key.
 
 Suppose you have settings
 
@@ -118,33 +141,33 @@ The output will be
 }
 ```
 
-**utc_index**
+### utc_index
 
 ```
 utc_index true
 ```
 
-By default, the records inserted into index `logstash-YYMMDD` with utc (Coordinated Universal Time). This option allows to use local time if you describe utc_index to false.
+By default, the records inserted into index `logstash-YYMMDD` with UTC (Coordinated Universal Time). This option allows to use local time if you describe utc_index to false.
 
-**request_timeout**
+### request_timeout
+
+You can specify HTTP request timeout.
+
+This is useful when ElasticSearch cannot return response for bulk request within the default of 5 seconds.
 
 ```
 request_timeout 15s # defaults to 5s
 ```
 
-You can specify HTTP request timeout.
-
-This is useful when Elasticsearch cannot return response for bulk request within the default of 5 seconds.
-
-**reload_connections**
+### reload_connections
 
 ```
 reload_connections false # defaults to true
 ```
 
-**reload_on_failure**
+### reload_on_failure
 
-You can tune how the elasticsearch-transport host reloading feature works. By default it will reload the host list from the server every 10,000th request to spread the load. This can be an issue if your ElasticSearch cluster is behind a Reverse Proxy, as fluentd process may not have direct network access to the ElasticSearch nodes.
+You can tune how the elasticsearch-transport host reloading feature works. By default it will reload the host list from the server every 10,000th request to spread the load. This can be an issue if your ElasticSearch cluster is behind a Reverse Proxy, as Fluentd process may not have direct network access to the ElasticSearch nodes.
 
 ```
 reload_on_failure true # defaults to false
@@ -153,14 +176,14 @@ reload_on_failure true # defaults to false
 Indicates that the elasticsearch-transport will try to reload the nodes addresses if there is a failure while making the
 request, this can be useful to quickly remove a dead node from the list of addresses.
 
-**include_tag_key, tag_key**
+### include_tag_key, tag_key
 
 ```
 include_tag_key true # defaults to false
 tag_key tag # defaults to tag
 ```
 
-This will add the fluentd tag in the json record. For instance, if you have a config like this:
+This will add the Fluentd tag in the JSON record. For instance, if you have a config like this:
 
 ```
 <match my.logs>
@@ -170,19 +193,19 @@ This will add the fluentd tag in the json record. For instance, if you have a co
 </match>
 ```
 
-The record inserted into elasticsearch would be
+The record inserted into ElasticSearch would be
 
 ```
 {"_key":"my.logs", "name":"Johnny Doeie"}
 ```
 
-**id_key**
+### id_key
 
 ```
 id_key request_id # use "request_id" field as a record id in ES
 ```
 
-By default, all records inserted into elasticsearch get a random _id. This option allows to use a field in the record as an identifier.
+By default, all records inserted into ElasticSearch get a random _id. This option allows to use a field in the record as an identifier.
 
 This following record `{"name":"Johnny","request_id":"87d89af7daffad6"}` will trigger the following ElasticSearch command
 
@@ -191,9 +214,23 @@ This following record `{"name":"Johnny","request_id":"87d89af7daffad6"}` will tr
 { "name": "Johnny", "request_id": "87d89af7daffad6" }
 ```
 
-**Buffered output options**
+### Client/host certificate options
+
+Need to verify ElasticSearch's certificate?  You can use the following parameter to specify a CA instead of using an environment variable.
+```
+ca_file /path/to/your/ca/cert
+```
+
+Does your ElasticSearch cluster want to verify client connections?  You can specify the following parameters to use your client certificate, key, and key password for your connection.
+```
+client_cert /path/to/your/client/cert
+client_key /path/to/your/private/key
+client_key_pass password
+```
+
+### Buffered output options
 
-fluentd-plugin-elasticsearch is a buffered output that uses elasticseach's bulk API. So additional buffer configuration would be (with default values):
+`fluentd-plugin-elasticsearch` extends [Fluentd's builtin Buffered Output plugin](http://docs.fluentd.org/articles/buffer-plugin-overview). It adds the following options:
 
 ```
 buffer_type memory
@@ -203,9 +240,11 @@ retry_wait 1.0
 num_threads 1
 ```
 
-**Not seeing a config you need?**
+### Not seeing a config you need?
 
-We try to keep the scope of this plugin small. If you need more configuration options, please consider using [fluent-plugin-forest](https://github.com/tagomoris/fluent-plugin-forest). For example, to configure multiple tags to be sent to different ElasticSearch indices:
+We try to keep the scope of this plugin small and not add too many configuration options. If you think an option would be useful to others, feel free to open an issue or contribute a Pull Request.
+
+Alternatively, consider using [fluent-plugin-forest](https://github.com/tagomoris/fluent-plugin-forest). For example, to configure multiple tags to be sent to different ElasticSearch indices:
 
 ```
 <match my.logs.*>
@@ -219,12 +258,39 @@ We try to keep the scope of this plugin small. If you need more configuration op
 </match>
 ```
 
-## Contributing
+And yet another option is described in Dynamic Configuration section.
+
+### Dynamic configuration
+
+If you want configurations to depend on information in messages, you can use `elasticsearch_dynamic`. This is an experimental variation of the ElasticSearch plugin allows configuration values to be specified in ways such as the below:
+
+```
+<match my.logs.*>
+  type elasticsearch_dynamic
+  hosts ${record['host1']}:9200,${record['host2']}:9200
+  index_name my_index.${Time.at(time).getutc.strftime(@logstash_dateformat)}
+  logstash_prefix ${tag_parts[3]}
+  port ${9200+rand(4)}
+  index_name ${tag_parts[2]}-${Time.at(time).getutc.strftime(@logstash_dateformat)}
+</match>
+```
+
+**Please note, this uses Ruby's `eval` for every message, so there are performance and security implications.**
 
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+## Contact
 
 If you have a question, [open an Issue](https://github.com/uken/fluent-plugin-elasticsearch/issues).
+
+## Contributing
+
+Pull Requests are welcomed.
+
+## Running tests
+
+Install dev dependencies:
+
+```sh
+$ gem install bundler
+$ bundle install
+$ bundle exec rake test
+```

data/fluent-plugin-elasticsearch.gemspec

@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-elasticsearch'
-  s.version       = '1.0.0'
+  s.version       = '1.1.0'
   s.authors       = ['diogo', 'pitr']
   s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com']
   s.description   = %q{ElasticSearch output plugin for Fluent event collector}

data/lib/fluent/plugin/out_elasticsearch.rb

@@ -29,6 +29,10 @@ class Fluent::ElasticsearchOutput < Fluent::BufferedOutput
   config_param :reload_on_failure, :bool, :default => false
   config_param :time_key, :string, :default => nil
   config_param :ssl_verify , :bool, :default => true
+  config_param :client_key, :string, :default => nil
+  config_param :client_cert, :string, :default => nil
+  config_param :client_key_pass, :string, :default => nil
+  config_param :ca_file, :string, :default => nil
 
   include Fluent::SetTagKeyMixin
   config_set_default :include_tag_key, false
@@ -47,7 +51,8 @@ class Fluent::ElasticsearchOutput < Fluent::BufferedOutput
 
   def client
     @_es ||= begin
-      adapter_conf = lambda {|f| f.adapter :excon }
+      excon_options = { client_key: @client_key, client_cert: @client_cert, client_key_pass: @client_key_pass }
+      adapter_conf = lambda {|f| f.adapter :excon, excon_options }
       transport = Elasticsearch::Transport::Transport::HTTP::Faraday.new(get_connection_options.merge(
                                                                           options: {
                                                                             reload_connections: @reload_connections,
@@ -55,7 +60,7 @@ class Fluent::ElasticsearchOutput < Fluent::BufferedOutput
                                                                             retry_on_failure: 5,
                                                                             transport_options: {
                                                                               request: { timeout: @request_timeout },
-                                                                              ssl: { verify: @ssl_verify }
+                                                                              ssl: { verify: @ssl_verify, ca_file: @ca_file }
                                                                             }
                                                                           }), &adapter_conf)
       es = Elasticsearch::Client.new transport: transport

data/lib/fluent/plugin/out_elasticsearch_dynamic.rb

@@ -0,0 +1,241 @@
+# encoding: UTF-8
+require_relative 'out_elasticsearch'
+
+class Fluent::ElasticsearchOutputDynamic < Fluent::ElasticsearchOutput
+
+  Fluent::Plugin.register_output('elasticsearch_dynamic', self)
+
+  config_param :delimiter, :string, :default => "."
+
+  # params overloaded as strings
+  config_param :port, :string, :default => "9200"
+  config_param :logstash_format, :string, :default => "false"
+  config_param :utc_index, :string, :default => "true"
+  config_param :reload_connections, :string, :default => "true"
+  config_param :reload_on_failure, :string, :default => "false"
+  config_param :ssl_verify, :string, :dfeault => "true"
+
+  def configure(conf)
+    super
+
+    # evaluate all configurations here
+    @dynamic_config = Hash.new
+    self.instance_variables.each { |var|
+      if is_valid_expand_param_type(var)
+        value = expand_param(self.instance_variable_get(var), nil, nil, nil)
+
+        var = var.to_s.gsub(/@(.+)/){ $1 }
+        @dynamic_config[var] = value
+      end
+    }
+    # end eval all configs
+    @current_config = nil
+  end
+
+  def client(host)
+
+    # check here to see if we already have a client connection for the given host
+    connection_options = get_connection_options(host)
+
+    @_es = nil unless is_existing_connection(connection_options[:hosts])
+
+    @_es ||= begin
+      @current_config = connection_options[:hosts].clone
+      excon_options = { client_key: @dynamic_config['client_key'], client_cert: @dynamic_config['client_cert'], client_key_pass: @dynamic_config['client_key_pass'] }
+      adapter_conf = lambda {|f| f.adapter :excon, excon_options }
+      transport = Elasticsearch::Transport::Transport::HTTP::Faraday.new(connection_options.merge(
+                                                                          options: {
+                                                                            reload_connections: @dynamic_config['reload_connections'],
+                                                                            reload_on_failure: @dynamic_config['reload_on_failure'],
+                                                                            retry_on_failure: 5,
+                                                                            transport_options: {
+                                                                              request: { timeout: @dynamic_config['request_timeout'] },
+                                                                              ssl: { verify: @dynamic_config['ssl_verify'], ca_file: @dynamic_config['ca_file'] }
+                                                                            }
+                                                                          }), &adapter_conf)
+      es = Elasticsearch::Client.new transport: transport
+
+      begin
+        raise ConnectionFailure, "Can not reach Elasticsearch cluster (#{connection_options_description(host)})!" unless es.ping
+      rescue *es.transport.host_unreachable_exceptions => e
+        raise ConnectionFailure, "Can not reach Elasticsearch cluster (#{connection_options_description(host)})! #{e.message}"
+      end
+
+      log.info "Connection opened to Elasticsearch cluster => #{connection_options_description(host)}"
+      es
+    end
+  end
+
+  def get_connection_options(con_host)
+    raise "`password` must be present if `user` is present" if @dynamic_config['user'] && !@dynamic_config['password']
+
+    hosts = if con_host || @dynamic_config['hosts']
+      (con_host || @dynamic_config['hosts']).split(',').map do |host_str|
+        # Support legacy hosts format host:port,host:port,host:port...
+        if host_str.match(%r{^[^:]+(\:\d+)?$})
+          {
+            host:   host_str.split(':')[0],
+            port:   (host_str.split(':')[1] || @dynamic_config['port']).to_i,
+            scheme: @dynamic_config['scheme']
+          }
+        else
+          # New hosts format expects URLs such as http://logs.foo.com,https://john:pass@logs2.foo.com/elastic
+          uri = URI(host_str)
+          %w(user password path).inject(host: uri.host, port: uri.port, scheme: uri.scheme) do |hash, key|
+            hash[key.to_sym] = uri.public_send(key) unless uri.public_send(key).nil? || uri.public_send(key) == ''
+            hash
+          end
+        end
+      end.compact
+    else
+      [{host: @dynamic_config['host'], port: @dynamic_config['port'].to_i, scheme: @dynamic_config['scheme']}]
+    end.each do |host|
+      host.merge!(user: @dynamic_config['user'], password: @dynamic_config['password']) if !host[:user] && @dynamic_config['user']
+      host.merge!(path: @dynamic_config['path']) if !host[:path] && @dynamic_config['path']
+    end
+
+    {
+      hosts: hosts
+    }
+  end
+
+  def connection_options_description(host)
+    get_connection_options(host)[:hosts].map do |host_info|
+      attributes = host_info.dup
+      attributes[:password] = 'obfuscated' if attributes.has_key?(:password)
+      attributes.inspect
+    end.join(', ')
+  end  
+
+  def write(chunk)
+    
+    bulk_message = Hash.new { |h,k| h[k] = [] }
+
+    chunk.msgpack_each do |tag, time, record|
+      next unless record.is_a? Hash
+
+      # evaluate all configurations here
+      self.instance_variables.each { |var|
+        if is_valid_expand_param_type(var)
+          # check here to determine if we should evaluate
+          if @dynamic_config[var[1,var.length-1]] != self.instance_variable_get(var)
+            value = expand_param(self.instance_variable_get(var), tag, time, record)
+            var = var.to_s.gsub(/@(.+)/){ $1 }
+            @dynamic_config[var] = value
+          end
+        end
+      }
+      # end eval all configs
+
+      if eval(@dynamic_config['logstash_format'])
+        if record.has_key?("@timestamp")
+          time = Time.parse record["@timestamp"]
+        elsif record.has_key?(@dynamic_config['time_key'])
+          time = Time.parse record[@dynamic_config['time_key']]
+          record['@timestamp'] = record[@dynamic_config['time_key']]
+        else
+          record.merge!({"@timestamp" => Time.at(time).to_datetime.to_s})
+        end
+
+        if eval(@dynamic_config['utc_index'])
+          target_index = "#{@dynamic_config['logstash_prefix']}-#{Time.at(time).getutc.strftime("#{@dynamic_config['logstash_dateformat']}")}"
+        else
+          target_index = "#{@dynamic_config['logstash_prefix']}-#{Time.at(time).strftime("#{@dynamic_config['logstash_dateformat']}")}"
+        end
+      else
+        target_index = @dynamic_config['index_name']
+      end
+
+      if @include_tag_key
+        record.merge!(@dynamic_config['tag_key'] => tag)
+      end
+
+      meta = { "index" => {"_index" => target_index, "_type" => @dynamic_config['type_name']} }
+      if @dynamic_config['id_key'] && record[@dynamic_config['id_key']]
+        meta['index']['_id'] = record[@dynamic_config['id_key']]
+      end
+
+      if @dynamic_config['parent_key'] && record[@dynamic_config['parent_key']]
+        meta['index']['_parent'] = record[@dynamic_config['parent_key']]
+      end
+
+      if @dynamic_config['hosts']
+        host = @dynamic_config['hosts']  
+      else
+        host = "#{@dynamic_config['host']}:#{@dynamic_config['port']}"
+      end
+
+      bulk_message[host] << meta
+      bulk_message[host] << record
+
+    end
+
+    bulk_message.each do | hKey, array |
+      send(array, hKey) unless array.empty?
+      array.clear
+    end
+
+  end
+
+  def send(data, host)
+    retries = 0
+    begin
+      client(host).bulk body: data
+    rescue *client(host).transport.host_unreachable_exceptions => e
+      if retries < 2
+        retries += 1
+        @_es = nil
+        log.warn "Could not push logs to Elasticsearch, resetting connection and trying again. #{e.message}"
+        sleep 2**retries
+        retry
+      end
+      raise ConnectionFailure, "Could not push logs to Elasticsearch after #{retries} retries. #{e.message}"
+    end
+  end
+
+  def expand_param(param, tag, time, record)
+    # check for '${ ... }'
+    #   yes => `eval`
+    #   no  => return param
+    return param if (param =~ /\${.+}/).nil?
+
+    # check for 'tag_parts[]'
+      # separated by a delimiter (default '.')
+    tag_parts = tag.split(@delimiter) unless (param =~ /tag_parts\[.+\]/).nil? || tag.nil?
+
+    # pull out section between ${} then eval
+    inner = param.clone
+    while inner.match(/\${.+}/)
+      to_eval = inner.match(/\${(.+?)}/){$1}
+
+      if !(to_eval =~ /record\[.+\]/).nil? && record.nil?
+        return to_eval
+      elsif !(to_eval =~/tag_parts\[.+\]/).nil? && tag_parts.nil?
+        return to_eval
+      elsif !(to_eval =~/time/).nil? && time.nil?
+        return to_eval
+      else
+        inner.sub!(/\${.+?}/, eval( to_eval ))
+      end
+    end
+    inner
+  end
+
+  def is_valid_expand_param_type(param)
+    return self.instance_variable_get(param).is_a?(String)
+  end
+
+  def is_existing_connection(host)
+    # check if the host provided match the current connection
+    return false if @_es.nil?
+    return false if host.length != @current_config.length
+
+    for i in 0...host.length
+      if !host[i][:host].eql? @current_config[i][:host] || host[i][:port] != @current_config[i][:port]
+        return false
+      end
+    end
+
+    return true
+  end
+end

data/test/plugin/test_out_elasticsearch_dynamic.rb

@@ -0,0 +1,452 @@
+require 'test/unit'
+
+require 'fluent/test'
+require 'fluent/plugin/out_elasticsearch_dynamic'
+
+require 'webmock/test_unit'
+require 'date'
+
+$:.push File.expand_path("../..", __FILE__)
+$:.push File.dirname(__FILE__)
+
+require 'helper'
+
+WebMock.disable_net_connect!
+
+class ElasticsearchOutputDynamic < Test::Unit::TestCase
+  attr_accessor :index_cmds, :index_command_counts
+
+  def setup
+    Fluent::Test.setup
+    @driver = nil
+  end
+
+  def driver(tag='test', conf='')
+    @driver ||= Fluent::Test::BufferedOutputTestDriver.new(Fluent::ElasticsearchOutputDynamic, tag).configure(conf)
+  end
+
+  def sample_record
+    {'age' => 26, 'request_id' => '42', 'parent_id' => 'parent'}
+  end
+
+  def stub_elastic_ping(url="http://localhost:9200")
+    stub_request(:head, url).to_return(:status => 200, :body => "", :headers => {})
+  end
+
+  def stub_elastic(url="http://localhost:9200/_bulk")
+    stub_request(:post, url).with do |req|
+      @index_cmds = req.body.split("\n").map {|r| JSON.parse(r) }
+    end
+  end
+
+  def stub_elastic_unavailable(url="http://localhost:9200/_bulk")
+    stub_request(:post, url).to_return(:status => [503, "Service Unavailable"])
+  end
+
+  def stub_elastic_with_store_index_command_counts(url="http://localhost:9200/_bulk")
+    if @index_command_counts == nil
+       @index_command_counts = {}
+       @index_command_counts.default = 0
+    end
+
+    stub_request(:post, url).with do |req|
+      index_cmds = req.body.split("\n").map {|r| JSON.parse(r) }
+      @index_command_counts[url] += index_cmds.size
+    end
+  end
+
+  def test_configure
+    config = %{
+      host     logs.google.com
+      port     777
+      scheme   https
+      path     /es/
+      user     john
+      password doe
+    }
+    instance = driver('test', config).instance
+
+    assert_equal 'logs.google.com', instance.host
+    assert_equal "777", instance.port
+    assert_equal 'https', instance.scheme
+    assert_equal '/es/', instance.path
+    assert_equal 'john', instance.user
+    assert_equal 'doe', instance.password
+  end
+
+  def test_legacy_hosts_list
+    config = %{
+      hosts    host1:50,host2:100,host3
+      scheme   https
+      path     /es/
+      port     123
+    }
+    instance = driver('test', config).instance
+
+    assert_equal 3, instance.get_connection_options(nil)[:hosts].length
+    host1, host2, host3 = instance.get_connection_options(nil)[:hosts]
+
+    assert_equal 'host1', host1[:host]
+    assert_equal 50, host1[:port]
+    assert_equal 'https', host1[:scheme]
+    assert_equal '/es/', host2[:path]
+    assert_equal 'host3', host3[:host]
+    assert_equal 123, host3[:port]
+    assert_equal 'https', host3[:scheme]
+    assert_equal '/es/', host3[:path]
+  end
+
+  def test_hosts_list
+    config = %{
+      hosts    https://john:password@host1:443/elastic/,http://host2
+      path     /default_path
+      user     default_user
+      password default_password
+    }
+    instance = driver('test', config).instance
+
+    assert_equal 2, instance.get_connection_options(nil)[:hosts].length
+    host1, host2 = instance.get_connection_options(nil)[:hosts]
+
+    assert_equal 'host1', host1[:host]
+    assert_equal 443, host1[:port]
+    assert_equal 'https', host1[:scheme]
+    assert_equal 'john', host1[:user]
+    assert_equal 'password', host1[:password]
+    assert_equal '/elastic/', host1[:path]
+
+    assert_equal 'host2', host2[:host]
+    assert_equal 'http', host2[:scheme]
+    assert_equal 'default_user', host2[:user]
+    assert_equal 'default_password', host2[:password]
+    assert_equal '/default_path', host2[:path]
+  end
+
+  def test_single_host_params_and_defaults
+    config = %{
+      host     logs.google.com
+      user     john
+      password doe
+    }
+    instance = driver('test', config).instance
+
+    assert_equal 1, instance.get_connection_options(nil)[:hosts].length
+    host1 = instance.get_connection_options(nil)[:hosts][0]
+
+    assert_equal 'logs.google.com', host1[:host]
+    assert_equal 9200, host1[:port]
+    assert_equal 'http', host1[:scheme]
+    assert_equal 'john', host1[:user]
+    assert_equal 'doe', host1[:password]
+    assert_equal nil, host1[:path]
+  end
+
+  def test_writes_to_default_index
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_equal('fluentd', index_cmds.first['index']['_index'])
+  end
+
+  def test_writes_to_default_type
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_equal('fluentd', index_cmds.first['index']['_type'])
+  end
+
+  def test_writes_to_speficied_index
+    driver.configure("index_name myindex\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_equal('myindex', index_cmds.first['index']['_index'])
+  end
+
+  def test_writes_to_speficied_type
+    driver.configure("type_name mytype\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_equal('mytype', index_cmds.first['index']['_type'])
+  end
+
+  def test_writes_to_speficied_host
+    driver.configure("host 192.168.33.50\n")
+    stub_elastic_ping("http://192.168.33.50:9200")
+    elastic_request = stub_elastic("http://192.168.33.50:9200/_bulk")
+    driver.emit(sample_record)
+    driver.run
+    assert_requested(elastic_request)
+  end
+
+  def test_writes_to_speficied_port
+    driver.configure("port 9201\n")
+    stub_elastic_ping("http://localhost:9201")
+    elastic_request = stub_elastic("http://localhost:9201/_bulk")
+    driver.emit(sample_record)
+    driver.run
+    assert_requested(elastic_request)
+  end
+
+  def test_writes_to_multi_hosts
+    hosts = [['192.168.33.50', 9201], ['192.168.33.51', 9201], ['192.168.33.52', 9201]]
+    hosts_string = hosts.map {|x| "#{x[0]}:#{x[1]}"}.compact.join(',')
+
+    driver.configure("hosts #{hosts_string}")
+
+    hosts.each do |host_info|
+      host, port = host_info
+      stub_elastic_ping("http://#{host}:#{port}")
+      stub_elastic_with_store_index_command_counts("http://#{host}:#{port}/_bulk")
+    end
+
+    1000.times do
+      driver.emit(sample_record.merge('age'=>rand(100)))
+    end
+
+    driver.run
+
+    # @note: we cannot make multi chunks with options (flush_interval, buffer_chunk_limit)
+    # it's Fluentd test driver's constraint
+    # so @index_command_counts.size is always 1
+
+    assert(@index_command_counts.size > 0, "not working with hosts options")
+
+    total = 0
+    @index_command_counts.each do |url, count|
+      total += count
+    end
+    assert_equal(2000, total)
+  end
+
+  def test_makes_bulk_request
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.emit(sample_record.merge('age' => 27))
+    driver.run
+    assert_equal(4, index_cmds.count)
+  end
+
+  def test_all_records_are_preserved_in_bulk
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.emit(sample_record.merge('age' => 27))
+    driver.run
+    assert_equal(26, index_cmds[1]['age'])
+    assert_equal(27, index_cmds[3]['age'])
+  end
+
+  def test_writes_to_logstash_index
+    driver.configure("logstash_format true\n")
+    time = Time.parse Date.today.to_s
+    logstash_index = "logstash-#{time.getutc.strftime("%Y.%m.%d")}"
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record, time)
+    driver.run
+    assert_equal(logstash_index, index_cmds.first['index']['_index'])
+  end
+
+  def test_writes_to_logstash_utc_index
+    driver.configure("logstash_format true
+                      utc_index false")
+    time = Time.parse Date.today.to_s
+    utc_index = "logstash-#{time.strftime("%Y.%m.%d")}"
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record, time)
+    driver.run
+    assert_equal(utc_index, index_cmds.first['index']['_index'])
+  end
+
+  def test_writes_to_logstash_index_with_specified_prefix
+    driver.configure("logstash_format true
+                      logstash_prefix myprefix")
+    time = Time.parse Date.today.to_s
+    logstash_index = "myprefix-#{time.getutc.strftime("%Y.%m.%d")}"
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record, time)
+    driver.run
+    assert_equal(logstash_index, index_cmds.first['index']['_index'])
+  end
+
+    def test_writes_to_logstash_index_with_specified_dateformat
+    driver.configure("logstash_format true
+                      logstash_dateformat %Y.%m")
+    time = Time.parse Date.today.to_s
+    logstash_index = "logstash-#{time.getutc.strftime("%Y.%m")}"
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record, time)
+    driver.run
+    assert_equal(logstash_index, index_cmds.first['index']['_index'])
+  end
+
+  def test_writes_to_logstash_index_with_specified_prefix_and_dateformat
+    driver.configure("logstash_format true
+                      logstash_prefix myprefix
+                      logstash_dateformat %Y.%m")
+    time = Time.parse Date.today.to_s
+    logstash_index = "myprefix-#{time.getutc.strftime("%Y.%m")}"
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record, time)
+    driver.run
+    assert_equal(logstash_index, index_cmds.first['index']['_index'])
+  end
+
+  def test_doesnt_add_logstash_timestamp_by_default
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_nil(index_cmds[1]['@timestamp'])
+  end
+
+  def test_adds_logstash_timestamp_when_configured
+    driver.configure("logstash_format true\n")
+    stub_elastic_ping
+    stub_elastic
+    ts = DateTime.now.to_s
+    driver.emit(sample_record)
+    driver.run
+    assert(index_cmds[1].has_key? '@timestamp')
+    assert_equal(index_cmds[1]['@timestamp'], ts)
+  end
+
+  def test_uses_custom_timestamp_when_included_in_record
+    driver.configure("logstash_format true\n")
+    stub_elastic_ping
+    stub_elastic
+    ts = DateTime.new(2001,2,3).to_s
+    driver.emit(sample_record.merge!('@timestamp' => ts))
+    driver.run
+    assert(index_cmds[1].has_key? '@timestamp')
+    assert_equal(index_cmds[1]['@timestamp'], ts)
+  end
+
+  def test_uses_custom_time_key
+    driver.configure("logstash_format true
+                      time_key vtm\n")
+    stub_elastic_ping
+    stub_elastic
+    ts = DateTime.new(2001,2,3).to_s
+    driver.emit(sample_record.merge!('vtm' => ts))
+    driver.run
+    assert(index_cmds[1].has_key? '@timestamp')
+    assert_equal(index_cmds[1]['@timestamp'], ts)
+  end
+
+  def test_doesnt_add_tag_key_by_default
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_nil(index_cmds[1]['tag'])
+  end
+
+  def test_adds_tag_key_when_configured
+    driver('mytag').configure("include_tag_key true\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert(index_cmds[1].has_key?('tag'))
+    assert_equal(index_cmds[1]['tag'], 'mytag')
+  end
+
+  def test_adds_id_key_when_configured
+    driver.configure("id_key request_id\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_equal(index_cmds[0]['index']['_id'], '42')
+  end
+
+  def test_doesnt_add_id_key_if_missing_when_configured
+    driver.configure("id_key another_request_id\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert(!index_cmds[0]['index'].has_key?('_id'))
+  end
+
+  def test_adds_id_key_when_not_configured
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert(!index_cmds[0]['index'].has_key?('_id'))
+  end
+
+  def test_adds_parent_key_when_configured
+    driver.configure("parent_key parent_id\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert_equal(index_cmds[0]['index']['_parent'], 'parent')
+  end
+
+  def test_doesnt_add_parent_key_if_missing_when_configured
+    driver.configure("parent_key another_parent_id\n")
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert(!index_cmds[0]['index'].has_key?('_parent'))
+  end
+
+  def test_adds_parent_key_when_not_configured
+    stub_elastic_ping
+    stub_elastic
+    driver.emit(sample_record)
+    driver.run
+    assert(!index_cmds[0]['index'].has_key?('_parent'))
+  end
+
+  def test_request_error
+    stub_elastic_ping
+    stub_elastic_unavailable
+    driver.emit(sample_record)
+    assert_raise(Elasticsearch::Transport::Transport::Errors::ServiceUnavailable) {
+      driver.run
+    }
+  end
+
+  def test_garbage_record_error
+    stub_elastic_ping
+    stub_elastic
+    driver.emit("some garbage string")
+    driver.run
+  end
+
+  def test_connection_failed_retry
+    connection_resets = 0
+
+    stub_elastic_ping(url="http://localhost:9200").with do |req|
+      connection_resets += 1
+    end
+
+    stub_request(:post, "http://localhost:9200/_bulk").with do |req|
+      raise Faraday::ConnectionFailed, "Test message"
+    end
+
+    driver.emit(sample_record)
+
+    assert_raise(Fluent::ElasticsearchOutput::ConnectionFailure) {
+      driver.run
+    }
+    assert_equal(connection_resets, 3)
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - diogo
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-07-27 00:00:00.000000000 Z
+date: 2015-10-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -127,8 +127,10 @@ files:
 - Rakefile
 - fluent-plugin-elasticsearch.gemspec
 - lib/fluent/plugin/out_elasticsearch.rb
+- lib/fluent/plugin/out_elasticsearch_dynamic.rb
 - test/helper.rb
 - test/plugin/test_out_elasticsearch.rb
+- test/plugin/test_out_elasticsearch_dynamic.rb
 homepage: https://github.com/uken/fluent-plugin-elasticsearch
 licenses:
 - MIT
@@ -149,10 +151,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.5
+rubygems_version: 2.2.2
 signing_key: 
 specification_version: 4
 summary: ElasticSearch output plugin for Fluent event collector
 test_files:
 - test/helper.rb
 - test/plugin/test_out_elasticsearch.rb
+- test/plugin/test_out_elasticsearch_dynamic.rb