fluent-plugin-elasticsearch 3.6.1 → 3.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1073f1770d5c380f496b57f111e1792812b870392c5996111465ce9cd2cfa0c6
-  data.tar.gz: 1f8e70ec6516ba252d7f2b722ec7c85d3830e3eb0b616b67c909663ac5db9461
+  metadata.gz: 3edaea3928f2cc09c7a908d870afdc2cac2a6d15f2e3f36cc7aa2f27962da79f
+  data.tar.gz: '09c000b05842db4f08a546f09adb3fe725d50fcf213435d3c7fc0dfa94753551'
 SHA512:
-  metadata.gz: 291e4a936373fc096718c4fa9db816874a7e53d2f950b9a8e5bfa3f7a6830d98a4d79ab4b36d16b5615cfd831e364b5e545ff91a49866f26805a5aae73670338
-  data.tar.gz: ea66679ef5406d950923e9f255532116f5378619b712f063c64b01997313aecf5ec53c3e04a7c1313125cda4370e058c200d539bd741c2afd55a08015a42c26b
+  metadata.gz: 148893479963ccf22397b41be92ea10dbe49e852d54412daa10659e8eb52c8bd1f0ac9546549ec28dde9df53a40f41e6fc45746eb95c10ab8e885e8e38d60537
+  data.tar.gz: f7e31c37963276c11162d78aa45b52c71feaaf409dd548b72cfd32c652b784008ccc0d0777da0f050817f3d1803023e523c3d693d193a21725aeeb3849518fd6

data/History.md

@@ -1,6 +1,11 @@
 ## Changelog [[tags]](https://github.com/uken/fluent-plugin-elasticsearch/tags)
 
 ### [Unreleased]
+### 3.7.0
+- Tweak for cosmetic change (#671)
+- Fix access to Elasticsearch::Transport::VERSION with explicit top level class path (#670)
+- Implement Elasticsearch Input plugin (#669)
+
 ### 3.6.1
 - retry upsert on recoverable error. (#667)
 - Allow `_index` in chunk_keys (#665)

data/README.ElasticsearchInput.md

@@ -0,0 +1,293 @@
+## Index
+
+* [Installation](#installation)
+* [Usage](#usage)
+* [Configuration](#configuration)
+  + [host](#host)
+  + [port](#port)
+  + [hosts](#hosts)
+  + [user, password, path, scheme, ssl_verify](#user-password-path-scheme-ssl_verify)
+  + [parse_timestamp](#parse_timestamp)
+  + [timestampkey_format](#timestampkey_format)
+  + [timestamp_key](#timestamp_key)
+  + [timestamp_parse_error_tag](#timestamp_parse_error_tag)
+  + [http_backend](#http_backend)
+  + [request_timeout](#request_timeout)
+  + [reload_connections](#reload_connections)
+  + [reload_on_failure](#reload_on_failure)
+  + [resurrect_after](#resurrect_after)
+  + [with_transporter_log](#with_transporter_log)
+  + [Client/host certificate options](#clienthost-certificate-options)
+  + [sniffer_class_name](#sniffer-class-name)
+  + [custom_headers](#custom_headers)
+  + [docinfo_fields](#docinfo_fields)
+  + [docinfo_target](#docinfo_target)
+  + [docinfo](#docinfo)
+* [Advanced Usage](#advanced-usage)
+
+## Usage
+
+In your Fluentd configuration, use `@type elasticsearch` and specify `tag your.awesome.tag`. Additional configuration is optional, default values would look like this:
+
+```
+<source>
+  @type elasticsearch
+  host localhost
+  port 9200
+  index_name fluentd
+  type_name fluentd
+  tag my.logs
+</match>
+```
+
+## Configuration
+
+### host
+
+```
+host user-custom-host.domain # default localhost
+```
+
+You can specify Elasticsearch host by this parameter.
+
+
+### port
+
+```
+port 9201 # defaults to 9200
+```
+
+You can specify Elasticsearch port by this parameter.
+
+### hosts
+
+```
+hosts host1:port1,host2:port2,host3:port3
+```
+
+You can specify multiple Elasticsearch hosts with separator ",".
+
+If you specify multiple hosts, this plugin will load balance updates to Elasticsearch. This is an [elasticsearch-ruby](https://github.com/elasticsearch/elasticsearch-ruby) feature, the default strategy is round-robin.
+
+If you specify `hosts` option, `host` and `port` options are ignored.
+
+```
+host user-custom-host.domain # ignored
+port 9200                    # ignored
+hosts host1:port1,host2:port2,host3:port3
+```
+
+If you specify `hosts` option without port, `port` option is used.
+
+```
+port 9200
+hosts host1:port1,host2:port2,host3 # port3 is 9200
+```
+
+**Note:** If you will use scheme https, do not include "https://" in your hosts ie. host "https://domain", this will cause ES cluster to be unreachable and you will receive an error "Can not reach Elasticsearch cluster"
+
+**Note:** Up until v2.8.5, it was allowed to embed the username/password in the URL. However, this syntax is deprecated as of v2.8.6 because it was found to cause serious connection problems (See #394). Please migrate your settings to use the `user` and `password` field (described below) instead.
+
+### user, password, path, scheme, ssl_verify
+
+```
+user demo
+password secret
+path /elastic_search/
+scheme https
+```
+
+You can specify user and password for HTTP Basic authentication.
+
+And this plugin will escape required URL encoded characters within `%{}` placeholders.
+
+```
+user %{demo+}
+password %{@secret}
+```
+
+Specify `ssl_verify false` to skip ssl verification (defaults to true)
+
+### parse_timestamp
+
+```
+parse_timestamp true # defaults to false
+```
+
+Parse a `@timestamp` field and add parsed time to the event.
+
+### timestamp_key_format
+
+The format of the time stamp field (`@timestamp` or what you specify in Elasticsearch). This parameter only has an effect when [parse_timestamp](#parse_timestamp) is true as it only affects the name of the index we write to. Please see [Time#strftime](http://ruby-doc.org/core-1.9.3/Time.html#method-i-strftime) for information about the value of this format.
+
+Setting this to a known format can vastly improve your log ingestion speed if all most of your logs are in the same format. If there is an error parsing this format the timestamp will default to the ingestion time. If you are on Ruby 2.0 or later you can get a further performance improvement by installing the "strptime" gem: `fluent-gem install strptime`.
+
+For example to parse ISO8601 times with sub-second precision:
+
+```
+timestamp_key_format %Y-%m-%dT%H:%M:%S.%N%z
+```
+
+### timestamp_parse_error_tag
+
+With `parse_timestamp true`, elasticsearch input plugin parses timestamp field for consuming event time. If the consumed record has invalid timestamp value, this plugin emits an error event to `@ERROR` label with `timestamp_parse_error_tag` configured tag.
+
+Default value is `elasticsearch_plugin.input.time.error`.
+
+### http_backend
+
+With `http_backend typhoeus`, elasticsearch plugin uses typhoeus faraday http backend.
+Typhoeus can handle HTTP keepalive.
+
+Default value is `excon` which is default http_backend of elasticsearch plugin.
+
+```
+http_backend typhoeus
+```
+
+
+### request_timeout
+
+You can specify HTTP request timeout.
+
+This is useful when Elasticsearch cannot return response for bulk request within the default of 5 seconds.
+
+```
+request_timeout 15s # defaults to 5s
+```
+
+### reload_connections
+
+You can tune how the elasticsearch-transport host reloading feature works. By default it will reload the host list from the server every 10,000th request to spread the load. This can be an issue if your Elasticsearch cluster is behind a Reverse Proxy, as Fluentd process may not have direct network access to the Elasticsearch nodes.
+
+```
+reload_connections false # defaults to true
+```
+
+### reload_on_failure
+
+Indicates that the elasticsearch-transport will try to reload the nodes addresses if there is a failure while making the
+request, this can be useful to quickly remove a dead node from the list of addresses.
+
+```
+reload_on_failure true # defaults to false
+```
+
+### resurrect_after
+
+You can set in the elasticsearch-transport how often dead connections from the elasticsearch-transport's pool will be resurrected.
+
+```
+resurrect_after 5s # defaults to 60s
+```
+
+### with_transporter_log
+
+This is debugging purpose option to enable to obtain transporter layer log.
+Default value is `false` for backward compatibility.
+
+We recommend to set this true if you start to debug this plugin.
+
+```
+with_transporter_log true
+```
+
+### Client/host certificate options
+
+Need to verify Elasticsearch's certificate?  You can use the following parameter to specify a CA instead of using an environment variable.
+```
+ca_file /path/to/your/ca/cert
+```
+
+Does your Elasticsearch cluster want to verify client connections?  You can specify the following parameters to use your client certificate, key, and key password for your connection.
+```
+client_cert /path/to/your/client/cert
+client_key /path/to/your/private/key
+client_key_pass password
+```
+
+If you want to configure SSL/TLS version, you can specify ssl\_version parameter.
+```
+ssl_version TLSv1_2 # or [SSLv23, TLSv1, TLSv1_1]
+```
+
+:warning: If SSL/TLS enabled, it might have to be required to set ssl\_version.
+
+### Sniffer Class Name
+
+The default Sniffer used by the `Elasticsearch::Transport` class works well when Fluentd has a direct connection
+to all of the Elasticsearch servers and can make effective use of the `_nodes` API.  This doesn't work well
+when Fluentd must connect through a load balancer or proxy.  The parameter `sniffer_class_name` gives you the
+ability to provide your own Sniffer class to implement whatever connection reload logic you require.  In addition,
+there is a new `Fluent::Plugin::ElasticsearchSimpleSniffer` class which reuses the hosts given in the configuration, which
+is typically the hostname of the load balancer or proxy.  For example, a configuration like this would cause
+connections to `logging-es` to reload every 100 operations:
+
+```
+host logging-es
+port 9200
+reload_connections true
+sniffer_class_name Fluent::Plugin::ElasticsearchSimpleSniffer
+reload_after 100
+```
+
+### custom_headers
+
+This parameter adds additional headers to request. The default value is `{}`.
+
+```
+custom_headers {"token":"secret"}
+```
+
+### docinfo_fields
+
+This parameter specifies docinfo record keys. The default values are `['_index', '_type', '_id']`.
+
+```
+docinfo_fields ['_index', '_id']
+```
+
+### docinfo_target
+
+This parameter specifies docinfo storing key. The default value is `@metadata`.
+
+```
+docinfo_target metadata
+```
+
+### docinfo
+
+This parameter specifies whether docinfo information including or not. The default value is `false`.
+
+```
+docinfo false
+```
+
+## Advanced Usage
+
+Elasticsearch Input plugin and Elasticsearch output plugin can combine to transfer records into another cluster.
+
+```aconf
+<source>
+  @type elasticsearch
+  host original-cluster.local
+  port 9200
+  tag raw.elasticsearch
+  index_name logstash-*
+  docinfo true
+  # repeat false
+  # num_slices 2
+  # with_transporter_log true
+</source>
+<match raw.elasticsearch>
+  @type elasticsearch
+  host transferred-cluster.local
+  port 9200
+  index_name ${$.@metadata._index}
+  # type_name ${$.@metadata._type} # This parameter is optional due to Removal of mapping types since ES7.
+  id_key ${$.@metadata._id} # This parameter is needed for prevent duplicated records.
+  <buffer tag, $.@metadata._index, $.@metadata._type, $.@metadata._id>
+    @type memory # should use file buffer for preventing chunk lost
+  </buffer>
+</match>
+```

data/README.md

@@ -91,6 +91,7 @@ Current maintainers: @cosmo0920
   + [enable_ilm](#enable_ilm)
   + [ilm_policy_id](#ilm_policy_id)
   + [ilm_policy](#ilm_policy)
+* [Configuration - Elasticsearch Input](#configuration---elasticsearch-input)
 * [Troubleshooting](#troubleshooting)
   + [Cannot send events to elasticsearch](#cannot-send-events-to-elasticsearch)
   + [Cannot see detailed failure log](#cannot-see-detailed-failure-log)
@@ -794,11 +795,11 @@ http_backend typhoeus
 ```
 
 ### compression_level
-You can add gzip compression of output data. In this case `default_compression`, `best_compression` or `best speed` option should be chosen. 
+You can add gzip compression of output data. In this case `default_compression`, `best_compression` or `best speed` option should be chosen.
 By default there is no compression, default value for this option is `no_compression`
 ```
 compression_level best_compression
-``` 
+```
 
 ### prefer_oj_serializer
 
@@ -1158,6 +1159,10 @@ Default value is `{}`.
 
 **NOTE:** This parameter requests to install elasticsearch-xpack gem.
 
+## Configuration - Elasticsearch Input
+
+See [Elasticsearch Input plugin document](README.ElasticsearchInput.md)
+
 ## Troubleshooting
 
 ### Cannot send events to Elasticsearch

data/fluent-plugin-elasticsearch.gemspec

@@ -3,9 +3,9 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-elasticsearch'
-  s.version       = '3.6.1'
-  s.authors       = ['diogo', 'pitr']
-  s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com']
+  s.version       = '3.7.0'
+  s.authors       = ['diogo', 'pitr', 'Hiroshi Hatake']
+  s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com', 'cosmo0920.wp@gmail.com']
   s.description   = %q{Elasticsearch output plugin for Fluent event collector}
   s.summary       = s.description
   s.homepage      = 'https://github.com/uken/fluent-plugin-elasticsearch'

data/lib/fluent/plugin/in_elasticsearch.rb

@@ -0,0 +1,325 @@
+require 'elasticsearch'
+
+require 'fluent/log-ext'
+require 'fluent/plugin/input'
+require_relative 'elasticsearch_constants'
+
+module Fluent::Plugin
+  class ElasticsearchInput < Input
+    class UnrecoverableRequestFailure < Fluent::UnrecoverableError; end
+
+    DEFAULT_RELOAD_AFTER = -1
+    DEFAULT_STORAGE_TYPE = 'local'
+    METADATA = "@metadata".freeze
+
+    helpers :timer, :thread
+
+    Fluent::Plugin.register_input('elasticsearch', self)
+
+    config_param :tag, :string
+    config_param :host, :string,  :default => 'localhost'
+    config_param :port, :integer, :default => 9200
+    config_param :user, :string, :default => nil
+    config_param :password, :string, :default => nil, :secret => true
+    config_param :path, :string, :default => nil
+    config_param :scheme, :enum, :list => [:https, :http], :default => :http
+    config_param :hosts, :string, :default => nil
+    config_param :index_name, :string, :default => "fluentd"
+    config_param :parse_timestamp, :bool, :default => false
+    config_param :timestamp_key_format, :string, :default => nil
+    config_param :timestamp_parse_error_tag, :string, :default => 'elasticsearch_plugin.input.time.error'
+    config_param :query, :hash, :default => {"sort" => [ "_doc" ]}
+    config_param :scroll, :string, :default => "1m"
+    config_param :size, :integer, :default => 1000
+    config_param :num_slices, :integer, :default => 1
+    config_param :interval, :size, :default => 5
+    config_param :repeat, :bool, :default => true
+    config_param :http_backend, :enum, list: [:excon, :typhoeus], :default => :excon
+    config_param :request_timeout, :time, :default => 5
+    config_param :reload_connections, :bool, :default => true
+    config_param :reload_on_failure, :bool, :default => false
+    config_param :resurrect_after, :time, :default => 60
+    config_param :reload_after, :integer, :default => DEFAULT_RELOAD_AFTER
+    config_param :ssl_verify , :bool, :default => true
+    config_param :client_key, :string, :default => nil
+    config_param :client_cert, :string, :default => nil
+    config_param :client_key_pass, :string, :default => nil, :secret => true
+    config_param :ca_file, :string, :default => nil
+    config_param :ssl_version, :enum, list: [:SSLv23, :TLSv1, :TLSv1_1, :TLSv1_2], :default => :TLSv1_2
+    config_param :with_transporter_log, :bool, :default => false
+    config_param :sniffer_class_name, :string, :default => nil
+    config_param :custom_headers, :hash, :default => {}
+    config_param :docinfo_fields, :array, :default => ['_index', '_type', '_id']
+    config_param :docinfo_target, :string, :default => METADATA
+    config_param :docinfo, :bool, :default => false
+
+    include Fluent::Plugin::ElasticsearchConstants
+
+    def initialize
+      super
+    end
+
+    def configure(conf)
+      super
+
+      @timestamp_parser = create_time_parser
+      @backend_options = backend_options
+
+      raise Fluent::ConfigError, "`password` must be present if `user` is present" if @user && @password.nil?
+
+      if @user && m = @user.match(/%{(?<user>.*)}/)
+        @user = URI.encode_www_form_component(m["user"])
+      end
+      if @password && m = @password.match(/%{(?<password>.*)}/)
+        @password = URI.encode_www_form_component(m["password"])
+      end
+
+      @transport_logger = nil
+      if @with_transporter_log
+        @transport_logger = log
+        log_level = conf['@log_level'] || conf['log_level']
+        log.warn "Consider to specify log_level with @log_level." unless log_level
+      end
+      @current_config = nil
+      # Specify @sniffer_class before calling #client.
+      @sniffer_class = nil
+      begin
+        @sniffer_class = Object.const_get(@sniffer_class_name) if @sniffer_class_name
+      rescue Exception => ex
+        raise Fluent::ConfigError, "Could not load sniffer class #{@sniffer_class_name}: #{ex}"
+      end
+
+      @options = {
+        :index => @index_name,
+        :scroll => @scroll,
+        :size => @size
+      }
+      @base_query = @query
+    end
+
+    def backend_options
+      case @http_backend
+      when :excon
+        { client_key: @client_key, client_cert: @client_cert, client_key_pass: @client_key_pass }
+      when :typhoeus
+        require 'typhoeus'
+        { sslkey: @client_key, sslcert: @client_cert, keypasswd: @client_key_pass }
+      end
+    rescue LoadError => ex
+      log.error_backtrace(ex.backtrace)
+      raise Fluent::ConfigError, "You must install #{@http_backend} gem. Exception: #{ex}"
+    end
+
+    def get_escaped_userinfo(host_str)
+      if m = host_str.match(/(?<scheme>.*)%{(?<user>.*)}:%{(?<password>.*)}(?<path>@.*)/)
+        m["scheme"] +
+          URI.encode_www_form_component(m["user"]) +
+          ':' +
+          URI.encode_www_form_component(m["password"]) +
+          m["path"]
+      else
+        host_str
+      end
+    end
+
+    def get_connection_options(con_host=nil)
+
+      hosts = if con_host || @hosts
+        (con_host || @hosts).split(',').map do |host_str|
+          # Support legacy hosts format host:port,host:port,host:port...
+          if host_str.match(%r{^[^:]+(\:\d+)?$})
+            {
+              host:   host_str.split(':')[0],
+              port:   (host_str.split(':')[1] || @port).to_i,
+              scheme: @scheme.to_s
+            }
+          else
+            # New hosts format expects URLs such as http://logs.foo.com,https://john:pass@logs2.foo.com/elastic
+            uri = URI(get_escaped_userinfo(host_str))
+            %w(user password path).inject(host: uri.host, port: uri.port, scheme: uri.scheme) do |hash, key|
+              hash[key.to_sym] = uri.public_send(key) unless uri.public_send(key).nil? || uri.public_send(key) == ''
+              hash
+            end
+          end
+        end.compact
+      else
+        [{host: @host, port: @port, scheme: @scheme.to_s}]
+      end.each do |host|
+        host.merge!(user: @user, password: @password) if !host[:user] && @user
+        host.merge!(path: @path) if !host[:path] && @path
+      end
+
+      {
+        hosts: hosts
+      }
+    end
+
+    def start
+      super
+
+      timer_execute(:in_elasticsearch_timer, @interval, repeat: @repeat, &method(:run))
+    end
+
+    # once fluent v0.14 is released we might be able to use
+    # Fluent::Parser::TimeParser, but it doesn't quite do what we want - if gives
+    # [sec,nsec] where as we want something we can call `strftime` on...
+    def create_time_parser
+      if @timestamp_key_format
+        begin
+          # Strptime doesn't support all formats, but for those it does it's
+          # blazingly fast.
+          strptime = Strptime.new(@timestamp_key_format)
+          Proc.new { |value|
+            value = convert_numeric_time_into_string(value, @timestamp_key_format) if value.is_a?(Numeric)
+            strptime.exec(value).to_time
+          }
+        rescue
+          # Can happen if Strptime doesn't recognize the format; or
+          # if strptime couldn't be required (because it's not installed -- it's
+          # ruby 2 only)
+          Proc.new { |value|
+            value = convert_numeric_time_into_string(value, @timestamp_key_format) if value.is_a?(Numeric)
+            DateTime.strptime(value, @timestamp_key_format).to_time
+          }
+        end
+      else
+        Proc.new { |value|
+          value = convert_numeric_time_into_string(value) if value.is_a?(Numeric)
+          DateTime.parse(value).to_time
+        }
+      end
+    end
+
+    def convert_numeric_time_into_string(numeric_time, timestamp_key_format = "%Y-%m-%dT%H:%M:%S.%N%z")
+      numeric_time_parser = Fluent::NumericTimeParser.new(:float)
+      Time.at(numeric_time_parser.parse(numeric_time).to_r).strftime(timestamp_key_format)
+    end
+
+    def parse_time(value, event_time, tag)
+      @timestamp_parser.call(value)
+    rescue => e
+      router.emit_error_event(@timestamp_parse_error_tag, Fluent::Engine.now, {'tag' => tag, 'time' => event_time, 'format' => @timestamp_key_format, 'value' => value}, e)
+      return Time.at(event_time).to_time
+    end
+
+    def client(host = nil)
+      # check here to see if we already have a client connection for the given host
+      connection_options = get_connection_options(host)
+
+      @_es = nil unless is_existing_connection(connection_options[:hosts])
+
+      @_es ||= begin
+        @current_config = connection_options[:hosts].clone
+        adapter_conf = lambda {|f| f.adapter @http_backend, @backend_options }
+        local_reload_connections = @reload_connections
+        if local_reload_connections && @reload_after > DEFAULT_RELOAD_AFTER
+          local_reload_connections = @reload_after
+        end
+
+        headers = { 'Content-Type' => "application/json" }.merge(@custom_headers)
+
+        transport = Elasticsearch::Transport::Transport::HTTP::Faraday.new(
+          connection_options.merge(
+            options: {
+              reload_connections: local_reload_connections,
+              reload_on_failure: @reload_on_failure,
+              resurrect_after: @resurrect_after,
+              logger: @transport_logger,
+              transport_options: {
+                headers: headers,
+                request: { timeout: @request_timeout },
+                ssl: { verify: @ssl_verify, ca_file: @ca_file, version: @ssl_version }
+              },
+              http: {
+                user: @user,
+                password: @password
+              },
+              sniffer_class: @sniffer_class,
+            }), &adapter_conf)
+        Elasticsearch::Client.new transport: transport
+      end
+    end
+
+    def is_existing_connection(host)
+      # check if the host provided match the current connection
+      return false if @_es.nil?
+      return false if @current_config.nil?
+      return false if host.length != @current_config.length
+
+      for i in 0...host.length
+        if !host[i][:host].eql? @current_config[i][:host] || host[i][:port] != @current_config[i][:port]
+          return false
+        end
+      end
+
+      return true
+    end
+
+    def run
+      return run_slice if @num_slices <= 1
+
+      log.warn("Large slice number is specified:(#{@num_slices}). Consider reducing num_slices") if @num_slices > 8
+
+      @num_slices.times.map do |slice_id|
+        thread_create(:"in_elasticsearch_thread_#{slice_id}") do
+          run_slice(slice_id)
+        end
+      end
+    end
+
+    def run_slice(slice_id=nil)
+      slice_query = @base_query
+      slice_query = slice_query.merge('slice' => { 'id' => slice_id, 'max' => @num_slices}) unless slice_id.nil?
+      result = client.search(@options.merge(:body => Yajl.dump(slice_query) ))
+      es = Fluent::MultiEventStream.new
+
+      result["hits"]["hits"].each {|hit| process_events(hit, es)}
+      has_hits = result['hits']['hits'].any?
+      scroll_id = result['_scroll_id']
+
+      while has_hits && scroll_id
+        result = process_next_scroll_request(es, scroll_id)
+        has_hits = result['has_hits']
+        scroll_id = result['_scroll_id']
+      end
+
+      router.emit_stream(@tag, es)
+      client.clear_scroll(scroll_id: scroll_id) if scroll_id
+    end
+
+    def process_scroll_request(scroll_id)
+      client.scroll(:body => { :scroll_id => scroll_id }, :scroll => @scroll)
+    end
+
+    def process_next_scroll_request(es, scroll_id)
+      result = process_scroll_request(scroll_id)
+      result['hits']['hits'].each { |hit| process_events(hit, es) }
+      {'has_hits' => result['hits']['hits'].any?, '_scroll_id' => result['_scroll_id']}
+    end
+
+    def process_events(hit, es)
+      event = hit["_source"]
+      time = Fluent::Engine.now
+      if @parse_timestamp
+        if event.has_key?(TIMESTAMP_FIELD)
+          rts = event[TIMESTAMP_FIELD]
+          time = parse_time(rts, time, @tag)
+        end
+      end
+      if @docinfo
+        docinfo_target = event[@docinfo_target] || {}
+
+        unless docinfo_target.is_a?(Hash)
+          raise UnrecoverableError, "incompatible type for the docinfo_target=#{@docinfo_target} field in the `_source` document, expected a hash got:", :type => docinfo_target.class, :event => event
+        end
+
+        @docinfo_fields.each do |field|
+          docinfo_target[field] = hit[field]
+        end
+
+        event[@docinfo_target] = docinfo_target
+      end
+      es.add(time, event)
+    end
+  end
+end

data/lib/fluent/plugin/out_elasticsearch.rb

@@ -330,7 +330,7 @@ EOC
         end
       end
 
-      version_arr = Elasticsearch::Transport::VERSION.split('.')
+      version_arr = ::Elasticsearch::Transport::VERSION.split('.')
 
       if (version_arr[0].to_i < 7) || (version_arr[0].to_i == 7 && version_arr[1].to_i < 2)
         if compression

data/test/plugin/test_elasticsearch_error_handler.rb

@@ -629,7 +629,6 @@ class TestElasticsearchErrorHandler < Test::Unit::TestCase
         next unless e.respond_to?(:retry_stream)
         e.retry_stream.each {|time, record| records << record}
       end
-      puts records
       assert_equal 3, records.length
       assert_equal 2, records[0]['_id']
       # upsert is retried in case of conflict error.

data/test/plugin/test_in_elasticsearch.rb

@@ -0,0 +1,454 @@
+require_relative '../helper'
+require 'date'
+require 'fluent/test/helpers'
+require 'json'
+require 'fluent/test/driver/input'
+require 'flexmock/test_unit'
+
+class ElasticsearchInputTest < Test::Unit::TestCase
+  include FlexMock::TestCase
+  include Fluent::Test::Helpers
+
+  CONFIG = %[
+    tag raw.elasticsearch
+    interval 2
+  ]
+
+  def setup
+    Fluent::Test.setup
+    require 'fluent/plugin/in_elasticsearch'
+    @driver = nil
+    log = Fluent::Engine.log
+    log.out.logs.slice!(0, log.out.logs.length)
+  end
+
+  def driver(conf='')
+    @driver ||= Fluent::Test::Driver::Input.new(Fluent::Plugin::ElasticsearchInput).configure(conf)
+  end
+
+  def sample_response(index_name="fluentd")
+    {
+      "took"=>4,
+      "timed_out"=>false,
+      "_shards"=>{
+        "total"=>2,
+        "successful"=>2,
+        "skipped"=>0,
+        "failed"=>0
+      },
+      "hits"=>{
+        "total"=>{
+          "value"=>1,
+          "relation"=>"eq"
+        },
+        "max_score"=>1,
+        "hits"=>[
+          {
+            "_index"=>"#{index_name}-2019.11.14",
+            "_type"=>"_doc",
+            "_id"=>"MJ_faG4B16RqUMOji_nH",
+            "_score"=>1,
+            "_source"=>{
+              "message"=>"Hi from Fluentd!",
+              "@timestamp"=>"2019-11-14T16:45:10.559841000+09:00"
+            }
+          }
+        ]
+      }
+    }.to_json
+  end
+
+  def sample_scroll_response
+    {
+      "_scroll_id"=>"WomkoUKG0QPB679Ulo6TqQgh3pIGRUmrl9qXXGK3EeiQh9rbYNasTkspZQcJ01uz",
+      "took"=>0,
+      "timed_out"=>false,
+      "_shards"=>{
+        "total"=>1,
+        "successful"=>1,
+        "skipped"=>0,
+        "failed"=>0
+      },
+      "hits"=>{
+        "total"=>{
+          "value"=>7,
+          "relation"=>"eq"
+        },
+        "max_score"=>nil,
+        "hits"=>[
+          {
+            "_index"=>"fluentd-2019.11.14",
+            "_type"=>"_doc",
+            "_id"=>"MJ_faG4B16RqUMOji_nH",
+            "_score"=>1,
+            "_source"=>{
+              "message"=>"Hi from Fluentd!",
+              "@timestamp"=>"2019-11-14T16:45:10.559841000+09:00"
+            },
+            "sort"=>[0]
+          }
+        ]
+      }
+    }.to_json
+  end
+
+  def sample_scroll_response_2
+    {
+      "_scroll_id"=>"WomkoUKG0QPB679Ulo6TqQgh3pIGRUmrl9qXXGK3EeiQh9rbYNasTkspZQcJ01uz",
+      "took"=>0,
+      "timed_out"=>false,
+      "_shards"=>{
+        "total"=>1,
+        "successful"=>1,
+        "skipped"=>0,
+        "failed"=>0
+      },
+      "hits"=>{
+        "total"=>{
+          "value"=>7,
+          "relation"=>"eq"
+        },
+        "max_score"=>nil,
+        "hits"=>[
+          {
+            "_index"=>"fluentd-2019.11.14",
+            "_type"=>"_doc",
+            "_id"=>"L5-saG4B16RqUMOjw_kb",
+            "_score"=>1,
+            "_source"=>{
+              "message"=>"Yaaaaaaay from Fluentd!",
+              "@timestamp"=>"2019-11-14T15:49:41.112023000+09:00"
+            },
+            "sort"=>[1]
+          }
+        ]
+      }
+    }.to_json
+  end
+
+  def sample_scroll_response_terminate
+    {
+      "_scroll_id"=>"WomkoUKG0QPB679Ulo6TqQgh3pIGRUmrl9qXXGK3EeiQh9rbYNasTkspZQcJ01uz",
+      "took"=>1,
+      "timed_out"=>false,
+      "terminated_early"=>true,
+      "_shards"=>{
+        "total"=>1,
+        "successful"=>1,
+        "skipped"=>0,
+        "failed"=>0
+      },
+      "hits"=>{
+        "total"=>{
+          "value"=>7,
+          "relation"=>"eq"
+        },
+        "max_score"=>nil,
+        "hits"=>[]
+      }
+    }.to_json
+  end
+
+  def test_configure
+    config = %{
+      host     logs.google.com
+      port     777
+      scheme   https
+      path     /es/
+      user     john
+      password doe
+      tag      raw.elasticsearch
+    }
+    instance = driver(config).instance
+
+    expected_query = { "sort" => [ "_doc" ]}
+    assert_equal 'logs.google.com', instance.host
+    assert_equal 777, instance.port
+    assert_equal :https, instance.scheme
+    assert_equal '/es/', instance.path
+    assert_equal 'john', instance.user
+    assert_equal 'doe', instance.password
+    assert_equal 'raw.elasticsearch', instance.tag
+    assert_equal :TLSv1_2, instance.ssl_version
+    assert_equal 'fluentd', instance.index_name
+    assert_equal expected_query, instance.query
+    assert_equal '1m', instance.scroll
+    assert_equal 1000, instance.size
+    assert_equal 1, instance.num_slices
+    assert_equal 5, instance.interval
+    assert_true instance.repeat
+    assert_nil instance.client_key
+    assert_nil instance.client_cert
+    assert_nil instance.client_key_pass
+    assert_nil instance.ca_file
+    assert_false instance.with_transporter_log
+    assert_equal :excon, instance.http_backend
+    assert_nil instance.sniffer_class_name
+    assert_true instance.custom_headers.empty?
+    assert_equal ['_index', '_type', '_id'], instance.docinfo_fields
+    assert_equal '@metadata', instance.docinfo_target
+    assert_false instance.docinfo
+  end
+
+  def test_single_host_params_and_defaults
+    config = %{
+      host     logs.google.com
+      user     john
+      password doe
+      tag      raw.elasticsearch
+    }
+    instance = driver(config).instance
+
+    assert_equal 1, instance.get_connection_options[:hosts].length
+    host1 = instance.get_connection_options[:hosts][0]
+
+    assert_equal 'logs.google.com', host1[:host]
+    assert_equal 9200, host1[:port]
+    assert_equal 'http', host1[:scheme]
+    assert_equal 'john', host1[:user]
+    assert_equal 'doe', host1[:password]
+    assert_equal nil, host1[:path]
+    assert_equal 'raw.elasticsearch', instance.tag
+  end
+
+  def test_single_host_params_and_defaults_with_escape_placeholders
+    config = %{
+      host     logs.google.com
+      user     %{j+hn}
+      password %{d@e}
+      tag      raw.elasticsearch
+    }
+    instance = driver(config).instance
+
+    assert_equal 1, instance.get_connection_options[:hosts].length
+    host1 = instance.get_connection_options[:hosts][0]
+
+    assert_equal 'logs.google.com', host1[:host]
+    assert_equal 9200, host1[:port]
+    assert_equal 'http', host1[:scheme]
+    assert_equal 'j%2Bhn', host1[:user]
+    assert_equal 'd%40e', host1[:password]
+    assert_equal nil, host1[:path]
+    assert_equal 'raw.elasticsearch', instance.tag
+  end
+
+  def test_legacy_hosts_list
+    config = %{
+      hosts    host1:50,host2:100,host3
+      scheme   https
+      path     /es/
+      port     123
+      tag      raw.elasticsearch
+    }
+    instance = driver(config).instance
+
+    assert_equal 3, instance.get_connection_options[:hosts].length
+    host1, host2, host3 = instance.get_connection_options[:hosts]
+
+    assert_equal 'host1', host1[:host]
+    assert_equal 50, host1[:port]
+    assert_equal 'https', host1[:scheme]
+    assert_equal '/es/', host2[:path]
+    assert_equal 'host3', host3[:host]
+    assert_equal 123, host3[:port]
+    assert_equal 'https', host3[:scheme]
+    assert_equal '/es/', host3[:path]
+    assert_equal 'raw.elasticsearch', instance.tag
+  end
+
+  def test_hosts_list
+    config = %{
+      hosts    https://john:password@host1:443/elastic/,http://host2
+      path     /default_path
+      user     default_user
+      password default_password
+      tag      raw.elasticsearch
+    }
+    instance = driver(config).instance
+
+    assert_equal 2, instance.get_connection_options[:hosts].length
+    host1, host2 = instance.get_connection_options[:hosts]
+
+    assert_equal 'host1', host1[:host]
+    assert_equal 443, host1[:port]
+    assert_equal 'https', host1[:scheme]
+    assert_equal 'john', host1[:user]
+    assert_equal 'password', host1[:password]
+    assert_equal '/elastic/', host1[:path]
+
+    assert_equal 'host2', host2[:host]
+    assert_equal 'http', host2[:scheme]
+    assert_equal 'default_user', host2[:user]
+    assert_equal 'default_password', host2[:password]
+    assert_equal '/default_path', host2[:path]
+    assert_equal 'raw.elasticsearch', instance.tag
+  end
+
+  def test_hosts_list_with_escape_placeholders
+    config = %{
+      hosts    https://%{j+hn}:%{passw@rd}@host1:443/elastic/,http://host2
+      path     /default_path
+      user     default_user
+      password default_password
+      tag      raw.elasticsearch
+    }
+    instance = driver(config).instance
+
+    assert_equal 2, instance.get_connection_options[:hosts].length
+    host1, host2 = instance.get_connection_options[:hosts]
+
+    assert_equal 'host1', host1[:host]
+    assert_equal 443, host1[:port]
+    assert_equal 'https', host1[:scheme]
+    assert_equal 'j%2Bhn', host1[:user]
+    assert_equal 'passw%40rd', host1[:password]
+    assert_equal '/elastic/', host1[:path]
+
+    assert_equal 'host2', host2[:host]
+    assert_equal 'http', host2[:scheme]
+    assert_equal 'default_user', host2[:user]
+    assert_equal 'default_password', host2[:password]
+    assert_equal '/default_path', host2[:path]
+    assert_equal 'raw.elasticsearch', instance.tag
+  end
+
+  def test_emit
+    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"]}").
+      to_return(status: 200, body: sample_response.to_s,
+                headers: {'Content-Type' => 'application/json'})
+
+    driver(CONFIG)
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = {"message"    => "Hi from Fluentd!",
+                "@timestamp" => "2019-11-14T16:45:10.559841000+09:00"}
+    event = driver.events.map {|e| e.last}.last
+    assert_equal expected, event
+  end
+
+  def test_emit_with_custom_index_name
+    index_name = "logstash"
+    stub_request(:get, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"]}").
+      to_return(status: 200, body: sample_response(index_name).to_s,
+                headers: {'Content-Type' => 'application/json'})
+
+    driver(CONFIG + %[index_name #{index_name}])
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = {"message"    => "Hi from Fluentd!",
+                "@timestamp" => "2019-11-14T16:45:10.559841000+09:00"}
+    event = driver.events.map {|e| e.last}.last
+    assert_equal expected, event
+  end
+
+  def test_emit_with_parse_timestamp
+    index_name = "fluentd"
+    stub_request(:get, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"]}").
+      to_return(status: 200, body: sample_response(index_name).to_s,
+                headers: {'Content-Type' => 'application/json'})
+
+    driver(CONFIG + %[parse_timestamp])
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = {"message"    => "Hi from Fluentd!",
+                "@timestamp" => "2019-11-14T16:45:10.559841000+09:00"}
+    event = driver.events.map {|e| e.last}.last
+    time = driver.events.map {|e| e[1]}.last
+    expected_time = event_time("2019-11-14T16:45:10.559841000+09:00")
+    assert_equal expected_time.to_time, time.to_time
+    assert_equal expected, event
+  end
+
+  def test_emit_with_parse_timestamp_and_timstamp_format
+    index_name = "fluentd"
+    stub_request(:get, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"]}").
+      to_return(status: 200, body: sample_response(index_name).to_s,
+                headers: {'Content-Type' => 'application/json'})
+
+    driver(CONFIG + %[parse_timestamp true
+                      timestamp_key_format %Y-%m-%dT%H:%M:%S.%N%z
+                      ])
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = {"message"    => "Hi from Fluentd!",
+                "@timestamp" => "2019-11-14T16:45:10.559841000+09:00"}
+    event = driver.events.map {|e| e.last}.last
+    time = driver.events.map {|e| e[1]}.last
+    expected_time = event_time("2019-11-14T16:45:10.559841000+09:00")
+    assert_equal expected_time.to_time, time.to_time
+    assert_equal expected, event
+  end
+
+  def test_emit_with_docinfo
+    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"]}").
+      to_return(status: 200, body: sample_response.to_s,
+                headers: {'Content-Type' => 'application/json'})
+
+    driver(CONFIG + %[docinfo true])
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = {"message"    => "Hi from Fluentd!",
+                "@timestamp" => "2019-11-14T16:45:10.559841000+09:00"}
+    expected.merge!({"@metadata"=>
+                     {"_id"=>"MJ_faG4B16RqUMOji_nH",
+                      "_index"=>"fluentd-2019.11.14",
+                      "_type"=>"_doc"}
+                    })
+    event = driver.events.map {|e| e.last}.last
+    assert_equal expected, event
+  end
+
+  def test_emit_with_slices
+    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"],\"slice\":{\"id\":0,\"max\":2}}").
+      to_return(status: 200, body: sample_response.to_s,
+                headers: {'Content-Type' => 'application/json'})
+    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+      with(body: "{\"sort\":[\"_doc\"],\"slice\":{\"id\":1,\"max\":2}}").
+      to_return(status: 200, body: sample_response.to_s,
+                headers: {'Content-Type' => 'application/json'})
+
+    driver(CONFIG + %[num_slices 2])
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = [
+      {"message"=>"Hi from Fluentd!", "@timestamp"=>"2019-11-14T16:45:10.559841000+09:00"},
+      {"message"=>"Hi from Fluentd!", "@timestamp"=>"2019-11-14T16:45:10.559841000+09:00"},
+    ]
+    events = driver.events.map {|e| e.last}
+    assert_equal expected, events
+  end
+
+  def test_emit_with_size
+    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1").
+      with(body: "{\"sort\":[\"_doc\"]}").
+      to_return(status: 200, body: sample_scroll_response.to_s,
+                headers: {'Content-Type' => 'application/json'})
+    connection = 0
+    scroll_request = stub_request(:get, "http://localhost:9200/_search/scroll?scroll=1m").
+      with(
+        body: "{\"scroll_id\":\"WomkoUKG0QPB679Ulo6TqQgh3pIGRUmrl9qXXGK3EeiQh9rbYNasTkspZQcJ01uz\"}") do
+      connection += 1
+    end
+    scroll_request.to_return(lambda do |req|
+                               if connection <= 1
+                                 {status: 200, body: sample_scroll_response_2.to_s,
+                                  headers: {'Content-Type' => 'application/json'}}
+                               else
+                                 {status: 200, body: sample_scroll_response_terminate.to_s,
+                                  headers: {'Content-Type' => 'application/json'}}
+                               end
+                             end)
+    stub_request(:delete, "http://localhost:9200/_search/scroll/WomkoUKG0QPB679Ulo6TqQgh3pIGRUmrl9qXXGK3EeiQh9rbYNasTkspZQcJ01uz").
+      to_return(status: 200, body: "", headers: {})
+
+    driver(CONFIG + %[size 1])
+    driver.run(expect_emits: 1, timeout: 10)
+    expected = [
+      {"message"=>"Hi from Fluentd!", "@timestamp"=>"2019-11-14T16:45:10.559841000+09:00"},
+      {"message"=>"Yaaaaaaay from Fluentd!", "@timestamp"=>"2019-11-14T15:49:41.112023000+09:00"}
+    ]
+    events = driver.events.map{|e| e.last}
+    assert_equal expected, events
+  end
+
+end

metadata

@@ -1,15 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 3.6.1
+  version: 3.7.0
 platform: ruby
 authors:
 - diogo
 - pitr
+- Hiroshi Hatake
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-11-11 00:00:00.000000000 Z
+date: 2019-11-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -127,6 +128,7 @@ description: Elasticsearch output plugin for Fluent event collector
 email:
 - pitr.vern@gmail.com
 - me@diogoterror.com
+- cosmo0920.wp@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -141,6 +143,7 @@ files:
 - ISSUE_TEMPLATE.md
 - LICENSE.txt
 - PULL_REQUEST_TEMPLATE.md
+- README.ElasticsearchInput.md
 - README.md
 - Rakefile
 - appveyor.yml
@@ -155,6 +158,7 @@ files:
 - lib/fluent/plugin/elasticsearch_index_template.rb
 - lib/fluent/plugin/elasticsearch_simple_sniffer.rb
 - lib/fluent/plugin/filter_elasticsearch_genid.rb
+- lib/fluent/plugin/in_elasticsearch.rb
 - lib/fluent/plugin/oj_serializer.rb
 - lib/fluent/plugin/out_elasticsearch.rb
 - lib/fluent/plugin/out_elasticsearch_dynamic.rb
@@ -163,6 +167,7 @@ files:
 - test/plugin/test_elasticsearch_error_handler.rb
 - test/plugin/test_elasticsearch_index_lifecycle_management.rb
 - test/plugin/test_filter_elasticsearch_genid.rb
+- test/plugin/test_in_elasticsearch.rb
 - test/plugin/test_out_elasticsearch.rb
 - test/plugin/test_out_elasticsearch_dynamic.rb
 - test/plugin/test_template.json
@@ -197,6 +202,7 @@ test_files:
 - test/plugin/test_elasticsearch_error_handler.rb
 - test/plugin/test_elasticsearch_index_lifecycle_management.rb
 - test/plugin/test_filter_elasticsearch_genid.rb
+- test/plugin/test_in_elasticsearch.rb
 - test/plugin/test_out_elasticsearch.rb
 - test/plugin/test_out_elasticsearch_dynamic.rb
 - test/plugin/test_template.json