fluent-plugin-elasticsearch 3.3.1 → 3.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 31cc9563debdbede8ae6e51e6e43089150c5ce736457433a9f4e8dc5652fb489
-  data.tar.gz: 021c1bde5bc10ffcd24536a7fcda38151dcc2508edcfb87b973dee8cf8f1fdd4
+  metadata.gz: 65a43866400112b8742307709fb258518efdee379cdbb56b324ed365d8a4e1d7
+  data.tar.gz: 716149f2c5aa5380bbfc95e916d730572546bfae2520ed0b792dc6b6a828d78c
 SHA512:
-  metadata.gz: faca5fb42a3a126f27b864a48588a1fd918b645b16287500b2793150ed83fe5d2378f1c0527917fe33309a9fcb7af0f7645190700043f30fd8a02d3fd04ddcf6
-  data.tar.gz: 363df0da34ce21414a993c4464ba60766c3e0b0dd171b61674d6bd8850d3ca8d4055563d231439b1004cfc5aaa0b54f311e9002a6689c938960f011ab70e051a
+  metadata.gz: 6d98dc4fef7b9fe208d549fc93f8a4809bfb56f156bee069bf0f599891d13b29adad5d119e9e7a2274f31aede5a13879dbf16223f98a3642d598f9f71eb2665f
+  data.tar.gz: ab08e35c256740009631cf5d8fa13c9f8ab9edc98f172e106d111177b4a069db13ba27a60f201c71411727bdb9cac63de716d0b6d9e4ae2184a3eac8ae12131f

data/History.md

@@ -2,6 +2,10 @@
 
 ### [Unreleased]
 
+### 3.3.2
+- Fix support for host, hosts placeholders (#560)
+- typo fixes in README.md (#559)
+
 ### 3.3.1
 - add new option to suppress doc wrapping (#557)
 - Include 2 (#555)

data/README.md

@@ -138,7 +138,7 @@ host user-custom-host.domain # default localhost
 
 You can specify Elasticsearch host by this parameter.
 
-**Note:** Since v3.3.0, `host` parameter supports builtin placeholders. If you want to send events dinamically into different hosts at runtime with `elasticsearch_dynamic` output plugin, please consider to switch to use plain `elasticsearch` output plugin. In more detail for builtin placeholders, please refer to [Placeholders](#placeholders) section.
+**Note:** Since v3.3.2, `host` parameter supports builtin placeholders. If you want to send events dynamically into different hosts at runtime with `elasticsearch_dynamic` output plugin, please consider to switch to use plain `elasticsearch` output plugin. In more detail for builtin placeholders, please refer to [Placeholders](#placeholders) section.
 
 ### emit_error_for_missing_id
 
@@ -158,7 +158,7 @@ hosts host1:port1,host2:port2,host3:port3
 You can specify multiple Elasticsearch hosts with separator ",".
 
 If you specify multiple hosts, this plugin will load balance updates to Elasticsearch. This is an [elasticsearch-ruby](https://github.com/elasticsearch/elasticsearch-ruby) feature, the default strategy is round-robin.
-**Note:** If you will use scheme https, do not include "https://" in your hosts ie. host "https://domain", this will cause ES cluster to be unreachable and you will recieve an error "Can not reach Elasticsearch cluster"
+**Note:** If you will use scheme https, do not include "https://" in your hosts ie. host "https://domain", this will cause ES cluster to be unreachable and you will receive an error "Can not reach Elasticsearch cluster"
 
 **Note:** Up until v2.8.5, it was allowed to embed the username/password in the URL. However, this syntax is deprecated as of v2.8.6 because it was found to cause serious connection problems (See #394). Please migrate your settings to use the `user` and `password` field (described below) instead.
 
@@ -190,7 +190,7 @@ Specify `ssl_verify false` to skip ssl verification (defaults to true)
 logstash_format true # defaults to false
 ```
 
-This is meant to make writing data into Elasticsearch indices compatible to what [Logstash](https://www.elastic.co/products/logstash) calls them. By doing this, one could take advantage of [Kibana](https://www.elastic.co/products/kibana). See logstash_prefix and logstash_dateformat to customize this index name pattern. The index name will be `#{logstash_prefix}-#{formated_date}`
+This is meant to make writing data into Elasticsearch indices compatible to what [Logstash](https://www.elastic.co/products/logstash) calls them. By doing this, one could take advantage of [Kibana](https://www.elastic.co/products/kibana). See logstash_prefix and logstash_dateformat to customize this index name pattern. The index name will be `#{logstash_prefix}-#{formatted_date}`
 
 :warning: Setting this option to `true` will ignore the `index_name` setting. The default index name prefix is `logstash-`.
 
@@ -236,7 +236,7 @@ pipeline pipeline_id
 
 The format of the time stamp field (`@timestamp` or what you specify with [time_key](#time_key)). This parameter only has an effect when [logstash_format](#logstash_format) is true as it only affects the name of the index we write to. Please see [Time#strftime](http://ruby-doc.org/core-1.9.3/Time.html#method-i-strftime) for information about the value of this format.
 
-Setting this to a known format can vastly improve your log ingestion speed if all most of your logs are in the same format. If there is an error parsing this format the timestamp will default to the ingestion time. If you are on Ruby 2.0 or later you can get a further performance improvment by installing the "strptime" gem: `fluent-gem install strptime`.
+Setting this to a known format can vastly improve your log ingestion speed if all most of your logs are in the same format. If there is an error parsing this format the timestamp will default to the ingestion time. If you are on Ruby 2.0 or later you can get a further performance improvement by installing the "strptime" gem: `fluent-gem install strptime`.
 
 For example to parse ISO8601 times with sub-second precision:
 
@@ -530,7 +530,7 @@ This following record `{"name": "Johnny", "request_id": "87d89af7daffad6"}` will
 { "name": "Johnny", "request_id": "87d89af7daffad6" }
 ```
 
-Fluentd re-emits events that failed to be indexed/ingested in Elasticsearch with a new and unique `_id` value, this means that congested Elasticsearch clusters that reject events (due to command queue overflow, for example) will cause Fluentd to re-emit the event with a new `_id`, however Elasticsearch may actually process both (or more) attempts (with some delay) and create duplicate events in the index (since each have a unique `_id` value), one possible workaround is to use the [fluent-plugin-genhashvalue](https://github.com/mtakemi/fluent-plugin-genhashvalue) plugin to generate a unique `_hash` key in the record of each event, this `_hash` record can be used as the `id_key` to prevent Elasticsearch from creating deplicate events.
+Fluentd re-emits events that failed to be indexed/ingested in Elasticsearch with a new and unique `_id` value, this means that congested Elasticsearch clusters that reject events (due to command queue overflow, for example) will cause Fluentd to re-emit the event with a new `_id`, however Elasticsearch may actually process both (or more) attempts (with some delay) and create duplicate events in the index (since each have a unique `_id` value), one possible workaround is to use the [fluent-plugin-genhashvalue](https://github.com/mtakemi/fluent-plugin-genhashvalue) plugin to generate a unique `_hash` key in the record of each event, this `_hash` record can be used as the `id_key` to prevent Elasticsearch from creating duplicate events.
 
 ```
 id_key _hash
@@ -540,7 +540,7 @@ Example configuration for [fluent-plugin-genhashvalue](https://github.com/mtakem
 ```
 <filter logs.**>
   @type genhashvalue
-  keys sessionid,requestid
+  keys session_id,request_id
   hash_type md5    # md5/sha1/sha256/sha512
   base64_enc true
   base91_enc false
@@ -745,7 +745,7 @@ http_backend typhoeus
 
 ### prefer_oj_serializer
 
-With default beavior, Elasticsearch client uses `Yajl` as JSON encoder/decoder.
+With default behavior, Elasticsearch client uses `Yajl` as JSON encoder/decoder.
 `Oj` is the alternative high performance JSON encoder/decoder.
 When this parameter sets as `true`, Elasticsearch client uses `Oj` as JSON encoder/decoder.
 
@@ -833,7 +833,7 @@ Here is a sample config:
   @type elasticsearch
   id_key _hash # specify same key name which is specified in hash_id_key
   remove_keys _hash # Elasticsearch doesn't like keys that start with _
-  # other settings are ommitted.
+  # other settings are omitted.
 </match>
 ```
 
@@ -961,7 +961,7 @@ If you want configurations to depend on information in messages, you can use `el
 
 v0.14 placeholders can handle `${tag}` for tag, `%Y%m%d` like strftime format, and custom record keys like as `record["mykey"]`.
 
-Note that custom chunk key is diffrent notations for `record_reformer` and `record_modifier`.
+Note that custom chunk key is different notations for `record_reformer` and `record_modifier`.
 They uses `record["some_key"]` to specify placeholders, but this feature uses `${key1}`, `${key2}` notation. And tag, time, and some arbitrary keys must be included in buffer directive attributes.
 
 They are used as below:
@@ -1052,7 +1052,7 @@ Or, fluent-plugin-elasticsearch v2.11.7 or later, users can inspect version inco
 validate_client_version true
 ```
 
-If you get the following error message, please consider to install compatibile elasticsearch client gems:
+If you get the following error message, please consider to install compatible elasticsearch client gems:
 
 ```
 Detected ES 5 but you use ES client 6.1.0.
@@ -1068,7 +1068,7 @@ A common cause of failure is that you are trying to connect to an Elasticsearch
 For example, `out_elasticsearch` set up ssl_version to TLSv1 due to historical reason.
 Modern Elasticsearch ecosystem requests to communicate with TLS v1.2 or later.
 But, in this case, `out_elasticsearch` conceals transporter part failure log by default.
-If you want to aquire transporter log, please consider to set the following configuration:
+If you want to acquire transporter log, please consider to set the following configuration:
 
 ```
 with_transporter_log true
@@ -1157,7 +1157,7 @@ Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/bin/f
 Oct 31 9:44:45 <ES-Host> systemd[1]: fluentd.service: Control process exited, code=exited status=1
 ```
 
-If you want to aquire transporter log, please consider to set the following configuration:
+If you want to acquire transporter log, please consider to set the following configuration:
 
 ```
 with_transporter_log true
@@ -1171,7 +1171,7 @@ Then, the following log is shown in Fluentd log:
 2018-10-31 10:00:57 +0900 [error]: #7 [Faraday::ConnectionFailed] Connection reset by peer - SSL_connect (Errno::ECONNRESET) {:host=>"<ES-Host>", :port=>9400, :scheme=>"https", :protocol=>"https"}
 ```
 
-The above logs indicates that using incompatibile SSL/TLS version between fluent-plugin-elasticsearch and nginx, which is reverse proxy, is root cause of this issue.
+The above logs indicates that using incompatible SSL/TLS version between fluent-plugin-elasticsearch and nginx, which is reverse proxy, is root cause of this issue.
 
 If you want to use TLS v1.2, please use `ssl_version` parameter like as:
 
@@ -1302,7 +1302,7 @@ If you use vanilla Fluentd, you can install it by:
 gem install typhoeus
 ```
 
-But, you use td-agent instead of vannila Fluentd, you have to use `td-agent-gem`:
+But, you use td-agent instead of vanilla Fluentd, you have to use `td-agent-gem`:
 
 ```
 td-agent-gem install typhoeus

data/fluent-plugin-elasticsearch.gemspec

@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-elasticsearch'
-  s.version       = '3.3.1'
+  s.version       = '3.3.2'
   s.authors       = ['diogo', 'pitr']
   s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com']
   s.description   = %q{Elasticsearch output plugin for Fluent event collector}

data/lib/fluent/plugin/out_elasticsearch.rb

@@ -274,6 +274,7 @@ EOC
       end
 
       @routing_key_name = configure_routing_key_name
+      @current_config = nil
     end
 
     def backend_options
@@ -353,7 +354,10 @@ EOC
       # check here to see if we already have a client connection for the given host
       connection_options = get_connection_options(host)
 
+      @_es = nil unless is_existing_connection(connection_options[:hosts])
+
       @_es ||= begin
+        @current_config = connection_options[:hosts].clone
         adapter_conf = lambda {|f| f.adapter @http_backend, @backend_options }
         local_reload_connections = @reload_connections
         if local_reload_connections && @reload_after > DEFAULT_RELOAD_AFTER
@@ -427,8 +431,8 @@ EOC
       }
     end
 
-    def connection_options_description
-      get_connection_options[:hosts].map do |host_info|
+    def connection_options_description(con_host=nil)
+      get_connection_options(con_host)[:hosts].map do |host_info|
         attributes = host_info.dup
         attributes[:password] = 'obfuscated' if attributes.has_key?(:password)
         attributes.inspect
@@ -565,7 +569,6 @@ EOC
         end
       end
 
-
       bulk_message.each do |info, msgs|
         send_bulk(msgs, tag, chunk, bulk_message_count[info], extracted_values, info) unless msgs.empty?
         msgs.clear
@@ -678,8 +681,23 @@ EOC
         @_es = nil if @reconnect_on_error
         @_es_info = nil if @reconnect_on_error
         # FIXME: identify unrecoverable errors and raise UnrecoverableRequestFailure instead
-        raise RecoverableRequestFailure, "could not push logs to Elasticsearch cluster (#{connection_options_description}): #{e.message}"
+        raise RecoverableRequestFailure, "could not push logs to Elasticsearch cluster (#{connection_options_description(info.host)}): #{e.message}"
+      end
+    end
+
+    def is_existing_connection(host)
+      # check if the host provided match the current connection
+      return false if @_es.nil?
+      return false if @current_config.nil?
+      return false if host.length != @current_config.length
+
+      for i in 0...host.length
+        if !host[i][:host].eql? @current_config[i][:host] || host[i][:port] != @current_config[i][:port]
+          return false
+        end
       end
+
+      return true
     end
   end
 end

data/lib/fluent/plugin/out_elasticsearch_dynamic.rb

@@ -28,7 +28,6 @@ module Fluent::Plugin
         @dynamic_config[key] = value.to_s
       }
       # end eval all configs
-      @current_config = nil
     end
 
     def create_meta_config_map
@@ -259,20 +258,5 @@ module Fluent::Plugin
       return false if [:@buffer_type].include?(param)
       return self.instance_variable_get(param).is_a?(String)
     end
-
-    def is_existing_connection(host)
-      # check if the host provided match the current connection
-      return false if @_es.nil?
-      return false if @current_config.nil?
-      return false if host.length != @current_config.length
-
-      for i in 0...host.length
-        if !host[i][:host].eql? @current_config[i][:host] || host[i][:port] != @current_config[i][:port]
-          return false
-        end
-      end
-
-      return true
-    end
   end
 end

data/test/plugin/test_out_elasticsearch.rb

@@ -1535,12 +1535,16 @@ class ElasticsearchOutput < Test::Unit::TestCase
                          ]
                        ))
       time = Time.parse Date.today.iso8601
-      pipeline_id = "5"
-      elastic_request = stub_elastic("http://myhost-5:9200/_bulk")
+      first_pipeline_id = "1"
+      second_pipeline_id = "2"
+      first_request = stub_elastic("http://myhost-1:9200/_bulk")
+      second_request = stub_elastic("http://myhost-2:9200/_bulk")
       driver.run(default_tag: 'test') do
-        driver.feed(time.to_i, sample_record.merge({"pipeline_id" => pipeline_id}))
+        driver.feed(time.to_i, sample_record.merge({"pipeline_id" => first_pipeline_id}))
+        driver.feed(time.to_i, sample_record.merge({"pipeline_id" => second_pipeline_id}))
       end
-      assert_requested(elastic_request)
+      assert_requested(first_request)
+      assert_requested(second_request)
     end
   end
 
@@ -1558,7 +1562,7 @@ class ElasticsearchOutput < Test::Unit::TestCase
     assert_equal(logstash_index, index_cmds.first['index']['_index'])
   end
 
-    def test_writes_to_logstash_index_with_specified_dateformat
+  def test_writes_to_logstash_index_with_specified_dateformat
     driver.configure("logstash_format true
                       logstash_dateformat %Y.%m")
     time = Time.parse Date.today.iso8601

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 3.3.1
+  version: 3.3.2
 platform: ruby
 authors:
 - diogo
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-03-08 00:00:00.000000000 Z
+date: 2019-03-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd