fluent-plugin-elasticsearch 4.0.10 → 4.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a12dc6596a56b7486d45129b64187021215433d2e7aa5428e96712fb63a345b5
-  data.tar.gz: 6fe5296f4a0afa23e863399f6d1cc9eeaa8f70c55e5cb979ca8db1bbf7a8988f
+  metadata.gz: 6b4183d1e2afbcf9e0e5f78b6cb6ed2fff3206f495a8676d38eb1dbc1ff7267f
+  data.tar.gz: 2f1644d04286ae42c51e869c6aadd928ec00b678a405dce838cb1d1802ebf5b1
 SHA512:
-  metadata.gz: 4f5d200eb1dee56f9f4fc05efc46cdbf440e0e2df7152e527985b124ac91ff74b0d263bee36f69fddec9f5ccc3730032145d51a95cc22fc00817f909fd4c2a7a
-  data.tar.gz: 11b0453ea6cbcfaf46e71866a96daf4ceba8f3877dd23e7aa124ea214f056b1b7f56bd8082ae273883a7403a2f1752c64beb377a2a3564c1d30757a8a5f101fa
+  metadata.gz: bc88d00e8800e7062dff4fdb9b6bc67c509ee851f5e726321783bec8902df6b548a2533d7c278151882b6f534e5adc680ba83964885d7bb9fd520340484c8478
+  data.tar.gz: 07f4379b3bfee4b22bb50f60c441fa60c87ee65967fa70523c3a6fdc7d0cc80c0fced8fd92958227eb64e353c8e623293f5c1e3d17fba848cdab707969d4f3d4

data/History.md

@@ -2,6 +2,23 @@
 
 ### [Unreleased]
 
+### 4.1.3
+- Load multiple templates even if template_name and template_file given (#799)
+- Handle elasticsearch-ruby 7.9.0 using HTTP method changes (#795)
+
+### 4.1.2
+- Use Hash#dig instead of Hash#[] on retrieving version information from Elasticsearch info API (#793)
+
+### 4.1.1
+- Correct ILM explain on logstash_format case (#786)
+
+### 4.1.0
+- Implement Fallback selector and configurable selector class (#782)
+
+### 4.0.11
+- Add custom and time placeholders combination testcase for ILM (#781)
+- Really support ILM on `logstash_format` enabled environment (#779)
+
 ### 4.0.10
 - filter_elasticsearch_genid: Use entire record as hash seed (#777)
 - Suppress type in meta with suppress_type_name parameter (#774)

data/README.md

@@ -78,6 +78,7 @@ Current maintainers: @cosmo0920
   + [Hash flattening](#hash-flattening)
   + [Generate Hash ID](#generate-hash-id)
   + [sniffer_class_name](#sniffer-class-name)
+  + [selector_class_name](#selector-class-name)
   + [reload_after](#reload-after)
   + [validate_client_version](#validate-client-version)
   + [unrecoverable_error_types](#unrecoverable-error-types)
@@ -120,10 +121,12 @@ Current maintainers: @cosmo0920
 
 ## Requirements
 
-| fluent-plugin-elasticsearch  | fluentd | ruby |
-|-------------------|---------|------|
-| >= 2.0.0 | >= v0.14.20 | >= 2.1 |
-|  < 2.0.0 | >= v0.12.0 | >= 1.9 |
+| fluent-plugin-elasticsearch  | fluentd     | ruby   |
+|:----------------------------:|:-----------:|:------:|
+| >= 4.0.1                     | >= v0.14.22 | >= 2.3 |
+| >= 3.2.4 && < 4.0.1          | >= v0.14.22 | >= 2.1 |
+| >= 2.0.0 && < 3.2.3          | >= v0.14.20 | >= 2.1 |
+|  < 2.0.0                     | >= v0.12.0  | >= 1.9 |
 
 NOTE: For v0.12 version, you should use 1.x.y version. Please send patch into v0.12 branch if you encountered 1.x version's bug.
 
@@ -428,7 +431,7 @@ Specify index templates in form of hash. Can contain multiple templates.
 templates { "template_name_1": "path_to_template_1_file", "template_name_2": "path_to_template_2_file"}
 ```
 
-If `template_file` and `template_name` are set, then this parameter will be ignored.
+**Note:** Before ES plugin v4.1.2, if `template_file` and `template_name` are set, then this parameter will be ignored. In 4.1.3 or later, `template_file` and `template_name` can work with `templates`.
 
 ### customize_template
 
@@ -478,6 +481,8 @@ deflector_alias test-current
 
 If [rollover_index](#rollover_index) is set, then this parameter will be in effect otherwise ignored.
 
+**NOTE:** Since 4.1.1, `deflector_alias` is prohibited to use with `enable_ilm`.
+
 ### index_prefix
 
 This parameter is marked as obsoleted.
@@ -989,7 +994,7 @@ reload_after 100
 
 #### Tips
 
-The included sniffer class does not required `out_elasticsearch`.
+The included sniffer class is not required `out_elasticsearch`.
 You should tell Fluentd where the sniffer class exists.
 
 If you use td-agent, you must put the following lines into `TD_AGENT_DEFAULT` file:
@@ -1006,6 +1011,38 @@ sniffer=$(td-agent-gem contents fluent-plugin-elasticsearch|grep elasticsearch_s
 $ fluentd -r $sniffer [AND YOUR OTHER OPTIONS]
 ```
 
+### Selector Class Name
+
+The default selector used by the `Elasticsearch::Transport` class works well when Fluentd should behave round robin and random selector cases. This doesn't work well when Fluentd should behave fallbacking from exhausted ES cluster to normal ES cluster.
+The parameter `selector_class_name` gives you the ability to provide your own Selector class to implement whatever selection nodes logic you require.
+
+The below configuration is using plugin built-in `ElasticseatchFallbackSelector`:
+
+```
+hosts exhausted-host:9201,normal-host:9200
+selector_class_name "Fluent::Plugin::ElasticseatchFallbackSelector"
+```
+
+#### Tips
+
+The included selector class is required in `out_elasticsearch` by default.
+But, your custom selector class is not required in `out_elasticsearch`.
+You should tell Fluentd where the selector class exists.
+
+If you use td-agent, you must put the following lines into `TD_AGENT_DEFAULT` file:
+
+```
+selector=/path/to/your_awesome_selector.rb
+TD_AGENT_OPTIONS="--use-v1-config -r $selector"
+```
+
+If you use Fluentd directly, you must pass the following lines as Fluentd command line option:
+
+```
+selector=/path/to/your_awesome_selector.rb
+$ fluentd -r $selector [AND YOUR OTHER OPTIONS]
+```
+
 ### Reload After
 
 When `reload_connections true`, this is the integer number of operations after which the plugin will
@@ -1720,7 +1757,7 @@ ILM target index alias is created with `index_name` or an index which is calcula
 
 From Elasticsearch plugin v4.0.0, ILM target index will be calculated from `index_name` (normal mode) or `logstash_prefix` (using with `logstash_format`as true).
 
-When using `deflector_alias` parameter, Elasticsearch plugin will create ILM target indices alias with `deflector_alias` instead of `index_name` or an index which is calculated from `logstash_prefix`. This behavior should be kept due to backward ILM feature compatibility.
+**NOTE:** Before Elasticsearch plugin v4.1.0, using `deflector_alias` parameter when ILM is enabled is permitted and handled, but, in the later releases such that 4.1.1 or later, it cannot use with when ILM is enabled.
 
 And also, ILM feature users should specify their Elasticsearch template for ILM enabled indices.
 Because ILM settings are injected into their Elasticsearch templates.
@@ -1733,7 +1770,13 @@ It usually should be used with default value which is `default`.
 
 Then, ILM parameters are used in alias index like as:
 
-`<index_name/logstash_prefix><index_separator><application_name>-000001`.
+##### Simple `index_name` case:
+
+`<index_name><index_separator><application_name>-000001`.
+
+##### `logstash_format` as `true` case:
+
+`<logstash_prefix><logstash_prefix_separator><application_name><logstash_prefix_separator><logstash_dateformat>-000001`.
 
 #### Example ILM settings
 

data/fluent-plugin-elasticsearch.gemspec

@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-elasticsearch'
-  s.version       = '4.0.10'
+  s.version       = '4.1.3'
   s.authors       = ['diogo', 'pitr', 'Hiroshi Hatake']
   s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com', 'cosmo0920.wp@gmail.com']
   s.description   = %q{Elasticsearch output plugin for Fluent event collector}

data/lib/fluent/plugin/elasticsearch_fallback_selector.rb

@@ -0,0 +1,9 @@
+require 'elasticsearch/transport/transport/connections/selector'
+
+class Fluent::Plugin::ElasticseatchFallbackSelector
+  include Elasticsearch::Transport::Transport::Connections::Selector::Base
+
+  def select(options={})
+    connections.first
+  end
+end

data/lib/fluent/plugin/elasticsearch_index_template.rb

@@ -65,11 +65,12 @@ module Fluent::ElasticsearchIndexTemplate
     end
   end
 
-  def template_install(name, template_file, overwrite, enable_ilm = false, deflector_alias_name = nil, ilm_policy_id = nil, host = nil)
+  def template_install(name, template_file, overwrite, enable_ilm = false, deflector_alias_name = nil, ilm_policy_id = nil, host = nil, target_index = nil)
     inject_template_name = get_template_name(enable_ilm, name, deflector_alias_name)
     if overwrite
       template_put(inject_template_name,
                    enable_ilm ? inject_ilm_settings_to_template(deflector_alias_name,
+                                                                target_index,
                                                                 ilm_policy_id,
                                                                 get_template(template_file)) :
                      get_template(template_file), host)
@@ -80,6 +81,7 @@ module Fluent::ElasticsearchIndexTemplate
     if !template_exists?(inject_template_name, host)
       template_put(inject_template_name,
                    enable_ilm ? inject_ilm_settings_to_template(deflector_alias_name,
+                                                                target_index,
                                                                 ilm_policy_id,
                                                                 get_template(template_file)) :
                      get_template(template_file), host)
@@ -89,10 +91,12 @@ module Fluent::ElasticsearchIndexTemplate
     end
   end
 
-  def template_custom_install(template_name, template_file, overwrite, customize_template, enable_ilm, deflector_alias_name, ilm_policy_id, host)
+  def template_custom_install(template_name, template_file, overwrite, customize_template, enable_ilm, deflector_alias_name, ilm_policy_id, host, target_index)
     template_custom_name = get_template_name(enable_ilm, template_name, deflector_alias_name)
     custom_template = if enable_ilm
-                        inject_ilm_settings_to_template(deflector_alias_name, ilm_policy_id,
+                        inject_ilm_settings_to_template(deflector_alias_name,
+                                                        target_index,
+                                                        ilm_policy_id,
                                                         get_custom_template(template_file,
                                                                             customize_template))
                       else
@@ -115,26 +119,30 @@ module Fluent::ElasticsearchIndexTemplate
     enable_ilm ? deflector_alias_name : template_name
   end
 
-  def inject_ilm_settings_to_template(deflector_alias_name, ilm_policy_id, template)
+  def inject_ilm_settings_to_template(deflector_alias, target_index, ilm_policy_id, template)
     log.debug("Overwriting index patterns when Index Lifecycle Management is enabled.")
     template.delete('template') if template.include?('template')
-    template['index_patterns'] = "#{deflector_alias_name}-*"
-    template['order'] = template['order'] ? template['order'] + deflector_alias_name.split('-').length : 50 + deflector_alias_name.split('-').length
+    template['index_patterns'] = "#{target_index}-*"
+    template['order'] = template['order'] ? template['order'] + target_index.split('-').length : 50 + target_index.split('-').length
     if template['settings'] && (template['settings']['index.lifecycle.name'] || template['settings']['index.lifecycle.rollover_alias'])
       log.debug("Overwriting index lifecycle name and rollover alias when Index Lifecycle Management is enabled.")
     end
-    template['settings'].update({ 'index.lifecycle.name' => ilm_policy_id, 'index.lifecycle.rollover_alias' => deflector_alias_name})
+    template['settings'].update({ 'index.lifecycle.name' => ilm_policy_id, 'index.lifecycle.rollover_alias' => deflector_alias})
     template
   end
 
-  def create_rollover_alias(index_prefix, rollover_index, deflector_alias_name, app_name, index_date_pattern, index_separator, enable_ilm, ilm_policy_id, ilm_policy, ilm_policy_overwrite, host)
+  def create_rollover_alias(target_index, rollover_index, deflector_alias_name, app_name, index_date_pattern, index_separator, enable_ilm, ilm_policy_id, ilm_policy, ilm_policy_overwrite, host)
      # ILM request to create alias.
     if rollover_index || enable_ilm
       if !client.indices.exists_alias(:name => deflector_alias_name)
-        if index_date_pattern.empty?
-          index_name_temp='<'+index_prefix.downcase+index_separator+app_name.downcase+'-000001>'
+        if @logstash_format
+          index_name_temp = '<'+target_index+'-000001>'
         else
-          index_name_temp='<'+index_prefix.downcase+index_separator+app_name.downcase+'-{'+index_date_pattern+'}-000001>'
+          if index_date_pattern.empty?
+            index_name_temp = '<'+target_index.downcase+index_separator+app_name.downcase+'-000001>'
+          else
+            index_name_temp = '<'+target_index.downcase+index_separator+app_name.downcase+'-{'+index_date_pattern+'}-000001>'
+          end
         end
         indexcreation(index_name_temp, host)
         body = rollover_alias_payload(deflector_alias_name)

data/lib/fluent/plugin/out_elasticsearch.rb

@@ -25,6 +25,7 @@ require_relative 'elasticsearch_error_handler'
 require_relative 'elasticsearch_index_template'
 require_relative 'elasticsearch_index_lifecycle_management'
 require_relative 'elasticsearch_tls'
+require_relative 'elasticsearch_fallback_selector'
 begin
   require_relative 'oj_serializer'
 rescue LoadError
@@ -50,7 +51,7 @@ module Fluent::Plugin
       end
     end
 
-    RequestInfo = Struct.new(:host, :index, :ilm_index)
+    RequestInfo = Struct.new(:host, :index, :ilm_index, :ilm_alias)
 
     attr_reader :alias_indexes
     attr_reader :template_names
@@ -137,6 +138,7 @@ EOC
     config_param :with_transporter_log, :bool, :default => false
     config_param :emit_error_for_missing_id, :bool, :default => false
     config_param :sniffer_class_name, :string, :default => nil
+    config_param :selector_class_name, :string, :default => nil
     config_param :reload_after, :integer, :default => DEFAULT_RELOAD_AFTER
     config_param :content_type, :enum, list: [:"application/json", :"application/x-ndjson"], :default => :"application/json",
                  :deprecated => <<EOC
@@ -232,7 +234,7 @@ EOC
       if !dry_run?
         if @template_name && @template_file
           if @enable_ilm
-            raise Fluent::ConfigError, "deflector_alias is prohibited to use with 'logstash_format at same time." if @logstash_format and @deflector_alias
+            raise Fluent::ConfigError, "deflector_alias is prohibited to use with enable_ilm at same time." if @deflector_alias
           end
           if @ilm_policy.empty? && @ilm_policy_overwrite
             raise Fluent::ConfigError, "ilm_policy_overwrite requires a non empty ilm_policy."
@@ -245,7 +247,8 @@ EOC
             template_installation_actual(@deflector_alias ? @deflector_alias : @index_name, @template_name, @customize_template, @application_name, @index_name, @ilm_policy_id)
           end
           verify_ilm_working if @enable_ilm
-        elsif @templates
+        end 
+        if @templates
           retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
             templates_hash_install(@templates, @template_overwrite)
           end
@@ -300,6 +303,13 @@ EOC
         raise Fluent::ConfigError, "Could not load sniffer class #{@sniffer_class_name}: #{ex}"
       end
 
+      @selector_class = nil
+      begin
+        @selector_class = Object.const_get(@selector_class_name) if @selector_class_name
+      rescue Exception => ex
+        raise Fluent::ConfigError, "Could not load selector class #{@selector_class_name}: #{ex}"
+      end
+
       @last_seen_major_version = if major_version = handle_last_seen_es_major_version
                                    major_version
                                  else
@@ -452,7 +462,10 @@ EOC
 
     def detect_es_major_version
       @_es_info ||= client.info
-      @_es_info["version"]["number"].to_i
+      unless version = @_es_info.dig("version", "number")
+        version = @default_elasticsearch_version
+      end
+      version.to_i
     end
 
     def client_library_version
@@ -566,6 +579,7 @@ EOC
                                                                               },
                                                                               sniffer_class: @sniffer_class,
                                                                               serializer_class: @serializer_class,
+                                                                              selector_class: @selector_class,
                                                                               compression: compress_connection,
                                                                             }), &adapter_conf)
         Elasticsearch::Client.new transport: transport
@@ -761,9 +775,9 @@ EOC
         begin
           meta, header, record = process_message(tag, meta, header, time, record, extracted_values)
           info = if @include_index_in_url
-                   RequestInfo.new(host, meta.delete("_index".freeze), meta["_index".freeze])
+                   RequestInfo.new(host, meta.delete("_index".freeze), meta["_index".freeze], meta.delete("_alias".freeze))
                  else
-                   RequestInfo.new(host, nil, meta["_index".freeze])
+                   RequestInfo.new(host, nil, meta["_index".freeze], meta.delete("_alias".freeze))
                  end
 
           if split_request?(bulk_message, info)
@@ -809,7 +823,7 @@ EOC
     end
 
     def process_message(tag, meta, header, time, record, extracted_values)
-      logstash_prefix, logstash_dateformat, index_name, type_name, _template_name, _customize_template, _deflector_alias, _application_name, pipeline, _ilm_policy_id = extracted_values
+      logstash_prefix, logstash_dateformat, index_name, type_name, _template_name, _customize_template, _deflector_alias, application_name, pipeline, _ilm_policy_id = extracted_values
 
       if @flatten_hashes
         record = flatten_record(record)
@@ -832,17 +846,19 @@ EOC
 
       target_index_parent, target_index_child_key = @target_index_key ? get_parent_of(record, @target_index_key) : nil
       if target_index_parent && target_index_parent[target_index_child_key]
-        target_index = target_index_parent.delete(target_index_child_key)
+        target_index_alias = target_index = target_index_parent.delete(target_index_child_key)
       elsif @logstash_format
         dt = dt.new_offset(0) if @utc_index
         target_index = "#{logstash_prefix}#{@logstash_prefix_separator}#{dt.strftime(logstash_dateformat)}"
+        target_index_alias = "#{logstash_prefix}#{@logstash_prefix_separator}#{application_name}#{@logstash_prefix_separator}#{dt.strftime(logstash_dateformat)}"
       else
-        target_index = index_name
+        target_index_alias = target_index = index_name
       end
 
       # Change target_index to lower-case since Elasticsearch doesn't
       # allow upper-case characters in index names.
       target_index = target_index.downcase
+      target_index_alias = target_index_alias.downcase
       if @include_tag_key
         record[@tag_key] = tag
       end
@@ -877,6 +893,7 @@ EOC
       meta.clear
       meta["_index".freeze] = target_index
       meta["_type".freeze] = target_type unless @last_seen_major_version >= 8
+      meta["_alias".freeze] = target_index_alias
 
       if @pipeline
         meta["pipeline".freeze] = pipeline
@@ -931,16 +948,16 @@ EOC
 
     def template_installation_actual(deflector_alias, template_name, customize_template, application_name, target_index, ilm_policy_id, host=nil)
       if template_name && @template_file
-        if @alias_indexes.include? deflector_alias
+        if !@logstash_format && @alias_indexes.include?(deflector_alias)
           log.debug("Index alias #{deflector_alias} already exists (cached)")
-        elsif @template_names.include? template_name
+        elsif !@logstash_format && @template_names.include?(template_name)
           log.debug("Template name #{template_name} already exists (cached)")
         else
           retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
             if customize_template
-              template_custom_install(template_name, @template_file, @template_overwrite, customize_template, @enable_ilm, deflector_alias, ilm_policy_id, host)
+              template_custom_install(template_name, @template_file, @template_overwrite, customize_template, @enable_ilm, deflector_alias, ilm_policy_id, host, target_index)
             else
-              template_install(template_name, @template_file, @template_overwrite, @enable_ilm, deflector_alias, ilm_policy_id, host)
+              template_install(template_name, @template_file, @template_overwrite, @enable_ilm, deflector_alias, ilm_policy_id, host, target_index)
             end
             ilm_policy = @ilm_policies[ilm_policy_id] || {}
             create_rollover_alias(target_index, @rollover_index, deflector_alias, application_name, @index_date_pattern, @index_separator, @enable_ilm, ilm_policy_id, ilm_policy, @ilm_policy_overwrite, host)
@@ -954,11 +971,11 @@ EOC
     # send_bulk given a specific bulk request, the original tag,
     # chunk, and bulk_message_count
     def send_bulk(data, tag, chunk, bulk_message_count, extracted_values, info)
-      logstash_prefix, _logstash_dateformat, index_name, _type_name, template_name, customize_template, deflector_alias, application_name, _pipeline, ilm_policy_id = extracted_values
+      _logstash_prefix, _logstash_dateformat, index_name, _type_name, template_name, customize_template, deflector_alias, application_name, _pipeline, ilm_policy_id = extracted_values
       if deflector_alias
         template_installation(deflector_alias, template_name, customize_template, application_name, index_name, ilm_policy_id, info.host)
       else
-        template_installation(info.ilm_index, template_name, customize_template, application_name, @logstash_format ? logstash_prefix : index_name, ilm_policy_id, info.host)
+        template_installation(info.ilm_index, template_name, customize_template, application_name, @logstash_format ? info.ilm_alias : index_name, ilm_policy_id, info.host)
       end
 
       begin

data/test/plugin/test_elasticsearch_fallback_selector.rb

@@ -0,0 +1,73 @@
+require_relative '../helper'
+require 'fluent/test/driver/output'
+require 'fluent/plugin/out_elasticsearch'
+
+class ElasticsearchFallbackSelectorTest < Test::Unit::TestCase
+  attr_accessor :index_cmds
+
+  def setup
+    Fluent::Test.setup
+    @driver = nil
+    log = Fluent::Engine.log
+    log.out.logs.slice!(0, log.out.logs.length)
+  end
+
+  def stub_elastic(url="http://localhost:9200/_bulk")
+    stub_request(:post, url).with do |req|
+      @index_cmds = req.body.split("\n").map {|r| JSON.parse(r) }
+    end
+  end
+
+  def stub_elastic_info(url="http://localhost:9200/", version="6.4.2")
+    body ="{\"version\":{\"number\":\"#{version}\"}}"
+    stub_request(:get, url).to_return({:status => 200, :body => body, :headers => { 'Content-Type' => 'json' } })
+  end
+
+  def stub_elastic_info_not_found(url="http://localhost:9200/", version="6.4.2")
+    stub_request(:get, url).to_return(:status => [404, "Not Found"])
+  end
+
+  def stub_elastic_info_unavailable(url="http://localhost:9200/", version="6.4.2")
+    stub_request(:get, url).to_return(:status => [503, "Service Unavailable"])
+  end
+
+  def sample_record(content={})
+    {'age' => 26, 'request_id' => '42', 'parent_id' => 'parent', 'routing_id' => 'routing'}.merge(content)
+  end
+
+  def driver(conf='')
+    @driver ||= Fluent::Test::Driver::Output.new(Fluent::Plugin::ElasticsearchOutput) {
+      # v0.12's test driver assume format definition. This simulates ObjectBufferedOutput format
+      if !defined?(Fluent::Plugin::Output)
+        def format(tag, time, record)
+          [time, record].to_msgpack
+        end
+      end
+    }.configure(conf)
+  end
+
+  def test_fallback_on_info
+    stub_elastic_info_not_found("http://localhost:9202/")
+    stub_elastic_info_unavailable("http://localhost:9201/")
+    stub_elastic_info
+    stub_elastic
+    config = %[
+      hosts localhost:9202,localhost:9201,localhost:9200
+      selector_class_name Fluent::Plugin::ElasticseatchFallbackSelector
+      @log_level debug
+      with_transporter_log true
+      reload_connections true
+      reload_after 10
+    ]
+    assert_raise(Elasticsearch::Transport::Transport::Errors::NotFound) do
+      driver(config)
+    end
+    driver.run(default_tag: 'test') do
+      driver.feed(sample_record)
+    end
+    assert_equal(2, index_cmds.length)
+    assert_equal("fluentd", index_cmds.first['index']['_index'])
+  end
+
+  # TODO: on feed phase test case
+end

data/test/plugin/test_in_elasticsearch.rb

@@ -20,6 +20,11 @@ class ElasticsearchInputTest < Test::Unit::TestCase
     @driver = nil
     log = Fluent::Engine.log
     log.out.logs.slice!(0, log.out.logs.length)
+    @http_method = if Gem::Version.new(Elasticsearch::VERSION) >= Gem::Version.new("7.9.0")
+                     :post
+                   else
+                     :get
+                   end
   end
 
   def driver(conf='')
@@ -313,7 +318,7 @@ class ElasticsearchInputTest < Test::Unit::TestCase
   end
 
   def test_emit
-    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"]}").
       to_return(status: 200, body: sample_response.to_s,
                 headers: {'Content-Type' => 'application/json'})
@@ -328,7 +333,7 @@ class ElasticsearchInputTest < Test::Unit::TestCase
 
   def test_emit_with_custom_index_name
     index_name = "logstash"
-    stub_request(:get, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"]}").
       to_return(status: 200, body: sample_response(index_name).to_s,
                 headers: {'Content-Type' => 'application/json'})
@@ -343,7 +348,7 @@ class ElasticsearchInputTest < Test::Unit::TestCase
 
   def test_emit_with_parse_timestamp
     index_name = "fluentd"
-    stub_request(:get, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"]}").
       to_return(status: 200, body: sample_response(index_name).to_s,
                 headers: {'Content-Type' => 'application/json'})
@@ -361,7 +366,7 @@ class ElasticsearchInputTest < Test::Unit::TestCase
 
   def test_emit_with_parse_timestamp_and_timstamp_format
     index_name = "fluentd"
-    stub_request(:get, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/#{index_name}/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"]}").
       to_return(status: 200, body: sample_response(index_name).to_s,
                 headers: {'Content-Type' => 'application/json'})
@@ -380,7 +385,7 @@ class ElasticsearchInputTest < Test::Unit::TestCase
   end
 
   def test_emit_with_docinfo
-    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"]}").
       to_return(status: 200, body: sample_response.to_s,
                 headers: {'Content-Type' => 'application/json'})
@@ -399,11 +404,11 @@ class ElasticsearchInputTest < Test::Unit::TestCase
   end
 
   def test_emit_with_slices
-    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"],\"slice\":{\"id\":0,\"max\":2}}").
       to_return(status: 200, body: sample_response.to_s,
                 headers: {'Content-Type' => 'application/json'})
-    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
+    stub_request(@http_method, "http://localhost:9200/fluentd/_search?scroll=1m&size=1000").
       with(body: "{\"sort\":[\"_doc\"],\"slice\":{\"id\":1,\"max\":2}}").
       to_return(status: 200, body: sample_response.to_s,
                 headers: {'Content-Type' => 'application/json'})
@@ -419,12 +424,12 @@ class ElasticsearchInputTest < Test::Unit::TestCase
   end
 
   def test_emit_with_size
-    stub_request(:get, "http://localhost:9200/fluentd/_search?scroll=1m&size=1").
+    stub_request(@http_method, "http://localhost:9200/fluentd/_search?scroll=1m&size=1").
       with(body: "{\"sort\":[\"_doc\"]}").
       to_return(status: 200, body: sample_scroll_response.to_s,
                 headers: {'Content-Type' => 'application/json'})
     connection = 0
-    scroll_request = stub_request(:get, "http://localhost:9200/_search/scroll?scroll=1m").
+    scroll_request = stub_request(@http_method, "http://localhost:9200/_search/scroll?scroll=1m").
       with(
         body: "{\"scroll_id\":\"WomkoUKG0QPB679Ulo6TqQgh3pIGRUmrl9qXXGK3EeiQh9rbYNasTkspZQcJ01uz\"}") do
       connection += 1

data/test/plugin/test_out_elasticsearch.rb

@@ -707,7 +707,10 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
       Fluent::Plugin::ElasticsearchOutput.module_eval(<<-CODE)
         def detect_es_major_version
           @_es_info ||= client.info
-          @_es_info["version"]["number"].to_i
+          unless version = @_es_info.dig("version", "number")
+            version = @default_elasticsearch_version
+          end
+          version.to_i
         end
       CODE
       Fluent::Plugin::ElasticsearchOutput.module_eval(<<-CODE)
@@ -753,7 +756,10 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
       Fluent::Plugin::ElasticsearchOutput.module_eval(<<-CODE)
         def detect_es_major_version
           @_es_info ||= client.info
-          @_es_info["version"]["number"].to_i
+          unless version = @_es_info.dig("version", "number")
+            version = @default_elasticsearch_version
+          end
+          version.to_i
         end
       CODE
       Fluent::Plugin::ElasticsearchOutput.module_eval(<<-CODE)
@@ -1053,7 +1059,7 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
       assert_requested(:put, "https://logs.google.com:777/es//_template/logstash", times: 1)
     end
 
-    def test_template_create_with_rollover_index_and_default_ilm_and_ilm_policy_overwrite
+    def test_template_create_with_rollover_index_and_default_ilm_on_logstash_format
       cwd = File.dirname(__FILE__)
       template_file = File.join(cwd, 'test_template.json')
 
@@ -1067,60 +1073,67 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
         template_name   logstash
         template_file   #{template_file}
         index_date_pattern now/w{xxxx.ww}
-        index_name      logstash
         enable_ilm      true
-        ilm_policy_overwrite true
-        ilm_policy {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_size":"60gb","max_age": "45d"}}}}}}
+        logstash_format true
+        application_name log
       }
 
+      date_str = Time.now.strftime("%Y.%m.%d")
       # connection start
       stub_request(:head, "https://logs.google.com:777/es//").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => "", :headers => {})
       # check if template exists
-      stub_request(:get, "https://logs.google.com:777/es//_template/logstash").
+      stub_request(:get, "https://logs.google.com:777/es//_template/logstash-#{date_str}").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 404, :body => "", :headers => {})
       # creation
-      stub_request(:put, "https://logs.google.com:777/es//_template/logstash").
+      stub_request(:put, "https://logs.google.com:777/es//_template/logstash-#{date_str}").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => "", :headers => {})
       # check if alias exists
-      stub_request(:head, "https://logs.google.com:777/es//_alias/logstash").
+      stub_request(:head, "https://logs.google.com:777/es//_alias/logstash-#{date_str}").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 404, :body => "", :headers => {})
-      stub_request(:get, "https://logs.google.com:777/es//_template/logstash").
+      stub_request(:get, "https://logs.google.com:777/es//_template/logstash-#{date_str}").
         with(basic_auth: ['john', 'doe']).
         to_return(status: 404, body: "", headers: {})
-      stub_request(:put, "https://logs.google.com:777/es//_template/logstash").
+      stub_request(:put, "https://logs.google.com:777/es//_template/logstash-#{date_str}").
         with(basic_auth: ['john', 'doe'],
-             body: "{\"settings\":{\"number_of_shards\":1,\"index.lifecycle.name\":\"logstash-policy\",\"index.lifecycle.rollover_alias\":\"logstash\"},\"mappings\":{\"type1\":{\"_source\":{\"enabled\":false},\"properties\":{\"host_name\":{\"type\":\"string\",\"index\":\"not_analyzed\"},\"created_at\":{\"type\":\"date\",\"format\":\"EEE MMM dd HH:mm:ss Z YYYY\"}}}},\"index_patterns\":\"logstash-*\",\"order\":51}").
+             body: "{\"settings\":{\"number_of_shards\":1,\"index.lifecycle.name\":\"logstash-policy\",\"index.lifecycle.rollover_alias\":\"logstash-log-#{date_str}\"},\"mappings\":{\"type1\":{\"_source\":{\"enabled\":false},\"properties\":{\"host_name\":{\"type\":\"string\",\"index\":\"not_analyzed\"},\"created_at\":{\"type\":\"date\",\"format\":\"EEE MMM dd HH:mm:ss Z YYYY\"}}}},\"index_patterns\":\"logstash-log-#{date_str}-*\",\"order\":53}").
         to_return(status: 200, body: "", headers: {})
       # put the alias for the index
-      stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E").
+      stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-log-#{date_str}-000001%3E").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => "", :headers => {})
-      stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E/#{alias_endpoint}/logstash").
+
+      stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-log-#{date_str}-000001%3E/#{alias_endpoint}/logstash-#{date_str}").
         with(basic_auth: ['john', 'doe'],
-             :body => "{\"aliases\":{\"logstash\":{\"is_write_index\":true}}}").
-        to_return(:status => 200, :body => "", :headers => {})
+             body: "{\"aliases\":{\"logstash-#{date_str}\":{\"is_write_index\":true}}}").
+        to_return(status: 200, body: "", headers: {})
       stub_request(:get, "https://logs.google.com:777/es//_xpack").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => '{"features":{"ilm":{"available":true,"enabled":true}}}', :headers => {"Content-Type"=> "application/json"})
       stub_request(:get, "https://logs.google.com:777/es//_ilm/policy/logstash-policy").
         with(basic_auth: ['john', 'doe']).
-        to_return(:status => 200, :body => "", :headers => {})
+        to_return(:status => 404, :body => "", :headers => {})
       stub_request(:put, "https://logs.google.com:777/es//_ilm/policy/logstash-policy").
         with(basic_auth: ['john', 'doe'],
-             :body => "{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"60gb\",\"max_age\":\"45d\"}}}}}}").
+             :body => "{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"50gb\",\"max_age\":\"30d\"}}}}}}").
         to_return(:status => 200, :body => "", :headers => {})
 
       driver(config)
 
-      assert_requested(:put, "https://logs.google.com:777/es//_template/logstash", times: 1)
+      elastic_request = stub_elastic("https://logs.google.com:777/es//_bulk")
+      driver.run(default_tag: 'test') do
+        driver.feed(sample_record)
+      end
+      assert_requested(:put, "https://logs.google.com:777/es//_template/logstash-#{date_str}", times: 1)
+
+      assert_requested(elastic_request)
     end
 
-    def test_template_create_with_rollover_index_and_default_ilm_with_deflector_alias
+    def test_template_create_with_rollover_index_and_default_ilm_and_ilm_policy_overwrite
       cwd = File.dirname(__FILE__)
       template_file = File.join(cwd, 'test_template.json')
 
@@ -1134,9 +1147,10 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
         template_name   logstash
         template_file   #{template_file}
         index_date_pattern now/w{xxxx.ww}
-        deflector_alias myapp_deflector
         index_name      logstash
         enable_ilm      true
+        ilm_policy_overwrite true
+        ilm_policy {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_size":"60gb","max_age": "45d"}}}}}}
       }
 
       # connection start
@@ -1152,38 +1166,63 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => "", :headers => {})
       # check if alias exists
-      stub_request(:head, "https://logs.google.com:777/es//_alias/myapp_deflector").
+      stub_request(:head, "https://logs.google.com:777/es//_alias/logstash").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 404, :body => "", :headers => {})
-      stub_request(:get, "https://logs.google.com:777/es//_template/myapp_deflector").
+      stub_request(:get, "https://logs.google.com:777/es//_template/logstash").
         with(basic_auth: ['john', 'doe']).
         to_return(status: 404, body: "", headers: {})
-      stub_request(:put, "https://logs.google.com:777/es//_template/myapp_deflector").
+      stub_request(:put, "https://logs.google.com:777/es//_template/logstash").
         with(basic_auth: ['john', 'doe'],
-             body: "{\"settings\":{\"number_of_shards\":1,\"index.lifecycle.name\":\"logstash-policy\",\"index.lifecycle.rollover_alias\":\"myapp_deflector\"},\"mappings\":{\"type1\":{\"_source\":{\"enabled\":false},\"properties\":{\"host_name\":{\"type\":\"string\",\"index\":\"not_analyzed\"},\"created_at\":{\"type\":\"date\",\"format\":\"EEE MMM dd HH:mm:ss Z YYYY\"}}}},\"index_patterns\":\"myapp_deflector-*\",\"order\":51}").
+             body: "{\"settings\":{\"number_of_shards\":1,\"index.lifecycle.name\":\"logstash-policy\",\"index.lifecycle.rollover_alias\":\"logstash\"},\"mappings\":{\"type1\":{\"_source\":{\"enabled\":false},\"properties\":{\"host_name\":{\"type\":\"string\",\"index\":\"not_analyzed\"},\"created_at\":{\"type\":\"date\",\"format\":\"EEE MMM dd HH:mm:ss Z YYYY\"}}}},\"index_patterns\":\"logstash-*\",\"order\":51}").
         to_return(status: 200, body: "", headers: {})
       # put the alias for the index
       stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => "", :headers => {})
-      stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E/#{alias_endpoint}/myapp_deflector").
+      stub_request(:put, "https://logs.google.com:777/es//%3Clogstash-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E/#{alias_endpoint}/logstash").
         with(basic_auth: ['john', 'doe'],
-             :body => "{\"aliases\":{\"myapp_deflector\":{\"is_write_index\":true}}}").
+             :body => "{\"aliases\":{\"logstash\":{\"is_write_index\":true}}}").
         to_return(:status => 200, :body => "", :headers => {})
       stub_request(:get, "https://logs.google.com:777/es//_xpack").
         with(basic_auth: ['john', 'doe']).
         to_return(:status => 200, :body => '{"features":{"ilm":{"available":true,"enabled":true}}}', :headers => {"Content-Type"=> "application/json"})
       stub_request(:get, "https://logs.google.com:777/es//_ilm/policy/logstash-policy").
         with(basic_auth: ['john', 'doe']).
-        to_return(:status => 404, :body => "", :headers => {})
+        to_return(:status => 200, :body => "", :headers => {})
       stub_request(:put, "https://logs.google.com:777/es//_ilm/policy/logstash-policy").
         with(basic_auth: ['john', 'doe'],
-             :body => "{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"50gb\",\"max_age\":\"30d\"}}}}}}").
+             :body => "{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"60gb\",\"max_age\":\"45d\"}}}}}}").
         to_return(:status => 200, :body => "", :headers => {})
 
       driver(config)
 
-      assert_requested(:put, "https://logs.google.com:777/es//_template/myapp_deflector", times: 1)
+      assert_requested(:put, "https://logs.google.com:777/es//_template/logstash", times: 1)
+    end
+
+    def test_template_create_with_rollover_index_and_default_ilm_with_deflector_alias
+      cwd = File.dirname(__FILE__)
+      template_file = File.join(cwd, 'test_template.json')
+
+      config = %{
+        host            logs.google.com
+        port            777
+        scheme          https
+        path            /es/
+        user            john
+        password        doe
+        template_name   logstash
+        template_file   #{template_file}
+        index_date_pattern now/w{xxxx.ww}
+        deflector_alias myapp_deflector
+        index_name      logstash
+        enable_ilm      true
+      }
+
+      # Should raise error because multiple alias indices IllegalArgument Error on executing ILM feature
+      assert_raise(Fluent::ConfigError) do
+        driver(config)
+      end
     end
 
     def test_template_create_with_rollover_index_and_default_ilm_with_empty_index_date_pattern
@@ -1607,6 +1646,83 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
 
       assert_requested(elastic_request)
     end
+
+    def test_template_create_with_rollover_index_and_default_ilm_and_custom_and_time_placeholders
+      cwd = File.dirname(__FILE__)
+      template_file = File.join(cwd, 'test_template.json')
+
+      config = Fluent::Config::Element.new(
+        'ROOT', '', {
+          '@type' => 'elasticsearch',
+          'host' => 'logs.google.com',
+          'port' => 777,
+          'scheme' => "https",
+          'path' => "/es/",
+          'user' => 'john',
+          'password' => 'doe',
+          'template_name' => 'logstash',
+          'template_file' => "#{template_file}",
+          'index_date_pattern' => 'now/w{xxxx.ww}',
+          'index_name' => "${taskDef}-%Y.%m",
+          'enable_ilm' => true,
+        }, [
+          Fluent::Config::Element.new('buffer', 'tag, time, taskDef', {
+                                        'chunk_keys' => ['tag', 'time', 'taskDef'],
+                                        'timekey' => 3600,
+                                      }, [])
+        ]
+      )
+
+      task_def_value = "task_definition"
+      date_str = Time.now.strftime("%Y.%m")
+      # connection start
+      stub_request(:head, "https://logs.google.com:777/es//").
+        with(basic_auth: ['john', 'doe']).
+        to_return(:status => 200, :body => "", :headers => {})
+      # check if template exists
+      stub_request(:get, "https://logs.google.com:777/es//_template/#{task_def_value}-#{date_str}").
+        with(basic_auth: ['john', 'doe']).
+        to_return(:status => 404, :body => "", :headers => {})
+      # creation
+      stub_request(:put, "https://logs.google.com:777/es//_template/#{task_def_value}-#{date_str}").
+        with(basic_auth: ['john', 'doe'],
+             body: "{\"settings\":{\"number_of_shards\":1,\"index.lifecycle.name\":\"logstash-policy\",\"index.lifecycle.rollover_alias\":\"#{task_def_value}-#{date_str}\"},\"mappings\":{\"type1\":{\"_source\":{\"enabled\":false},\"properties\":{\"host_name\":{\"type\":\"string\",\"index\":\"not_analyzed\"},\"created_at\":{\"type\":\"date\",\"format\":\"EEE MMM dd HH:mm:ss Z YYYY\"}}}},\"index_patterns\":\"#{task_def_value}-#{date_str}-*\",\"order\":52}").
+        to_return(:status => 200, :body => "", :headers => {})
+      # check if alias exists
+      stub_request(:head, "https://logs.google.com:777/es//_alias/#{task_def_value}-#{date_str}").
+        with(basic_auth: ['john', 'doe']).
+        to_return(:status => 404, :body => "", :headers => {})
+      # put the alias for the index
+      stub_request(:put, "https://logs.google.com:777/es//%3C#{task_def_value}-#{date_str}-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E").
+        with(basic_auth: ['john', 'doe']).
+        to_return(:status => 200, :body => "", :headers => {})
+      stub_request(:put, "https://logs.google.com:777/es//%3C#{task_def_value}-#{date_str}-default-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E/#{alias_endpoint}/#{task_def_value}-#{date_str}").
+        with(basic_auth: ['john', 'doe'],
+             :body => "{\"aliases\":{\"#{task_def_value}-#{date_str}\":{\"is_write_index\":true}}}").
+        to_return(:status => 200, :body => "", :headers => {})
+      stub_request(:get, "https://logs.google.com:777/es//_xpack").
+        with(basic_auth: ['john', 'doe']).
+        to_return(:status => 200, :body => '{"features":{"ilm":{"available":true,"enabled":true}}}', :headers => {"Content-Type"=> "application/json"})
+      stub_request(:get, "https://logs.google.com:777/es//_ilm/policy/logstash-policy").
+        with(basic_auth: ['john', 'doe']).
+        to_return(:status => 404, :body => "", :headers => {})
+      stub_request(:put, "https://logs.google.com:777/es//_ilm/policy/logstash-policy").
+        with(basic_auth: ['john', 'doe'],
+             :body => "{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"50gb\",\"max_age\":\"30d\"}}}}}}").
+        to_return(:status => 200, :body => "", :headers => {})
+
+      driver(config)
+
+      elastic_request = stub_elastic("https://logs.google.com:777/es//_bulk")
+      driver.run(default_tag: 'test') do
+        driver.feed(sample_record.merge("taskDef" => task_def_value))
+      end
+      assert_equal("#{task_def_value}-#{date_str}", index_cmds.first['index']['_index'])
+
+      assert_equal ["#{task_def_value}-#{date_str}"], driver.instance.alias_indexes
+
+      assert_requested(elastic_request)
+    end
   end
 
   def test_custom_template_create
@@ -1845,6 +1961,7 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
       application_name myapp
     }
 
+    timestr = Time.now.strftime("%Y.%m.%d")
     # connection start
     stub_request(:head, "https://logs.google.com:777/es//").
       with(basic_auth: ['john', 'doe']).
@@ -1858,17 +1975,17 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
       with(basic_auth: ['john', 'doe']).
       to_return(:status => 200, :body => "", :headers => {})
     # creation of index which can rollover
-    stub_request(:put, "https://logs.google.com:777/es//%3Cmylogs-myapp-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E").
+    stub_request(:put, "https://logs.google.com:777/es//%3Cmylogs-myapp-#{timestr}-000001%3E").
       with(basic_auth: ['john', 'doe']).
       to_return(:status => 200, :body => "", :headers => {})
     # check if alias exists
-    timestr = Time.now.strftime("%Y.%m.%d")
     stub_request(:head, "https://logs.google.com:777/es//_alias/mylogs-#{timestr}").
       with(basic_auth: ['john', 'doe']).
       to_return(:status => 404, :body => "", :headers => {})
     # put the alias for the index
-    stub_request(:put, "https://logs.google.com:777/es//%3Cmylogs-myapp-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E/#{alias_endpoint}/mylogs-#{timestr}").
-      with(basic_auth: ['john', 'doe']).
+    stub_request(:put, "https://logs.google.com:777/es//%3Cmylogs-myapp-#{timestr}-000001%3E/#{alias_endpoint}/mylogs-#{timestr}").
+      with(basic_auth: ['john', 'doe'],
+           body: "{\"aliases\":{\"mylogs-#{timestr}\":{\"is_write_index\":true}}}").
       to_return(:status => 200, :body => "", :headers => {})
 
     driver(config)
@@ -2051,52 +2168,10 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
         enable_ilm      true
       }
 
-      # connection start
-      stub_request(:head, "https://logs.google.com:777/es//").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 200, :body => "", :headers => {})
-      # check if template exists
-      stub_request(:get, "https://logs.google.com:777/es//_template/myapp_alias_template").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 404, :body => "", :headers => {})
-      # creation
-      stub_request(:put, "https://logs.google.com:777/es//_template/myapp_alias_template").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 200, :body => "", :headers => {})
-      # creation of index which can rollover
-      stub_request(:put, "https://logs.google.com:777/es//%3Cmylogs-myapp-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 200, :body => "", :headers => {})
-      # check if alias exists
-      stub_request(:head, "https://logs.google.com:777/es//_alias/myapp_deflector").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 404, :body => "", :headers => {})
-      stub_request(:get, "https://logs.google.com:777/es//_template/myapp_deflector").
-        with(basic_auth: ['john', 'doe']).
-        to_return(status: 404, body: "", headers: {})
-      stub_request(:put, "https://logs.google.com:777/es//_template/myapp_deflector").
-        with(basic_auth: ['john', 'doe'],
-             body: "{\"order\":6,\"settings\":{\"index.lifecycle.name\":\"fluentd-policy\",\"index.lifecycle.rollover_alias\":\"myapp_deflector\"},\"mappings\":{},\"aliases\":{\"myapp-logs-alias\":{}},\"index_patterns\":\"myapp_deflector-*\"}").
-        to_return(status: 200, body: "", headers: {})
-      # put the alias for the index
-      stub_request(:put, "https://logs.google.com:777/es//%3Cmylogs-myapp-%7Bnow%2Fw%7Bxxxx.ww%7D%7D-000001%3E/#{alias_endpoint}/myapp_deflector").
-        with(basic_auth: ['john', 'doe'],
-             :body => "{\"aliases\":{\"myapp_deflector\":{\"is_write_index\":true}}}").
-        to_return(:status => 200, :body => "", :headers => {})
-      stub_request(:get, "https://logs.google.com:777/es//_xpack").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 200, :body => '{"features":{"ilm":{"available":true,"enabled":true}}}', :headers => {"Content-Type"=> "application/json"})
-      stub_request(:get, "https://logs.google.com:777/es//_ilm/policy/fluentd-policy").
-        with(basic_auth: ['john', 'doe']).
-        to_return(:status => 404, :body => "", :headers => {})
-      stub_request(:put, "https://logs.google.com:777/es//_ilm/policy/fluentd-policy").
-        with(basic_auth: ['john', 'doe'],
-             :body => "{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"50gb\",\"max_age\":\"30d\"}}}}}}").
-        to_return(:status => 200, :body => "", :headers => {})
-
-      driver(config)
-
-      assert_requested(:put, "https://logs.google.com:777/es//_template/myapp_deflector", times: 1)
+      # Should raise error because multiple alias indices IllegalArgument Error on executing ILM feature
+      assert_raise(Fluent::ConfigError) do
+        driver(config)
+      end
     end
 
     def test_custom_template_with_rollover_index_create_and_default_ilm_and_placeholders
@@ -2584,7 +2659,7 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
     assert_requested(:put, "https://logs.google.com:777/es//_template/logstash3", times: 1)
   end
 
-  def test_templates_not_used
+  def test_templates_are_also_used
     cwd = File.dirname(__FILE__)
     template_file = File.join(cwd, 'test_template.json')
 
@@ -2628,8 +2703,8 @@ class ElasticsearchOutputTest < Test::Unit::TestCase
 
     assert_requested(:put, "https://logs.google.com:777/es//_template/logstash", times: 1)
 
-    assert_not_requested(:put, "https://logs.google.com:777/es//_template/logstash1")
-    assert_not_requested(:put, "https://logs.google.com:777/es//_template/logstash2")
+    assert_requested(:put, "https://logs.google.com:777/es//_template/logstash1")
+    assert_requested(:put, "https://logs.google.com:777/es//_template/logstash2")
   end
 
   def test_templates_can_be_partially_created_if_error_occurs

metadata

@@ -1,16 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 4.0.10
+  version: 4.1.3
 platform: ruby
 authors:
 - diogo
 - pitr
 - Hiroshi Hatake
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-07-06 00:00:00.000000000 Z
+date: 2020-09-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -160,6 +160,7 @@ files:
 - lib/fluent/plugin/elasticsearch_constants.rb
 - lib/fluent/plugin/elasticsearch_error.rb
 - lib/fluent/plugin/elasticsearch_error_handler.rb
+- lib/fluent/plugin/elasticsearch_fallback_selector.rb
 - lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb
 - lib/fluent/plugin/elasticsearch_index_template.rb
 - lib/fluent/plugin/elasticsearch_simple_sniffer.rb
@@ -172,6 +173,7 @@ files:
 - test/helper.rb
 - test/plugin/test_alias_template.json
 - test/plugin/test_elasticsearch_error_handler.rb
+- test/plugin/test_elasticsearch_fallback_selector.rb
 - test/plugin/test_elasticsearch_index_lifecycle_management.rb
 - test/plugin/test_elasticsearch_tls.rb
 - test/plugin/test_filter_elasticsearch_genid.rb
@@ -186,7 +188,7 @@ licenses:
 - Apache-2.0
 metadata:
   changelog_uri: https://github.com/uken/fluent-plugin-elasticsearch/blob/master/History.md
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -202,13 +204,14 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.1.2
-signing_key: 
+signing_key:
 specification_version: 4
 summary: Elasticsearch output plugin for Fluent event collector
 test_files:
 - test/helper.rb
 - test/plugin/test_alias_template.json
 - test/plugin/test_elasticsearch_error_handler.rb
+- test/plugin/test_elasticsearch_fallback_selector.rb
 - test/plugin/test_elasticsearch_index_lifecycle_management.rb
 - test/plugin/test_elasticsearch_tls.rb
 - test/plugin/test_filter_elasticsearch_genid.rb