fluent-plugin-opensearch 1.0.0 → 1.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 56dc0da995d5f55c59e13afd75452b46e824f8337de6b074655dc8c92926eba8
-  data.tar.gz: dffab1c953b558fe1001df5c59844f1d71f8a2847199afade1b74ff9a05f739a
+  metadata.gz: 217feaf328809b259c6eb716e968b02417df7b10075a9a1e278c6e337d8a660b
+  data.tar.gz: 351327bd18d5dd3aa7b6a360ebba4a4f574a86b65ed470cffdf15b38e5d16022
 SHA512:
-  metadata.gz: da67be6908b8000b6904655b0492cfedbedddae45dd7b408a41698e7c02ea1778b1fb8c64be4097b5cbfed1e58601c30a57892d07706de4782526cb0a6edd0c4
-  data.tar.gz: f002f9ccec9f8dd7c292ef3d88d8425e0f05e3651cf831eb86204059726643db43502c3285b4ad8410840e7bb29d814f2e243c1b6671c24909abb1f1b7b14824
+  metadata.gz: c374adb923d9533e50c1c11355b304ba04ca92112dee36dcacd886d53be0ec910481559c7fdb0f6cdb3b61c7c949b5b917c3540d4bbd2789a9d568015cb52cbb
+  data.tar.gz: 11259ac1ee5e4dfe0de5b18336fd2167b40b8c10c381a283068557da1d206e06c03d5d76ef46411a8803b00c333370f47c21fbfdf4331e528cb05a145803a9b4

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,37 +1,29 @@
 ---
 name:  Bug Report
-about: Create a report to help us improve. If you have questions about ES plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
+about: Create a report to help us improve. If you have questions about OpenSearch plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
 
 ---
 
 (check apply)
-- [ ] read [the contribution guideline](https://github.com/uken/fluent-plugin-elasticsearch/blob/master/CONTRIBUTING.md)
+- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
 - [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
 
-#### Problem
-
-...
-
 #### Steps to replicate
 
-Either clone and modify https://gist.github.com/pitr/9a518e840db58f435911
-
-**OR**
-
 Provide example config and message
 
 #### Expected Behavior or What you need to ask
 
 ...
 
-#### Using Fluentd and ES plugin versions
+#### Using Fluentd and OpenSearch plugin versions
 
 * OS version
 * Bare Metal or within Docker or Kubernetes or others?
-* Fluentd v0.12 or v0.14/v1.0
+* Fluentd v1.0 or later
   * paste result of ``fluentd --version`` or ``td-agent --version``
-* ES plugin 3.x.y/2.x.y or 1.x.y
+* OpenSearch plugin version
   * paste boot log of fluentd or td-agent
   * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
-* ES version (optional)
-* ES template(s) (optional)
+* OpenSearch version (optional)
+* OpenSearch template(s) (optional)

data/History.md

@@ -2,5 +2,22 @@
 
 ### [Unreleased]
 
+### 1.0.3
+ - Configurable unrecoverable record types (#40)
+ - Handle exceptions on retrieving AWS credentials (#39)
+ - Suppress emit error label events (#38)
+ - Provide suppress_type_name parameter (#29)
+ - Honor @time_key (data streams) (#28)
+
+### 1.0.2
+ - Honor @hosts parameter for Data Streams (#21)
+ - Use template_file for Data Streams (#20)
+ - Specify host argument if needed (#11)
+
+### 1.0.1
+ -  Add testcases for hosts parameter (#10)
+ - Permit to handle @hosts parameter (#9)
+ - Retry creating data stream/template when OpenSearch is unreachable (#8)
+
 ### 1.0.0
  - Initial public gem release.

data/README.OpenSearchInput.md

@@ -17,6 +17,7 @@
   + [reload_on_failure](#reload_on_failure)
   + [resurrect_after](#resurrect_after)
   + [with_transporter_log](#with_transporter_log)
+  + [emit_error_label_event](#emit-error-label-event)
   + [Client/host certificate options](#clienthost-certificate-options)
   + [sniffer_class_name](#sniffer-class-name)
   + [custom_headers](#custom_headers)
@@ -190,6 +191,18 @@ We recommend to set this true if you start to debug this plugin.
 with_transporter_log true
 ```
 
+### emit_error_label_event
+
+Default `emit_error_label_event` value is `true`.
+
+Emitting error label events is default behavior.
+
+When using the followin configuration, OpenSearch plugin will cut error events on error handler:
+
+```aconf
+emit_error_label_event false
+```
+
 ### Client/host certificate options
 
 Need to verify OpenSearch's certificate?  You can use the following parameter to specify a CA instead of using an environment variable.

data/README.md

@@ -6,7 +6,7 @@
 ![Testing on Ubuntu](https://github.com/fluent/fluent-plugin-opensearch/workflows/Testing%20on%20Ubuntu/badge.svg?branch=main)
 [![Coverage Status](https://coveralls.io/repos/github/fluent/fluent-plugin-opensearch/badge.svg?branch=upload-coverage-into-coveralls)](https://coveralls.io/github/fluent/fluent-plugin-opensearch?branch=main)
 
-Send your logs to OpenSearch (and search them with OpenSearch Dashboard maybe?)
+Send your logs to OpenSearch (and search them with OpenSearch Dashboards maybe?)
 
 * [Installation](#installation)
 * [Usage](#usage)
@@ -76,6 +76,8 @@ Send your logs to OpenSearch (and search them with OpenSearch Dashboard maybe?)
   + [reload_after](#reload-after)
   + [validate_client_version](#validate-client-version)
   + [unrecoverable_error_types](#unrecoverable-error-types)
+  + [unrecoverable_record_types](#unrecoverable-record-types)
+  + [emit_error_label_event](#emit-error-label-event)
   + [verify os version at startup](#verify_os_version_at_startup)
   + [default_opensearch_version](#default_opensearch_version)
   + [custom_headers](#custom_headers)
@@ -356,6 +358,15 @@ utc_index true
 
 By default, the records inserted into index `logstash-YYMMDD` with UTC (Coordinated Universal Time). This option allows to use local time if you describe utc_index to false.
 
+
+### suppress_type_name
+
+If OpenSearch cluster complains types removal warnings, this can be suppressed with:
+
+```
+suppress_type_name true
+```
+
 ### target_index_key
 
 Tell this plugin to find the index name to write to in the record under this key in preference to other mechanisms. Key can be specified as path to nested record using dot ('.') as a separator.
@@ -1097,6 +1108,33 @@ Then, remove `rejected_execution_exception` from `unrecoverable_error_types` par
 unrecoverable_error_types ["out_of_memory_error"]
 ```
 
+
+### Unrecoverable Record Types
+
+Default `unrecoverable_record_types` parameter is set up loosely.
+Because `json_parse_exception` is caused by invalid JSON record.
+Another possible unrecoverable record error should be included within this paramater.
+
+If you want to handle `security_exception` as _unrecoverable exceptions_, please consider to change `unrecoverable_record_types` parameter from default value:
+
+```
+unrecoverable_record_types ["json_parse_exception", "security_exception"]
+```
+
+If this error type is included in OpenSearch response, an invalid record should be sent in `@ERROR` data pipeline.
+
+### emit_error_label_event
+
+Default `emit_error_label_event` value is `true`.
+
+Emitting error label events is default behavior.
+
+When using the followin configuration, OpenSearch plugin will cut error events on error handler:
+
+```aconf
+emit_error_label_event false
+```
+
 ### verify_os_version_at_startup
 
 Because OpenSearch plugin will ought to change behavior each of OpenSearch major versions.

data/fluent-plugin-opensearch.gemspec

@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-opensearch'
-  s.version       = '1.0.0'
+  s.version       = '1.0.3'
   s.authors       = ['Hiroshi Hatake']
   s.email         = ['cosmo0920.wp@gmail.com']
   s.description   = %q{Opensearch output plugin for Fluent event collector}

data/lib/fluent/plugin/in_opensearch.rb

@@ -73,6 +73,7 @@ module Fluent::Plugin
     config_param :ca_file, :string, :default => nil
     config_param :ssl_version, :enum, list: [:SSLv23, :TLSv1, :TLSv1_1, :TLSv1_2], :default => :TLSv1_2
     config_param :with_transporter_log, :bool, :default => false
+    config_param :emit_error_label_event, :bool, :default => true
     config_param :sniffer_class_name, :string, :default => nil
     config_param :custom_headers, :hash, :default => {}
     config_param :docinfo_fields, :array, :default => ['_index', '_type', '_id']
@@ -180,6 +181,13 @@ module Fluent::Plugin
       }
     end
 
+    def emit_error_label_event(&block)
+      # If `emit_error_label_event` is specified as false, error event emittions are not occurred.
+      if emit_error_label_event
+        block.call
+      end
+    end
+
     def start
       super
 
@@ -224,7 +232,9 @@ module Fluent::Plugin
     def parse_time(value, event_time, tag)
       @timestamp_parser.call(value)
     rescue => e
-      router.emit_error_event(@timestamp_parse_error_tag, Fluent::Engine.now, {'tag' => tag, 'time' => event_time, 'format' => @timestamp_key_format, 'value' => value}, e)
+      emit_error_label_event do
+        router.emit_error_event(@timestamp_parse_error_tag, Fluent::Engine.now, {'tag' => tag, 'time' => event_time, 'format' => @timestamp_key_format, 'value' => value}, e)
+      end
       return Time.at(event_time).to_time
     end
 

data/lib/fluent/plugin/opensearch_error_handler.rb

@@ -49,8 +49,12 @@ class Fluent::Plugin::OpenSearchErrorHandler
     unrecoverable_error_types.include?(type)
   end
 
+  def unrecoverable_record_types
+    @plugin.unrecoverable_record_types
+  end
+
   def unrecoverable_record_error?(type)
-    ['json_parse_exception'].include?(type)
+    unrecoverable_record_types.include?(type)
   end
 
   def log_os_400_reason(&block)
@@ -61,6 +65,13 @@ class Fluent::Plugin::OpenSearchErrorHandler
     end
   end
 
+  def emit_error_label_event(&block)
+    # If `emit_error_label_event` is specified as false, error event emittions are not occurred.
+    if @plugin.emit_error_label_event
+      block.call
+    end
+  end
+
   def handle_error(response, tag, chunk, bulk_message_count, extracted_values)
     items = response['items']
     if items.nil? || !items.is_a?(Array)
@@ -127,12 +138,16 @@ class Fluent::Plugin::OpenSearchErrorHandler
             reason += " [reason]: \'#{item[write_operation]['error']['reason']}\'"
           end
         end
-        @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("400 - Rejected by OpenSearch#{reason}"))
+        emit_error_label_event do
+          @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("400 - Rejected by OpenSearch#{reason}"))
+        end
       else
         if item[write_operation]['error'].is_a?(String)
           reason = item[write_operation]['error']
           stats[:errors_block_resp] += 1
-          @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("#{status} - #{reason}"))
+          emit_error_label_event do
+            @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("#{status} - #{reason}"))
+          end
           next
         elsif item[write_operation].has_key?('error') && item[write_operation]['error'].has_key?('type')
           type = item[write_operation]['error']['type']
@@ -141,7 +156,9 @@ class Fluent::Plugin::OpenSearchErrorHandler
             raise OpenSearchRequestAbortError, "Rejected OpenSearch due to #{type}"
           end
           if unrecoverable_record_error?(type)
-            @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("#{status} - #{type}: #{reason}"))
+            emit_error_label_event do
+              @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("#{status} - #{type}: #{reason}"))
+            end
             next
           else
             retry_stream.add(time, rawrecord) unless unrecoverable_record_error?(type)
@@ -150,7 +167,9 @@ class Fluent::Plugin::OpenSearchErrorHandler
           # When we don't have a type field, something changed in the API
           # expected return values.
           stats[:errors_bad_resp] += 1
-          @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("#{status} - No error type provided in the response"))
+          emit_error_label_event do
+            @plugin.router.emit_error_event(tag, time, rawrecord, OpenSearchError.new("#{status} - No error type provided in the response"))
+          end
           next
         end
         stats[type] += 1

data/lib/fluent/plugin/out_opensearch.rb

@@ -110,6 +110,7 @@ module Fluent::Plugin
     config_param :logstash_prefix_separator, :string, :default => '-'
     config_param :logstash_dateformat, :string, :default => "%Y.%m.%d"
     config_param :utc_index, :bool, :default => true
+    config_param :suppress_type_name, :bool, :default => false
     config_param :index_name, :string, :default => "fluentd"
     config_param :id_key, :string, :default => nil
     config_param :write_operation, :string, :default => "index"
@@ -160,6 +161,8 @@ module Fluent::Plugin
     config_param :validate_client_version, :bool, :default => false
     config_param :prefer_oj_serializer, :bool, :default => false
     config_param :unrecoverable_error_types, :array, :default => ["out_of_memory_error", "rejected_execution_exception"]
+    config_param :unrecoverable_record_types, :array, :default => ["json_parse_exception"]
+    config_param :emit_error_label_event, :bool, :default => true
     config_param :verify_os_version_at_startup, :bool, :default => true
     config_param :default_opensearch_version, :integer, :default => DEFAULT_OPENSEARCH_VERSION
     config_param :log_os_400_reason, :bool, :default => false
@@ -219,8 +222,8 @@ module Fluent::Plugin
         if conf[:assume_role_arn].nil?
           aws_container_credentials_relative_uri = conf[:ecs_container_credentials_relative_uri] || ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
           if aws_container_credentials_relative_uri.nil?
-            credentials = Aws::SharedCredentials.new({retries: 2}).credentials
-            credentials ||= Aws::InstanceProfileCredentials.new.credentials
+            credentials = Aws::SharedCredentials.new({retries: 2}).credentials rescue nil
+            credentials ||= Aws::InstanceProfileCredentials.new.credentials rescue nil
             credentials ||= Aws::ECSCredentials.new.credentials
           else
             credentials = Aws::ECSCredentials.new({
@@ -462,6 +465,13 @@ module Fluent::Plugin
       placeholder_validities.include?(true)
     end
 
+    def emit_error_label_event(&block)
+      # If `emit_error_label_event` is specified as false, error event emittions are not occurred.
+      if @emit_error_label_event
+        block.call
+      end
+    end
+
     def compression
       !(@compression_level == :no_compression)
     end
@@ -578,7 +588,9 @@ module Fluent::Plugin
     def parse_time(value, event_time, tag)
       @time_parser.call(value)
     rescue => e
-      router.emit_error_event(@time_parse_error_tag, Fluent::Engine.now, {'tag' => tag, 'time' => event_time, 'format' => @time_key_format, 'value' => value}, e)
+      emit_error_label_event do
+        router.emit_error_event(@time_parse_error_tag, Fluent::Engine.now, {'tag' => tag, 'time' => event_time, 'format' => @time_key_format, 'value' => value}, e)
+      end
       return Time.at(event_time).to_datetime
     end
 
@@ -870,7 +882,9 @@ module Fluent::Plugin
             end
           end
         rescue => e
-          router.emit_error_event(tag, time, record, e)
+          emit_error_label_event do
+            router.emit_error_event(tag, time, record, e)
+          end
         end
       end
 
@@ -978,8 +992,12 @@ module Fluent::Plugin
         end
       end
 
-      # OpenSearch only supports "_doc".
-      target_type = DEFAULT_TYPE_NAME
+      if @suppress_type_name
+        target_type = nil
+      else
+        # OpenSearch only supports "_doc".
+        target_type = DEFAULT_TYPE_NAME
+      end
 
       meta.clear
       meta["_index".freeze] = target_index

data/lib/fluent/plugin/out_opensearch_data_stream.rb

@@ -29,12 +29,15 @@ module Fluent::Plugin
         @data_stream_names = []
       end
 
-      @client = client
       unless @use_placeholder
         begin
           @data_stream_names = [@data_stream_name]
-          create_index_template(@data_stream_name, @data_stream_template_name, @host)
-          create_data_stream(@data_stream_name)
+          retry_operate(@max_retry_putting_template,
+                        @fail_on_putting_template_retry_exceed,
+                        @catch_transport_exception_on_retry) do
+            create_index_template(@data_stream_name, @data_stream_template_name)
+            create_data_stream(@data_stream_name)
+          end
         rescue => e
           raise Fluent::ConfigError, "Failed to create data stream: <#{@data_stream_name}> #{e.message}"
         end
@@ -43,7 +46,7 @@ module Fluent::Plugin
 
     def validate_data_stream_parameters
       {"data_stream_name" => @data_stream_name,
-       "data_stream_template_name"=> @data_stream_template_name}.each do |parameter, value|
+       "data_stream_template_name" => @data_stream_template_name}.each do |parameter, value|
         unless valid_data_stream_parameters?(value)
           unless start_with_valid_characters?(value)
             if not_dots?(value)
@@ -65,30 +68,36 @@ module Fluent::Plugin
       end
     end
 
-    def create_index_template(datastream_name, template_name, host)
-      return if data_stream_exist?(datastream_name) or template_exists?(template_name, host)
-      body = {
-        "index_patterns" => ["#{datastream_name}*"],
-        "data_stream" => {},
-      }
-      params = {
-        name: template_name,
-        body: body
-      }
-      retry_operate(@max_retry_putting_template,
-                    @fail_on_putting_template_retry_exceed,
-                    @catch_transport_exception_on_retry) do
-        @client.indices.put_index_template(params)
+    def create_index_template(datastream_name, template_name, host = nil)
+      # Create index template from file
+      if @template_file
+        template_installation_actual(template_name, @customize_template, @application_name, datastream_name, host)
+      else # Create default index template
+        return if data_stream_exist?(datastream_name, host) or template_exists?(template_name, host)
+        body = {
+          "index_patterns" => ["#{datastream_name}*"],
+          "data_stream" => {},
+        }
+
+        params = {
+          name: template_name,
+          body: body
+        }
+        retry_operate(@max_retry_putting_template,
+                      @fail_on_putting_template_retry_exceed,
+                      @catch_transport_exception_on_retry) do
+          client(host).indices.put_index_template(params)
+        end
       end
     end
 
-    def data_stream_exist?(datastream_name)
+    def data_stream_exist?(datastream_name, host = nil)
       params = {
         name: datastream_name
       }
       begin
         # TODO: Use X-Pack equivalent performing DataStream operation method on the following line
-        response = @client.perform_request('GET', "/_data_stream/#{datastream_name}", {}, params)
+        response = client(host).perform_request('GET', "/_data_stream/#{datastream_name}", {}, params)
         return (not response.is_a?(OpenSearch::Transport::Transport::Errors::NotFound))
       rescue OpenSearch::Transport::Transport::Errors::NotFound => e
         log.info "Specified data stream does not exist. Will be created: <#{e}>"
@@ -96,8 +105,8 @@ module Fluent::Plugin
       end
     end
 
-    def create_data_stream(datastream_name)
-      return if data_stream_exist?(datastream_name)
+    def create_data_stream(datastream_name, host = nil)
+      return if data_stream_exist?(datastream_name, host)
       params = {
         name: datastream_name
       }
@@ -105,7 +114,7 @@ module Fluent::Plugin
                     @fail_on_putting_template_retry_exceed,
                     @catch_transport_exception_on_retry) do
         # TODO: Use X-Pack equivalent performing DataStream operation method on the following line
-        @client.perform_request('PUT', "/_data_stream/#{datastream_name}", {}, params)
+        client(host).perform_request('PUT', "/_data_stream/#{datastream_name}", {}, params)
       end
     end
 
@@ -155,14 +164,19 @@ module Fluent::Plugin
     def write(chunk)
       data_stream_name = @data_stream_name
       data_stream_template_name = @data_stream_template_name
-      host = @host
+      host = nil
       if @use_placeholder
+        host = if @hosts
+                 extract_placeholders(@hosts, chunk)
+               else
+                 extract_placeholders(@host, chunk)
+               end
         data_stream_name = extract_placeholders(@data_stream_name, chunk)
         data_stream_template_name = extract_placeholders(@data_stream_template_name, chunk)
         unless @data_stream_names.include?(data_stream_name)
           begin
             create_index_template(data_stream_name, data_stream_template_name, host)
-            create_data_stream(data_stream_name)
+            create_data_stream(data_stream_name, host)
             @data_stream_names << data_stream_name
           rescue => e
             raise Fluent::ConfigError, "Failed to create data stream: <#{data_stream_name}> #{e.message}"
@@ -177,12 +191,22 @@ module Fluent::Plugin
       tag = chunk.metadata.tag
       chunk.msgpack_each do |time, record|
         next unless record.is_a? Hash
-
         begin
-          record.merge!({"@timestamp" => Time.at(time).iso8601(@time_precision)})
+          if record.has_key?(TIMESTAMP_FIELD)
+            rts = record[TIMESTAMP_FIELD]
+            dt = parse_time(rts, time, tag)
+          elsif record.has_key?(@time_key)
+            rts = record[@time_key]
+            dt = parse_time(rts, time, tag)
+          else
+            dt = Time.at(time).to_datetime
+          end
+          record.merge!({"@timestamp" => dt.iso8601(@time_precision)})
           bulk_message = append_record_to_messages(CREATE_OP, {}, headers, record, bulk_message)
         rescue => e
-          router.emit_error_event(tag, time, record, e)
+          emit_error_label_event do
+            router.emit_error_event(tag, time, record, e)
+          end
         end
       end
 
@@ -191,7 +215,7 @@ module Fluent::Plugin
         body: bulk_message
       }
       begin
-        response = @client.bulk(params)
+        response = client(host).bulk(params)
         if response['errors']
           log.error "Could not bulk insert to Data Stream: #{data_stream_name} #{response}"
         end

data/test/plugin/datastream_template.json

@@ -0,0 +1,4 @@
+{
+  "index_patterns": ["foo*"],
+  "data_stream": {}
+}

data/test/plugin/test_opensearch_error_handler.rb

@@ -35,14 +35,18 @@ class TestOpenSearchErrorHandler < Test::Unit::TestCase
     attr_reader :log
     attr_reader :error_events
     attr_accessor :unrecoverable_error_types
+    attr_accessor :unrecoverable_record_types
     attr_accessor :log_os_400_reason
     attr_accessor :write_operation
+    attr_accessor :emit_error_label_event
     def initialize(log, log_os_400_reason = false)
       @log = log
       @write_operation = 'index'
       @error_events = []
       @unrecoverable_error_types = ["out_of_memory_error", "rejected_execution_exception"]
+      @unrecoverable_record_types = ["json_parse_exception"]
       @log_os_400_reason = log_os_400_reason
+      @emit_error_label_event = true
     end
 
     def router
@@ -135,6 +139,44 @@ class TestOpenSearchErrorHandler < Test::Unit::TestCase
     end
   end
 
+  class TEST400ResponseReasonWithoutErrorLog < self
+    def setup
+      Fluent::Test.setup
+      @log_device = Fluent::Test::DummyLogDevice.new
+      dl_opts = {:log_level => ServerEngine::DaemonLogger::DEBUG}
+      logger = ServerEngine::DaemonLogger.new(@log_device, dl_opts)
+      @log = Fluent::Log.new(logger)
+      @plugin = TestPlugin.new(@log)
+      @handler = Fluent::Plugin::OpenSearchErrorHandler.new(@plugin)
+      @plugin.emit_error_label_event = false
+    end
+
+    def test_400_responses_reason_log
+      records = [{time: 123, record: {"foo" => "bar", '_id' => 'abc'}}]
+      response = parse_response(%({
+      "took" : 0,
+      "errors" : true,
+      "items" : [
+        {
+          "create" : {
+            "_index" : "foo",
+            "status" : 400,
+            "error" : {
+              "type"  : "mapper_parsing_exception",
+              "reason" : "failed to parse"
+            }
+          }
+        }
+      ]
+     }))
+      chunk = MockChunk.new(records)
+      dummy_extracted_values = []
+      @handler.handle_error(response, 'atag', chunk, records.length, dummy_extracted_values)
+      assert_equal(0, @plugin.error_events.size)
+      assert_true(@plugin.error_events.empty?)
+    end
+  end
+
   class TEST400ResponseReasonNoDebug < self
     def setup
       Fluent::Test.setup
@@ -177,6 +219,45 @@ class TestOpenSearchErrorHandler < Test::Unit::TestCase
     end
   end
 
+  class TEST400ResponseReasonNoDebugAndNoErrorLog < self
+    def setup
+      Fluent::Test.setup
+      @log_device = Fluent::Test::DummyLogDevice.new
+      dl_opts = {:log_level => ServerEngine::DaemonLogger::INFO}
+      logger = ServerEngine::DaemonLogger.new(@log_device, dl_opts)
+      @log = Fluent::Log.new(logger)
+      @plugin = TestPlugin.new(@log)
+      @handler = Fluent::Plugin::OpenSearchErrorHandler.new(@plugin)
+      @plugin.log_os_400_reason = true
+      @plugin.emit_error_label_event = false
+    end
+
+    def test_400_responses_reason_log
+      records = [{time: 123, record: {"foo" => "bar", '_id' => 'abc'}}]
+      response = parse_response(%({
+      "took" : 0,
+      "errors" : true,
+      "items" : [
+        {
+          "create" : {
+            "_index" : "foo",
+            "status" : 400,
+            "error" : {
+              "type"  : "mapper_parsing_exception",
+              "reason" : "failed to parse"
+            }
+          }
+        }
+      ]
+     }))
+      chunk = MockChunk.new(records)
+      dummy_extracted_values = []
+      @handler.handle_error(response, 'atag', chunk, records.length, dummy_extracted_values)
+      assert_equal(0, @plugin.error_events.size)
+      assert_true(@plugin.error_events.empty?)
+    end
+  end
+
   def test_nil_items_responses
     records = [{time: 123, record: {"foo" => "bar", '_id' => 'abc'}}]
     response = parse_response(%({

data/test/plugin/test_out_opensearch.rb

@@ -2320,23 +2320,11 @@ class OpenSearchOutputTest < Test::Unit::TestCase
   end
 
   data(
-    "OpenSearch default"=> {"os_version" => 1, "_type" => "_doc"},
+    "OpenSearch default"=> {"os_version" => 1, "_type" => "_doc", "suppress_type" => false},
+    "Suppressed type" => {"os_version" => 1, "_type" => nil, "suppress_type" => true},
   )
   def test_writes_to_speficied_type(data)
-    driver('', data["os_version"]).configure("")
-    stub_opensearch
-    stub_opensearch_info
-    driver.run(default_tag: 'test') do
-      driver.feed(sample_record)
-    end
-    assert_equal(data['_type'], index_cmds.first['index']['_type'])
-  end
-
-  data(
-    "OpenSearch 1"=> {"os_version" => 1, "_type" => "_doc"},
-  )
-  def test_writes_to_speficied_type_with_placeholders(data)
-    driver('', data["os_version"]).configure("")
+    driver('', data["os_version"]).configure("suppress_type_name #{data['suppress_type']}")
     stub_opensearch
     stub_opensearch_info
     driver.run(default_tag: 'test') do

data/test/plugin/test_out_opensearch_data_stream.rb

@@ -9,7 +9,7 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
   include FlexMock::TestCase
   include Fluent::Test::Helpers
 
-  attr_accessor :bulk_records
+  attr_accessor :bulk_records, :index_cmds
 
   OPENSEARCH_DATA_STREAM_TYPE = "opensearch_data_stream"
 
@@ -61,58 +61,77 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
   DUPLICATED_DATA_STREAM_EXCEPTION = {"error": {}, "status": 400}
   NONEXISTENT_DATA_STREAM_EXCEPTION = {"error": {}, "status": 404}
 
-  def stub_index_template(name="foo_tpl")
-    stub_request(:put, "http://localhost:9200/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+  def stub_index_template(name="foo_tpl", url="http://localhost:9200")
+    stub_request(:put, "#{url}/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
   end
 
-  def stub_data_stream(name="foo")
-    stub_request(:put, "http://localhost:9200/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+  def stub_data_stream(name="foo", url="localhost:9200")
+    stub_request(:put, "#{url}/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
   end
 
-  def stub_existent_data_stream?(name="foo")
-    stub_request(:get, "http://localhost:9200/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+  def stub_existent_data_stream?(name="foo", url="localhost:9200")
+    stub_request(:get, "#{url}/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
   end
 
-  def stub_existent_template?(name="foo_tpl")
-    stub_request(:get, "http://localhost:9200/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+  def stub_existent_template?(name="foo_tpl", url="localhost:9200")
+    stub_request(:get, "#{url}/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
   end
 
-  def stub_nonexistent_data_stream?(name="foo")
-    stub_request(:get, "http://localhost:9200/_data_stream/#{name}").to_return(:status => [404, OpenSearch::Transport::Transport::Errors::NotFound])
+  def stub_nonexistent_data_stream?(name="foo", url="localhost:9200")
+    stub_request(:get, "#{url}/_data_stream/#{name}").to_return(:status => [404, OpenSearch::Transport::Transport::Errors::NotFound])
   end
 
-  def stub_nonexistent_template?(name="foo_tpl")
-    stub_request(:get, "http://localhost:9200/_index_template/#{name}").to_return(:status => [404, OpenSearch::Transport::Transport::Errors::NotFound])
+  def stub_nonexistent_template?(name="foo_tpl", url="http://localhost:9200")
+    stub_request(:get, "#{url}/_index_template/#{name}").to_return(:status => [404, OpenSearch::Transport::Transport::Errors::NotFound])
   end
 
-  def stub_bulk_feed(datastream_name="foo", template_name="foo_tpl")
-    stub_request(:post, "http://localhost:9200/#{datastream_name}/_bulk").with do |req|
+  def stub_nonexistent_template_retry?(name="foo_tpl", url="http://localhost:9200")
+    stub_request(:get, "#{url}/_index_template/#{name}").
+      to_return({ status: 500, body: 'Internal Server Error' }, { status: 404, body: '{}' })
+  end
+
+  def stub_bulk_feed(datastream_name="foo", template_name="foo_tpl", url="http://localhost:9200")
+    stub_request(:post, "#{url}/#{datastream_name}/_bulk").with do |req|
       # bulk data must be pair of OP and records
       # {"create": {}}\nhttp://localhost:9200/_data_stream/foo_bar
       # {"@timestamp": ...}
       @bulk_records += req.body.split("\n").size / 2
+      @index_cmds = req.body.split("\n").map {|r| JSON.parse(r) }
     end
-    stub_request(:post, "http://localhost:9200/#{template_name}/_bulk").with do |req|
+    stub_request(:post, "http://#{url}#{template_name}/_bulk").with do |req|
       # bulk data must be pair of OP and records
       # {"create": {}}\nhttp://localhost:9200/_data_stream/foo_bar
       # {"@timestamp": ...}
       @bulk_records += req.body.split("\n").size / 2
+      @index_cmds = req.body.split("\n").map {|r| JSON.parse(r) }
     end
   end
 
-  def stub_elastic_info(url="http://localhost:9200/", version="1.2.2")
+  def stub_opensearch_info(url="http://localhost:9200/", version="1.2.2", headers={})
     body ="{\"version\":{\"number\":\"#{version}\", \"distribution\":\"opensearch\"},\"tagline\":\"The OpenSearch Project: https://opensearch.org/\"}"
-    stub_request(:get, url).to_return({:status => 200, :body => body, :headers => { 'Content-Type' => 'json' } })
+    stub_request(:get, url).to_return({:status => 200, :body => body, :headers => { 'Content-Type' => 'json' }.merge(headers) })
   end
 
   def stub_default(datastream_name="foo", template_name="foo_tpl", host="http://localhost:9200")
-    stub_elastic_info(host)
+    stub_opensearch_info(host)
     stub_nonexistent_template?(template_name)
     stub_index_template(template_name)
     stub_nonexistent_data_stream?(datastream_name)
     stub_data_stream(datastream_name)
   end
 
+  def stub_opensearch_with_store_index_command_counts(url="http://localhost:9200/_bulk")
+    if @index_command_counts == nil
+      @index_command_counts = {}
+      @index_command_counts.default = 0
+    end
+
+    stub_request(:post, url).with do |req|
+      index_cmds = req.body.split("\n").map {|r| JSON.parse(r) }
+      @index_command_counts[url] += index_cmds.size
+    end
+  end
+
   # ref. https://opensearch.org/docs/latest/opensearch/data-streams/
   class DataStreamNameTest < self
 
@@ -315,11 +334,71 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
     assert_equal "foo", driver(conf).instance.data_stream_name
   end
 
+  def test_hosts_list_configure
+    config = %{
+      hosts            https://john:password@host1:443/elastic/,http://host2
+      path             /default_path
+      user             default_user
+      password         default_password
+      data_stream_name default
+    }
+    stub_opensearch_info("https://host1:443/elastic//", "1.2.2",
+                         {'Authorization'=>"Basic #{Base64.encode64('john:password').split.first}"})
+    stub_opensearch_info("http://host2/default_path/_data_stream/default", "1.2.2",
+                         {'Authorization'=>"Basic #{Base64.encode64('john:password').split.first}"})
+    stub_existent_data_stream?("default", "https://host1/elastic/")
+    instance = driver(config).instance
+
+    assert_equal 2, instance.get_connection_options[:hosts].length
+    host1, host2 = instance.get_connection_options[:hosts]
+
+    assert_equal 'host1', host1[:host]
+    assert_equal 443, host1[:port]
+    assert_equal 'https', host1[:scheme]
+    assert_equal 'john', host1[:user]
+    assert_equal 'password', host1[:password]
+    assert_equal '/elastic/', host1[:path]
+
+    assert_equal 'host2', host2[:host]
+    assert_equal 'http', host2[:scheme]
+    assert_equal 'default_user', host2[:user]
+    assert_equal 'default_password', host2[:password]
+    assert_equal '/default_path', host2[:path]
+  end
+
+  def test_datastream_configure_retry
+    stub_opensearch_info
+    stub_nonexistent_template_retry?
+    stub_index_template
+    stub_nonexistent_data_stream?
+    stub_data_stream
+    conf = config_element(
+      'ROOT', '', {
+      '@type' => OPENSEARCH_DATA_STREAM_TYPE,
+      'data_stream_name' => 'foo',
+      'data_stream_template_name' => "foo_tpl"
+    })
+    assert_equal "foo", driver(conf).instance.data_stream_name
+  end
+
+  def test_template_file
+    stub_default
+    cwd = File.dirname(__FILE__)
+    conf = config_element(
+      'ROOT', '', {
+      '@type' => OPENSEARCH_DATA_STREAM_TYPE,
+      'data_stream_name' => 'foo',
+      'data_stream_template_name' => "foo_tpl",
+      'template_file' => File.join(cwd, 'datastream_template.json')
+    })
+    assert_equal "foo", driver(conf).instance.data_stream_name
+  end
+
   def test_existent_data_stream
     stub_index_template
     stub_existent_data_stream?
     stub_data_stream
-    stub_elastic_info
+    stub_opensearch_info
     conf = config_element(
       'ROOT', '', {
         '@type' => OPENSEARCH_DATA_STREAM_TYPE,
@@ -333,7 +412,7 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
     stub_index_template
     stub_existent_data_stream?
     stub_data_stream
-    stub_elastic_info
+    stub_opensearch_info
     conf = config_element(
       'ROOT', '', {
         '@type' => OPENSEARCH_DATA_STREAM_TYPE,
@@ -432,13 +511,56 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
         '@type' => OPENSEARCH_DATA_STREAM_TYPE,
         'data_stream_name' => 'foo',
         'data_stream_template_name' => 'foo_tpl'
-      })
+    })
     driver(conf).run(default_tag: 'test') do
       driver.feed(sample_record)
     end
     assert_equal 1, @bulk_records
   end
 
+  def test_placeholder_writes_to_multi_hosts
+    stub_default("foo_bar", "foo_tpl_bar")
+    hosts = [['192.168.33.50', 9201], ['192.168.33.51', 9201], ['192.168.33.52', 9201]]
+    hosts_string = hosts.map {|x| "#{x[0]}:#{x[1]}"}.compact.join(',')
+    hosts.each do |host_info|
+      host, port = host_info
+      stub_opensearch_with_store_index_command_counts("http://#{host}:#{port}/foo_bar/_bulk")
+      stub_opensearch_info("http://#{host}:#{port}/")
+      stub_request(:get, "http://#{host}:#{port}/_data_stream/foo_bar").
+        to_return(status: 200, body: "", headers: {})
+    end
+
+    conf = config_element(
+      'ROOT', '', {
+      '@type' => OPENSEARCH_DATA_STREAM_TYPE,
+      'data_stream_name' => 'foo_${key1}',
+      'data_stream_template_name' => 'foo_tpl_${key1}',
+      'hosts' => "#{hosts_string}"
+    }, [config_element('buffer', 'tag,key1', {
+      'timekey' => '1d'
+    }, [])])
+    driver(conf).run(default_tag: 'test') do
+      hashes = {
+        'age' => rand(100),
+        'key1' => 'bar'
+      }
+      1000.times do
+        driver.feed(sample_record.merge(hashes))
+      end
+    end
+
+    # @note: we cannot make multi chunks with options (flush_interval, buffer_chunk_limit)
+    # it's Fluentd test driver's constraint
+    # so @index_command_counts.size is always 1
+    assert(@index_command_counts.size > 0, "not working with hosts options")
+
+    total = 0
+    @index_command_counts.each do |_, count|
+      total += count
+    end
+    assert_equal(2000, total)
+  end
+
   def test_template_retry_install_fails
     cwd = File.dirname(__FILE__)
     template_file = File.join(cwd, 'test_index_template.json')
@@ -463,7 +585,7 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
       connection_resets += 1
       raise Faraday::ConnectionFailed, "Test message"
     end
-    stub_elastic_info("https://logs.google.com:778/")
+    stub_opensearch_info("https://logs.google.com:778/")
 
     assert_raise(Fluent::Plugin::OpenSearchError::RetryableOperationExhaustedFailure) do
       driver(config)
@@ -471,4 +593,72 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
 
     assert_equal(4, connection_resets)
   end
+
+  def test_uses_custom_time_key
+    stub_default
+    stub_bulk_feed
+    conf = config_element(
+      'ROOT', '', {
+      '@type' => OPENSEARCH_DATA_STREAM_TYPE,
+      'data_stream_name' => 'foo',
+      'data_stream_template_name' => 'foo_tpl',
+      'time_key' => 'vtm'
+    })
+
+    ts = DateTime.new(2021,2,3).iso8601(9)
+    record = {
+      'vtm' => ts,
+      'message' => 'Sample Record'
+    }
+
+    driver(conf).run(default_tag: 'test') do
+      driver.feed(record)
+    end
+    assert(index_cmds[1].has_key? '@timestamp')
+    assert_equal(ts, index_cmds[1]['@timestamp'])
+  end
+
+  def test_uses_custom_time_key_with_format
+    stub_default
+    stub_bulk_feed
+    conf = config_element(
+      'ROOT', '', {
+      '@type' => OPENSEARCH_DATA_STREAM_TYPE,
+      'data_stream_name' => 'foo',
+      'data_stream_template_name' => 'foo_tpl',
+      'time_key' => 'vtm',
+      'time_key_format' => '%Y-%m-%d %H:%M:%S.%N%z'
+    })
+    ts = "2021-02-03 13:14:01.673+02:00"
+    record = {
+      'vtm' => ts,
+      'message' => 'Sample Record'
+    }
+    driver(conf).run(default_tag: 'test') do
+      driver.feed(record)
+    end
+    assert(index_cmds[1].has_key? '@timestamp')
+    assert_equal(DateTime.parse(ts).iso8601(9), index_cmds[1]['@timestamp'])
+  end
+
+  def test_record_no_timestamp
+    stub_default
+    stub_bulk_feed
+    stub_default
+    stub_bulk_feed
+    conf = config_element(
+      'ROOT', '', {
+      '@type' => OPENSEARCH_DATA_STREAM_TYPE,
+      'data_stream_name' => 'foo',
+      'data_stream_template_name' => 'foo_tpl'
+    })
+    record = {
+      'message' => 'Sample Record'
+    }
+    driver(conf).run(default_tag: 'test') do
+      driver.feed(record)
+    end
+    assert(index_cmds[1].has_key? '@timestamp')
+  end
+
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-opensearch
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.3
 platform: ruby
 authors:
 - Hiroshi Hatake
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-01-13 00:00:00.000000000 Z
+date: 2022-03-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -208,6 +208,7 @@ files:
 - lib/fluent/plugin/out_opensearch.rb
 - lib/fluent/plugin/out_opensearch_data_stream.rb
 - test/helper.rb
+- test/plugin/datastream_template.json
 - test/plugin/test_alias_template.json
 - test/plugin/test_filter_opensearch_genid.rb
 - test/plugin/test_in_opensearch.rb
@@ -226,7 +227,7 @@ licenses:
 - Apache-2.0
 metadata:
   changelog_uri: https://github.com/fluent/fluent-plugin-opensearch/blob/master/History.md
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -241,12 +242,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.22
-signing_key: 
+rubygems_version: 3.2.30
+signing_key:
 specification_version: 4
 summary: Opensearch output plugin for Fluent event collector
 test_files:
 - test/helper.rb
+- test/plugin/datastream_template.json
 - test/plugin/test_alias_template.json
 - test/plugin/test_filter_opensearch_genid.rb
 - test/plugin/test_in_opensearch.rb