logstash-output-opensearch 1.0.0-java → 1.3.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1442c3f5b382cf09b452e6e4464dc04de5add0c1ef5a2d0ffd575f765364a166
-  data.tar.gz: 1a6055265e1be76ab8f4b54ecea6a3136e0b375f4578463609f2ccbab162c67a
+  metadata.gz: bbe40e1f039f2d91a4adfb8b8fb9a28fd877156bf270060fa1d294986a769983
+  data.tar.gz: 9c2bd44c3e32cbf31d3ffac173a5e9b0ff2e4bcb9c4bd27267ac70a3a88ee6da
 SHA512:
-  metadata.gz: 4007705f78e0d2e4f4872f3a9704d1a936a9e086dbebda06877dc0db5ebcc522dbf4550d30acf60317baea74813e813d26ae36355130d42d373e02e8d423971d
-  data.tar.gz: ecd5d013db35fd54a3faa7c7dbda5c4e36c588798245909f49f25aedb4a0639d0bd479608510d2e3d382dcd8c731f201449304771821141cf9e543ade9a16cf6
+  metadata.gz: ed114d31a8dd15ebe57cba0d87c278efe3be2a47743aa2cdfe99bf861e0934edb2772d09aa59eda0d9428b119dde77a16c82371ce19ca1cbfb5c6ce736692690
+  data.tar.gz: df6263f5109d5834bc19fa9804c082c95dc1867eed7926be99b7682c4c1cfb3abffa0ff23b16f123f55f196e353527d31d8d5a61f3011bbf56805f43fa890fdc

data/COMPATIBILITY.md

@@ -0,0 +1,27 @@
+## Compatibility
+
+### Matrix for Logstash OSS
+
+|  logstash-output-opensearch | Logstash OSS|
+| ------------- | ------------- |
+| 1.0.0+  | 7.13.2  |
+
+### Matrix for OpenSearch
+
+|  logstash-output-opensearch | OpenSearch  |
+| ------------- | ------------- |
+| 1.0.0+  | 1.0.0  |  
+
+
+### Matrix for ODFE
+
+|  logstash-output-opensearch |  ODFE |
+| ------------- | ------------- |
+| 1.0.0+   |   1.x - 1.13.2 |
+
+
+### Matrix for Elasticsearch OSS
+
+|  logstash-output-opensearch |  Elasticsearch OSS|
+| ------------- | ------------- |
+| 1.0.0+   |  7.x - 7.10.2      |

data/CONTRIBUTING.md

@@ -6,6 +6,7 @@
     - [Documentation Changes](#documentation-changes)
     - [Contributing Code](#contributing-code)
 - [Developer Certificate of Origin](#developer-certificate-of-origin)
+- [License Headers](#license-headers)
 - [Review Process](#review-process)
 
 ## Contributing to logstash-output-opensearch
@@ -35,17 +36,13 @@ If you've upgraded to the latest version and you can't find it in our open issue
 
 If you've thought of a way that logstash-output-opensearch could be better, we want to hear about it.  We track feature requests using GitHub, so please feel free to open an issue which describes the feature you would like to see, why you need it, and how it should work.
 
-### Documentation Changes
-
-//TODO
-
 ### Contributing Code
 
 As with other types of contributions, the first step is to [**open an issue on GitHub**](https://github.com/opensearch-project/logstash-output-opensearch/issues/new/choose).  Opening an issue before you make changes makes sure that someone else isn't already working on that particular problem.  It also lets us all work together to find the right approach before you spend a bunch of time on a PR.  So again, when in doubt, open an issue.
 
 ## Developer Certificate of Origin
 
-logstash-output-opensearch is an open source product released under the Apache 2.0 license (see either [the Apache site](https://www.apache.org/licenses/LICENSE-2.0) or the [LICENSE.txt file](./LICENSE.txt)).  The Apache 2.0 license allows you to freely use, modify, distribute, and sell your own products that include Apache 2.0 licensed software.
+logstash-output-opensearch is an open source product released under the Apache 2.0 license (see either [the Apache site](https://www.apache.org/licenses/LICENSE-2.0) or the [LICENSE file](./LICENSE)).  The Apache 2.0 license allows you to freely use, modify, distribute, and sell your own products that include Apache 2.0 licensed software.
 
 We respect intellectual property rights of others and we want to make sure all incoming contributions are correctly attributed and licensed. A Developer Certificate of Origin (DCO) is a lightweight mechanism to do that.
 
@@ -88,6 +85,31 @@ Signed-off-by: Jane Smith <jane.smith@email.com>
 ```
 You may type this line on your own when writing your commit messages.  However, if your user.name and user.email are set in your git configs, you can use `-s` or `– – signoff` to add the `Signed-off-by` line to the end of the commit message.
 
+## License Headers
+
+New files in your code contributions should contain the following license header. If you are modifying existing files with license headers, or including new files that already have license headers, do not remove or modify them without guidance.
+
+### Java
+
+```
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+```
+
+### Python
+```
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+```
+
+### Shell
+```
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+```
+
 ## Review Process
 
 We deeply appreciate everyone who takes the time to make a contribution.  We will review all contributions as quickly as possible.  As a reminder, [opening an issue](https://github.com/opensearch-project/logstash-output-opensearch/issues/new/choose) discussing your change before you make it is the best way to smooth the PR process.  This will prevent a rejection because someone else is already working on the problem, or because the solution is incompatible with the architectural direction.

data/DEVELOPER_GUIDE.md

@@ -8,6 +8,7 @@
         - [Run plugin](#run-plugin-in-logstash)
         - [Configuration for Logstash Output OpenSearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
     - [Submitting Changes](#submitting-changes)
+    - [Backports](#backports) 
 
 # Developer Guide
 
@@ -59,7 +60,7 @@ Perform the following from your project root directory eg: `~/workspace/logstash
 
 ```bash
 # Set up Environment variable for docker to pull your test environment
-export LOGSTASH_VERSION=7.13.2 # will use latest version if not specified.
+export LOGSTASH_VERSION=7.13.2 # will use 7.13.2 version if not specified.
 
 # Set up docker ( this will build, install into Logstash )
 scripts/unit-test/docker-setup.sh
@@ -73,8 +74,8 @@ scripts/unit-test/docker-run.sh
 
 ```bash
 # Set up Environment variable for Docker to pull your test environment
-export LOGSTASH_VERSION=7.13.2 # will use latest version if not specified.
-export OPENSEARCH_VERSION=1.0.0-rc1 # will use latest if not specified.
+export LOGSTASH_VERSION=7.13.2 # will use 7.13.2 version if not specified.
+export OPENSEARCH_VERSION=1.0.0 # will use latest if not specified.
 
 # Set up docker ( this will build, install into Logstash )
 scripts/opensearch/docker-setup.sh
@@ -87,8 +88,8 @@ scripts/opensearch/docker-run.sh
 
 ```bash
 # Set up Environment variable for Docker to pull your test environment
-export LOGSTASH_VERSION=7.13.2 # will use latest version if not specified.
-export OPENSEARCH_VERSION=1.0.0-rc1 # will use latest if not specified.
+export LOGSTASH_VERSION=7.13.2 # will use 7.13.2 version if not specified.
+export OPENSEARCH_VERSION=1.0.0 # will use latest if not specified.
 export SECURE_INTEGRATION=true # to run against cluster with security plugin
 # Set up docker ( this will build, install into Logstash )
 scripts/opensearch/docker-setup.sh
@@ -101,7 +102,7 @@ scripts/opensearch/docker-run.sh
 
 ```bash
 # Set up Environment variable for docker to pull your test environment
-export LOGSTASH_VERSION=7.13.2 # will use latest version if not specified.
+export LOGSTASH_VERSION=7.13.2 # will use 7.13.2 version if not specified.
 export OPENDISTRO_VERSION=1.13.2 # will use latest if not specified.
 
 # Set up docker ( this will build, install into Logstash )
@@ -188,7 +189,7 @@ To run the Logstash Output Opensearch plugin, add following configuration in you
 ```
 output {
     opensearch {
-        hosts       => "https://hostname:port"
+        hosts       => ["hostname:port"]
         user        => "admin"
         password    => "admin"
         index       => "logstash-logs-%{+YYYY.MM.dd}"
@@ -206,3 +207,11 @@ Authorization to a secure OpenSearch cluster requires read permission at [index
 ## Submitting Changes
 
 See [CONTRIBUTING](CONTRIBUTING.md).
+
+## Backports
+
+The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the
+original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow
+run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label
+`backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
+merged to main, the workflow will create a backport PR to the `1.x` branch.

data/Gemfile

@@ -17,4 +17,4 @@ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "
 if Dir.exist?(logstash_path) && use_logstash_source
   gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
   gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
-end
+end

data/MAINTAINERS.md

@@ -22,6 +22,9 @@ This document explains who the maintainers are (see below), what they do in this
 | Jack Mazanec             | [jmazanec15](https://github.com/jmazanec15) | Amazon |
 | Vamshi Vijay Nakkirtha   | [vamshin](https://github.com/vamshin)   |   Amazon    |
 | Vijayan Balasubramanian  | [VijayanB](https://github.com/VijayanB) |   Amazon    |
+| Deep Datta               | [deepdatta](https://github.com/deepdatta) | Amazon    |
+| David Venable            | [dlvenable](https://github.com/dlvenable) | Amazon    |
+| Shivani Shukla           | [sshivanii](https://github.com/sshivanii) | Amazon    |
 
 ## Maintainer Responsibilities
 

data/NOTICE

@@ -1,2 +1,9 @@
-logstash-output-opensearch
-Copyright 2021 logstash-output-opensearch Contributors
+OpenSearch (https://opensearch.org/)
+Copyright OpenSearch Contributors
+
+This product includes software developed by
+Elasticsearch (http://www.elastic.co).
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -1,21 +1,25 @@
 [![Build and Test logstash-output-opensearch plugin](https://github.com/opensearch-project/logstash-output-opensearch/actions/workflows/CI.yml/badge.svg)](https://github.com/opensearch-project/logstash-output-opensearch/actions/workflows/CI.yml)
 ![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)
-# Logstash Plugin
+# Logstash Output OpenSearch
 
 - [Welcome!](#welcome)
 - [Project Resources](#project-resources)
+- [Configuration for Logstash Output Opensearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
 - [Code of Conduct](#code-of-conduct)
 - [License](#license)
 - [Copyright](#copyright)
 
 ## Welcome!
 
-**logstash-output-opensearch** is a community-driven, open source fork logstash-output-elasticsearch licensed under the [Apache v2.0 License](LICENSE.txt). For more information, see [opensearch.org](https://opensearch.org/).
+**logstash-output-opensearch** is a community-driven, open source fork logstash-output-elasticsearch licensed under the [Apache v2.0 License](LICENSE). For more information, see [opensearch.org](https://opensearch.org/).
+
+The logstash-output-opensearch plugin helps to ship events from Logstash to OpenSearch cluster.
 
 ## Project Resources
 
 * [Project Website](https://opensearch.org/)
-* [Documentation](https://opensearch.org/)
+* [Documentation](https://opensearch.org/docs/clients/logstash/index/)
+* [Developer Guide](DEVELOPER_GUIDE.md)
 * Need help? Try [Forums](https://discuss.opendistrocommunity.dev/)
 * [Project Principles](https://opensearch.org/#principles)
 * [Contributing to OpenSearch](CONTRIBUTING.md)
@@ -24,14 +28,80 @@
 * [Admin Responsibilities](ADMINS.md)
 * [Security](SECURITY.md)
 
+## Configuration for Logstash Output Opensearch Plugin
+
+To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
+```
+output {
+    opensearch {
+        hosts       => ["hostname:port"]
+        user        => "admin"
+        password    => "admin"
+        index       => "logstash-logs-%{+YYYY.MM.dd}"
+    }
+}
+```
+
+To run the Logstash Output Opensearch plugin using aws_iam authentication, refer to the sample configuration shown below:
+```
+output {        
+   opensearch {     
+          hosts => ["hostname:port"]              
+          auth_type => {    
+              type => 'aws_iam'     
+              aws_access_key_id => 'ACCESS_KEY'     
+              aws_secret_access_key => 'SECRET_KEY'     
+              region => 'us-west-2'         
+          }         
+          index  => "logstash-logs-%{+YYYY.MM.dd}"      
+   }            
+}
+```
+
+In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type.
+
+Example Configuration for basic authentication:
+```
+output {    
+    opensearch {        
+          hosts  => ["hostname:port"]     
+          auth_type => {            
+              type => 'basic'           
+              user => 'admin'           
+              password => 'admin'           
+          }             
+          index => "logstash-logs-%{+YYYY.MM.dd}"       
+   }            
+}  
+```
+
+To ingest data into a `data stream` through logstash, we need to create the data stream and specify the name of data stream and the `op_type` of `create` in the output configuration. The sample configuration is shown below:
+
+```yml
+output {    
+    opensearch {        
+          hosts  => ["https://hostname:port"]     
+          auth_type => {            
+              type => 'basic'           
+              user => 'admin'           
+              password => 'admin'           
+          }
+          index => "my-data-stream"
+          action => "create"
+   }            
+}               
+```
+
+For more details refer to this [documentation](https://opensearch.org/docs/latest/clients/logstash/ship-to-opensearch/#opensearch-output-plugin)
+
 ## Code of Conduct
 
 This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUCT.md). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact [opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments.
 
 ## License
 
-This project is licensed under the [Apache v2.0 License](LICENSE.txt).
+This project is licensed under the [Apache v2.0 License](LICENSE).
 
 ## Copyright
 
-Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+Copyright OpenSearch Contributors. See [NOTICE](NOTICE) for details.

data/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

@@ -7,12 +7,31 @@
 #  Modifications Copyright OpenSearch Contributors. See
 #  GitHub history for details.
 
-require 'manticore'
+require 'aws-sdk-core'
 require 'cgi'
+require 'manticore'
+require 'uri'
 
 module LogStash; module Outputs; class OpenSearch; class HttpClient;
-  DEFAULT_HEADERS = { "Content-Type" => "application/json" }
-  
+  AWS_DEFAULT_PORT = 443
+  AWS_DEFAULT_PROFILE = 'default'
+  AWS_DEFAULT_PROFILE_CREDENTIAL_RETRY = 0
+  AWS_DEFAULT_PROFILE_CREDENTIAL_TIMEOUT = 1
+  AWS_DEFAULT_REGION = 'us-east-1'
+  AWS_IAM_AUTH_TYPE = "aws_iam"
+  AWS_SERVICE = 'es'
+  BASIC_AUTH_TYPE = 'basic'
+  DEFAULT_HEADERS = { "content-type" => "application/json" }
+
+  AWSIAMCredential = Struct.new(
+    :access_key_id,
+    :secret_access_key,
+    :session_token,
+    :profile,
+    :instance_profile_credentials_retries,
+    :instance_profile_credentials_timeout,
+    :region)
+
   class ManticoreAdapter
     attr_reader :manticore, :logger
 
@@ -27,13 +46,65 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
       options[:cookies] = false
 
       @client_params = {:headers => DEFAULT_HEADERS.merge(options[:headers] || {})}
-      
+      @type = get_auth_type(options) || nil
+
+      if @type == AWS_IAM_AUTH_TYPE
+        aws_iam_auth_initialization(options)
+      elsif @type == BASIC_AUTH_TYPE
+        basic_auth_initialization(options)
+      end
+
       if options[:proxy]
         options[:proxy] = manticore_proxy_hash(options[:proxy])
       end
       
       @manticore = ::Manticore::Client.new(options)
     end
+
+    def get_auth_type(options)
+      if options[:auth_type] != nil
+        options[:auth_type]["type"]
+      end
+    end
+
+    def aws_iam_auth_initialization(options)
+      aws_access_key_id =  options[:auth_type]["aws_access_key_id"] || nil
+      aws_secret_access_key = options[:auth_type]["aws_secret_access_key"] || nil
+      session_token = options[:auth_type]["session_token"] || nil
+      profile = options[:auth_type]["profile"] || AWS_DEFAULT_PROFILE
+      instance_cred_retries = options[:auth_type]["instance_profile_credentials_retries"] || AWS_DEFAULT_PROFILE_CREDENTIAL_RETRY
+      instance_cred_timeout = options[:auth_type]["instance_profile_credentials_timeout"] || AWS_DEFAULT_PROFILE_CREDENTIAL_TIMEOUT
+      region = options[:auth_type]["region"] || AWS_DEFAULT_REGION
+      set_aws_region(region)
+
+      credential_config = AWSIAMCredential.new(aws_access_key_id, aws_secret_access_key, session_token, profile, instance_cred_retries, instance_cred_timeout, region)
+      @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
+    end
+
+    def basic_auth_initialization(options)
+      set_user_password(options)
+    end
+
+    def set_aws_region(region)
+      @region = region
+    end
+
+    def get_aws_region()
+      @region
+    end
+
+    def set_user_password(options)
+      @user = options[:auth_type]["user"]
+      @password = options[:auth_type]["password"]
+    end
+
+    def get_user()
+      @user
+    end
+
+    def get_password()
+      @password
+    end
     
     # Transform the proxy option to a hash. Manticore's support for non-hash
     # proxy options is broken. This was fixed in https://github.com/cheald/manticore/commit/34a00cee57a56148629ed0a47c329181e7319af5
@@ -61,6 +132,8 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
       params = (params || {}).merge(@client_params) { |key, oldval, newval|
         (oldval.is_a?(Hash) && newval.is_a?(Hash)) ? oldval.merge(newval) : newval
       }
+
+      params[:headers] = params[:headers].clone
       params[:body] = body if body
 
       if url.user
@@ -71,9 +144,16 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
           :password => CGI.unescape(url.password), 
           :eager => true 
         }
+      elsif @type == BASIC_AUTH_TYPE
+        add_basic_auth_to_params(params)
       end
 
       request_uri = format_url(url, path)
+
+      if @type == AWS_IAM_AUTH_TYPE
+        sign_aws_request(request_uri, path, method, params)
+      end
+
       request_uri_as_string = remove_double_escaping(request_uri.to_s)
       resp = @manticore.send(method.downcase, request_uri_as_string, params)
 
@@ -92,6 +172,23 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
       resp
     end
 
+    def sign_aws_request(request_uri, path, method, params)
+      url = URI::HTTPS.build({:host=>URI(request_uri.to_s).host, :port=>AWS_DEFAULT_PORT.to_s, :path=>path})
+      key = Seahorse::Client::Http::Request.new(options={:endpoint=>url, :http_method => method.to_s.upcase,
+                                                         :headers => params[:headers],:body => params[:body]})
+      aws_signer = Aws::Signers::V4.new(@credentials,  AWS_SERVICE, get_aws_region )
+      signed_key =  aws_signer.sign(key)
+      params[:headers] =  params[:headers].merge(signed_key.headers)
+    end
+
+    def add_basic_auth_to_params(params)
+      params[:auth] = {
+        :user => get_user(),
+        :password => get_password(),
+        :eager => true
+      }
+    end
+
     # Returned urls from this method should be checked for double escaping.
     def format_url(url, path_and_query=nil)
       request_uri = url.clone

data/lib/logstash/outputs/opensearch/http_client/pool.rb

@@ -8,7 +8,6 @@
 #  GitHub history for details.
 
 require "concurrent/atomic/atomic_reference"
-require "logstash/plugin_mixins/opensearch/noop_distribution_checker"
 
 module LogStash; module Outputs; class OpenSearch; class HttpClient;
   class Pool
@@ -41,7 +40,6 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
     end
 
     attr_reader :logger, :adapter, :sniffing, :sniffer_delay, :resurrect_delay, :healthcheck_path, :sniffing_path, :bulk_path
-    attr_reader :distribution_checker
 
     ROOT_URI_PATH = '/'.freeze
 
@@ -80,7 +78,6 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
       @stopping = false
 
       @last_version = Concurrent::AtomicReference.new
-      @distribution_checker = options[:distribution_checker] || LogStash::PluginMixins::OpenSearch::NoopDistributionChecker::INSTANCE
     end
 
     def start
@@ -239,8 +236,7 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
           @state_mutex.synchronize do
             meta[:version] = version
             set_last_version(version, url)
-            alive = @distribution_checker.is_supported?(self, url, @maximum_seen_major_version)
-            meta[:state] = alive ? :alive : :dead
+            meta[:state] = :alive
           end
         rescue HostUnreachableError, BadResponseCodeError => e
           logger.warn("Attempted to resurrect connection to dead OpenSearch instance, but got an error", url: url.sanitized.to_s, exception: e.class, message: e.message)
@@ -415,18 +411,9 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
       end
     end
 
-    def get_version_map(url)
-      request = perform_request_to_url(url, :get, ROOT_URI_PATH)
-      LogStash::Json.load(request.body)['version']
-    end
-
     def get_version(url)
-      get_version_map(url)['number'] # e.g. "7.10.0"
-    end
-
-    def get_distribution(url)
-      version_map = get_version_map(url)
-      version_map.has_key?('distribution') ? version_map['distribution'] : version_map['build_flavor'] # e.g. "opensearch or oss"
+      request = perform_request_to_url(url, :get, ROOT_URI_PATH)
+      LogStash::Json.load(request.body)["version"]["number"] # e.g. "7.10.0"
     end
 
     def last_version

data/lib/logstash/outputs/opensearch/http_client.rb

@@ -15,23 +15,9 @@ require 'zlib'
 require 'stringio'
 
 module LogStash; module Outputs; class OpenSearch;
-  # This is a constant instead of a config option because
-  # there really isn't a good reason to configure it.
-  #
-  # The criteria used are:
-  # 1. We need a number that's less than 100MiB because OpenSearch
-  #    won't accept bulks larger than that.
-  # 2. It must be large enough to amortize the connection constant
-  #    across multiple requests.
-  # 3. It must be small enough that even if multiple threads hit this size
-  #    we won't use a lot of heap.
-  #
-  # We wound up agreeing that a number greater than 10 MiB and less than 100MiB
-  # made sense. We picked one on the lowish side to not use too much heap.
-  TARGET_BULK_BYTES = 20 * 1024 * 1024 # 20MiB
-
   class HttpClient
-    attr_reader :client, :options, :logger, :pool, :action_count, :recv_count
+    attr_reader :client, :options, :logger, :pool, :action_count, :recv_count, :target_bulk_bytes
+
     # This is here in case we use DEFAULT_OPTIONS in the future
     # DEFAULT_OPTIONS = {
     #   :setting => value
@@ -41,13 +27,14 @@ module LogStash; module Outputs; class OpenSearch;
     # The `options` is a hash where the following symbol keys have meaning:
     #
     # * `:hosts` - array of String. Set a list of hosts to use for communication.
-    # * `:port` - number. set the port to use to communicate with OpenSearch
     # * `:user` - String. The user to use for authentication.
     # * `:password` - String. The password to use for authentication.
     # * `:timeout` - Float. A duration value, in seconds, after which a socket
     #    operation or request will be aborted if not yet successfull
+    # * `:auth_type` - hash of String. It contains the type of authentication
+    #     and it's respective credentials
     # * `:client_settings` - a hash; see below for keys.
-    #
+
     # The `client_settings` key is a has that can contain other settings:
     #
     # * `:ssl` - Boolean. Enable or disable SSL/TLS.
@@ -72,6 +59,8 @@ module LogStash; module Outputs; class OpenSearch;
       # mutex to prevent requests and sniffing to access the
       # connection pool at the same time
       @bulk_path = @options[:bulk_path]
+
+      @target_bulk_bytes = @options[:target_bulk_bytes]
     end
 
     def build_url_template
@@ -104,7 +93,6 @@ module LogStash; module Outputs; class OpenSearch;
     def bulk(actions)
       @action_count ||= 0
       @action_count += actions.size
-
       return if actions.empty?
 
       bulk_actions = actions.collect do |action, args, source|
@@ -131,7 +119,7 @@ module LogStash; module Outputs; class OpenSearch;
                     action.map {|line| LogStash::Json.dump(line)}.join("\n") :
                     LogStash::Json.dump(action)
         as_json << "\n"
-        if (stream_writer.pos + as_json.bytesize) > TARGET_BULK_BYTES && stream_writer.pos > 0
+        if (stream_writer.pos + as_json.bytesize) > @target_bulk_bytes && stream_writer.pos > 0
           stream_writer.flush # ensure writer has sync'd buffers before reporting sizes
           logger.debug("Sending partial bulk request for batch with one or more actions remaining.",
                        :action_count => batch_actions.size,
@@ -324,6 +312,8 @@ module LogStash; module Outputs; class OpenSearch;
 
       adapter_options[:headers] = client_settings[:headers] if client_settings[:headers]
 
+      adapter_options[:auth_type] = options[:auth_type]
+
       adapter_class = ::LogStash::Outputs::OpenSearch::HttpClient::ManticoreAdapter
       adapter = adapter_class.new(@logger, adapter_options)
     end
@@ -332,7 +322,6 @@ module LogStash; module Outputs; class OpenSearch;
       adapter = build_adapter(options)
 
       pool_options = {
-        :distribution_checker => options[:distribution_checker],
         :sniffing => sniffing,
         :sniffer_delay => options[:sniffer_delay],
         :sniffing_path => options[:sniffing_path],
@@ -394,14 +383,14 @@ module LogStash; module Outputs; class OpenSearch;
     end
 
     def template_put(name, template)
-      path = "#{template_endpoint}/#{name}"
+      path = "/#{template_endpoint}/#{name}"
       logger.info("Installing OpenSearch template", name: name)
       @pool.put(path, nil, LogStash::Json.dump(template))
     end
 
     def template_endpoint
       # TODO: Check Version < 7.8 and use index template for >= 7.8 & OpenSearch
-      # https://docs-beta.opensearch.org/opensearch/index-templates/
+      # https://opensearch.org/docs/opensearch/index-templates/
       '_template'
     end
 

data/lib/logstash/outputs/opensearch/http_client_builder.rb

@@ -24,7 +24,6 @@ module LogStash; module Outputs; class OpenSearch;
       client_settings[:proxy] = params["proxy"] if params["proxy"]
       
       common_options = {
-        :distribution_checker => params["distribution_checker"],
         :client_settings => client_settings,
         :metric => params["metric"],
         :resurrect_delay => params["resurrect_delay"]
@@ -36,6 +35,7 @@ module LogStash; module Outputs; class OpenSearch;
       end
 
       common_options[:timeout] = params["timeout"] if params["timeout"]
+      common_options[:target_bulk_bytes] = params["target_bulk_bytes"]
 
       if params["path"]
         client_settings[:path] = dedup_slashes("/#{params["path"]}/")
@@ -107,7 +107,10 @@ module LogStash; module Outputs; class OpenSearch;
       }
       common_options.merge! update_options if params["action"] == 'update'
 
-      create_http_client(common_options.merge(:hosts => hosts, :logger => logger))
+      create_http_client(common_options.merge(:hosts => hosts,
+                                              :logger => logger,
+                                              :auth_type => params["auth_type"]
+                                              ))
     end
 
     def self.create_http_client(options)