logstash-output-opensearch 2.0.2-java → 2.0.3-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a9700db92f60b264d7edf26bb34281f9fac13c27d93564d80ce007e5e7b73928
-  data.tar.gz: a4feb8d63b747cd2db45ec1b47df5e30a6aea6dbee3a30dde51c94773b192ec8
+  metadata.gz: c4d023016ab9043a845b0c4c35fc0fea899235e8f15d32f8cc9799a29dd4a84a
+  data.tar.gz: 1eef3e646ab7a2ec0ac175fa26b3fe7983755133de4bb0580ea55299d55ce18c
 SHA512:
-  metadata.gz: a7ad196b7d23790cf9bed976a3c51b0a2ebbd2d1bb9fa92617a9994620d0be287c142c5ce944cb941626f598491d38b112d68797f7953e63180204522a639af3
-  data.tar.gz: cdd308cd216f67382e466e98546f04e6ac1a011e6d4f64e4b27e23d44809ae6a069c3849e62986a0f79ccb26d40eaa12fb998271072efaf87be1a5ca765da48a
+  metadata.gz: e22c44f48115c0df680dc9dc1e886daa5ac1521daf1a3b1ee3c9f5574eed59e7974356bff2055dfc44c0ed01115cb8838b073ffa7b81d70172b5da227c718584
+  data.tar.gz: 98556ece3e0e8f771a8168bffb32d6b8be5d4e21d356cbd64ddb814b26ac5cb91c3b58a94f49dec72e377a67cb3896008a7441f9db2373bcf3c15de9f7ece2a9

data/DEVELOPER_GUIDE.md

@@ -9,6 +9,7 @@
         - [Configuration for Logstash Output OpenSearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
     - [Submitting Changes](#submitting-changes)
     - [Backports](#backports) 
+- [Building Custom Docker Images](docs/building_custom_docker_images.md)
 
 # Developer Guide
 
@@ -185,13 +186,14 @@ Build the gem locally and install it using:
 ## Configuration for Logstash Output OpenSearch Plugin
 
 To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
 
 ```
 output {
     opensearch {
         hosts       => ["hostname:port"]
         user        => "admin"
-        password    => "admin"
+        password    => "<your-admin-password>"
         index       => "logstash-logs-%{+YYYY.MM.dd}"
     }
 }
@@ -215,3 +217,6 @@ original PR with an appropriate label `backport <backport-branch-name>` is merge
 run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label
 `backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
 merged to main, the workflow will create a backport PR to the `1.x` branch.
+
+# [Building Custom Docker Images](docs/building_custom_docker_images.md)
+

data/MAINTAINERS.md

@@ -5,11 +5,12 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 ## Current Maintainers
 
 
-| Maintainer              | GitHub ID                                   | Affiliation |
-| ----------------------- | ------------------------------------------- | ----------- |
-| Asif Sohail Mohammed | [asifsmohammed](https://github.com/asifsmohammed)     | Amazon      |
-| David Venable        | [dlvenable](https://github.com/dlvenable)             | Amazon      |
-| Hai Yan              | [oeyh](https://github.com/oeyh)                       | Amazon      |
+| Maintainer               | GitHub ID                                           | Affiliation |
+| ------------------------ | --------------------------------------------------- | ----------- |
+| Daniel "dB." Doubrovkine | [dblock](https://github.com/dblock)                 | Amazon      |
+| Asif Sohail Mohammed     | [asifsmohammed](https://github.com/asifsmohammed)   | Amazon      |
+| David Venable            | [dlvenable](https://github.com/dlvenable)           | Amazon      |
+| Hai Yan                  | [oeyh](https://github.com/oeyh)                     | Amazon      |
 
 
 

data/README.md

@@ -32,12 +32,14 @@ The logstash-output-opensearch plugin helps to ship events from Logstash to Open
 ## Configuration for Logstash Output Opensearch Plugin
 
 To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
+
 ```
 output {
     opensearch {
         hosts       => ["hostname:port"]
         user        => "admin"
-        password    => "admin"
+        password    => "<your-admin-password>"
         index       => "logstash-logs-%{+YYYY.MM.dd}"
     }
 }
@@ -62,6 +64,8 @@ output {
 In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type.
 
 Example Configuration for basic authentication:
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
+
 ```
 output {
     opensearch {
@@ -69,7 +73,7 @@ output {
           auth_type => {
               type => 'basic'
               user => 'admin'
-              password => 'admin'
+              password => '<your-admin-password>'
           }
           index => "logstash-logs-%{+YYYY.MM.dd}"
    }
@@ -77,6 +81,7 @@ output {
 ```
 
 To ingest data into a `data stream` through logstash, we need to create the data stream and specify the name of data stream and the `op_type` of `create` in the output configuration. The sample configuration is shown below:
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
 
 ```yml
 output {
@@ -85,7 +90,7 @@ output {
           auth_type => {
               type => 'basic'
               user => 'admin'
-              password => 'admin'
+              password => '<your-admin-password>'
           }
           index => "my-data-stream"
           action => "create"
@@ -105,7 +110,7 @@ Starting in 2.0.0, the aws sdk version is bumped to v3. In order for all other A
 /usr/share/logstash/bin/logstash-plugin remove logstash-output-cloudwatch
 
 /usr/share/logstash/bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws
-bin/logstash-plugin install --version 2.0.0 logstash-output-opensearch
+/usr/share/logstash/bin/logstash-plugin install --version 2.0.0 logstash-output-opensearch
 ```
 ## ECS Compatibility
 [Elastic Common Schema(ECS)](https://www.elastic.co/guide/en/ecs/current/index.html]) compatibility for V8 was added in 1.3.0. For more details on ECS support refer to this [documentation](docs/ecs_compatibility.md).

data/docs/building_custom_docker_images.md

@@ -0,0 +1,79 @@
+- [Building Custom Docker Images](#building-custom-docker-images)
+  - [Logstash 8.x](#logstash-8x)
+  - [Logstash 7.x](#logstash-7x)
+    - [Build Logstash Output OpenSearch Plugin Gem](#build-logstash-output-opensearch-plugin-gem)
+    - [Dockerfile](#dockerfile)
+
+
+# Building Custom Docker Images
+
+To build an image that is not available in the [official Docker repository tags](https://hub.docker.com/r/opensearchproject/logstash-oss-with-opensearch-output-plugin/tags), or to add specific plugins to your image, you can create a custom Dockerfile.
+
+The process varies depending on whether you want to build an image with `Logstash 8.x` versions or `Logstash 7.x` versions
+
+## Logstash 8.x
+
+Create this `Dockerfile` to build an image with `OpenSearch 2.0.2` for **`Logstash 8.x`**
+
+``` Dockerfile
+ARG APP_VERSION
+
+FROM docker.elastic.co/logstash/logstash-oss:${APP_VERSION}
+RUN logstash-plugin install --version 7.1.1 logstash-integration-aws
+RUN logstash-plugin install --version 2.0.2 logstash-output-opensearch
+```
+
+## Logstash 7.x
+
+For **`Logstash 7.x`** , the Logstash output OpenSearch gem needs to be build.
+
+### Build Logstash Output OpenSearch Plugin Gem
+
+1. Clone `logstash-output-opensearch repo` 
+
+    ```sh
+    git clone https://github.com/opensearch-project/logstash-output-opensearch.git
+    ```
+
+2. Checkout the tag for the plugin version you want to build. For the version [2.0.2](https://github.com/opensearch-project/logstash-output-opensearch/tree/2.0.2) for example
+
+    ```sh
+    git checkout 2.0.2
+    ```
+
+
+3. Remove [this line that adds the json version spec](https://github.com/opensearch-project/logstash-output-opensearch/blob/2.0.2/logstash-output-opensearch.gemspec#L49). This version of the JSON gem is incompatible with `Logstash version 7.x`.
+
+4. Build the gem by running the following command:
+
+    ```sh
+    gem build logstash-output-opensearch.gemspec
+    ```
+
+    The Gemfile `logstash-output-opensearch-2.0.2-x86_64-linux.gem` will be generated.
+
+### Dockerfile
+
+Create this Dockerfile to build an image with `logstash version 7.x` and the previously generated `Gemfile` :
+
+```Dockerfile
+ARG APP_VERSION
+
+FROM docker.elastic.co/logstash/logstash-oss:${APP_VERSION}
+
+USER logstash
+# Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
+# https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
+# https://github.com/opensearch-project/logstash-output-opensearch#configuration-for-logstash-output-opensearch-plugin
+RUN logstash-plugin remove logstash-input-s3
+RUN logstash-plugin remove logstash-input-sqs
+RUN logstash-plugin remove logstash-output-s3
+RUN logstash-plugin remove logstash-output-sns
+RUN logstash-plugin remove logstash-output-sqs
+RUN logstash-plugin remove logstash-output-cloudwatch
+
+RUN logstash-plugin install --version 7.1.1 logstash-integration-aws
+
+COPY logstash-output-opensearch-2.0.2-x86_64-linux.gem /usr/share
+RUN logstash-plugin install /usr/share/logstash-output-opensearch-2.0.2-x86_64-linux.gem
+```

data/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

@@ -33,7 +33,8 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
     :profile,
     :instance_profile_credentials_retries,
     :instance_profile_credentials_timeout,
-    :region)
+    :region,
+    :account_id)
 
   class ManticoreAdapter
     attr_reader :manticore, :logger
@@ -78,10 +79,11 @@ module LogStash; module Outputs; class OpenSearch; class HttpClient;
       instance_cred_retries = options[:auth_type]["instance_profile_credentials_retries"] || AWS_DEFAULT_PROFILE_CREDENTIAL_RETRY
       instance_cred_timeout = options[:auth_type]["instance_profile_credentials_timeout"] || AWS_DEFAULT_PROFILE_CREDENTIAL_TIMEOUT
       region = options[:auth_type]["region"] || AWS_DEFAULT_REGION
+      account_id = nil
       set_aws_region(region)
       set_service_name(options[:auth_type]["service_name"] || AWS_SERVICE)
 
-      credential_config = AWSIAMCredential.new(aws_access_key_id, aws_secret_access_key, session_token, profile, instance_cred_retries, instance_cred_timeout, region)
+      credential_config = AWSIAMCredential.new(aws_access_key_id, aws_secret_access_key, session_token, profile, instance_cred_retries, instance_cred_timeout, region, account_id)
       @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
     end
 

data/lib/logstash/plugin_mixins/opensearch/common.rb

@@ -87,6 +87,7 @@ module LogStash; module PluginMixins; module OpenSearch
 
     def discover_cluster_uuid
       return unless defined?(plugin_metadata)
+      return if params && params['auth_type'] && params['auth_type']['service_name'] == "aoss" # AOSS doesn't support GET /
       cluster_info = client.get('/')
       plugin_metadata.set(:cluster_uuid, cluster_info['cluster_uuid'])
     rescue => e

data/logstash-output-opensearch.gemspec

@@ -11,7 +11,7 @@ signing_key_path = "gem-private_key.pem"
 
 Gem::Specification.new do |s|
   s.name            = 'logstash-output-opensearch'
-  s.version         = '2.0.2'
+  s.version         = '2.0.3'
 
   s.licenses        = ['Apache-2.0']
   s.summary         = "Stores logs in OpenSearch"

data/spec/integration/outputs/index_spec.rb

@@ -140,7 +140,7 @@ describe "indexing" do
   end
   describe "a secured indexer", :secure_integration => true do
     let(:user) { "admin" }
-    let(:password) { "admin" }
+    let(:password) { OpenSearchHelper.admin_password }
     let(:opensearch_url) {"https://integration:9200"}
     let(:config) do
       {
@@ -172,7 +172,8 @@ describe "indexing" do
       :auth_type => {
         "type"=>"basic",
         "user" => "admin",
-        "password" => "admin"}
+        "password" => OpenSearchHelper.admin_password
+        }
     } }
     let(:user) {options[:auth_type]["user"]}
     let(:password) {options[:auth_type]["password"]}

data/spec/opensearch_spec_helper.rb

@@ -72,6 +72,15 @@ module OpenSearchHelper
     end
   end
 
+  # set admin password based on version
+  def self.admin_password
+    if check_version?('< 2.12.0') || check_version?('> 7')
+      "admin"
+    else
+      "myStrongPassword123!"
+    end
+  end
+
   def clean(client)
     client.indices.delete_template(:name => "*")
     client.indices.delete_index_template(:name => "logstash*") rescue nil

data/spec/unit/outputs/opensearch_spec.rb

@@ -776,6 +776,30 @@ describe LogStash::Outputs::OpenSearch do
       expect(logger).to have_received(:error).with(/Unable to retrieve OpenSearch cluster uuid/i, anything)
     end
 
+    context 'with iam auth' do
+      context 'es' do
+        let(:options) { { 'hosts' => '127.0.0.1:9999', 'auth_type' => { 'service_name' => 'es' } } }
+        it "logs inability to retrieve uuid" do
+          allow(subject).to receive(:install_template)
+          subject.register
+          subject.send :wait_for_successful_connection
+    
+          expect(logger).to have_received(:error).with(/Unable to retrieve OpenSearch cluster uuid/i, anything)
+        end
+      end
+
+      context 'aoss' do
+        let(:options) { { 'hosts' => '127.0.0.1:9999', 'auth_type' => { 'service_name' => 'aoss' } } }
+        it "does not attempt to retrieve uuid" do
+          allow(subject).to receive(:install_template)
+          subject.register
+          subject.send :wait_for_successful_connection
+
+          expect(logger).to_not have_received(:error)
+        end
+      end
+    end
+
     it "logs template install failure" do
       allow(subject).to receive(:discover_cluster_uuid)
       subject.register

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-opensearch
 version: !ruby/object:Gem::Version
-  version: 2.0.2
+  version: 2.0.3
 platform: java
 authors:
 - Elastic
@@ -13,7 +13,7 @@ cert_chain:
   -----BEGIN CERTIFICATE-----
   MIIDfDCCAmSgAwIBAgIBATANBgkqhkiG9w0BAQsFADBCMRMwEQYDVQQDDApvcGVu
   c2VhcmNoMRYwFAYKCZImiZPyLGQBGRYGYW1hem9uMRMwEQYKCZImiZPyLGQBGRYD
-  Y29tMB4XDTIzMDgyNDIwNDIwNFoXDTI0MDgyMzIwNDIwNFowQjETMBEGA1UEAwwK
+  Y29tMB4XDTI0MDcyNjIzMjIyN1oXDTI1MDcyNjIzMjIyN1owQjETMBEGA1UEAwwK
   b3BlbnNlYXJjaDEWMBQGCgmSJomT8ixkARkWBmFtYXpvbjETMBEGCgmSJomT8ixk
   ARkWA2NvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM1z3/jcitjV
   umXwRFn+JSBBd36qZB54Dtucf6+E2fmNPzBRhgYN5XJy/+clQJ9NIJV7C8H52P3V
@@ -24,14 +24,14 @@ cert_chain:
   zfR37/NQFkECAwEAAaN9MHswCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0O
   BBYEFJJ2myhLXK742btavNbG0IWrMNBIMCAGA1UdEQQZMBeBFW9wZW5zZWFyY2hA
   YW1hem9uLmNvbTAgBgNVHRIEGTAXgRVvcGVuc2VhcmNoQGFtYXpvbi5jb20wDQYJ
-  KoZIhvcNAQELBQADggEBABAQpnELuY5AgdNUIlIVRVATO6iZOXTbt3a9oVbQdLPe
-  BfMObZyJydg0+leyR3oFyN9ZIFiEFwpd0biFf39DuC0M6Oge0Rv4oO9GRI3yyv77
-  9m59he+5DI3hbqtGje108oqRe61NZMlKcy/DCBVkzzZFsJ17GC09tY/gwhmNRtLV
-  3vYIEY6vmn57wrGn1NUzWdG+x/XVjYPw5Kwo+/rCxxZqpVTklMqVWV43N/lFrUOe
-  1DlemA1SsUBIoF7CwtVd/RTG/K1iT6nBD08fdKxodMhI5ujkP3N7gkxzRf6aKN4z
-  glnDJYZjluKBUsKTOLdPW1CZpb0AHLpNqDf8SVHsPFk=
+  KoZIhvcNAQELBQADggEBAB1D6Ba88KkTApeUl0Iv/WKie1WNi+6o3KQhqqLt+xjB
+  oiBLLdQpKwH7k/TkCmfo9/8lY1sa3Pxckuw2fNVDaVhEHX56fmTUPoOjkyPS4H19
+  YEKFBeY6U1aF7a2piN58j4EcPP7Kv0KD3RlaEwKbYJTiOy/0f5XjWqVYKOS+DrjM
+  EMhX0gWR5oucydDNSi5hkggPYqCZnW9q6yo+k/FW+DK33CeUm69D6elFx7qPGhEx
+  m0rB25J1sO2yjlalRdBoV5p7OKaWG4CGMr+q6vCC2za7yrtZiYOOv+KpWmheTwds
+  P1nbdIL2eAXGfNU3cbtYisS1sSYi7VSkS52pVk8Jgtw=
   -----END CERTIFICATE-----
-date: 2023-08-25 00:00:00.000000000 Z
+date: 2024-11-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -43,8 +43,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 1.0.0
   name: manticore
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -63,8 +63,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0.0'
   name: stud
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -83,8 +83,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.99'
   name: logstash-core-plugin-api
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -100,8 +100,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '1.0'
   name: logstash-mixin-ecs_compatibility_support
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -114,8 +114,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '3'
   name: aws-sdk
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -131,8 +131,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2'
   name: json
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -148,8 +148,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
   name: logstash-codec-plain
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -162,8 +162,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
   name: logstash-devutils
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -176,8 +176,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
   name: flores
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -190,8 +190,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0.6'
   name: cabin
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -204,8 +204,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '1'
   name: opensearch-ruby
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -231,6 +231,7 @@ files:
 - README.md
 - RELEASING.md
 - SECURITY.md
+- docs/building_custom_docker_images.md
 - docs/ecs_compatibility.md
 - lib/logstash/outputs/opensearch.rb
 - lib/logstash/outputs/opensearch/http_client.rb