logstash-output-opensearch 1.2.0-java → 1.3.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2d7793b7f8cb45bd6d1d6a0893357497b1080fa46733e7806bd7bd6d0ec63ef3
-  data.tar.gz: d577bb202ad1e25d6ef34358dfad0d26cb22a0a579af4ba45343702c90768450
+  metadata.gz: bbe40e1f039f2d91a4adfb8b8fb9a28fd877156bf270060fa1d294986a769983
+  data.tar.gz: 9c2bd44c3e32cbf31d3ffac173a5e9b0ff2e4bcb9c4bd27267ac70a3a88ee6da
 SHA512:
-  metadata.gz: a14b81ed9f97bd3c518227b5fc2e53cfbabab713dedd174aa128ac93674bb00e467e41eff9d11c60da3edd1e563f0da7e31fcdfc0d05f97f591adb93524fb82e
-  data.tar.gz: fb586673825bcc572f1e6b47fefa43540d6515e47542e2415838b56e7a9091586b16ebc07a2fb000955aab70c0459b7496564b88483064807c199711be42f5bf
+  metadata.gz: ed114d31a8dd15ebe57cba0d87c278efe3be2a47743aa2cdfe99bf861e0934edb2772d09aa59eda0d9428b119dde77a16c82371ce19ca1cbfb5c6ce736692690
+  data.tar.gz: df6263f5109d5834bc19fa9804c082c95dc1867eed7926be99b7682c4c1cfb3abffa0ff23b16f123f55f196e353527d31d8d5a61f3011bbf56805f43fa890fdc

data/DEVELOPER_GUIDE.md

@@ -8,6 +8,7 @@
         - [Run plugin](#run-plugin-in-logstash)
         - [Configuration for Logstash Output OpenSearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
     - [Submitting Changes](#submitting-changes)
+    - [Backports](#backports) 
 
 # Developer Guide
 
@@ -206,3 +207,11 @@ Authorization to a secure OpenSearch cluster requires read permission at [index
 ## Submitting Changes
 
 See [CONTRIBUTING](CONTRIBUTING.md).
+
+## Backports
+
+The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the
+original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow
+run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label
+`backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
+merged to main, the workflow will create a backport PR to the `1.x` branch.

data/Gemfile

@@ -17,4 +17,4 @@ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "
 if Dir.exist?(logstash_path) && use_logstash_source
   gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
   gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
-end
+end

data/MAINTAINERS.md

@@ -22,6 +22,9 @@ This document explains who the maintainers are (see below), what they do in this
 | Jack Mazanec             | [jmazanec15](https://github.com/jmazanec15) | Amazon |
 | Vamshi Vijay Nakkirtha   | [vamshin](https://github.com/vamshin)   |   Amazon    |
 | Vijayan Balasubramanian  | [VijayanB](https://github.com/VijayanB) |   Amazon    |
+| Deep Datta               | [deepdatta](https://github.com/deepdatta) | Amazon    |
+| David Venable            | [dlvenable](https://github.com/dlvenable) | Amazon    |
+| Shivani Shukla           | [sshivanii](https://github.com/sshivanii) | Amazon    |
 
 ## Maintainer Responsibilities
 

data/README.md

@@ -1,9 +1,10 @@
 [![Build and Test logstash-output-opensearch plugin](https://github.com/opensearch-project/logstash-output-opensearch/actions/workflows/CI.yml/badge.svg)](https://github.com/opensearch-project/logstash-output-opensearch/actions/workflows/CI.yml)
 ![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)
-# Logstash Plugin
+# Logstash Output OpenSearch
 
 - [Welcome!](#welcome)
 - [Project Resources](#project-resources)
+- [Configuration for Logstash Output Opensearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
 - [Code of Conduct](#code-of-conduct)
 - [License](#license)
 - [Copyright](#copyright)
@@ -12,6 +13,8 @@
 
 **logstash-output-opensearch** is a community-driven, open source fork logstash-output-elasticsearch licensed under the [Apache v2.0 License](LICENSE). For more information, see [opensearch.org](https://opensearch.org/).
 
+The logstash-output-opensearch plugin helps to ship events from Logstash to OpenSearch cluster.
+
 ## Project Resources
 
 * [Project Website](https://opensearch.org/)
@@ -25,6 +28,72 @@
 * [Admin Responsibilities](ADMINS.md)
 * [Security](SECURITY.md)
 
+## Configuration for Logstash Output Opensearch Plugin
+
+To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
+```
+output {
+    opensearch {
+        hosts       => ["hostname:port"]
+        user        => "admin"
+        password    => "admin"
+        index       => "logstash-logs-%{+YYYY.MM.dd}"
+    }
+}
+```
+
+To run the Logstash Output Opensearch plugin using aws_iam authentication, refer to the sample configuration shown below:
+```
+output {        
+   opensearch {     
+          hosts => ["hostname:port"]              
+          auth_type => {    
+              type => 'aws_iam'     
+              aws_access_key_id => 'ACCESS_KEY'     
+              aws_secret_access_key => 'SECRET_KEY'     
+              region => 'us-west-2'         
+          }         
+          index  => "logstash-logs-%{+YYYY.MM.dd}"      
+   }            
+}
+```
+
+In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type.
+
+Example Configuration for basic authentication:
+```
+output {    
+    opensearch {        
+          hosts  => ["hostname:port"]     
+          auth_type => {            
+              type => 'basic'           
+              user => 'admin'           
+              password => 'admin'           
+          }             
+          index => "logstash-logs-%{+YYYY.MM.dd}"       
+   }            
+}  
+```
+
+To ingest data into a `data stream` through logstash, we need to create the data stream and specify the name of data stream and the `op_type` of `create` in the output configuration. The sample configuration is shown below:
+
+```yml
+output {    
+    opensearch {        
+          hosts  => ["https://hostname:port"]     
+          auth_type => {            
+              type => 'basic'           
+              user => 'admin'           
+              password => 'admin'           
+          }
+          index => "my-data-stream"
+          action => "create"
+   }            
+}               
+```
+
+For more details refer to this [documentation](https://opensearch.org/docs/latest/clients/logstash/ship-to-opensearch/#opensearch-output-plugin)
+
 ## Code of Conduct
 
 This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUCT.md). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact [opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments.

data/lib/logstash/outputs/opensearch/http_client.rb

@@ -383,7 +383,7 @@ module LogStash; module Outputs; class OpenSearch;
     end
 
     def template_put(name, template)
-      path = "#{template_endpoint}/#{name}"
+      path = "/#{template_endpoint}/#{name}"
       logger.info("Installing OpenSearch template", name: name)
       @pool.put(path, nil, LogStash::Json.dump(template))
     end

data/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json

@@ -0,0 +1,44 @@
+{
+  "index_patterns" : "logstash-*",
+  "version" : 60001,
+  "settings" : {
+    "index.refresh_interval" : "5s",
+    "number_of_shards": 1
+  },
+  "mappings" : {
+    "dynamic_templates" : [ {
+      "message_field" : {
+        "path_match" : "message",
+        "match_mapping_type" : "string",
+        "mapping" : {
+          "type" : "text",
+          "norms" : false
+        }
+      }
+    }, {
+      "string_fields" : {
+        "match" : "*",
+        "match_mapping_type" : "string",
+        "mapping" : {
+          "type" : "text", "norms" : false,
+          "fields" : {
+            "keyword" : { "type": "keyword", "ignore_above": 256 }
+          }
+        }
+      }
+    } ],
+    "properties" : {
+      "@timestamp": { "type": "date"},
+      "@version": { "type": "keyword"},
+      "geoip"  : {
+        "dynamic": true,
+        "properties" : {
+          "ip": { "type": "ip" },
+          "location" : { "type" : "geo_point" },
+          "latitude" : { "type" : "half_float" },
+          "longitude" : { "type" : "half_float" }
+        }
+      }
+    }
+  }
+}