logstash-output-opensearch 1.2.0-java → 1.3.0-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/DEVELOPER_GUIDE.md +9 -0
- data/Gemfile +1 -1
- data/MAINTAINERS.md +3 -0
- data/README.md +70 -1
- data/lib/logstash/outputs/opensearch/http_client.rb +1 -1
- data/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json +44 -0
- data/lib/logstash/outputs/opensearch/templates/ecs-v8/1x.json +5252 -0
- data/lib/logstash/outputs/opensearch/templates/ecs-v8/2x.json +5252 -0
- data/lib/logstash/outputs/opensearch/templates/ecs-v8/7x.json +5252 -0
- data/lib/logstash/outputs/opensearch.rb +2 -2
- data/logstash-output-opensearch.gemspec +3 -2
- data/spec/integration/outputs/compressed_indexing_spec.rb +7 -1
- data/spec/integration/outputs/index_spec.rb +7 -1
- data/spec/integration/outputs/ingest_pipeline_spec.rb +2 -1
- data/spec/integration/outputs/parent_spec.rb +1 -1
- data/spec/integration/outputs/retry_spec.rb +2 -1
- data/spec/integration/outputs/templates_spec.rb +83 -57
- data/spec/opensearch_spec_helper.rb +10 -0
- data/spec/unit/outputs/opensearch/http_client_spec.rb +1 -1
- data/spec/unit/outputs/opensearch/template_manager_spec.rb +23 -4
- data/spec/unit/outputs/opensearch_spec.rb +16 -0
- data.tar.gz.sig +0 -0
- metadata +48 -25
- metadata.gz.sig +0 -0
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: bbe40e1f039f2d91a4adfb8b8fb9a28fd877156bf270060fa1d294986a769983
|
4
|
+
data.tar.gz: 9c2bd44c3e32cbf31d3ffac173a5e9b0ff2e4bcb9c4bd27267ac70a3a88ee6da
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ed114d31a8dd15ebe57cba0d87c278efe3be2a47743aa2cdfe99bf861e0934edb2772d09aa59eda0d9428b119dde77a16c82371ce19ca1cbfb5c6ce736692690
|
7
|
+
data.tar.gz: df6263f5109d5834bc19fa9804c082c95dc1867eed7926be99b7682c4c1cfb3abffa0ff23b16f123f55f196e353527d31d8d5a61f3011bbf56805f43fa890fdc
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/DEVELOPER_GUIDE.md
CHANGED
@@ -8,6 +8,7 @@
|
|
8
8
|
- [Run plugin](#run-plugin-in-logstash)
|
9
9
|
- [Configuration for Logstash Output OpenSearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
|
10
10
|
- [Submitting Changes](#submitting-changes)
|
11
|
+
- [Backports](#backports)
|
11
12
|
|
12
13
|
# Developer Guide
|
13
14
|
|
@@ -206,3 +207,11 @@ Authorization to a secure OpenSearch cluster requires read permission at [index
|
|
206
207
|
## Submitting Changes
|
207
208
|
|
208
209
|
See [CONTRIBUTING](CONTRIBUTING.md).
|
210
|
+
|
211
|
+
## Backports
|
212
|
+
|
213
|
+
The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the
|
214
|
+
original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow
|
215
|
+
run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label
|
216
|
+
`backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
|
217
|
+
merged to main, the workflow will create a backport PR to the `1.x` branch.
|
data/Gemfile
CHANGED
@@ -17,4 +17,4 @@ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "
|
|
17
17
|
if Dir.exist?(logstash_path) && use_logstash_source
|
18
18
|
gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
|
19
19
|
gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
|
20
|
-
end
|
20
|
+
end
|
data/MAINTAINERS.md
CHANGED
@@ -22,6 +22,9 @@ This document explains who the maintainers are (see below), what they do in this
|
|
22
22
|
| Jack Mazanec | [jmazanec15](https://github.com/jmazanec15) | Amazon |
|
23
23
|
| Vamshi Vijay Nakkirtha | [vamshin](https://github.com/vamshin) | Amazon |
|
24
24
|
| Vijayan Balasubramanian | [VijayanB](https://github.com/VijayanB) | Amazon |
|
25
|
+
| Deep Datta | [deepdatta](https://github.com/deepdatta) | Amazon |
|
26
|
+
| David Venable | [dlvenable](https://github.com/dlvenable) | Amazon |
|
27
|
+
| Shivani Shukla | [sshivanii](https://github.com/sshivanii) | Amazon |
|
25
28
|
|
26
29
|
## Maintainer Responsibilities
|
27
30
|
|
data/README.md
CHANGED
@@ -1,9 +1,10 @@
|
|
1
1
|
[![Build and Test logstash-output-opensearch plugin](https://github.com/opensearch-project/logstash-output-opensearch/actions/workflows/CI.yml/badge.svg)](https://github.com/opensearch-project/logstash-output-opensearch/actions/workflows/CI.yml)
|
2
2
|
![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)
|
3
|
-
# Logstash
|
3
|
+
# Logstash Output OpenSearch
|
4
4
|
|
5
5
|
- [Welcome!](#welcome)
|
6
6
|
- [Project Resources](#project-resources)
|
7
|
+
- [Configuration for Logstash Output Opensearch Plugin](#configuration-for-logstash-output-opensearch-plugin)
|
7
8
|
- [Code of Conduct](#code-of-conduct)
|
8
9
|
- [License](#license)
|
9
10
|
- [Copyright](#copyright)
|
@@ -12,6 +13,8 @@
|
|
12
13
|
|
13
14
|
**logstash-output-opensearch** is a community-driven, open source fork logstash-output-elasticsearch licensed under the [Apache v2.0 License](LICENSE). For more information, see [opensearch.org](https://opensearch.org/).
|
14
15
|
|
16
|
+
The logstash-output-opensearch plugin helps to ship events from Logstash to OpenSearch cluster.
|
17
|
+
|
15
18
|
## Project Resources
|
16
19
|
|
17
20
|
* [Project Website](https://opensearch.org/)
|
@@ -25,6 +28,72 @@
|
|
25
28
|
* [Admin Responsibilities](ADMINS.md)
|
26
29
|
* [Security](SECURITY.md)
|
27
30
|
|
31
|
+
## Configuration for Logstash Output Opensearch Plugin
|
32
|
+
|
33
|
+
To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
|
34
|
+
```
|
35
|
+
output {
|
36
|
+
opensearch {
|
37
|
+
hosts => ["hostname:port"]
|
38
|
+
user => "admin"
|
39
|
+
password => "admin"
|
40
|
+
index => "logstash-logs-%{+YYYY.MM.dd}"
|
41
|
+
}
|
42
|
+
}
|
43
|
+
```
|
44
|
+
|
45
|
+
To run the Logstash Output Opensearch plugin using aws_iam authentication, refer to the sample configuration shown below:
|
46
|
+
```
|
47
|
+
output {
|
48
|
+
opensearch {
|
49
|
+
hosts => ["hostname:port"]
|
50
|
+
auth_type => {
|
51
|
+
type => 'aws_iam'
|
52
|
+
aws_access_key_id => 'ACCESS_KEY'
|
53
|
+
aws_secret_access_key => 'SECRET_KEY'
|
54
|
+
region => 'us-west-2'
|
55
|
+
}
|
56
|
+
index => "logstash-logs-%{+YYYY.MM.dd}"
|
57
|
+
}
|
58
|
+
}
|
59
|
+
```
|
60
|
+
|
61
|
+
In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type.
|
62
|
+
|
63
|
+
Example Configuration for basic authentication:
|
64
|
+
```
|
65
|
+
output {
|
66
|
+
opensearch {
|
67
|
+
hosts => ["hostname:port"]
|
68
|
+
auth_type => {
|
69
|
+
type => 'basic'
|
70
|
+
user => 'admin'
|
71
|
+
password => 'admin'
|
72
|
+
}
|
73
|
+
index => "logstash-logs-%{+YYYY.MM.dd}"
|
74
|
+
}
|
75
|
+
}
|
76
|
+
```
|
77
|
+
|
78
|
+
To ingest data into a `data stream` through logstash, we need to create the data stream and specify the name of data stream and the `op_type` of `create` in the output configuration. The sample configuration is shown below:
|
79
|
+
|
80
|
+
```yml
|
81
|
+
output {
|
82
|
+
opensearch {
|
83
|
+
hosts => ["https://hostname:port"]
|
84
|
+
auth_type => {
|
85
|
+
type => 'basic'
|
86
|
+
user => 'admin'
|
87
|
+
password => 'admin'
|
88
|
+
}
|
89
|
+
index => "my-data-stream"
|
90
|
+
action => "create"
|
91
|
+
}
|
92
|
+
}
|
93
|
+
```
|
94
|
+
|
95
|
+
For more details refer to this [documentation](https://opensearch.org/docs/latest/clients/logstash/ship-to-opensearch/#opensearch-output-plugin)
|
96
|
+
|
28
97
|
## Code of Conduct
|
29
98
|
|
30
99
|
This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUCT.md). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact [opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments.
|
@@ -383,7 +383,7 @@ module LogStash; module Outputs; class OpenSearch;
|
|
383
383
|
end
|
384
384
|
|
385
385
|
def template_put(name, template)
|
386
|
-
path = "
|
386
|
+
path = "/#{template_endpoint}/#{name}"
|
387
387
|
logger.info("Installing OpenSearch template", name: name)
|
388
388
|
@pool.put(path, nil, LogStash::Json.dump(template))
|
389
389
|
end
|
@@ -0,0 +1,44 @@
|
|
1
|
+
{
|
2
|
+
"index_patterns" : "logstash-*",
|
3
|
+
"version" : 60001,
|
4
|
+
"settings" : {
|
5
|
+
"index.refresh_interval" : "5s",
|
6
|
+
"number_of_shards": 1
|
7
|
+
},
|
8
|
+
"mappings" : {
|
9
|
+
"dynamic_templates" : [ {
|
10
|
+
"message_field" : {
|
11
|
+
"path_match" : "message",
|
12
|
+
"match_mapping_type" : "string",
|
13
|
+
"mapping" : {
|
14
|
+
"type" : "text",
|
15
|
+
"norms" : false
|
16
|
+
}
|
17
|
+
}
|
18
|
+
}, {
|
19
|
+
"string_fields" : {
|
20
|
+
"match" : "*",
|
21
|
+
"match_mapping_type" : "string",
|
22
|
+
"mapping" : {
|
23
|
+
"type" : "text", "norms" : false,
|
24
|
+
"fields" : {
|
25
|
+
"keyword" : { "type": "keyword", "ignore_above": 256 }
|
26
|
+
}
|
27
|
+
}
|
28
|
+
}
|
29
|
+
} ],
|
30
|
+
"properties" : {
|
31
|
+
"@timestamp": { "type": "date"},
|
32
|
+
"@version": { "type": "keyword"},
|
33
|
+
"geoip" : {
|
34
|
+
"dynamic": true,
|
35
|
+
"properties" : {
|
36
|
+
"ip": { "type": "ip" },
|
37
|
+
"location" : { "type" : "geo_point" },
|
38
|
+
"latitude" : { "type" : "half_float" },
|
39
|
+
"longitude" : { "type" : "half_float" }
|
40
|
+
}
|
41
|
+
}
|
42
|
+
}
|
43
|
+
}
|
44
|
+
}
|