fluent-plugin-opensearch 1.1.5 → 1.1.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.yaml +73 -0
- data/.github/ISSUE_TEMPLATE/config.yml +5 -0
- data/.github/ISSUE_TEMPLATE/feature_request.yaml +38 -0
- data/.github/dependabot.yml +23 -0
- data/.github/workflows/add-to-project.yml +24 -0
- data/.github/workflows/coverage.yaml +3 -4
- data/.github/workflows/issue-auto-closer.yml +1 -1
- data/.github/workflows/linux.yml +17 -6
- data/.github/workflows/macos.yml +17 -6
- data/.github/workflows/windows.yml +17 -6
- data/History.md +11 -0
- data/README.md +40 -7
- data/fluent-plugin-opensearch.gemspec +2 -2
- data/lib/fluent/plugin/in_opensearch.rb +1 -1
- data/lib/fluent/plugin/opensearch_index_template.rb +2 -2
- data/lib/fluent/plugin/out_opensearch.rb +19 -12
- data/lib/fluent/plugin/out_opensearch_data_stream.rb +15 -1
- data/test/plugin/test_opensearch_error_handler.rb +0 -1
- data/test/plugin/test_out_opensearch.rb +120 -2
- data/test/plugin/test_out_opensearch_data_stream.rb +127 -0
- metadata +10 -10
- data/.github/ISSUE_TEMPLATE/bug_report.md +0 -29
- data/.github/ISSUE_TEMPLATE/feature_request.md +0 -24
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 59ef255b7e5cb5ceae9b97b0678cdc7151f0514263934f21e1028a45220180d6
|
|
4
|
+
data.tar.gz: 7603936854d8a36c74b6fa6ee4187b089cd781c997da8b97eed21fafbd52833c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4caf0aa8ff2803a46536db82defff7b203f837dea3e2fc97b46487a04c9d85c5b4f71b9145985e9616e4db54f4bdef93a61ca85d941612474794276bdbf37531
|
|
7
|
+
data.tar.gz: b9fb24c499f3c84ece032e744a567ee9c34b7b8a2f47c39a706408e80fb7c8f19624499d447ed4191e75333c4050ef79237e51037dd0122ea95fc7aa6ad6f097
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
name: Bug Report
|
|
2
|
+
description: Create a report with a procedure for reproducing the bug
|
|
3
|
+
labels: "waiting-for-triage"
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Please check the [README](https://github.com/fluent/fluent-plugin-opensearch/blob/main/README.md) first. To help us investigate the problem, please provide the following information.
|
|
9
|
+
- type: textarea
|
|
10
|
+
id: description
|
|
11
|
+
attributes:
|
|
12
|
+
label: Describe the bug
|
|
13
|
+
description: A clear and concise description of what the bug is
|
|
14
|
+
validations:
|
|
15
|
+
required: true
|
|
16
|
+
- type: textarea
|
|
17
|
+
id: reproduce
|
|
18
|
+
attributes:
|
|
19
|
+
label: To Reproduce
|
|
20
|
+
description: Steps to reproduce the behavior
|
|
21
|
+
validations:
|
|
22
|
+
required: true
|
|
23
|
+
- type: textarea
|
|
24
|
+
id: expected
|
|
25
|
+
attributes:
|
|
26
|
+
label: Expected behavior
|
|
27
|
+
description: A clear and concise description of what you expected to happen
|
|
28
|
+
validations:
|
|
29
|
+
required: true
|
|
30
|
+
- type: textarea
|
|
31
|
+
id: environment
|
|
32
|
+
attributes:
|
|
33
|
+
label: Your Environment
|
|
34
|
+
description: |
|
|
35
|
+
- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
|
|
36
|
+
- Operating system: `cat /etc/os-release`
|
|
37
|
+
- Kernel version: `uname -r`
|
|
38
|
+
|
|
39
|
+
Tip: If you hit the problem with older fluentd version, try latest version first.
|
|
40
|
+
value: |
|
|
41
|
+
- Fluentd version:
|
|
42
|
+
- TD Agent version:
|
|
43
|
+
- fluent-plugin-opensearch version:
|
|
44
|
+
- OpenSearch version (optional)
|
|
45
|
+
- OpenSearch template(s) (optional)
|
|
46
|
+
- Operating system:
|
|
47
|
+
- Kernel version:
|
|
48
|
+
render: markdown
|
|
49
|
+
validations:
|
|
50
|
+
required: true
|
|
51
|
+
- type: textarea
|
|
52
|
+
id: configuration
|
|
53
|
+
attributes:
|
|
54
|
+
label: Your Configuration
|
|
55
|
+
description: |
|
|
56
|
+
Write your configuration here. Minimum reproducible fluentd.conf is recommended.
|
|
57
|
+
validations:
|
|
58
|
+
required: true
|
|
59
|
+
- type: textarea
|
|
60
|
+
id: logs
|
|
61
|
+
attributes:
|
|
62
|
+
label: Your Error Log
|
|
63
|
+
description: Write your ALL error log here
|
|
64
|
+
render: shell
|
|
65
|
+
validations:
|
|
66
|
+
required: true
|
|
67
|
+
- type: textarea
|
|
68
|
+
id: addtional-context
|
|
69
|
+
attributes:
|
|
70
|
+
label: Additional context
|
|
71
|
+
description: Add any other context about the problem here.
|
|
72
|
+
validations:
|
|
73
|
+
required: false
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
name: Feature request
|
|
2
|
+
description: Suggest an idea for this project
|
|
3
|
+
labels: "waiting-for-triage"
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Please check the [README](https://github.com/fluent/fluent-plugin-opensearch/blob/main/README.md) first. To help us understand the request, please provide the following information.
|
|
9
|
+
- type: textarea
|
|
10
|
+
id: description
|
|
11
|
+
attributes:
|
|
12
|
+
label: Is your feature request related to a problem? Please describe.
|
|
13
|
+
description: |
|
|
14
|
+
A clear and concise description of what the problem is.
|
|
15
|
+
Ex. I'm always frustrated when [...]
|
|
16
|
+
validations:
|
|
17
|
+
required: true
|
|
18
|
+
- type: textarea
|
|
19
|
+
id: solution
|
|
20
|
+
attributes:
|
|
21
|
+
label: Describe the solution you'd like
|
|
22
|
+
description: A clear and concise description of what you want to happen.
|
|
23
|
+
validations:
|
|
24
|
+
required: true
|
|
25
|
+
- type: textarea
|
|
26
|
+
id: alternative
|
|
27
|
+
attributes:
|
|
28
|
+
label: Describe alternatives you've considered
|
|
29
|
+
description: A clear and concise description of any alternative solutions or features you've considered.
|
|
30
|
+
validations:
|
|
31
|
+
required: true
|
|
32
|
+
- type: textarea
|
|
33
|
+
id: addtional-context
|
|
34
|
+
attributes:
|
|
35
|
+
label: Additional context
|
|
36
|
+
description: Add any other context or screenshots about the feature request here.
|
|
37
|
+
validations:
|
|
38
|
+
required: false
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
version: 2
|
|
2
|
+
updates:
|
|
3
|
+
- package-ecosystem: 'github-actions'
|
|
4
|
+
directory: '/'
|
|
5
|
+
schedule:
|
|
6
|
+
interval: 'monthly'
|
|
7
|
+
groups:
|
|
8
|
+
# PR: "Security update [package] from [old] to [new]"
|
|
9
|
+
# This PR should be merged in hurry
|
|
10
|
+
security-updates:
|
|
11
|
+
applies-to: security-updates
|
|
12
|
+
patterns:
|
|
13
|
+
- '*'
|
|
14
|
+
|
|
15
|
+
# PR: "Bump [package] from [old] to [new]"
|
|
16
|
+
# No need to be merged this PR in hurry. It is enough to merge
|
|
17
|
+
# once in a month.
|
|
18
|
+
monthly-updates:
|
|
19
|
+
applies-to: version-updates
|
|
20
|
+
patterns:
|
|
21
|
+
- '*'
|
|
22
|
+
# Allow to create PR both of security and normal updates.
|
|
23
|
+
open-pull-requests-limit: 1
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
name: Add bugs to fluent project
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
issues:
|
|
5
|
+
types:
|
|
6
|
+
- opened
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
add-to-project:
|
|
10
|
+
name: Add issue to project
|
|
11
|
+
runs-on: ubuntu-latest
|
|
12
|
+
steps:
|
|
13
|
+
- name: Generate token
|
|
14
|
+
id: generate_token
|
|
15
|
+
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
|
|
16
|
+
with:
|
|
17
|
+
app-id: ${{ secrets.PROJECT_APP_ID }}
|
|
18
|
+
private-key: ${{ secrets.PROJECT_APP_PRIVATE_KEY }}
|
|
19
|
+
owner: fluent
|
|
20
|
+
- uses: actions/add-to-project@5afcf98fcd03f1c2f92c3c83f58ae24323cc57fd # v2.0.0
|
|
21
|
+
with:
|
|
22
|
+
project-url: https://github.com/orgs/fluent/projects/4
|
|
23
|
+
github-token: ${{ steps.generate_token.outputs.token }}
|
|
24
|
+
labeled: waiting-for-triage
|
|
@@ -6,17 +6,16 @@ jobs:
|
|
|
6
6
|
coverage:
|
|
7
7
|
runs-on: ubuntu-latest
|
|
8
8
|
steps:
|
|
9
|
-
- uses: actions/checkout@
|
|
9
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
10
10
|
- name: Set up Ruby 3.0
|
|
11
|
-
uses: ruby/setup-ruby@v1
|
|
11
|
+
uses: ruby/setup-ruby@95ef2b042f9d7a56d8268cba8559e2842e2ad01b # v1.321.0
|
|
12
12
|
with:
|
|
13
13
|
ruby-version: '3.1'
|
|
14
14
|
- name: Build and test with Rake
|
|
15
15
|
run: |
|
|
16
|
-
gem install bundler
|
|
17
16
|
bundle install
|
|
18
17
|
bundle exec rake
|
|
19
18
|
- name: Coveralls GitHub Action
|
|
20
|
-
uses: coverallsapp/github-action@
|
|
19
|
+
uses: coverallsapp/github-action@8d6379e14d29928660c4ba802d8e85393440b329 # v2.3.8
|
|
21
20
|
with:
|
|
22
21
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
@@ -5,7 +5,7 @@ jobs:
|
|
|
5
5
|
runs-on: ubuntu-latest
|
|
6
6
|
steps:
|
|
7
7
|
- name: Autoclose issues that did not follow issue template
|
|
8
|
-
uses: roots/issue-closer-action@v1.
|
|
8
|
+
uses: roots/issue-closer-action@fec85f2a845cd6cf33eda6e6a4c93026e71f86d1 # v1.2
|
|
9
9
|
with:
|
|
10
10
|
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
|
11
11
|
issue-close-message: "@${issue.user.login} this issue was automatically closed because it did not follow the issue template."
|
data/.github/workflows/linux.yml
CHANGED
|
@@ -1,27 +1,38 @@
|
|
|
1
1
|
name: Testing on Ubuntu
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- ubuntu-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
29
|
+
- uses: ruby/setup-ruby@95ef2b042f9d7a56d8268cba8559e2842e2ad01b # v1.321.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle config set path 'vendor/bundle'
|
|
26
37
|
bundle install --jobs 4 --retry 3
|
|
27
38
|
bundle exec rake test
|
data/.github/workflows/macos.yml
CHANGED
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
name: Testing on macOS
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- macOS-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
29
|
+
- uses: ruby/setup-ruby@95ef2b042f9d7a56d8268cba8559e2842e2ad01b # v1.321.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle install --jobs 4 --retry 3
|
|
26
37
|
bundle exec rake test
|
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
name: Testing on Windows
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- windows-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
29
|
+
- uses: ruby/setup-ruby@95ef2b042f9d7a56d8268cba8559e2842e2ad01b # v1.321.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle install --jobs 4 --retry 3
|
|
26
37
|
bundle exec rake test
|
data/History.md
CHANGED
|
@@ -2,6 +2,17 @@
|
|
|
2
2
|
|
|
3
3
|
### [Unreleased]
|
|
4
4
|
|
|
5
|
+
### 1.1.7
|
|
6
|
+
- test: remove duplicate keys from bulk response fixtures (#190)
|
|
7
|
+
- out\_opensearch: keep AWS credential providers refreshable to fix expired SigV4 tokens with assume\_role\_arn (#188)
|
|
8
|
+
- docs: document OpenSearch Serverless startup version-check workaround (#186)
|
|
9
|
+
- Allow duplicate keys in JSON parser (#185)
|
|
10
|
+
- Replace yajl-ruby with the json gem for JSON handling (#183)
|
|
11
|
+
|
|
12
|
+
### 1.1.6
|
|
13
|
+
- out\_opensearch\_data\_stream: filter failed items in data stream bulk error log (#175)
|
|
14
|
+
- out\_opensearch: fix default value of `refresh_credentials_interval` (#159)
|
|
15
|
+
|
|
5
16
|
### 1.1.5
|
|
6
17
|
- Fix bundle installation error on GitHub Action (#143)
|
|
7
18
|
- Prevent AWS credentials refresh from stopping on exception (#142)
|
data/README.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Fluent::Plugin::OpenSearch, a plugin for [Fluentd](http://fluentd.org)
|
|
2
2
|
|
|
3
|
-
[](https://badge.fury.io/rb/fluent-plugin-opensearch)
|
|
4
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/windows.yml)
|
|
5
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/macos.yml)
|
|
6
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/linux.yml)
|
|
7
7
|
[](https://coveralls.io/github/fluent/fluent-plugin-opensearch?branch=main)
|
|
8
8
|
|
|
9
9
|
Send your logs to OpenSearch (and search them with OpenSearch Dashboards maybe?)
|
|
@@ -902,7 +902,7 @@ compression_level best_compression
|
|
|
902
902
|
|
|
903
903
|
### prefer_oj_serializer
|
|
904
904
|
|
|
905
|
-
With default behavior, OpenSearch client uses `
|
|
905
|
+
With default behavior, OpenSearch client uses `JSON` as JSON encoder/decoder.
|
|
906
906
|
`Oj` is the alternative high performance JSON encoder/decoder.
|
|
907
907
|
When this parameter sets as `true`, OpenSearch client uses `Oj` as JSON encoder/decoder.
|
|
908
908
|
|
|
@@ -1568,7 +1568,7 @@ In this case, the endpoint configuration looks like:
|
|
|
1568
1568
|
|
|
1569
1569
|
### Expiring AWS credentials
|
|
1570
1570
|
|
|
1571
|
-
If you want to
|
|
1571
|
+
Credentials obtained from STS, IMDS or ECS are refreshed by the AWS SDK before they expire, so you usually do not need to change this parameter. If you want to recreate the credential provider forcibly in a certain interval, you should specify `refresh_credentials_interval` parameter under `endpoint` section:
|
|
1572
1572
|
|
|
1573
1573
|
```aconf
|
|
1574
1574
|
<endpoint>
|
|
@@ -1581,7 +1581,7 @@ If you want to expire AWS credentials in certain interval, you should specify `r
|
|
|
1581
1581
|
|
|
1582
1582
|
### Use OpenSearch Serverless
|
|
1583
1583
|
|
|
1584
|
-
If you want to use Serverless version of OpenSearch service, you have to specify `aoss` in `aws_service_name` under `endpoint` section:
|
|
1584
|
+
If you want to use the Serverless version of OpenSearch service, you have to specify `aoss` in `aws_service_name` under the `endpoint` section:
|
|
1585
1585
|
|
|
1586
1586
|
```aconf
|
|
1587
1587
|
<endpoint>
|
|
@@ -1592,6 +1592,39 @@ If you want to use Serverless version of OpenSearch service, you have to specify
|
|
|
1592
1592
|
</endpoint>
|
|
1593
1593
|
```
|
|
1594
1594
|
|
|
1595
|
+
**NOTE:** OpenSearch Serverless only supports a subset of the OpenSearch APIs and does **not** serve the root/info endpoint (`GET /`) that this plugin uses to detect the OpenSearch major version at startup. If you leave the default `verify_os_version_at_startup true`, the plugin fails the version probe with `404`, keeps retrying, and never starts sending data:
|
|
1596
|
+
|
|
1597
|
+
```
|
|
1598
|
+
Could not communicate to OpenSearch, resetting connection and trying again. [404]
|
|
1599
|
+
```
|
|
1600
|
+
|
|
1601
|
+
To avoid this, disable the startup version check and pin the version (Serverless is OpenSearch 2.x compatible):
|
|
1602
|
+
|
|
1603
|
+
```aconf
|
|
1604
|
+
<match es.**>
|
|
1605
|
+
@type opensearch
|
|
1606
|
+
logstash_format true
|
|
1607
|
+
include_tag_key true
|
|
1608
|
+
flush_interval 1s
|
|
1609
|
+
|
|
1610
|
+
# Serverless does not serve GET /, so skip the version probe.
|
|
1611
|
+
verify_os_version_at_startup false
|
|
1612
|
+
default_opensearch_version 2
|
|
1613
|
+
|
|
1614
|
+
<endpoint>
|
|
1615
|
+
url https://CLUSTER_ENDPOINT_URL
|
|
1616
|
+
region us-east-2
|
|
1617
|
+
aws_service_name aoss
|
|
1618
|
+
</endpoint>
|
|
1619
|
+
</match>
|
|
1620
|
+
```
|
|
1621
|
+
|
|
1622
|
+
**Other Serverless limitations to keep in mind:**
|
|
1623
|
+
|
|
1624
|
+
- Your IAM identity needs a **data access policy** on the collection (at least `aoss:CreateIndex` and `aoss:WriteDocument`). An IAM permission policy alone is not sufficient.
|
|
1625
|
+
- For `TIMESERIES` collections, custom document IDs are not supported. Do not set `id_key`, and keep the default `write_operation index`, since `create` / `update` / `upsert` require an `_id`.
|
|
1626
|
+
- Index templates, rollover/ILM, and data streams rely on APIs that Serverless does not expose, so template-related options and `@type opensearch_data_stream` are not usable against a Serverless collection.
|
|
1627
|
+
|
|
1595
1628
|
## Troubleshooting
|
|
1596
1629
|
|
|
1597
1630
|
See [Troubleshooting document](README.Troubleshooting.md)
|
|
@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
|
|
|
3
3
|
|
|
4
4
|
Gem::Specification.new do |s|
|
|
5
5
|
s.name = 'fluent-plugin-opensearch'
|
|
6
|
-
s.version = '1.1.
|
|
6
|
+
s.version = '1.1.7'
|
|
7
7
|
s.authors = ['Hiroshi Hatake']
|
|
8
8
|
s.email = ['cosmo0920.wp@gmail.com']
|
|
9
9
|
s.description = %q{Opensearch output plugin for Fluent event collector}
|
|
@@ -31,7 +31,7 @@ Gem::Specification.new do |s|
|
|
|
31
31
|
s.add_runtime_dependency "faraday_middleware-aws-sigv4", "~> 1.0.1"
|
|
32
32
|
|
|
33
33
|
s.add_development_dependency 'rake', '>= 0'
|
|
34
|
-
s.add_development_dependency 'webrick', '~> 1.
|
|
34
|
+
s.add_development_dependency 'webrick', '~> 1.9'
|
|
35
35
|
s.add_development_dependency 'webmock', '~> 3.18.1'
|
|
36
36
|
s.add_development_dependency 'test-unit', '~> 3.3.0'
|
|
37
37
|
s.add_development_dependency 'minitest', '~> 5.8'
|
|
@@ -374,7 +374,7 @@ module Fluent::Plugin
|
|
|
374
374
|
def run_slice(slice_id=nil)
|
|
375
375
|
slice_query = @base_query
|
|
376
376
|
slice_query = slice_query.merge('slice' => { 'id' => slice_id, 'max' => @num_slices}) unless slice_id.nil?
|
|
377
|
-
result = client.search(@options.merge(:body =>
|
|
377
|
+
result = client.search(@options.merge(:body => JSON.generate(slice_query) ))
|
|
378
378
|
es = Fluent::MultiEventStream.new
|
|
379
379
|
|
|
380
380
|
result["hits"]["hits"].each {|hit| process_events(hit, es)}
|
|
@@ -33,7 +33,7 @@ module Fluent::OpenSearchIndexTemplate
|
|
|
33
33
|
raise "If you specify a template_name you must specify a valid template file (checked '#{template_file}')!"
|
|
34
34
|
end
|
|
35
35
|
file_contents = IO.read(template_file).gsub(/\n/,'')
|
|
36
|
-
JSON.parse(file_contents)
|
|
36
|
+
JSON.parse(file_contents, allow_duplicate_key: true)
|
|
37
37
|
end
|
|
38
38
|
|
|
39
39
|
def get_custom_template(template_file, customize_template)
|
|
@@ -44,7 +44,7 @@ module Fluent::OpenSearchIndexTemplate
|
|
|
44
44
|
customize_template.each do |key, value|
|
|
45
45
|
file_contents = file_contents.gsub(key,value.downcase)
|
|
46
46
|
end
|
|
47
|
-
JSON.parse(file_contents)
|
|
47
|
+
JSON.parse(file_contents, allow_duplicate_key: true)
|
|
48
48
|
end
|
|
49
49
|
|
|
50
50
|
def template_exists?(name, host = nil)
|
|
@@ -195,7 +195,7 @@ module Fluent::Plugin
|
|
|
195
195
|
config_param :assume_role_session_name, :string, :default => "fluentd"
|
|
196
196
|
config_param :assume_role_web_identity_token_file, :string, :default => nil
|
|
197
197
|
config_param :sts_credentials_region, :string, :default => nil
|
|
198
|
-
config_param :refresh_credentials_interval, :time, :default =>
|
|
198
|
+
config_param :refresh_credentials_interval, :time, :default => 18000 # 5 hours
|
|
199
199
|
config_param :aws_service_name, :enum, list: [:es, :aoss], :default => :es
|
|
200
200
|
end
|
|
201
201
|
|
|
@@ -225,13 +225,13 @@ module Fluent::Plugin
|
|
|
225
225
|
if conf[:assume_role_arn].nil?
|
|
226
226
|
aws_container_credentials_relative_uri = conf[:ecs_container_credentials_relative_uri] || ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
|
|
227
227
|
if aws_container_credentials_relative_uri.nil?
|
|
228
|
-
credentials = Aws::SharedCredentials.new
|
|
229
|
-
credentials ||= Aws::InstanceProfileCredentials.new
|
|
230
|
-
credentials ||= Aws::ECSCredentials.new
|
|
228
|
+
credentials = usable_credentials_provider { Aws::SharedCredentials.new }
|
|
229
|
+
credentials ||= usable_credentials_provider { Aws::InstanceProfileCredentials.new }
|
|
230
|
+
credentials ||= usable_credentials_provider { Aws::ECSCredentials.new }
|
|
231
231
|
else
|
|
232
232
|
credentials = Aws::ECSCredentials.new({
|
|
233
233
|
credential_path: aws_container_credentials_relative_uri
|
|
234
|
-
})
|
|
234
|
+
})
|
|
235
235
|
end
|
|
236
236
|
else
|
|
237
237
|
if conf[:assume_role_web_identity_token_file].nil?
|
|
@@ -239,21 +239,28 @@ module Fluent::Plugin
|
|
|
239
239
|
role_arn: conf[:assume_role_arn],
|
|
240
240
|
role_session_name: conf[:assume_role_session_name],
|
|
241
241
|
region: sts_creds_region(conf)
|
|
242
|
-
})
|
|
242
|
+
})
|
|
243
243
|
else
|
|
244
244
|
credentials = Aws::AssumeRoleWebIdentityCredentials.new({
|
|
245
245
|
role_arn: conf[:assume_role_arn],
|
|
246
246
|
web_identity_token_file: conf[:assume_role_web_identity_token_file],
|
|
247
247
|
region: sts_creds_region(conf)
|
|
248
|
-
})
|
|
248
|
+
})
|
|
249
249
|
end
|
|
250
250
|
end
|
|
251
251
|
end
|
|
252
|
-
raise "No valid AWS credentials found." unless credentials
|
|
252
|
+
raise "No valid AWS credentials found." unless credentials&.credentials&.set?
|
|
253
253
|
|
|
254
254
|
credentials
|
|
255
255
|
end
|
|
256
256
|
|
|
257
|
+
def usable_credentials_provider
|
|
258
|
+
provider = yield
|
|
259
|
+
provider if provider.credentials&.set?
|
|
260
|
+
rescue
|
|
261
|
+
nil
|
|
262
|
+
end
|
|
263
|
+
|
|
257
264
|
def sts_creds_region(conf)
|
|
258
265
|
conf[:sts_credentials_region] || conf[:region]
|
|
259
266
|
end
|
|
@@ -339,7 +346,7 @@ module Fluent::Plugin
|
|
|
339
346
|
}
|
|
340
347
|
end
|
|
341
348
|
end
|
|
342
|
-
#
|
|
349
|
+
# Credential providers refresh themselves near expiration. This timer recreates them forcibly.
|
|
343
350
|
@credential_mutex = Mutex.new
|
|
344
351
|
if @endpoint
|
|
345
352
|
@_aws_credentials = aws_credentials(@endpoint)
|
|
@@ -369,7 +376,7 @@ module Fluent::Plugin
|
|
|
369
376
|
OpenSearch::API.settings[:serializer] = Fluent::Plugin::Serializer::Oj
|
|
370
377
|
end
|
|
371
378
|
rescue LoadError
|
|
372
|
-
@dump_proc =
|
|
379
|
+
@dump_proc = JSON.method(:generate)
|
|
373
380
|
end
|
|
374
381
|
|
|
375
382
|
raise Fluent::ConfigError, "`password` must be present if `user` is present" if @user && @password.nil?
|
|
@@ -630,7 +637,7 @@ module Fluent::Plugin
|
|
|
630
637
|
:aws_sigv4,
|
|
631
638
|
service: @endpoint.aws_service_name.to_s,
|
|
632
639
|
region: @endpoint.region,
|
|
633
|
-
|
|
640
|
+
credentials_provider: @_aws_credentials,
|
|
634
641
|
)
|
|
635
642
|
|
|
636
643
|
f.adapter @http_backend, @backend_options
|
|
@@ -940,7 +947,7 @@ module Fluent::Plugin
|
|
|
940
947
|
{"_index" => {"order" => "desc"}}
|
|
941
948
|
]
|
|
942
949
|
}
|
|
943
|
-
result = client.search(options.merge(:body =>
|
|
950
|
+
result = client.search(options.merge(:body => JSON.generate(query)))
|
|
944
951
|
# There should be just one hit per _id, but in case there still is multiple, just the oldest index is stored to map
|
|
945
952
|
result['hits']['hits'].each do |hit|
|
|
946
953
|
indices[hit["_id"]] = hit["_index"]
|
|
@@ -174,6 +174,7 @@ module Fluent::Plugin
|
|
|
174
174
|
CREATE_OP => {}
|
|
175
175
|
}
|
|
176
176
|
tag = chunk.metadata.tag
|
|
177
|
+
records = []
|
|
177
178
|
chunk.msgpack_each do |time, record|
|
|
178
179
|
next unless record.is_a? Hash
|
|
179
180
|
begin
|
|
@@ -194,6 +195,7 @@ module Fluent::Plugin
|
|
|
194
195
|
@remove_keys.each { |key| record.delete(key) }
|
|
195
196
|
end
|
|
196
197
|
bulk_message = append_record_to_messages(CREATE_OP, {}, headers, record, bulk_message)
|
|
198
|
+
records << [time, record]
|
|
197
199
|
rescue => e
|
|
198
200
|
emit_error_label_event do
|
|
199
201
|
router.emit_error_event(tag, time, record, e)
|
|
@@ -210,7 +212,18 @@ module Fluent::Plugin
|
|
|
210
212
|
begin
|
|
211
213
|
response = client(host).bulk(params)
|
|
212
214
|
if response['errors']
|
|
213
|
-
|
|
215
|
+
failed_count = response['items'].count { |item| item.values.first['status'] >= 300 }
|
|
216
|
+
log.error "Could not bulk insert to Data Stream: #{data_stream_name}, #{failed_count} item(s) failed."
|
|
217
|
+
if emit_error_label_event?
|
|
218
|
+
response['items'].each_with_index do |item, idx|
|
|
219
|
+
status = item.values.first['status']
|
|
220
|
+
next unless status >= 300
|
|
221
|
+
time, record = records[idx]
|
|
222
|
+
next if record.nil?
|
|
223
|
+
error_info = item.values.first['error']
|
|
224
|
+
router.emit_error_event(tag, time, record, RuntimeError.new("status #{status}: #{error_info}"))
|
|
225
|
+
end
|
|
226
|
+
end
|
|
214
227
|
end
|
|
215
228
|
rescue => e
|
|
216
229
|
raise RecoverableRequestFailure, "could not push logs to OpenSearch cluster (#{data_stream_name}): #{e.message}"
|
|
@@ -229,3 +242,4 @@ module Fluent::Plugin
|
|
|
229
242
|
end
|
|
230
243
|
end
|
|
231
244
|
end
|
|
245
|
+
|
|
@@ -85,6 +85,27 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
85
85
|
}
|
|
86
86
|
end
|
|
87
87
|
|
|
88
|
+
def chained_credentials_conf
|
|
89
|
+
{
|
|
90
|
+
access_key_id: '',
|
|
91
|
+
secret_access_key: '',
|
|
92
|
+
assume_role_arn: nil,
|
|
93
|
+
ecs_container_credentials_relative_uri: nil,
|
|
94
|
+
}
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def with_stubbed_aws_responses
|
|
98
|
+
saved = Aws.config.key?(:stub_responses) ? Aws.config[:stub_responses] : :__unset__
|
|
99
|
+
Aws.config[:stub_responses] = true
|
|
100
|
+
yield
|
|
101
|
+
ensure
|
|
102
|
+
if saved == :__unset__
|
|
103
|
+
Aws.config.delete(:stub_responses)
|
|
104
|
+
else
|
|
105
|
+
Aws.config[:stub_responses] = saved
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
|
|
88
109
|
def stub_opensearch_info(url="http://localhost:9200/", version="1.2.2")
|
|
89
110
|
body ="{\"version\":{\"number\":\"#{version}\", \"distribution\":\"opensearch\"},\"tagline\":\"The OpenSearch Project: https://opensearch.org/\"}"
|
|
90
111
|
stub_request(:get, url).to_return({:status => 200, :body => body, :headers => { 'Content-Type' => 'json' } })
|
|
@@ -315,9 +336,106 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
315
336
|
assert_equal "fluentd", instance.endpoint.assume_role_session_name
|
|
316
337
|
assert_nil instance.endpoint.assume_role_web_identity_token_file
|
|
317
338
|
assert_nil instance.endpoint.sts_credentials_region
|
|
339
|
+
assert_equal 18000, instance.endpoint.refresh_credentials_interval
|
|
318
340
|
assert_equal :es, instance.endpoint.aws_service_name
|
|
319
341
|
end
|
|
320
342
|
|
|
343
|
+
test 'aws_credentials returns credential provider with access key' do
|
|
344
|
+
config = Fluent::Config::Element.new(
|
|
345
|
+
'ROOT', '', {
|
|
346
|
+
'@type' => 'opensearch',
|
|
347
|
+
}, [
|
|
348
|
+
Fluent::Config::Element.new('endpoint', '', {
|
|
349
|
+
'url' => "https://search-opensearch.aws.example.com/",
|
|
350
|
+
'region' => "local",
|
|
351
|
+
'access_key_id' => 'YOUR_AWESOME_KEY',
|
|
352
|
+
'secret_access_key' => 'YOUR_AWESOME_SECRET',
|
|
353
|
+
}, []),
|
|
354
|
+
Fluent::Config::Element.new('buffer', 'tag', {}, [])
|
|
355
|
+
|
|
356
|
+
])
|
|
357
|
+
instance = driver(config).instance
|
|
358
|
+
credentials = instance.aws_credentials(instance.endpoint)
|
|
359
|
+
|
|
360
|
+
assert_instance_of Aws::Credentials, credentials
|
|
361
|
+
assert_equal 'YOUR_AWESOME_KEY', credentials.credentials.access_key_id
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
test 'aws_credentials returns refreshable credential provider with assume_role_arn' do
|
|
365
|
+
with_stubbed_aws_responses do
|
|
366
|
+
config = Fluent::Config::Element.new(
|
|
367
|
+
'ROOT', '', {
|
|
368
|
+
'@type' => 'opensearch',
|
|
369
|
+
}, [
|
|
370
|
+
Fluent::Config::Element.new('endpoint', '', {
|
|
371
|
+
'url' => "https://search-opensearch.aws.example.com/",
|
|
372
|
+
'region' => "local",
|
|
373
|
+
'assume_role_arn' => 'arn:aws:iam::123456789012:role/some-role',
|
|
374
|
+
}, []),
|
|
375
|
+
Fluent::Config::Element.new('buffer', 'tag', {}, [])
|
|
376
|
+
|
|
377
|
+
])
|
|
378
|
+
instance = driver(config).instance
|
|
379
|
+
credentials = instance.aws_credentials(instance.endpoint)
|
|
380
|
+
|
|
381
|
+
assert_instance_of Aws::AssumeRoleCredentials, credentials
|
|
382
|
+
assert_kind_of Aws::RefreshingCredentials, credentials
|
|
383
|
+
assert_true credentials.credentials.set?
|
|
384
|
+
end
|
|
385
|
+
end
|
|
386
|
+
|
|
387
|
+
test 'aws_credentials falls back to the next provider when a provider has no credentials' do
|
|
388
|
+
unset_provider = flexmock('unset provider', credentials: Aws::Credentials.new(nil, nil))
|
|
389
|
+
ecs_provider = flexmock('ecs provider', credentials: Aws::Credentials.new('ECS_KEY', 'ECS_SECRET'))
|
|
390
|
+
flexmock(Aws::SharedCredentials).should_receive(:new).and_return(unset_provider)
|
|
391
|
+
flexmock(Aws::InstanceProfileCredentials).should_receive(:new).and_return(unset_provider)
|
|
392
|
+
flexmock(Aws::ECSCredentials).should_receive(:new).and_return(ecs_provider)
|
|
393
|
+
instance = driver.instance
|
|
394
|
+
|
|
395
|
+
assert_same ecs_provider, instance.aws_credentials(chained_credentials_conf)
|
|
396
|
+
end
|
|
397
|
+
|
|
398
|
+
test 'aws_credentials raises when no provider has usable credentials' do
|
|
399
|
+
unset_provider = flexmock('unset provider', credentials: Aws::Credentials.new(nil, nil))
|
|
400
|
+
flexmock(Aws::SharedCredentials).should_receive(:new).and_return(unset_provider)
|
|
401
|
+
flexmock(Aws::InstanceProfileCredentials).should_receive(:new).and_return(unset_provider)
|
|
402
|
+
flexmock(Aws::ECSCredentials).should_receive(:new)
|
|
403
|
+
.and_raise(ArgumentError.new("Cannot instantiate an ECS Credential Provider without a credential path or endpoint."))
|
|
404
|
+
instance = driver.instance
|
|
405
|
+
|
|
406
|
+
assert_raise(RuntimeError.new("No valid AWS credentials found.")) do
|
|
407
|
+
instance.aws_credentials(chained_credentials_conf)
|
|
408
|
+
end
|
|
409
|
+
end
|
|
410
|
+
|
|
411
|
+
test 'credentials provider is passed to aws_sigv4 middleware' do
|
|
412
|
+
with_stubbed_aws_responses do
|
|
413
|
+
config = Fluent::Config::Element.new(
|
|
414
|
+
'ROOT', '', {
|
|
415
|
+
'@type' => 'opensearch',
|
|
416
|
+
}, [
|
|
417
|
+
Fluent::Config::Element.new('endpoint', '', {
|
|
418
|
+
'url' => "https://search-opensearch.aws.example.com/",
|
|
419
|
+
'region' => "local",
|
|
420
|
+
'assume_role_arn' => 'arn:aws:iam::123456789012:role/some-role',
|
|
421
|
+
}, []),
|
|
422
|
+
Fluent::Config::Element.new('buffer', 'tag', {}, [])
|
|
423
|
+
|
|
424
|
+
])
|
|
425
|
+
instance = driver(config).instance
|
|
426
|
+
# Reaching into Faraday internals. If this breaks after a Faraday or
|
|
427
|
+
# faraday_middleware-aws-sigv4 update, check RackBuilder::Handler.
|
|
428
|
+
handler = instance.client.transport.transport.connections.first.connection.builder.handlers.find { |h|
|
|
429
|
+
h.klass == FaradayMiddleware::AwsSigV4
|
|
430
|
+
}
|
|
431
|
+
|
|
432
|
+
assert_not_nil handler
|
|
433
|
+
signer = handler.build(proc {}).instance_variable_get(:@signer)
|
|
434
|
+
assert_instance_of Aws::AssumeRoleCredentials, signer.credentials_provider
|
|
435
|
+
assert_same instance.instance_variable_get(:@_aws_credentials), signer.credentials_provider
|
|
436
|
+
end
|
|
437
|
+
end
|
|
438
|
+
|
|
321
439
|
data("OpenSearch Service" => [:es, 'es'],
|
|
322
440
|
"OpenSearch Serverless" => [:aoss, 'aoss'])
|
|
323
441
|
test 'configure endpoint section w/ aws_service_name' do |data|
|
|
@@ -2755,7 +2873,8 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
2755
2873
|
driver.feed(time.to_i, sample_record.merge({"pipeline_id" => pipeline_id}))
|
|
2756
2874
|
end
|
|
2757
2875
|
}
|
|
2758
|
-
|
|
2876
|
+
connection_spec = {host: "myhost-1", port: 9200, scheme: "http"}
|
|
2877
|
+
assert_equal("could not push logs to OpenSearch cluster (#{connection_spec.inspect}): [503] ", exception.message)
|
|
2759
2878
|
end
|
|
2760
2879
|
end
|
|
2761
2880
|
|
|
@@ -3647,7 +3766,6 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
3647
3766
|
"_index" : "foo",
|
|
3648
3767
|
"_type" : "bar",
|
|
3649
3768
|
"_id" : "abc",
|
|
3650
|
-
"_id" : "abc",
|
|
3651
3769
|
"status" : 409
|
|
3652
3770
|
}
|
|
3653
3771
|
}
|
|
@@ -743,4 +743,131 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
|
|
|
743
743
|
assert(!index_cmds[1].has_key?('remove_me'))
|
|
744
744
|
end
|
|
745
745
|
|
|
746
|
+
def test_bulk_error_logs_only_failed_items
|
|
747
|
+
stub_default
|
|
748
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
749
|
+
status: 200,
|
|
750
|
+
body: {
|
|
751
|
+
'errors' => true,
|
|
752
|
+
'items' => [
|
|
753
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
754
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
755
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
756
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
757
|
+
]
|
|
758
|
+
}.to_json,
|
|
759
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
760
|
+
)
|
|
761
|
+
conf = config_element(
|
|
762
|
+
'ROOT', '', {
|
|
763
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
764
|
+
'data_stream_name' => 'foo',
|
|
765
|
+
'data_stream_template_name' => 'foo_tpl'
|
|
766
|
+
})
|
|
767
|
+
driver(conf).run(default_tag: 'test') do
|
|
768
|
+
driver.feed(sample_record)
|
|
769
|
+
end
|
|
770
|
+
error_log = driver.logs.find { |l| l.include?("Could not bulk insert") }
|
|
771
|
+
assert_not_nil error_log, "Expected an error log entry for bulk insert failure"
|
|
772
|
+
assert_match(/1 item\(s\) failed/, error_log)
|
|
773
|
+
assert_no_match(/201/, error_log)
|
|
774
|
+
end
|
|
775
|
+
|
|
776
|
+
def test_bulk_error_logs_5xx_items
|
|
777
|
+
stub_default
|
|
778
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
779
|
+
status: 200,
|
|
780
|
+
body: {
|
|
781
|
+
'errors' => true,
|
|
782
|
+
'items' => [
|
|
783
|
+
{ 'create' => { 'status' => 200, '_index' => 'foo' } },
|
|
784
|
+
{ 'create' => { 'status' => 500, '_index' => 'foo',
|
|
785
|
+
'error' => { 'type' => 'internal_server_error' } } }
|
|
786
|
+
]
|
|
787
|
+
}.to_json,
|
|
788
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
789
|
+
)
|
|
790
|
+
conf = config_element(
|
|
791
|
+
'ROOT', '', {
|
|
792
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
793
|
+
'data_stream_name' => 'foo',
|
|
794
|
+
'data_stream_template_name' => 'foo_tpl'
|
|
795
|
+
})
|
|
796
|
+
driver(conf).run(default_tag: 'test') do
|
|
797
|
+
driver.feed(sample_record)
|
|
798
|
+
end
|
|
799
|
+
error_log = driver.logs.find { |l| l.include?("Could not bulk insert") }
|
|
800
|
+
assert_not_nil error_log, "Expected an error log entry for bulk insert failure"
|
|
801
|
+
assert_match(/1 item\(s\) failed/, error_log)
|
|
802
|
+
assert_no_match(/200/, error_log)
|
|
803
|
+
end
|
|
804
|
+
|
|
805
|
+
def test_bulk_error_emits_failed_records_to_error_label
|
|
806
|
+
stub_default
|
|
807
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
808
|
+
status: 200,
|
|
809
|
+
body: {
|
|
810
|
+
'errors' => true,
|
|
811
|
+
'items' => [
|
|
812
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
813
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
814
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
815
|
+
]
|
|
816
|
+
}.to_json,
|
|
817
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
818
|
+
)
|
|
819
|
+
conf = config_element(
|
|
820
|
+
'ROOT', '', {
|
|
821
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
822
|
+
'data_stream_name' => 'foo',
|
|
823
|
+
'data_stream_template_name' => 'foo_tpl',
|
|
824
|
+
'emit_error_label_event' => 'true'
|
|
825
|
+
})
|
|
826
|
+
error_events = []
|
|
827
|
+
flexmock(Fluent::Engine).should_receive(:emit_error_event).and_return { |tag, time, record, e|
|
|
828
|
+
error_events << { tag: tag, record: record, error: e }
|
|
829
|
+
}
|
|
830
|
+
d = driver(conf)
|
|
831
|
+
flexmock(d.instance.router).should_receive(:emit_error_event).and_return { |tag, time, record, e|
|
|
832
|
+
error_events << { tag: tag, record: record, error: e }
|
|
833
|
+
}
|
|
834
|
+
d.run(default_tag: 'test') do
|
|
835
|
+
d.feed(event_time, sample_record)
|
|
836
|
+
d.feed(event_time, sample_record)
|
|
837
|
+
end
|
|
838
|
+
assert_equal 1, error_events.size, "Expected exactly one record routed to @ERROR"
|
|
839
|
+
assert_match(/400/, error_events.first[:error].message)
|
|
840
|
+
end
|
|
841
|
+
|
|
842
|
+
def test_bulk_error_does_not_emit_error_events_when_disabled
|
|
843
|
+
stub_default
|
|
844
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
845
|
+
status: 200,
|
|
846
|
+
body: {
|
|
847
|
+
'errors' => true,
|
|
848
|
+
'items' => [
|
|
849
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
850
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
851
|
+
]
|
|
852
|
+
}.to_json,
|
|
853
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
854
|
+
)
|
|
855
|
+
conf = config_element(
|
|
856
|
+
'ROOT', '', {
|
|
857
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
858
|
+
'data_stream_name' => 'foo',
|
|
859
|
+
'data_stream_template_name' => 'foo_tpl',
|
|
860
|
+
'emit_error_label_event' => 'false'
|
|
861
|
+
})
|
|
862
|
+
error_events = []
|
|
863
|
+
d = driver(conf)
|
|
864
|
+
flexmock(d.instance.router).should_receive(:emit_error_event).never
|
|
865
|
+
d.run(default_tag: 'test') do
|
|
866
|
+
d.feed(event_time, sample_record)
|
|
867
|
+
end
|
|
868
|
+
error_log = d.logs.find { |l| l.include?("Could not bulk insert") }
|
|
869
|
+
assert_not_nil error_log, "Expected summary error log even when emit_error_label_event is false"
|
|
870
|
+
end
|
|
871
|
+
|
|
746
872
|
end
|
|
873
|
+
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-opensearch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.1.
|
|
4
|
+
version: 1.1.7
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Hiroshi Hatake
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: fluentd
|
|
@@ -128,14 +127,14 @@ dependencies:
|
|
|
128
127
|
requirements:
|
|
129
128
|
- - "~>"
|
|
130
129
|
- !ruby/object:Gem::Version
|
|
131
|
-
version: 1.
|
|
130
|
+
version: '1.9'
|
|
132
131
|
type: :development
|
|
133
132
|
prerelease: false
|
|
134
133
|
version_requirements: !ruby/object:Gem::Requirement
|
|
135
134
|
requirements:
|
|
136
135
|
- - "~>"
|
|
137
136
|
- !ruby/object:Gem::Version
|
|
138
|
-
version: 1.
|
|
137
|
+
version: '1.9'
|
|
139
138
|
- !ruby/object:Gem::Dependency
|
|
140
139
|
name: webmock
|
|
141
140
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -201,9 +200,12 @@ extra_rdoc_files: []
|
|
|
201
200
|
files:
|
|
202
201
|
- ".coveralls.yml"
|
|
203
202
|
- ".editorconfig"
|
|
204
|
-
- ".github/ISSUE_TEMPLATE/bug_report.
|
|
205
|
-
- ".github/ISSUE_TEMPLATE/
|
|
203
|
+
- ".github/ISSUE_TEMPLATE/bug_report.yaml"
|
|
204
|
+
- ".github/ISSUE_TEMPLATE/config.yml"
|
|
205
|
+
- ".github/ISSUE_TEMPLATE/feature_request.yaml"
|
|
206
206
|
- ".github/PULL_REQUEST_TEMPLATE/pull_request_template.md"
|
|
207
|
+
- ".github/dependabot.yml"
|
|
208
|
+
- ".github/workflows/add-to-project.yml"
|
|
207
209
|
- ".github/workflows/coverage.yaml"
|
|
208
210
|
- ".github/workflows/issue-auto-closer.yml"
|
|
209
211
|
- ".github/workflows/linux.yml"
|
|
@@ -254,7 +256,6 @@ licenses:
|
|
|
254
256
|
- Apache-2.0
|
|
255
257
|
metadata:
|
|
256
258
|
changelog_uri: https://github.com/fluent/fluent-plugin-opensearch/blob/master/History.md
|
|
257
|
-
post_install_message:
|
|
258
259
|
rdoc_options: []
|
|
259
260
|
require_paths:
|
|
260
261
|
- lib
|
|
@@ -269,8 +270,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
269
270
|
- !ruby/object:Gem::Version
|
|
270
271
|
version: '0'
|
|
271
272
|
requirements: []
|
|
272
|
-
rubygems_version:
|
|
273
|
-
signing_key:
|
|
273
|
+
rubygems_version: 4.0.18
|
|
274
274
|
specification_version: 4
|
|
275
275
|
summary: Opensearch output plugin for Fluent event collector
|
|
276
276
|
test_files:
|
|
@@ -1,29 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Bug Report
|
|
3
|
-
about: Create a report to help us improve. If you have questions about OpenSearch plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
(check apply)
|
|
8
|
-
- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
|
|
9
|
-
- [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
|
|
10
|
-
|
|
11
|
-
#### Steps to replicate
|
|
12
|
-
|
|
13
|
-
Provide example config and message
|
|
14
|
-
|
|
15
|
-
#### Expected Behavior or What you need to ask
|
|
16
|
-
|
|
17
|
-
...
|
|
18
|
-
|
|
19
|
-
#### Using Fluentd and OpenSearch plugin versions
|
|
20
|
-
|
|
21
|
-
* OS version
|
|
22
|
-
* Bare Metal or within Docker or Kubernetes or others?
|
|
23
|
-
* Fluentd v1.0 or later
|
|
24
|
-
* paste result of ``fluentd --version`` or ``td-agent --version``
|
|
25
|
-
* OpenSearch plugin version
|
|
26
|
-
* paste boot log of fluentd or td-agent
|
|
27
|
-
* paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
|
|
28
|
-
* OpenSearch version (optional)
|
|
29
|
-
* OpenSearch template(s) (optional)
|
|
@@ -1,24 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Feature request
|
|
3
|
-
about: Suggest an idea for this project
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
(check apply)
|
|
8
|
-
- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
|
|
9
|
-
|
|
10
|
-
**Is your feature request related to a problem? Please describe.**
|
|
11
|
-
|
|
12
|
-
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
|
|
13
|
-
|
|
14
|
-
**Describe the solution you'd like**
|
|
15
|
-
|
|
16
|
-
<!-- A clear and concise description of what you want to happen. -->
|
|
17
|
-
|
|
18
|
-
**Describe alternatives you've considered**
|
|
19
|
-
|
|
20
|
-
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
|
|
21
|
-
|
|
22
|
-
**Additional context**
|
|
23
|
-
|
|
24
|
-
<!-- Add any other context or screenshots about the feature request here. -->
|