fluent-plugin-opensearch 1.1.5 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.yaml +73 -0
- data/.github/ISSUE_TEMPLATE/config.yml +5 -0
- data/.github/ISSUE_TEMPLATE/feature_request.yaml +38 -0
- data/.github/dependabot.yml +6 -0
- data/.github/workflows/add-to-project.yml +24 -0
- data/.github/workflows/coverage.yaml +3 -4
- data/.github/workflows/issue-auto-closer.yml +1 -1
- data/.github/workflows/linux.yml +17 -6
- data/.github/workflows/macos.yml +17 -6
- data/.github/workflows/windows.yml +17 -6
- data/History.md +4 -0
- data/README.md +4 -4
- data/fluent-plugin-opensearch.gemspec +2 -2
- data/lib/fluent/plugin/out_opensearch.rb +1 -1
- data/lib/fluent/plugin/out_opensearch_data_stream.rb +15 -1
- data/test/plugin/test_out_opensearch.rb +2 -1
- data/test/plugin/test_out_opensearch_data_stream.rb +127 -0
- metadata +10 -10
- data/.github/ISSUE_TEMPLATE/bug_report.md +0 -29
- data/.github/ISSUE_TEMPLATE/feature_request.md +0 -24
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 26499682224f8d7dc4a9b204f99172b512057fb5c9b67d03b0bf1637d39f4224
|
|
4
|
+
data.tar.gz: b6bc5ff55815d050f50a9c8658e98e0dbfce5511ade14342fa26d31c4dba59d0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a0ffa39d8a11088809008e5d74997d1765ca624153890bf0699ea2ab9f5522d80b2226427291bee42b04abf370636abd8a7dc5d8a13af1baf6cc4089e345da91
|
|
7
|
+
data.tar.gz: eb8983700b0a7af66010a5e8ec5f2f5ad5a30532fd877c7ab547aabfef015b45b012b83069c9775bb0965ebb95e01da8b75a8eb303d481ec768d076673679a3f
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
name: Bug Report
|
|
2
|
+
description: Create a report with a procedure for reproducing the bug
|
|
3
|
+
labels: "waiting-for-triage"
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Please check the [README](https://github.com/fluent/fluent-plugin-opensearch/blob/main/README.md) first. To help us investigate the problem, please provide the following information.
|
|
9
|
+
- type: textarea
|
|
10
|
+
id: description
|
|
11
|
+
attributes:
|
|
12
|
+
label: Describe the bug
|
|
13
|
+
description: A clear and concise description of what the bug is
|
|
14
|
+
validations:
|
|
15
|
+
required: true
|
|
16
|
+
- type: textarea
|
|
17
|
+
id: reproduce
|
|
18
|
+
attributes:
|
|
19
|
+
label: To Reproduce
|
|
20
|
+
description: Steps to reproduce the behavior
|
|
21
|
+
validations:
|
|
22
|
+
required: true
|
|
23
|
+
- type: textarea
|
|
24
|
+
id: expected
|
|
25
|
+
attributes:
|
|
26
|
+
label: Expected behavior
|
|
27
|
+
description: A clear and concise description of what you expected to happen
|
|
28
|
+
validations:
|
|
29
|
+
required: true
|
|
30
|
+
- type: textarea
|
|
31
|
+
id: environment
|
|
32
|
+
attributes:
|
|
33
|
+
label: Your Environment
|
|
34
|
+
description: |
|
|
35
|
+
- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
|
|
36
|
+
- Operating system: `cat /etc/os-release`
|
|
37
|
+
- Kernel version: `uname -r`
|
|
38
|
+
|
|
39
|
+
Tip: If you hit the problem with older fluentd version, try latest version first.
|
|
40
|
+
value: |
|
|
41
|
+
- Fluentd version:
|
|
42
|
+
- TD Agent version:
|
|
43
|
+
- fluent-plugin-opensearch version:
|
|
44
|
+
- OpenSearch version (optional)
|
|
45
|
+
- OpenSearch template(s) (optional)
|
|
46
|
+
- Operating system:
|
|
47
|
+
- Kernel version:
|
|
48
|
+
render: markdown
|
|
49
|
+
validations:
|
|
50
|
+
required: true
|
|
51
|
+
- type: textarea
|
|
52
|
+
id: configuration
|
|
53
|
+
attributes:
|
|
54
|
+
label: Your Configuration
|
|
55
|
+
description: |
|
|
56
|
+
Write your configuration here. Minimum reproducible fluentd.conf is recommended.
|
|
57
|
+
validations:
|
|
58
|
+
required: true
|
|
59
|
+
- type: textarea
|
|
60
|
+
id: logs
|
|
61
|
+
attributes:
|
|
62
|
+
label: Your Error Log
|
|
63
|
+
description: Write your ALL error log here
|
|
64
|
+
render: shell
|
|
65
|
+
validations:
|
|
66
|
+
required: true
|
|
67
|
+
- type: textarea
|
|
68
|
+
id: addtional-context
|
|
69
|
+
attributes:
|
|
70
|
+
label: Additional context
|
|
71
|
+
description: Add any other context about the problem here.
|
|
72
|
+
validations:
|
|
73
|
+
required: false
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
name: Feature request
|
|
2
|
+
description: Suggest an idea for this project
|
|
3
|
+
labels: "waiting-for-triage"
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Please check the [README](https://github.com/fluent/fluent-plugin-opensearch/blob/main/README.md) first. To help us understand the request, please provide the following information.
|
|
9
|
+
- type: textarea
|
|
10
|
+
id: description
|
|
11
|
+
attributes:
|
|
12
|
+
label: Is your feature request related to a problem? Please describe.
|
|
13
|
+
description: |
|
|
14
|
+
A clear and concise description of what the problem is.
|
|
15
|
+
Ex. I'm always frustrated when [...]
|
|
16
|
+
validations:
|
|
17
|
+
required: true
|
|
18
|
+
- type: textarea
|
|
19
|
+
id: solution
|
|
20
|
+
attributes:
|
|
21
|
+
label: Describe the solution you'd like
|
|
22
|
+
description: A clear and concise description of what you want to happen.
|
|
23
|
+
validations:
|
|
24
|
+
required: true
|
|
25
|
+
- type: textarea
|
|
26
|
+
id: alternative
|
|
27
|
+
attributes:
|
|
28
|
+
label: Describe alternatives you've considered
|
|
29
|
+
description: A clear and concise description of any alternative solutions or features you've considered.
|
|
30
|
+
validations:
|
|
31
|
+
required: true
|
|
32
|
+
- type: textarea
|
|
33
|
+
id: addtional-context
|
|
34
|
+
attributes:
|
|
35
|
+
label: Additional context
|
|
36
|
+
description: Add any other context or screenshots about the feature request here.
|
|
37
|
+
validations:
|
|
38
|
+
required: false
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
name: Add bugs to fluent project
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
issues:
|
|
5
|
+
types:
|
|
6
|
+
- opened
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
add-to-project:
|
|
10
|
+
name: Add issue to project
|
|
11
|
+
runs-on: ubuntu-latest
|
|
12
|
+
steps:
|
|
13
|
+
- name: Generate token
|
|
14
|
+
id: generate_token
|
|
15
|
+
uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1
|
|
16
|
+
with:
|
|
17
|
+
app-id: ${{ secrets.PROJECT_APP_ID }}
|
|
18
|
+
private-key: ${{ secrets.PROJECT_APP_PRIVATE_KEY }}
|
|
19
|
+
owner: fluent
|
|
20
|
+
- uses: actions/add-to-project@244f685bbc3b7adfa8466e08b698b5577571133e # v1.0.2
|
|
21
|
+
with:
|
|
22
|
+
project-url: https://github.com/orgs/fluent/projects/4
|
|
23
|
+
github-token: ${{ steps.generate_token.outputs.token }}
|
|
24
|
+
labeled: waiting-for-triage
|
|
@@ -6,17 +6,16 @@ jobs:
|
|
|
6
6
|
coverage:
|
|
7
7
|
runs-on: ubuntu-latest
|
|
8
8
|
steps:
|
|
9
|
-
- uses: actions/checkout@
|
|
9
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
10
10
|
- name: Set up Ruby 3.0
|
|
11
|
-
uses: ruby/setup-ruby@v1
|
|
11
|
+
uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
12
12
|
with:
|
|
13
13
|
ruby-version: '3.1'
|
|
14
14
|
- name: Build and test with Rake
|
|
15
15
|
run: |
|
|
16
|
-
gem install bundler
|
|
17
16
|
bundle install
|
|
18
17
|
bundle exec rake
|
|
19
18
|
- name: Coveralls GitHub Action
|
|
20
|
-
uses: coverallsapp/github-action@
|
|
19
|
+
uses: coverallsapp/github-action@648a8eb78e6d50909eff900e4ec85cab4524a45b # v2.3.6
|
|
21
20
|
with:
|
|
22
21
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
@@ -5,7 +5,7 @@ jobs:
|
|
|
5
5
|
runs-on: ubuntu-latest
|
|
6
6
|
steps:
|
|
7
7
|
- name: Autoclose issues that did not follow issue template
|
|
8
|
-
uses: roots/issue-closer-action@v1.
|
|
8
|
+
uses: roots/issue-closer-action@fec85f2a845cd6cf33eda6e6a4c93026e71f86d1 # v1.2
|
|
9
9
|
with:
|
|
10
10
|
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
|
11
11
|
issue-close-message: "@${issue.user.login} this issue was automatically closed because it did not follow the issue template."
|
data/.github/workflows/linux.yml
CHANGED
|
@@ -1,27 +1,38 @@
|
|
|
1
1
|
name: Testing on Ubuntu
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- ubuntu-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
29
|
+
- uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle config set path 'vendor/bundle'
|
|
26
37
|
bundle install --jobs 4 --retry 3
|
|
27
38
|
bundle exec rake test
|
data/.github/workflows/macos.yml
CHANGED
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
name: Testing on macOS
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- macOS-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
29
|
+
- uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle install --jobs 4 --retry 3
|
|
26
37
|
bundle exec rake test
|
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
name: Testing on Windows
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- windows-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
29
|
+
- uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle install --jobs 4 --retry 3
|
|
26
37
|
bundle exec rake test
|
data/History.md
CHANGED
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
### [Unreleased]
|
|
4
4
|
|
|
5
|
+
### 1.1.6
|
|
6
|
+
- out\_opensearch\_data\_stream: filter failed items in data stream bulk error log (#175)
|
|
7
|
+
- out\_opensearch: fix default value of `refresh_credentials_interval` (#159)
|
|
8
|
+
|
|
5
9
|
### 1.1.5
|
|
6
10
|
- Fix bundle installation error on GitHub Action (#143)
|
|
7
11
|
- Prevent AWS credentials refresh from stopping on exception (#142)
|
data/README.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Fluent::Plugin::OpenSearch, a plugin for [Fluentd](http://fluentd.org)
|
|
2
2
|
|
|
3
|
-
[](https://badge.fury.io/rb/fluent-plugin-opensearch)
|
|
4
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/windows.yml)
|
|
5
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/macos.yml)
|
|
6
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/linux.yml)
|
|
7
7
|
[](https://coveralls.io/github/fluent/fluent-plugin-opensearch?branch=main)
|
|
8
8
|
|
|
9
9
|
Send your logs to OpenSearch (and search them with OpenSearch Dashboards maybe?)
|
|
@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
|
|
|
3
3
|
|
|
4
4
|
Gem::Specification.new do |s|
|
|
5
5
|
s.name = 'fluent-plugin-opensearch'
|
|
6
|
-
s.version = '1.1.
|
|
6
|
+
s.version = '1.1.6'
|
|
7
7
|
s.authors = ['Hiroshi Hatake']
|
|
8
8
|
s.email = ['cosmo0920.wp@gmail.com']
|
|
9
9
|
s.description = %q{Opensearch output plugin for Fluent event collector}
|
|
@@ -31,7 +31,7 @@ Gem::Specification.new do |s|
|
|
|
31
31
|
s.add_runtime_dependency "faraday_middleware-aws-sigv4", "~> 1.0.1"
|
|
32
32
|
|
|
33
33
|
s.add_development_dependency 'rake', '>= 0'
|
|
34
|
-
s.add_development_dependency 'webrick', '~> 1.
|
|
34
|
+
s.add_development_dependency 'webrick', '~> 1.9'
|
|
35
35
|
s.add_development_dependency 'webmock', '~> 3.18.1'
|
|
36
36
|
s.add_development_dependency 'test-unit', '~> 3.3.0'
|
|
37
37
|
s.add_development_dependency 'minitest', '~> 5.8'
|
|
@@ -195,7 +195,7 @@ module Fluent::Plugin
|
|
|
195
195
|
config_param :assume_role_session_name, :string, :default => "fluentd"
|
|
196
196
|
config_param :assume_role_web_identity_token_file, :string, :default => nil
|
|
197
197
|
config_param :sts_credentials_region, :string, :default => nil
|
|
198
|
-
config_param :refresh_credentials_interval, :time, :default =>
|
|
198
|
+
config_param :refresh_credentials_interval, :time, :default => 18000 # 5 hours
|
|
199
199
|
config_param :aws_service_name, :enum, list: [:es, :aoss], :default => :es
|
|
200
200
|
end
|
|
201
201
|
|
|
@@ -174,6 +174,7 @@ module Fluent::Plugin
|
|
|
174
174
|
CREATE_OP => {}
|
|
175
175
|
}
|
|
176
176
|
tag = chunk.metadata.tag
|
|
177
|
+
records = []
|
|
177
178
|
chunk.msgpack_each do |time, record|
|
|
178
179
|
next unless record.is_a? Hash
|
|
179
180
|
begin
|
|
@@ -194,6 +195,7 @@ module Fluent::Plugin
|
|
|
194
195
|
@remove_keys.each { |key| record.delete(key) }
|
|
195
196
|
end
|
|
196
197
|
bulk_message = append_record_to_messages(CREATE_OP, {}, headers, record, bulk_message)
|
|
198
|
+
records << [time, record]
|
|
197
199
|
rescue => e
|
|
198
200
|
emit_error_label_event do
|
|
199
201
|
router.emit_error_event(tag, time, record, e)
|
|
@@ -210,7 +212,18 @@ module Fluent::Plugin
|
|
|
210
212
|
begin
|
|
211
213
|
response = client(host).bulk(params)
|
|
212
214
|
if response['errors']
|
|
213
|
-
|
|
215
|
+
failed_count = response['items'].count { |item| item.values.first['status'] >= 300 }
|
|
216
|
+
log.error "Could not bulk insert to Data Stream: #{data_stream_name}, #{failed_count} item(s) failed."
|
|
217
|
+
if emit_error_label_event?
|
|
218
|
+
response['items'].each_with_index do |item, idx|
|
|
219
|
+
status = item.values.first['status']
|
|
220
|
+
next unless status >= 300
|
|
221
|
+
time, record = records[idx]
|
|
222
|
+
next if record.nil?
|
|
223
|
+
error_info = item.values.first['error']
|
|
224
|
+
router.emit_error_event(tag, time, record, RuntimeError.new("status #{status}: #{error_info}"))
|
|
225
|
+
end
|
|
226
|
+
end
|
|
214
227
|
end
|
|
215
228
|
rescue => e
|
|
216
229
|
raise RecoverableRequestFailure, "could not push logs to OpenSearch cluster (#{data_stream_name}): #{e.message}"
|
|
@@ -229,3 +242,4 @@ module Fluent::Plugin
|
|
|
229
242
|
end
|
|
230
243
|
end
|
|
231
244
|
end
|
|
245
|
+
|
|
@@ -2755,7 +2755,8 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
2755
2755
|
driver.feed(time.to_i, sample_record.merge({"pipeline_id" => pipeline_id}))
|
|
2756
2756
|
end
|
|
2757
2757
|
}
|
|
2758
|
-
|
|
2758
|
+
connection_spec = {host: "myhost-1", port: 9200, scheme: "http"}
|
|
2759
|
+
assert_equal("could not push logs to OpenSearch cluster (#{connection_spec.inspect}): [503] ", exception.message)
|
|
2759
2760
|
end
|
|
2760
2761
|
end
|
|
2761
2762
|
|
|
@@ -743,4 +743,131 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
|
|
|
743
743
|
assert(!index_cmds[1].has_key?('remove_me'))
|
|
744
744
|
end
|
|
745
745
|
|
|
746
|
+
def test_bulk_error_logs_only_failed_items
|
|
747
|
+
stub_default
|
|
748
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
749
|
+
status: 200,
|
|
750
|
+
body: {
|
|
751
|
+
'errors' => true,
|
|
752
|
+
'items' => [
|
|
753
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
754
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
755
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
756
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
757
|
+
]
|
|
758
|
+
}.to_json,
|
|
759
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
760
|
+
)
|
|
761
|
+
conf = config_element(
|
|
762
|
+
'ROOT', '', {
|
|
763
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
764
|
+
'data_stream_name' => 'foo',
|
|
765
|
+
'data_stream_template_name' => 'foo_tpl'
|
|
766
|
+
})
|
|
767
|
+
driver(conf).run(default_tag: 'test') do
|
|
768
|
+
driver.feed(sample_record)
|
|
769
|
+
end
|
|
770
|
+
error_log = driver.logs.find { |l| l.include?("Could not bulk insert") }
|
|
771
|
+
assert_not_nil error_log, "Expected an error log entry for bulk insert failure"
|
|
772
|
+
assert_match(/1 item\(s\) failed/, error_log)
|
|
773
|
+
assert_no_match(/201/, error_log)
|
|
774
|
+
end
|
|
775
|
+
|
|
776
|
+
def test_bulk_error_logs_5xx_items
|
|
777
|
+
stub_default
|
|
778
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
779
|
+
status: 200,
|
|
780
|
+
body: {
|
|
781
|
+
'errors' => true,
|
|
782
|
+
'items' => [
|
|
783
|
+
{ 'create' => { 'status' => 200, '_index' => 'foo' } },
|
|
784
|
+
{ 'create' => { 'status' => 500, '_index' => 'foo',
|
|
785
|
+
'error' => { 'type' => 'internal_server_error' } } }
|
|
786
|
+
]
|
|
787
|
+
}.to_json,
|
|
788
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
789
|
+
)
|
|
790
|
+
conf = config_element(
|
|
791
|
+
'ROOT', '', {
|
|
792
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
793
|
+
'data_stream_name' => 'foo',
|
|
794
|
+
'data_stream_template_name' => 'foo_tpl'
|
|
795
|
+
})
|
|
796
|
+
driver(conf).run(default_tag: 'test') do
|
|
797
|
+
driver.feed(sample_record)
|
|
798
|
+
end
|
|
799
|
+
error_log = driver.logs.find { |l| l.include?("Could not bulk insert") }
|
|
800
|
+
assert_not_nil error_log, "Expected an error log entry for bulk insert failure"
|
|
801
|
+
assert_match(/1 item\(s\) failed/, error_log)
|
|
802
|
+
assert_no_match(/200/, error_log)
|
|
803
|
+
end
|
|
804
|
+
|
|
805
|
+
def test_bulk_error_emits_failed_records_to_error_label
|
|
806
|
+
stub_default
|
|
807
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
808
|
+
status: 200,
|
|
809
|
+
body: {
|
|
810
|
+
'errors' => true,
|
|
811
|
+
'items' => [
|
|
812
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
813
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
814
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
815
|
+
]
|
|
816
|
+
}.to_json,
|
|
817
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
818
|
+
)
|
|
819
|
+
conf = config_element(
|
|
820
|
+
'ROOT', '', {
|
|
821
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
822
|
+
'data_stream_name' => 'foo',
|
|
823
|
+
'data_stream_template_name' => 'foo_tpl',
|
|
824
|
+
'emit_error_label_event' => 'true'
|
|
825
|
+
})
|
|
826
|
+
error_events = []
|
|
827
|
+
flexmock(Fluent::Engine).should_receive(:emit_error_event).and_return { |tag, time, record, e|
|
|
828
|
+
error_events << { tag: tag, record: record, error: e }
|
|
829
|
+
}
|
|
830
|
+
d = driver(conf)
|
|
831
|
+
flexmock(d.instance.router).should_receive(:emit_error_event).and_return { |tag, time, record, e|
|
|
832
|
+
error_events << { tag: tag, record: record, error: e }
|
|
833
|
+
}
|
|
834
|
+
d.run(default_tag: 'test') do
|
|
835
|
+
d.feed(event_time, sample_record)
|
|
836
|
+
d.feed(event_time, sample_record)
|
|
837
|
+
end
|
|
838
|
+
assert_equal 1, error_events.size, "Expected exactly one record routed to @ERROR"
|
|
839
|
+
assert_match(/400/, error_events.first[:error].message)
|
|
840
|
+
end
|
|
841
|
+
|
|
842
|
+
def test_bulk_error_does_not_emit_error_events_when_disabled
|
|
843
|
+
stub_default
|
|
844
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
845
|
+
status: 200,
|
|
846
|
+
body: {
|
|
847
|
+
'errors' => true,
|
|
848
|
+
'items' => [
|
|
849
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
850
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
851
|
+
]
|
|
852
|
+
}.to_json,
|
|
853
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
854
|
+
)
|
|
855
|
+
conf = config_element(
|
|
856
|
+
'ROOT', '', {
|
|
857
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
858
|
+
'data_stream_name' => 'foo',
|
|
859
|
+
'data_stream_template_name' => 'foo_tpl',
|
|
860
|
+
'emit_error_label_event' => 'false'
|
|
861
|
+
})
|
|
862
|
+
error_events = []
|
|
863
|
+
d = driver(conf)
|
|
864
|
+
flexmock(d.instance.router).should_receive(:emit_error_event).never
|
|
865
|
+
d.run(default_tag: 'test') do
|
|
866
|
+
d.feed(event_time, sample_record)
|
|
867
|
+
end
|
|
868
|
+
error_log = d.logs.find { |l| l.include?("Could not bulk insert") }
|
|
869
|
+
assert_not_nil error_log, "Expected summary error log even when emit_error_label_event is false"
|
|
870
|
+
end
|
|
871
|
+
|
|
746
872
|
end
|
|
873
|
+
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-opensearch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.1.
|
|
4
|
+
version: 1.1.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Hiroshi Hatake
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: fluentd
|
|
@@ -128,14 +127,14 @@ dependencies:
|
|
|
128
127
|
requirements:
|
|
129
128
|
- - "~>"
|
|
130
129
|
- !ruby/object:Gem::Version
|
|
131
|
-
version: 1.
|
|
130
|
+
version: '1.9'
|
|
132
131
|
type: :development
|
|
133
132
|
prerelease: false
|
|
134
133
|
version_requirements: !ruby/object:Gem::Requirement
|
|
135
134
|
requirements:
|
|
136
135
|
- - "~>"
|
|
137
136
|
- !ruby/object:Gem::Version
|
|
138
|
-
version: 1.
|
|
137
|
+
version: '1.9'
|
|
139
138
|
- !ruby/object:Gem::Dependency
|
|
140
139
|
name: webmock
|
|
141
140
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -201,9 +200,12 @@ extra_rdoc_files: []
|
|
|
201
200
|
files:
|
|
202
201
|
- ".coveralls.yml"
|
|
203
202
|
- ".editorconfig"
|
|
204
|
-
- ".github/ISSUE_TEMPLATE/bug_report.
|
|
205
|
-
- ".github/ISSUE_TEMPLATE/
|
|
203
|
+
- ".github/ISSUE_TEMPLATE/bug_report.yaml"
|
|
204
|
+
- ".github/ISSUE_TEMPLATE/config.yml"
|
|
205
|
+
- ".github/ISSUE_TEMPLATE/feature_request.yaml"
|
|
206
206
|
- ".github/PULL_REQUEST_TEMPLATE/pull_request_template.md"
|
|
207
|
+
- ".github/dependabot.yml"
|
|
208
|
+
- ".github/workflows/add-to-project.yml"
|
|
207
209
|
- ".github/workflows/coverage.yaml"
|
|
208
210
|
- ".github/workflows/issue-auto-closer.yml"
|
|
209
211
|
- ".github/workflows/linux.yml"
|
|
@@ -254,7 +256,6 @@ licenses:
|
|
|
254
256
|
- Apache-2.0
|
|
255
257
|
metadata:
|
|
256
258
|
changelog_uri: https://github.com/fluent/fluent-plugin-opensearch/blob/master/History.md
|
|
257
|
-
post_install_message:
|
|
258
259
|
rdoc_options: []
|
|
259
260
|
require_paths:
|
|
260
261
|
- lib
|
|
@@ -269,8 +270,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
269
270
|
- !ruby/object:Gem::Version
|
|
270
271
|
version: '0'
|
|
271
272
|
requirements: []
|
|
272
|
-
rubygems_version:
|
|
273
|
-
signing_key:
|
|
273
|
+
rubygems_version: 4.0.10
|
|
274
274
|
specification_version: 4
|
|
275
275
|
summary: Opensearch output plugin for Fluent event collector
|
|
276
276
|
test_files:
|
|
@@ -1,29 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Bug Report
|
|
3
|
-
about: Create a report to help us improve. If you have questions about OpenSearch plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
(check apply)
|
|
8
|
-
- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
|
|
9
|
-
- [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
|
|
10
|
-
|
|
11
|
-
#### Steps to replicate
|
|
12
|
-
|
|
13
|
-
Provide example config and message
|
|
14
|
-
|
|
15
|
-
#### Expected Behavior or What you need to ask
|
|
16
|
-
|
|
17
|
-
...
|
|
18
|
-
|
|
19
|
-
#### Using Fluentd and OpenSearch plugin versions
|
|
20
|
-
|
|
21
|
-
* OS version
|
|
22
|
-
* Bare Metal or within Docker or Kubernetes or others?
|
|
23
|
-
* Fluentd v1.0 or later
|
|
24
|
-
* paste result of ``fluentd --version`` or ``td-agent --version``
|
|
25
|
-
* OpenSearch plugin version
|
|
26
|
-
* paste boot log of fluentd or td-agent
|
|
27
|
-
* paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
|
|
28
|
-
* OpenSearch version (optional)
|
|
29
|
-
* OpenSearch template(s) (optional)
|
|
@@ -1,24 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Feature request
|
|
3
|
-
about: Suggest an idea for this project
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
(check apply)
|
|
8
|
-
- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
|
|
9
|
-
|
|
10
|
-
**Is your feature request related to a problem? Please describe.**
|
|
11
|
-
|
|
12
|
-
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
|
|
13
|
-
|
|
14
|
-
**Describe the solution you'd like**
|
|
15
|
-
|
|
16
|
-
<!-- A clear and concise description of what you want to happen. -->
|
|
17
|
-
|
|
18
|
-
**Describe alternatives you've considered**
|
|
19
|
-
|
|
20
|
-
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
|
|
21
|
-
|
|
22
|
-
**Additional context**
|
|
23
|
-
|
|
24
|
-
<!-- Add any other context or screenshots about the feature request here. -->
|