fluent-plugin-opensearch 1.1.4 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.yaml +73 -0
- data/.github/ISSUE_TEMPLATE/config.yml +5 -0
- data/.github/ISSUE_TEMPLATE/feature_request.yaml +38 -0
- data/.github/dependabot.yml +6 -0
- data/.github/workflows/add-to-project.yml +24 -0
- data/.github/workflows/coverage.yaml +3 -4
- data/.github/workflows/issue-auto-closer.yml +1 -1
- data/.github/workflows/linux.yml +18 -6
- data/.github/workflows/macos.yml +17 -6
- data/.github/workflows/windows.yml +17 -6
- data/History.md +12 -0
- data/README.OpenSearchInput.md +92 -0
- data/README.md +4 -4
- data/fluent-plugin-opensearch.gemspec +4 -4
- data/lib/fluent/plugin/in_opensearch.rb +78 -3
- data/lib/fluent/plugin/out_opensearch.rb +6 -2
- data/lib/fluent/plugin/out_opensearch_data_stream.rb +17 -1
- data/test/plugin/test_in_opensearch.rb +7 -0
- data/test/plugin/test_out_opensearch.rb +24 -1
- data/test/plugin/test_out_opensearch_data_stream.rb +127 -0
- metadata +15 -16
- data/.github/ISSUE_TEMPLATE/bug_report.md +0 -29
- data/.github/ISSUE_TEMPLATE/feature_request.md +0 -24
- data/ISSUE_TEMPLATE.md +0 -26
- /data/{PULL_REQUEST_TEMPLATE.md → .github/PULL_REQUEST_TEMPLATE/pull_request_template.md} +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 26499682224f8d7dc4a9b204f99172b512057fb5c9b67d03b0bf1637d39f4224
|
|
4
|
+
data.tar.gz: b6bc5ff55815d050f50a9c8658e98e0dbfce5511ade14342fa26d31c4dba59d0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a0ffa39d8a11088809008e5d74997d1765ca624153890bf0699ea2ab9f5522d80b2226427291bee42b04abf370636abd8a7dc5d8a13af1baf6cc4089e345da91
|
|
7
|
+
data.tar.gz: eb8983700b0a7af66010a5e8ec5f2f5ad5a30532fd877c7ab547aabfef015b45b012b83069c9775bb0965ebb95e01da8b75a8eb303d481ec768d076673679a3f
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
name: Bug Report
|
|
2
|
+
description: Create a report with a procedure for reproducing the bug
|
|
3
|
+
labels: "waiting-for-triage"
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Please check the [README](https://github.com/fluent/fluent-plugin-opensearch/blob/main/README.md) first. To help us investigate the problem, please provide the following information.
|
|
9
|
+
- type: textarea
|
|
10
|
+
id: description
|
|
11
|
+
attributes:
|
|
12
|
+
label: Describe the bug
|
|
13
|
+
description: A clear and concise description of what the bug is
|
|
14
|
+
validations:
|
|
15
|
+
required: true
|
|
16
|
+
- type: textarea
|
|
17
|
+
id: reproduce
|
|
18
|
+
attributes:
|
|
19
|
+
label: To Reproduce
|
|
20
|
+
description: Steps to reproduce the behavior
|
|
21
|
+
validations:
|
|
22
|
+
required: true
|
|
23
|
+
- type: textarea
|
|
24
|
+
id: expected
|
|
25
|
+
attributes:
|
|
26
|
+
label: Expected behavior
|
|
27
|
+
description: A clear and concise description of what you expected to happen
|
|
28
|
+
validations:
|
|
29
|
+
required: true
|
|
30
|
+
- type: textarea
|
|
31
|
+
id: environment
|
|
32
|
+
attributes:
|
|
33
|
+
label: Your Environment
|
|
34
|
+
description: |
|
|
35
|
+
- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
|
|
36
|
+
- Operating system: `cat /etc/os-release`
|
|
37
|
+
- Kernel version: `uname -r`
|
|
38
|
+
|
|
39
|
+
Tip: If you hit the problem with older fluentd version, try latest version first.
|
|
40
|
+
value: |
|
|
41
|
+
- Fluentd version:
|
|
42
|
+
- TD Agent version:
|
|
43
|
+
- fluent-plugin-opensearch version:
|
|
44
|
+
- OpenSearch version (optional)
|
|
45
|
+
- OpenSearch template(s) (optional)
|
|
46
|
+
- Operating system:
|
|
47
|
+
- Kernel version:
|
|
48
|
+
render: markdown
|
|
49
|
+
validations:
|
|
50
|
+
required: true
|
|
51
|
+
- type: textarea
|
|
52
|
+
id: configuration
|
|
53
|
+
attributes:
|
|
54
|
+
label: Your Configuration
|
|
55
|
+
description: |
|
|
56
|
+
Write your configuration here. Minimum reproducible fluentd.conf is recommended.
|
|
57
|
+
validations:
|
|
58
|
+
required: true
|
|
59
|
+
- type: textarea
|
|
60
|
+
id: logs
|
|
61
|
+
attributes:
|
|
62
|
+
label: Your Error Log
|
|
63
|
+
description: Write your ALL error log here
|
|
64
|
+
render: shell
|
|
65
|
+
validations:
|
|
66
|
+
required: true
|
|
67
|
+
- type: textarea
|
|
68
|
+
id: addtional-context
|
|
69
|
+
attributes:
|
|
70
|
+
label: Additional context
|
|
71
|
+
description: Add any other context about the problem here.
|
|
72
|
+
validations:
|
|
73
|
+
required: false
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
name: Feature request
|
|
2
|
+
description: Suggest an idea for this project
|
|
3
|
+
labels: "waiting-for-triage"
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Please check the [README](https://github.com/fluent/fluent-plugin-opensearch/blob/main/README.md) first. To help us understand the request, please provide the following information.
|
|
9
|
+
- type: textarea
|
|
10
|
+
id: description
|
|
11
|
+
attributes:
|
|
12
|
+
label: Is your feature request related to a problem? Please describe.
|
|
13
|
+
description: |
|
|
14
|
+
A clear and concise description of what the problem is.
|
|
15
|
+
Ex. I'm always frustrated when [...]
|
|
16
|
+
validations:
|
|
17
|
+
required: true
|
|
18
|
+
- type: textarea
|
|
19
|
+
id: solution
|
|
20
|
+
attributes:
|
|
21
|
+
label: Describe the solution you'd like
|
|
22
|
+
description: A clear and concise description of what you want to happen.
|
|
23
|
+
validations:
|
|
24
|
+
required: true
|
|
25
|
+
- type: textarea
|
|
26
|
+
id: alternative
|
|
27
|
+
attributes:
|
|
28
|
+
label: Describe alternatives you've considered
|
|
29
|
+
description: A clear and concise description of any alternative solutions or features you've considered.
|
|
30
|
+
validations:
|
|
31
|
+
required: true
|
|
32
|
+
- type: textarea
|
|
33
|
+
id: addtional-context
|
|
34
|
+
attributes:
|
|
35
|
+
label: Additional context
|
|
36
|
+
description: Add any other context or screenshots about the feature request here.
|
|
37
|
+
validations:
|
|
38
|
+
required: false
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
name: Add bugs to fluent project
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
issues:
|
|
5
|
+
types:
|
|
6
|
+
- opened
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
add-to-project:
|
|
10
|
+
name: Add issue to project
|
|
11
|
+
runs-on: ubuntu-latest
|
|
12
|
+
steps:
|
|
13
|
+
- name: Generate token
|
|
14
|
+
id: generate_token
|
|
15
|
+
uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1
|
|
16
|
+
with:
|
|
17
|
+
app-id: ${{ secrets.PROJECT_APP_ID }}
|
|
18
|
+
private-key: ${{ secrets.PROJECT_APP_PRIVATE_KEY }}
|
|
19
|
+
owner: fluent
|
|
20
|
+
- uses: actions/add-to-project@244f685bbc3b7adfa8466e08b698b5577571133e # v1.0.2
|
|
21
|
+
with:
|
|
22
|
+
project-url: https://github.com/orgs/fluent/projects/4
|
|
23
|
+
github-token: ${{ steps.generate_token.outputs.token }}
|
|
24
|
+
labeled: waiting-for-triage
|
|
@@ -6,17 +6,16 @@ jobs:
|
|
|
6
6
|
coverage:
|
|
7
7
|
runs-on: ubuntu-latest
|
|
8
8
|
steps:
|
|
9
|
-
- uses: actions/checkout@
|
|
9
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
10
10
|
- name: Set up Ruby 3.0
|
|
11
|
-
uses: ruby/setup-ruby@v1
|
|
11
|
+
uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
12
12
|
with:
|
|
13
13
|
ruby-version: '3.1'
|
|
14
14
|
- name: Build and test with Rake
|
|
15
15
|
run: |
|
|
16
|
-
gem install bundler
|
|
17
16
|
bundle install
|
|
18
17
|
bundle exec rake
|
|
19
18
|
- name: Coveralls GitHub Action
|
|
20
|
-
uses: coverallsapp/github-action@
|
|
19
|
+
uses: coverallsapp/github-action@648a8eb78e6d50909eff900e4ec85cab4524a45b # v2.3.6
|
|
21
20
|
with:
|
|
22
21
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
@@ -5,7 +5,7 @@ jobs:
|
|
|
5
5
|
runs-on: ubuntu-latest
|
|
6
6
|
steps:
|
|
7
7
|
- name: Autoclose issues that did not follow issue template
|
|
8
|
-
uses: roots/issue-closer-action@v1.
|
|
8
|
+
uses: roots/issue-closer-action@fec85f2a845cd6cf33eda6e6a4c93026e71f86d1 # v1.2
|
|
9
9
|
with:
|
|
10
10
|
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
|
11
11
|
issue-close-message: "@${issue.user.login} this issue was automatically closed because it did not follow the issue template."
|
data/.github/workflows/linux.yml
CHANGED
|
@@ -1,26 +1,38 @@
|
|
|
1
1
|
name: Testing on Ubuntu
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- ubuntu-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
29
|
+
- uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
|
|
36
|
+
bundle config set path 'vendor/bundle'
|
|
25
37
|
bundle install --jobs 4 --retry 3
|
|
26
38
|
bundle exec rake test
|
data/.github/workflows/macos.yml
CHANGED
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
name: Testing on macOS
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- macOS-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
29
|
+
- uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle install --jobs 4 --retry 3
|
|
26
37
|
bundle exec rake test
|
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
name: Testing on Windows
|
|
2
2
|
on:
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
branches: [main]
|
|
7
|
+
schedule:
|
|
8
|
+
- cron: '0 0 1 * *'
|
|
5
9
|
jobs:
|
|
10
|
+
ruby-versions:
|
|
11
|
+
uses: ruby/actions/.github/workflows/ruby_versions.yml@master
|
|
12
|
+
with:
|
|
13
|
+
engine: cruby
|
|
14
|
+
min_version: 3.1
|
|
6
15
|
build:
|
|
16
|
+
needs: ruby-versions
|
|
7
17
|
runs-on: ${{ matrix.os }}
|
|
8
18
|
strategy:
|
|
9
19
|
fail-fast: false
|
|
10
20
|
matrix:
|
|
11
|
-
ruby:
|
|
21
|
+
ruby: ${{ fromJson(needs.ruby-versions.outputs.versions) }}
|
|
22
|
+
exclude:
|
|
23
|
+
- ruby: head
|
|
12
24
|
os:
|
|
13
25
|
- windows-latest
|
|
14
26
|
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
27
|
steps:
|
|
16
|
-
- uses: actions/checkout@
|
|
17
|
-
- uses: ruby/setup-ruby@v1
|
|
28
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
29
|
+
- uses: ruby/setup-ruby@c4e5b1316158f92e3d49443a9d58b31d25ac0f8f # v1.306.0
|
|
18
30
|
with:
|
|
19
31
|
ruby-version: ${{ matrix.ruby }}
|
|
20
32
|
- name: unit testing
|
|
21
33
|
env:
|
|
22
34
|
CI: true
|
|
23
35
|
run: |
|
|
24
|
-
gem install bundler rake
|
|
25
36
|
bundle install --jobs 4 --retry 3
|
|
26
37
|
bundle exec rake test
|
data/History.md
CHANGED
|
@@ -2,6 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
### [Unreleased]
|
|
4
4
|
|
|
5
|
+
### 1.1.6
|
|
6
|
+
- out\_opensearch\_data\_stream: filter failed items in data stream bulk error log (#175)
|
|
7
|
+
- out\_opensearch: fix default value of `refresh_credentials_interval` (#159)
|
|
8
|
+
|
|
9
|
+
### 1.1.5
|
|
10
|
+
- Fix bundle installation error on GitHub Action (#143)
|
|
11
|
+
- Prevent AWS credentials refresh from stopping on exception (#142)
|
|
12
|
+
- Added a retry logic and a service availability check function for high availability. (#136)
|
|
13
|
+
- out\_opensearch\_data\_stream: Early return on empty body (#131)
|
|
14
|
+
- Docs/cleanup issue templates (#119)
|
|
15
|
+
- pin dependency opensearch-ruby to 3.0.1 (#116)
|
|
16
|
+
|
|
5
17
|
### 1.1.4
|
|
6
18
|
- test: remove minitest to correct misjudge of the framework by flexmock (#114)
|
|
7
19
|
- Add logic to write method of out_opensearch_data_stream (#109)
|
data/README.OpenSearchInput.md
CHANGED
|
@@ -24,6 +24,16 @@
|
|
|
24
24
|
+ [docinfo_fields](#docinfo_fields)
|
|
25
25
|
+ [docinfo_target](#docinfo_target)
|
|
26
26
|
+ [docinfo](#docinfo)
|
|
27
|
+
+ [check_connection](#check_connection)
|
|
28
|
+
+ [retry_forever](#retry_forever)
|
|
29
|
+
+ [retry_timeout](#retry_timeout)
|
|
30
|
+
+ [retry_max_times](#retry_max_times)
|
|
31
|
+
+ [retry_type](#retry_type)
|
|
32
|
+
+ [retry_wait](#retry_wait)
|
|
33
|
+
+ [retry_exponential_backoff_base](#retry_exponential_backoff_base)
|
|
34
|
+
+ [retry_max_interval](#retry_max_interval)
|
|
35
|
+
+ [retry_randomize](#retry_randomize)
|
|
36
|
+
|
|
27
37
|
* [Advanced Usage](#advanced-usage)
|
|
28
38
|
|
|
29
39
|
## Usage
|
|
@@ -274,6 +284,88 @@ This parameter specifies whether docinfo information including or not. The defau
|
|
|
274
284
|
docinfo false
|
|
275
285
|
```
|
|
276
286
|
|
|
287
|
+
### check_connection
|
|
288
|
+
|
|
289
|
+
The parameter for checking on connection availability with Elasticsearch or Opensearch hosts. The default value is `true`.
|
|
290
|
+
|
|
291
|
+
```
|
|
292
|
+
check_connection true
|
|
293
|
+
```
|
|
294
|
+
### retry_forever
|
|
295
|
+
|
|
296
|
+
The parameter If true, plugin will ignore retry_timeout and retry_max_times options and retry forever. The default value is `true`.
|
|
297
|
+
|
|
298
|
+
```
|
|
299
|
+
retry_forever true
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
### retry_timeout
|
|
303
|
+
|
|
304
|
+
The parameter maximum time (seconds) to retry again the failed try, until the plugin discards the retry.
|
|
305
|
+
If the next retry is going to exceed this time limit, the last retry will be made at exactly this time limit..
|
|
306
|
+
The default value is `72h`.
|
|
307
|
+
72hours == 17 times with exponential backoff (not to change default behavior)
|
|
308
|
+
|
|
309
|
+
```
|
|
310
|
+
retry_timeout 72 * 60 * 60
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
### retry_max_times
|
|
314
|
+
|
|
315
|
+
The parameter maximum number of times to retry the failed try. The default value is `5`
|
|
316
|
+
|
|
317
|
+
```
|
|
318
|
+
retry_max_times 5
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
### retry_type
|
|
322
|
+
|
|
323
|
+
The parameter needs for how long need to wait (time in seconds) to retry again:
|
|
324
|
+
`exponential_backoff`: wait in seconds will become large exponentially per failure,
|
|
325
|
+
`periodic`: plugin will retry periodically with fixed intervals (configured via retry_wait). The default value is `:exponential_backoff`
|
|
326
|
+
Periodic -> fixed :retry_wait
|
|
327
|
+
Exponential backoff: k is number of retry times
|
|
328
|
+
c: constant factor, @retry_wait
|
|
329
|
+
b: base factor, @retry_exponential_backoff_base
|
|
330
|
+
k: times
|
|
331
|
+
total retry time: c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1
|
|
332
|
+
|
|
333
|
+
```
|
|
334
|
+
retry_type exponential_backoff
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
### retry_wait
|
|
338
|
+
|
|
339
|
+
The parameter needs for wait in seconds before the next retry to again or constant factor of exponential backoff. The default value is `5`
|
|
340
|
+
|
|
341
|
+
```
|
|
342
|
+
retry_wait 5
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+
### retry_exponential_backoff_base
|
|
346
|
+
|
|
347
|
+
The parameter The base number of exponential backoff for retries. The default value is `2`
|
|
348
|
+
|
|
349
|
+
```
|
|
350
|
+
retry_exponential_backoff_base 2
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
### retry_max_interval
|
|
354
|
+
|
|
355
|
+
The parameter maximum interval (seconds) for exponential backoff between retries while failing. The default value is `nil`
|
|
356
|
+
|
|
357
|
+
```
|
|
358
|
+
retry_max_interval nil
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
### retry_randomize
|
|
362
|
+
|
|
363
|
+
The parameter If true, the plugin will retry after randomized interval not to do burst retries. The default value is `false`
|
|
364
|
+
|
|
365
|
+
```
|
|
366
|
+
retry_randomize false
|
|
367
|
+
```
|
|
368
|
+
|
|
277
369
|
## Advanced Usage
|
|
278
370
|
|
|
279
371
|
OpenSearch Input plugin and OpenSearch output plugin can combine to transfer records into another cluster.
|
data/README.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Fluent::Plugin::OpenSearch, a plugin for [Fluentd](http://fluentd.org)
|
|
2
2
|
|
|
3
|
-
[](https://badge.fury.io/rb/fluent-plugin-opensearch)
|
|
4
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/windows.yml)
|
|
5
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/macos.yml)
|
|
6
|
+
[](https://github.com/fluent/fluent-plugin-opensearch/actions/workflows/linux.yml)
|
|
7
7
|
[](https://coveralls.io/github/fluent/fluent-plugin-opensearch?branch=main)
|
|
8
8
|
|
|
9
9
|
Send your logs to OpenSearch (and search them with OpenSearch Dashboards maybe?)
|
|
@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
|
|
|
3
3
|
|
|
4
4
|
Gem::Specification.new do |s|
|
|
5
5
|
s.name = 'fluent-plugin-opensearch'
|
|
6
|
-
s.version = '1.1.
|
|
6
|
+
s.version = '1.1.6'
|
|
7
7
|
s.authors = ['Hiroshi Hatake']
|
|
8
8
|
s.email = ['cosmo0920.wp@gmail.com']
|
|
9
9
|
s.description = %q{Opensearch output plugin for Fluent event collector}
|
|
@@ -23,7 +23,7 @@ Gem::Specification.new do |s|
|
|
|
23
23
|
s.required_ruby_version = Gem::Requirement.new(">= 2.3".freeze)
|
|
24
24
|
|
|
25
25
|
s.add_runtime_dependency 'fluentd', '>= 0.14.22'
|
|
26
|
-
s.add_runtime_dependency 'opensearch-ruby'
|
|
26
|
+
s.add_runtime_dependency 'opensearch-ruby', '>= 3.0.1'
|
|
27
27
|
s.add_runtime_dependency "aws-sdk-core", "~> 3"
|
|
28
28
|
s.add_runtime_dependency 'excon', '>= 0'
|
|
29
29
|
s.add_runtime_dependency 'faraday', '>= 2.0.0'
|
|
@@ -31,8 +31,8 @@ Gem::Specification.new do |s|
|
|
|
31
31
|
s.add_runtime_dependency "faraday_middleware-aws-sigv4", "~> 1.0.1"
|
|
32
32
|
|
|
33
33
|
s.add_development_dependency 'rake', '>= 0'
|
|
34
|
-
s.add_development_dependency 'webrick', '~> 1.
|
|
35
|
-
s.add_development_dependency 'webmock', '~> 3'
|
|
34
|
+
s.add_development_dependency 'webrick', '~> 1.9'
|
|
35
|
+
s.add_development_dependency 'webmock', '~> 3.18.1'
|
|
36
36
|
s.add_development_dependency 'test-unit', '~> 3.3.0'
|
|
37
37
|
s.add_development_dependency 'minitest', '~> 5.8'
|
|
38
38
|
s.add_development_dependency 'flexmock', '~> 2.0'
|
|
@@ -29,6 +29,7 @@ require 'opensearch'
|
|
|
29
29
|
require 'faraday/excon'
|
|
30
30
|
require 'fluent/log-ext'
|
|
31
31
|
require 'fluent/plugin/input'
|
|
32
|
+
require 'fluent/plugin_helper'
|
|
32
33
|
require_relative 'opensearch_constants'
|
|
33
34
|
|
|
34
35
|
module Fluent::Plugin
|
|
@@ -39,7 +40,7 @@ module Fluent::Plugin
|
|
|
39
40
|
DEFAULT_STORAGE_TYPE = 'local'
|
|
40
41
|
METADATA = "@metadata".freeze
|
|
41
42
|
|
|
42
|
-
helpers :timer, :thread
|
|
43
|
+
helpers :timer, :thread, :retry_state
|
|
43
44
|
|
|
44
45
|
Fluent::Plugin.register_input('opensearch', self)
|
|
45
46
|
|
|
@@ -80,6 +81,23 @@ module Fluent::Plugin
|
|
|
80
81
|
config_param :docinfo_fields, :array, :default => ['_index', '_type', '_id']
|
|
81
82
|
config_param :docinfo_target, :string, :default => METADATA
|
|
82
83
|
config_param :docinfo, :bool, :default => false
|
|
84
|
+
config_param :check_connection, :bool, :default => true
|
|
85
|
+
config_param :retry_forever, :bool, default: true, desc: 'If true, plugin will ignore retry_timeout and retry_max_times options and retry forever.'
|
|
86
|
+
config_param :retry_timeout, :time, default: 72 * 60 * 60, desc: 'The maximum seconds to retry'
|
|
87
|
+
# 72hours == 17 times with exponential backoff (not to change default behavior)
|
|
88
|
+
config_param :retry_max_times, :integer, default: 5, desc: 'The maximum number of times to retry'
|
|
89
|
+
# exponential backoff sequence will be initialized at the time of this threshold
|
|
90
|
+
config_param :retry_type, :enum, list: [:exponential_backoff, :periodic], default: :exponential_backoff
|
|
91
|
+
### Periodic -> fixed :retry_wait
|
|
92
|
+
### Exponential backoff: k is number of retry times
|
|
93
|
+
# c: constant factor, @retry_wait
|
|
94
|
+
# b: base factor, @retry_exponential_backoff_base
|
|
95
|
+
# k: times
|
|
96
|
+
# total retry time: c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1
|
|
97
|
+
config_param :retry_wait, :time, default: 5, desc: 'Seconds to wait before next retry , or constant factor of exponential backoff.'
|
|
98
|
+
config_param :retry_exponential_backoff_base, :float, default: 2, desc: 'The base number of exponential backoff for retries.'
|
|
99
|
+
config_param :retry_max_interval, :time, default: nil, desc: 'The maximum interval seconds for exponential backoff between retries while failing.'
|
|
100
|
+
config_param :retry_randomize, :bool, default: false, desc: 'If true, output plugin will retry after randomized interval not to do burst retries.'
|
|
83
101
|
|
|
84
102
|
include Fluent::Plugin::OpenSearchConstants
|
|
85
103
|
|
|
@@ -92,6 +110,7 @@ module Fluent::Plugin
|
|
|
92
110
|
|
|
93
111
|
@timestamp_parser = create_time_parser
|
|
94
112
|
@backend_options = backend_options
|
|
113
|
+
@retry = nil
|
|
95
114
|
|
|
96
115
|
raise Fluent::ConfigError, "`password` must be present if `user` is present" if @user && @password.nil?
|
|
97
116
|
|
|
@@ -138,6 +157,15 @@ module Fluent::Plugin
|
|
|
138
157
|
raise Fluent::ConfigError, "You must install #{@http_backend} gem. Exception: #{ex}"
|
|
139
158
|
end
|
|
140
159
|
|
|
160
|
+
def retry_state(randomize)
|
|
161
|
+
retry_state_create(
|
|
162
|
+
:input_retries, @retry_type, @retry_wait, @retry_timeout,
|
|
163
|
+
forever: @retry_forever, max_steps: @retry_max_times,
|
|
164
|
+
max_interval: @retry_max_interval, backoff_base: @retry_exponential_backoff_base,
|
|
165
|
+
randomize: randomize
|
|
166
|
+
)
|
|
167
|
+
end
|
|
168
|
+
|
|
141
169
|
def get_escaped_userinfo(host_str)
|
|
142
170
|
if m = host_str.match(/(?<scheme>.*)%{(?<user>.*)}:%{(?<password>.*)}(?<path>@.*)/)
|
|
143
171
|
m["scheme"] +
|
|
@@ -176,12 +204,29 @@ module Fluent::Plugin
|
|
|
176
204
|
host.merge!(user: @user, password: @password) if !host[:user] && @user
|
|
177
205
|
host.merge!(path: @path) if !host[:path] && @path
|
|
178
206
|
end
|
|
179
|
-
|
|
207
|
+
live_hosts = @check_connection ? hosts.select { |host| reachable_host?(host) } : hosts
|
|
180
208
|
{
|
|
181
|
-
hosts:
|
|
209
|
+
hosts: live_hosts
|
|
182
210
|
}
|
|
183
211
|
end
|
|
184
212
|
|
|
213
|
+
def reachable_host?(host)
|
|
214
|
+
client = OpenSearch::Client.new(
|
|
215
|
+
host: ["#{host[:scheme]}://#{host[:host]}:#{host[:port]}"],
|
|
216
|
+
user: host[:user],
|
|
217
|
+
password: host[:password],
|
|
218
|
+
reload_connections: @reload_connections,
|
|
219
|
+
request_timeout: @request_timeout,
|
|
220
|
+
resurrect_after: @resurrect_after,
|
|
221
|
+
reload_on_failure: @reload_on_failure,
|
|
222
|
+
transport_options: { ssl: { verify: @ssl_verify, ca_file: @ca_file, version: @ssl_version } }
|
|
223
|
+
)
|
|
224
|
+
client.ping
|
|
225
|
+
rescue => e
|
|
226
|
+
log.warn "Failed to connect to #{host[:scheme]}://#{host[:host]}:#{host[:port]}: #{e.message}"
|
|
227
|
+
false
|
|
228
|
+
end
|
|
229
|
+
|
|
185
230
|
def emit_error_label_event(&block)
|
|
186
231
|
# If `emit_error_label_event` is specified as false, error event emittions are not occurred.
|
|
187
232
|
if emit_error_label_event
|
|
@@ -292,6 +337,25 @@ module Fluent::Plugin
|
|
|
292
337
|
return true
|
|
293
338
|
end
|
|
294
339
|
|
|
340
|
+
def update_retry_state(error=nil)
|
|
341
|
+
if error
|
|
342
|
+
unless @retry
|
|
343
|
+
@retry = retry_state(@retry_randomize)
|
|
344
|
+
end
|
|
345
|
+
@retry.step
|
|
346
|
+
#Raise error if the retry limit has been reached
|
|
347
|
+
raise "Hit limit for retries. retry_times: #{@retry.steps}, error: #{error.message}" if @retry.limit?
|
|
348
|
+
#Retry if the limit hasn't been reached
|
|
349
|
+
log.warn("failed to connect or search.", retry_times: @retry.steps, next_retry_time: @retry.next_time.round, error: error.message)
|
|
350
|
+
sleep(@retry.next_time - Time.now)
|
|
351
|
+
else
|
|
352
|
+
unless @retry.nil?
|
|
353
|
+
log.debug("retry succeeded.")
|
|
354
|
+
@retry = nil
|
|
355
|
+
end
|
|
356
|
+
end
|
|
357
|
+
end
|
|
358
|
+
|
|
295
359
|
def run
|
|
296
360
|
return run_slice if @num_slices <= 1
|
|
297
361
|
|
|
@@ -302,6 +366,9 @@ module Fluent::Plugin
|
|
|
302
366
|
run_slice(slice_id)
|
|
303
367
|
end
|
|
304
368
|
end
|
|
369
|
+
rescue Faraday::ConnectionFailed, OpenSearch::Transport::Transport::Error => error
|
|
370
|
+
update_retry_state(error)
|
|
371
|
+
retry
|
|
305
372
|
end
|
|
306
373
|
|
|
307
374
|
def run_slice(slice_id=nil)
|
|
@@ -321,7 +388,15 @@ module Fluent::Plugin
|
|
|
321
388
|
end
|
|
322
389
|
|
|
323
390
|
router.emit_stream(@tag, es)
|
|
391
|
+
clear_scroll(scroll_id)
|
|
392
|
+
update_retry_state
|
|
393
|
+
end
|
|
394
|
+
|
|
395
|
+
def clear_scroll(scroll_id)
|
|
324
396
|
client.clear_scroll(scroll_id: scroll_id) if scroll_id
|
|
397
|
+
rescue => e
|
|
398
|
+
# ignore & log any clear_scroll errors
|
|
399
|
+
log.warn("Ignoring clear_scroll exception", message: e.message, exception: e.class)
|
|
325
400
|
end
|
|
326
401
|
|
|
327
402
|
def process_scroll_request(scroll_id)
|
|
@@ -195,7 +195,7 @@ module Fluent::Plugin
|
|
|
195
195
|
config_param :assume_role_session_name, :string, :default => "fluentd"
|
|
196
196
|
config_param :assume_role_web_identity_token_file, :string, :default => nil
|
|
197
197
|
config_param :sts_credentials_region, :string, :default => nil
|
|
198
|
-
config_param :refresh_credentials_interval, :time, :default =>
|
|
198
|
+
config_param :refresh_credentials_interval, :time, :default => 18000 # 5 hours
|
|
199
199
|
config_param :aws_service_name, :enum, list: [:es, :aoss], :default => :es
|
|
200
200
|
end
|
|
201
201
|
|
|
@@ -350,7 +350,11 @@ module Fluent::Plugin
|
|
|
350
350
|
|
|
351
351
|
@credential_mutex.synchronize do
|
|
352
352
|
@_os = nil
|
|
353
|
-
|
|
353
|
+
begin
|
|
354
|
+
@_aws_credentials = aws_credentials(@endpoint)
|
|
355
|
+
rescue => e
|
|
356
|
+
log.error("Failed to get new AWS credentials: #{e}")
|
|
357
|
+
end
|
|
354
358
|
end
|
|
355
359
|
end
|
|
356
360
|
end
|
|
@@ -174,6 +174,7 @@ module Fluent::Plugin
|
|
|
174
174
|
CREATE_OP => {}
|
|
175
175
|
}
|
|
176
176
|
tag = chunk.metadata.tag
|
|
177
|
+
records = []
|
|
177
178
|
chunk.msgpack_each do |time, record|
|
|
178
179
|
next unless record.is_a? Hash
|
|
179
180
|
begin
|
|
@@ -194,6 +195,7 @@ module Fluent::Plugin
|
|
|
194
195
|
@remove_keys.each { |key| record.delete(key) }
|
|
195
196
|
end
|
|
196
197
|
bulk_message = append_record_to_messages(CREATE_OP, {}, headers, record, bulk_message)
|
|
198
|
+
records << [time, record]
|
|
197
199
|
rescue => e
|
|
198
200
|
emit_error_label_event do
|
|
199
201
|
router.emit_error_event(tag, time, record, e)
|
|
@@ -201,6 +203,8 @@ module Fluent::Plugin
|
|
|
201
203
|
end
|
|
202
204
|
end
|
|
203
205
|
|
|
206
|
+
return if bulk_message.to_s.empty?
|
|
207
|
+
|
|
204
208
|
params = {
|
|
205
209
|
index: data_stream_name,
|
|
206
210
|
body: bulk_message
|
|
@@ -208,7 +212,18 @@ module Fluent::Plugin
|
|
|
208
212
|
begin
|
|
209
213
|
response = client(host).bulk(params)
|
|
210
214
|
if response['errors']
|
|
211
|
-
|
|
215
|
+
failed_count = response['items'].count { |item| item.values.first['status'] >= 300 }
|
|
216
|
+
log.error "Could not bulk insert to Data Stream: #{data_stream_name}, #{failed_count} item(s) failed."
|
|
217
|
+
if emit_error_label_event?
|
|
218
|
+
response['items'].each_with_index do |item, idx|
|
|
219
|
+
status = item.values.first['status']
|
|
220
|
+
next unless status >= 300
|
|
221
|
+
time, record = records[idx]
|
|
222
|
+
next if record.nil?
|
|
223
|
+
error_info = item.values.first['error']
|
|
224
|
+
router.emit_error_event(tag, time, record, RuntimeError.new("status #{status}: #{error_info}"))
|
|
225
|
+
end
|
|
226
|
+
end
|
|
212
227
|
end
|
|
213
228
|
rescue => e
|
|
214
229
|
raise RecoverableRequestFailure, "could not push logs to OpenSearch cluster (#{data_stream_name}): #{e.message}"
|
|
@@ -227,3 +242,4 @@ module Fluent::Plugin
|
|
|
227
242
|
end
|
|
228
243
|
end
|
|
229
244
|
end
|
|
245
|
+
|
|
@@ -39,6 +39,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
39
39
|
CONFIG = %[
|
|
40
40
|
tag raw.opensearch
|
|
41
41
|
interval 2
|
|
42
|
+
check_connection false
|
|
42
43
|
]
|
|
43
44
|
|
|
44
45
|
def setup
|
|
@@ -190,6 +191,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
190
191
|
user john
|
|
191
192
|
password doe
|
|
192
193
|
tag raw.opensearch
|
|
194
|
+
check_connection false
|
|
193
195
|
}
|
|
194
196
|
instance = driver(config).instance
|
|
195
197
|
|
|
@@ -228,6 +230,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
228
230
|
user john
|
|
229
231
|
password doe
|
|
230
232
|
tag raw.opensearch
|
|
233
|
+
check_connection false
|
|
231
234
|
}
|
|
232
235
|
instance = driver(config).instance
|
|
233
236
|
|
|
@@ -249,6 +252,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
249
252
|
user %{j+hn}
|
|
250
253
|
password %{d@e}
|
|
251
254
|
tag raw.opensearch
|
|
255
|
+
check_connection false
|
|
252
256
|
}
|
|
253
257
|
instance = driver(config).instance
|
|
254
258
|
|
|
@@ -271,6 +275,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
271
275
|
path /es/
|
|
272
276
|
port 123
|
|
273
277
|
tag raw.opensearch
|
|
278
|
+
check_connection false
|
|
274
279
|
}
|
|
275
280
|
instance = driver(config).instance
|
|
276
281
|
|
|
@@ -295,6 +300,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
295
300
|
user default_user
|
|
296
301
|
password default_password
|
|
297
302
|
tag raw.opensearch
|
|
303
|
+
check_connection false
|
|
298
304
|
}
|
|
299
305
|
instance = driver(config).instance
|
|
300
306
|
|
|
@@ -323,6 +329,7 @@ class OpenSearchInputTest < Test::Unit::TestCase
|
|
|
323
329
|
user default_user
|
|
324
330
|
password default_password
|
|
325
331
|
tag raw.opensearch
|
|
332
|
+
check_connection false
|
|
326
333
|
}
|
|
327
334
|
instance = driver(config).instance
|
|
328
335
|
|
|
@@ -2755,7 +2755,8 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
2755
2755
|
driver.feed(time.to_i, sample_record.merge({"pipeline_id" => pipeline_id}))
|
|
2756
2756
|
end
|
|
2757
2757
|
}
|
|
2758
|
-
|
|
2758
|
+
connection_spec = {host: "myhost-1", port: 9200, scheme: "http"}
|
|
2759
|
+
assert_equal("could not push logs to OpenSearch cluster (#{connection_spec.inspect}): [503] ", exception.message)
|
|
2759
2760
|
end
|
|
2760
2761
|
end
|
|
2761
2762
|
|
|
@@ -3977,4 +3978,26 @@ class OpenSearchOutputTest < Test::Unit::TestCase
|
|
|
3977
3978
|
end
|
|
3978
3979
|
}
|
|
3979
3980
|
end
|
|
3981
|
+
|
|
3982
|
+
def test_no_aws_credentials_refresh_exception
|
|
3983
|
+
# See https://github.com/fluent/fluent-plugin-opensearch/issues/129
|
|
3984
|
+
endpoint_config =
|
|
3985
|
+
Fluent::Config::Element.new('endpoint', '', {
|
|
3986
|
+
'url' => "https://search-opensearch.aws.example.com/",
|
|
3987
|
+
'region' => "local",
|
|
3988
|
+
'access_key_id' => 'YOUR_AWESOME_KEY',
|
|
3989
|
+
'secret_access_key' => 'YOUR_AWESOME_SECRET',
|
|
3990
|
+
'refresh_credentials_interval' => '0',
|
|
3991
|
+
}, [])
|
|
3992
|
+
config = Fluent::Config::Element.new('ROOT', '**', { '@type' => 'opensearch' },
|
|
3993
|
+
[endpoint_config])
|
|
3994
|
+
# aws_credentials will be called twice in
|
|
3995
|
+
# OpenSearchOutput#configure call, and in the 2nd call was changed not
|
|
3996
|
+
# to emit exception. (instead, logging error) so check the error logs
|
|
3997
|
+
flexmock(Fluent::Plugin::OpenSearchOutput).new_instances.should_receive(:aws_credentials)
|
|
3998
|
+
.and_return(true).and_raise(::RuntimeError.new("No valid AWS credentials found."))
|
|
3999
|
+
d = driver(config)
|
|
4000
|
+
d.run
|
|
4001
|
+
assert { d.logs.any?(/\[error\]: Failed to get new AWS credentials: No valid AWS credentials found.\n/) }
|
|
4002
|
+
end
|
|
3980
4003
|
end
|
|
@@ -743,4 +743,131 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
|
|
|
743
743
|
assert(!index_cmds[1].has_key?('remove_me'))
|
|
744
744
|
end
|
|
745
745
|
|
|
746
|
+
def test_bulk_error_logs_only_failed_items
|
|
747
|
+
stub_default
|
|
748
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
749
|
+
status: 200,
|
|
750
|
+
body: {
|
|
751
|
+
'errors' => true,
|
|
752
|
+
'items' => [
|
|
753
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
754
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
755
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
756
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
757
|
+
]
|
|
758
|
+
}.to_json,
|
|
759
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
760
|
+
)
|
|
761
|
+
conf = config_element(
|
|
762
|
+
'ROOT', '', {
|
|
763
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
764
|
+
'data_stream_name' => 'foo',
|
|
765
|
+
'data_stream_template_name' => 'foo_tpl'
|
|
766
|
+
})
|
|
767
|
+
driver(conf).run(default_tag: 'test') do
|
|
768
|
+
driver.feed(sample_record)
|
|
769
|
+
end
|
|
770
|
+
error_log = driver.logs.find { |l| l.include?("Could not bulk insert") }
|
|
771
|
+
assert_not_nil error_log, "Expected an error log entry for bulk insert failure"
|
|
772
|
+
assert_match(/1 item\(s\) failed/, error_log)
|
|
773
|
+
assert_no_match(/201/, error_log)
|
|
774
|
+
end
|
|
775
|
+
|
|
776
|
+
def test_bulk_error_logs_5xx_items
|
|
777
|
+
stub_default
|
|
778
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
779
|
+
status: 200,
|
|
780
|
+
body: {
|
|
781
|
+
'errors' => true,
|
|
782
|
+
'items' => [
|
|
783
|
+
{ 'create' => { 'status' => 200, '_index' => 'foo' } },
|
|
784
|
+
{ 'create' => { 'status' => 500, '_index' => 'foo',
|
|
785
|
+
'error' => { 'type' => 'internal_server_error' } } }
|
|
786
|
+
]
|
|
787
|
+
}.to_json,
|
|
788
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
789
|
+
)
|
|
790
|
+
conf = config_element(
|
|
791
|
+
'ROOT', '', {
|
|
792
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
793
|
+
'data_stream_name' => 'foo',
|
|
794
|
+
'data_stream_template_name' => 'foo_tpl'
|
|
795
|
+
})
|
|
796
|
+
driver(conf).run(default_tag: 'test') do
|
|
797
|
+
driver.feed(sample_record)
|
|
798
|
+
end
|
|
799
|
+
error_log = driver.logs.find { |l| l.include?("Could not bulk insert") }
|
|
800
|
+
assert_not_nil error_log, "Expected an error log entry for bulk insert failure"
|
|
801
|
+
assert_match(/1 item\(s\) failed/, error_log)
|
|
802
|
+
assert_no_match(/200/, error_log)
|
|
803
|
+
end
|
|
804
|
+
|
|
805
|
+
def test_bulk_error_emits_failed_records_to_error_label
|
|
806
|
+
stub_default
|
|
807
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
808
|
+
status: 200,
|
|
809
|
+
body: {
|
|
810
|
+
'errors' => true,
|
|
811
|
+
'items' => [
|
|
812
|
+
{ 'create' => { 'status' => 201, '_index' => 'foo' } },
|
|
813
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
814
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
815
|
+
]
|
|
816
|
+
}.to_json,
|
|
817
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
818
|
+
)
|
|
819
|
+
conf = config_element(
|
|
820
|
+
'ROOT', '', {
|
|
821
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
822
|
+
'data_stream_name' => 'foo',
|
|
823
|
+
'data_stream_template_name' => 'foo_tpl',
|
|
824
|
+
'emit_error_label_event' => 'true'
|
|
825
|
+
})
|
|
826
|
+
error_events = []
|
|
827
|
+
flexmock(Fluent::Engine).should_receive(:emit_error_event).and_return { |tag, time, record, e|
|
|
828
|
+
error_events << { tag: tag, record: record, error: e }
|
|
829
|
+
}
|
|
830
|
+
d = driver(conf)
|
|
831
|
+
flexmock(d.instance.router).should_receive(:emit_error_event).and_return { |tag, time, record, e|
|
|
832
|
+
error_events << { tag: tag, record: record, error: e }
|
|
833
|
+
}
|
|
834
|
+
d.run(default_tag: 'test') do
|
|
835
|
+
d.feed(event_time, sample_record)
|
|
836
|
+
d.feed(event_time, sample_record)
|
|
837
|
+
end
|
|
838
|
+
assert_equal 1, error_events.size, "Expected exactly one record routed to @ERROR"
|
|
839
|
+
assert_match(/400/, error_events.first[:error].message)
|
|
840
|
+
end
|
|
841
|
+
|
|
842
|
+
def test_bulk_error_does_not_emit_error_events_when_disabled
|
|
843
|
+
stub_default
|
|
844
|
+
stub_request(:post, "http://localhost:9200/foo/_bulk").to_return(
|
|
845
|
+
status: 200,
|
|
846
|
+
body: {
|
|
847
|
+
'errors' => true,
|
|
848
|
+
'items' => [
|
|
849
|
+
{ 'create' => { 'status' => 400, '_index' => 'foo',
|
|
850
|
+
'error' => { 'type' => 'mapper_parsing_exception', 'reason' => 'failed to parse' } } }
|
|
851
|
+
]
|
|
852
|
+
}.to_json,
|
|
853
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
854
|
+
)
|
|
855
|
+
conf = config_element(
|
|
856
|
+
'ROOT', '', {
|
|
857
|
+
'@type' => OPENSEARCH_DATA_STREAM_TYPE,
|
|
858
|
+
'data_stream_name' => 'foo',
|
|
859
|
+
'data_stream_template_name' => 'foo_tpl',
|
|
860
|
+
'emit_error_label_event' => 'false'
|
|
861
|
+
})
|
|
862
|
+
error_events = []
|
|
863
|
+
d = driver(conf)
|
|
864
|
+
flexmock(d.instance.router).should_receive(:emit_error_event).never
|
|
865
|
+
d.run(default_tag: 'test') do
|
|
866
|
+
d.feed(event_time, sample_record)
|
|
867
|
+
end
|
|
868
|
+
error_log = d.logs.find { |l| l.include?("Could not bulk insert") }
|
|
869
|
+
assert_not_nil error_log, "Expected summary error log even when emit_error_label_event is false"
|
|
870
|
+
end
|
|
871
|
+
|
|
746
872
|
end
|
|
873
|
+
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-opensearch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.1.
|
|
4
|
+
version: 1.1.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Hiroshi Hatake
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: fluentd
|
|
@@ -30,14 +29,14 @@ dependencies:
|
|
|
30
29
|
requirements:
|
|
31
30
|
- - ">="
|
|
32
31
|
- !ruby/object:Gem::Version
|
|
33
|
-
version:
|
|
32
|
+
version: 3.0.1
|
|
34
33
|
type: :runtime
|
|
35
34
|
prerelease: false
|
|
36
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
36
|
requirements:
|
|
38
37
|
- - ">="
|
|
39
38
|
- !ruby/object:Gem::Version
|
|
40
|
-
version:
|
|
39
|
+
version: 3.0.1
|
|
41
40
|
- !ruby/object:Gem::Dependency
|
|
42
41
|
name: aws-sdk-core
|
|
43
42
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -128,28 +127,28 @@ dependencies:
|
|
|
128
127
|
requirements:
|
|
129
128
|
- - "~>"
|
|
130
129
|
- !ruby/object:Gem::Version
|
|
131
|
-
version: 1.
|
|
130
|
+
version: '1.9'
|
|
132
131
|
type: :development
|
|
133
132
|
prerelease: false
|
|
134
133
|
version_requirements: !ruby/object:Gem::Requirement
|
|
135
134
|
requirements:
|
|
136
135
|
- - "~>"
|
|
137
136
|
- !ruby/object:Gem::Version
|
|
138
|
-
version: 1.
|
|
137
|
+
version: '1.9'
|
|
139
138
|
- !ruby/object:Gem::Dependency
|
|
140
139
|
name: webmock
|
|
141
140
|
requirement: !ruby/object:Gem::Requirement
|
|
142
141
|
requirements:
|
|
143
142
|
- - "~>"
|
|
144
143
|
- !ruby/object:Gem::Version
|
|
145
|
-
version:
|
|
144
|
+
version: 3.18.1
|
|
146
145
|
type: :development
|
|
147
146
|
prerelease: false
|
|
148
147
|
version_requirements: !ruby/object:Gem::Requirement
|
|
149
148
|
requirements:
|
|
150
149
|
- - "~>"
|
|
151
150
|
- !ruby/object:Gem::Version
|
|
152
|
-
version:
|
|
151
|
+
version: 3.18.1
|
|
153
152
|
- !ruby/object:Gem::Dependency
|
|
154
153
|
name: test-unit
|
|
155
154
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -201,8 +200,12 @@ extra_rdoc_files: []
|
|
|
201
200
|
files:
|
|
202
201
|
- ".coveralls.yml"
|
|
203
202
|
- ".editorconfig"
|
|
204
|
-
- ".github/ISSUE_TEMPLATE/bug_report.
|
|
205
|
-
- ".github/ISSUE_TEMPLATE/
|
|
203
|
+
- ".github/ISSUE_TEMPLATE/bug_report.yaml"
|
|
204
|
+
- ".github/ISSUE_TEMPLATE/config.yml"
|
|
205
|
+
- ".github/ISSUE_TEMPLATE/feature_request.yaml"
|
|
206
|
+
- ".github/PULL_REQUEST_TEMPLATE/pull_request_template.md"
|
|
207
|
+
- ".github/dependabot.yml"
|
|
208
|
+
- ".github/workflows/add-to-project.yml"
|
|
206
209
|
- ".github/workflows/coverage.yaml"
|
|
207
210
|
- ".github/workflows/issue-auto-closer.yml"
|
|
208
211
|
- ".github/workflows/linux.yml"
|
|
@@ -212,9 +215,7 @@ files:
|
|
|
212
215
|
- CONTRIBUTING.md
|
|
213
216
|
- Gemfile
|
|
214
217
|
- History.md
|
|
215
|
-
- ISSUE_TEMPLATE.md
|
|
216
218
|
- LICENSE.txt
|
|
217
|
-
- PULL_REQUEST_TEMPLATE.md
|
|
218
219
|
- README.OpenSearchGenID.md
|
|
219
220
|
- README.OpenSearchInput.md
|
|
220
221
|
- README.Troubleshooting.md
|
|
@@ -255,7 +256,6 @@ licenses:
|
|
|
255
256
|
- Apache-2.0
|
|
256
257
|
metadata:
|
|
257
258
|
changelog_uri: https://github.com/fluent/fluent-plugin-opensearch/blob/master/History.md
|
|
258
|
-
post_install_message:
|
|
259
259
|
rdoc_options: []
|
|
260
260
|
require_paths:
|
|
261
261
|
- lib
|
|
@@ -270,8 +270,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
270
270
|
- !ruby/object:Gem::Version
|
|
271
271
|
version: '0'
|
|
272
272
|
requirements: []
|
|
273
|
-
rubygems_version:
|
|
274
|
-
signing_key:
|
|
273
|
+
rubygems_version: 4.0.10
|
|
275
274
|
specification_version: 4
|
|
276
275
|
summary: Opensearch output plugin for Fluent event collector
|
|
277
276
|
test_files:
|
|
@@ -1,29 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Bug Report
|
|
3
|
-
about: Create a report to help us improve. If you have questions about OpenSearch plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
(check apply)
|
|
8
|
-
- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
|
|
9
|
-
- [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
|
|
10
|
-
|
|
11
|
-
#### Steps to replicate
|
|
12
|
-
|
|
13
|
-
Provide example config and message
|
|
14
|
-
|
|
15
|
-
#### Expected Behavior or What you need to ask
|
|
16
|
-
|
|
17
|
-
...
|
|
18
|
-
|
|
19
|
-
#### Using Fluentd and OpenSearch plugin versions
|
|
20
|
-
|
|
21
|
-
* OS version
|
|
22
|
-
* Bare Metal or within Docker or Kubernetes or others?
|
|
23
|
-
* Fluentd v1.0 or later
|
|
24
|
-
* paste result of ``fluentd --version`` or ``td-agent --version``
|
|
25
|
-
* OpenSearch plugin version
|
|
26
|
-
* paste boot log of fluentd or td-agent
|
|
27
|
-
* paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
|
|
28
|
-
* OpenSearch version (optional)
|
|
29
|
-
* OpenSearch template(s) (optional)
|
|
@@ -1,24 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Feature request
|
|
3
|
-
about: Suggest an idea for this project
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
(check apply)
|
|
8
|
-
- [ ] read [the contribution guideline](https://github.com/uken/fluent-plugin-elasticsearch/blob/master/CONTRIBUTING.md)
|
|
9
|
-
|
|
10
|
-
**Is your feature request related to a problem? Please describe.**
|
|
11
|
-
|
|
12
|
-
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
|
|
13
|
-
|
|
14
|
-
**Describe the solution you'd like**
|
|
15
|
-
|
|
16
|
-
<!-- A clear and concise description of what you want to happen. -->
|
|
17
|
-
|
|
18
|
-
**Describe alternatives you've considered**
|
|
19
|
-
|
|
20
|
-
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
|
|
21
|
-
|
|
22
|
-
**Additional context**
|
|
23
|
-
|
|
24
|
-
<!-- Add any other context or screenshots about the feature request here. -->
|
data/ISSUE_TEMPLATE.md
DELETED
|
@@ -1,26 +0,0 @@
|
|
|
1
|
-
(check apply)
|
|
2
|
-
- [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
|
|
3
|
-
|
|
4
|
-
#### Problem
|
|
5
|
-
|
|
6
|
-
...
|
|
7
|
-
|
|
8
|
-
#### Steps to replicate
|
|
9
|
-
|
|
10
|
-
Provide example config and message
|
|
11
|
-
|
|
12
|
-
#### Expected Behavior or What you need to ask
|
|
13
|
-
|
|
14
|
-
...
|
|
15
|
-
|
|
16
|
-
#### Using Fluentd and OpenSearch plugin versions
|
|
17
|
-
|
|
18
|
-
* OS version
|
|
19
|
-
* Bare Metal or within Docker or Kubernetes or others?
|
|
20
|
-
* Fluentd v1.0 or later
|
|
21
|
-
* paste result of ``fluentd --version`` or ``td-agent --version``
|
|
22
|
-
* OpenSearch plugin version
|
|
23
|
-
* paste boot log of fluentd or td-agent
|
|
24
|
-
* paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
|
|
25
|
-
* OpenSearch version (optional)
|
|
26
|
-
* OpenSearch template(s) (optional)
|
|
File without changes
|