fluent-plugin-input-opensearch 1.1.9

Sign up to get free protection for your applications and to get access to all the features.
Files changed (52) hide show
  1. checksums.yaml +7 -0
  2. data/.coveralls.yml +1 -0
  3. data/.editorconfig +9 -0
  4. data/.github/ISSUE_TEMPLATE/bug_report.md +29 -0
  5. data/.github/ISSUE_TEMPLATE/feature_request.md +24 -0
  6. data/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md +9 -0
  7. data/.github/workflows/coverage.yaml +22 -0
  8. data/.github/workflows/issue-auto-closer.yml +12 -0
  9. data/.github/workflows/linux.yml +26 -0
  10. data/.github/workflows/macos.yml +26 -0
  11. data/.github/workflows/windows.yml +26 -0
  12. data/.gitignore +18 -0
  13. data/CONTRIBUTING.md +24 -0
  14. data/Gemfile +10 -0
  15. data/History.md +67 -0
  16. data/LICENSE.txt +201 -0
  17. data/README.OpenSearchGenID.md +116 -0
  18. data/README.OpenSearchInput.md +314 -0
  19. data/README.Troubleshooting.md +482 -0
  20. data/README.md +1622 -0
  21. data/Rakefile +37 -0
  22. data/fluent-plugin-opensearch.gemspec +39 -0
  23. data/gemfiles/Gemfile.elasticsearch.v6 +12 -0
  24. data/lib/fluent/log-ext.rb +64 -0
  25. data/lib/fluent/plugin/filter_opensearch_genid.rb +103 -0
  26. data/lib/fluent/plugin/in_opensearch.rb +410 -0
  27. data/lib/fluent/plugin/oj_serializer.rb +48 -0
  28. data/lib/fluent/plugin/opensearch_constants.rb +39 -0
  29. data/lib/fluent/plugin/opensearch_error.rb +31 -0
  30. data/lib/fluent/plugin/opensearch_error_handler.rb +182 -0
  31. data/lib/fluent/plugin/opensearch_fallback_selector.rb +36 -0
  32. data/lib/fluent/plugin/opensearch_index_template.rb +155 -0
  33. data/lib/fluent/plugin/opensearch_simple_sniffer.rb +36 -0
  34. data/lib/fluent/plugin/opensearch_tls.rb +96 -0
  35. data/lib/fluent/plugin/out_opensearch.rb +1158 -0
  36. data/lib/fluent/plugin/out_opensearch_data_stream.rb +229 -0
  37. data/test/helper.rb +60 -0
  38. data/test/plugin/datastream_template.json +4 -0
  39. data/test/plugin/test_alias_template.json +9 -0
  40. data/test/plugin/test_filter_opensearch_genid.rb +241 -0
  41. data/test/plugin/test_in_opensearch.rb +500 -0
  42. data/test/plugin/test_index_alias_template.json +11 -0
  43. data/test/plugin/test_index_template.json +25 -0
  44. data/test/plugin/test_oj_serializer.rb +45 -0
  45. data/test/plugin/test_opensearch_error_handler.rb +770 -0
  46. data/test/plugin/test_opensearch_fallback_selector.rb +100 -0
  47. data/test/plugin/test_opensearch_tls.rb +171 -0
  48. data/test/plugin/test_out_opensearch.rb +3980 -0
  49. data/test/plugin/test_out_opensearch_data_stream.rb +746 -0
  50. data/test/plugin/test_template.json +23 -0
  51. data/test/test_log-ext.rb +61 -0
  52. metadata +291 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 52ccf524b6427d6a8191410da0ed878ed988ac91881ad741d1b9464356ebdb0a
4
+ data.tar.gz: 0da3d50a8c427537fd307fabc5400775d4c26bb0e8eda131fd7eb4b89174c249
5
+ SHA512:
6
+ metadata.gz: 849df809ddd94adc13b5a66ee21302c9fa59e7a4bf732678c15ca0a8acbb2d2e57e52c54f03f76a4537e2a363b5f6eb1a13ebc41a741dfc625ef12b5ce449641
7
+ data.tar.gz: 7b4172bfa6cdeb847e8a595f64086d2f26171aca71ed762acc3f746d5af1be19e5b06df400118a9bd35ed57e73d0ad8460871b8b1244571cd252a4345b5a1a60
data/.coveralls.yml ADDED
@@ -0,0 +1 @@
1
+ service_name: github-ci
data/.editorconfig ADDED
@@ -0,0 +1,9 @@
1
+ # EditorConfig: http://EditorConfig.org
2
+
3
+ root = true
4
+
5
+ [*]
6
+ end_of_line = lf
7
+ insert_final_newline = true
8
+ indent_style = space
9
+ indent_size = 2
@@ -0,0 +1,29 @@
1
+ ---
2
+ name: Bug Report
3
+ about: Create a report to help us improve. If you have questions about OpenSearch plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
4
+
5
+ ---
6
+
7
+ (check apply)
8
+ - [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
9
+ - [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
10
+
11
+ #### Steps to replicate
12
+
13
+ Provide example config and message
14
+
15
+ #### Expected Behavior or What you need to ask
16
+
17
+ ...
18
+
19
+ #### Using Fluentd and OpenSearch plugin versions
20
+
21
+ * OS version
22
+ * Bare Metal or within Docker or Kubernetes or others?
23
+ * Fluentd v1.0 or later
24
+ * paste result of ``fluentd --version`` or ``td-agent --version``
25
+ * OpenSearch plugin version
26
+ * paste boot log of fluentd or td-agent
27
+ * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
28
+ * OpenSearch version (optional)
29
+ * OpenSearch template(s) (optional)
@@ -0,0 +1,24 @@
1
+ ---
2
+ name: Feature request
3
+ about: Suggest an idea for this project
4
+
5
+ ---
6
+
7
+ (check apply)
8
+ - [ ] read [the contribution guideline](https://github.com/fluent/fluent-plugin-opensearch/blob/master/CONTRIBUTING.md)
9
+
10
+ **Is your feature request related to a problem? Please describe.**
11
+
12
+ <!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
13
+
14
+ **Describe the solution you'd like**
15
+
16
+ <!-- A clear and concise description of what you want to happen. -->
17
+
18
+ **Describe alternatives you've considered**
19
+
20
+ <!-- A clear and concise description of any alternative solutions or features you've considered. -->
21
+
22
+ **Additional context**
23
+
24
+ <!-- Add any other context or screenshots about the feature request here. -->
@@ -0,0 +1,9 @@
1
+ DESCRIPTION HERE
2
+
3
+ (check all that apply)
4
+ - [ ] tests added
5
+ - [ ] tests passing
6
+ - [ ] README updated (if needed)
7
+ - [ ] README Table of Contents updated (if needed)
8
+ - [ ] History.md and `version` in gemspec are untouched
9
+ - [ ] backward compatible
@@ -0,0 +1,22 @@
1
+ name: coverage
2
+
3
+ on: [push, pull_request]
4
+
5
+ jobs:
6
+ coverage:
7
+ runs-on: ubuntu-latest
8
+ steps:
9
+ - uses: actions/checkout@v2
10
+ - name: Set up Ruby 3.0
11
+ uses: ruby/setup-ruby@v1
12
+ with:
13
+ ruby-version: '3.1'
14
+ - name: Build and test with Rake
15
+ run: |
16
+ gem install bundler
17
+ bundle install
18
+ bundle exec rake
19
+ - name: Coveralls GitHub Action
20
+ uses: coverallsapp/github-action@v1.1.2
21
+ with:
22
+ github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -0,0 +1,12 @@
1
+ name: Autocloser
2
+ on: [issues]
3
+ jobs:
4
+ autoclose:
5
+ runs-on: ubuntu-latest
6
+ steps:
7
+ - name: Autoclose issues that did not follow issue template
8
+ uses: roots/issue-closer-action@v1.1
9
+ with:
10
+ repo-token: ${{ secrets.GITHUB_TOKEN }}
11
+ issue-close-message: "@${issue.user.login} this issue was automatically closed because it did not follow the issue template."
12
+ issue-pattern: "(.*Problem.*)|(.*Expected Behavior or What you need to ask.*)|(.*Using Fluentd and ES plugin versions.*)|(.*Is your feature request related to a problem? Please describe.*)|(.*Describe the solution you'd like.*)|(.*Describe alternatives you've considered.*)"
@@ -0,0 +1,26 @@
1
+ name: Testing on Ubuntu
2
+ on:
3
+ - push
4
+ - pull_request
5
+ jobs:
6
+ build:
7
+ runs-on: ${{ matrix.os }}
8
+ strategy:
9
+ fail-fast: false
10
+ matrix:
11
+ ruby: [ '3.0', '3.1', '3.2' ]
12
+ os:
13
+ - ubuntu-latest
14
+ name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
15
+ steps:
16
+ - uses: actions/checkout@v2
17
+ - uses: ruby/setup-ruby@v1
18
+ with:
19
+ ruby-version: ${{ matrix.ruby }}
20
+ - name: unit testing
21
+ env:
22
+ CI: true
23
+ run: |
24
+ gem install bundler rake
25
+ bundle install --jobs 4 --retry 3
26
+ bundle exec rake test
@@ -0,0 +1,26 @@
1
+ name: Testing on macOS
2
+ on:
3
+ - push
4
+ - pull_request
5
+ jobs:
6
+ build:
7
+ runs-on: ${{ matrix.os }}
8
+ strategy:
9
+ fail-fast: false
10
+ matrix:
11
+ ruby: [ '3.0', '3.1', '3.2' ]
12
+ os:
13
+ - macOS-latest
14
+ name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
15
+ steps:
16
+ - uses: actions/checkout@v2
17
+ - uses: ruby/setup-ruby@v1
18
+ with:
19
+ ruby-version: ${{ matrix.ruby }}
20
+ - name: unit testing
21
+ env:
22
+ CI: true
23
+ run: |
24
+ gem install bundler rake
25
+ bundle install --jobs 4 --retry 3
26
+ bundle exec rake test
@@ -0,0 +1,26 @@
1
+ name: Testing on Windows
2
+ on:
3
+ - push
4
+ - pull_request
5
+ jobs:
6
+ build:
7
+ runs-on: ${{ matrix.os }}
8
+ strategy:
9
+ fail-fast: false
10
+ matrix:
11
+ ruby: [ '3.0', '3.1', '3.2' ]
12
+ os:
13
+ - windows-latest
14
+ name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
15
+ steps:
16
+ - uses: actions/checkout@v2
17
+ - uses: ruby/setup-ruby@v1
18
+ with:
19
+ ruby-version: ${{ matrix.ruby }}
20
+ - name: unit testing
21
+ env:
22
+ CI: true
23
+ run: |
24
+ gem install bundler rake
25
+ bundle install --jobs 4 --retry 3
26
+ bundle exec rake test
data/.gitignore ADDED
@@ -0,0 +1,18 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
18
+ .DS_Store
data/CONTRIBUTING.md ADDED
@@ -0,0 +1,24 @@
1
+ Contributing to fluent-plugin-elasticsearch
2
+ ===
3
+
4
+ We'd love your contribution. Here are the guidelines!
5
+
6
+ ## Got a question or problem?
7
+
8
+ RESOURCES of [OpenSearch plugin documentation](https://github.com/fluent/fluent-plugin-opensearch/blob/master/README.md) may help you.
9
+
10
+ If you have further questions about OpenSearch plugin, feel free to [open an Issue](https://github.com/fluent/fluent-plugin-opensearch/issues).
11
+
12
+ ## Found a bug?
13
+
14
+ If you find a bug of OpenSearch plugin or a mistake in the documentation, you can help us by submitting an issue to OpenSearch plugin. Even better you can submit a pull request with a fix.
15
+
16
+ Note: Please follow the issue template/pull request template when sending your issue. Otherwise, your issue _might going to be closed with no response_.
17
+
18
+ ## Patch Guidelines
19
+
20
+ Here are some things that would increase a chance that your patch is accepted:
21
+
22
+ * Write tests
23
+ * Run tests before sending pull request with `bundle exec rake test`
24
+ * Write [a good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)
data/Gemfile ADDED
@@ -0,0 +1,10 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in fluent-plugin-opensearch.gemspec
4
+ gemspec
5
+
6
+ gem 'simplecov', require: false
7
+ gem 'simplecov-lcov', require: false
8
+ gem 'strptime', require: false if RUBY_ENGINE == "ruby" && RUBY_VERSION =~ /^2/
9
+ gem "irb" if RUBY_ENGINE == "ruby" && RUBY_VERSION >= "2.6"
10
+ gem "oj"
data/History.md ADDED
@@ -0,0 +1,67 @@
1
+ ## Changelog [[tags]](https://github.com/fluent/fluent-plugin-opensearch/tags)
2
+
3
+ ### [Unreleased]
4
+
5
+ ### 1.1.4
6
+ - test: remove minitest to correct misjudge of the framework by flexmock (#114)
7
+ - Add logic to write method of out_opensearch_data_stream (#109)
8
+
9
+ ### 1.1.3
10
+ - Revert the behavior of passing duration second (#108)
11
+
12
+ ### 1.1.2
13
+ - Check OS cluster for data streams and templates for index template creation (#106)
14
+ - out\_opensearch\_data\_stream: Don't connect to opensearch on dry-run (#105)
15
+
16
+ ### 1.1.1
17
+ - Pass a value of refresh\_credentials\_interval as duration\_seconds (#78)
18
+
19
+ ### 1.1.0
20
+ - Unpin `faraday` from v1, upgrade to v2.
21
+ Note that if you can't migrate other plugins from `faraday` v1 yet, need to keep
22
+ fluent-plugin-opensearch v1.0.10.
23
+
24
+ ### 1.0.10
25
+ - Replace File.exists? with File.exist? to work with Ruby 3.2 (#93)
26
+ - Add a constraint for dependent gem to stay on Faraday v1 (#90)
27
+ - README.md: Fix a link to opensearch-ruby (#85)
28
+
29
+ ### 1.0.9
30
+ - Adjust GitHub workflows (#89)
31
+ - out\_opensearch: Provide service_name choices for handling serverless (#88)
32
+
33
+ ### 1.0.8
34
+ - Use faraday 1.x explicitly (#71)
35
+
36
+ ### 1.0.7
37
+ - Expire AWS credentials with a certain interval (#52)
38
+
39
+ ### 1.0.6
40
+ - out\_opensearch: Handle suppress\_type\_name operation correctly (#61)
41
+
42
+ ### 1.0.5
43
+ - Use if clause to detect requirements for emitting error label events (#57)
44
+ - opensearch_data_stream: Align lowercases for data_stream and its template names (#50)
45
+
46
+ ### 1.0.4
47
+ - Automatically create data streams (#44)
48
+
49
+ ### 1.0.3
50
+ - Configurable unrecoverable record types (#40)
51
+ - Handle exceptions on retrieving AWS credentials (#39)
52
+ - Suppress emit error label events (#38)
53
+ - Provide suppress_type_name parameter (#29)
54
+ - Honor @time_key (data streams) (#28)
55
+
56
+ ### 1.0.2
57
+ - Honor @hosts parameter for Data Streams (#21)
58
+ - Use template_file for Data Streams (#20)
59
+ - Specify host argument if needed (#11)
60
+
61
+ ### 1.0.1
62
+ - Add testcases for hosts parameter (#10)
63
+ - Permit to handle @hosts parameter (#9)
64
+ - Retry creating data stream/template when OpenSearch is unreachable (#8)
65
+
66
+ ### 1.0.0
67
+ - Initial public gem release.
data/LICENSE.txt ADDED
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "{}"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
@@ -0,0 +1,116 @@
1
+ ## Index
2
+
3
+ * [Usage](#usage)
4
+ * [Configuration](#configuration)
5
+ + [hash_id_key](#hash_id_key)
6
+ + [include_tag_in_seed](#include_tag_in_seed)
7
+ + [include_time_in_seed](#include_time_in_seed)
8
+ + [use_record_as_seed](#use_record_as_seed)
9
+ + [use_entire_record](#use_entire_record)
10
+ + [record_keys](#record_keys)
11
+ + [separator](#separator)
12
+ + [hash_type](#hash_type)
13
+ * [Advanced Usage](#advanced-usage)
14
+
15
+ ## Usage
16
+
17
+ In your Fluentd configuration, use `@type opensearch_genid`. Additional configuration is optional, default values would look like this:
18
+
19
+ ```
20
+ <filter>
21
+ @type opensearch_genid
22
+ hash_id_key _hash
23
+ include_tag_in_seed false
24
+ include_time_in_seed false
25
+ use_record_as_seed false
26
+ use_entire_record false
27
+ record_keys []
28
+ separator _
29
+ hash_type sha1
30
+ </filter>
31
+ ```
32
+
33
+ ## Configuration
34
+
35
+ ### hash_id_key
36
+
37
+ ```
38
+ hash_id_key _id
39
+ ```
40
+
41
+ You can specify generated hash storing key.
42
+
43
+ ### include_tag_in_seed
44
+
45
+ ```
46
+ include_tag_in_seed true
47
+ ```
48
+
49
+ You can specify to use tag for hash generation seed.
50
+
51
+ ### include_time_in_seed
52
+
53
+ ```
54
+ include_time_in_seed true
55
+ ```
56
+
57
+ You can specify to use time for hash generation seed.
58
+
59
+ ### use_record_as_seed
60
+
61
+ ```
62
+ use_record_as_seed true
63
+ ```
64
+
65
+ You can specify to use record in events for hash generation seed. This parameter should be used with [record_keys](#record_keys) parameter in practice.
66
+
67
+ ### record_keys
68
+
69
+ ```
70
+ record_keys request_id,pipeline_id
71
+ ```
72
+
73
+ You can specify keys which are record in events for hash generation seed. This parameter should be used with [use_record_as_seed](#use_record_as_seed) parameter in practice.
74
+
75
+ ### use_entire_record
76
+
77
+ ```
78
+ use_entire_record true
79
+ ```
80
+
81
+ You can specify to use entire record in events for hash generation seed.
82
+
83
+
84
+ ### separator
85
+
86
+ ```
87
+ separator |
88
+ ```
89
+
90
+ You can specify separator charactor to creating seed for hash generation.
91
+
92
+ ### hash_type
93
+
94
+ ```
95
+ hash_type sha1
96
+ ```
97
+
98
+ You can specify hash algorithm. Support algorithms `md5`, `sha1`, `sha256`, `sha512`. Default: `sha1`
99
+
100
+ ## Advanced Usage
101
+
102
+ Opensearch GenID plugin can handle record contents differing with the following parameters:
103
+
104
+ ```aconf
105
+ <filter the.awesome.your.routing.tag>
106
+ @type opensearch_genid
107
+ use_entire_record true
108
+ hash_type sha1
109
+ hash_id_key _hash
110
+ separator _
111
+ inc_time_as_key true
112
+ inc_tag_as_key true
113
+ </filter>
114
+ ```
115
+
116
+ The above configuration can handle tag, time, and record differing and generate different base64 encoded hash per record.