fluent-plugin-elasticsearch 4.1.1 → 5.4.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 616365bcf32cc6d302718907d145639cbb4786b33cb8a837da24f3934cf7a837
-  data.tar.gz: 77df51b64aa07d12b57b82a093f4a5b8cbd1c1bf9c5c82384163118c050c68cf
+  metadata.gz: 542a7706becd1b24f8d3e6ad8957015dba5f721b3a93047ec4bea80313df8200
+  data.tar.gz: 8104645d96578abf0f4357a22c83a7f9cbc548e7bbc35ef5eeb8726deaada6f8
 SHA512:
-  metadata.gz: ae68cc2f0a2abbf7cf7a4c11a9bb5dac9a9e09ba9c053109748c671acd304198b45ad949782691a28eadb418e15bc66f1ef16897f810c5f0b5dd0ce04dd7913e
-  data.tar.gz: 484df0670840045d6c55bd1971142cf13a2a0c35cefc0ff96ada3259404d08529d97adbdd736bf9eaf99ad84c51d2124af1d0160c35042a736c875eef8aa5f3c
+  metadata.gz: 93a2a932a766170b197f3ded762c89912f743cec3ddcef3e7719d8ca57b504f6f7b51bc17bac02ef5073fd1fade3366140a7d775351c8b86b1f05693a76be122
+  data.tar.gz: 35b82c172909a1f47dc74507c7ea9df451576b9cb1c8dcba85fdb094b3568b2187aafa1aecbd3367ef2ab0a347a30f90d8b663780e3261bdefc5dcb2b24f0133

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,37 @@
+---
+name:  Bug Report
+about: Create a report to help us improve. If you have questions about ES plugin on kubernetes, please direct these to https://discuss.kubernetes.io/ before sumbit kubernetes related issue.
+
+---
+
+(check apply)
+- [ ] read [the contribution guideline](https://github.com/uken/fluent-plugin-elasticsearch/blob/master/CONTRIBUTING.md)
+- [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
+
+#### Problem
+
+...
+
+#### Steps to replicate
+
+Either clone and modify https://gist.github.com/pitr/9a518e840db58f435911
+
+**OR**
+
+Provide example config and message
+
+#### Expected Behavior or What you need to ask
+
+...
+
+#### Using Fluentd and ES plugin versions
+
+* OS version
+* Bare Metal or within Docker or Kubernetes or others?
+* Fluentd v0.12 or v0.14/v1.0
+  * paste result of ``fluentd --version`` or ``td-agent --version``
+* ES plugin 3.x.y/2.x.y or 1.x.y
+  * paste boot log of fluentd or td-agent
+  * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
+* ES version (optional)
+* ES template(s) (optional)

data/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,24 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+(check apply)
+- [ ] read [the contribution guideline](https://github.com/uken/fluent-plugin-elasticsearch/blob/master/CONTRIBUTING.md)
+
+**Is your feature request related to a problem? Please describe.**
+
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
+
+**Describe the solution you'd like**
+
+<!-- A clear and concise description of what you want to happen. -->
+
+**Describe alternatives you've considered**
+
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
+
+**Additional context**
+
+<!-- Add any other context or screenshots about the feature request here. -->

data/.github/dependabot.yml

@@ -0,0 +1,6 @@
+version: 2
+updates:
+- package-ecosystem: "github-actions"
+  directory: "/"
+  schedule:
+      interval: "weekly"

data/.github/workflows/issue-auto-closer.yml

@@ -5,8 +5,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Autoclose issues that did not follow issue template
-      uses: roots/issue-closer-action@v1.1
+      uses: roots/issue-closer-action@v1.2
       with:
         repo-token: ${{ secrets.GITHUB_TOKEN }}
         issue-close-message: "@${issue.user.login} this issue was automatically closed because it did not follow the issue template."
-        issue-pattern: "(.*Problem.*)|(.*Expected Behavior or What you need to ask.*)|(.*Using Fluentd and ES plugin versions.*)"
+        issue-pattern: "(.*Problem.*)|(.*Expected Behavior or What you need to ask.*)|(.*Using Fluentd and ES plugin versions.*)|(.*Is your feature request related to a problem? Please describe.*)|(.*Describe the solution you'd like.*)|(.*Describe alternatives you've considered.*)"

data/.github/workflows/linux.yml

@@ -2,18 +2,21 @@ name: Testing on Ubuntu
 on:
   - push
   - pull_request
+permissions:
+  contents: read
+
 jobs:
   build:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        ruby: [ '2.4', '2.5', '2.6' ]
+        ruby: [ '3.0', '3.1', '3.2' ]
         os:
           - ubuntu-latest
     name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - uses: ruby/setup-ruby@v1
       with:
         ruby-version: ${{ matrix.ruby }}

data/.github/workflows/macos.yml

@@ -2,18 +2,21 @@ name: Testing on macOS
 on:
   - push
   - pull_request
+permissions:
+  contents: read
+
 jobs:
   build:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        ruby: [ '2.4', '2.5', '2.6' ]
+        ruby: [ '3.0', '3.1', '3.2' ]
         os:
           - macOS-latest
     name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - uses: ruby/setup-ruby@v1
       with:
         ruby-version: ${{ matrix.ruby }}

data/.github/workflows/windows.yml

@@ -2,18 +2,21 @@ name: Testing on Windows
 on:
   - push
   - pull_request
+permissions:
+  contents: read
+
 jobs:
   build:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        ruby: [ '2.4', '2.5', '2.6' ]
+        ruby: [ '3.0', '3.1', '3.2' ]
         os:
           - windows-latest
     name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - uses: ruby/setup-ruby@v1
       with:
         ruby-version: ${{ matrix.ruby }}

data/Gemfile

@@ -4,8 +4,7 @@ source 'https://rubygems.org'
 gemspec
 
 gem 'simplecov', require: false
-gem 'coveralls', ">= 0.8.0", require: false
 gem 'strptime', require: false if RUBY_ENGINE == "ruby" && RUBY_VERSION =~ /^2/
 gem "irb" if RUBY_ENGINE == "ruby" && RUBY_VERSION >= "2.6"
-gem "elasticsearch-xpack"
+gem "elasticsearch-xpack" if ENV["USE_XPACK"]
 gem "oj"

data/History.md

@@ -2,6 +2,152 @@
 
 ### [Unreleased]
 
+### 5.4.3
+- add data_stream_template\_use\_index\_patterns\_wildcard in elasticsearch\_data\_stream (#1040)
+
+### 5.4.2
+- in\_elasticsearch: Avoid to use deprecated endpoint for clear\_scroll (#1039)
+
+### 5.4.1
+- Adjust ilm endpoint (#1036)
+
+### 5.4.0
+- Fix support for host/hosts placeholders for ipv6 addresses (#1030)
+- Handle newer es library (#1032)
+
+### 5.3.0
+
+- Unpin `faraday` from v1, upgrade to v2. (#1012)
+  Note that if you can't migrate other plugins from `faraday` v1 yet, need to keep
+  fluent-plugin-elasticsearch v5.2.5.
+
+### 5.2.5
+
+- Update the name of parameter about `thread_pool`.
+  It should be `thread_pool.write.queue_size` instead of `thread_pool.bulk.queue_size` (#1000)
+- Replace obsolete `File.exists?` with `File.exist?` (#998)
+- Support `compression_level` for `elasticsearch_data_stream` (#995)
+- Honor `hosts` parameter for `elasticsearch_data_stream` (#1008)
+- Drop outdated coveralls (#1009)
+
+### 5.2.4
+- Pin Faraday 1.10 (#987)
+- Increase errors metric on error response in data stream (#986)
+
+### 5.2.3
+- Bump actions/checkout from 2 to 3 (#978)
+- chore: Included githubactions in the dependabot config (#977)
+- chore: Set permissions for GitHub actions (#972)
+- Remove nested msgpack\_each in handle\_error (#970)
+- do not overwrite @timestamp in data stream if it already exists in the record (#968)
+
+### 5.2.2
+-  Add missing top level class markers (#961)
+- Ensure use_record_as_seed for same records (#960)
+
+### 5.2.1
+- respect include\_tag\_key and tag\_key setting when using data streams (#936)
+- Handle unsupported version error (#956)
+- Display deprecated warning on ES dynamic plugin (#955)
+
+### 5.2.0
+-  Migrate to handle Elasticsearch 8 (#949)
+
+### 5.1.5
+- Make retryable DataStreams creation at configure phase (#943)
+- Handle @hosts parameter on data_stream plugin (#942)
+- allow specifying custom ILM policies for data streams (#933)
+
+### 5.1.4
+- Handle ES8 or above more strictly (#931)
+- fixing double "\_policy" in index lifecycle management policy for elasticsearch\_data\_stream output (#930)
+
+### 5.1.3
+-  fixing execution order for dynamic data stream creation (#928)
+
+### 5.1.2
+-  Fix default values of datastream parameters (#926)
+
+### 5.1.1
+- Report appropriate error for data_stream parameters (#922)
+- Add ILM and template parameters for data streams (#920)
+- Support Buffer in Data Stream Output (#917)
+
+### 5.1.0
+- Correct default target bytes value (#914)
+- Handle elasticsearch-ruby 7.14 properly (#913)
+
+### 5.0.5
+- Drop json_parse_exception messages for bulk failures (#900)
+- GitHub Actions: Drop Ruby 2.5 due to EOL (#894)
+
+### 5.0.4
+- test: out_elasticsearch: Remove a needless headers from affinity stub (#888)
+- Target Index Affinity (#883)
+
+### 5.0.3
+- Fix use_legacy_template documentation (#880)
+- Add FAQ for dynamic index/template (#878)
+- Handle IPv6 address string on host and hosts parameters (#877)
+
+### 5.0.2
+- GitHub Actions: Tweak Ruby versions on test (#875)
+- test: datastreams: Set nonexistent datastream as default (#874)
+- Fix overwriting of index template and index lifecycle policy on existing data streams (#872)
+
+### 5.0.1
+- Use elasticsearch/api instead of elasticsearch/xpack (#870)
+
+### 5.0.0
+- Support #retry_operate on data stream (#863)
+- Support placeholder in @data\_stream\_name for @type elasticsearch\_data\_stream (#862)
+- Extract troubleshooting section (#861)
+- Fix unmatched `<source>` close tag (#860)
+- Initial support for Elasticsearch Data Stream (#859)
+
+### 4.3.3
+- Handle invalid Elasticsearch::Client#info response (#855)
+
+### 4.3.2
+- Use log.debug to dump ES 8.x _type information (#852)
+
+### 4.3.1
+- Unsplit huge records by default (#851)
+
+### 4.3.0
+- Add cloud_id & cloud_auth settings (#850)
+- Fix failing tests on es6 client (#848)
+- feat: retry on transport errors (#846)
+- fix typo (#838)
+- Remove unnecessary nil check (#826)
+
+### 4.2.2
+- Remove unnecessary nil check (#826)
+- Fix ILM rollover index template pattern to apply index_separator (#825)
+- Fix ILM rollover alias creation when a placeholder is used in index_name (#823)
+- Add a note about the pitfalls of per-date indexes used with ILM (#822)
+
+### 4.2.1
+- Update a broken link (#821)
+- Include chunk_id in records (#820)
+- Fix a failing testcase (#818)
+
+### 4.2.0
+- ci: Add Ruby 2.7 jobs (#812)
+- Support Elasticsearch new style template (#810)
+
+### 4.1.4
+- ElasticsearchGenID update docs for hash_type (#809)
+- Handle api key header (#808)
+- index_template: Handle error object entirely on index creation failure (#807)
+
+### 4.1.3
+- Load multiple templates even if template_name and template_file given (#799)
+- Handle elasticsearch-ruby 7.9.0 using HTTP method changes (#795)
+
+### 4.1.2
+- Use Hash#dig instead of Hash#[] on retrieving version information from Elasticsearch info API (#793)
+
 ### 4.1.1
 - Correct ILM explain on logstash_format case (#786)
 

data/README.ElasticsearchGenID.md

@@ -17,7 +17,7 @@
 In your Fluentd configuration, use `@type elasticsearch_genid`. Additional configuration is optional, default values would look like this:
 
 ```
-<source>
+<filter>
   @type elasticsearch_genid
   hash_id_key _hash
   include_tag_in_seed false
@@ -26,8 +26,8 @@ In your Fluentd configuration, use `@type elasticsearch_genid`. Additional confi
   use_entire_record false
   record_keys []
   separator _
-  hash_type md5
-</match>
+  hash_type sha1
+</filter>
 ```
 
 ## Configuration
@@ -95,7 +95,7 @@ You can specify separator charactor to creating seed for hash generation.
 hash_type sha1
 ```
 
-You can specify hash algorithm.
+You can specify hash algorithm. Support algorithms `md5`, `sha1`, `sha256`, `sha512`. Default: `sha1`
 
 ## Advanced Usage
 

data/README.ElasticsearchInput.md

@@ -37,7 +37,7 @@ In your Fluentd configuration, use `@type elasticsearch` and specify `tag your.a
   index_name fluentd
   type_name fluentd
   tag my.logs
-</match>
+</source>
 ```
 
 ## Configuration