fluent-plugin-elasticsearch 1.9.4 → 5.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.github/ISSUE_TEMPLATE/bug_report.md +37 -0
- data/.github/ISSUE_TEMPLATE/feature_request.md +24 -0
- data/.github/workflows/issue-auto-closer.yml +12 -0
- data/.github/workflows/linux.yml +26 -0
- data/.github/workflows/macos.yml +26 -0
- data/.github/workflows/windows.yml +26 -0
- data/.travis.yml +33 -6
- data/CONTRIBUTING.md +24 -0
- data/Gemfile +4 -1
- data/History.md +445 -1
- data/ISSUE_TEMPLATE.md +19 -0
- data/README.ElasticsearchGenID.md +116 -0
- data/README.ElasticsearchInput.md +293 -0
- data/README.Troubleshooting.md +692 -0
- data/README.md +1013 -38
- data/appveyor.yml +20 -0
- data/fluent-plugin-elasticsearch.gemspec +15 -9
- data/{Gemfile.v0.12 → gemfiles/Gemfile.elasticsearch.v6} +6 -5
- data/lib/fluent/log-ext.rb +38 -0
- data/lib/fluent/plugin/default-ilm-policy.json +14 -0
- data/lib/fluent/plugin/elasticsearch_constants.rb +13 -0
- data/lib/fluent/plugin/elasticsearch_error.rb +5 -0
- data/lib/fluent/plugin/elasticsearch_error_handler.rb +129 -0
- data/lib/fluent/plugin/elasticsearch_fallback_selector.rb +9 -0
- data/lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb +67 -0
- data/lib/fluent/plugin/elasticsearch_index_template.rb +186 -12
- data/lib/fluent/plugin/elasticsearch_simple_sniffer.rb +10 -0
- data/lib/fluent/plugin/elasticsearch_tls.rb +70 -0
- data/lib/fluent/plugin/filter_elasticsearch_genid.rb +77 -0
- data/lib/fluent/plugin/in_elasticsearch.rb +325 -0
- data/lib/fluent/plugin/oj_serializer.rb +22 -0
- data/lib/fluent/plugin/out_elasticsearch.rb +1008 -267
- data/lib/fluent/plugin/out_elasticsearch_data_stream.rb +218 -0
- data/lib/fluent/plugin/out_elasticsearch_dynamic.rb +232 -214
- data/test/plugin/test_alias_template.json +9 -0
- data/test/plugin/test_elasticsearch_error_handler.rb +646 -0
- data/test/plugin/test_elasticsearch_fallback_selector.rb +74 -0
- data/test/plugin/test_elasticsearch_index_lifecycle_management.rb +66 -0
- data/test/plugin/test_elasticsearch_tls.rb +145 -0
- data/test/plugin/test_filter_elasticsearch_genid.rb +215 -0
- data/test/plugin/test_in_elasticsearch.rb +459 -0
- data/test/plugin/test_index_alias_template.json +11 -0
- data/test/plugin/test_index_template.json +25 -0
- data/test/plugin/test_oj_serializer.rb +19 -0
- data/test/plugin/test_out_elasticsearch.rb +5029 -387
- data/test/plugin/test_out_elasticsearch_data_stream.rb +337 -0
- data/test/plugin/test_out_elasticsearch_dynamic.rb +681 -208
- data/test/test_log-ext.rb +35 -0
- metadata +97 -19
data/History.md
CHANGED
|
@@ -1,8 +1,452 @@
|
|
|
1
1
|
## Changelog [[tags]](https://github.com/uken/fluent-plugin-elasticsearch/tags)
|
|
2
2
|
|
|
3
3
|
### [Unreleased]
|
|
4
|
-
-
|
|
4
|
+
- Fix use_legacy_template documentation (#880)
|
|
5
|
+
- Add FAQ for dynamic index/template (#878)
|
|
6
|
+
- Handle IPv6 address string on host and hosts parameters (#877)
|
|
7
|
+
|
|
8
|
+
### 5.0.2
|
|
9
|
+
- GitHub Actions: Tweak Ruby versions on test (#875)
|
|
10
|
+
- test: datastreams: Set nonexistent datastream as default (#874)
|
|
11
|
+
- Fix overwriting of index template and index lifecycle policy on existing data streams (#872)
|
|
12
|
+
|
|
13
|
+
### 5.0.1
|
|
14
|
+
- Use elasticsearch/api instead of elasticsearch/xpack (#870)
|
|
15
|
+
|
|
16
|
+
### 5.0.0
|
|
17
|
+
- Support #retry_operate on data stream (#863)
|
|
18
|
+
- Support placeholder in @data\_stream\_name for @type elasticsearch\_data\_stream (#862)
|
|
19
|
+
- Extract troubleshooting section (#861)
|
|
20
|
+
- Fix unmatched `<source>` close tag (#860)
|
|
21
|
+
- Initial support for Elasticsearch Data Stream (#859)
|
|
22
|
+
|
|
23
|
+
### 4.3.3
|
|
24
|
+
- Handle invalid Elasticsearch::Client#info response (#855)
|
|
25
|
+
|
|
26
|
+
### 4.3.2
|
|
27
|
+
- Use log.debug to dump ES 8.x _type information (#852)
|
|
28
|
+
|
|
29
|
+
### 4.3.1
|
|
30
|
+
- Unsplit huge records by default (#851)
|
|
31
|
+
|
|
32
|
+
### 4.3.0
|
|
33
|
+
- Add cloud_id & cloud_auth settings (#850)
|
|
34
|
+
- Fix failing tests on es6 client (#848)
|
|
35
|
+
- feat: retry on transport errors (#846)
|
|
36
|
+
- fix typo (#838)
|
|
37
|
+
- Remove unnecessary nil check (#826)
|
|
38
|
+
|
|
39
|
+
### 4.2.2
|
|
40
|
+
- Remove unnecessary nil check (#826)
|
|
41
|
+
- Fix ILM rollover index template pattern to apply index_separator (#825)
|
|
42
|
+
- Fix ILM rollover alias creation when a placeholder is used in index_name (#823)
|
|
43
|
+
- Add a note about the pitfalls of per-date indexes used with ILM (#822)
|
|
44
|
+
|
|
45
|
+
### 4.2.1
|
|
46
|
+
- Update a broken link (#821)
|
|
47
|
+
- Include chunk_id in records (#820)
|
|
48
|
+
- Fix a failing testcase (#818)
|
|
49
|
+
|
|
50
|
+
### 4.2.0
|
|
51
|
+
- ci: Add Ruby 2.7 jobs (#812)
|
|
52
|
+
- Support Elasticsearch new style template (#810)
|
|
53
|
+
|
|
54
|
+
### 4.1.4
|
|
55
|
+
- ElasticsearchGenID update docs for hash_type (#809)
|
|
56
|
+
- Handle api key header (#808)
|
|
57
|
+
- index_template: Handle error object entirely on index creation failure (#807)
|
|
58
|
+
|
|
59
|
+
### 4.1.3
|
|
60
|
+
- Load multiple templates even if template_name and template_file given (#799)
|
|
61
|
+
- Handle elasticsearch-ruby 7.9.0 using HTTP method changes (#795)
|
|
62
|
+
|
|
63
|
+
### 4.1.2
|
|
64
|
+
- Use Hash#dig instead of Hash#[] on retrieving version information from Elasticsearch info API (#793)
|
|
65
|
+
|
|
66
|
+
### 4.1.1
|
|
67
|
+
- Correct ILM explain on logstash_format case (#786)
|
|
68
|
+
|
|
69
|
+
### 4.1.0
|
|
70
|
+
- Implement Fallback selector and configurable selector class (#782)
|
|
71
|
+
|
|
72
|
+
### 4.0.11
|
|
73
|
+
- Add custom and time placeholders combination testcase for ILM (#781)
|
|
74
|
+
- Really support ILM on `logstash_format` enabled environment (#779)
|
|
75
|
+
|
|
76
|
+
### 4.0.10
|
|
77
|
+
- filter_elasticsearch_genid: Use entire record as hash seed (#777)
|
|
78
|
+
- Suppress type in meta with suppress_type_name parameter (#774)
|
|
79
|
+
- filter\_elasticsearch\_genid: Add hash generation mechanism from events (#773)
|
|
80
|
+
- Clean up error text (#772)
|
|
81
|
+
- Use GitHub Actions badges instead of Travis' (#760)
|
|
82
|
+
- Add issue auto closer workflow (#759)
|
|
83
|
+
- Document required permissions (#757)
|
|
84
|
+
|
|
85
|
+
### 4.0.9
|
|
86
|
+
- Add possibility to configure multiple ILM policies (#753)
|
|
87
|
+
- Document required permissions (#757)
|
|
88
|
+
|
|
89
|
+
### 4.0.8
|
|
90
|
+
- Handle compressable connection usable state (#743)
|
|
91
|
+
- Use newer tls protocol versions (#739)
|
|
92
|
+
- Add GitHub Actions file (#740)
|
|
93
|
+
|
|
94
|
+
### 4.0.7
|
|
95
|
+
- Added http_backend_excon_nonblock config in out_elasticsearch (#733)
|
|
96
|
+
|
|
97
|
+
### 4.0.6
|
|
98
|
+
- Add fallback mechanism for handling to detect es version (#730)
|
|
99
|
+
- Remove needless section (#728)
|
|
100
|
+
- Handle exception if index already exists (#727)
|
|
101
|
+
- Tweak test cases (#726)
|
|
102
|
+
|
|
103
|
+
### 4.0.5
|
|
104
|
+
- add logstash_dateformat as placeholder (#718)
|
|
105
|
+
- Tweak travis.yml for suppressing validator warnings and add CI for Linux Arm64 architecture and macOS 10.14 (#724)
|
|
106
|
+
- Elasticsearch ruby v7.5 (#723)
|
|
107
|
+
- Add Oj serializer testcases for all job (#722)
|
|
108
|
+
- Update documentation for ILM (#721)
|
|
109
|
+
|
|
110
|
+
### 4.0.4
|
|
111
|
+
- Provide clearing caches timer (#719)
|
|
112
|
+
|
|
113
|
+
### 4.0.3
|
|
114
|
+
- Use http.scheme settings for persisent scheme setup (#713)
|
|
115
|
+
|
|
116
|
+
### 4.0.2
|
|
117
|
+
- Support TLSv1.3 (#710)
|
|
118
|
+
|
|
119
|
+
### 4.0.1
|
|
120
|
+
- Placeholders for template name and customize template (#708)
|
|
121
|
+
- Add overwriting ilm policy config parameter (#707)
|
|
122
|
+
- Fix a failing ILM config testcase (#706)
|
|
123
|
+
|
|
124
|
+
### 4.0.0
|
|
125
|
+
- Restructuring ILM related features (#701)
|
|
126
|
+
- Extract placeholders in pipeline parameter (#695)
|
|
127
|
+
- fix typo in `README.md` (#698)
|
|
128
|
+
- Reduce log noise when not using rollover_index (#692)
|
|
129
|
+
|
|
130
|
+
### 3.8.0
|
|
131
|
+
- Add FAQ for specifying index.codec (#679)
|
|
132
|
+
- Add FAQ for connect_write timeout reached error (#687)
|
|
133
|
+
- Unblocking buffer overflow with block action (#688)
|
|
134
|
+
|
|
135
|
+
### 3.7.1
|
|
136
|
+
- Make conpatible for Fluentd v1.8 (#677)
|
|
137
|
+
- Handle flatten_hashes in elasticsearch_dynamic (#675)
|
|
138
|
+
- Handle empty index_date_pattern parameter (#674)
|
|
139
|
+
|
|
140
|
+
### 3.7.0
|
|
141
|
+
- Tweak for cosmetic change (#671)
|
|
142
|
+
- Fix access to Elasticsearch::Transport::VERSION with explicit top level class path (#670)
|
|
143
|
+
- Implement Elasticsearch Input plugin (#669)
|
|
144
|
+
|
|
145
|
+
### 3.6.1
|
|
146
|
+
- retry upsert on recoverable error. (#667)
|
|
147
|
+
- Allow `_index` in chunk_keys (#665)
|
|
148
|
+
- Support compression feature (#664)
|
|
149
|
+
|
|
150
|
+
### 3.6.0
|
|
151
|
+
- Set order in newly created templates (#660)
|
|
152
|
+
- Merge Support index lifecycle management into master (#659)
|
|
153
|
+
- Support template installation with host placeholder (#654)
|
|
154
|
+
- Support index lifecycle management (#651)
|
|
155
|
+
|
|
156
|
+
### 3.5.6
|
|
157
|
+
- Support elasticsearch8 removal of mapping types (#656)
|
|
158
|
+
- Upgrade webmock to 3 (#652)
|
|
159
|
+
- Suppress `ruby -c` warnings (#649)
|
|
160
|
+
- Add tips for sniffer class (#644)
|
|
161
|
+
- Make `client_key_pass` secret (#637)
|
|
162
|
+
- Validate `user` and `password` early (#636)
|
|
163
|
+
|
|
164
|
+
### 3.5.5
|
|
165
|
+
- Fix arguments order of `assert_equal` (#635)
|
|
166
|
+
- Add description for `port` option (#634)
|
|
167
|
+
- Fix description position and add examples for `hosts` option (#633)
|
|
168
|
+
- Use upper-case to compare before and after conversion (#630)
|
|
169
|
+
- Validate `max_retry_get_es_version` (#629)
|
|
170
|
+
- Remove unused getting plugin instance (#628)
|
|
171
|
+
- Fix error message for `max_retry_putting_template` (#627)
|
|
172
|
+
- Fix DST-breaking unit test (#623)
|
|
173
|
+
- Handle ClusterBlockException (#621)
|
|
174
|
+
- Add contribution guideline document (#618)
|
|
175
|
+
|
|
176
|
+
### 3.5.4
|
|
177
|
+
- Add FAQ for Fluentd seems to hang if it unable to connect Elasticsearch (#617)
|
|
178
|
+
- Check bulk_message size before appending (#616)
|
|
179
|
+
- Add FAQ for Elasticsearch index mapping glitch (#614)
|
|
180
|
+
- Display retry counts and interval (#613)
|
|
181
|
+
|
|
182
|
+
### 3.5.3
|
|
183
|
+
- Handle nil items response (#611)
|
|
184
|
+
|
|
185
|
+
### 3.5.2
|
|
186
|
+
- Fix `@meta_config_map` creation timing (#592)
|
|
187
|
+
|
|
188
|
+
### 3.5.1
|
|
189
|
+
- Configurable split request size threshold (#586)
|
|
190
|
+
|
|
191
|
+
### 3.5.0
|
|
192
|
+
- Adopt Elasticsearch ruby client v7 loggable class (#583)
|
|
193
|
+
|
|
194
|
+
### 3.4.3
|
|
195
|
+
- Add fail_on_putting_template_retry_exceed config (#579)
|
|
196
|
+
|
|
197
|
+
### 3.4.2
|
|
198
|
+
- Comparing DEFAULT_TYPE_NAME_ES_7x to target_type instead of type_name (#573)
|
|
199
|
+
|
|
200
|
+
### 3.4.1
|
|
201
|
+
- Handle non-String value on parse_time (#570)
|
|
202
|
+
|
|
203
|
+
### 3.4.0
|
|
204
|
+
- Check exclusive feature on #configure (#569)
|
|
205
|
+
- Modify FAQ for highly load k8s EFK stack (#566)
|
|
206
|
+
- Add FAQ for high load k8s EFK stack (#564)
|
|
207
|
+
|
|
208
|
+
### 3.3.3
|
|
209
|
+
- Add unit test for exception message (#563)
|
|
210
|
+
- Add ignore_exceptions config (#562)
|
|
211
|
+
|
|
212
|
+
### 3.3.2
|
|
213
|
+
- Fix support for host, hosts placeholders (#560)
|
|
214
|
+
- typo fixes in README.md (#559)
|
|
215
|
+
|
|
216
|
+
### 3.3.1
|
|
217
|
+
- add new option to suppress doc wrapping (#557)
|
|
218
|
+
- Include 2 (#555)
|
|
219
|
+
|
|
220
|
+
### 3.3.0
|
|
221
|
+
- Support builtin placeholders for host and hosts parameter (#554)
|
|
222
|
+
|
|
223
|
+
### 3.2.4
|
|
224
|
+
- Pass chunk for built in placeholders (#553)
|
|
225
|
+
|
|
226
|
+
### 3.2.3
|
|
227
|
+
- Expose exception backtrace for typhoeus gem loading error (#550)
|
|
228
|
+
|
|
229
|
+
### 3.2.2
|
|
230
|
+
- Don't validate ES cliuent version under dry-run mode (#547)
|
|
231
|
+
|
|
232
|
+
### 3.2.1
|
|
233
|
+
- Don't attempt to connect to Elasticsearch in dry run mode (#543)
|
|
234
|
+
- Add FAQ for typhoeus gem installation (#544)
|
|
235
|
+
|
|
236
|
+
### 3.2.0
|
|
237
|
+
- Split huge record requests (#539)
|
|
238
|
+
|
|
239
|
+
### 3.1.1
|
|
240
|
+
- Add document for custom_headers (#538)
|
|
241
|
+
- out_elasticsearch: Add custom_headers parameter (#529)
|
|
242
|
+
- Bundle irb on Ruby 2.6 or later (#537)
|
|
243
|
+
|
|
244
|
+
### 3.1.0
|
|
245
|
+
- Retry obtaining Elasticsearch version (#532)
|
|
246
|
+
- Fix broken id links (#530)
|
|
247
|
+
|
|
248
|
+
### 3.0.2
|
|
249
|
+
- appveyor: Remove Ruby 2.1 CI targets on AppVeyor (#524)
|
|
250
|
+
- Follow removal of _routing field change on recent Elasticsearch (#523)
|
|
251
|
+
- Travis: Tweak to use Ruby versions (#522)
|
|
252
|
+
|
|
253
|
+
### 3.0.1
|
|
254
|
+
- Remove needless Elasticsearch version detection (#520)
|
|
255
|
+
|
|
256
|
+
### 3.0.0
|
|
257
|
+
- Use fluentd core retry mechanism (#519)
|
|
258
|
+
- Depends on builtin retrying mechanism (#518)
|
|
259
|
+
- Loosen ConnectionRetryFailure condition when flush_thread_count > 1 and depends on Fluentd core retrying mechanism (#516)
|
|
260
|
+
|
|
261
|
+
### 2.12.5
|
|
262
|
+
- Ensure sniffer class constants definition before calling #client (#515)
|
|
263
|
+
|
|
264
|
+
### 2.12.4
|
|
265
|
+
- #506 Rollover index will be in effect in case of template overwrite also. (#513)
|
|
266
|
+
|
|
267
|
+
### 2.12.3
|
|
268
|
+
- Added log_es_400_reason configuration item (#511)
|
|
269
|
+
- Allow a user to specify the rollover index date pattern (#510)
|
|
270
|
+
|
|
271
|
+
### 2.12.2
|
|
272
|
+
- Verify connection at startup (#504)
|
|
273
|
+
- Add faq for glob pattern tag routing (#502)
|
|
274
|
+
|
|
275
|
+
### 2.12.1
|
|
276
|
+
- Make configurable unrecoverable types (#501)
|
|
277
|
+
- Add FAQ for TLS enabled nginx proxy TLS version incompatibility trouble (#496)
|
|
278
|
+
- Add FAQs (#492)
|
|
279
|
+
- Remove issuestats.com badges (#489)
|
|
280
|
+
|
|
281
|
+
### 2.12.0
|
|
282
|
+
- Decoupling the custom template and rollover index creation #485 (#486)
|
|
283
|
+
|
|
284
|
+
### 2.11.11
|
|
285
|
+
- Handle error not to acquire version information (#479)
|
|
286
|
+
|
|
287
|
+
### 2.11.10
|
|
288
|
+
- Verbose error reason output (#469)
|
|
289
|
+
|
|
290
|
+
### 2.11.9
|
|
291
|
+
- Use ConnectionRetryFailure in plugin specific retrying for consistency (#468)
|
|
292
|
+
- Remove outdated generating hash_id_key code (#466)
|
|
293
|
+
- Tweak behavior for UnrecoverableError and #detect_es_major_version (#465)
|
|
294
|
+
|
|
295
|
+
### 2.11.8
|
|
296
|
+
- Serialize requests with Oj (#464)
|
|
297
|
+
|
|
298
|
+
### 2.11.7
|
|
299
|
+
- Add mechanism to detect ES and its client version mismatch (#463)
|
|
300
|
+
|
|
301
|
+
### 2.11.6
|
|
302
|
+
- 355 customize template (#431)
|
|
303
|
+
|
|
304
|
+
### 2.11.5
|
|
305
|
+
- Uplift Merge pull request #459 from richm/v0.12-simple-sniffer (#461)
|
|
306
|
+
|
|
307
|
+
### 2.11.4
|
|
308
|
+
- Persistent backend (#456)
|
|
309
|
+
|
|
310
|
+
### 2.11.3
|
|
311
|
+
- Implement the `include_index_in_url` option for out_elasticsearch (#451)
|
|
312
|
+
- Add an option `include_index_in_url` to allow URL-based conrtrols (#450)
|
|
313
|
+
|
|
314
|
+
### 2.11.2
|
|
315
|
+
- Strictness scheme (#445)
|
|
316
|
+
|
|
317
|
+
### 2.11.0
|
|
318
|
+
- Uplift Merge pull request #437 from jcantrill/fix_bulk_count (#438)
|
|
319
|
+
|
|
320
|
+
### 2.10.5
|
|
321
|
+
- Uplift Merge pull request #435 from jcantrill/add_trace_logging (#436)
|
|
322
|
+
|
|
323
|
+
### 2.10.4
|
|
324
|
+
- Use Fluent::UnrecoverableError as unrecoverable error class ancestors (#433)
|
|
325
|
+
- Add parameter validation for retrying template installation (#429)
|
|
326
|
+
|
|
327
|
+
### 2.10.3
|
|
328
|
+
- Add retry mechanism for template installation (#428)
|
|
329
|
+
|
|
330
|
+
### 2.10.2
|
|
331
|
+
- Use logstash_prefix_separator on elasticsearch_dynamic (#427)
|
|
332
|
+
|
|
333
|
+
### 2.10.1
|
|
334
|
+
- Uplift Merge pull request #419 from jcantrill/retry_prefix (#421)
|
|
335
|
+
- Uplift Merge pull request #418 from jcantrill/emit_exception (#420)
|
|
336
|
+
|
|
337
|
+
### 2.10.0
|
|
338
|
+
- Uplift Merge pull request #405 from jcantrill/sanitize_bulk (#414)
|
|
339
|
+
|
|
340
|
+
### 2.9.2
|
|
341
|
+
- Uplift Merge pull request #410 from richm/v0.12-consistent-errors-and-tests (#411)
|
|
342
|
+
- Add correct link for include_timestamp (#408)
|
|
343
|
+
|
|
344
|
+
### 2.9.1
|
|
345
|
+
- Uplift Merge pull request #406 from richm/v0.12-successes-duplicates-no-retry (#407)
|
|
346
|
+
|
|
347
|
+
### 2.9.0
|
|
348
|
+
- DLQ revisited v1 uplifted #398, #402 (#404)
|
|
349
|
+
|
|
350
|
+
### 2.8.6
|
|
351
|
+
- auth: Fix missing auth tokens after reloading connections (#394)
|
|
352
|
+
|
|
353
|
+
### 2.8.5
|
|
354
|
+
- Add deprecated option into content_type parameter (#391)
|
|
355
|
+
|
|
356
|
+
### 2.8.4
|
|
357
|
+
- Use nanosecond precision in elasticsearch_dynamic (#387)
|
|
358
|
+
|
|
359
|
+
### 2.8.3
|
|
360
|
+
- Specify SSL/TLS version in out_elasticsearch_dynamic (#385)
|
|
361
|
+
|
|
362
|
+
### 2.8.2
|
|
363
|
+
- Revert content type header default value (#383)
|
|
364
|
+
|
|
365
|
+
### 2.8.1
|
|
366
|
+
- Restore default value of type name #(377)
|
|
367
|
+
|
|
368
|
+
### 2.8.0
|
|
369
|
+
- Remove outdated generating hash id support module (#373)
|
|
370
|
+
- Check Elasticsearch major version (#371)
|
|
371
|
+
|
|
372
|
+
### 2.7.0
|
|
373
|
+
- Configureable content type (#367)
|
|
374
|
+
|
|
375
|
+
### 2.6.1
|
|
376
|
+
- Prevent error when using template in elasticsearch_dynamic for elementally use case (#363)
|
|
377
|
+
|
|
378
|
+
### 2.6.0
|
|
379
|
+
- Handle high precision time format when using custom time_key (#360)
|
|
380
|
+
|
|
381
|
+
### 2.5.0
|
|
382
|
+
- Using nested record in `id_key`, `parent_key`, and `routing_key` (#351)
|
|
383
|
+
- Fix inverted case of a proper noun "Elasticsearch" (#349)
|
|
384
|
+
|
|
385
|
+
### 2.4.1
|
|
386
|
+
- Add config parameter to enable elasticsearch-ruby's transporter logging (#342)
|
|
387
|
+
|
|
388
|
+
### 2.4.0
|
|
389
|
+
- Add built-in placeholders support against type_name parameter (#338)
|
|
390
|
+
|
|
391
|
+
### 2.3.0
|
|
392
|
+
- Allow overwriting existing index template (#239)
|
|
393
|
+
|
|
394
|
+
### 2.2.0
|
|
395
|
+
- GA release 2.2.0.
|
|
396
|
+
|
|
397
|
+
### 2.2.0.rc.1
|
|
398
|
+
- Separate generate hash id module and bundled new plugin for generating unique hash id (#331)
|
|
399
|
+
|
|
400
|
+
### 2.1.1
|
|
401
|
+
- Raise ConfigError when specifying different @hash_config.hash_id_key and id_key configration (#327)
|
|
402
|
+
- Small typo fix in README.md (#325)
|
|
403
|
+
|
|
404
|
+
### 2.1.0
|
|
405
|
+
- Retry on certain errors from Elasticsearch (#322)
|
|
406
|
+
|
|
407
|
+
### 2.0.1
|
|
408
|
+
- Releasing generating hash id mechanism to avoid records duplication feature.
|
|
409
|
+
|
|
410
|
+
### 2.0.1.rc.1
|
|
411
|
+
- Add generating hash id mechanism to avoid records duplication (#318)
|
|
412
|
+
|
|
413
|
+
### 2.0.0
|
|
414
|
+
- Release for Fluentd v0.14 stable.
|
|
415
|
+
|
|
416
|
+
### 2.0.0.rc.7
|
|
417
|
+
- Add `include_timestamp` option (#310)
|
|
418
|
+
|
|
419
|
+
### 2.0.0.rc.6
|
|
420
|
+
- Improve documentation (#304)
|
|
421
|
+
- Handle dynamic_config misconfigurations (#305)
|
|
422
|
+
- Escape basic authentication user information placeholders (#306)
|
|
423
|
+
|
|
424
|
+
### 2.0.0.rc.5
|
|
425
|
+
- make configurable with `ssl_version` parameter (#299)
|
|
426
|
+
- add `logstash_prefix_separator` config parameter (#297)
|
|
427
|
+
|
|
428
|
+
### 2.0.0.rc.4
|
|
429
|
+
- fix license identifier in gemspec (#294)
|
|
430
|
+
|
|
431
|
+
### 2.0.0.rc.3
|
|
432
|
+
- add built-in placeholders support (#288, #293)
|
|
433
|
+
- permit multi workers feature (#291)
|
|
434
|
+
|
|
435
|
+
### 2.0.0.rc.2
|
|
436
|
+
- add pipeline parameter (#290)
|
|
437
|
+
|
|
438
|
+
### 2.0.0.rc.1
|
|
439
|
+
- Use v0.14 API to support nanosecond precision (#223)
|
|
440
|
+
|
|
441
|
+
### 1.9.5
|
|
442
|
+
- sub-second time precision [(#249)](https://github.com/uken/fluent-plugin-elasticsearch/pull/249)
|
|
443
|
+
|
|
444
|
+
### 1.9.4
|
|
445
|
+
- Include 'Content-Type' header in `transport_options`
|
|
446
|
+
|
|
447
|
+
### 1.9.3
|
|
5
448
|
- Use latest elasticsearch-ruby (#240)
|
|
449
|
+
- Log ES response errors (#230)
|
|
6
450
|
|
|
7
451
|
### 1.9.2
|
|
8
452
|
- Fix elasticsearch_dynamic for v0.14 (#224)
|
data/ISSUE_TEMPLATE.md
CHANGED
|
@@ -1,3 +1,6 @@
|
|
|
1
|
+
(check apply)
|
|
2
|
+
- [ ] read [the contribution guideline](https://github.com/uken/fluent-plugin-elasticsearch/blob/master/CONTRIBUTING.md)
|
|
3
|
+
|
|
1
4
|
#### Problem
|
|
2
5
|
|
|
3
6
|
...
|
|
@@ -9,3 +12,19 @@ Either clone and modify https://gist.github.com/pitr/9a518e840db58f435911
|
|
|
9
12
|
**OR**
|
|
10
13
|
|
|
11
14
|
Provide example config and message
|
|
15
|
+
|
|
16
|
+
#### Expected Behavior or What you need to ask
|
|
17
|
+
|
|
18
|
+
...
|
|
19
|
+
|
|
20
|
+
#### Using Fluentd and ES plugin versions
|
|
21
|
+
|
|
22
|
+
* OS version
|
|
23
|
+
* Bare Metal or within Docker or Kubernetes or others?
|
|
24
|
+
* Fluentd v0.12 or v0.14/v1.0
|
|
25
|
+
* paste result of ``fluentd --version`` or ``td-agent --version``
|
|
26
|
+
* ES plugin 3.x.y/2.x.y or 1.x.y
|
|
27
|
+
* paste boot log of fluentd or td-agent
|
|
28
|
+
* paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
|
|
29
|
+
* ES version (optional)
|
|
30
|
+
* ES template(s) (optional)
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
## Index
|
|
2
|
+
|
|
3
|
+
* [Usage](#usage)
|
|
4
|
+
* [Configuration](#configuration)
|
|
5
|
+
+ [hash_id_key](#hash_id_key)
|
|
6
|
+
+ [include_tag_in_seed](#include_tag_in_seed)
|
|
7
|
+
+ [include_time_in_seed](#include_time_in_seed)
|
|
8
|
+
+ [use_record_as_seed](#use_record_as_seed)
|
|
9
|
+
+ [use_entire_record](#use_entire_record)
|
|
10
|
+
+ [record_keys](#record_keys)
|
|
11
|
+
+ [separator](#separator)
|
|
12
|
+
+ [hash_type](#hash_type)
|
|
13
|
+
* [Advanced Usage](#advanced-usage)
|
|
14
|
+
|
|
15
|
+
## Usage
|
|
16
|
+
|
|
17
|
+
In your Fluentd configuration, use `@type elasticsearch_genid`. Additional configuration is optional, default values would look like this:
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
<filter>
|
|
21
|
+
@type elasticsearch_genid
|
|
22
|
+
hash_id_key _hash
|
|
23
|
+
include_tag_in_seed false
|
|
24
|
+
include_time_in_seed false
|
|
25
|
+
use_record_as_seed false
|
|
26
|
+
use_entire_record false
|
|
27
|
+
record_keys []
|
|
28
|
+
separator _
|
|
29
|
+
hash_type sha1
|
|
30
|
+
</filter>
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Configuration
|
|
34
|
+
|
|
35
|
+
### hash_id_key
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
hash_id_key _id
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
You can specify generated hash storing key.
|
|
42
|
+
|
|
43
|
+
### include_tag_in_seed
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
include_tag_in_seed true
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
You can specify to use tag for hash generation seed.
|
|
50
|
+
|
|
51
|
+
### include_time_in_seed
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
include_time_in_seed true
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
You can specify to use time for hash generation seed.
|
|
58
|
+
|
|
59
|
+
### use_record_as_seed
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
use_record_as_seed true
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
You can specify to use record in events for hash generation seed. This parameter should be used with [record_keys](#record_keys) parameter in practice.
|
|
66
|
+
|
|
67
|
+
### record_keys
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
record_keys request_id,pipeline_id
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
You can specify keys which are record in events for hash generation seed. This parameter should be used with [use_record_as_seed](#use_record_as_seed) parameter in practice.
|
|
74
|
+
|
|
75
|
+
### use_entire_record
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
use_entire_record true
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
You can specify to use entire record in events for hash generation seed.
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
### separator
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
separator |
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
You can specify separator charactor to creating seed for hash generation.
|
|
91
|
+
|
|
92
|
+
### hash_type
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
hash_type sha1
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
You can specify hash algorithm. Support algorithms `md5`, `sha1`, `sha256`, `sha512`. Default: `sha1`
|
|
99
|
+
|
|
100
|
+
## Advanced Usage
|
|
101
|
+
|
|
102
|
+
Elasticsearch GenID plugin can handle record contents differing with the following parameters:
|
|
103
|
+
|
|
104
|
+
```aconf
|
|
105
|
+
<filter the.awesome.your.routing.tag>
|
|
106
|
+
@type elasticsearch_genid
|
|
107
|
+
use_entire_record true
|
|
108
|
+
hash_type sha1
|
|
109
|
+
hash_id_key _hash
|
|
110
|
+
separator _
|
|
111
|
+
inc_time_as_key true
|
|
112
|
+
inc_tag_as_key true
|
|
113
|
+
</filter>
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
The above configuration can handle tag, time, and record differing and generate different base64 encoded hash per record.
|