fluent-plugin-opensearch 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (52) hide show
  1. checksums.yaml +7 -0
  2. data/.coveralls.yml +1 -0
  3. data/.editorconfig +9 -0
  4. data/.github/ISSUE_TEMPLATE/bug_report.md +37 -0
  5. data/.github/ISSUE_TEMPLATE/feature_request.md +24 -0
  6. data/.github/workflows/coverage.yaml +22 -0
  7. data/.github/workflows/issue-auto-closer.yml +12 -0
  8. data/.github/workflows/linux.yml +26 -0
  9. data/.github/workflows/macos.yml +26 -0
  10. data/.github/workflows/windows.yml +26 -0
  11. data/.gitignore +18 -0
  12. data/CONTRIBUTING.md +24 -0
  13. data/Gemfile +10 -0
  14. data/History.md +6 -0
  15. data/ISSUE_TEMPLATE.md +26 -0
  16. data/LICENSE.txt +201 -0
  17. data/PULL_REQUEST_TEMPLATE.md +9 -0
  18. data/README.OpenSearchGenID.md +116 -0
  19. data/README.OpenSearchInput.md +291 -0
  20. data/README.Troubleshooting.md +482 -0
  21. data/README.md +1556 -0
  22. data/Rakefile +37 -0
  23. data/fluent-plugin-opensearch.gemspec +38 -0
  24. data/gemfiles/Gemfile.elasticsearch.v6 +12 -0
  25. data/lib/fluent/log-ext.rb +64 -0
  26. data/lib/fluent/plugin/filter_opensearch_genid.rb +103 -0
  27. data/lib/fluent/plugin/in_opensearch.rb +351 -0
  28. data/lib/fluent/plugin/oj_serializer.rb +48 -0
  29. data/lib/fluent/plugin/opensearch_constants.rb +39 -0
  30. data/lib/fluent/plugin/opensearch_error.rb +31 -0
  31. data/lib/fluent/plugin/opensearch_error_handler.rb +166 -0
  32. data/lib/fluent/plugin/opensearch_fallback_selector.rb +36 -0
  33. data/lib/fluent/plugin/opensearch_index_template.rb +155 -0
  34. data/lib/fluent/plugin/opensearch_simple_sniffer.rb +36 -0
  35. data/lib/fluent/plugin/opensearch_tls.rb +96 -0
  36. data/lib/fluent/plugin/out_opensearch.rb +1124 -0
  37. data/lib/fluent/plugin/out_opensearch_data_stream.rb +214 -0
  38. data/test/helper.rb +61 -0
  39. data/test/plugin/test_alias_template.json +9 -0
  40. data/test/plugin/test_filter_opensearch_genid.rb +241 -0
  41. data/test/plugin/test_in_opensearch.rb +493 -0
  42. data/test/plugin/test_index_alias_template.json +11 -0
  43. data/test/plugin/test_index_template.json +25 -0
  44. data/test/plugin/test_oj_serializer.rb +45 -0
  45. data/test/plugin/test_opensearch_error_handler.rb +689 -0
  46. data/test/plugin/test_opensearch_fallback_selector.rb +100 -0
  47. data/test/plugin/test_opensearch_tls.rb +171 -0
  48. data/test/plugin/test_out_opensearch.rb +3953 -0
  49. data/test/plugin/test_out_opensearch_data_stream.rb +474 -0
  50. data/test/plugin/test_template.json +23 -0
  51. data/test/test_log-ext.rb +61 -0
  52. metadata +262 -0
@@ -0,0 +1,291 @@
1
+ ## Index
2
+
3
+ * [Installation](#installation)
4
+ * [Usage](#usage)
5
+ * [Configuration](#configuration)
6
+ + [host](#host)
7
+ + [port](#port)
8
+ + [hosts](#hosts)
9
+ + [user, password, path, scheme, ssl_verify](#user-password-path-scheme-ssl_verify)
10
+ + [parse_timestamp](#parse_timestamp)
11
+ + [timestampkey_format](#timestampkey_format)
12
+ + [timestamp_key](#timestamp_key)
13
+ + [timestamp_parse_error_tag](#timestamp_parse_error_tag)
14
+ + [http_backend](#http_backend)
15
+ + [request_timeout](#request_timeout)
16
+ + [reload_connections](#reload_connections)
17
+ + [reload_on_failure](#reload_on_failure)
18
+ + [resurrect_after](#resurrect_after)
19
+ + [with_transporter_log](#with_transporter_log)
20
+ + [Client/host certificate options](#clienthost-certificate-options)
21
+ + [sniffer_class_name](#sniffer-class-name)
22
+ + [custom_headers](#custom_headers)
23
+ + [docinfo_fields](#docinfo_fields)
24
+ + [docinfo_target](#docinfo_target)
25
+ + [docinfo](#docinfo)
26
+ * [Advanced Usage](#advanced-usage)
27
+
28
+ ## Usage
29
+
30
+ In your Fluentd configuration, use `@type opensearch` and specify `tag your.awesome.tag`. Additional configuration is optional, default values would look like this:
31
+
32
+ ```
33
+ <source>
34
+ @type opensearch
35
+ host localhost
36
+ port 9200
37
+ index_name fluentd
38
+ type_name fluentd
39
+ tag my.logs
40
+ </source>
41
+ ```
42
+
43
+ ## Configuration
44
+
45
+ ### host
46
+
47
+ ```
48
+ host user-custom-host.domain # default localhost
49
+ ```
50
+
51
+ You can specify OpenSearch host by this parameter.
52
+
53
+
54
+ ### port
55
+
56
+ ```
57
+ port 9201 # defaults to 9200
58
+ ```
59
+
60
+ You can specify OpenSearch port by this parameter.
61
+
62
+ ### hosts
63
+
64
+ ```
65
+ hosts host1:port1,host2:port2,host3:port3
66
+ ```
67
+
68
+ You can specify multiple OpenSearch hosts with separator ",".
69
+
70
+ If you specify multiple hosts, this plugin will load balance updates to OpenSearch. This is an [opensearch-ruby](https://github.com/opensearch/opensearch-ruby) feature, the default strategy is round-robin.
71
+
72
+ If you specify `hosts` option, `host` and `port` options are ignored.
73
+
74
+ ```
75
+ host user-custom-host.domain # ignored
76
+ port 9200 # ignored
77
+ hosts host1:port1,host2:port2,host3:port3
78
+ ```
79
+
80
+ If you specify `hosts` option without port, `port` option is used.
81
+
82
+ ```
83
+ port 9200
84
+ hosts host1:port1,host2:port2,host3 # port3 is 9200
85
+ ```
86
+
87
+ **Note:** If you will use scheme https, do not include "https://" in your hosts ie. host "https://domain", this will cause ES cluster to be unreachable and you will receive an error "Can not reach OpenSearch cluster"
88
+
89
+ ### user, password, path, scheme, ssl_verify
90
+
91
+ ```
92
+ user demo
93
+ password secret
94
+ path /elastic_search/
95
+ scheme https
96
+ ```
97
+
98
+ You can specify user and password for HTTP Basic authentication.
99
+
100
+ And this plugin will escape required URL encoded characters within `%{}` placeholders.
101
+
102
+ ```
103
+ user %{demo+}
104
+ password %{@secret}
105
+ ```
106
+
107
+ Specify `ssl_verify false` to skip ssl verification (defaults to true)
108
+
109
+ ### parse_timestamp
110
+
111
+ ```
112
+ parse_timestamp true # defaults to false
113
+ ```
114
+
115
+ Parse a `@timestamp` field and add parsed time to the event.
116
+
117
+ ### timestamp_key_format
118
+
119
+ The format of the time stamp field (`@timestamp` or what you specify in OpenSearch). This parameter only has an effect when [parse_timestamp](#parse_timestamp) is true as it only affects the name of the index we write to. Please see [Time#strftime](http://ruby-doc.org/core-1.9.3/Time.html#method-i-strftime) for information about the value of this format.
120
+
121
+ Setting this to a known format can vastly improve your log ingestion speed if all most of your logs are in the same format. If there is an error parsing this format the timestamp will default to the ingestion time. If you are on Ruby 2.0 or later you can get a further performance improvement by installing the "strptime" gem: `fluent-gem install strptime`.
122
+
123
+ For example to parse ISO8601 times with sub-second precision:
124
+
125
+ ```
126
+ timestamp_key_format %Y-%m-%dT%H:%M:%S.%N%z
127
+ ```
128
+
129
+ ### timestamp_parse_error_tag
130
+
131
+ With `parse_timestamp true`, opensearch input plugin parses timestamp field for consuming event time. If the consumed record has invalid timestamp value, this plugin emits an error event to `@ERROR` label with `timestamp_parse_error_tag` configured tag.
132
+
133
+ Default value is `opensearch_plugin.input.time.error`.
134
+
135
+ ### http_backend
136
+
137
+ With `http_backend typhoeus`, opensearch plugin uses typhoeus faraday http backend.
138
+ Typhoeus can handle HTTP keepalive.
139
+
140
+ Default value is `excon` which is default http_backend of opensearch plugin.
141
+
142
+ ```
143
+ http_backend typhoeus
144
+ ```
145
+
146
+
147
+ ### request_timeout
148
+
149
+ You can specify HTTP request timeout.
150
+
151
+ This is useful when OpenSearch cannot return response for bulk request within the default of 5 seconds.
152
+
153
+ ```
154
+ request_timeout 15s # defaults to 5s
155
+ ```
156
+
157
+ ### reload_connections
158
+
159
+ You can tune how the opensearch-transport host reloading feature works. By default it will reload the host list from the server every 10,000th request to spread the load. This can be an issue if your OpenSearch cluster is behind a Reverse Proxy, as Fluentd process may not have direct network access to the OpenSearch nodes.
160
+
161
+ ```
162
+ reload_connections false # defaults to true
163
+ ```
164
+
165
+ ### reload_on_failure
166
+
167
+ Indicates that the opensearch-transport will try to reload the nodes addresses if there is a failure while making the
168
+ request, this can be useful to quickly remove a dead node from the list of addresses.
169
+
170
+ ```
171
+ reload_on_failure true # defaults to false
172
+ ```
173
+
174
+ ### resurrect_after
175
+
176
+ You can set in the opensearch-transport how often dead connections from the opensearch-transport's pool will be resurrected.
177
+
178
+ ```
179
+ resurrect_after 5s # defaults to 60s
180
+ ```
181
+
182
+ ### with_transporter_log
183
+
184
+ This is debugging purpose option to enable to obtain transporter layer log.
185
+ Default value is `false` for backward compatibility.
186
+
187
+ We recommend to set this true if you start to debug this plugin.
188
+
189
+ ```
190
+ with_transporter_log true
191
+ ```
192
+
193
+ ### Client/host certificate options
194
+
195
+ Need to verify OpenSearch's certificate? You can use the following parameter to specify a CA instead of using an environment variable.
196
+ ```
197
+ ca_file /path/to/your/ca/cert
198
+ ```
199
+
200
+ Does your OpenSearch cluster want to verify client connections? You can specify the following parameters to use your client certificate, key, and key password for your connection.
201
+ ```
202
+ client_cert /path/to/your/client/cert
203
+ client_key /path/to/your/private/key
204
+ client_key_pass password
205
+ ```
206
+
207
+ If you want to configure SSL/TLS version, you can specify ssl\_version parameter.
208
+ ```
209
+ ssl_version TLSv1_2 # or [SSLv23, TLSv1, TLSv1_1]
210
+ ```
211
+
212
+ :warning: If SSL/TLS enabled, it might have to be required to set ssl\_version.
213
+
214
+ ### Sniffer Class Name
215
+
216
+ The default Sniffer used by the `OpenSearch::Transport` class works well when Fluentd has a direct connection
217
+ to all of the OpenSearch servers and can make effective use of the `_nodes` API. This doesn't work well
218
+ when Fluentd must connect through a load balancer or proxy. The parameter `sniffer_class_name` gives you the
219
+ ability to provide your own Sniffer class to implement whatever connection reload logic you require. In addition,
220
+ there is a new `Fluent::Plugin::OpenSearchSimpleSniffer` class which reuses the hosts given in the configuration, which
221
+ is typically the hostname of the load balancer or proxy. For example, a configuration like this would cause
222
+ connections to `logging-es` to reload every 100 operations:
223
+
224
+ ```
225
+ host logging-es
226
+ port 9200
227
+ reload_connections true
228
+ sniffer_class_name Fluent::Plugin::OpenSearchSimpleSniffer
229
+ reload_after 100
230
+ ```
231
+
232
+ ### custom_headers
233
+
234
+ This parameter adds additional headers to request. The default value is `{}`.
235
+
236
+ ```
237
+ custom_headers {"token":"secret"}
238
+ ```
239
+
240
+ ### docinfo_fields
241
+
242
+ This parameter specifies docinfo record keys. The default values are `['_index', '_type', '_id']`.
243
+
244
+ ```
245
+ docinfo_fields ['_index', '_id']
246
+ ```
247
+
248
+ ### docinfo_target
249
+
250
+ This parameter specifies docinfo storing key. The default value is `@metadata`.
251
+
252
+ ```
253
+ docinfo_target metadata
254
+ ```
255
+
256
+ ### docinfo
257
+
258
+ This parameter specifies whether docinfo information including or not. The default value is `false`.
259
+
260
+ ```
261
+ docinfo false
262
+ ```
263
+
264
+ ## Advanced Usage
265
+
266
+ OpenSearch Input plugin and OpenSearch output plugin can combine to transfer records into another cluster.
267
+
268
+ ```aconf
269
+ <source>
270
+ @type opensearch
271
+ host original-cluster.local
272
+ port 9200
273
+ tag raw.opensearch
274
+ index_name logstash-*
275
+ docinfo true
276
+ # repeat false
277
+ # num_slices 2
278
+ # with_transporter_log true
279
+ </source>
280
+ <match raw.opensearch>
281
+ @type opensearch
282
+ host transferred-cluster.local
283
+ port 9200
284
+ index_name ${$.@metadata._index}
285
+ type_name ${$.@metadata._type} # This parameter will be deprecated due to Removal of mapping types since ES7.
286
+ id_key ${$.@metadata._id} # This parameter is needed for prevent duplicated records.
287
+ <buffer tag, $.@metadata._index, $.@metadata._type, $.@metadata._id>
288
+ @type memory # should use file buffer for preventing chunk lost
289
+ </buffer>
290
+ </match>
291
+ ```