test-opensearch 0.0.0.18 → 0.0.0.21
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.OpenSearchInput.md +86 -4
- data/fluent-plugin-opensearch.gemspec +1 -1
- data/lib/fluent/plugin/in_opensearch.rb +19 -25
- metadata +1 -1
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 833739abc81a6562071c9047e266225855f059853b5071cd11452cff393b4a1a
|
4
|
+
data.tar.gz: 75c81ae847a0d5a3e12d8e3e211a7614778ee419ee5ef10f8aa2c194b3063d6f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e6d93a2323842bf0258558d0da05da7bd76a3b95bfbbd191943cfc9c6203e131db9ee49e0455a7426c7626ca8668d4800c3cb0d3dc0b9ec283d171698b44b3d5
|
7
|
+
data.tar.gz: ce93e37096e8ebf4c502990fc172b1c6523b8ba6f285a8e3bc7e9bdb9ef219a45d4385dee1ab4b377e686e51e949990b8f6b95bf22c16e0cf49e20a312cb0e8d
|
data/README.OpenSearchInput.md
CHANGED
@@ -24,7 +24,16 @@
|
|
24
24
|
+ [docinfo_fields](#docinfo_fields)
|
25
25
|
+ [docinfo_target](#docinfo_target)
|
26
26
|
+ [docinfo](#docinfo)
|
27
|
-
+ [
|
27
|
+
+ [check_connection](#check_connection)
|
28
|
+
+ [retry_forever](#retry_forever)
|
29
|
+
+ [retry_timeout](#retry_timeout)
|
30
|
+
+ [retry_max_times](#retry_max_times)
|
31
|
+
+ [retry_type](#retry_type)
|
32
|
+
+ [retry_wait](#retry_wait)
|
33
|
+
+ [retry_exponential_backoff_base](#retry_exponential_backoff_base)
|
34
|
+
+ [retry_max_interval](#retry_max_interval)
|
35
|
+
+ [retry_randomize](#retry_randomize)
|
36
|
+
|
28
37
|
* [Advanced Usage](#advanced-usage)
|
29
38
|
|
30
39
|
## Usage
|
@@ -275,14 +284,87 @@ This parameter specifies whether docinfo information including or not. The defau
|
|
275
284
|
docinfo false
|
276
285
|
```
|
277
286
|
|
278
|
-
###
|
287
|
+
### check_connection
|
288
|
+
|
289
|
+
The parameter for checking on connection availability with Elasticsearch or Opensearch hosts. The default value is `true`.
|
290
|
+
|
291
|
+
```
|
292
|
+
check_connection true
|
293
|
+
```
|
294
|
+
### retry_forever
|
295
|
+
|
296
|
+
The parameter If true, plugin will ignore retry_timeout and retry_max_times options and retry forever. The default value is `true`.
|
297
|
+
|
298
|
+
```
|
299
|
+
retry_forever true
|
300
|
+
```
|
301
|
+
|
302
|
+
### retry_timeout
|
303
|
+
|
304
|
+
The parameter maximum time (seconds) to retry again the failed try, until the plugin discards the retry.
|
305
|
+
If the next retry is going to exceed this time limit, the last retry will be made at exactly this time limit..
|
306
|
+
The default value is `72h`.
|
307
|
+
72hours == 17 times with exponential backoff (not to change default behavior)
|
308
|
+
|
309
|
+
```
|
310
|
+
retry_timeout 72 * 60 * 60
|
311
|
+
```
|
312
|
+
|
313
|
+
### retry_max_times
|
314
|
+
|
315
|
+
The parameter maximum number of times to retry the failed try. The default value is `5`
|
316
|
+
|
317
|
+
```
|
318
|
+
retry_max_times 5
|
319
|
+
```
|
320
|
+
|
321
|
+
### retry_type
|
279
322
|
|
280
|
-
The parameter
|
323
|
+
The parameter needs for how long need to wait (time in seconds) to retry again:
|
324
|
+
`exponential_backoff`: wait in seconds will become large exponentially per failure,
|
325
|
+
`periodic`: plugin will retry periodically with fixed intervals (configured via retry_wait). The default value is `:exponential_backoff`
|
326
|
+
Periodic -> fixed :retry_wait
|
327
|
+
Exponential backoff: k is number of retry times
|
328
|
+
c: constant factor, @retry_wait
|
329
|
+
b: base factor, @retry_exponential_backoff_base
|
330
|
+
k: times
|
331
|
+
total retry time: c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1
|
281
332
|
|
282
333
|
```
|
283
|
-
|
334
|
+
retry_type :exponential_backoff
|
284
335
|
```
|
285
336
|
|
337
|
+
### retry_wait
|
338
|
+
|
339
|
+
The parameter needs for wait in seconds before the next retry to again or constant factor of exponential backoff. The default value is `5`
|
340
|
+
|
341
|
+
```
|
342
|
+
retry_wait 5
|
343
|
+
```
|
344
|
+
|
345
|
+
### retry_exponential_backoff_base
|
346
|
+
|
347
|
+
The parameter The base number of exponential backoff for retries. The default value is `2`
|
348
|
+
|
349
|
+
```
|
350
|
+
retry_exponential_backoff_base 2
|
351
|
+
```
|
352
|
+
|
353
|
+
### retry_max_interval
|
354
|
+
|
355
|
+
The parameter maximum interval (seconds) for exponential backoff between retries while failing. The default value is `nil`
|
356
|
+
|
357
|
+
```
|
358
|
+
retry_max_interval nil
|
359
|
+
```
|
360
|
+
|
361
|
+
### retry_randomize
|
362
|
+
|
363
|
+
The parameter If true, the output plugin will retry after randomized interval not to do burst retries. The default value is `false`
|
364
|
+
|
365
|
+
```
|
366
|
+
retry_randomize false
|
367
|
+
```
|
286
368
|
|
287
369
|
## Advanced Usage
|
288
370
|
|
@@ -204,28 +204,24 @@ module Fluent::Plugin
|
|
204
204
|
host.merge!(user: @user, password: @password) if !host[:user] && @user
|
205
205
|
host.merge!(path: @path) if !host[:path] && @path
|
206
206
|
end
|
207
|
-
live_hosts = hosts.select { |host| reachable_host?(host) }
|
207
|
+
live_hosts = @check_connection ? hosts.select { |host| reachable_host?(host) } : hosts
|
208
208
|
{
|
209
209
|
hosts: live_hosts
|
210
210
|
}
|
211
211
|
end
|
212
212
|
|
213
213
|
def reachable_host?(host)
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
|
220
|
-
|
221
|
-
|
222
|
-
|
223
|
-
|
224
|
-
|
225
|
-
client.ping
|
226
|
-
else
|
227
|
-
true
|
228
|
-
end
|
214
|
+
client = OpenSearch::Client.new(
|
215
|
+
host: ["#{host[:scheme]}://#{host[:host]}:#{host[:port]}"],
|
216
|
+
user: host[:user],
|
217
|
+
password: host[:password],
|
218
|
+
reload_connections: @reload_connections,
|
219
|
+
request_timeout: @request_timeout,
|
220
|
+
resurrect_after: @resurrect_after,
|
221
|
+
reload_on_failure: @reload_on_failure,
|
222
|
+
transport_options: { ssl: { verify: @ssl_verify, ca_file: @ca_file, version: @ssl_version } }
|
223
|
+
)
|
224
|
+
client.ping
|
229
225
|
rescue => e
|
230
226
|
log.warn "Failed to connect to #{host[:scheme]}://#{host[:host]}:#{host[:port]}: #{e.message}"
|
231
227
|
false
|
@@ -353,15 +349,13 @@ module Fluent::Plugin
|
|
353
349
|
end
|
354
350
|
rescue Faraday::ConnectionFailed, OpenSearch::Transport::Transport::Error => e
|
355
351
|
@retry.step
|
356
|
-
#
|
357
|
-
if @retry.limit?
|
358
|
-
|
359
|
-
|
360
|
-
|
361
|
-
|
362
|
-
|
363
|
-
retry
|
364
|
-
end
|
352
|
+
#Raise error if the retry limit has been reached
|
353
|
+
raise "Hit limit for retries. retry_times: #{@retry.steps}, error: #{e.message}" if @retry.limit?
|
354
|
+
|
355
|
+
#Retry if the retry limit hasn't been reached
|
356
|
+
log.warn("failed to connect or search.", retry_times: @retry.steps, next_retry_time: @retry.next_time.round, error: e.message)
|
357
|
+
sleep(@retry.next_time - Time.now)
|
358
|
+
retry
|
365
359
|
end
|
366
360
|
|
367
361
|
def run_slice(slice_id=nil)
|