test-opensearch 0.0.0.17 → 0.0.0.20
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.OpenSearchInput.md +86 -4
- data/fluent-plugin-opensearch.gemspec +1 -1
- data/lib/fluent/plugin/in_opensearch.rb +22 -28
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: e64802826ae07863828975be01dbb8fee1bc0e4d8fef81ad195fe658ede210e0
|
4
|
+
data.tar.gz: cd18ef4da46e20bd059a3fcf5d9b2791e630e4c37704cc4af17e2480e717113b
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d79fbfa7dd1f59e79db7c53e422cf3f6f4ad8d62b09f7b20566cfec33077230971083b2691dbb57b081344005b0ad09200a2916eba1a35628b18d3cbcf039858
|
7
|
+
data.tar.gz: ade76a78db35219afeb6cba5cd4db1bcfb14470c9a0f7d1daf82d9a7ae065602fcb9577d36f6e81448f456f2e773837ac48f35be70627c761f3dd6f76ba0108c
|
data/README.OpenSearchInput.md
CHANGED
@@ -24,7 +24,16 @@
|
|
24
24
|
+ [docinfo_fields](#docinfo_fields)
|
25
25
|
+ [docinfo_target](#docinfo_target)
|
26
26
|
+ [docinfo](#docinfo)
|
27
|
-
+ [
|
27
|
+
+ [check_connection](#check_connection)
|
28
|
+
+ [retry_forever](#retry_forever)
|
29
|
+
+ [retry_timeout](#retry_timeout)
|
30
|
+
+ [retry_max_times](#retry_max_times)
|
31
|
+
+ [retry_type](#retry_type)
|
32
|
+
+ [retry_wait](#retry_wait)
|
33
|
+
+ [retry_exponential_backoff_base](#retry_exponential_backoff_base)
|
34
|
+
+ [retry_max_interval](#retry_max_interval)
|
35
|
+
+ [retry_randomize](#retry_randomize)
|
36
|
+
|
28
37
|
* [Advanced Usage](#advanced-usage)
|
29
38
|
|
30
39
|
## Usage
|
@@ -275,14 +284,87 @@ This parameter specifies whether docinfo information including or not. The defau
|
|
275
284
|
docinfo false
|
276
285
|
```
|
277
286
|
|
278
|
-
###
|
287
|
+
### check_connection
|
288
|
+
|
289
|
+
The parameter for checking on connection availability with Elasticsearch or Opensearch hosts. The default value is `true`.
|
290
|
+
|
291
|
+
```
|
292
|
+
check_connection true
|
293
|
+
```
|
294
|
+
### retry_forever
|
295
|
+
|
296
|
+
The parameter If true, plugin will ignore retry_timeout and retry_max_times options and retry forever. The default value is `true`.
|
297
|
+
|
298
|
+
```
|
299
|
+
retry_forever true
|
300
|
+
```
|
301
|
+
|
302
|
+
### retry_timeout
|
303
|
+
|
304
|
+
The parameter maximum time (seconds) to retry again the failed try, until the plugin discards the retry.
|
305
|
+
If the next retry is going to exceed this time limit, the last retry will be made at exactly this time limit..
|
306
|
+
The default value is `72h`.
|
307
|
+
72hours == 17 times with exponential backoff (not to change default behavior)
|
308
|
+
|
309
|
+
```
|
310
|
+
retry_timeout 72 * 60 * 60
|
311
|
+
```
|
312
|
+
|
313
|
+
### retry_max_times
|
314
|
+
|
315
|
+
The parameter maximum number of times to retry the failed try. The default value is `5`
|
316
|
+
|
317
|
+
```
|
318
|
+
retry_max_times 5
|
319
|
+
```
|
320
|
+
|
321
|
+
### retry_type
|
279
322
|
|
280
|
-
The parameter
|
323
|
+
The parameter needs for how long need to wait (time in seconds) to retry again:
|
324
|
+
`exponential_backoff`: wait in seconds will become large exponentially per failure,
|
325
|
+
`periodic:` plugin will retry periodically with fixed intervals (configured via retry_wait). The default value is `:periodic`
|
326
|
+
Periodic -> fixed :retry_wait
|
327
|
+
Exponential backoff: k is number of retry times
|
328
|
+
c: constant factor, @retry_wait
|
329
|
+
b: base factor, @retry_exponential_backoff_base
|
330
|
+
k: times
|
331
|
+
total retry time: c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1
|
281
332
|
|
282
333
|
```
|
283
|
-
|
334
|
+
retry_type :periodic
|
284
335
|
```
|
285
336
|
|
337
|
+
### retry_wait
|
338
|
+
|
339
|
+
The parameter needs for wait in seconds before the next retry to again or constant factor of exponential backoff. The default value is `5`
|
340
|
+
|
341
|
+
```
|
342
|
+
retry_wait 5
|
343
|
+
```
|
344
|
+
|
345
|
+
### retry_exponential_backoff_base
|
346
|
+
|
347
|
+
The parameter The base number of exponential backoff for retries. The default value is `2`
|
348
|
+
|
349
|
+
```
|
350
|
+
retry_exponential_backoff_base 2
|
351
|
+
```
|
352
|
+
|
353
|
+
### retry_max_interval
|
354
|
+
|
355
|
+
The parameter maximum interval (seconds) for exponential backoff between retries while failing. The default value is `nil`
|
356
|
+
|
357
|
+
```
|
358
|
+
retry_max_interval nil
|
359
|
+
```
|
360
|
+
|
361
|
+
### retry_randomize
|
362
|
+
|
363
|
+
The parameter If true, the output plugin will retry after randomized interval not to do burst retries. The default value is `false`
|
364
|
+
|
365
|
+
```
|
366
|
+
retry_randomize false
|
367
|
+
```
|
286
368
|
|
287
369
|
## Advanced Usage
|
288
370
|
|
@@ -83,9 +83,9 @@ module Fluent::Plugin
|
|
83
83
|
config_param :docinfo, :bool, :default => false
|
84
84
|
config_param :check_connection, :bool, :default => true
|
85
85
|
config_param :retry_forever, :bool, default: true, desc: 'If true, plugin will ignore retry_timeout and retry_max_times options and retry forever.'
|
86
|
-
config_param :retry_timeout, :time, default: 72 * 60 * 60, desc: 'The maximum seconds to retry
|
86
|
+
config_param :retry_timeout, :time, default: 72 * 60 * 60, desc: 'The maximum seconds to retry'
|
87
87
|
# 72hours == 17 times with exponential backoff (not to change default behavior)
|
88
|
-
config_param :retry_max_times, :integer, default: 5, desc: 'The maximum number of times to retry
|
88
|
+
config_param :retry_max_times, :integer, default: 5, desc: 'The maximum number of times to retry'
|
89
89
|
# exponential backoff sequence will be initialized at the time of this threshold
|
90
90
|
config_param :retry_type, :enum, list: [:exponential_backoff, :periodic], default: :exponential_backoff
|
91
91
|
### Periodic -> fixed :retry_wait
|
@@ -94,7 +94,7 @@ module Fluent::Plugin
|
|
94
94
|
# b: base factor, @retry_exponential_backoff_base
|
95
95
|
# k: times
|
96
96
|
# total retry time: c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1
|
97
|
-
config_param :retry_wait, :time, default:
|
97
|
+
config_param :retry_wait, :time, default: 5, desc: 'Seconds to wait before next retry , or constant factor of exponential backoff.'
|
98
98
|
config_param :retry_exponential_backoff_base, :float, default: 2, desc: 'The base number of exponential backoff for retries.'
|
99
99
|
config_param :retry_max_interval, :time, default: nil, desc: 'The maximum interval seconds for exponential backoff between retries while failing.'
|
100
100
|
config_param :retry_randomize, :bool, default: false, desc: 'If true, output plugin will retry after randomized interval not to do burst retries.'
|
@@ -204,28 +204,24 @@ module Fluent::Plugin
|
|
204
204
|
host.merge!(user: @user, password: @password) if !host[:user] && @user
|
205
205
|
host.merge!(path: @path) if !host[:path] && @path
|
206
206
|
end
|
207
|
-
live_hosts = hosts.select { |host| reachable_host?(host) }
|
207
|
+
live_hosts = @check_connection ? hosts.select { |host| reachable_host?(host) } : hosts
|
208
208
|
{
|
209
209
|
hosts: live_hosts
|
210
210
|
}
|
211
211
|
end
|
212
212
|
|
213
213
|
def reachable_host?(host)
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
|
220
|
-
|
221
|
-
|
222
|
-
|
223
|
-
|
224
|
-
|
225
|
-
client.ping
|
226
|
-
else
|
227
|
-
true
|
228
|
-
end
|
214
|
+
client = OpenSearch::Client.new(
|
215
|
+
host: ["#{host[:scheme]}://#{host[:host]}:#{host[:port]}"],
|
216
|
+
user: host[:user],
|
217
|
+
password: host[:password],
|
218
|
+
reload_connections: @reload_connections,
|
219
|
+
request_timeout: @request_timeout,
|
220
|
+
resurrect_after: @resurrect_after,
|
221
|
+
reload_on_failure: @reload_on_failure,
|
222
|
+
transport_options: { ssl: { verify: @ssl_verify, ca_file: @ca_file, version: @ssl_version } }
|
223
|
+
)
|
224
|
+
client.ping
|
229
225
|
rescue => e
|
230
226
|
log.warn "Failed to connect to #{host[:scheme]}://#{host[:host]}:#{host[:port]}: #{e.message}"
|
231
227
|
false
|
@@ -353,15 +349,13 @@ module Fluent::Plugin
|
|
353
349
|
end
|
354
350
|
rescue Faraday::ConnectionFailed, OpenSearch::Transport::Transport::Error => e
|
355
351
|
@retry.step
|
356
|
-
#
|
357
|
-
if @retry.limit?
|
358
|
-
|
359
|
-
|
360
|
-
|
361
|
-
|
362
|
-
|
363
|
-
retry
|
364
|
-
end
|
352
|
+
#Raise error if the retry limit has been reached
|
353
|
+
raise "Hit limit for retries. retry_times: #{@retry.limit?}, error: #{e.message}" if @retry.limit?
|
354
|
+
|
355
|
+
#Retry if the retry limit hasn't been reached
|
356
|
+
log.warn("failed to connect or search.", retry_times: @retry.steps, next_retry_time: @retry.next_time.round, error: e.message)
|
357
|
+
sleep(@retry.next_time - Time.now)
|
358
|
+
retry
|
365
359
|
end
|
366
360
|
|
367
361
|
def run_slice(slice_id=nil)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: test-opensearch
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.0.
|
4
|
+
version: 0.0.0.20
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Oleh
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-04-
|
11
|
+
date: 2024-04-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: fluentd
|