fluent-plugin-s3 1.2.0 → 1.3.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/ChangeLog +21 -0
- data/README.md +96 -14
- data/VERSION +1 -1
- data/fluent-plugin-s3.gemspec +2 -2
- data/lib/fluent/plugin/in_s3.rb +32 -3
- data/lib/fluent/plugin/out_s3.rb +51 -3
- data/test/test_in_s3.rb +2 -0
- data/test/test_out_s3.rb +59 -0
- metadata +6 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6b9e71e48fe8776d96c49e62012af3fc76ad2dbc6d7ded8a5514a82b1e9d0fc6
|
4
|
+
data.tar.gz: c1cba63a0f740b46557fe3e99ae86415a395df09fa573e9c12fe96de058f6971
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 705a0c4eff9cac0ea5f8bd1b36b081a2c282b6cfb1b4c54ebc1e94e819560fc51b4d7192041d67a7054c89f645448a33af4f76c71f0050c026cd535f97c1309a
|
7
|
+
data.tar.gz: 0eb1a170b8d8155a156c7d7b786a2ff5a9a3a618e63a98fca1eb75ae93b32fe71daffd513a08a41597f69d3639a604cb60780058a4db0aed8e70aa86000d5d43
|
data/ChangeLog
CHANGED
@@ -1,3 +1,24 @@
|
|
1
|
+
Release 1.3.3 - 2020/06/25
|
2
|
+
|
3
|
+
* Allow fips/gov included endpoint
|
4
|
+
* Support sts_region parameter
|
5
|
+
|
6
|
+
Release 1.3.2 - 2020/05/18
|
7
|
+
|
8
|
+
* out_s3: Show warning message for object conflict case.
|
9
|
+
|
10
|
+
Release 1.3.1 - 2020/04/15
|
11
|
+
|
12
|
+
* out_s3: Support S3 Dual-Stack Endpoints in output plugin via enable_dual_stack parameter
|
13
|
+
|
14
|
+
Release 1.3.0 - 2020/02/10
|
15
|
+
|
16
|
+
* in_s3/out_s3: Support AssumeRoleWebIdentityCredentials via `web_identity_credentials` section for EKS.
|
17
|
+
|
18
|
+
Release 1.2.1 - 2019/11/10
|
19
|
+
|
20
|
+
* in_s3: Support ECSCredentials
|
21
|
+
|
1
22
|
Release 1.2.0 - 2019/10/17
|
2
23
|
|
3
24
|
* out_s3: Add bucket_lifecycle_rule section to set bucket's lifecycle
|
data/README.md
CHANGED
@@ -31,8 +31,13 @@ We must setup SQS queue and S3 event notification before use this plugin.
|
|
31
31
|
|
32
32
|
Simply use RubyGems:
|
33
33
|
|
34
|
-
|
35
|
-
$ gem install fluent-plugin-s3
|
34
|
+
# install latest version
|
35
|
+
$ gem install fluent-plugin-s3 --no-document # for fluentd v1.0 or later
|
36
|
+
# If you need to install specifiv version, use -v option
|
37
|
+
$ gem install fluent-plugin-s3 -v 1.3.0 --no-document
|
38
|
+
# For v0.12. This is for old v0.12 users. Don't use v0.12 for new deployment
|
39
|
+
$ gem install fluent-plugin-s3 -v "~> 0.8" --no-document # for fluentd v0.12
|
40
|
+
|
36
41
|
|
37
42
|
## Configuration: credentials
|
38
43
|
|
@@ -93,6 +98,51 @@ is set to 3600 seconds.
|
|
93
98
|
A unique identifier that is used by third parties when assuming roles in
|
94
99
|
their customers' accounts.
|
95
100
|
|
101
|
+
### web_identity_credentials
|
102
|
+
|
103
|
+
Similar to the assume_role_credentials, but for usage in EKS.
|
104
|
+
|
105
|
+
<match *>
|
106
|
+
@type s3
|
107
|
+
|
108
|
+
<web_identity_credentials>
|
109
|
+
role_arn ROLE_ARN
|
110
|
+
role_session_name ROLE_SESSION_NAME
|
111
|
+
web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILE
|
112
|
+
</web_identity_credentials>
|
113
|
+
</match>
|
114
|
+
|
115
|
+
See also:
|
116
|
+
|
117
|
+
* [Using IAM Roles - AWS Identity and Access
|
118
|
+
Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
|
119
|
+
* [IAM Roles For Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
|
120
|
+
* [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
|
121
|
+
* [Aws::AssumeRoleWebIdentityCredentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/AssumeRoleWebIdentityCredentials.html)
|
122
|
+
|
123
|
+
**role_arn (required)**
|
124
|
+
|
125
|
+
The Amazon Resource Name (ARN) of the role to assume.
|
126
|
+
|
127
|
+
**role_session_name (required)**
|
128
|
+
|
129
|
+
An identifier for the assumed role session.
|
130
|
+
|
131
|
+
**web_identity_token_file (required)**
|
132
|
+
|
133
|
+
The absolute path to the file on disk containing the OIDC token
|
134
|
+
|
135
|
+
**policy**
|
136
|
+
|
137
|
+
An IAM policy in JSON format.
|
138
|
+
|
139
|
+
**duration_seconds**
|
140
|
+
|
141
|
+
The duration, in seconds, of the role session. The value can range from
|
142
|
+
900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value
|
143
|
+
is set to 3600 seconds.
|
144
|
+
|
145
|
+
|
96
146
|
### instance_profile_credentials
|
97
147
|
|
98
148
|
Retrieve temporary security credentials via HTTP request. This is useful on
|
@@ -165,7 +215,7 @@ Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
|
|
165
215
|
|
166
216
|
### v1.0 style
|
167
217
|
|
168
|
-
With fluentd v1
|
218
|
+
With fluentd v1 and fluent-plugin-s3 v1.0.0 or later, use new buffer configuration to dynamic parameters.
|
169
219
|
|
170
220
|
<match pattern>
|
171
221
|
@type s3
|
@@ -248,12 +298,18 @@ recommend using `s3_region` instead of `s3_endpoint`.
|
|
248
298
|
**s3_endpoint**
|
249
299
|
|
250
300
|
endpoint for S3 compatible services. For example, Riak CS based storage or
|
251
|
-
something. This option
|
301
|
+
something. This option is deprecated for AWS S3, use `s3_region` instead.
|
302
|
+
|
303
|
+
See also AWS article: [Working with Regions](https://aws.amazon.com/blogs/developer/working-with-regions/).
|
252
304
|
|
253
305
|
**enable_transfer_acceleration**
|
254
306
|
|
255
307
|
Enable [S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) for uploads. **IMPORTANT**: For this to work, you must first enable this feature on your destination S3 bucket.
|
256
308
|
|
309
|
+
**enable_dual_stack**
|
310
|
+
|
311
|
+
Enable [Amazon S3 Dual-Stack Endpoints](https://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html) for uploads. Will make it possible to use either IPv4 or IPv6 when connecting to S3.
|
312
|
+
|
257
313
|
**use_bundled_cert**
|
258
314
|
|
259
315
|
For cases where the default SSL certificate is unavailable (e.g. Windows), you can set this option to true in order to use the AWS SDK bundled certificate. Default is false.
|
@@ -297,6 +353,13 @@ You can configure the length of string with a
|
|
297
353
|
`hex_random_length` parameter (Default: 4).
|
298
354
|
|
299
355
|
The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
|
356
|
+
In addition, you can use [buffer placeholders](https://docs.fluentd.org/configuration/buffer-section#placeholders) in this parameter,
|
357
|
+
so you can embed tag, time and record value like below:
|
358
|
+
|
359
|
+
s3_object_key_format %{path}/events/%Y%m%d/${tag}_%{index}.%{file_extension}
|
360
|
+
<buffer tag,time>
|
361
|
+
# buffer parameters...
|
362
|
+
</buffer>
|
300
363
|
|
301
364
|
For instance, using the example configuration above, actual object keys on S3
|
302
365
|
will be something like:
|
@@ -352,7 +415,7 @@ See `Use your compression algorithm` section for adding another format.
|
|
352
415
|
**`<format>` or format**
|
353
416
|
|
354
417
|
Change one line format in the S3 object. Supported formats are "out_file",
|
355
|
-
"json", "ltsv"
|
418
|
+
"json", "ltsv", "single_value" and other formatter plugins. See also [official Formatter article](https://docs.fluentd.org/formatter).
|
356
419
|
|
357
420
|
* out_file (default).
|
358
421
|
|
@@ -368,11 +431,18 @@ Change one line format in the S3 object. Supported formats are "out_file",
|
|
368
431
|
|
369
432
|
|
370
433
|
At this format, "time" and "tag" are omitted. But you can set these
|
371
|
-
information to the record by setting
|
372
|
-
"include_time_key" / "time_key" option. If you set following configuration in
|
434
|
+
information to the record by setting `<inject>` option. If you set following configuration in
|
373
435
|
S3 output:
|
374
436
|
|
375
|
-
|
437
|
+
# v1
|
438
|
+
<format>
|
439
|
+
@type json
|
440
|
+
</format>
|
441
|
+
<inject>
|
442
|
+
time_key log_time
|
443
|
+
</inject>
|
444
|
+
# v0.12
|
445
|
+
@format json
|
376
446
|
include_time_key true
|
377
447
|
time_key log_time # default is time
|
378
448
|
|
@@ -380,15 +450,14 @@ then the record has log_time field.
|
|
380
450
|
|
381
451
|
{"log_time":"time string",...}
|
382
452
|
|
453
|
+
See also [official Inject Section article](https://docs.fluentd.org/configuration/inject-section).
|
454
|
+
|
383
455
|
* ltsv
|
384
456
|
|
385
457
|
key1:value1\tkey2:value2
|
386
458
|
key1:value1\tkey2:value2
|
387
459
|
...
|
388
460
|
|
389
|
-
|
390
|
-
"ltsv" format also accepts "include_xxx" related options. See "json" section.
|
391
|
-
|
392
461
|
* single_value
|
393
462
|
|
394
463
|
|
@@ -431,21 +500,34 @@ uri of proxy environment.
|
|
431
500
|
**path**
|
432
501
|
|
433
502
|
path prefix of the files on S3. Default is "" (no prefix).
|
503
|
+
[buffer placeholder](https://docs.fluentd.org/configuration/buffer-section#placeholders) is supported,
|
504
|
+
so you can embed tag, time and record value like below.
|
505
|
+
|
506
|
+
path logs/%Y%m%d/${tag}/
|
507
|
+
<buffer tag,time>
|
508
|
+
# buffer parameters...
|
509
|
+
</buffer>
|
434
510
|
|
435
|
-
**buffer_path (
|
511
|
+
**buffer_path (for v0.12)**
|
436
512
|
|
437
513
|
path prefix of the files to buffer logs.
|
438
514
|
|
439
|
-
|
515
|
+
This parameter is for v0.12. Use `<buffer>`'s `path` in v1.
|
516
|
+
|
517
|
+
**time_slice_format(for v0.12)**
|
440
518
|
|
441
519
|
Format of the time used as the file name. Default is '%Y%m%d'. Use
|
442
520
|
'%Y%m%d%H' to split files hourly.
|
443
521
|
|
444
|
-
|
522
|
+
This parameter is for v0.12. Use buffer placeholder for `path` / `s3_object_key_format` in v1.
|
523
|
+
|
524
|
+
**time_slice_wait(for v0.12)**
|
445
525
|
|
446
526
|
The time to wait old logs. Default is 10 minutes. Specify larger value if
|
447
527
|
old logs may reach.
|
448
528
|
|
529
|
+
This parameter is for v0.12. Use `<buffer>`'s `timekey_wait` in v1.
|
530
|
+
|
449
531
|
**utc**
|
450
532
|
|
451
533
|
Use UTC instead of local time.
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
1.
|
1
|
+
1.3.3
|
data/fluent-plugin-s3.gemspec
CHANGED
@@ -17,8 +17,8 @@ Gem::Specification.new do |gem|
|
|
17
17
|
gem.require_paths = ['lib']
|
18
18
|
|
19
19
|
gem.add_dependency "fluentd", [">= 0.14.22", "< 2"]
|
20
|
-
gem.add_dependency "aws-sdk-s3", "~> 1.
|
21
|
-
gem.add_dependency "aws-sdk-sqs", "~> 1.
|
20
|
+
gem.add_dependency "aws-sdk-s3", "~> 1.60"
|
21
|
+
gem.add_dependency "aws-sdk-sqs", "~> 1.23"
|
22
22
|
gem.add_development_dependency "rake", ">= 0.9.2"
|
23
23
|
gem.add_development_dependency "test-unit", ">= 3.0.8"
|
24
24
|
gem.add_development_dependency "test-unit-rr", ">= 1.0.3"
|
data/lib/fluent/plugin/in_s3.rb
CHANGED
@@ -42,6 +42,20 @@ module Fluent::Plugin
|
|
42
42
|
desc "A unique identifier that is used by third parties when assuming roles in their customers' accounts."
|
43
43
|
config_param :external_id, :string, default: nil
|
44
44
|
end
|
45
|
+
# See the following link for additional params that could be added:
|
46
|
+
# https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/STS/Client.html#assume_role_with_web_identity-instance_method
|
47
|
+
config_section :web_identity_credentials, multi: false do
|
48
|
+
desc "The Amazon Resource Name (ARN) of the role to assume"
|
49
|
+
config_param :role_arn, :string # required
|
50
|
+
desc "An identifier for the assumed role session"
|
51
|
+
config_param :role_session_name, :string #required
|
52
|
+
desc "The absolute path to the file on disk containing the OIDC token"
|
53
|
+
config_param :web_identity_token_file, :string #required
|
54
|
+
desc "An IAM policy in JSON format"
|
55
|
+
config_param :policy, :string, default: nil
|
56
|
+
desc "The duration, in seconds, of the role session (900-43200)"
|
57
|
+
config_param :duration_seconds, :integer, default: nil
|
58
|
+
end
|
45
59
|
config_section :instance_profile_credentials, multi: false do
|
46
60
|
desc "Number of times to retry when retrieving credentials"
|
47
61
|
config_param :retries, :integer, default: nil
|
@@ -104,11 +118,11 @@ module Fluent::Plugin
|
|
104
118
|
def configure(conf)
|
105
119
|
super
|
106
120
|
|
107
|
-
if @s3_endpoint && @s3_endpoint.end_with?('amazonaws.com')
|
121
|
+
if @s3_endpoint && (@s3_endpoint.end_with?('amazonaws.com') && !['fips', 'gov'].any? { |e| @s3_endpoint.include?(e) })
|
108
122
|
raise Fluent::ConfigError, "s3_endpoint parameter is not supported for S3, use s3_region instead. This parameter is for S3 compatible services"
|
109
123
|
end
|
110
124
|
|
111
|
-
if @sqs.endpoint && @sqs.endpoint.end_with?('amazonaws.com')
|
125
|
+
if @sqs.endpoint && (@sqs.endpoint.end_with?('amazonaws.com') && !['fips', 'gov'].any? { |e| @sqs.endpoint.include?(e) })
|
112
126
|
raise Fluent::ConfigError, "sqs/endpoint parameter is not supported for SQS, use s3_region instead. This parameter is for SQS compatible services"
|
113
127
|
end
|
114
128
|
|
@@ -202,6 +216,17 @@ module Fluent::Plugin
|
|
202
216
|
credentials_options[:client] = Aws::STS::Client.new(:region => @s3_region)
|
203
217
|
end
|
204
218
|
options[:credentials] = Aws::AssumeRoleCredentials.new(credentials_options)
|
219
|
+
when @web_identity_credentials
|
220
|
+
c = @web_identity_credentials
|
221
|
+
credentials_options[:role_arn] = c.role_arn
|
222
|
+
credentials_options[:role_session_name] = c.role_session_name
|
223
|
+
credentials_options[:web_identity_token_file] = c.web_identity_token_file
|
224
|
+
credentials_options[:policy] = c.policy if c.policy
|
225
|
+
credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
|
226
|
+
if @s3_region
|
227
|
+
credentials_options[:client] = Aws::STS::Client.new(:region => @s3_region)
|
228
|
+
end
|
229
|
+
options[:credentials] = Aws::AssumeRoleWebIdentityCredentials.new(credentials_options)
|
205
230
|
when @instance_profile_credentials
|
206
231
|
c = @instance_profile_credentials
|
207
232
|
credentials_options[:retries] = c.retries if c.retries
|
@@ -209,7 +234,11 @@ module Fluent::Plugin
|
|
209
234
|
credentials_options[:port] = c.port if c.port
|
210
235
|
credentials_options[:http_open_timeout] = c.http_open_timeout if c.http_open_timeout
|
211
236
|
credentials_options[:http_read_timeout] = c.http_read_timeout if c.http_read_timeout
|
212
|
-
|
237
|
+
if ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
|
238
|
+
options[:credentials] = Aws::ECSCredentials.new(credentials_options)
|
239
|
+
else
|
240
|
+
options[:credentials] = Aws::InstanceProfileCredentials.new(credentials_options)
|
241
|
+
end
|
213
242
|
when @shared_credentials
|
214
243
|
c = @shared_credentials
|
215
244
|
credentials_options[:path] = c.path if c.path
|
data/lib/fluent/plugin/out_s3.rb
CHANGED
@@ -39,6 +39,24 @@ module Fluent::Plugin
|
|
39
39
|
config_param :duration_seconds, :integer, default: nil
|
40
40
|
desc "A unique identifier that is used by third parties when assuming roles in their customers' accounts."
|
41
41
|
config_param :external_id, :string, default: nil, secret: true
|
42
|
+
desc "The region of the STS endpoint to use."
|
43
|
+
config_param :sts_region, :string, default: nil
|
44
|
+
end
|
45
|
+
# See the following link for additional params that could be added:
|
46
|
+
# https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/STS/Client.html#assume_role_with_web_identity-instance_method
|
47
|
+
config_section :web_identity_credentials, multi: false do
|
48
|
+
desc "The Amazon Resource Name (ARN) of the role to assume"
|
49
|
+
config_param :role_arn, :string # required
|
50
|
+
desc "An identifier for the assumed role session"
|
51
|
+
config_param :role_session_name, :string #required
|
52
|
+
desc "The absolute path to the file on disk containing the OIDC token"
|
53
|
+
config_param :web_identity_token_file, :string #required
|
54
|
+
desc "An IAM policy in JSON format"
|
55
|
+
config_param :policy, :string, default: nil
|
56
|
+
desc "The duration, in seconds, of the role session (900-43200)"
|
57
|
+
config_param :duration_seconds, :integer, default: nil
|
58
|
+
desc "The region of the STS endpoint to use."
|
59
|
+
config_param :sts_region, :string, default: nil
|
42
60
|
end
|
43
61
|
config_section :instance_profile_credentials, multi: false do
|
44
62
|
desc "Number of times to retry when retrieving credentials"
|
@@ -70,6 +88,8 @@ module Fluent::Plugin
|
|
70
88
|
config_param :s3_endpoint, :string, default: nil
|
71
89
|
desc "If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket"
|
72
90
|
config_param :enable_transfer_acceleration, :bool, default: false
|
91
|
+
desc "If true, use Amazon S3 Dual-Stack Endpoints. Will make it possible to use either IPv4 or IPv6 when connecting to S3."
|
92
|
+
config_param :enable_dual_stack, :bool, default: false
|
73
93
|
desc "If false, the certificate of endpoint will not be verified"
|
74
94
|
config_param :ssl_verify_peer, :bool, :default => true
|
75
95
|
desc "The format of S3 object keys"
|
@@ -155,7 +175,7 @@ module Fluent::Plugin
|
|
155
175
|
|
156
176
|
Aws.use_bundled_cert! if @use_bundled_cert
|
157
177
|
|
158
|
-
if @s3_endpoint && @s3_endpoint.end_with?('amazonaws.com')
|
178
|
+
if @s3_endpoint && (@s3_endpoint.end_with?('amazonaws.com') && !['fips', 'gov'].any? { |e| @s3_endpoint.include?(e) })
|
159
179
|
raise Fluent::ConfigError, "s3_endpoint parameter is not supported for S3, use s3_region instead. This parameter is for S3 compatible services"
|
160
180
|
end
|
161
181
|
|
@@ -175,7 +195,7 @@ module Fluent::Plugin
|
|
175
195
|
end
|
176
196
|
|
177
197
|
unless @index_format =~ /^%(0\d*)?[dxX]$/
|
178
|
-
raise Fluent::ConfigError, "index_format parameter should follow `%[flags][width]type`. `0` is the only supported flag, and is mandatory if width is specified. `d`, `x` and `X` are supported types"
|
198
|
+
raise Fluent::ConfigError, "index_format parameter should follow `%[flags][width]type`. `0` is the only supported flag, and is mandatory if width is specified. `d`, `x` and `X` are supported types"
|
179
199
|
end
|
180
200
|
|
181
201
|
if @reduced_redundancy
|
@@ -193,6 +213,8 @@ module Fluent::Plugin
|
|
193
213
|
end
|
194
214
|
end
|
195
215
|
|
216
|
+
check_s3_path_safety(conf)
|
217
|
+
|
196
218
|
# For backward compatibility
|
197
219
|
# TODO: Remove time_slice_format when end of support compat_parameters
|
198
220
|
@configured_time_slice_format = conf['time_slice_format']
|
@@ -209,6 +231,7 @@ module Fluent::Plugin
|
|
209
231
|
options[:region] = @s3_region if @s3_region
|
210
232
|
options[:endpoint] = @s3_endpoint if @s3_endpoint
|
211
233
|
options[:use_accelerate_endpoint] = @enable_transfer_acceleration
|
234
|
+
options[:use_dualstack_endpoint] = @enable_dual_stack
|
212
235
|
options[:http_proxy] = @proxy_uri if @proxy_uri
|
213
236
|
options[:force_path_style] = @force_path_style
|
214
237
|
options[:compute_checksums] = @compute_checksums unless @compute_checksums.nil?
|
@@ -434,6 +457,16 @@ module Fluent::Plugin
|
|
434
457
|
}
|
435
458
|
end
|
436
459
|
|
460
|
+
def check_s3_path_safety(conf)
|
461
|
+
unless conf.has_key?('s3_object_key_format')
|
462
|
+
log.warn "The default value of s3_object_key_format will use ${chunk_id} instead of %{index} to avoid object conflict in v2"
|
463
|
+
end
|
464
|
+
|
465
|
+
if (@buffer_config.flush_thread_count > 1) && ['${chunk_id}', '%{uuid_flush}'].none? { |key| @s3_object_key_format.include?(key) }
|
466
|
+
log.warn "No ${chunk_id} or %{uuid_flush} in s3_object_key_format with multiple flush threads. Recommend to set ${chunk_id} or %{uuid_flush} to avoid data lost by object conflict"
|
467
|
+
end
|
468
|
+
end
|
469
|
+
|
437
470
|
def check_apikeys
|
438
471
|
@bucket.objects(prefix: @path, :max_keys => 1).first
|
439
472
|
rescue Aws::S3::Errors::NoSuchBucket
|
@@ -456,10 +489,25 @@ module Fluent::Plugin
|
|
456
489
|
credentials_options[:policy] = c.policy if c.policy
|
457
490
|
credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
|
458
491
|
credentials_options[:external_id] = c.external_id if c.external_id
|
459
|
-
if
|
492
|
+
if c.sts_region
|
493
|
+
credentials_options[:client] = Aws::STS::Client.new(region: c.sts_region)
|
494
|
+
elsif @s3_region
|
460
495
|
credentials_options[:client] = Aws::STS::Client.new(region: @s3_region)
|
461
496
|
end
|
462
497
|
options[:credentials] = Aws::AssumeRoleCredentials.new(credentials_options)
|
498
|
+
when @web_identity_credentials
|
499
|
+
c = @web_identity_credentials
|
500
|
+
credentials_options[:role_arn] = c.role_arn
|
501
|
+
credentials_options[:role_session_name] = c.role_session_name
|
502
|
+
credentials_options[:web_identity_token_file] = c.web_identity_token_file
|
503
|
+
credentials_options[:policy] = c.policy if c.policy
|
504
|
+
credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
|
505
|
+
if c.sts_region
|
506
|
+
credentials_options[:client] = Aws::STS::Client.new(:region => c.sts_region)
|
507
|
+
elsif @s3_region
|
508
|
+
credentials_options[:client] = Aws::STS::Client.new(:region => @s3_region)
|
509
|
+
end
|
510
|
+
options[:credentials] = Aws::AssumeRoleWebIdentityCredentials.new(credentials_options)
|
463
511
|
when @instance_profile_credentials
|
464
512
|
c = @instance_profile_credentials
|
465
513
|
credentials_options[:retries] = c.retries if c.retries
|
data/test/test_in_s3.rb
CHANGED
@@ -11,6 +11,7 @@ require 'fluent/plugin/in_s3'
|
|
11
11
|
require 'test/unit/rr'
|
12
12
|
require 'zlib'
|
13
13
|
require 'fileutils'
|
14
|
+
require 'ostruct'
|
14
15
|
|
15
16
|
include Fluent::Test::Helpers
|
16
17
|
|
@@ -152,6 +153,7 @@ EOS
|
|
152
153
|
|
153
154
|
def setup_mocks
|
154
155
|
@s3_client = stub(Aws::S3::Client.new(stub_responses: true))
|
156
|
+
stub(@s3_client).config { OpenStruct.new({region: "us-east-1"}) }
|
155
157
|
mock(Aws::S3::Client).new(anything).at_least(0) { @s3_client }
|
156
158
|
@s3_resource = mock(Aws::S3::Resource.new(client: @s3_client))
|
157
159
|
mock(Aws::S3::Resource).new(client: @s3_client) { @s3_resource }
|
data/test/test_out_s3.rb
CHANGED
@@ -10,6 +10,7 @@ require 'zlib'
|
|
10
10
|
require 'fileutils'
|
11
11
|
require 'timecop'
|
12
12
|
require 'uuidtools'
|
13
|
+
require 'ostruct'
|
13
14
|
|
14
15
|
include Fluent::Test::Helpers
|
15
16
|
|
@@ -427,6 +428,7 @@ EOC
|
|
427
428
|
|
428
429
|
def setup_mocks(exists_return = false)
|
429
430
|
@s3_client = stub(Aws::S3::Client.new(stub_responses: true))
|
431
|
+
stub(@s3_client).config { OpenStruct.new({region: "us-east-1"}) }
|
430
432
|
# aws-sdk-s3 calls Client#put_object inside Object#put
|
431
433
|
mock(@s3_client).put_object(anything).at_least(0) { MockResponse.new({}) }
|
432
434
|
mock(Aws::S3::Client).new(anything).at_least(0) { @s3_client }
|
@@ -464,6 +466,7 @@ EOC
|
|
464
466
|
|
465
467
|
def setup_mocks_hardened_policy()
|
466
468
|
@s3_client = stub(Aws::S3::Client.new(:stub_responses => true))
|
469
|
+
stub(@s3_client).config { OpenStruct.new({region: "us-east-1"}) }
|
467
470
|
mock(@s3_client).put_object(anything).at_least(0) { MockResponse.new({}) }
|
468
471
|
mock(Aws::S3::Client).new(anything).at_least(0) { @s3_client }
|
469
472
|
@s3_resource = mock(Aws::S3::Resource.new(:client => @s3_client))
|
@@ -564,6 +567,62 @@ EOC
|
|
564
567
|
assert_equal(expected_credentials, credentials)
|
565
568
|
end
|
566
569
|
|
570
|
+
def test_web_identity_credentials
|
571
|
+
expected_credentials = Aws::Credentials.new("test_key", "test_secret")
|
572
|
+
mock(Aws::AssumeRoleWebIdentityCredentials).new(
|
573
|
+
role_arn: "test_arn",
|
574
|
+
role_session_name: "test_session",
|
575
|
+
web_identity_token_file: "test_file",
|
576
|
+
client: anything
|
577
|
+
){
|
578
|
+
expected_credentials
|
579
|
+
}
|
580
|
+
|
581
|
+
config = CONFIG_TIME_SLICE.split("\n").reject{|x| x =~ /.+aws_.+/}.join("\n")
|
582
|
+
config += %[
|
583
|
+
<web_identity_credentials>
|
584
|
+
role_arn test_arn
|
585
|
+
role_session_name test_session
|
586
|
+
web_identity_token_file test_file
|
587
|
+
</web_identity_credentials>
|
588
|
+
]
|
589
|
+
d = create_time_sliced_driver(config)
|
590
|
+
assert_nothing_raised { d.run {} }
|
591
|
+
client = d.instance.instance_variable_get(:@s3).client
|
592
|
+
credentials = client.config.credentials
|
593
|
+
assert_equal(expected_credentials, credentials)
|
594
|
+
end
|
595
|
+
|
596
|
+
def test_web_identity_credentials_with_sts_region
|
597
|
+
expected_credentials = Aws::Credentials.new("test_key", "test_secret")
|
598
|
+
sts_client = Aws::STS::Client.new(region: 'us-east-1')
|
599
|
+
mock(Aws::STS::Client).new(region: 'us-east-1'){ sts_client }
|
600
|
+
mock(Aws::AssumeRoleWebIdentityCredentials).new(
|
601
|
+
role_arn: "test_arn",
|
602
|
+
role_session_name: "test_session",
|
603
|
+
web_identity_token_file: "test_file",
|
604
|
+
client: sts_client
|
605
|
+
){
|
606
|
+
expected_credentials
|
607
|
+
}
|
608
|
+
|
609
|
+
config = CONFIG_TIME_SLICE.split("\n").reject{|x| x =~ /.+aws_.+/}.join("\n")
|
610
|
+
config += %[
|
611
|
+
s3_region us-west-2
|
612
|
+
<web_identity_credentials>
|
613
|
+
role_arn test_arn
|
614
|
+
role_session_name test_session
|
615
|
+
web_identity_token_file test_file
|
616
|
+
sts_region us-east-1
|
617
|
+
</web_identity_credentials>
|
618
|
+
]
|
619
|
+
d = create_time_sliced_driver(config)
|
620
|
+
assert_nothing_raised { d.run {} }
|
621
|
+
client = d.instance.instance_variable_get(:@s3).client
|
622
|
+
credentials = client.config.credentials
|
623
|
+
assert_equal(expected_credentials, credentials)
|
624
|
+
end
|
625
|
+
|
567
626
|
def test_instance_profile_credentials
|
568
627
|
expected_credentials = Aws::Credentials.new("test_key", "test_secret")
|
569
628
|
mock(Aws::InstanceProfileCredentials).new({}).returns(expected_credentials)
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-s3
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.3.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sadayuki Furuhashi
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2020-06-25 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: fluentd
|
@@ -37,28 +37,28 @@ dependencies:
|
|
37
37
|
requirements:
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version: '1.
|
40
|
+
version: '1.60'
|
41
41
|
type: :runtime
|
42
42
|
prerelease: false
|
43
43
|
version_requirements: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
45
|
- - "~>"
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '1.
|
47
|
+
version: '1.60'
|
48
48
|
- !ruby/object:Gem::Dependency
|
49
49
|
name: aws-sdk-sqs
|
50
50
|
requirement: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
52
|
- - "~>"
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '1.
|
54
|
+
version: '1.23'
|
55
55
|
type: :runtime
|
56
56
|
prerelease: false
|
57
57
|
version_requirements: !ruby/object:Gem::Requirement
|
58
58
|
requirements:
|
59
59
|
- - "~>"
|
60
60
|
- !ruby/object:Gem::Version
|
61
|
-
version: '1.
|
61
|
+
version: '1.23'
|
62
62
|
- !ruby/object:Gem::Dependency
|
63
63
|
name: rake
|
64
64
|
requirement: !ruby/object:Gem::Requirement
|