fluent-plugin-s3 1.2.0 → 1.3.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 26e0fd779938910c499dfdd043c253b90167c05c585945fddcaa37985302cbff
4
- data.tar.gz: fe09d0a34580eeea82b3538f7e7fe8fcbc85e42dcb1d7dd5cedfbfeeefdfbe56
3
+ metadata.gz: 6b9e71e48fe8776d96c49e62012af3fc76ad2dbc6d7ded8a5514a82b1e9d0fc6
4
+ data.tar.gz: c1cba63a0f740b46557fe3e99ae86415a395df09fa573e9c12fe96de058f6971
5
5
  SHA512:
6
- metadata.gz: aec5991826999db113542e9e313b03a2ca018d3b7143ca745f45a211f0ce3e3c66c1b5b84c087b3e5d654aeb8b840db1580e5b6911bd182b7fb1a4131ee317ce
7
- data.tar.gz: 575fcd019697834d1ff5aef35a456fa7b5a54ec8c3f565285b50195ca1f5d67a24738d39973e9f7d7e9e15bac398ac96d494ffc1907e5cd6271385242cf36521
6
+ metadata.gz: 705a0c4eff9cac0ea5f8bd1b36b081a2c282b6cfb1b4c54ebc1e94e819560fc51b4d7192041d67a7054c89f645448a33af4f76c71f0050c026cd535f97c1309a
7
+ data.tar.gz: 0eb1a170b8d8155a156c7d7b786a2ff5a9a3a618e63a98fca1eb75ae93b32fe71daffd513a08a41597f69d3639a604cb60780058a4db0aed8e70aa86000d5d43
data/ChangeLog CHANGED
@@ -1,3 +1,24 @@
1
+ Release 1.3.3 - 2020/06/25
2
+
3
+ * Allow fips/gov included endpoint
4
+ * Support sts_region parameter
5
+
6
+ Release 1.3.2 - 2020/05/18
7
+
8
+ * out_s3: Show warning message for object conflict case.
9
+
10
+ Release 1.3.1 - 2020/04/15
11
+
12
+ * out_s3: Support S3 Dual-Stack Endpoints in output plugin via enable_dual_stack parameter
13
+
14
+ Release 1.3.0 - 2020/02/10
15
+
16
+ * in_s3/out_s3: Support AssumeRoleWebIdentityCredentials via `web_identity_credentials` section for EKS.
17
+
18
+ Release 1.2.1 - 2019/11/10
19
+
20
+ * in_s3: Support ECSCredentials
21
+
1
22
  Release 1.2.0 - 2019/10/17
2
23
 
3
24
  * out_s3: Add bucket_lifecycle_rule section to set bucket's lifecycle
data/README.md CHANGED
@@ -31,8 +31,13 @@ We must setup SQS queue and S3 event notification before use this plugin.
31
31
 
32
32
  Simply use RubyGems:
33
33
 
34
- $ gem install fluent-plugin-s3 -v "~> 0.8" --no-document # for fluentd v0.12 or later
35
- $ gem install fluent-plugin-s3 -v 1.0.0 --no-document # for fluentd v1.0 or later
34
+ # install latest version
35
+ $ gem install fluent-plugin-s3 --no-document # for fluentd v1.0 or later
36
+ # If you need to install specifiv version, use -v option
37
+ $ gem install fluent-plugin-s3 -v 1.3.0 --no-document
38
+ # For v0.12. This is for old v0.12 users. Don't use v0.12 for new deployment
39
+ $ gem install fluent-plugin-s3 -v "~> 0.8" --no-document # for fluentd v0.12
40
+
36
41
 
37
42
  ## Configuration: credentials
38
43
 
@@ -93,6 +98,51 @@ is set to 3600 seconds.
93
98
  A unique identifier that is used by third parties when assuming roles in
94
99
  their customers' accounts.
95
100
 
101
+ ### web_identity_credentials
102
+
103
+ Similar to the assume_role_credentials, but for usage in EKS.
104
+
105
+ <match *>
106
+ @type s3
107
+
108
+ <web_identity_credentials>
109
+ role_arn ROLE_ARN
110
+ role_session_name ROLE_SESSION_NAME
111
+ web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILE
112
+ </web_identity_credentials>
113
+ </match>
114
+
115
+ See also:
116
+
117
+ * [Using IAM Roles - AWS Identity and Access
118
+ Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
119
+ * [IAM Roles For Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
120
+ * [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
121
+ * [Aws::AssumeRoleWebIdentityCredentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/AssumeRoleWebIdentityCredentials.html)
122
+
123
+ **role_arn (required)**
124
+
125
+ The Amazon Resource Name (ARN) of the role to assume.
126
+
127
+ **role_session_name (required)**
128
+
129
+ An identifier for the assumed role session.
130
+
131
+ **web_identity_token_file (required)**
132
+
133
+ The absolute path to the file on disk containing the OIDC token
134
+
135
+ **policy**
136
+
137
+ An IAM policy in JSON format.
138
+
139
+ **duration_seconds**
140
+
141
+ The duration, in seconds, of the role session. The value can range from
142
+ 900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value
143
+ is set to 3600 seconds.
144
+
145
+
96
146
  ### instance_profile_credentials
97
147
 
98
148
  Retrieve temporary security credentials via HTTP request. This is useful on
@@ -165,7 +215,7 @@ Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
165
215
 
166
216
  ### v1.0 style
167
217
 
168
- With fluentd v1.0 and fluent-plugin-s3 v1.0.0, use new buffer configuration to dynamic parameters.
218
+ With fluentd v1 and fluent-plugin-s3 v1.0.0 or later, use new buffer configuration to dynamic parameters.
169
219
 
170
220
  <match pattern>
171
221
  @type s3
@@ -248,12 +298,18 @@ recommend using `s3_region` instead of `s3_endpoint`.
248
298
  **s3_endpoint**
249
299
 
250
300
  endpoint for S3 compatible services. For example, Riak CS based storage or
251
- something. This option doesn't work on S3, use `s3_region` instead.
301
+ something. This option is deprecated for AWS S3, use `s3_region` instead.
302
+
303
+ See also AWS article: [Working with Regions](https://aws.amazon.com/blogs/developer/working-with-regions/).
252
304
 
253
305
  **enable_transfer_acceleration**
254
306
 
255
307
  Enable [S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) for uploads. **IMPORTANT**: For this to work, you must first enable this feature on your destination S3 bucket.
256
308
 
309
+ **enable_dual_stack**
310
+
311
+ Enable [Amazon S3 Dual-Stack Endpoints](https://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html) for uploads. Will make it possible to use either IPv4 or IPv6 when connecting to S3.
312
+
257
313
  **use_bundled_cert**
258
314
 
259
315
  For cases where the default SSL certificate is unavailable (e.g. Windows), you can set this option to true in order to use the AWS SDK bundled certificate. Default is false.
@@ -297,6 +353,13 @@ You can configure the length of string with a
297
353
  `hex_random_length` parameter (Default: 4).
298
354
 
299
355
  The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
356
+ In addition, you can use [buffer placeholders](https://docs.fluentd.org/configuration/buffer-section#placeholders) in this parameter,
357
+ so you can embed tag, time and record value like below:
358
+
359
+ s3_object_key_format %{path}/events/%Y%m%d/${tag}_%{index}.%{file_extension}
360
+ <buffer tag,time>
361
+ # buffer parameters...
362
+ </buffer>
300
363
 
301
364
  For instance, using the example configuration above, actual object keys on S3
302
365
  will be something like:
@@ -352,7 +415,7 @@ See `Use your compression algorithm` section for adding another format.
352
415
  **`<format>` or format**
353
416
 
354
417
  Change one line format in the S3 object. Supported formats are "out_file",
355
- "json", "ltsv" and "single_value". See also [official Formatter article](https://docs.fluentd.org/formatter).
418
+ "json", "ltsv", "single_value" and other formatter plugins. See also [official Formatter article](https://docs.fluentd.org/formatter).
356
419
 
357
420
  * out_file (default).
358
421
 
@@ -368,11 +431,18 @@ Change one line format in the S3 object. Supported formats are "out_file",
368
431
 
369
432
 
370
433
  At this format, "time" and "tag" are omitted. But you can set these
371
- information to the record by setting "include_tag_key" / "tag_key" and
372
- "include_time_key" / "time_key" option. If you set following configuration in
434
+ information to the record by setting `<inject>` option. If you set following configuration in
373
435
  S3 output:
374
436
 
375
- format json
437
+ # v1
438
+ <format>
439
+ @type json
440
+ </format>
441
+ <inject>
442
+ time_key log_time
443
+ </inject>
444
+ # v0.12
445
+ @format json
376
446
  include_time_key true
377
447
  time_key log_time # default is time
378
448
 
@@ -380,15 +450,14 @@ then the record has log_time field.
380
450
 
381
451
  {"log_time":"time string",...}
382
452
 
453
+ See also [official Inject Section article](https://docs.fluentd.org/configuration/inject-section).
454
+
383
455
  * ltsv
384
456
 
385
457
  key1:value1\tkey2:value2
386
458
  key1:value1\tkey2:value2
387
459
  ...
388
460
 
389
-
390
- "ltsv" format also accepts "include_xxx" related options. See "json" section.
391
-
392
461
  * single_value
393
462
 
394
463
 
@@ -431,21 +500,34 @@ uri of proxy environment.
431
500
  **path**
432
501
 
433
502
  path prefix of the files on S3. Default is "" (no prefix).
503
+ [buffer placeholder](https://docs.fluentd.org/configuration/buffer-section#placeholders) is supported,
504
+ so you can embed tag, time and record value like below.
505
+
506
+ path logs/%Y%m%d/${tag}/
507
+ <buffer tag,time>
508
+ # buffer parameters...
509
+ </buffer>
434
510
 
435
- **buffer_path (required)**
511
+ **buffer_path (for v0.12)**
436
512
 
437
513
  path prefix of the files to buffer logs.
438
514
 
439
- **time_slice_format**
515
+ This parameter is for v0.12. Use `<buffer>`'s `path` in v1.
516
+
517
+ **time_slice_format(for v0.12)**
440
518
 
441
519
  Format of the time used as the file name. Default is '%Y%m%d'. Use
442
520
  '%Y%m%d%H' to split files hourly.
443
521
 
444
- **time_slice_wait**
522
+ This parameter is for v0.12. Use buffer placeholder for `path` / `s3_object_key_format` in v1.
523
+
524
+ **time_slice_wait(for v0.12)**
445
525
 
446
526
  The time to wait old logs. Default is 10 minutes. Specify larger value if
447
527
  old logs may reach.
448
528
 
529
+ This parameter is for v0.12. Use `<buffer>`'s `timekey_wait` in v1.
530
+
449
531
  **utc**
450
532
 
451
533
  Use UTC instead of local time.
data/VERSION CHANGED
@@ -1 +1 @@
1
- 1.2.0
1
+ 1.3.3
@@ -17,8 +17,8 @@ Gem::Specification.new do |gem|
17
17
  gem.require_paths = ['lib']
18
18
 
19
19
  gem.add_dependency "fluentd", [">= 0.14.22", "< 2"]
20
- gem.add_dependency "aws-sdk-s3", "~> 1.0"
21
- gem.add_dependency "aws-sdk-sqs", "~> 1.0"
20
+ gem.add_dependency "aws-sdk-s3", "~> 1.60"
21
+ gem.add_dependency "aws-sdk-sqs", "~> 1.23"
22
22
  gem.add_development_dependency "rake", ">= 0.9.2"
23
23
  gem.add_development_dependency "test-unit", ">= 3.0.8"
24
24
  gem.add_development_dependency "test-unit-rr", ">= 1.0.3"
@@ -42,6 +42,20 @@ module Fluent::Plugin
42
42
  desc "A unique identifier that is used by third parties when assuming roles in their customers' accounts."
43
43
  config_param :external_id, :string, default: nil
44
44
  end
45
+ # See the following link for additional params that could be added:
46
+ # https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/STS/Client.html#assume_role_with_web_identity-instance_method
47
+ config_section :web_identity_credentials, multi: false do
48
+ desc "The Amazon Resource Name (ARN) of the role to assume"
49
+ config_param :role_arn, :string # required
50
+ desc "An identifier for the assumed role session"
51
+ config_param :role_session_name, :string #required
52
+ desc "The absolute path to the file on disk containing the OIDC token"
53
+ config_param :web_identity_token_file, :string #required
54
+ desc "An IAM policy in JSON format"
55
+ config_param :policy, :string, default: nil
56
+ desc "The duration, in seconds, of the role session (900-43200)"
57
+ config_param :duration_seconds, :integer, default: nil
58
+ end
45
59
  config_section :instance_profile_credentials, multi: false do
46
60
  desc "Number of times to retry when retrieving credentials"
47
61
  config_param :retries, :integer, default: nil
@@ -104,11 +118,11 @@ module Fluent::Plugin
104
118
  def configure(conf)
105
119
  super
106
120
 
107
- if @s3_endpoint && @s3_endpoint.end_with?('amazonaws.com')
121
+ if @s3_endpoint && (@s3_endpoint.end_with?('amazonaws.com') && !['fips', 'gov'].any? { |e| @s3_endpoint.include?(e) })
108
122
  raise Fluent::ConfigError, "s3_endpoint parameter is not supported for S3, use s3_region instead. This parameter is for S3 compatible services"
109
123
  end
110
124
 
111
- if @sqs.endpoint && @sqs.endpoint.end_with?('amazonaws.com')
125
+ if @sqs.endpoint && (@sqs.endpoint.end_with?('amazonaws.com') && !['fips', 'gov'].any? { |e| @sqs.endpoint.include?(e) })
112
126
  raise Fluent::ConfigError, "sqs/endpoint parameter is not supported for SQS, use s3_region instead. This parameter is for SQS compatible services"
113
127
  end
114
128
 
@@ -202,6 +216,17 @@ module Fluent::Plugin
202
216
  credentials_options[:client] = Aws::STS::Client.new(:region => @s3_region)
203
217
  end
204
218
  options[:credentials] = Aws::AssumeRoleCredentials.new(credentials_options)
219
+ when @web_identity_credentials
220
+ c = @web_identity_credentials
221
+ credentials_options[:role_arn] = c.role_arn
222
+ credentials_options[:role_session_name] = c.role_session_name
223
+ credentials_options[:web_identity_token_file] = c.web_identity_token_file
224
+ credentials_options[:policy] = c.policy if c.policy
225
+ credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
226
+ if @s3_region
227
+ credentials_options[:client] = Aws::STS::Client.new(:region => @s3_region)
228
+ end
229
+ options[:credentials] = Aws::AssumeRoleWebIdentityCredentials.new(credentials_options)
205
230
  when @instance_profile_credentials
206
231
  c = @instance_profile_credentials
207
232
  credentials_options[:retries] = c.retries if c.retries
@@ -209,7 +234,11 @@ module Fluent::Plugin
209
234
  credentials_options[:port] = c.port if c.port
210
235
  credentials_options[:http_open_timeout] = c.http_open_timeout if c.http_open_timeout
211
236
  credentials_options[:http_read_timeout] = c.http_read_timeout if c.http_read_timeout
212
- options[:credentials] = Aws::InstanceProfileCredentials.new(credentials_options)
237
+ if ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
238
+ options[:credentials] = Aws::ECSCredentials.new(credentials_options)
239
+ else
240
+ options[:credentials] = Aws::InstanceProfileCredentials.new(credentials_options)
241
+ end
213
242
  when @shared_credentials
214
243
  c = @shared_credentials
215
244
  credentials_options[:path] = c.path if c.path
@@ -39,6 +39,24 @@ module Fluent::Plugin
39
39
  config_param :duration_seconds, :integer, default: nil
40
40
  desc "A unique identifier that is used by third parties when assuming roles in their customers' accounts."
41
41
  config_param :external_id, :string, default: nil, secret: true
42
+ desc "The region of the STS endpoint to use."
43
+ config_param :sts_region, :string, default: nil
44
+ end
45
+ # See the following link for additional params that could be added:
46
+ # https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/STS/Client.html#assume_role_with_web_identity-instance_method
47
+ config_section :web_identity_credentials, multi: false do
48
+ desc "The Amazon Resource Name (ARN) of the role to assume"
49
+ config_param :role_arn, :string # required
50
+ desc "An identifier for the assumed role session"
51
+ config_param :role_session_name, :string #required
52
+ desc "The absolute path to the file on disk containing the OIDC token"
53
+ config_param :web_identity_token_file, :string #required
54
+ desc "An IAM policy in JSON format"
55
+ config_param :policy, :string, default: nil
56
+ desc "The duration, in seconds, of the role session (900-43200)"
57
+ config_param :duration_seconds, :integer, default: nil
58
+ desc "The region of the STS endpoint to use."
59
+ config_param :sts_region, :string, default: nil
42
60
  end
43
61
  config_section :instance_profile_credentials, multi: false do
44
62
  desc "Number of times to retry when retrieving credentials"
@@ -70,6 +88,8 @@ module Fluent::Plugin
70
88
  config_param :s3_endpoint, :string, default: nil
71
89
  desc "If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket"
72
90
  config_param :enable_transfer_acceleration, :bool, default: false
91
+ desc "If true, use Amazon S3 Dual-Stack Endpoints. Will make it possible to use either IPv4 or IPv6 when connecting to S3."
92
+ config_param :enable_dual_stack, :bool, default: false
73
93
  desc "If false, the certificate of endpoint will not be verified"
74
94
  config_param :ssl_verify_peer, :bool, :default => true
75
95
  desc "The format of S3 object keys"
@@ -155,7 +175,7 @@ module Fluent::Plugin
155
175
 
156
176
  Aws.use_bundled_cert! if @use_bundled_cert
157
177
 
158
- if @s3_endpoint && @s3_endpoint.end_with?('amazonaws.com')
178
+ if @s3_endpoint && (@s3_endpoint.end_with?('amazonaws.com') && !['fips', 'gov'].any? { |e| @s3_endpoint.include?(e) })
159
179
  raise Fluent::ConfigError, "s3_endpoint parameter is not supported for S3, use s3_region instead. This parameter is for S3 compatible services"
160
180
  end
161
181
 
@@ -175,7 +195,7 @@ module Fluent::Plugin
175
195
  end
176
196
 
177
197
  unless @index_format =~ /^%(0\d*)?[dxX]$/
178
- raise Fluent::ConfigError, "index_format parameter should follow `%[flags][width]type`. `0` is the only supported flag, and is mandatory if width is specified. `d`, `x` and `X` are supported types"
198
+ raise Fluent::ConfigError, "index_format parameter should follow `%[flags][width]type`. `0` is the only supported flag, and is mandatory if width is specified. `d`, `x` and `X` are supported types"
179
199
  end
180
200
 
181
201
  if @reduced_redundancy
@@ -193,6 +213,8 @@ module Fluent::Plugin
193
213
  end
194
214
  end
195
215
 
216
+ check_s3_path_safety(conf)
217
+
196
218
  # For backward compatibility
197
219
  # TODO: Remove time_slice_format when end of support compat_parameters
198
220
  @configured_time_slice_format = conf['time_slice_format']
@@ -209,6 +231,7 @@ module Fluent::Plugin
209
231
  options[:region] = @s3_region if @s3_region
210
232
  options[:endpoint] = @s3_endpoint if @s3_endpoint
211
233
  options[:use_accelerate_endpoint] = @enable_transfer_acceleration
234
+ options[:use_dualstack_endpoint] = @enable_dual_stack
212
235
  options[:http_proxy] = @proxy_uri if @proxy_uri
213
236
  options[:force_path_style] = @force_path_style
214
237
  options[:compute_checksums] = @compute_checksums unless @compute_checksums.nil?
@@ -434,6 +457,16 @@ module Fluent::Plugin
434
457
  }
435
458
  end
436
459
 
460
+ def check_s3_path_safety(conf)
461
+ unless conf.has_key?('s3_object_key_format')
462
+ log.warn "The default value of s3_object_key_format will use ${chunk_id} instead of %{index} to avoid object conflict in v2"
463
+ end
464
+
465
+ if (@buffer_config.flush_thread_count > 1) && ['${chunk_id}', '%{uuid_flush}'].none? { |key| @s3_object_key_format.include?(key) }
466
+ log.warn "No ${chunk_id} or %{uuid_flush} in s3_object_key_format with multiple flush threads. Recommend to set ${chunk_id} or %{uuid_flush} to avoid data lost by object conflict"
467
+ end
468
+ end
469
+
437
470
  def check_apikeys
438
471
  @bucket.objects(prefix: @path, :max_keys => 1).first
439
472
  rescue Aws::S3::Errors::NoSuchBucket
@@ -456,10 +489,25 @@ module Fluent::Plugin
456
489
  credentials_options[:policy] = c.policy if c.policy
457
490
  credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
458
491
  credentials_options[:external_id] = c.external_id if c.external_id
459
- if @s3_region
492
+ if c.sts_region
493
+ credentials_options[:client] = Aws::STS::Client.new(region: c.sts_region)
494
+ elsif @s3_region
460
495
  credentials_options[:client] = Aws::STS::Client.new(region: @s3_region)
461
496
  end
462
497
  options[:credentials] = Aws::AssumeRoleCredentials.new(credentials_options)
498
+ when @web_identity_credentials
499
+ c = @web_identity_credentials
500
+ credentials_options[:role_arn] = c.role_arn
501
+ credentials_options[:role_session_name] = c.role_session_name
502
+ credentials_options[:web_identity_token_file] = c.web_identity_token_file
503
+ credentials_options[:policy] = c.policy if c.policy
504
+ credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
505
+ if c.sts_region
506
+ credentials_options[:client] = Aws::STS::Client.new(:region => c.sts_region)
507
+ elsif @s3_region
508
+ credentials_options[:client] = Aws::STS::Client.new(:region => @s3_region)
509
+ end
510
+ options[:credentials] = Aws::AssumeRoleWebIdentityCredentials.new(credentials_options)
463
511
  when @instance_profile_credentials
464
512
  c = @instance_profile_credentials
465
513
  credentials_options[:retries] = c.retries if c.retries
@@ -11,6 +11,7 @@ require 'fluent/plugin/in_s3'
11
11
  require 'test/unit/rr'
12
12
  require 'zlib'
13
13
  require 'fileutils'
14
+ require 'ostruct'
14
15
 
15
16
  include Fluent::Test::Helpers
16
17
 
@@ -152,6 +153,7 @@ EOS
152
153
 
153
154
  def setup_mocks
154
155
  @s3_client = stub(Aws::S3::Client.new(stub_responses: true))
156
+ stub(@s3_client).config { OpenStruct.new({region: "us-east-1"}) }
155
157
  mock(Aws::S3::Client).new(anything).at_least(0) { @s3_client }
156
158
  @s3_resource = mock(Aws::S3::Resource.new(client: @s3_client))
157
159
  mock(Aws::S3::Resource).new(client: @s3_client) { @s3_resource }
@@ -10,6 +10,7 @@ require 'zlib'
10
10
  require 'fileutils'
11
11
  require 'timecop'
12
12
  require 'uuidtools'
13
+ require 'ostruct'
13
14
 
14
15
  include Fluent::Test::Helpers
15
16
 
@@ -427,6 +428,7 @@ EOC
427
428
 
428
429
  def setup_mocks(exists_return = false)
429
430
  @s3_client = stub(Aws::S3::Client.new(stub_responses: true))
431
+ stub(@s3_client).config { OpenStruct.new({region: "us-east-1"}) }
430
432
  # aws-sdk-s3 calls Client#put_object inside Object#put
431
433
  mock(@s3_client).put_object(anything).at_least(0) { MockResponse.new({}) }
432
434
  mock(Aws::S3::Client).new(anything).at_least(0) { @s3_client }
@@ -464,6 +466,7 @@ EOC
464
466
 
465
467
  def setup_mocks_hardened_policy()
466
468
  @s3_client = stub(Aws::S3::Client.new(:stub_responses => true))
469
+ stub(@s3_client).config { OpenStruct.new({region: "us-east-1"}) }
467
470
  mock(@s3_client).put_object(anything).at_least(0) { MockResponse.new({}) }
468
471
  mock(Aws::S3::Client).new(anything).at_least(0) { @s3_client }
469
472
  @s3_resource = mock(Aws::S3::Resource.new(:client => @s3_client))
@@ -564,6 +567,62 @@ EOC
564
567
  assert_equal(expected_credentials, credentials)
565
568
  end
566
569
 
570
+ def test_web_identity_credentials
571
+ expected_credentials = Aws::Credentials.new("test_key", "test_secret")
572
+ mock(Aws::AssumeRoleWebIdentityCredentials).new(
573
+ role_arn: "test_arn",
574
+ role_session_name: "test_session",
575
+ web_identity_token_file: "test_file",
576
+ client: anything
577
+ ){
578
+ expected_credentials
579
+ }
580
+
581
+ config = CONFIG_TIME_SLICE.split("\n").reject{|x| x =~ /.+aws_.+/}.join("\n")
582
+ config += %[
583
+ <web_identity_credentials>
584
+ role_arn test_arn
585
+ role_session_name test_session
586
+ web_identity_token_file test_file
587
+ </web_identity_credentials>
588
+ ]
589
+ d = create_time_sliced_driver(config)
590
+ assert_nothing_raised { d.run {} }
591
+ client = d.instance.instance_variable_get(:@s3).client
592
+ credentials = client.config.credentials
593
+ assert_equal(expected_credentials, credentials)
594
+ end
595
+
596
+ def test_web_identity_credentials_with_sts_region
597
+ expected_credentials = Aws::Credentials.new("test_key", "test_secret")
598
+ sts_client = Aws::STS::Client.new(region: 'us-east-1')
599
+ mock(Aws::STS::Client).new(region: 'us-east-1'){ sts_client }
600
+ mock(Aws::AssumeRoleWebIdentityCredentials).new(
601
+ role_arn: "test_arn",
602
+ role_session_name: "test_session",
603
+ web_identity_token_file: "test_file",
604
+ client: sts_client
605
+ ){
606
+ expected_credentials
607
+ }
608
+
609
+ config = CONFIG_TIME_SLICE.split("\n").reject{|x| x =~ /.+aws_.+/}.join("\n")
610
+ config += %[
611
+ s3_region us-west-2
612
+ <web_identity_credentials>
613
+ role_arn test_arn
614
+ role_session_name test_session
615
+ web_identity_token_file test_file
616
+ sts_region us-east-1
617
+ </web_identity_credentials>
618
+ ]
619
+ d = create_time_sliced_driver(config)
620
+ assert_nothing_raised { d.run {} }
621
+ client = d.instance.instance_variable_get(:@s3).client
622
+ credentials = client.config.credentials
623
+ assert_equal(expected_credentials, credentials)
624
+ end
625
+
567
626
  def test_instance_profile_credentials
568
627
  expected_credentials = Aws::Credentials.new("test_key", "test_secret")
569
628
  mock(Aws::InstanceProfileCredentials).new({}).returns(expected_credentials)
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-s3
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.0
4
+ version: 1.3.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sadayuki Furuhashi
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2019-10-18 00:00:00.000000000 Z
12
+ date: 2020-06-25 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: fluentd
@@ -37,28 +37,28 @@ dependencies:
37
37
  requirements:
38
38
  - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: '1.0'
40
+ version: '1.60'
41
41
  type: :runtime
42
42
  prerelease: false
43
43
  version_requirements: !ruby/object:Gem::Requirement
44
44
  requirements:
45
45
  - - "~>"
46
46
  - !ruby/object:Gem::Version
47
- version: '1.0'
47
+ version: '1.60'
48
48
  - !ruby/object:Gem::Dependency
49
49
  name: aws-sdk-sqs
50
50
  requirement: !ruby/object:Gem::Requirement
51
51
  requirements:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
- version: '1.0'
54
+ version: '1.23'
55
55
  type: :runtime
56
56
  prerelease: false
57
57
  version_requirements: !ruby/object:Gem::Requirement
58
58
  requirements:
59
59
  - - "~>"
60
60
  - !ruby/object:Gem::Version
61
- version: '1.0'
61
+ version: '1.23'
62
62
  - !ruby/object:Gem::Dependency
63
63
  name: rake
64
64
  requirement: !ruby/object:Gem::Requirement