fluent-plugin-s3 1.1.8 → 1.1.9
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/ChangeLog +5 -0
- data/README.md +29 -23
- data/VERSION +1 -1
- data/fluent-plugin-s3.gemspec +1 -1
- data/lib/fluent/plugin/out_s3.rb +6 -3
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9946aa9a708d9fe24ef1a4582334bea3683efb1b
|
4
|
+
data.tar.gz: ff8eb6d8aca6f8fbcdc7d8dd20b354419e248150
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: aa61687fed5f22827e94fde41098336c3ae3935352e294f70d490ccca5763563c5d1426f3a0a55335ab2b8d5e7a49575bdc762c644b77f57fac73bee4d06dd09
|
7
|
+
data.tar.gz: 43958485397cf76a352352312890c4445c26122c81f0b251c6d42704e60bc28087963b1fce4f0e3f311cc62e92ad7b6fb3c33e1c3dce51f17cc4eff31c4f5bd9
|
data/ChangeLog
CHANGED
data/README.md
CHANGED
@@ -253,6 +253,10 @@ recommend using `s3_region` instead of `s3_endpoint`.
|
|
253
253
|
endpoint for S3 compatible services. For example, Riak CS based storage or
|
254
254
|
something. This option doesn't work on S3, use `s3_region` instead.
|
255
255
|
|
256
|
+
**enable_transfer_acceleration**
|
257
|
+
|
258
|
+
Enable [S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) for uploads. **IMPORTANT**: For this to work, you must first enable this feature on your destination S3 bucket.
|
259
|
+
|
256
260
|
**use_bundled_cert**
|
257
261
|
|
258
262
|
For cases where the default SSL certificate is unavailable (e.g. Windows), you can set this option to true in order to use the AWS SDK bundled certificate. Default is false.
|
@@ -406,29 +410,16 @@ Create S3 bucket if it does not exists. Default is true.
|
|
406
410
|
|
407
411
|
Check mentioned bucket if it exists in AWS or not. Default is true.
|
408
412
|
|
409
|
-
When it is false,
|
410
|
-
|
411
|
-
case where bucket will be pre-created before running fluentd.
|
413
|
+
When it is false, fluentd will not check aws s3 for the existence of the mentioned bucket.
|
414
|
+
This is the case where bucket will be pre-created before running fluentd.
|
412
415
|
|
413
416
|
**check_object**
|
414
417
|
|
415
418
|
Check object before creation if it exists or not. Default is true.
|
416
419
|
|
417
|
-
When it is false,
|
418
|
-
|
419
|
-
|
420
|
-
Example object name, assuming it is created on 2016/16/11 3:30:54 PM
|
421
|
-
20161611_153054.txt (extension can be anything as per user's choice)
|
422
|
-
|
423
|
-
**Example when check_bucket=false and check_object=false**
|
424
|
-
|
425
|
-
When the mentioned configuration will be made, fluentd will work with the
|
426
|
-
minimum IAM poilcy, like:
|
427
|
-
"Statement": [{
|
428
|
-
"Effect": "Allow",
|
429
|
-
"Action": "s3:PutObject",
|
430
|
-
"Resource": ["*"]
|
431
|
-
}]
|
420
|
+
When it is false, s3_object_key_format will be %{path}%{time_slice}_%{hms_slice}.%{file_extension} by default where,
|
421
|
+
hms_slice will be time-slice in hhmmss format, so that each object will be unique.
|
422
|
+
Example object name, assuming it is created on 2016/16/11 3:30:54 PM 20161611_153054.txt (extension can be anything as per user's choice)
|
432
423
|
|
433
424
|
**check_apikey_on_start**
|
434
425
|
|
@@ -656,8 +647,7 @@ Interval to retry polling SQS if polling unsuccessful, in seconds. Default is 30
|
|
656
647
|
|
657
648
|
## IAM Policy
|
658
649
|
|
659
|
-
The following is an example for a
|
660
|
-
bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
|
650
|
+
The following is an example for a IAM policy needed to write to an s3 bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
|
661
651
|
|
662
652
|
{
|
663
653
|
"Version": "2012-10-17",
|
@@ -672,15 +662,18 @@ bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
|
|
672
662
|
{
|
673
663
|
"Effect": "Allow",
|
674
664
|
"Action": [
|
675
|
-
"s3:PutObject"
|
665
|
+
"s3:PutObject",
|
666
|
+
"s3:GetObject"
|
676
667
|
],
|
677
668
|
"Resource": "arn:aws:s3:::my-s3bucket/*"
|
678
669
|
}
|
679
670
|
]
|
680
671
|
}
|
681
672
|
|
682
|
-
Note that the bucket must already exist and **auto_create_bucket** has no
|
683
|
-
|
673
|
+
Note that the bucket must already exist and **auto_create_bucket** has no effect in this case.
|
674
|
+
|
675
|
+
`s3:GetObject` is needed for object check to avoid object overwritten.
|
676
|
+
If you set `check_object false`, `s3:GetObject` is not needed.
|
684
677
|
|
685
678
|
Refer to the [AWS
|
686
679
|
documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolicies.html) for example policies.
|
@@ -690,6 +683,19 @@ roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
|
|
690
683
|
with a properly configured IAM policy are preferred over embedding access keys
|
691
684
|
on EC2 instances.
|
692
685
|
|
686
|
+
### Example when `check_bucket false` and `check_object false`
|
687
|
+
|
688
|
+
When the mentioned configuration will be made, fluentd will work with the
|
689
|
+
minimum IAM poilcy, like:
|
690
|
+
|
691
|
+
|
692
|
+
"Statement": [{
|
693
|
+
"Effect": "Allow",
|
694
|
+
"Action": "s3:PutObject",
|
695
|
+
"Resource": ["*"]
|
696
|
+
}]
|
697
|
+
|
698
|
+
|
693
699
|
## Use your (de)compression algorithm
|
694
700
|
|
695
701
|
s3 plugin has pluggable compression mechanizm like Fluentd's input / output
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
1.1.
|
1
|
+
1.1.9
|
data/fluent-plugin-s3.gemspec
CHANGED
@@ -17,7 +17,7 @@ Gem::Specification.new do |gem|
|
|
17
17
|
gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
18
18
|
gem.require_paths = ['lib']
|
19
19
|
|
20
|
-
gem.add_dependency "fluentd", [">= 0.14.
|
20
|
+
gem.add_dependency "fluentd", [">= 0.14.22", "< 2"]
|
21
21
|
gem.add_dependency "aws-sdk-s3", "~> 1.0"
|
22
22
|
gem.add_dependency "aws-sdk-sqs", "~> 1.0"
|
23
23
|
gem.add_development_dependency "rake", ">= 0.9.2"
|
data/lib/fluent/plugin/out_s3.rb
CHANGED
@@ -68,6 +68,8 @@ module Fluent::Plugin
|
|
68
68
|
config_param :s3_region, :string, default: ENV["AWS_REGION"] || "us-east-1"
|
69
69
|
desc "Use 's3_region' instead"
|
70
70
|
config_param :s3_endpoint, :string, default: nil
|
71
|
+
desc "If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket"
|
72
|
+
config_param :enable_transfer_acceleration, :bool, default: false
|
71
73
|
desc "If false, the certificate of endpoint will not be verified"
|
72
74
|
config_param :ssl_verify_peer, :bool, :default => true
|
73
75
|
desc "The format of S3 object keys"
|
@@ -198,6 +200,7 @@ module Fluent::Plugin
|
|
198
200
|
options = setup_credentials
|
199
201
|
options[:region] = @s3_region if @s3_region
|
200
202
|
options[:endpoint] = @s3_endpoint if @s3_endpoint
|
203
|
+
options[:use_accelerate_endpoint] = @enable_transfer_acceleration
|
201
204
|
options[:http_proxy] = @proxy_uri if @proxy_uri
|
202
205
|
options[:force_path_style] = @force_path_style
|
203
206
|
options[:compute_checksums] = @compute_checksums unless @compute_checksums.nil?
|
@@ -251,7 +254,7 @@ module Fluent::Plugin
|
|
251
254
|
s3path = @s3_object_key_format.gsub(%r(%{[^}]+})) do |matched_key|
|
252
255
|
values_for_s3_object_key_pre.fetch(matched_key, matched_key)
|
253
256
|
end
|
254
|
-
s3path = extract_placeholders(s3path,
|
257
|
+
s3path = extract_placeholders(s3path, chunk)
|
255
258
|
s3path = s3path.gsub(%r(%{[^}]+}), values_for_s3_object_key_post)
|
256
259
|
if (i > 0) && (s3path == previous_path)
|
257
260
|
if @overwrite
|
@@ -289,7 +292,7 @@ module Fluent::Plugin
|
|
289
292
|
s3path = @s3_object_key_format.gsub(%r(%{[^}]+})) do |matched_key|
|
290
293
|
values_for_s3_object_key_pre.fetch(matched_key, matched_key)
|
291
294
|
end
|
292
|
-
s3path = extract_placeholders(s3path,
|
295
|
+
s3path = extract_placeholders(s3path, chunk)
|
293
296
|
s3path = s3path.gsub(%r(%{[^}]+}), values_for_s3_object_key_post)
|
294
297
|
end
|
295
298
|
|
@@ -319,7 +322,7 @@ module Fluent::Plugin
|
|
319
322
|
if @s3_metadata
|
320
323
|
put_options[:metadata] = {}
|
321
324
|
@s3_metadata.each do |k, v|
|
322
|
-
put_options[:metadata][k] = extract_placeholders(v,
|
325
|
+
put_options[:metadata][k] = extract_placeholders(v, chunk).gsub(%r(%{[^}]+}), {"%{index}" => sprintf(@index_format, i - 1)})
|
323
326
|
end
|
324
327
|
end
|
325
328
|
@bucket.object(s3path).put(put_options)
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-s3
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.1.
|
4
|
+
version: 1.1.9
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sadayuki Furuhashi
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2019-
|
12
|
+
date: 2019-03-27 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: fluentd
|
@@ -17,7 +17,7 @@ dependencies:
|
|
17
17
|
requirements:
|
18
18
|
- - ">="
|
19
19
|
- !ruby/object:Gem::Version
|
20
|
-
version: 0.14.
|
20
|
+
version: 0.14.22
|
21
21
|
- - "<"
|
22
22
|
- !ruby/object:Gem::Version
|
23
23
|
version: '2'
|
@@ -27,7 +27,7 @@ dependencies:
|
|
27
27
|
requirements:
|
28
28
|
- - ">="
|
29
29
|
- !ruby/object:Gem::Version
|
30
|
-
version: 0.14.
|
30
|
+
version: 0.14.22
|
31
31
|
- - "<"
|
32
32
|
- !ruby/object:Gem::Version
|
33
33
|
version: '2'
|