fluent-plugin-s3 1.4.0 → 1.6.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.yaml +72 -0
- data/.github/ISSUE_TEMPLATE/config.yml +5 -0
- data/.github/ISSUE_TEMPLATE/feature_request.yaml +38 -0
- data/.github/workflows/linux.yml +26 -0
- data/.github/workflows/stale-actions.yml +22 -0
- data/ChangeLog +16 -0
- data/README.md +13 -781
- data/VERSION +1 -1
- data/docs/credentials.md +171 -0
- data/docs/howto.md +92 -0
- data/docs/input.md +90 -0
- data/docs/output.md +449 -0
- data/docs/v0.12.md +52 -0
- data/lib/fluent/plugin/in_s3.rb +6 -1
- data/lib/fluent/plugin/out_s3.rb +33 -15
- data/lib/fluent/plugin/s3_compressor_parquet.rb +83 -0
- data/test/test_in_s3.rb +9 -4
- data/test/test_out_s3.rb +131 -0
- metadata +15 -7
- data/.travis.yml +0 -24
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: fbb79ec06cc2a084f5043825a68b9e15b3d3eed9008f0906ff6fedede8659a22
|
|
4
|
+
data.tar.gz: '09eb263937f9f5dd397296579426a744f701eee6ba45d4a2b05b0f5c7990fe5b'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5a0685a1d250f12608bf432c4157d6a1b9cd7a627161cf2278c703ae07cce9fe858d5468bbe5056d060f40914570ea9ec39bbec04828c1dbf193737edf72193a
|
|
7
|
+
data.tar.gz: 0caf64221c557536e5e3d1d6f21f28cfdce5fc865a4a4d5b55ec490fd716701675b49999faf81b9e030348abbd68e265df690154f2aac155613efbfcf4ba144d
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
name: Bug Report
|
|
2
|
+
description: Create a report with a procedure for reproducing the bug
|
|
3
|
+
body:
|
|
4
|
+
- type: markdown
|
|
5
|
+
attributes:
|
|
6
|
+
value: |
|
|
7
|
+
Check [README](https://github.com/fluent/fluent-plugin-s3#readme) first and here is the list to help us investigate the problem.
|
|
8
|
+
- type: textarea
|
|
9
|
+
id: description
|
|
10
|
+
attributes:
|
|
11
|
+
label: Describe the bug
|
|
12
|
+
description: A clear and concise description of what the bug is
|
|
13
|
+
validations:
|
|
14
|
+
required: true
|
|
15
|
+
- type: textarea
|
|
16
|
+
id: reproduce
|
|
17
|
+
attributes:
|
|
18
|
+
label: To Reproduce
|
|
19
|
+
description: Steps to reproduce the behavior
|
|
20
|
+
validations:
|
|
21
|
+
required: true
|
|
22
|
+
- type: textarea
|
|
23
|
+
id: expected
|
|
24
|
+
attributes:
|
|
25
|
+
label: Expected behavior
|
|
26
|
+
description: A clear and concise description of what you expected to happen
|
|
27
|
+
validations:
|
|
28
|
+
required: true
|
|
29
|
+
- type: textarea
|
|
30
|
+
id: environment
|
|
31
|
+
attributes:
|
|
32
|
+
label: Your Environment
|
|
33
|
+
description: |
|
|
34
|
+
- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
|
|
35
|
+
- Operating system: `cat /etc/os-release`
|
|
36
|
+
- Kernel version: `uname -r`
|
|
37
|
+
|
|
38
|
+
Tip: If you hit the problem with older fluentd version, try latest version first.
|
|
39
|
+
value: |
|
|
40
|
+
- Fluentd version:
|
|
41
|
+
- TD Agent version:
|
|
42
|
+
- fluent-plugin-s3 version:
|
|
43
|
+
- aws-sdk-s3 version:
|
|
44
|
+
- aws-sdk-sqs version:
|
|
45
|
+
- Operating system:
|
|
46
|
+
- Kernel version:
|
|
47
|
+
render: markdown
|
|
48
|
+
validations:
|
|
49
|
+
required: true
|
|
50
|
+
- type: textarea
|
|
51
|
+
id: configuration
|
|
52
|
+
attributes:
|
|
53
|
+
label: Your Configuration
|
|
54
|
+
description: |
|
|
55
|
+
Write your configuration here. Minimum reproducible fluentd.conf is recommended.
|
|
56
|
+
validations:
|
|
57
|
+
required: true
|
|
58
|
+
- type: textarea
|
|
59
|
+
id: logs
|
|
60
|
+
attributes:
|
|
61
|
+
label: Your Error Log
|
|
62
|
+
description: Write your ALL error log here
|
|
63
|
+
render: shell
|
|
64
|
+
validations:
|
|
65
|
+
required: true
|
|
66
|
+
- type: textarea
|
|
67
|
+
id: addtional-context
|
|
68
|
+
attributes:
|
|
69
|
+
label: Additional context
|
|
70
|
+
description: Add any other context about the problem here.
|
|
71
|
+
validations:
|
|
72
|
+
required: false
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
name: Feature request
|
|
2
|
+
description: Suggest an idea for this project
|
|
3
|
+
body:
|
|
4
|
+
- type: markdown
|
|
5
|
+
attributes:
|
|
6
|
+
value: |
|
|
7
|
+
Check [README.md](https://github.com/fluent/fluent-plugin-s3/blob/master/README.md) first and here is the list to help us investigate the problem.
|
|
8
|
+
- type: textarea
|
|
9
|
+
id: description
|
|
10
|
+
attributes:
|
|
11
|
+
label: Is your feature request related to a problem? Please describe.
|
|
12
|
+
description: |
|
|
13
|
+
A clear and concise description of what the problem is.
|
|
14
|
+
Ex. I'm always frustrated when [...]
|
|
15
|
+
validations:
|
|
16
|
+
required: true
|
|
17
|
+
- type: textarea
|
|
18
|
+
id: solution
|
|
19
|
+
attributes:
|
|
20
|
+
label: Describe the solution you'd like
|
|
21
|
+
description: A clear and concise description of what you want to happen.
|
|
22
|
+
validations:
|
|
23
|
+
required: true
|
|
24
|
+
- type: textarea
|
|
25
|
+
id: alternative
|
|
26
|
+
attributes:
|
|
27
|
+
label: Describe alternatives you've considered
|
|
28
|
+
description: A clear and concise description of any alternative solutions or features you've considered.
|
|
29
|
+
validations:
|
|
30
|
+
required: true
|
|
31
|
+
- type: textarea
|
|
32
|
+
id: addtional-context
|
|
33
|
+
attributes:
|
|
34
|
+
label: Additional context
|
|
35
|
+
description: Add any other context or screenshots about the feature request here.
|
|
36
|
+
validations:
|
|
37
|
+
required: false
|
|
38
|
+
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
name: linux
|
|
2
|
+
on:
|
|
3
|
+
- push
|
|
4
|
+
- pull_request
|
|
5
|
+
jobs:
|
|
6
|
+
build:
|
|
7
|
+
runs-on: ${{ matrix.os }}
|
|
8
|
+
strategy:
|
|
9
|
+
fail-fast: false
|
|
10
|
+
matrix:
|
|
11
|
+
ruby: [ '2.4', '2.5', '2.6', '2.7' ]
|
|
12
|
+
os:
|
|
13
|
+
- ubuntu-latest
|
|
14
|
+
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v2
|
|
17
|
+
- uses: ruby/setup-ruby@v1
|
|
18
|
+
with:
|
|
19
|
+
ruby-version: ${{ matrix.ruby }}
|
|
20
|
+
- name: unit testing
|
|
21
|
+
env:
|
|
22
|
+
CI: true
|
|
23
|
+
run: |
|
|
24
|
+
gem install bundler rake
|
|
25
|
+
bundle install --jobs 4 --retry 3
|
|
26
|
+
bundle exec rake test
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
name: "Mark or close stale issues and PRs"
|
|
2
|
+
on:
|
|
3
|
+
schedule:
|
|
4
|
+
- cron: "00 10 * * *"
|
|
5
|
+
|
|
6
|
+
jobs:
|
|
7
|
+
stale:
|
|
8
|
+
runs-on: ubuntu-latest
|
|
9
|
+
steps:
|
|
10
|
+
- uses: actions/stale@v3
|
|
11
|
+
with:
|
|
12
|
+
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
|
13
|
+
days-before-stale: 90
|
|
14
|
+
days-before-close: 30
|
|
15
|
+
stale-issue-message: "This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days"
|
|
16
|
+
stale-pr-message: "This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days"
|
|
17
|
+
close-issue-message: "This issue was automatically closed because of stale in 30 days"
|
|
18
|
+
close-pr-message: "This PR was automatically closed because of stale in 30 days"
|
|
19
|
+
stale-pr-label: "stale"
|
|
20
|
+
stale-issue-label: "stale"
|
|
21
|
+
exempt-issue-labels: "bug,help wanted"
|
|
22
|
+
exempt-pr-labels: "bug,help wanted"
|
data/ChangeLog
CHANGED
|
@@ -1,3 +1,19 @@
|
|
|
1
|
+
Release 1.6.1 - 2021/08/19
|
|
2
|
+
|
|
3
|
+
* in_s3/out_s3: Don't raise error when s3_endpoint is used for VPC endpoint (GitHub: #384)
|
|
4
|
+
|
|
5
|
+
Release 1.6.0 - 2021/04/08
|
|
6
|
+
|
|
7
|
+
* out_s3: Add support for Parquet compressor. Use `<compress>` section to configure columnify command behavior.
|
|
8
|
+
|
|
9
|
+
Release 1.5.1 - 2021/02/16
|
|
10
|
+
|
|
11
|
+
* out_s3: Fix assume_role_credentials regression
|
|
12
|
+
|
|
13
|
+
Release 1.5.0 - 2020/12/18
|
|
14
|
+
|
|
15
|
+
* out_s3: Fix IAM credentials handling. Prefer assume_role_credentials and region parameter bug.
|
|
16
|
+
|
|
1
17
|
Release 1.4.0 - 2020/08/02
|
|
2
18
|
|
|
3
19
|
* Remove uuidtools dependency
|
data/README.md
CHANGED
|
@@ -20,6 +20,8 @@ the former one is stored in "20110102.gz" file, and latter one in
|
|
|
20
20
|
SQS queue on the region same as S3 bucket.
|
|
21
21
|
We must setup SQS queue and S3 event notification before use this plugin.
|
|
22
22
|
|
|
23
|
+
:warning: Be sure to keep a close eye on S3 costs, as a few user have reported [unexpectedly high costs](https://github.com/fluent/fluent-plugin-s3/issues/160).
|
|
24
|
+
|
|
23
25
|
## Requirements
|
|
24
26
|
|
|
25
27
|
| fluent-plugin-s3 | fluentd | ruby |
|
|
@@ -43,795 +45,25 @@ Simply use RubyGems:
|
|
|
43
45
|
|
|
44
46
|
Both S3 input/output plugin provide several credential methods for authentication/authorization.
|
|
45
47
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
These parameters are required when your agent is not running on EC2 instance with an IAM Role. When using an IAM role, make sure to configure `instance_profile_credentials`. Usage can be found below.
|
|
49
|
-
|
|
50
|
-
**aws_key_id**
|
|
51
|
-
|
|
52
|
-
AWS access key id.
|
|
53
|
-
|
|
54
|
-
**aws_sec_key**
|
|
55
|
-
|
|
56
|
-
AWS secret key.
|
|
57
|
-
|
|
58
|
-
### assume_role_credentials
|
|
59
|
-
|
|
60
|
-
Typically, you use AssumeRole for cross-account access or federation.
|
|
61
|
-
|
|
62
|
-
<match *>
|
|
63
|
-
@type s3
|
|
64
|
-
|
|
65
|
-
<assume_role_credentials>
|
|
66
|
-
role_arn ROLE_ARN
|
|
67
|
-
role_session_name ROLE_SESSION_NAME
|
|
68
|
-
</assume_role_credentials>
|
|
69
|
-
</match>
|
|
70
|
-
|
|
71
|
-
See also:
|
|
72
|
-
|
|
73
|
-
* [Using IAM Roles - AWS Identity and Access
|
|
74
|
-
Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
|
|
75
|
-
* [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
|
|
76
|
-
* [Aws::AssumeRoleCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html)
|
|
77
|
-
|
|
78
|
-
**role_arn (required)**
|
|
79
|
-
|
|
80
|
-
The Amazon Resource Name (ARN) of the role to assume.
|
|
81
|
-
|
|
82
|
-
**role_session_name (required)**
|
|
83
|
-
|
|
84
|
-
An identifier for the assumed role session.
|
|
85
|
-
|
|
86
|
-
**policy**
|
|
87
|
-
|
|
88
|
-
An IAM policy in JSON format.
|
|
89
|
-
|
|
90
|
-
**duration_seconds**
|
|
91
|
-
|
|
92
|
-
The duration, in seconds, of the role session. The value can range from
|
|
93
|
-
900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value
|
|
94
|
-
is set to 3600 seconds.
|
|
95
|
-
|
|
96
|
-
**external_id**
|
|
97
|
-
|
|
98
|
-
A unique identifier that is used by third parties when assuming roles in
|
|
99
|
-
their customers' accounts.
|
|
100
|
-
|
|
101
|
-
### web_identity_credentials
|
|
102
|
-
|
|
103
|
-
Similar to the assume_role_credentials, but for usage in EKS.
|
|
104
|
-
|
|
105
|
-
<match *>
|
|
106
|
-
@type s3
|
|
107
|
-
|
|
108
|
-
<web_identity_credentials>
|
|
109
|
-
role_arn ROLE_ARN
|
|
110
|
-
role_session_name ROLE_SESSION_NAME
|
|
111
|
-
web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILE
|
|
112
|
-
</web_identity_credentials>
|
|
113
|
-
</match>
|
|
114
|
-
|
|
115
|
-
See also:
|
|
116
|
-
|
|
117
|
-
* [Using IAM Roles - AWS Identity and Access
|
|
118
|
-
Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
|
|
119
|
-
* [IAM Roles For Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
|
|
120
|
-
* [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
|
|
121
|
-
* [Aws::AssumeRoleWebIdentityCredentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/AssumeRoleWebIdentityCredentials.html)
|
|
122
|
-
|
|
123
|
-
**role_arn (required)**
|
|
124
|
-
|
|
125
|
-
The Amazon Resource Name (ARN) of the role to assume.
|
|
126
|
-
|
|
127
|
-
**role_session_name (required)**
|
|
128
|
-
|
|
129
|
-
An identifier for the assumed role session.
|
|
130
|
-
|
|
131
|
-
**web_identity_token_file (required)**
|
|
132
|
-
|
|
133
|
-
The absolute path to the file on disk containing the OIDC token
|
|
134
|
-
|
|
135
|
-
**policy**
|
|
136
|
-
|
|
137
|
-
An IAM policy in JSON format.
|
|
138
|
-
|
|
139
|
-
**duration_seconds**
|
|
140
|
-
|
|
141
|
-
The duration, in seconds, of the role session. The value can range from
|
|
142
|
-
900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value
|
|
143
|
-
is set to 3600 seconds.
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
### instance_profile_credentials
|
|
147
|
-
|
|
148
|
-
Retrieve temporary security credentials via HTTP request. This is useful on
|
|
149
|
-
EC2 instance.
|
|
150
|
-
|
|
151
|
-
<match *>
|
|
152
|
-
@type s3
|
|
153
|
-
|
|
154
|
-
<instance_profile_credentials>
|
|
155
|
-
ip_address IP_ADDRESS
|
|
156
|
-
port PORT
|
|
157
|
-
</instance_profile_credentials>
|
|
158
|
-
</match>
|
|
159
|
-
|
|
160
|
-
See also:
|
|
161
|
-
|
|
162
|
-
* [Aws::InstanceProfileCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html)
|
|
163
|
-
* [Temporary Security Credentials - AWS Identity and Access
|
|
164
|
-
Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
|
|
165
|
-
* [Instance Metadata and User Data - Amazon Elastic Compute
|
|
166
|
-
Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
|
|
167
|
-
|
|
168
|
-
**retries**
|
|
169
|
-
|
|
170
|
-
Number of times to retry when retrieving credentials. Default is 5.
|
|
171
|
-
|
|
172
|
-
**ip_address**
|
|
173
|
-
|
|
174
|
-
Default is 169.254.169.254.
|
|
175
|
-
|
|
176
|
-
**port**
|
|
177
|
-
|
|
178
|
-
Default is 80.
|
|
179
|
-
|
|
180
|
-
**http_open_timeout**
|
|
181
|
-
|
|
182
|
-
Default is 5.
|
|
183
|
-
|
|
184
|
-
**http_read_timeout**
|
|
185
|
-
|
|
186
|
-
Default is 5.
|
|
187
|
-
|
|
188
|
-
### shared_credentials
|
|
189
|
-
|
|
190
|
-
This loads AWS access credentials from local ini file. This is useful for
|
|
191
|
-
local developing.
|
|
192
|
-
|
|
193
|
-
<match *>
|
|
194
|
-
@type s3
|
|
195
|
-
|
|
196
|
-
<shared_credentials>
|
|
197
|
-
path PATH
|
|
198
|
-
profile_name PROFILE_NAME
|
|
199
|
-
</shared_credentials>
|
|
200
|
-
</match>
|
|
201
|
-
|
|
202
|
-
See also:
|
|
203
|
-
|
|
204
|
-
* [Aws::SharedCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html)
|
|
205
|
-
|
|
206
|
-
**path**
|
|
207
|
-
|
|
208
|
-
Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials".
|
|
209
|
-
|
|
210
|
-
**profile_name**
|
|
211
|
-
|
|
212
|
-
Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
|
|
213
|
-
|
|
214
|
-
## Configuration: Output
|
|
215
|
-
|
|
216
|
-
### v1.0 style
|
|
217
|
-
|
|
218
|
-
With fluentd v1 and fluent-plugin-s3 v1.0.0 or later, use new buffer configuration to dynamic parameters.
|
|
219
|
-
|
|
220
|
-
<match pattern>
|
|
221
|
-
@type s3
|
|
222
|
-
|
|
223
|
-
aws_key_id YOUR_AWS_KEY_ID
|
|
224
|
-
aws_sec_key YOUR_AWS_SECRET_KEY
|
|
225
|
-
s3_bucket YOUR_S3_BUCKET_NAME
|
|
226
|
-
s3_region ap-northeast-1
|
|
227
|
-
|
|
228
|
-
path logs/${tag}/%Y/%m/%d/
|
|
229
|
-
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
|
|
230
|
-
|
|
231
|
-
# if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
|
|
232
|
-
# need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
|
|
233
|
-
<buffer tag,time>
|
|
234
|
-
@type file
|
|
235
|
-
path /var/log/fluent/s3
|
|
236
|
-
timekey 3600 # 1 hour partition
|
|
237
|
-
timekey_wait 10m
|
|
238
|
-
timekey_use_utc true # use utc
|
|
239
|
-
</buffer>
|
|
240
|
-
<format>
|
|
241
|
-
@type json
|
|
242
|
-
</format>
|
|
243
|
-
</match>
|
|
244
|
-
|
|
245
|
-
For `<buffer>`, you can use any record field in `path` / `s3_object_key_format`.
|
|
246
|
-
|
|
247
|
-
path logs/${tag}/${foo}
|
|
248
|
-
<buffer tag,foo>
|
|
249
|
-
# parameters...
|
|
250
|
-
</buffer>
|
|
251
|
-
|
|
252
|
-
See official article for more detail: [Config: Buffer Section - Fluentd](https://docs.fluentd.org/configuration/buffer-section)
|
|
253
|
-
|
|
254
|
-
Note that this configuration doesn't work with fluentd v0.12.
|
|
255
|
-
|
|
256
|
-
### v0.12 style
|
|
257
|
-
|
|
258
|
-
This configuration works with both fluentd v0.12 and v1.0.
|
|
259
|
-
|
|
260
|
-
<match pattern>
|
|
261
|
-
@type s3
|
|
262
|
-
|
|
263
|
-
aws_key_id YOUR_AWS_KEY_ID
|
|
264
|
-
aws_sec_key YOUR_AWS_SECRET_KEY
|
|
265
|
-
s3_bucket YOUR_S3_BUCKET_NAME
|
|
266
|
-
s3_region ap-northeast-1
|
|
267
|
-
|
|
268
|
-
path logs/
|
|
269
|
-
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
|
|
270
|
-
buffer_path /var/log/fluent/s3
|
|
271
|
-
time_slice_format %Y%m%d-%H
|
|
272
|
-
time_slice_wait 10m
|
|
273
|
-
utc
|
|
274
|
-
format json
|
|
275
|
-
</match>
|
|
276
|
-
|
|
277
|
-
If you want to embed tag in `path` / `s3_object_key_format`, you need to use `fluent-plugin-forest` plugin.
|
|
278
|
-
|
|
279
|
-
**aws_iam_retries**
|
|
280
|
-
|
|
281
|
-
This parameter is deprecated. Use `instance_profile_credentials` instead.
|
|
282
|
-
|
|
283
|
-
The number of attempts to make (with exponential backoff) when loading
|
|
284
|
-
instance profile credentials from the EC2 metadata service using an IAM
|
|
285
|
-
role. Defaults to 5 retries.
|
|
286
|
-
|
|
287
|
-
**s3_bucket (required)**
|
|
288
|
-
|
|
289
|
-
S3 bucket name.
|
|
290
|
-
|
|
291
|
-
**s3_region**
|
|
292
|
-
|
|
293
|
-
s3 region name. For example, US West (Oregon) Region is "us-west-2". The
|
|
294
|
-
full list of regions are available here. >
|
|
295
|
-
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
|
|
296
|
-
recommend using `s3_region` instead of `s3_endpoint`.
|
|
297
|
-
|
|
298
|
-
**s3_endpoint**
|
|
299
|
-
|
|
300
|
-
endpoint for S3 compatible services. For example, Riak CS based storage or
|
|
301
|
-
something. This option is deprecated for AWS S3, use `s3_region` instead.
|
|
302
|
-
|
|
303
|
-
See also AWS article: [Working with Regions](https://aws.amazon.com/blogs/developer/working-with-regions/).
|
|
304
|
-
|
|
305
|
-
**enable_transfer_acceleration**
|
|
306
|
-
|
|
307
|
-
Enable [S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) for uploads. **IMPORTANT**: For this to work, you must first enable this feature on your destination S3 bucket.
|
|
308
|
-
|
|
309
|
-
**enable_dual_stack**
|
|
310
|
-
|
|
311
|
-
Enable [Amazon S3 Dual-Stack Endpoints](https://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html) for uploads. Will make it possible to use either IPv4 or IPv6 when connecting to S3.
|
|
312
|
-
|
|
313
|
-
**use_bundled_cert**
|
|
314
|
-
|
|
315
|
-
For cases where the default SSL certificate is unavailable (e.g. Windows), you can set this option to true in order to use the AWS SDK bundled certificate. Default is false.
|
|
316
|
-
|
|
317
|
-
This fixes the following error often seen in Windows:
|
|
318
|
-
|
|
319
|
-
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Seahorse::Client::NetworkingError)
|
|
320
|
-
|
|
321
|
-
**ssl_verify_peer**
|
|
322
|
-
|
|
323
|
-
Verify SSL certificate of the endpoint. Default is true. Set false when you want to ignore the endpoint SSL certificate.
|
|
324
|
-
|
|
325
|
-
**s3_object_key_format**
|
|
326
|
-
|
|
327
|
-
The format of S3 object keys. You can use several built-in variables:
|
|
328
|
-
|
|
329
|
-
* %{path}
|
|
330
|
-
* %{time_slice}
|
|
331
|
-
* %{index}
|
|
332
|
-
* %{file_extension}
|
|
333
|
-
* %{hex_random}
|
|
334
|
-
* %{uuid_flush}
|
|
335
|
-
* %{hostname}
|
|
336
|
-
|
|
337
|
-
to decide keys dynamically.
|
|
338
|
-
|
|
339
|
-
* %{path} is exactly the value of **path** configured in the configuration file.
|
|
340
|
-
E.g., "logs/" in the example configuration above.
|
|
341
|
-
* %{time_slice} is the
|
|
342
|
-
time-slice in text that are formatted with **time_slice_format**.
|
|
343
|
-
* %{index} is the sequential number starts from 0, increments when multiple files are uploaded to S3 in the same time slice.
|
|
344
|
-
* %{file_extension} depends on **store_as** parameter.
|
|
345
|
-
* %{uuid_flush} a uuid that is replaced everytime the buffer will be flushed.
|
|
346
|
-
* %{hostname} is replaced with `Socket.gethostname` result.
|
|
347
|
-
* %{hex_random} a random hex string that is replaced for each buffer chunk, not
|
|
348
|
-
assured to be unique. This is used to follow a way of performance tuning, `Add
|
|
349
|
-
a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance
|
|
350
|
-
Considerations - Amazon Simple Storage
|
|
351
|
-
Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
|
|
352
|
-
You can configure the length of string with a
|
|
353
|
-
`hex_random_length` parameter (Default: 4).
|
|
354
|
-
|
|
355
|
-
The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
|
|
356
|
-
In addition, you can use [buffer placeholders](https://docs.fluentd.org/configuration/buffer-section#placeholders) in this parameter,
|
|
357
|
-
so you can embed tag, time and record value like below:
|
|
358
|
-
|
|
359
|
-
s3_object_key_format %{path}/events/%Y%m%d/${tag}_%{index}.%{file_extension}
|
|
360
|
-
<buffer tag,time>
|
|
361
|
-
# buffer parameters...
|
|
362
|
-
</buffer>
|
|
363
|
-
|
|
364
|
-
For instance, using the example configuration above, actual object keys on S3
|
|
365
|
-
will be something like:
|
|
366
|
-
|
|
367
|
-
"logs/20130111-22_0.gz"
|
|
368
|
-
"logs/20130111-23_0.gz"
|
|
369
|
-
"logs/20130111-23_1.gz"
|
|
370
|
-
"logs/20130112-00_0.gz"
|
|
371
|
-
|
|
372
|
-
With the configuration:
|
|
373
|
-
|
|
374
|
-
s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
|
|
375
|
-
path log
|
|
376
|
-
time_slice_format %Y%m%d-%H
|
|
377
|
-
|
|
378
|
-
You get:
|
|
379
|
-
|
|
380
|
-
"log/events/ts=20130111-22/events_0.gz"
|
|
381
|
-
"log/events/ts=20130111-23/events_0.gz"
|
|
382
|
-
"log/events/ts=20130111-23/events_1.gz"
|
|
383
|
-
"log/events/ts=20130112-00/events_0.gz"
|
|
384
|
-
|
|
385
|
-
NOTE: ${hostname} placeholder is deprecated since v0.8. You can get same result by using [configuration's embedded ruby code feature](https://docs.fluentd.org/configuration/config-file#embedded-ruby-code).
|
|
386
|
-
|
|
387
|
-
s3_object_key_format %{path}%{time_slice}_%{hostname}%{index}.%{file_extension}
|
|
388
|
-
s3_object_key_format "%{path}%{time_slice}_#{Socket.gethostname}%{index}.%{file_extension}"
|
|
389
|
-
|
|
390
|
-
Above two configurations are same. The important point is wrapping `""` is needed for `#{Socket.gethostname}`.
|
|
391
|
-
|
|
392
|
-
**force_path_style**
|
|
393
|
-
|
|
394
|
-
:force_path_style (Boolean) — default: false — When set to true, the
|
|
395
|
-
bucket name is always left in the request URI and never moved to the host
|
|
396
|
-
as a sub-domain. See Plugins::S3BucketDns for more details.
|
|
397
|
-
|
|
398
|
-
This parameter is deprecated. See AWS announcement: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/
|
|
399
|
-
|
|
400
|
-
**store_as**
|
|
401
|
-
|
|
402
|
-
archive format on S3. You can use several format:
|
|
403
|
-
|
|
404
|
-
* gzip (default)
|
|
405
|
-
* json
|
|
406
|
-
* text
|
|
407
|
-
* lzo (Need lzop command)
|
|
408
|
-
* lzma2 (Need xz command)
|
|
409
|
-
* gzip_command (Need gzip command)
|
|
410
|
-
* This compressor uses an external gzip command, hence would result in
|
|
411
|
-
utilizing CPU cores well compared with `gzip`
|
|
412
|
-
|
|
413
|
-
See `Use your compression algorithm` section for adding another format.
|
|
414
|
-
|
|
415
|
-
**`<format>` or format**
|
|
416
|
-
|
|
417
|
-
Change one line format in the S3 object. Supported formats are "out_file",
|
|
418
|
-
"json", "ltsv", "single_value" and other formatter plugins. See also [official Formatter article](https://docs.fluentd.org/formatter).
|
|
419
|
-
|
|
420
|
-
* out_file (default).
|
|
421
|
-
|
|
422
|
-
time\ttag\t{..json1..}
|
|
423
|
-
time\ttag\t{..json2..}
|
|
424
|
-
...
|
|
425
|
-
|
|
426
|
-
* json
|
|
427
|
-
|
|
428
|
-
{..json1..}
|
|
429
|
-
{..json2..}
|
|
430
|
-
...
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
At this format, "time" and "tag" are omitted. But you can set these
|
|
434
|
-
information to the record by setting `<inject>` option. If you set following configuration in
|
|
435
|
-
S3 output:
|
|
436
|
-
|
|
437
|
-
# v1
|
|
438
|
-
<format>
|
|
439
|
-
@type json
|
|
440
|
-
</format>
|
|
441
|
-
<inject>
|
|
442
|
-
time_key log_time
|
|
443
|
-
</inject>
|
|
444
|
-
# v0.12
|
|
445
|
-
@format json
|
|
446
|
-
include_time_key true
|
|
447
|
-
time_key log_time # default is time
|
|
448
|
-
|
|
449
|
-
then the record has log_time field.
|
|
450
|
-
|
|
451
|
-
{"log_time":"time string",...}
|
|
452
|
-
|
|
453
|
-
See also [official Inject Section article](https://docs.fluentd.org/configuration/inject-section).
|
|
454
|
-
|
|
455
|
-
* ltsv
|
|
456
|
-
|
|
457
|
-
key1:value1\tkey2:value2
|
|
458
|
-
key1:value1\tkey2:value2
|
|
459
|
-
...
|
|
460
|
-
|
|
461
|
-
* single_value
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
Use specified value instead of entire recode. If you get '{"message":"my
|
|
465
|
-
log"}', then contents are
|
|
466
|
-
|
|
467
|
-
my log1
|
|
468
|
-
my log2
|
|
469
|
-
...
|
|
470
|
-
|
|
471
|
-
You can change key name by "message_key" option.
|
|
472
|
-
|
|
473
|
-
**auto_create_bucket**
|
|
474
|
-
|
|
475
|
-
Create S3 bucket if it does not exists. Default is true.
|
|
476
|
-
|
|
477
|
-
**check_bucket**
|
|
478
|
-
|
|
479
|
-
Check mentioned bucket if it exists in AWS or not. Default is true.
|
|
480
|
-
|
|
481
|
-
When it is false, fluentd will not check aws s3 for the existence of the mentioned bucket.
|
|
482
|
-
This is the case where bucket will be pre-created before running fluentd.
|
|
483
|
-
|
|
484
|
-
**check_object**
|
|
485
|
-
|
|
486
|
-
Check object before creation if it exists or not. Default is true.
|
|
487
|
-
|
|
488
|
-
When it is false, s3_object_key_format will be %{path}%{time_slice}_%{hms_slice}.%{file_extension} by default where,
|
|
489
|
-
hms_slice will be time-slice in hhmmss format, so that each object will be unique.
|
|
490
|
-
Example object name, assuming it is created on 2016/16/11 3:30:54 PM 20161611_153054.txt (extension can be anything as per user's choice)
|
|
491
|
-
|
|
492
|
-
**check_apikey_on_start**
|
|
493
|
-
|
|
494
|
-
Check AWS key on start. Default is true.
|
|
495
|
-
|
|
496
|
-
**proxy_uri**
|
|
497
|
-
|
|
498
|
-
uri of proxy environment.
|
|
499
|
-
|
|
500
|
-
**path**
|
|
501
|
-
|
|
502
|
-
path prefix of the files on S3. Default is "" (no prefix).
|
|
503
|
-
[buffer placeholder](https://docs.fluentd.org/configuration/buffer-section#placeholders) is supported,
|
|
504
|
-
so you can embed tag, time and record value like below.
|
|
505
|
-
|
|
506
|
-
path logs/%Y%m%d/${tag}/
|
|
507
|
-
<buffer tag,time>
|
|
508
|
-
# buffer parameters...
|
|
509
|
-
</buffer>
|
|
510
|
-
|
|
511
|
-
**buffer_path (for v0.12)**
|
|
512
|
-
|
|
513
|
-
path prefix of the files to buffer logs.
|
|
514
|
-
|
|
515
|
-
This parameter is for v0.12. Use `<buffer>`'s `path` in v1.
|
|
516
|
-
|
|
517
|
-
**time_slice_format(for v0.12)**
|
|
518
|
-
|
|
519
|
-
Format of the time used as the file name. Default is '%Y%m%d'. Use
|
|
520
|
-
'%Y%m%d%H' to split files hourly.
|
|
521
|
-
|
|
522
|
-
This parameter is for v0.12. Use buffer placeholder for `path` / `s3_object_key_format` in v1.
|
|
523
|
-
|
|
524
|
-
**time_slice_wait(for v0.12)**
|
|
525
|
-
|
|
526
|
-
The time to wait old logs. Default is 10 minutes. Specify larger value if
|
|
527
|
-
old logs may reach.
|
|
528
|
-
|
|
529
|
-
This parameter is for v0.12. Use `<buffer>`'s `timekey_wait` in v1.
|
|
530
|
-
|
|
531
|
-
**utc**
|
|
532
|
-
|
|
533
|
-
Use UTC instead of local time.
|
|
534
|
-
|
|
535
|
-
**storage_class**
|
|
536
|
-
|
|
537
|
-
Set storage class. Possible values are `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA` from [Ruby SDK](http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#storage_class-instance_method).
|
|
538
|
-
|
|
539
|
-
**reduced_redundancy**
|
|
540
|
-
|
|
541
|
-
Use S3 reduced redundancy storage for 33% cheaper pricing. Default is
|
|
542
|
-
false.
|
|
543
|
-
|
|
544
|
-
This is deprecated. Use `storage_class REDUCED_REDUNDANCY` instead.
|
|
545
|
-
|
|
546
|
-
**acl**
|
|
547
|
-
|
|
548
|
-
Permission for the object in S3. This is useful for cross-account access
|
|
549
|
-
using IAM roles. Valid values are:
|
|
550
|
-
|
|
551
|
-
* private (default)
|
|
552
|
-
* public-read
|
|
553
|
-
* public-read-write (not recommended - see [Canned
|
|
554
|
-
ACL](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl))
|
|
555
|
-
* authenticated-read
|
|
556
|
-
* bucket-owner-read
|
|
557
|
-
* bucket-owner-full-control
|
|
558
|
-
|
|
559
|
-
To use cross-account access, you will need to create a bucket policy granting
|
|
560
|
-
the specific access required. Refer to the [AWS
|
|
561
|
-
documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example3.html) for examples.
|
|
562
|
-
|
|
563
|
-
**grant_full_control**
|
|
564
|
-
|
|
565
|
-
Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object.
|
|
566
|
-
This is useful for cross-account access using IAM roles.
|
|
567
|
-
|
|
568
|
-
Valid values are `id="Grantee-CanonicalUserID"`. Please specify the grantee's canonical user ID.
|
|
569
|
-
|
|
570
|
-
e.g. `id="79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"`
|
|
571
|
-
|
|
572
|
-
Note that a canonical user ID is different from an AWS account ID.
|
|
573
|
-
Please refer to [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html) for more details.
|
|
574
|
-
|
|
575
|
-
**grant_read**
|
|
576
|
-
|
|
577
|
-
Allows grantee to read the object data and its metadata.
|
|
578
|
-
Valid values are `id="Grantee-CanonicalUserID"`. Please specify the grantee's canonical user ID.
|
|
579
|
-
|
|
580
|
-
e.g. `id="79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"`
|
|
581
|
-
|
|
582
|
-
**grant_read_acp**
|
|
583
|
-
|
|
584
|
-
Allows grantee to read the object ACL.
|
|
585
|
-
Valid values are `id="Grantee-CanonicalUserID"`. Please specify the grantee's canonical user ID.
|
|
586
|
-
|
|
587
|
-
e.g. `id="79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"`
|
|
588
|
-
|
|
589
|
-
**grant_write_acp**
|
|
590
|
-
|
|
591
|
-
Allows grantee to write the ACL for the applicable object.
|
|
592
|
-
Valid values are `id="Grantee-CanonicalUserID"`. Please specify the grantee's canonical user ID.
|
|
593
|
-
|
|
594
|
-
e.g. `id="79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"`
|
|
595
|
-
|
|
596
|
-
**hex_random_length**
|
|
597
|
-
|
|
598
|
-
The length of `%{hex_random}` placeholder. Default is 4 as written in
|
|
599
|
-
[Request Rate and Performance Considerations - Amazon Simple Storage
|
|
600
|
-
Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
|
|
601
|
-
The maximum length is 16.
|
|
602
|
-
|
|
603
|
-
**index_format**
|
|
604
|
-
|
|
605
|
-
`%{index}` is formatted by [sprintf](http://ruby-doc.org/core-2.2.0/Kernel.html#method-i-sprintf) using this format_string. Default is '%d'. Zero padding is supported e.g. `%04d` to ensure minimum length four digits. `%{index}` can be in lowercase or uppercase hex using '%x' or '%X'
|
|
606
|
-
|
|
607
|
-
**overwrite**
|
|
608
|
-
|
|
609
|
-
Overwrite already existing path. Default is false, which raises an error
|
|
610
|
-
if a s3 object of the same path already exists, or increment the
|
|
611
|
-
`%{index}` placeholder until finding an absent path.
|
|
612
|
-
|
|
613
|
-
**use_server_side_encryption**
|
|
614
|
-
|
|
615
|
-
The Server-side encryption algorithm used when storing this object in S3
|
|
616
|
-
(e.g., AES256, aws:kms)
|
|
617
|
-
|
|
618
|
-
**ssekms_key_id**
|
|
619
|
-
|
|
620
|
-
Specifies the AWS KMS key ID to use for object encryption. You have to
|
|
621
|
-
set "aws:kms" to `use_server_side_encryption` to use the KMS encryption.
|
|
622
|
-
|
|
623
|
-
**sse_customer_algorithm**
|
|
624
|
-
|
|
625
|
-
Specifies the algorithm to use to when encrypting the object (e.g., AES256).
|
|
626
|
-
|
|
627
|
-
**sse_customer_key**
|
|
628
|
-
|
|
629
|
-
Specifies the AWS KMS key ID to use for object encryption.
|
|
630
|
-
|
|
631
|
-
**sse_customer_key_md5**
|
|
632
|
-
|
|
633
|
-
Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
|
|
634
|
-
|
|
635
|
-
**compute_checksums**
|
|
636
|
-
|
|
637
|
-
AWS SDK uses MD5 for API request/response by default. On FIPS enabled environment,
|
|
638
|
-
OpenSSL returns an error because MD5 is disabled. If you want to use
|
|
639
|
-
this plugin on FIPS enabled environment, set `compute_checksums false`.
|
|
640
|
-
|
|
641
|
-
**signature_version**
|
|
642
|
-
|
|
643
|
-
Signature version for API request. `s3` means signature version 2 and
|
|
644
|
-
`v4` means signature version 4. Default is `nil` (Following SDK's default).
|
|
645
|
-
It would be useful when you use S3 compatible storage that accepts only signature version 2.
|
|
646
|
-
|
|
647
|
-
**warn_for_delay**
|
|
648
|
-
|
|
649
|
-
Given a threshold to treat events as delay, output warning logs if delayed events were put into s3.
|
|
650
|
-
|
|
651
|
-
**bucket_lifecycle_rule**
|
|
652
|
-
|
|
653
|
-
Specify one or more lifecycle rules for the bucket
|
|
654
|
-
|
|
655
|
-
<bucket_lifecycle_rule>
|
|
656
|
-
id UNIQUE_ID_FOR_THE_RULE
|
|
657
|
-
prefix OPTIONAL_PREFIX # Objects whose keys begin with this prefix will be affected by the rule. If not specified all objects of the bucket will be affected
|
|
658
|
-
expiration_days NUMBER_OF_DAYS # The number of days before the object will expire
|
|
659
|
-
</bucket_lifecycle_rule>
|
|
660
|
-
|
|
661
|
-
## Input: Setup
|
|
662
|
-
|
|
663
|
-
1. Create new [SQS](https://aws.amazon.com/documentation/sqs/) queue (use same region as S3)
|
|
664
|
-
2. Set proper permission to new queue
|
|
665
|
-
3. [Configure S3 event notification](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html)
|
|
666
|
-
4. Write configuration file such as fluent.conf
|
|
667
|
-
5. Run fluentd
|
|
668
|
-
|
|
669
|
-
## Configuration: Input
|
|
670
|
-
|
|
671
|
-
<source>
|
|
672
|
-
@type s3
|
|
673
|
-
|
|
674
|
-
aws_key_id YOUR_AWS_KEY_ID
|
|
675
|
-
aws_sec_key YOUR_AWS_SECRET_KEY
|
|
676
|
-
s3_bucket YOUR_S3_BUCKET_NAME
|
|
677
|
-
s3_region ap-northeast-1
|
|
678
|
-
add_object_metadata true
|
|
679
|
-
|
|
680
|
-
<sqs>
|
|
681
|
-
queue_name YOUR_SQS_QUEUE_NAME
|
|
682
|
-
</sqs>
|
|
683
|
-
</source>
|
|
684
|
-
|
|
685
|
-
**add_object_metadata**
|
|
686
|
-
|
|
687
|
-
Whether or not object metadata should be added to the record. Defaults to `false`. See below for details.
|
|
688
|
-
|
|
689
|
-
**s3_bucket (required)**
|
|
690
|
-
|
|
691
|
-
S3 bucket name.
|
|
692
|
-
|
|
693
|
-
**s3_region**
|
|
694
|
-
|
|
695
|
-
S3 region name. For example, US West (Oregon) Region is
|
|
696
|
-
"us-west-2". The full list of regions are available here. >
|
|
697
|
-
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
|
|
698
|
-
recommend using `s3_region` instead of `s3_endpoint`.
|
|
699
|
-
|
|
700
|
-
**store_as**
|
|
701
|
-
|
|
702
|
-
archive format on S3. You can use serveral format:
|
|
703
|
-
|
|
704
|
-
* gzip (default)
|
|
705
|
-
* json
|
|
706
|
-
* text
|
|
707
|
-
* lzo (Need lzop command)
|
|
708
|
-
* lzma2 (Need xz command)
|
|
709
|
-
* gzip_command (Need gzip command)
|
|
710
|
-
* This compressor uses an external gzip command, hence would result in utilizing CPU cores well compared with `gzip`
|
|
711
|
-
|
|
712
|
-
See 'Use your compression algorithm' section for adding another format.
|
|
713
|
-
|
|
714
|
-
**format**
|
|
715
|
-
|
|
716
|
-
Parse a line as this format in the S3 object. Supported formats are
|
|
717
|
-
"apache_error", "apache2", "syslog", "json", "tsv", "ltsv", "csv",
|
|
718
|
-
"nginx" and "none".
|
|
719
|
-
|
|
720
|
-
**check_apikey_on_start**
|
|
721
|
-
|
|
722
|
-
Check AWS key on start. Default is true.
|
|
723
|
-
|
|
724
|
-
**proxy_uri**
|
|
725
|
-
|
|
726
|
-
URI of proxy environment.
|
|
727
|
-
|
|
728
|
-
**sqs/queue_name (required)**
|
|
729
|
-
|
|
730
|
-
SQS queue name. Need to create SQS queue on the region same as S3 bucket.
|
|
731
|
-
|
|
732
|
-
**sqs/skip_delete**
|
|
733
|
-
|
|
734
|
-
When true, messages are not deleted after polling block. Default is false.
|
|
735
|
-
|
|
736
|
-
**sqs/wait_time_seconds**
|
|
737
|
-
|
|
738
|
-
The long polling interval. Default is 20.
|
|
739
|
-
|
|
740
|
-
**sqs/retry_error_interval**
|
|
741
|
-
|
|
742
|
-
Interval to retry polling SQS if polling unsuccessful, in seconds. Default is 300.
|
|
743
|
-
|
|
744
|
-
## Object Metadata Added To Records
|
|
745
|
-
|
|
746
|
-
If the `add_object_metadata` option is set to true, then the name of the bucket
|
|
747
|
-
and the key for a given object will be added to each log record as `s3_bucket`
|
|
748
|
-
and `s3_key`, respectively. This metadata can be used by filter plugins or other
|
|
749
|
-
downstream processors to better identify the source of a given record.
|
|
750
|
-
|
|
751
|
-
## IAM Policy
|
|
752
|
-
|
|
753
|
-
The following is an example for a IAM policy needed to write to an s3 bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
|
|
754
|
-
|
|
755
|
-
{
|
|
756
|
-
"Version": "2012-10-17",
|
|
757
|
-
"Statement": [
|
|
758
|
-
{
|
|
759
|
-
"Effect": "Allow",
|
|
760
|
-
"Action": [
|
|
761
|
-
"s3:ListBucket"
|
|
762
|
-
],
|
|
763
|
-
"Resource": "arn:aws:s3:::my-s3bucket"
|
|
764
|
-
},
|
|
765
|
-
{
|
|
766
|
-
"Effect": "Allow",
|
|
767
|
-
"Action": [
|
|
768
|
-
"s3:PutObject",
|
|
769
|
-
"s3:GetObject"
|
|
770
|
-
],
|
|
771
|
-
"Resource": "arn:aws:s3:::my-s3bucket/*"
|
|
772
|
-
}
|
|
773
|
-
]
|
|
774
|
-
}
|
|
775
|
-
|
|
776
|
-
Note that the bucket must already exist and **auto_create_bucket** has no effect in this case.
|
|
777
|
-
|
|
778
|
-
`s3:GetObject` is needed for object check to avoid object overwritten.
|
|
779
|
-
If you set `check_object false`, `s3:GetObject` is not needed.
|
|
780
|
-
|
|
781
|
-
Refer to the [AWS
|
|
782
|
-
documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolicies.html) for example policies.
|
|
783
|
-
|
|
784
|
-
Using [IAM
|
|
785
|
-
roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
|
|
786
|
-
with a properly configured IAM policy are preferred over embedding access keys
|
|
787
|
-
on EC2 instances.
|
|
788
|
-
|
|
789
|
-
### Example when `check_bucket false` and `check_object false`
|
|
790
|
-
|
|
791
|
-
When the mentioned configuration will be made, fluentd will work with the
|
|
792
|
-
minimum IAM poilcy, like:
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
"Statement": [{
|
|
796
|
-
"Effect": "Allow",
|
|
797
|
-
"Action": "s3:PutObject",
|
|
798
|
-
"Resource": ["*"]
|
|
799
|
-
}]
|
|
48
|
+
See [Configuration: credentials](docs/credentials.md) about details.
|
|
800
49
|
|
|
50
|
+
## Output Plugin
|
|
801
51
|
|
|
802
|
-
|
|
52
|
+
See [Configuration: Output](docs/output.md) about details.
|
|
803
53
|
|
|
804
|
-
|
|
805
|
-
plugin. If you set 'store_as xxx', `out_s3` plugin searches
|
|
806
|
-
`fluent/plugin/s3_compressor_xxx.rb` and `in_s3` plugin searches
|
|
807
|
-
`fluent/plugin/s3_extractor_xxx.rb`. You can define your (de)compression with
|
|
808
|
-
'S3Output::Compressor'/`S3Input::Extractor` classes. Compressor API is here:
|
|
54
|
+
## Input Plugin
|
|
809
55
|
|
|
810
|
-
|
|
811
|
-
class S3Output
|
|
812
|
-
class XXXCompressor < Compressor
|
|
813
|
-
S3Output.register_compressor('xxx', self)
|
|
56
|
+
See [Configuration: Input](docs/input.md) about details.
|
|
814
57
|
|
|
815
|
-
|
|
816
|
-
def ext
|
|
817
|
-
'xxx'
|
|
818
|
-
end
|
|
58
|
+
## Tips and How to
|
|
819
59
|
|
|
820
|
-
|
|
821
|
-
|
|
822
|
-
|
|
823
|
-
end
|
|
60
|
+
* [Object Metadata Added To Records](docs/howto.md#object-metadata-added-to-records)
|
|
61
|
+
* [IAM Policy](docs/howto.md#iam-policy)
|
|
62
|
+
* [Use your (de)compression algorithm](docs/howto.md#use-your-decompression-algorithm)
|
|
824
63
|
|
|
825
|
-
|
|
826
|
-
def compress(chunk, tmp)
|
|
827
|
-
# call command or something
|
|
828
|
-
end
|
|
829
|
-
end
|
|
830
|
-
end
|
|
831
|
-
end
|
|
64
|
+
## Migration guide
|
|
832
65
|
|
|
833
|
-
|
|
834
|
-
See bundled `Compressor`/`Extractor` classes for more detail.
|
|
66
|
+
See [Migration guide from v0.12](docs/v0.12.md) about details.
|
|
835
67
|
|
|
836
68
|
## Website, license, et. al.
|
|
837
69
|
|