fluent-plugin-s3-hubspot 1.0.0.rc5
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +13 -0
- data/.travis.yml +25 -0
- data/AUTHORS +2 -0
- data/ChangeLog +280 -0
- data/Gemfile +5 -0
- data/Gemfile.v0.12 +6 -0
- data/README.md +684 -0
- data/Rakefile +14 -0
- data/VERSION +1 -0
- data/appveyor.yml +25 -0
- data/fluent-plugin-s3.gemspec +26 -0
- data/lib/fluent/log-ext.rb +12 -0
- data/lib/fluent/plugin/in_s3.rb +329 -0
- data/lib/fluent/plugin/out_s3.rb +512 -0
- data/lib/fluent/plugin/s3_compressor_gzip_command.rb +52 -0
- data/lib/fluent/plugin/s3_compressor_lzma2.rb +35 -0
- data/lib/fluent/plugin/s3_compressor_lzo.rb +35 -0
- data/lib/fluent/plugin/s3_extractor_gzip_command.rb +46 -0
- data/lib/fluent/plugin/s3_extractor_lzma2.rb +40 -0
- data/lib/fluent/plugin/s3_extractor_lzo.rb +40 -0
- data/test/test_in_s3.rb +262 -0
- data/test/test_out_s3.rb +640 -0
- metadata +178 -0
data/.gitignore
ADDED
data/.travis.yml
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
language: ruby
|
2
|
+
|
3
|
+
rvm:
|
4
|
+
- 2.1.10
|
5
|
+
- 2.2.5
|
6
|
+
- 2.3.1
|
7
|
+
- 2.4.0
|
8
|
+
- ruby-head
|
9
|
+
|
10
|
+
gemfile:
|
11
|
+
- Gemfile
|
12
|
+
|
13
|
+
branches:
|
14
|
+
only:
|
15
|
+
- master
|
16
|
+
- v0.12
|
17
|
+
|
18
|
+
before_install: gem update bundler
|
19
|
+
script: bundle exec rake test
|
20
|
+
|
21
|
+
sudo: false
|
22
|
+
|
23
|
+
matrix:
|
24
|
+
allow_failures:
|
25
|
+
- rvm: ruby-head
|
data/AUTHORS
ADDED
data/ChangeLog
ADDED
@@ -0,0 +1,280 @@
|
|
1
|
+
Release 0.8.0 - 2016/12/20
|
2
|
+
|
3
|
+
* out_s3: Add check_object / check_bucket parameters for only put permission
|
4
|
+
* Remove fluent-mixin-config-placeholders dependency
|
5
|
+
|
6
|
+
|
7
|
+
Release 0.7.2 - 2016/10/20
|
8
|
+
|
9
|
+
* in_s3: Replace old parser API with new one
|
10
|
+
* in_s3: Don't stop SQS polling when error happens
|
11
|
+
|
12
|
+
|
13
|
+
Release 0.7.1 - 2016/09/02
|
14
|
+
|
15
|
+
* Support IAM role for Amazon ECS task
|
16
|
+
|
17
|
+
|
18
|
+
Release 0.7.0 - 2016/08/10
|
19
|
+
|
20
|
+
* Add s3 input plugin
|
21
|
+
|
22
|
+
|
23
|
+
Release 0.6.9 - 2016/07/28
|
24
|
+
|
25
|
+
* Support v0.14
|
26
|
+
* Fix aws_iam_retries warning
|
27
|
+
* Fix race condition at start with buffers
|
28
|
+
|
29
|
+
|
30
|
+
Release 0.6.8 - 2016/04/19
|
31
|
+
|
32
|
+
* Add sse customer key options
|
33
|
+
|
34
|
+
|
35
|
+
Release 0.6.7 - 2016/03/31
|
36
|
+
|
37
|
+
* Add signature_version parameter
|
38
|
+
* Add warn_for_delay parameter
|
39
|
+
|
40
|
+
|
41
|
+
Release 0.6.6 - 2016/03/16
|
42
|
+
|
43
|
+
* Fix ACL handling in PUT operation
|
44
|
+
|
45
|
+
|
46
|
+
Release 0.6.5 - 2016/01/13
|
47
|
+
|
48
|
+
* Add description to parameters
|
49
|
+
* Use path as prefix in API key check
|
50
|
+
|
51
|
+
|
52
|
+
Release 0.6.4 - 2015/12/03
|
53
|
+
|
54
|
+
* Add secret parameters to role_arn, external_id and ssekms_key_id
|
55
|
+
* Fix region handling in assume_role_credentials
|
56
|
+
|
57
|
+
|
58
|
+
Release 0.6.3 - 2015/11/25
|
59
|
+
|
60
|
+
* Add compute_checksums parameter
|
61
|
+
|
62
|
+
|
63
|
+
Release 0.6.2 - 2015/11/24
|
64
|
+
|
65
|
+
* Add ssekms_key_id option to use KMS encryption
|
66
|
+
|
67
|
+
|
68
|
+
Release 0.6.1 - 2015/10/30
|
69
|
+
|
70
|
+
* Fix server_side_encryption error
|
71
|
+
* Keep hex random identity on rebooting
|
72
|
+
* Fix Tempfile handling on windows
|
73
|
+
|
74
|
+
|
75
|
+
Release 0.6.0 - 2015/10/09
|
76
|
+
|
77
|
+
* Allow path based calling format
|
78
|
+
* Add hex_random placeholder
|
79
|
+
* Add overwrite option
|
80
|
+
|
81
|
+
|
82
|
+
Release 0.6.0.pre1 - 2015/09/10
|
83
|
+
|
84
|
+
* Use AWS SDK v2
|
85
|
+
|
86
|
+
|
87
|
+
Release 0.5.11 - 2015/08/04
|
88
|
+
|
89
|
+
* Add acl parameter
|
90
|
+
* Fix use_server_side_encryption parameter
|
91
|
+
|
92
|
+
|
93
|
+
Release 0.5.10 - 2015/07/27
|
94
|
+
|
95
|
+
* Add "uuid_flush" placeholder to s3_object_key_format for creating unique object
|
96
|
+
|
97
|
+
|
98
|
+
Release 0.5.9 - 2015/06/10
|
99
|
+
|
100
|
+
* Add secret option to AWS key related parameters
|
101
|
+
|
102
|
+
|
103
|
+
Release 0.5.8 - 2015/06/09
|
104
|
+
|
105
|
+
* Fix credential provider selection for env vars
|
106
|
+
* Add aws_iam_retries parameter to set number of attemps to the EC2 metadata service
|
107
|
+
|
108
|
+
|
109
|
+
Release 0.5.7 - 2015/03/31
|
110
|
+
|
111
|
+
* Use s3_endpoint instead of endpoint for S3 Client configuration for S3 compatible services
|
112
|
+
* Increase the number of retry when use IAM role
|
113
|
+
|
114
|
+
|
115
|
+
Release 0.5.6 - 2015/03/02
|
116
|
+
|
117
|
+
* Force to use aws-sdk-v1, not aws-sdk
|
118
|
+
|
119
|
+
|
120
|
+
Release 0.5.5 - 2015/02/19
|
121
|
+
|
122
|
+
* Revive s3_endpoint parameter for S3 compatible services
|
123
|
+
|
124
|
+
|
125
|
+
Release 0.5.4 - 2015/02/12
|
126
|
+
|
127
|
+
* Add use_server_side_encryption parameter
|
128
|
+
|
129
|
+
|
130
|
+
Release 0.5.3 - 2015/02/06
|
131
|
+
|
132
|
+
* Add error information in API check
|
133
|
+
* Add GzipWriter fallback to gzip_command
|
134
|
+
|
135
|
+
|
136
|
+
Release 0.5.2 - 2015/02/05
|
137
|
+
|
138
|
+
* Add experimental gzip_command compressor
|
139
|
+
|
140
|
+
|
141
|
+
Release 0.5.1 - 2014/12/18
|
142
|
+
|
143
|
+
* Skip apikey_check error when auto_create_bucket is true and bucket doesn't exist
|
144
|
+
|
145
|
+
|
146
|
+
Release 0.5.0 - 2014/12/07
|
147
|
+
|
148
|
+
* Make compression algorithm pluggable
|
149
|
+
* Remove format_json parameter
|
150
|
+
* Remove s3_endpoint parameter
|
151
|
+
* Relax fluentd version restriction to support 0.12 or later
|
152
|
+
|
153
|
+
|
154
|
+
Release 0.4.3 - 2014/11/10
|
155
|
+
|
156
|
+
* Change API check message to consider region mismatch
|
157
|
+
|
158
|
+
|
159
|
+
Release 0.4.2 - 2014/10/22
|
160
|
+
|
161
|
+
* Update fluent-mixin-config-placeholders to v0.3.0
|
162
|
+
|
163
|
+
|
164
|
+
Release 0.4.1 - 2014/10/16
|
165
|
+
|
166
|
+
* Add 's3_region' option to specify S3 region
|
167
|
+
* Restrict aws-sdk gem dependency version to use v1
|
168
|
+
* Fix infinite loop when same object path is generated
|
169
|
+
|
170
|
+
|
171
|
+
Release 0.4.0 - 2014/06/06
|
172
|
+
|
173
|
+
* Add 'format' option to change one line format
|
174
|
+
* Update fluentd dependency to v0.10.49
|
175
|
+
|
176
|
+
|
177
|
+
Release 0.3.7 - 2014/03/07
|
178
|
+
|
179
|
+
* Support lzma2 compression using 'xz' command
|
180
|
+
https://github.com/fluent/fluent-plugin-s3/pull/41
|
181
|
+
* Relax aws-sdk gem requirement
|
182
|
+
https://github.com/fluent/fluent-plugin-s3/pull/42
|
183
|
+
|
184
|
+
|
185
|
+
Release 0.3.6 - 2014/02/05
|
186
|
+
|
187
|
+
* Support 'log_level' option
|
188
|
+
|
189
|
+
|
190
|
+
Release 0.3.5 - 2013/12/05
|
191
|
+
|
192
|
+
* Add 'reduced_redundancy' option to store logs in reduced redundancy
|
193
|
+
https://github.com/fluent/fluent-plugin-s3/pull/33
|
194
|
+
|
195
|
+
|
196
|
+
Release 0.3.4 - 2013/07/31
|
197
|
+
|
198
|
+
* Add dynamic path slicing by time formatted string
|
199
|
+
https://github.com/fluent/fluent-plugin-s3/pull/24
|
200
|
+
|
201
|
+
|
202
|
+
Release 0.3.3 - 2013/06/18
|
203
|
+
|
204
|
+
* Fix require bug on case-sensitive environment
|
205
|
+
|
206
|
+
|
207
|
+
Release 0.3.2 - 2013/06/18
|
208
|
+
|
209
|
+
* Support lzo mime-type
|
210
|
+
https://github.com/fluent/fluent-plugin-s3/pull/29
|
211
|
+
* Add proxy_uri option
|
212
|
+
https://github.com/fluent/fluent-plugin-s3/issues/25
|
213
|
+
* Add check_apikey_on_start option
|
214
|
+
https://github.com/fluent/fluent-plugin-s3/pull/28
|
215
|
+
|
216
|
+
|
217
|
+
Release 0.3.1 - 2013/03/28
|
218
|
+
|
219
|
+
* Support json and text mime-types
|
220
|
+
https://github.com/fluent/fluent-plugin-s3/pull/20
|
221
|
+
|
222
|
+
|
223
|
+
Release 0.3.0 - 2013/02/19
|
224
|
+
|
225
|
+
* Enable dynamic and configurable S3 object kyes
|
226
|
+
https://github.com/fluent/fluent-plugin-s3/pull/12
|
227
|
+
* Fix a lot of temporary files were left on /tmp when the plugin failed to write to S3
|
228
|
+
https://github.com/fluent/fluent-plugin-s3/pull/15
|
229
|
+
* Enable fluent-mixin-config-placeholders to support hostname, uuid and other parameters in configuration
|
230
|
+
https://github.com/fluent/fluent-plugin-s3/pull/19
|
231
|
+
* Update 'aws-sdk' version requirement to '~> 1.8.2'
|
232
|
+
https://github.com/fluent/fluent-plugin-s3/pull/21
|
233
|
+
* Create new S3 bucket if not exists
|
234
|
+
https://github.com/fluent/fluent-plugin-s3/pull/22
|
235
|
+
* Check the permission and bucket existence at start method, not write method.
|
236
|
+
|
237
|
+
|
238
|
+
Release 0.2.6 - 2013/01/15
|
239
|
+
|
240
|
+
* Add use_ssl option
|
241
|
+
|
242
|
+
|
243
|
+
Release 0.2.5 - 2012/12/06
|
244
|
+
|
245
|
+
* Add format_json and time/tag mixin options [#9]
|
246
|
+
|
247
|
+
|
248
|
+
Release 0.2.4 - 2012/11/21
|
249
|
+
|
250
|
+
* Set content type when writing file to s3
|
251
|
+
|
252
|
+
|
253
|
+
Release 0.2.3 - 2012/11/19
|
254
|
+
|
255
|
+
* Loosen 'aws-sdk' version requirement from "~> 1.1.3" to "~> 1.1"
|
256
|
+
* Support aws-sdk facility to load credentials from ENV vars or IAM Instance Profile by making the credentials non-mandatory
|
257
|
+
* Use Yajl instead of to_json not to raise exceptions when it got invalid bytes as UTF-8.
|
258
|
+
|
259
|
+
|
260
|
+
Release 0.2.2 - 2011/12/15
|
261
|
+
|
262
|
+
* Add s3_endpoint option
|
263
|
+
|
264
|
+
|
265
|
+
Release 0.2.1 - 2011/10/24
|
266
|
+
|
267
|
+
* Add sequential number to the file to avoid overwriting
|
268
|
+
* Use bundler instead of jeweler for packaging
|
269
|
+
* Updated README
|
270
|
+
|
271
|
+
|
272
|
+
Release 0.2.0 - 2011/10/16
|
273
|
+
|
274
|
+
* Updated to fluentd-0.10.0
|
275
|
+
|
276
|
+
|
277
|
+
Release 0.1.1 - 2011/09/27
|
278
|
+
|
279
|
+
* First release
|
280
|
+
|
data/Gemfile
ADDED
data/Gemfile.v0.12
ADDED
data/README.md
ADDED
@@ -0,0 +1,684 @@
|
|
1
|
+
# Amazon S3 plugin for [Fluentd](http://github.com/fluent/fluentd)
|
2
|
+
|
3
|
+
[<img src="https://travis-ci.org/fluent/fluent-plugin-s3.svg?branch=master"
|
4
|
+
alt="Build Status" />](https://travis-ci.org/fluent/fluent-plugin-s3) [<img
|
5
|
+
src="https://codeclimate.com/github/fluent/fluent-plugin-s3/badges/gpa.svg"
|
6
|
+
/>](https://codeclimate.com/github/fluent/fluent-plugin-s3)
|
7
|
+
|
8
|
+
## PR
|
9
|
+
**it will be greatful that the plugin can push log to owner S3 Storage like Ceph S3.**
|
10
|
+
|
11
|
+
## Overview
|
12
|
+
|
13
|
+
**s3** output plugin buffers event logs in local file and upload it to S3
|
14
|
+
periodically.
|
15
|
+
|
16
|
+
This plugin splits files exactly by using the time of event logs (not the time
|
17
|
+
when the logs are received). For example, a log '2011-01-02 message B' is
|
18
|
+
reached, and then another log '2011-01-03 message B' is reached in this order,
|
19
|
+
the former one is stored in "20110102.gz" file, and latter one in
|
20
|
+
"20110103.gz" file.
|
21
|
+
|
22
|
+
**s3** input plugin reads data from S3 periodically. This plugin uses
|
23
|
+
SQS queue on the region same as S3 bucket.
|
24
|
+
We must setup SQS queue and S3 event notification before use this plugin.
|
25
|
+
|
26
|
+
## Requirements
|
27
|
+
|
28
|
+
| fluent-plugin-s3 | fluentd | ruby |
|
29
|
+
|-------------------|---------|------|
|
30
|
+
| >= 1.0.0 | >= v0.14.0 | >= 2.1 |
|
31
|
+
| < 1.0.0 | >= v0.12.0 | >= 1.9 |
|
32
|
+
|
33
|
+
NOTE: fluent-plugin-s3 v1.0.0 is now RC. We will release stable v1.0.0 soon.
|
34
|
+
|
35
|
+
## Installation
|
36
|
+
|
37
|
+
Simply use RubyGems:
|
38
|
+
|
39
|
+
$ gem install fluent-plugin-s3 -v "~> 0.8" --no-document # for fluentd v0.12 or later
|
40
|
+
$ gem install fluent-plugin-s3 -v 1.0.0.rc2 --no-document # for fluentd v0.14 or later
|
41
|
+
|
42
|
+
## Output: Configuration
|
43
|
+
|
44
|
+
### v0.14 style
|
45
|
+
|
46
|
+
With fluentd v0.14 and fluent-plugin-s3 v1.0.0, use new buffer configuration to dynamic parameters.
|
47
|
+
|
48
|
+
<match pattern>
|
49
|
+
@type s3
|
50
|
+
|
51
|
+
aws_key_id YOUR_AWS_KEY_ID
|
52
|
+
aws_sec_key YOUR_AWS_SECRET_KEY
|
53
|
+
s3_bucket YOUR_S3_BUCKET_NAME
|
54
|
+
s3_region ap-northeast-1
|
55
|
+
|
56
|
+
path logs/${tag}/%Y/%m/%d/
|
57
|
+
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
|
58
|
+
|
59
|
+
# if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
|
60
|
+
# need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
|
61
|
+
<buffer tag,time>
|
62
|
+
@type file
|
63
|
+
path /var/log/fluent/s3
|
64
|
+
timekey 3600 # 1 hour partition
|
65
|
+
timekey_wait 10m
|
66
|
+
timekey_use_utc true # use utc
|
67
|
+
</buffer>
|
68
|
+
<format>
|
69
|
+
@type json
|
70
|
+
</format>
|
71
|
+
</match>
|
72
|
+
|
73
|
+
For `<buffer>`, you can use any record field in `path` / `s3_object_key_format`.
|
74
|
+
|
75
|
+
path logs/${tag}/${foo}
|
76
|
+
<buffer tag,foo>
|
77
|
+
# parameters...
|
78
|
+
</buffer>
|
79
|
+
|
80
|
+
See official article for more detail: [Buffer section configurations](http://docs.fluentd.org/v0.14/articles/buffer-section)
|
81
|
+
|
82
|
+
Note that this configuration doesn't work with fluentd v0.12.
|
83
|
+
|
84
|
+
### v0.12 style
|
85
|
+
|
86
|
+
This configuration works with both fluentd v0.12 and v0.14.
|
87
|
+
|
88
|
+
<match pattern>
|
89
|
+
@type s3
|
90
|
+
|
91
|
+
aws_key_id YOUR_AWS_KEY_ID
|
92
|
+
aws_sec_key YOUR_AWS_SECRET_KEY
|
93
|
+
s3_bucket YOUR_S3_BUCKET_NAME
|
94
|
+
s3_region ap-northeast-1
|
95
|
+
|
96
|
+
path logs/
|
97
|
+
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
|
98
|
+
buffer_path /var/log/fluent/s3
|
99
|
+
time_slice_format %Y%m%d-%H
|
100
|
+
time_slice_wait 10m
|
101
|
+
utc
|
102
|
+
format json
|
103
|
+
</match>
|
104
|
+
|
105
|
+
If you want to embed tag in `path` / `s3_object_key_format`, you need to use `fluent-plugin-forest` plugin.
|
106
|
+
|
107
|
+
**aws_key_id**
|
108
|
+
|
109
|
+
AWS access key id. This parameter is required when your agent is not
|
110
|
+
running on EC2 instance with an IAM Role. When using an IAM role, make
|
111
|
+
sure to configure `instance_profile_credentials`. Usage can be found below.
|
112
|
+
|
113
|
+
**aws_sec_key**
|
114
|
+
|
115
|
+
AWS secret key. This parameter is required when your agent is not running
|
116
|
+
on EC2 instance with an IAM Role.
|
117
|
+
|
118
|
+
**aws_iam_retries**
|
119
|
+
|
120
|
+
The number of attempts to make (with exponential backoff) when loading
|
121
|
+
instance profile credentials from the EC2 metadata service using an IAM
|
122
|
+
role. Defaults to 5 retries.
|
123
|
+
|
124
|
+
**s3_bucket (required)**
|
125
|
+
|
126
|
+
S3 bucket name.
|
127
|
+
|
128
|
+
**s3_region**
|
129
|
+
|
130
|
+
s3 region name. For example, US West (Oregon) Region is "us-west-2". The
|
131
|
+
full list of regions are available here. >
|
132
|
+
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
|
133
|
+
recommend using `s3_region` instead of `s3_endpoint`.
|
134
|
+
|
135
|
+
**s3_endpoint**
|
136
|
+
|
137
|
+
endpoint for S3 compatible services. For example, Riak CS based storage or
|
138
|
+
something. This option doesn't work on S3, use `s3_region` instead.
|
139
|
+
|
140
|
+
**ssl_verify_peer**
|
141
|
+
|
142
|
+
Verify SSL certificate of the endpoint. Default is true. Set false when you want to ignore the endpoint SSL certificate.
|
143
|
+
|
144
|
+
**s3_object_key_format**
|
145
|
+
|
146
|
+
The format of S3 object keys. You can use several built-in variables:
|
147
|
+
|
148
|
+
* %{path}
|
149
|
+
* %{time_slice}
|
150
|
+
* %{index}
|
151
|
+
* %{file_extension}
|
152
|
+
* %{hex_random}
|
153
|
+
* %{uuid_flush}
|
154
|
+
* %{hostname}
|
155
|
+
|
156
|
+
to decide keys dynamically.
|
157
|
+
|
158
|
+
* %{path} is exactly the value of **path** configured in the configuration file.
|
159
|
+
E.g., "logs/" in the example configuration above.
|
160
|
+
* %{time_slice} is the
|
161
|
+
time-slice in text that are formatted with **time_slice_format**. %{index} is
|
162
|
+
the sequential number starts from 0, increments when multiple files are
|
163
|
+
uploaded to S3 in the same time slice.
|
164
|
+
* %{file_extention} is always "gz" for
|
165
|
+
now.
|
166
|
+
* %{uuid_flush} a uuid that is replaced everytime the buffer will be flushed. If you want to use this placeholder, install `uuidtools` gem first.
|
167
|
+
* %{hostname} is replaced with `Socket.gethostname` result.
|
168
|
+
* %{hex_random} a random hex string that is replaced for each buffer chunk, not
|
169
|
+
assured to be unique. This is used to follow a way of performance tuning, `Add
|
170
|
+
a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance
|
171
|
+
Considerations - Amazon Simple Storage
|
172
|
+
Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
|
173
|
+
You can configure the length of string with a
|
174
|
+
`hex_random_length` parameter (Default: 4).
|
175
|
+
|
176
|
+
The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
|
177
|
+
|
178
|
+
For instance, using the example configuration above, actual object keys on S3
|
179
|
+
will be something like:
|
180
|
+
|
181
|
+
"logs/20130111-22_0.gz"
|
182
|
+
"logs/20130111-23_0.gz"
|
183
|
+
"logs/20130111-23_1.gz"
|
184
|
+
"logs/20130112-00_0.gz"
|
185
|
+
|
186
|
+
With the configuration:
|
187
|
+
|
188
|
+
s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
|
189
|
+
path log
|
190
|
+
time_slice_format %Y%m%d-%H
|
191
|
+
|
192
|
+
You get:
|
193
|
+
|
194
|
+
"log/events/ts=20130111-22/events_0.gz"
|
195
|
+
"log/events/ts=20130111-23/events_0.gz"
|
196
|
+
"log/events/ts=20130111-23/events_1.gz"
|
197
|
+
"log/events/ts=20130112-00/events_0.gz"
|
198
|
+
|
199
|
+
NOTE: ${hostname} placeholder is deprecated since v0.8. You can get same result by using [configuration's embedded ruby code feature](http://docs.fluentd.org/articles/config-file#embedded-ruby-code).
|
200
|
+
|
201
|
+
s3_object_key_format %{path}%{time_slice}_%{hostname}%{index}.%{file_extension}
|
202
|
+
s3_object_key_format "%{path}%{time_slice}_#{Socket.gethostname}%{index}.%{file_extension}"
|
203
|
+
|
204
|
+
Above two configurations are same. The important point is wrapping `""` is needed for `#{Socket.gethostname}`.
|
205
|
+
|
206
|
+
**force_path_style**
|
207
|
+
|
208
|
+
:force_path_style (Boolean) — default: false — When set to true, the
|
209
|
+
bucket name is always left in the request URI and never moved to the host
|
210
|
+
as a sub-domain. See Plugins::S3BucketDns for more details.
|
211
|
+
|
212
|
+
**store_as**
|
213
|
+
|
214
|
+
archive format on S3. You can use several format:
|
215
|
+
|
216
|
+
* gzip (default)
|
217
|
+
* json
|
218
|
+
* text
|
219
|
+
* lzo (Need lzop command)
|
220
|
+
* lzma2 (Need xz command)
|
221
|
+
* gzip_command (Need gzip command)
|
222
|
+
* This compressor uses an external gzip command, hence would result in
|
223
|
+
utilizing CPU cores well compared with `gzip`
|
224
|
+
|
225
|
+
See `Use your compression algorithm` section for adding another format.
|
226
|
+
|
227
|
+
**`<format>` or format**
|
228
|
+
|
229
|
+
Change one line format in the S3 object. Supported formats are "out_file",
|
230
|
+
"json", "ltsv" and "single_value". See also [official Formatter article](http://docs.fluentd.org/articles/formatter-plugin-overview).
|
231
|
+
|
232
|
+
* out_file (default).
|
233
|
+
|
234
|
+
time\ttag\t{..json1..}
|
235
|
+
time\ttag\t{..json2..}
|
236
|
+
...
|
237
|
+
|
238
|
+
* json
|
239
|
+
|
240
|
+
{..json1..}
|
241
|
+
{..json2..}
|
242
|
+
...
|
243
|
+
|
244
|
+
|
245
|
+
At this format, "time" and "tag" are omitted. But you can set these
|
246
|
+
information to the record by setting "include_tag_key" / "tag_key" and
|
247
|
+
"include_time_key" / "time_key" option. If you set following configuration in
|
248
|
+
S3 output:
|
249
|
+
|
250
|
+
format json
|
251
|
+
include_time_key true
|
252
|
+
time_key log_time # default is time
|
253
|
+
|
254
|
+
then the record has log_time field.
|
255
|
+
|
256
|
+
{"log_time":"time string",...}
|
257
|
+
|
258
|
+
* ltsv
|
259
|
+
|
260
|
+
key1:value1\tkey2:value2
|
261
|
+
key1:value1\tkey2:value2
|
262
|
+
...
|
263
|
+
|
264
|
+
|
265
|
+
"ltsv" format also accepts "include_xxx" related options. See "json" section.
|
266
|
+
|
267
|
+
* single_value
|
268
|
+
|
269
|
+
|
270
|
+
Use specified value instead of entire recode. If you get '{"message":"my
|
271
|
+
log"}', then contents are
|
272
|
+
|
273
|
+
my log1
|
274
|
+
my log2
|
275
|
+
...
|
276
|
+
|
277
|
+
You can change key name by "message_key" option.
|
278
|
+
|
279
|
+
**auto_create_bucket**
|
280
|
+
|
281
|
+
Create S3 bucket if it does not exists. Default is true.
|
282
|
+
|
283
|
+
**check_bucket**
|
284
|
+
|
285
|
+
Check mentioned bucket if it exists in AWS or not. Default is true.
|
286
|
+
|
287
|
+
When it is false,
|
288
|
+
fluentd will not check aws s3 for the existence of the mentioned bucket. This is the
|
289
|
+
case where bucket will be pre-created before running fluentd.
|
290
|
+
|
291
|
+
**check_object**
|
292
|
+
|
293
|
+
Check object before creation if it exists or not. Default is true.
|
294
|
+
|
295
|
+
When it is false,
|
296
|
+
s3_object_key_format will be %{path}%{date_slice}_%{time_slice}.%{file_extension}
|
297
|
+
where, time_slice will be in hhmmss format, so that each object will be unique.
|
298
|
+
Example object name, assuming it is created on 2016/16/11 3:30:54 PM
|
299
|
+
20161611_153054.txt (extension can be anything as per user's choice)
|
300
|
+
|
301
|
+
**Example when check_bucket=false and check_object=false**
|
302
|
+
|
303
|
+
When the mentioned configuration will be made, fluentd will work with the
|
304
|
+
minimum IAM poilcy, like:
|
305
|
+
"Statement": [{
|
306
|
+
"Effect": "Allow",
|
307
|
+
"Action": "s3:PutObject",
|
308
|
+
"Resource": ["*"]
|
309
|
+
}]
|
310
|
+
|
311
|
+
**check_apikey_on_start**
|
312
|
+
|
313
|
+
Check AWS key on start. Default is true.
|
314
|
+
|
315
|
+
**proxy_uri**
|
316
|
+
|
317
|
+
uri of proxy environment.
|
318
|
+
|
319
|
+
**path**
|
320
|
+
|
321
|
+
path prefix of the files on S3. Default is "" (no prefix).
|
322
|
+
|
323
|
+
**buffer_path (required)**
|
324
|
+
|
325
|
+
path prefix of the files to buffer logs.
|
326
|
+
|
327
|
+
**time_slice_format**
|
328
|
+
|
329
|
+
Format of the time used as the file name. Default is '%Y%m%d'. Use
|
330
|
+
'%Y%m%d%H' to split files hourly.
|
331
|
+
|
332
|
+
**time_slice_wait**
|
333
|
+
|
334
|
+
The time to wait old logs. Default is 10 minutes. Specify larger value if
|
335
|
+
old logs may reach.
|
336
|
+
|
337
|
+
**utc**
|
338
|
+
|
339
|
+
Use UTC instead of local time.
|
340
|
+
|
341
|
+
**storage_class**
|
342
|
+
|
343
|
+
Set storage class. Possible values are `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA` from [Ruby SDK](http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#storage_class-instance_method).
|
344
|
+
|
345
|
+
**reduced_redundancy**
|
346
|
+
|
347
|
+
Use S3 reduced redundancy storage for 33% cheaper pricing. Default is
|
348
|
+
false.
|
349
|
+
|
350
|
+
This is deprecated. Use `storage_class REDUCED_REDUNDANCY` instead.
|
351
|
+
|
352
|
+
**acl**
|
353
|
+
|
354
|
+
Permission for the object in S3. This is useful for cross-account access
|
355
|
+
using IAM roles. Valid values are:
|
356
|
+
|
357
|
+
* private (default)
|
358
|
+
* public-read
|
359
|
+
* public-read-write (not recommended - see [Canned
|
360
|
+
ACL](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl))
|
361
|
+
* authenticated-read
|
362
|
+
* bucket-owner-read
|
363
|
+
* bucket-owner-full-control
|
364
|
+
|
365
|
+
To use cross-account access, you will need to create a bucket policy granting
|
366
|
+
the specific access required. Refer to the [AWS
|
367
|
+
documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example3.html) for examples.
|
368
|
+
|
369
|
+
**hex_random_length**
|
370
|
+
|
371
|
+
The length of `%{hex_random}` placeholder. Default is 4 as written in
|
372
|
+
[Request Rate and Performance Considerations - Amazon Simple Storage
|
373
|
+
Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
|
374
|
+
The maximum length is 16.
|
375
|
+
|
376
|
+
**overwrite**
|
377
|
+
|
378
|
+
Overwrite already existing path. Default is false, which raises an error
|
379
|
+
if a s3 object of the same path already exists, or increment the
|
380
|
+
`%{index}` placeholder until finding an absent path.
|
381
|
+
|
382
|
+
**use_server_side_encryption**
|
383
|
+
|
384
|
+
The Server-side encryption algorithm used when storing this object in S3
|
385
|
+
(e.g., AES256, aws:kms)
|
386
|
+
|
387
|
+
**ssekms_key_id**
|
388
|
+
|
389
|
+
Specifies the AWS KMS key ID to use for object encryption. You have to
|
390
|
+
set "aws:kms" to `use_server_side_encryption` to use the KMS encryption.
|
391
|
+
|
392
|
+
**sse_customer_algorithm**
|
393
|
+
|
394
|
+
Specifies the algorithm to use to when encrypting the object (e.g., AES256).
|
395
|
+
|
396
|
+
**sse_customer_key**
|
397
|
+
|
398
|
+
Specifies the AWS KMS key ID to use for object encryption.
|
399
|
+
|
400
|
+
**sse_customer_key_md5**
|
401
|
+
|
402
|
+
Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
|
403
|
+
|
404
|
+
**compute_checksums**
|
405
|
+
|
406
|
+
AWS SDK uses MD5 for API request/response by default. On FIPS enabled environment,
|
407
|
+
OpenSSL returns an error because MD5 is disabled. If you want to use
|
408
|
+
this plugin on FIPS enabled environment, set `compute_checksums false`.
|
409
|
+
|
410
|
+
**signature_version**
|
411
|
+
|
412
|
+
Signature version for API request. `s3` means signature version 2 and
|
413
|
+
`v4` means signature version 4. Default is `nil` (Following SDK's default).
|
414
|
+
It would be useful when you use S3 compatible storage that accepts only signature version 2.
|
415
|
+
|
416
|
+
**warn_for_delay**
|
417
|
+
|
418
|
+
Given a threshold to treat events as delay, output warning logs if delayed events were put into s3.
|
419
|
+
|
420
|
+
### assume_role_credentials
|
421
|
+
|
422
|
+
Typically, you use AssumeRole for cross-account access or federation.
|
423
|
+
|
424
|
+
<match *>
|
425
|
+
@type s3
|
426
|
+
|
427
|
+
<assume_role_credentials>
|
428
|
+
role_arn ROLE_ARN
|
429
|
+
role_session_name ROLE_SESSION_NAME
|
430
|
+
</assume_role_credentials>
|
431
|
+
</match>
|
432
|
+
|
433
|
+
See also:
|
434
|
+
|
435
|
+
* [Using IAM Roles - AWS Identity and Access
|
436
|
+
Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
|
437
|
+
* [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
|
438
|
+
* [Aws::AssumeRoleCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html)
|
439
|
+
|
440
|
+
**role_arn (required)**
|
441
|
+
|
442
|
+
The Amazon Resource Name (ARN) of the role to assume.
|
443
|
+
|
444
|
+
**role_session_name (required)**
|
445
|
+
|
446
|
+
An identifier for the assumed role session.
|
447
|
+
|
448
|
+
**policy**
|
449
|
+
|
450
|
+
An IAM policy in JSON format.
|
451
|
+
|
452
|
+
**duration_seconds**
|
453
|
+
|
454
|
+
The duration, in seconds, of the role session. The value can range from
|
455
|
+
900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value
|
456
|
+
is set to 3600 seconds.
|
457
|
+
|
458
|
+
**external_id**
|
459
|
+
|
460
|
+
A unique identifier that is used by third parties when assuming roles in
|
461
|
+
their customers' accounts.
|
462
|
+
|
463
|
+
### instance_profile_credentials
|
464
|
+
|
465
|
+
Retrieve temporary security credentials via HTTP request. This is useful on
|
466
|
+
EC2 instance.
|
467
|
+
|
468
|
+
<match *>
|
469
|
+
@type s3
|
470
|
+
|
471
|
+
<instance_profile_credentials>
|
472
|
+
ip_address IP_ADDRESS
|
473
|
+
port PORT
|
474
|
+
</instance_profile_credentials>
|
475
|
+
</match>
|
476
|
+
|
477
|
+
See also:
|
478
|
+
|
479
|
+
* [Aws::InstanceProfileCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html)
|
480
|
+
* [Temporary Security Credentials - AWS Identity and Access
|
481
|
+
Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
|
482
|
+
* [Instance Metadata and User Data - Amazon Elastic Compute
|
483
|
+
Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
|
484
|
+
|
485
|
+
**retries**
|
486
|
+
|
487
|
+
Number of times to retry when retrieving credentials. Default is 5.
|
488
|
+
|
489
|
+
**ip_address**
|
490
|
+
|
491
|
+
Default is 169.254.169.254.
|
492
|
+
|
493
|
+
**port**
|
494
|
+
|
495
|
+
Default is 80.
|
496
|
+
|
497
|
+
**http_open_timeout**
|
498
|
+
|
499
|
+
Default is 5.
|
500
|
+
|
501
|
+
**http_read_timeout**
|
502
|
+
|
503
|
+
Default is 5.
|
504
|
+
|
505
|
+
### shared_credentials
|
506
|
+
|
507
|
+
This loads AWS access credentials from local ini file. This is useful for
|
508
|
+
local developing.
|
509
|
+
|
510
|
+
<match *>
|
511
|
+
@type s3
|
512
|
+
|
513
|
+
<shared_credentials>
|
514
|
+
path PATH
|
515
|
+
profile_name PROFILE_NAME
|
516
|
+
</shared_credentials>
|
517
|
+
</match>
|
518
|
+
|
519
|
+
See also:
|
520
|
+
|
521
|
+
* [Aws::SharedCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html)
|
522
|
+
|
523
|
+
**path**
|
524
|
+
|
525
|
+
Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials".
|
526
|
+
|
527
|
+
**profile_name**
|
528
|
+
|
529
|
+
Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
|
530
|
+
|
531
|
+
## Input: Setup
|
532
|
+
|
533
|
+
1. Create new [SQS](https://aws.amazon.com/documentation/sqs/) queue (use same region as S3)
|
534
|
+
2. Set proper permission to new queue
|
535
|
+
3. [Configure S3 event notification](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html)
|
536
|
+
4. Write configuration file such as fluent.conf
|
537
|
+
5. Run fluentd
|
538
|
+
|
539
|
+
## Input: Configuration
|
540
|
+
|
541
|
+
<source>
|
542
|
+
@type s3
|
543
|
+
|
544
|
+
aws_key_id YOUR_AWS_KEY_ID
|
545
|
+
aws_sec_key YOUR_AWS_SECRET_KEY
|
546
|
+
s3_bucket YOUR_S3_BUCKET_NAME
|
547
|
+
s3_region ap-northeast-1
|
548
|
+
|
549
|
+
<sqs>
|
550
|
+
queue_name YOUR_SQS_QUEUE_NAME
|
551
|
+
</sqs>
|
552
|
+
</source>
|
553
|
+
|
554
|
+
**aws_key_id**
|
555
|
+
|
556
|
+
AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Role.
|
557
|
+
|
558
|
+
**aws_sec_key**
|
559
|
+
|
560
|
+
AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Role.
|
561
|
+
|
562
|
+
**aws_iam_retries**
|
563
|
+
|
564
|
+
The number of attempts to make (with exponential backoff) when loading instance profile credentials from the EC2 metadata
|
565
|
+
service using an IAM role. Defaults to 5 retries.
|
566
|
+
|
567
|
+
**s3_bucket (required)**
|
568
|
+
|
569
|
+
S3 bucket name.
|
570
|
+
|
571
|
+
**s3_region**
|
572
|
+
|
573
|
+
S3 region name. For example, US West (Oregon) Region is
|
574
|
+
"us-west-2". The full list of regions are available here. >
|
575
|
+
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
|
576
|
+
recommend using `s3_region` instead of `s3_endpoint`.
|
577
|
+
|
578
|
+
**store_as**
|
579
|
+
|
580
|
+
archive format on S3. You can use serveral format:
|
581
|
+
|
582
|
+
* gzip (default)
|
583
|
+
* json
|
584
|
+
* text
|
585
|
+
* lzo (Need lzop command)
|
586
|
+
* lzma2 (Need xz command)
|
587
|
+
* gzip_command (Need gzip command)
|
588
|
+
* This compressor uses an external gzip command, hence would result in utilizing CPU cores well compared with `gzip`
|
589
|
+
|
590
|
+
See 'Use your compression algorithm' section for adding another format.
|
591
|
+
|
592
|
+
**format**
|
593
|
+
|
594
|
+
Parse a line as this format in the S3 object. Supported formats are
|
595
|
+
"apache_error", "apache2", "syslog", "json", "tsv", "ltsv", "csv",
|
596
|
+
"nginx" and "none".
|
597
|
+
|
598
|
+
**check_apikey_on_start**
|
599
|
+
|
600
|
+
Check AWS key on start. Default is true.
|
601
|
+
|
602
|
+
**proxy_uri**
|
603
|
+
|
604
|
+
URI of proxy environment.
|
605
|
+
|
606
|
+
**sqs/queue_name (required)**
|
607
|
+
|
608
|
+
SQS queue name. Need to create SQS queue on the region same as S3 bucket.
|
609
|
+
|
610
|
+
**sqs/skip_delete**
|
611
|
+
|
612
|
+
When true, messages are not deleted after polling block. Default is false.
|
613
|
+
|
614
|
+
**sqs/wait_time_seconds**
|
615
|
+
|
616
|
+
The long polling interval. Default is 20.
|
617
|
+
|
618
|
+
## IAM Policy
|
619
|
+
|
620
|
+
The following is an example for a minimal IAM policy needed to write to an s3
|
621
|
+
bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
|
622
|
+
|
623
|
+
{ "Statement": [
|
624
|
+
{ "Effect":"Allow",
|
625
|
+
"Action":"s3:*",
|
626
|
+
"Resource":"arn:aws:s3:::my-s3bucket*"
|
627
|
+
} ]
|
628
|
+
}
|
629
|
+
|
630
|
+
Note that the bucket must already exist and **auto_create_bucket** has no
|
631
|
+
effect in this case.
|
632
|
+
|
633
|
+
Refer to the [AWS
|
634
|
+
documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolicies.html) for example policies.
|
635
|
+
|
636
|
+
Using [IAM
|
637
|
+
roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
|
638
|
+
with a properly configured IAM policy are preferred over embedding access keys
|
639
|
+
on EC2 instances.
|
640
|
+
|
641
|
+
## Use your (de)compression algorithm
|
642
|
+
|
643
|
+
s3 plugin has plugabble compression mechanizm like Fleuntd's input / output
|
644
|
+
plugin. If you set 'store_as xxx', `out_s3` plugin searches
|
645
|
+
`fluent/plugin/s3_compressor_xxx.rb` and `in_s3` plugin searches
|
646
|
+
`fluent/plugin/s3_extractor_xxx.rb`. You can define your (de)compression with
|
647
|
+
'S3Output::Compressor'/`S3Input::Extractor` classes. Compressor API is here:
|
648
|
+
|
649
|
+
module Fluent # Since fluent-plugin-s3 v1.0.0 or later, use Fluent::Plugin instead of Fluent
|
650
|
+
class S3Output
|
651
|
+
class XXXCompressor < Compressor
|
652
|
+
S3Output.register_compressor('xxx', self)
|
653
|
+
|
654
|
+
# Used to file extension
|
655
|
+
def ext
|
656
|
+
'xxx'
|
657
|
+
end
|
658
|
+
|
659
|
+
# Used to file content type
|
660
|
+
def content_type
|
661
|
+
'application/x-xxx'
|
662
|
+
end
|
663
|
+
|
664
|
+
# chunk is buffer chunk. tmp is destination file for upload
|
665
|
+
def compress(chunk, tmp)
|
666
|
+
# call command or something
|
667
|
+
end
|
668
|
+
end
|
669
|
+
end
|
670
|
+
end
|
671
|
+
|
672
|
+
`Extractor` is similar to `Compressor`
|
673
|
+
See bundled `Compressor`/`Extractor` classes for more detail.
|
674
|
+
|
675
|
+
## Website, license, et. al.
|
676
|
+
|
677
|
+
| Web site | http://fluentd.org/ |
|
678
|
+
|-------------------|-------------------------------------------|
|
679
|
+
| Documents | http://docs.fluentd.org/ |
|
680
|
+
| Source repository | http://github.com/fluent/fluent-plugin-s3 |
|
681
|
+
| Discussion | http://groups.google.com/group/fluentd |
|
682
|
+
| Author | Sadayuki Furuhashi |
|
683
|
+
| Copyright | (c) 2011 FURUHASHI Sadayuki |
|
684
|
+
| License | Apache License, Version 2.0 |
|