fluent-plugin-s3-hubspot 1.0.0.rc5

data/.gitignore

@@ -0,0 +1,13 @@
+~*
+#*
+*~
+[._]*.s[a-w][a-z]
+.DS_Store
+
+*.gem
+.bundle
+Gemfile.lock
+vendor
+.ruby-version
+
+test/tmp/

data/.travis.yml

@@ -0,0 +1,25 @@
+language: ruby
+
+rvm:
+  - 2.1.10
+  - 2.2.5
+  - 2.3.1
+  - 2.4.0
+  - ruby-head
+
+gemfile:
+ - Gemfile
+
+branches:
+  only:
+    - master
+    - v0.12
+
+before_install: gem update bundler
+script: bundle exec rake test
+
+sudo: false
+
+matrix:
+  allow_failures:
+    - rvm: ruby-head

data/AUTHORS

@@ -0,0 +1,2 @@
+FURUHASHI Sadayuki <frsyuki _at_ gmail.com>
+MASAHIRO Nakagawa <repeatedly _at_ gmail.com>

data/ChangeLog

@@ -0,0 +1,280 @@
+Release 0.8.0 - 2016/12/20
+
+  * out_s3: Add check_object / check_bucket parameters for only put permission
+  * Remove fluent-mixin-config-placeholders dependency
+
+
+Release 0.7.2 - 2016/10/20
+
+  * in_s3: Replace old parser API with new one
+  * in_s3: Don't stop SQS polling when error happens
+
+
+Release 0.7.1 - 2016/09/02
+
+  * Support IAM role for Amazon ECS task
+
+
+Release 0.7.0 - 2016/08/10
+
+  * Add s3 input plugin
+
+
+Release 0.6.9 - 2016/07/28
+
+  * Support v0.14
+  * Fix aws_iam_retries warning
+  * Fix race condition at start with buffers
+
+
+Release 0.6.8 - 2016/04/19
+
+  * Add sse customer key options
+
+
+Release 0.6.7 - 2016/03/31
+
+  * Add signature_version parameter
+  * Add warn_for_delay parameter
+
+
+Release 0.6.6 - 2016/03/16
+
+  * Fix ACL handling in PUT operation
+
+
+Release 0.6.5 - 2016/01/13
+
+  * Add description to parameters
+  * Use path as prefix in API key check
+
+
+Release 0.6.4 - 2015/12/03
+
+  * Add secret parameters to role_arn, external_id and ssekms_key_id
+  * Fix region handling in assume_role_credentials
+
+
+Release 0.6.3 - 2015/11/25
+
+  * Add compute_checksums parameter
+
+
+Release 0.6.2 - 2015/11/24
+
+  * Add ssekms_key_id option to use KMS encryption
+
+
+Release 0.6.1 - 2015/10/30
+
+  * Fix server_side_encryption error
+  * Keep hex random identity on rebooting
+  * Fix Tempfile handling on windows
+
+
+Release 0.6.0 - 2015/10/09
+
+  * Allow path based calling format
+  * Add hex_random placeholder
+  * Add overwrite option
+
+
+Release 0.6.0.pre1 - 2015/09/10
+
+  * Use AWS SDK v2
+
+
+Release 0.5.11 - 2015/08/04
+
+  * Add acl parameter
+  * Fix use_server_side_encryption parameter
+
+
+Release 0.5.10 - 2015/07/27
+
+  * Add "uuid_flush" placeholder to s3_object_key_format for creating unique object
+
+
+Release 0.5.9 - 2015/06/10
+
+  * Add secret option to AWS key related parameters
+
+
+Release 0.5.8 - 2015/06/09
+
+  * Fix credential provider selection for env vars
+  * Add aws_iam_retries parameter to set number of attemps to the EC2 metadata service
+
+
+Release 0.5.7 - 2015/03/31
+
+  * Use s3_endpoint instead of endpoint for S3 Client configuration for S3 compatible services
+  * Increase the number of retry when use IAM role
+
+
+Release 0.5.6 - 2015/03/02
+
+  * Force to use aws-sdk-v1, not aws-sdk
+
+
+Release 0.5.5 - 2015/02/19
+
+  * Revive s3_endpoint parameter for S3 compatible services
+
+
+Release 0.5.4 - 2015/02/12
+
+  * Add use_server_side_encryption parameter
+
+
+Release 0.5.3 - 2015/02/06
+
+  * Add error information in API check
+  * Add GzipWriter fallback to gzip_command
+
+
+Release 0.5.2 - 2015/02/05
+
+  * Add experimental gzip_command compressor
+
+
+Release 0.5.1 - 2014/12/18
+
+  * Skip apikey_check error when auto_create_bucket is true and bucket doesn't exist
+
+
+Release 0.5.0 - 2014/12/07
+
+  * Make compression algorithm pluggable
+  * Remove format_json parameter
+  * Remove s3_endpoint parameter
+  * Relax fluentd version restriction to support 0.12 or later
+
+
+Release 0.4.3 - 2014/11/10
+
+  * Change API check message to consider region mismatch
+
+
+Release 0.4.2 - 2014/10/22
+
+  * Update fluent-mixin-config-placeholders to v0.3.0
+
+
+Release 0.4.1 - 2014/10/16
+
+  * Add 's3_region' option to specify S3 region
+  * Restrict aws-sdk gem dependency version to use v1
+  * Fix infinite loop when same object path is generated
+
+
+Release 0.4.0 - 2014/06/06
+
+  * Add 'format' option to change one line format
+  * Update fluentd dependency to v0.10.49
+
+
+Release 0.3.7 - 2014/03/07
+
+  * Support lzma2 compression using 'xz' command
+    https://github.com/fluent/fluent-plugin-s3/pull/41
+  * Relax aws-sdk gem requirement
+    https://github.com/fluent/fluent-plugin-s3/pull/42
+
+
+Release 0.3.6 - 2014/02/05
+
+  * Support 'log_level' option
+
+
+Release 0.3.5 - 2013/12/05
+
+  * Add 'reduced_redundancy' option to store logs in reduced redundancy
+    https://github.com/fluent/fluent-plugin-s3/pull/33
+
+
+Release 0.3.4 - 2013/07/31
+
+  * Add dynamic path slicing by time formatted string
+    https://github.com/fluent/fluent-plugin-s3/pull/24
+
+
+Release 0.3.3 - 2013/06/18
+
+  * Fix require bug on case-sensitive environment
+
+
+Release 0.3.2 - 2013/06/18
+
+  * Support lzo mime-type
+    https://github.com/fluent/fluent-plugin-s3/pull/29
+  * Add proxy_uri option
+    https://github.com/fluent/fluent-plugin-s3/issues/25
+  * Add check_apikey_on_start option
+    https://github.com/fluent/fluent-plugin-s3/pull/28
+
+
+Release 0.3.1 - 2013/03/28
+
+  * Support json and text mime-types
+    https://github.com/fluent/fluent-plugin-s3/pull/20
+
+
+Release 0.3.0 - 2013/02/19
+
+  * Enable dynamic and configurable S3 object kyes
+    https://github.com/fluent/fluent-plugin-s3/pull/12
+  * Fix a lot of temporary files were left on /tmp when the plugin failed to write to S3
+    https://github.com/fluent/fluent-plugin-s3/pull/15
+  * Enable fluent-mixin-config-placeholders to support hostname, uuid and other parameters in configuration
+    https://github.com/fluent/fluent-plugin-s3/pull/19
+  * Update 'aws-sdk' version requirement to '~> 1.8.2'
+    https://github.com/fluent/fluent-plugin-s3/pull/21
+  * Create new S3 bucket if not exists
+    https://github.com/fluent/fluent-plugin-s3/pull/22
+  * Check the permission and bucket existence at start method, not write method.
+
+
+Release 0.2.6 - 2013/01/15
+
+  * Add use_ssl option
+
+
+Release 0.2.5 - 2012/12/06
+
+  * Add format_json and time/tag mixin options [#9]
+
+
+Release 0.2.4 - 2012/11/21
+
+  * Set content type when writing file to s3
+
+
+Release 0.2.3 - 2012/11/19
+
+  * Loosen 'aws-sdk' version requirement from "~> 1.1.3" to "~> 1.1"
+  * Support aws-sdk facility to load credentials from ENV vars or IAM Instance Profile by making the credentials non-mandatory
+  * Use Yajl instead of to_json not to raise exceptions when it got invalid bytes as UTF-8.
+
+
+Release 0.2.2 - 2011/12/15
+
+  * Add s3_endpoint option
+
+
+Release 0.2.1 - 2011/10/24
+
+  * Add sequential number to the file to avoid overwriting
+  * Use bundler instead of jeweler for packaging
+  * Updated README
+
+
+Release 0.2.0 - 2011/10/16
+
+  * Updated to fluentd-0.10.0
+
+
+Release 0.1.1 - 2011/09/27
+
+  * First release
+

data/Gemfile

@@ -0,0 +1,5 @@
+source "http://rubygems.org"
+
+gem 'uuidtools'
+
+gemspec

data/Gemfile.v0.12

@@ -0,0 +1,6 @@
+source "http://rubygems.org"
+
+gem 'json', '= 1.8.3'
+gem 'fluentd', '~> 0.12.0'
+
+gemspec

data/README.md

@@ -0,0 +1,684 @@
+# Amazon S3 plugin for [Fluentd](http://github.com/fluent/fluentd)
+
+[<img src="https://travis-ci.org/fluent/fluent-plugin-s3.svg?branch=master"
+alt="Build Status" />](https://travis-ci.org/fluent/fluent-plugin-s3) [<img
+src="https://codeclimate.com/github/fluent/fluent-plugin-s3/badges/gpa.svg"
+/>](https://codeclimate.com/github/fluent/fluent-plugin-s3)
+
+## PR
+**it will be greatful that the plugin can push log to  owner S3 Storage like Ceph S3.**
+
+## Overview
+
+**s3** output plugin buffers event logs in local file and upload it to S3
+periodically.
+
+This plugin splits files exactly by using the time of event logs (not the time
+when the logs are received). For example, a log '2011-01-02 message B' is
+reached, and then another log '2011-01-03 message B' is reached in this order,
+the former one is stored in "20110102.gz" file, and latter one in
+"20110103.gz" file.
+
+**s3** input plugin reads data from S3 periodically. This plugin uses
+SQS queue on the region same as S3 bucket.
+We must setup SQS queue and S3 event notification before use this plugin.
+
+## Requirements
+
+| fluent-plugin-s3  | fluentd | ruby |
+|-------------------|---------|------|
+| >= 1.0.0 | >= v0.14.0 | >= 2.1 |
+|  < 1.0.0 | >= v0.12.0 | >= 1.9 |
+
+NOTE: fluent-plugin-s3 v1.0.0 is now RC. We will release stable v1.0.0 soon.
+
+## Installation
+
+Simply use RubyGems:
+
+    $ gem install fluent-plugin-s3 -v "~> 0.8"  --no-document # for fluentd v0.12 or later
+    $ gem install fluent-plugin-s3 -v 1.0.0.rc2 --no-document # for fluentd v0.14 or later
+
+## Output: Configuration
+
+### v0.14 style
+
+With fluentd v0.14 and fluent-plugin-s3 v1.0.0, use new buffer configuration to dynamic parameters.
+
+    <match pattern>
+      @type s3
+
+      aws_key_id YOUR_AWS_KEY_ID
+      aws_sec_key YOUR_AWS_SECRET_KEY
+      s3_bucket YOUR_S3_BUCKET_NAME
+      s3_region ap-northeast-1
+
+      path logs/${tag}/%Y/%m/%d/
+      s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
+
+      # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
+      # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
+      <buffer tag,time>
+        @type file
+        path /var/log/fluent/s3
+        timekey 3600 # 1 hour partition
+        timekey_wait 10m
+        timekey_use_utc true # use utc
+      </buffer>
+      <format>
+        @type json
+      </format>
+    </match>
+
+For `<buffer>`, you can use any record field in `path` / `s3_object_key_format`.
+
+    path logs/${tag}/${foo}
+    <buffer tag,foo>
+      # parameters...
+    </buffer>
+
+See official article for more detail: [Buffer section configurations](http://docs.fluentd.org/v0.14/articles/buffer-section)
+
+Note that this configuration doesn't work with fluentd v0.12.
+
+### v0.12 style
+
+This configuration works with both fluentd v0.12 and v0.14.
+
+    <match pattern>
+      @type s3
+
+      aws_key_id YOUR_AWS_KEY_ID
+      aws_sec_key YOUR_AWS_SECRET_KEY
+      s3_bucket YOUR_S3_BUCKET_NAME
+      s3_region ap-northeast-1
+
+      path logs/
+      s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
+      buffer_path /var/log/fluent/s3
+      time_slice_format %Y%m%d-%H
+      time_slice_wait 10m
+      utc
+      format json
+    </match>
+
+If you want to embed tag in `path` / `s3_object_key_format`, you need to use `fluent-plugin-forest` plugin.
+
+**aws_key_id**
+
+AWS access key id. This parameter is required when your agent is not
+running on EC2 instance with an IAM Role. When using an IAM role, make 
+sure to configure `instance_profile_credentials`. Usage can be found below.
+
+**aws_sec_key**
+
+AWS secret key. This parameter is required when your agent is not running
+on EC2 instance with an IAM Role.
+
+**aws_iam_retries**
+
+The number of attempts to make (with exponential backoff) when loading
+instance profile credentials from the EC2 metadata service using an IAM
+role. Defaults to 5 retries.
+
+**s3_bucket (required)**
+
+S3 bucket name.
+
+**s3_region**
+
+s3 region name. For example, US West (Oregon) Region is "us-west-2". The
+full list of regions are available here. >
+http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
+recommend using `s3_region` instead of `s3_endpoint`.
+
+**s3_endpoint**
+
+endpoint for S3 compatible services. For example, Riak CS based storage or
+something. This option doesn't work on S3, use `s3_region` instead.
+
+**ssl_verify_peer**
+
+Verify SSL certificate of the endpoint. Default is true. Set false when you want to ignore the endpoint SSL certificate.
+
+**s3_object_key_format**
+
+The format of S3 object keys. You can use several built-in variables:
+
+*   %{path}
+*   %{time_slice}
+*   %{index}
+*   %{file_extension}
+*   %{hex_random}
+*   %{uuid_flush}
+*   %{hostname}
+
+to decide keys dynamically.
+
+* %{path} is exactly the value of **path** configured in the configuration file.
+E.g., "logs/" in the example configuration above.
+* %{time_slice} is the
+time-slice in text that are formatted with **time_slice_format**. %{index} is
+the sequential number starts from 0, increments when multiple files are
+uploaded to S3 in the same time slice.
+* %{file_extention} is always "gz" for
+now.
+* %{uuid_flush} a uuid that is replaced everytime the buffer will be flushed. If you want to use this placeholder, install `uuidtools` gem first.
+* %{hostname} is replaced with `Socket.gethostname` result.
+* %{hex_random} a random hex string that is replaced for each buffer chunk, not
+assured to be unique. This is used to follow a way of performance tuning, `Add
+a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance
+Considerations - Amazon Simple Storage
+Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
+You can configure the length of string with a
+`hex_random_length` parameter (Default: 4).
+
+The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
+
+For instance, using the example configuration above, actual object keys on S3
+will be something like:
+
+    "logs/20130111-22_0.gz"
+    "logs/20130111-23_0.gz"
+    "logs/20130111-23_1.gz"
+    "logs/20130112-00_0.gz"
+
+With the configuration:
+
+    s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
+    path log
+    time_slice_format %Y%m%d-%H
+
+You get:
+
+    "log/events/ts=20130111-22/events_0.gz"
+    "log/events/ts=20130111-23/events_0.gz"
+    "log/events/ts=20130111-23/events_1.gz"
+    "log/events/ts=20130112-00/events_0.gz"
+
+NOTE: ${hostname} placeholder is deprecated since v0.8. You can get same result by using [configuration's embedded ruby code feature](http://docs.fluentd.org/articles/config-file#embedded-ruby-code).
+
+    s3_object_key_format %{path}%{time_slice}_%{hostname}%{index}.%{file_extension}
+    s3_object_key_format "%{path}%{time_slice}_#{Socket.gethostname}%{index}.%{file_extension}"
+
+Above two configurations are same. The important point is wrapping `""` is needed for `#{Socket.gethostname}`.
+
+**force_path_style**
+
+:force_path_style (Boolean) — default: false — When set to true, the
+bucket name is always left in the request URI and never moved to the host
+as a sub-domain. See Plugins::S3BucketDns for more details.
+
+**store_as**
+
+archive format on S3. You can use several format:
+
+*   gzip (default)
+*   json
+*   text
+*   lzo (Need lzop command)
+*   lzma2 (Need xz command)
+*   gzip_command (Need gzip command)
+    *   This compressor uses an external gzip command, hence would result in
+        utilizing CPU cores well compared with `gzip`
+
+See `Use your compression algorithm` section for adding another format.
+
+**`<format>` or format**
+
+Change one line format in the S3 object. Supported formats are "out_file",
+"json", "ltsv" and "single_value". See also [official Formatter article](http://docs.fluentd.org/articles/formatter-plugin-overview).
+
+* out_file (default).
+
+        time\ttag\t{..json1..}
+        time\ttag\t{..json2..}
+        ...
+
+* json
+
+        {..json1..}
+        {..json2..}
+        ...
+
+
+At this format, "time" and "tag" are omitted. But you can set these
+information to the record by setting "include_tag_key" / "tag_key" and
+"include_time_key" / "time_key" option. If you set following configuration in
+S3 output:
+
+    format json
+    include_time_key true
+    time_key log_time # default is time
+
+then the record has log_time field.
+
+    {"log_time":"time string",...}
+
+* ltsv
+
+        key1:value1\tkey2:value2
+        key1:value1\tkey2:value2
+        ...
+
+
+"ltsv" format also accepts "include_xxx" related options. See "json" section.
+
+* single_value
+
+
+Use specified value instead of entire recode. If you get '{"message":"my
+log"}', then contents are
+
+    my log1
+    my log2
+    ...
+
+You can change key name by "message_key" option.
+
+**auto_create_bucket**
+
+Create S3 bucket if it does not exists. Default is true.
+
+**check_bucket**
+
+Check mentioned bucket if it exists in AWS or not. Default is true.
+
+When it is false,
+	fluentd will not check aws s3 for the existence of the mentioned bucket. This is the
+	case where bucket will be pre-created before running fluentd.
+
+**check_object**
+
+Check object before creation if it exists or not. Default is true.
+
+When it is false,
+	s3_object_key_format will be %{path}%{date_slice}_%{time_slice}.%{file_extension}
+	where, time_slice will be in hhmmss format, so that each object will be unique.
+	Example object name, assuming it is created on 2016/16/11 3:30:54 PM
+		20161611_153054.txt (extension can be anything as per user's choice)
+
+**Example when check_bucket=false and check_object=false**
+
+When the mentioned configuration will be made, fluentd will work with the
+minimum IAM poilcy, like:
+				"Statement": [{
+					"Effect": "Allow",
+					"Action": "s3:PutObject",
+					"Resource": ["*"]
+				}]
+
+**check_apikey_on_start**
+
+Check AWS key on start. Default is true.
+
+**proxy_uri**
+
+uri of proxy environment.
+
+**path**
+
+path prefix of the files on S3. Default is "" (no prefix).
+
+**buffer_path (required)**
+
+path prefix of the files to buffer logs.
+
+**time_slice_format**
+
+Format of the time used as the file name. Default is '%Y%m%d'. Use
+'%Y%m%d%H' to split files hourly.
+
+**time_slice_wait**
+
+The time to wait old logs. Default is 10 minutes. Specify larger value if
+old logs may reach.
+
+**utc**
+
+Use UTC instead of local time.
+
+**storage_class**
+
+Set storage class. Possible values are `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA` from [Ruby SDK](http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#storage_class-instance_method).
+
+**reduced_redundancy**
+
+Use S3 reduced redundancy storage for 33% cheaper pricing. Default is
+false.
+
+This is deprecated. Use `storage_class REDUCED_REDUNDANCY` instead.
+
+**acl**
+
+Permission for the object in S3. This is useful for cross-account access
+using IAM roles. Valid values are:
+
+*   private (default)
+*   public-read
+*   public-read-write (not recommended - see [Canned
+    ACL](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl))
+*   authenticated-read
+*   bucket-owner-read
+*   bucket-owner-full-control
+
+To use cross-account access, you will need to create a bucket policy granting
+the specific access required. Refer to the [AWS
+documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example3.html) for examples.
+
+**hex_random_length**
+
+The length of `%{hex_random}` placeholder. Default is 4 as written in
+[Request Rate and Performance Considerations - Amazon Simple Storage
+Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
+The maximum length is 16.
+
+**overwrite**
+
+Overwrite already existing path. Default is false, which raises an error
+if a s3 object of the same path already exists, or increment the
+`%{index}` placeholder until finding an absent path.
+
+**use_server_side_encryption**
+
+The Server-side encryption algorithm used when storing this object in S3
+(e.g., AES256, aws:kms)
+
+**ssekms_key_id**
+
+Specifies the AWS KMS key ID to use for object encryption. You have to
+set "aws:kms" to `use_server_side_encryption` to use the KMS encryption.
+
+**sse_customer_algorithm**
+
+Specifies the algorithm to use to when encrypting the object (e.g., AES256).
+
+**sse_customer_key**
+
+Specifies the AWS KMS key ID to use for object encryption.
+
+**sse_customer_key_md5**
+
+Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
+
+**compute_checksums**
+
+AWS SDK uses MD5 for API request/response by default. On FIPS enabled environment,
+OpenSSL returns an error because MD5 is disabled. If you want to use
+this plugin on FIPS enabled environment, set `compute_checksums false`.
+
+**signature_version**
+
+Signature version for API request. `s3` means signature version 2 and
+`v4` means signature version 4. Default is `nil` (Following SDK's default).
+It would be useful when you use S3 compatible storage that accepts only signature version 2.
+
+**warn_for_delay**
+
+Given a threshold to treat events as delay, output warning logs if delayed events were put into s3.
+
+### assume_role_credentials
+
+Typically, you use AssumeRole for cross-account access or federation.
+
+    <match *>
+      @type s3
+
+      <assume_role_credentials>
+        role_arn          ROLE_ARN
+        role_session_name ROLE_SESSION_NAME
+      </assume_role_credentials>
+    </match>
+
+See also:
+
+*   [Using IAM Roles - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
+*   [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
+*   [Aws::AssumeRoleCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html)
+
+**role_arn (required)**
+
+The Amazon Resource Name (ARN) of the role to assume.
+
+**role_session_name (required)**
+
+An identifier for the assumed role session.
+
+**policy**
+
+An IAM policy in JSON format.
+
+**duration_seconds**
+
+The duration, in seconds, of the role session. The value can range from
+900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value
+is set to 3600 seconds.
+
+**external_id**
+
+A unique identifier that is used by third parties when assuming roles in
+their customers' accounts.
+
+### instance_profile_credentials
+
+Retrieve temporary security credentials via HTTP request. This is useful on
+EC2 instance.
+
+    <match *>
+      @type s3
+
+      <instance_profile_credentials>
+        ip_address IP_ADDRESS
+        port       PORT
+      </instance_profile_credentials>
+    </match>
+
+See also:
+
+*   [Aws::InstanceProfileCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html)
+*   [Temporary Security Credentials - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
+*   [Instance Metadata and User Data - Amazon Elastic Compute
+    Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
+
+**retries**
+
+Number of times to retry when retrieving credentials. Default is 5.
+
+**ip_address**
+
+Default is 169.254.169.254.
+
+**port**
+
+Default is 80.
+
+**http_open_timeout**
+
+Default is 5.
+
+**http_read_timeout**
+
+Default is 5.
+
+### shared_credentials
+
+This loads AWS access credentials from local ini file. This is useful for
+local developing.
+
+    <match *>
+      @type s3
+
+      <shared_credentials>
+        path         PATH
+        profile_name PROFILE_NAME
+      </shared_credentials>
+    </match>
+
+See also:
+
+*   [Aws::SharedCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html)
+
+**path**
+
+Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials".
+
+**profile_name**
+
+Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
+
+## Input: Setup
+
+1. Create new [SQS](https://aws.amazon.com/documentation/sqs/) queue (use same region as S3)
+2. Set proper permission to new queue
+3. [Configure S3 event notification](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html)
+4. Write configuration file such as fluent.conf
+5. Run fluentd
+
+## Input: Configuration
+
+    <source>
+      @type s3
+
+      aws_key_id YOUR_AWS_KEY_ID
+      aws_sec_key YOUR_AWS_SECRET_KEY
+      s3_bucket YOUR_S3_BUCKET_NAME
+      s3_region ap-northeast-1
+
+      <sqs>
+        queue_name YOUR_SQS_QUEUE_NAME
+      </sqs>
+    </source>
+
+**aws_key_id**
+
+AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Role.
+
+**aws_sec_key**
+
+AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Role.
+
+**aws_iam_retries**
+
+The number of attempts to make (with exponential backoff) when loading instance profile credentials from the EC2 metadata
+service using an IAM role. Defaults to 5 retries.
+
+**s3_bucket (required)**
+
+S3 bucket name.
+
+**s3_region**
+
+S3 region name. For example, US West (Oregon) Region is
+"us-west-2". The full list of regions are available here. >
+http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
+recommend using `s3_region` instead of `s3_endpoint`.
+
+**store_as**
+
+archive format on S3. You can use serveral format:
+
+* gzip (default)
+* json
+* text
+* lzo (Need lzop command)
+* lzma2 (Need xz command)
+* gzip_command (Need gzip command)
+  * This compressor uses an external gzip command, hence would result in utilizing CPU cores well compared with `gzip`
+
+See 'Use your compression algorithm' section for adding another format.
+
+**format**
+
+Parse a line as this format in the S3 object. Supported formats are
+"apache_error", "apache2", "syslog", "json", "tsv", "ltsv", "csv",
+"nginx" and "none".
+
+**check_apikey_on_start**
+
+Check AWS key on start. Default is true.
+
+**proxy_uri**
+
+URI of proxy environment.
+
+**sqs/queue_name (required)**
+
+SQS queue name. Need to create SQS queue on the region same as S3 bucket.
+
+**sqs/skip_delete**
+
+When true, messages are not deleted after polling block. Default is false.
+
+**sqs/wait_time_seconds**
+
+The long polling interval. Default is 20.
+
+## IAM Policy
+
+The following is an example for a minimal IAM policy needed to write to an s3
+bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
+
+    { "Statement": [
+     { "Effect":"Allow",
+       "Action":"s3:*",
+       "Resource":"arn:aws:s3:::my-s3bucket*"
+      } ]
+    }
+
+Note that the bucket must already exist and **auto_create_bucket** has no
+effect in this case.
+
+Refer to the [AWS
+documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolicies.html) for example policies.
+
+Using [IAM
+roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
+with a properly configured IAM policy are preferred over embedding access keys
+on EC2 instances.
+
+## Use your (de)compression algorithm
+
+s3 plugin has plugabble compression mechanizm like Fleuntd's input / output
+plugin. If you set 'store_as xxx', `out_s3` plugin searches
+`fluent/plugin/s3_compressor_xxx.rb` and `in_s3` plugin searches
+`fluent/plugin/s3_extractor_xxx.rb`. You can define your (de)compression with
+'S3Output::Compressor'/`S3Input::Extractor` classes. Compressor API is here:
+
+    module Fluent # Since fluent-plugin-s3 v1.0.0 or later, use Fluent::Plugin instead of Fluent
+      class S3Output
+        class XXXCompressor < Compressor
+          S3Output.register_compressor('xxx', self)
+
+          # Used to file extension
+          def ext
+            'xxx'
+          end
+
+          # Used to file content type
+          def content_type
+            'application/x-xxx'
+          end
+
+          # chunk is buffer chunk. tmp is destination file for upload
+          def compress(chunk, tmp)
+            # call command or something
+          end
+        end
+      end
+    end
+
+`Extractor` is similar to `Compressor`
+See bundled `Compressor`/`Extractor` classes for more detail.
+
+## Website, license, et. al.
+
+| Web site          | http://fluentd.org/                       |
+|-------------------|-------------------------------------------|
+| Documents         | http://docs.fluentd.org/                  |
+| Source repository | http://github.com/fluent/fluent-plugin-s3 |
+| Discussion        | http://groups.google.com/group/fluentd    |
+| Author            | Sadayuki Furuhashi                        |
+| Copyright         | (c) 2011 FURUHASHI Sadayuki               |
+| License           | Apache License, Version 2.0               |