fluent-plugin-s3 0.6.0 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9a324977d9062b01736950c76f666d42161e93d6
-  data.tar.gz: 6d70b12fabf887f6ac991798ac4e5a0200c5afe5
+  metadata.gz: fd2f564114dd0e1f90168baa3f8ed983a9eba33d
+  data.tar.gz: 259cb9cbe6213eeaf6e2734706eaf0701cb17812
 SHA512:
-  metadata.gz: 23a7cb16fca1b7dbe7f7c785d3096d8670c1bc7af033685e57ebfe7b8bcecc225bd3ad55abb72e71514d983e7319bea13629df0f1f51638fffffd60ac23c2e27
-  data.tar.gz: 30b2f703798b58b3b866b1f3fe1bdcf64081c4c73eede9d570a631d4c297a7be662267c59bbd46a17c286047a00481898493e8bf1cefed2acf2c15c3fc55b00f
+  metadata.gz: 7437b11fcd3ac8ce9ce0aabc34b2959eee14f003ba5c54ec3e5feed94658f18094811490c2505e548c0f9ea00136b333a0f86a5c914729a1e9d31a85ed0535e6
+  data.tar.gz: 3037249250cfed2cbe07343f826097ecdfabf33a3e3ebb9dac7a10c574b8224f4fefeceb1ae5bde87409b74d87da84355559eb93fcd91d1a52539868ba0c5589

data/.gitignore

@@ -9,3 +9,5 @@
 Gemfile.lock
 vendor
 .ruby-version
+
+test/tmp/

data/ChangeLog

@@ -1,3 +1,10 @@
+Release 0.6.1 - 2015/10/30
+
+  * Fix server_side_encryption error
+  * Keep hex random identity on rebooting
+  * Fix Tempfile handling on windows
+
+
 Release 0.6.0 - 2015/10/09
 
   * Allow path based calling format

data/README.md

@@ -0,0 +1,465 @@
+# Amazon S3 output plugin for [Fluentd](http://github.com/fluent/fluentd)
+
+[<img src="https://travis-ci.org/fluent/fluent-plugin-s3.svg?branch=master"
+alt="Build Status" />](https://travis-ci.org/fluent/fluent-plugin-s3) [<img
+src="https://codeclimate.com/github/fluent/fluent-plugin-s3/badges/gpa.svg"
+/>](https://codeclimate.com/github/fluent/fluent-plugin-s3)
+
+## Overview
+
+**s3** output plugin buffers event logs in local file and upload it to S3
+periodically.
+
+This plugin splits files exactly by using the time of event logs (not the time
+when the logs are received). For example, a log '2011-01-02 message B' is
+reached, and then another log '2011-01-03 message B' is reached in this order,
+the former one is stored in "20110102.gz" file, and latter one in
+"20110103.gz" file.
+
+## Installation
+
+Simply use RubyGems:
+
+    gem install fluent-plugin-s3
+
+## Configuration
+
+    <match pattern>
+      type s3
+
+      aws_key_id YOUR_AWS_KEY_ID
+      aws_sec_key YOUR_AWS_SECRET_KEY
+      s3_bucket YOUR_S3_BUCKET_NAME
+      s3_region ap-northeast-1
+      s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
+      path logs/
+      buffer_path /var/log/fluent/s3
+
+      time_slice_format %Y%m%d-%H
+      time_slice_wait 10m
+      utc
+    </match>
+
+**aws_key_id**
+
+AWS access key id. This parameter is required when your agent is not
+running on EC2 instance with an IAM Role.
+
+**aws_sec_key**
+
+AWS secret key. This parameter is required when your agent is not running
+on EC2 instance with an IAM Role.
+
+**aws_iam_retries**
+
+The number of attempts to make (with exponential backoff) when loading
+instance profile credentials from the EC2 metadata service using an IAM
+role. Defaults to 5 retries.
+
+**s3_bucket (required)**
+
+S3 bucket name.
+
+**s3_region**
+
+s3 region name. For example, US West (Oregon) Region is "us-west-2". The
+full list of regions are available here. >
+http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
+recommend using `s3_region` instead of `s3_endpoint`.
+
+**s3_endpoint**
+
+endpoint for S3 compatible services. For example, Riak CS based storage or
+something. This option doesn't work on S3, use `s3_region` instead.
+
+**s3_object_key_format**
+
+The format of S3 object keys. You can use several built-in variables:
+
+*   %{path}
+*   %{time_slice}
+*   %{index}
+*   %{file_extension}
+*   %{uuid_flush}
+*   %{hex_random}
+
+to decide keys dynamically.
+
+* %{path} is exactly the value of **path** configured in the configuration file.
+E.g., "logs/" in the example configuration above.
+* %{time_slice} is the
+time-slice in text that are formatted with **time_slice_format**. %{index} is
+the sequential number starts from 0, increments when multiple files are
+uploaded to S3 in the same time slice.
+* %{file_extention} is always "gz" for
+now.
+* %{uuid_flush} a uuid that is replaced everytime the buffer will be flushed
+* %{hex_random} a random hex string that is replaced for each buffer chunk, not
+assured to be unique. This is used to follow a way of peformance tuning, `Add
+a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance
+Considerations - Amazon Simple Storage
+Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-con
+siderations.html). You can configure the length of string with a
+`hex_random_length` parameter (Default: 4).
+
+The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
+
+For instance, using the example configuration above, actual object keys on S3
+will be something like:
+
+    "logs/20130111-22_0.gz"
+    "logs/20130111-23_0.gz"
+    "logs/20130111-23_1.gz"
+    "logs/20130112-00_0.gz"
+
+With the configuration:
+
+    s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
+    path log
+    time_slice_format %Y%m%d-%H
+
+You get:
+
+    "log/events/ts=20130111-22/events_0.gz"
+    "log/events/ts=20130111-23/events_0.gz"
+    "log/events/ts=20130111-23/events_1.gz"
+    "log/events/ts=20130112-00/events_0.gz"
+
+The
+[fluent-mixin-config-placeholders](https://github.com/tagomoris/fluent-mixin-c
+onfig-placeholders) mixin is also incorporated, so additional variables such
+as %{hostname}, %{uuid}, etc. can be used in the s3_object_key_format. This
+could prove useful in preventing filename conflicts when writing from multiple
+servers.
+
+    s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
+
+**force_path_style**
+
+:force_path_style (Boolean) — default: false — When set to true, the
+bucket name is always left in the request URI and never moved to the host
+as a sub-domain. See Plugins::S3BucketDns for more details.
+
+**store_as**
+
+archive format on S3. You can use serveral format:
+
+*   gzip (default)
+*   json
+*   text
+*   lzo (Need lzop command)
+*   lzma2 (Need xz command)
+*   gzip_command (Need gzip command)
+    *   This compressor uses an external gzip command, hence would result in
+        utilizing CPU cores well compared with `gzip`
+
+See `Use your compression algorithm` section for adding another format.
+
+**format**
+
+Change one line format in the S3 object. Supported formats are "out_file",
+"json", "ltsv" and "single_value".
+
+* out_file (default).
+
+        time\ttag\t{..json1..}
+        time\ttag\t{..json2..}
+        ...
+
+* json
+
+        {..json1..}
+        {..json2..}
+        ...
+
+
+At this format, "time" and "tag" are omitted. But you can set these
+information to the record by setting "include_tag_key" / "tag_key" and
+"include_time_key" / "time_key" option. If you set following configuration in
+S3 output:
+
+    format json
+    include_time_key true
+    time_key log_time # default is time
+
+then the record has log_time field.
+
+    {"log_time":"time string",...}
+
+* ltsv
+
+        key1:value1\tkey2:value2
+        key1:value1\tkey2:value2
+        ...
+
+
+"ltsv" format also accepts "include_xxx" related options. See "json" section.
+
+* single_value
+
+
+Use specified value instead of entire recode. If you get '{"message":"my
+log"}', then contents are
+
+    my log1
+    my log2
+    ...
+
+You can change key name by "message_key" option.
+
+**auto_create_bucket**
+
+Create S3 bucket if it does not exists. Default is true.
+
+**check_apikey_on_start**
+
+Check AWS key on start. Default is true.
+
+**proxy_uri**
+
+uri of proxy environment.
+
+**path**
+
+path prefix of the files on S3. Default is "" (no prefix).
+
+**buffer_path (required)**
+
+path prefix of the files to buffer logs.
+
+**time_slice_format**
+
+Format of the time used as the file name. Default is '%Y%m%d'. Use
+'%Y%m%d%H' to split files hourly.
+
+**time_slice_wait**
+
+The time to wait old logs. Default is 10 minutes. Specify larger value if
+old logs may reache.
+
+**utc**
+
+Use UTC instead of local time.
+
+**reduced_redundancy**
+
+Use S3 reduced redundancy storage for 33% cheaper pricing. Default is
+false.
+
+**acl**
+
+Permission for the object in S3. This is useful for cross-account access
+using IAM roles. Valid values are:
+
+*   private (default)
+*   public_read
+*   public_read_write (not recommended - see [Canned
+    ACL](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#cann
+    ed-acl))
+*   authenticated_read
+*   bucket_owner_read
+*   bucket_owner_full_control
+
+To use cross-account access, you will need to create a bucket policy granting
+the specific access required. Refer to the [AWS
+documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthro
+ughs-managing-access-example3.html) for examples.
+
+**hex_random_length**
+
+The length of `%{hex_random}` placeholder. Default is 4 as written in
+[Request Rate and Performance Considerations - Amazon Simple Storage
+Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
+The maximum length is 16.
+
+**overwrite**
+
+Overwrite already existing path. Default is false, which raises an error
+if a s3 object of the same path already exists, or increment the
+`%{index}` placeholder until finding an absent path.
+
+### assume_role_credentials
+
+Typically, you use AssumeRole for cross-account access or federation.
+
+    <match *>
+      type s3
+
+      <assume_role_credentials>
+        role_arn          ROLE_ARN
+        role_session_name ROLE_SESSION_NAME
+      </assume_role_credentials>
+    </match>
+
+See also:
+
+*   [Using IAM Roles - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.h
+    tml)
+*   [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Clien
+    t.html)
+*   [Aws::AssumeRoleCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws
+    /AssumeRoleCredentials.html)
+
+**role_arn (required)**
+
+The Amazon Resource Name (ARN) of the role to assume.
+
+**role_session_name (required)**
+
+An identifier for the assumed role session.
+
+**policy**
+
+An IAM policy in JSON format.
+
+**duration_seconds**
+
+The duration, in seconds, of the role session. The value can range from
+900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value
+is set to 3600 seconds.
+
+**external_id**
+
+A unique identifier that is used by third parties when assuming roles in
+their customers' accounts.
+
+### instance_profile_credentials
+
+Retrieve temporary security credentials via HTTP request. This is useful on
+EC2 instance.
+
+    <match *>
+      type s3
+
+      <instance_profile_credentials>
+        ip_address IP_ADDRESS
+        port       PORT
+      </instance_profile_credentials>
+    </match>
+
+See also:
+
+*   [Aws::InstanceProfileCredentials](http://docs.aws.amazon.com/sdkforruby/ap
+    i/Aws/InstanceProfileCredentials.html)
+*   [Temporary Security Credentials - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials
+    _temp.html)
+*   [Instance Metadata and User Data - Amazon Elastic Compute
+    Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-met
+    adata.html)
+
+**retries**
+
+Number of times to retry when retrieving credentials. Default is 5.
+
+**ip_address**
+
+Default is 169.254.169.254.
+
+**port**
+
+Default is 80.
+
+**http_open_timeout**
+
+Default is 5.
+
+**http_read_timeout**
+
+Default is 5.
+
+### shared_credentials
+
+This loads AWS access credentials from local ini file. This is useful for
+local developing.
+
+    <match *>
+      type s3
+
+      <shared_credentials>
+        path         PATH
+        profile_name PROFILE_NAME
+      </shared_credentials>
+    </match>
+
+See also:
+
+*   [Aws::SharedCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/Sha
+    redCredentials.html)
+
+**path**
+
+Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials".
+
+**profile_name**
+
+Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
+
+
+## IAM Policy
+
+The following is an example for a minimal IAM policy needed to write to an s3
+bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
+
+    { "Statement": [
+     { "Effect":"Allow",
+       "Action":"s3:*",
+       "Resource":"arn:aws:s3:::my-s3bucket*"
+      } ]
+    }
+
+Note that the bucket must already exist and **auto_create_bucket** has no
+effect in this case.
+
+Refer to the [AWS
+documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolic
+ies.html) for example policies.
+
+Using [IAM
+roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
+with a properly configured IAM policy are preferred over embedding access keys
+on EC2 instances.
+
+## Use your compression algorithm
+
+s3 plugin has plugabble compression mechanizm like Fleuntd's input / output
+plugin. If you set 'store_as xxx', s3 plugin searches
+`fluent/plugin/s3_compressor_xxx.rb`. You can define your compression with
+'S3Output::Compressor' class. Compressor API is here:
+
+    module Fluent
+      class S3Output
+        class XXXCompressor < Compressor
+          S3Output.register_compressor('xxx', self)
+
+          # Used to file extension
+          def ext
+            'xxx'
+          end
+
+          # Used to file content type
+          def content_type
+            'application/x-xxx'
+          end
+
+          # chunk is buffer chunk. tmp is destination file for upload
+          def compress(chunk, tmp)
+            # call command or something
+          end
+        end
+      end
+    end
+
+See bundled Compressor classes for more detail.
+
+## Website, license, et. al.
+
+| Web site          | http://fluentd.org/                       |
+|-------------------|-------------------------------------------|
+| Documents         | http://docs.fluentd.org/                  |
+| Source repository | http://github.com/fluent/fluent-plugin-s3 |
+| Discussion        | http://groups.google.com/group/fluentd    |
+| Author            | Sadayuki Furuhashi                        |
+| Copyright         | (c) 2011 FURUHASHI Sadayuki               |
+| License           | Apache License, Version 2.0               |

data/VERSION

@@ -1 +1 @@
-0.6.0
+0.6.1

data/appveyor.yml

@@ -0,0 +1,25 @@
+version: '{build}'
+
+install:
+  - SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
+  - "%devkit%\\devkitvars.bat"
+  - ruby --version
+  - gem --version
+  - bundle install
+build: off
+test_script:
+  - bundle exec rake test TESTOPTS=-v
+
+environment:
+  matrix:
+    - ruby_version: "22-x64"
+      devkit: C:\Ruby21-x64\DevKit
+    - ruby_version: "22"
+      devkit: C:\Ruby21\DevKit
+    - ruby_version: "21-x64"
+      devkit: C:\Ruby21-x64\DevKit
+    - ruby_version: "21"
+      devkit: C:\Ruby21\DevKit
+matrix:
+  allow_failures:
+    - ruby_version: "21"

data/lib/fluent/plugin/out_s3.rb

@@ -59,6 +59,7 @@ module Fluent
     attr_reader :bucket
 
     include Fluent::Mixin::ConfigPlaceholders
+    MAX_HEX_RANDOM_LENGTH = 16
 
     def placeholders
       [:percent]
@@ -92,6 +93,10 @@ module Fluent
         }
       end
 
+      if @hex_random_length > MAX_HEX_RANDOM_LENGTH
+        raise ConfigError, "hex_random_length parameter must be less than or equal to #{MAX_HEX_RANDOM_LENGTH}"
+      end
+
       @storage_class = "REDUCED_REDUNDANCY" if @reduced_redundancy
       @values_for_s3_object_chunk = {}
     end
@@ -102,7 +107,6 @@ module Fluent
       options[:region] = @s3_region if @s3_region
       options[:endpoint] = @s3_endpoint if @s3_endpoint
       options[:http_proxy] = @proxy_uri if @proxy_uri
-      options[:s3_server_side_encryption] = @use_server_side_encryption.to_sym if @use_server_side_encryption
       options[:force_path_style] = @force_path_style
 
       s3_client = Aws::S3::Client.new(options)
@@ -127,16 +131,16 @@ module Fluent
       begin
         path = @path_slicer.call(@path)
 
-        @values_for_s3_object_chunk[chunk.key] ||= {
-          "hex_random" => hex_random,
-          "uuid_flush" => uuid_random,
+        @values_for_s3_object_chunk[chunk.unique_id] ||= {
+          "hex_random" => hex_random(chunk),
         }
         values_for_s3_object_key = {
           "path" => path,
           "time_slice" => chunk.key,
           "file_extension" => @compressor.ext,
           "index" => i,
-        }.merge!(@values_for_s3_object_chunk[chunk.key])
+          "uuid_flush" => uuid_random,
+        }.merge!(@values_for_s3_object_chunk[chunk.unique_id])
 
         s3path = @s3_object_key_format.gsub(%r(%{[^}]+})) { |expr|
           values_for_s3_object_key[expr[2...expr.size-1]]
@@ -155,23 +159,43 @@ module Fluent
       end while @bucket.object(s3path).exists?
 
       tmp = Tempfile.new("s3-")
+      tmp.binmode
       begin
         @compressor.compress(chunk, tmp)
         tmp.rewind
-        log.debug { "out_s3: trying to write {object_id:#{chunk.object_id},time_slice:#{chunk.key}} to s3://#{@s3_bucket}/#{s3path}" }
-        @bucket.object(s3path).put(:body => tmp,
-                                   :content_type => @compressor.content_type,
-                                   :storage_class => @storage_class)
+        log.debug { "out_s3: write chunk: {key:#{chunk.key},tsuffix:#{tsuffix(chunk)}} to s3://#{@s3_bucket}/#{s3path}" }
+
+        put_options = {:body => tmp, :content_type => @compressor.content_type, :storage_class => @storage_class}
+        put_options[:server_side_encryption] = @use_server_side_encryption if @use_server_side_encryption
+        @bucket.object(s3path).put(put_options)
+
+        @values_for_s3_object_chunk.delete(chunk.unique_id)
       ensure
-        @values_for_s3_object_chunk.delete(chunk.key)
         tmp.close(true) rescue nil
       end
     end
 
     private
 
-    def hex_random
-      SecureRandom.hex(@hex_random_n)[0...@hex_random_length]
+    # tsuffix is the one which file buffer filename has
+    def tsuffix(chunk)
+      if chunk.is_a?(Fluent::FileBufferChunk)
+        unique_id = chunk.unique_id
+        tsuffix = unique_id[0...(unique_id.size/2)].unpack('C*').map {|x| x.to_s(16) }.join('') # size: 16
+      else
+        nil
+      end
+    end
+
+    def hex_random(chunk)
+      if chunk.is_a?(Fluent::FileBufferChunk)
+        # let me use tsuffix because its value is kept on retrying even after rebooting
+        tsuffix = tsuffix(chunk)
+        tsuffix.reverse! # tsuffix is like (time_sec, time_usec, rand) => reversing gives more randomness
+        tsuffix[0...@hex_random_length]
+      else
+        SecureRandom.hex(@hex_random_n)[0...@hex_random_length]
+      end
     end
 
     def ensure_bucket

data/lib/fluent/plugin/s3_compressor_gzip_command.rb

@@ -24,6 +24,7 @@ module Fluent
                  chunk.path
                else
                  w = Tempfile.new("chunk-gzip-tmp")
+                 w.binmode
                  chunk.write_to(w)
                  w.close
                  w.path

data/lib/fluent/plugin/s3_compressor_lzma2.rb

@@ -20,6 +20,7 @@ module Fluent
 
       def compress(chunk, tmp)
         w = Tempfile.new("chunk-xz-tmp")
+        w.binmode
         chunk.write_to(w)
         w.close
 

data/lib/fluent/plugin/s3_compressor_lzo.rb

@@ -20,6 +20,7 @@ module Fluent
 
       def compress(chunk, tmp)
         w = Tempfile.new("chunk-tmp")
+        w.binmode
         chunk.write_to(w)
         w.close
 

data/test/test_out_s3.rb

@@ -11,6 +11,10 @@ class S3OutputTest < Test::Unit::TestCase
     Fluent::Test.setup
   end
 
+  def teardown
+    Dir.glob('test/tmp/*').each {|file| FileUtils.rm_f(file) }
+  end
+
   CONFIG = %[
     aws_key_id test_key_id
     aws_sec_key test_sec_key
@@ -95,6 +99,16 @@ class S3OutputTest < Test::Unit::TestCase
     assert d.instance.force_path_style
   end
 
+  def test_configure_with_hex_random_length
+    conf = CONFIG.clone
+    assert_raise Fluent::ConfigError do
+      create_driver(conf + "\nhex_random_length 17\n")
+    end
+    assert_nothing_raised do
+      create_driver(conf + "\nhex_random_length 16\n")
+    end
+  end
+
   def test_path_slicing
     config = CONFIG.clone.gsub(/path\slog/, "path log/%Y/%m/%d")
     d = create_driver(config)
@@ -323,17 +337,36 @@ class S3OutputTest < Test::Unit::TestCase
                    data
     end
     FileUtils.rm_f(s3_test_file_path)
+    Dir.glob('tmp/*').each {|file| FileUtils.rm_f(file) }
+  end
+
+  def test_write_with_custom_s3_object_key_format_containing_hex_random_placeholder_memory_buffer
+    hex = "012345"
+    mock(SecureRandom).hex(3) { hex }
+
+    config = CONFIG_TIME_SLICE.gsub(/%{hostname}/,"%{hex_random}") << "\nhex_random_length 5"
+    write_with_custom_s3_object_key_format_containing_hex_random_placeholder(config, hex[0...5])
+  end
+
+  def test_write_with_custom_s3_object_key_format_containing_hex_random_placeholder_file_buffer
+    tsuffix = "5226c3c4fb3d49b1"
+    any_instance_of(Fluent::FileBufferChunk) do |klass|
+      unique_id = "R&\xC3\xC4\xFB=I\xB1R&\xC3\xC4\xFB=I\xB1" # corresponding unique_id with tsuffxi
+      stub(klass).unique_id { unique_id }
+    end
+    hex = tsuffix.reverse
+
+    config = CONFIG_TIME_SLICE.gsub(/%{hostname}/,"%{hex_random}") << "\nhex_random_length 16"
+    config = config.gsub(/buffer_type memory/, "buffer_type file\nbuffer_path test/tmp/buf")
+    write_with_custom_s3_object_key_format_containing_hex_random_placeholder(config, hex)
   end
 
   # ToDo: need to test hex_random does not change on retry, but it is difficult with
   # the current fluentd test helper because it does not provide a way to run with the same chunks
-  def test_write_with_custom_s3_object_key_format_containing_hex_random_placeholder
+  def write_with_custom_s3_object_key_format_containing_hex_random_placeholder(config, hex)
     # Partial mock the S3Bucket, not to make an actual connection to Amazon S3
     setup_mocks(true)
 
-    hex = "012345"
-    mock(SecureRandom).hex(3) { hex }
-
     # Assert content of event logs which are being sent to S3
     s3obj = stub(Aws::S3::Object.new(:bucket_name => "test_bucket",
                                      :key => "test",
@@ -345,11 +378,8 @@ class S3OutputTest < Test::Unit::TestCase
     s3obj.put(:body => tempfile,
               :content_type => "application/x-gzip",
               :storage_class => "STANDARD")
-    @s3_bucket.object("log/events/ts=20110102-13/events_0-#{hex[0...5]}.gz") { s3obj }
+    @s3_bucket.object("log/events/ts=20110102-13/events_0-#{hex}.gz") { s3obj }
 
-    # We must use TimeSlicedOutputTestDriver instead of BufferedOutputTestDriver,
-    # to make assertions on chunks' keys
-    config = CONFIG_TIME_SLICE.gsub(/%{hostname}/,"%{hex_random}") << "\nhex_random_length 5"
     d = create_time_sliced_driver(config)
 
     time = Time.parse("2011-01-02 13:14:15 UTC").to_i

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-s3
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.6.1
 platform: ruby
 authors:
 - Sadayuki Furuhashi
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-10-09 00:00:00.000000000 Z
+date: 2015-10-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -126,9 +126,10 @@ files:
 - AUTHORS
 - ChangeLog
 - Gemfile
-- README.rdoc
+- README.md
 - Rakefile
 - VERSION
+- appveyor.yml
 - fluent-plugin-s3.gemspec
 - lib/fluent/plugin/out_s3.rb
 - lib/fluent/plugin/s3_compressor_gzip_command.rb
@@ -155,7 +156,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.3
+rubygems_version: 2.4.5.1
 signing_key: 
 specification_version: 4
 summary: Amazon S3 output plugin for Fluentd event collector

data/README.rdoc

@@ -1,319 +0,0 @@
-= Amazon S3 output plugin for {Fluentd}[http://github.com/fluent/fluentd]
-
-{<img src="https://travis-ci.org/fluent/fluent-plugin-s3.svg?branch=master" alt="Build Status" />}[https://travis-ci.org/fluent/fluent-plugin-s3] {<img src="https://codeclimate.com/github/fluent/fluent-plugin-s3/badges/gpa.svg" />}[https://codeclimate.com/github/fluent/fluent-plugin-s3]
-
-== Overview
-
-*s3* output plugin buffers event logs in local file and upload it to S3 periodically.
-
-This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log '2011-01-02 message B' is reached, and then another log '2011-01-03 message B' is reached in this order, the former one is stored in "20110102.gz" file, and latter one in "20110103.gz" file.
-
-
-== Installation
-
-Simply use RubyGems:
-
-    gem install fluent-plugin-s3
-
-== Configuration
-
-    <match pattern>
-      type s3
-
-      aws_key_id YOUR_AWS_KEY_ID
-      aws_sec_key YOUR_AWS_SECRET_KEY
-      s3_bucket YOUR_S3_BUCKET_NAME
-      s3_region ap-northeast-1
-      s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
-      path logs/
-      buffer_path /var/log/fluent/s3
-
-      time_slice_format %Y%m%d-%H
-      time_slice_wait 10m
-      utc
-    </match>
-
-[aws_key_id] AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Role.
-
-[aws_sec_key] AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Role.
-
-[aws_iam_retries] The number of attempts to make (with exponential backoff) when loading instance profile credentials from the EC2 metadata service using an IAM role. Defaults to 5 retries.
-
-[s3_bucket (required)] S3 bucket name.
-
-[s3_region] s3 region name. For example, US West (Oregon) Region is "us-west-2". The full list of regions are available here. > http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We recommend using `s3_region` instead of `s3_endpoint`.
-
-[s3_endpoint] endpoint for S3 compatible services. For example, Riak CS based storage or something. This option doesn't work on S3, use `s3_region` instead.
-
-[s3_object_key_format] The format of S3 object keys. You can use several built-in variables:
-
-- %{path}
-- %{time_slice}
-- %{index}
-- %{file_extension}
-- %{uuid_flush}
-- %{hex_random}
-
-to decide keys dynamically.
-
-%{path} is exactly the value of *path* configured in the configuration file. E.g., "logs/" in the example configuration above.
-%{time_slice} is the time-slice in text that are formatted with *time_slice_format*.
-%{index} is the sequential number starts from 0, increments when multiple files are uploaded to S3 in the same time slice.
-%{file_extention} is always "gz" for now.
-%{uuid_flush} a uuid that is replaced for each buffer chunk to be flushed
-%{hex_random} a random hex string that is replaced for each buffer chunk, not assured to be unique. This is used to follow a way of peformance tuning, `Add a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance Considerations - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html). You can configure the length of string with a `hex_random_length` parameter (Default: 4).
-
-The default format is "%{path}%{time_slice}_%{index}.%{file_extension}".
-
-For instance, using the example configuration above, actual object keys on S3 will be something like:
-
-    "logs/20130111-22_0.gz"
-    "logs/20130111-23_0.gz"
-    "logs/20130111-23_1.gz"
-    "logs/20130112-00_0.gz"
-
-With the configuration:
-
-    s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
-    path log
-    time_slice_format %Y%m%d-%H
-
-You get:
-
-    "log/events/ts=20130111-22/events_0.gz"
-    "log/events/ts=20130111-23/events_0.gz"
-    "log/events/ts=20130111-23/events_1.gz"
-    "log/events/ts=20130112-00/events_0.gz"
-
-The {fluent-mixin-config-placeholders}[https://github.com/tagomoris/fluent-mixin-config-placeholders] mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the s3_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.
-
-    s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
-
-[force_path_style] :force_path_style (Boolean) — default: false — When set to true, the bucket name is always left in the request URI and never moved to the host as a sub-domain. See Plugins::S3BucketDns for more details.
-
-[store_as] archive format on S3. You can use serveral format:
-
-- gzip (default)
-- json
-- text
-- lzo (Need lzop command)
-- lzma2 (Need xz command)
-- gzip_command (Need gzip command)
-  - This compressor uses an external gzip command, hence would result in utilizing CPU cores well compared with `gzip`
-
-See 'Use your compression algorithm' section for adding another format.
-
-[format] Change one line format in the S3 object. Supported formats are "out_file", "json", "ltsv" and "single_value".
-
-- out_file (default).
-
-    time\ttag\t{..json1..}
-    time\ttag\t{..json2..}
-    ...
-
-- json
-
-    {..json1..}
-    {..json2..}
-    ...
-
-At this format, "time" and "tag" are omitted.
-But you can set these information to the record by setting "include_tag_key" / "tag_key" and "include_time_key" / "time_key" option.
-If you set following configuration in S3 output:
-
-    format json
-    include_time_key true
-    time_key log_time # default is time
-
-then the record has log_time field.
-
-    {"log_time":"time string",...}
-
-- ltsv
-
-    key1:value1\tkey2:value2
-    key1:value1\tkey2:value2
-    ...
-
-"ltsv" format also accepts "include_xxx" related options. See "json" section.
-
-- single_value
-
-Use specified value instead of entire recode. If you get '{"message":"my log"}', then contents are
-
-    my log1
-    my log2
-    ...
-
-You can change key name by "message_key" option.
-
-[auto_create_bucket] Create S3 bucket if it does not exists. Default is true.
-
-[check_apikey_on_start] Check AWS key on start. Default is true.
-
-[proxy_uri] uri of proxy environment.
-
-[path] path prefix of the files on S3. Default is "" (no prefix).
-
-[buffer_path (required)] path prefix of the files to buffer logs.
-
-[time_slice_format] Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.
-
-[time_slice_wait] The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
-
-[utc] Use UTC instead of local time.
-
-[reduced_redundancy] Use S3 reduced redundancy storage for 33% cheaper pricing. Default is false.
-
-[acl] Permission for the object in S3. This is useful for cross-account access using IAM roles. Valid values are:
-
-- private (default)
-- public_read
-- public_read_write (not recommended - see {Canned ACL}[http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl])
-- authenticated_read
-- bucket_owner_read
-- bucket_owner_full_control
-
-To use cross-account access, you will need to create a bucket policy granting
-the specific access required. Refer to the {AWS documentation}[http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example3.html] for examples.
-
-[hex_random_length] The length of `%{hex_random}` placeholder. Default is 4 as written in [Request Rate and Performance Considerations - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).
-
-[overwrite] Overwrite already existing path. Default is false, which raises an error if a s3 object of the same path already exists, or increment the `%{index}` placeholder until finding an absent path.
-
-=== assume_role_credentials
-
-Typically, you use AssumeRole for cross-account access or federation.
-
-    <match *>
-      type s3
-
-      <assume_role_credentials>
-        role_arn          ROLE_ARN
-        role_session_name ROLE_SESSION_NAME
-      </assume_role_credentials>
-    </match>
-
-See also:
-
-- {Using IAM Roles - AWS Identity and Access Management}[http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html]
-- {Aws::STS::Client}[http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html]
-- {Aws::AssumeRoleCredentials}[http://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html]
-
-[role_arn (required)] The Amazon Resource Name (ARN) of the role to assume.
-
-[role_session_name (required)] An identifier for the assumed role session.
-
-[policy] An IAM policy in JSON format.
-
-[duration_seconds] The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value is set to 3600 seconds.
-
-[external_id] A unique identifier that is used by third parties when assuming roles in their customers' accounts.
-
-=== instance_profile_credentials
-
-Retrieve temporary security credentials via HTTP request. This is useful on EC2 instance.
-
-    <match *>
-      type s3
-
-      <instance_profile_credentials>
-        ip_address IP_ADDRESS
-        port       PORT
-      </instance_profile_credentials>
-    </match>
-
-See also:
-
-- {Aws::InstanceProfileCredentials}[http://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html]
-- {Temporary Security Credentials - AWS Identity and Access Management}[http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html]
-- {Instance Metadata and User Data - Amazon Elastic Compute Cloud}[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html]
-
-[retries] Number of times to retry when retrieving credentials. Default is 5.
-
-[ip_address] Default is 169.254.169.254.
-
-[port] Default is 80.
-
-[http_open_timeout] Default is 5.
-
-[http_read_timeout] Default is 5.
-
-=== shared_credentials
-
-This loads AWS access credentials from local ini file. This is useful for local developing.
-
-    <match *>
-      type s3
-
-      <shared_credentials>
-        path         PATH
-        profile_name PROFILE_NAME
-      </shared_credentials>
-    </match>
-
-See also:
-
-- {Aws::SharedCredentials}[http://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html]
-
-[path] Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials".
-
-[profile_name] Defaults to 'default' or `ENV['AWS_PROFILE']`.
-
-== IAM Policy
-
-The following is an example for a minimal IAM policy needed to write to an s3 bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
-
-    { "Statement": [
-     { "Effect":"Allow",
-       "Action":"s3:*",
-       "Resource":"arn:aws:s3:::my-s3bucket*"
-      } ]
-    }
-
-Note that the bucket must already exist and *auto_create_bucket* has no effect in this case.
-
-Refer to the {AWS documentation}[http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolicies.html] for example policies.
-
-Using {IAM roles}[http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html] with a properly configured IAM policy are preferred over embedding access keys on EC2 instances.
-
-== Use your compression algorithm
-
-s3 plugin has plugabble compression mechanizm like Fleuntd\'s input / output plugin.
-If you set 'store_as xxx', s3 plugin searches `fluent/plugin/s3_compressor_xxx.rb`.
-You can define your compression with 'S3Output::Compressor' class. Compressor API is here:
-
-    module Fluent
-      class S3Output
-        class XXXCompressor < Compressor
-          S3Output.register_compressor('xxx', self)
-
-          # Used to file extension
-          def ext
-            'xxx'
-          end
-
-          # Used to file content type
-          def content_type
-            'application/x-xxx'
-          end
-
-          # chunk is buffer chunk. tmp is destination file for upload
-          def compress(chunk, tmp)
-            # call command or something
-          end
-        end
-      end
-    end
-
-See bundled Compressor classes for more detail.
-
-== Website, license, et. al.
-
-Web site::  http://fluentd.org/
-Documents:: http://docs.fluentd.org/
-Source repository:: http://github.com/fluent
-Discussion:: http://groups.google.com/group/fluentd
-Author:: Sadayuki Furuhashi
-Copyright:: (c) 2011 FURUHASHI Sadayuki
-License:: Apache License, Version 2.0