fluent-plugin-s3 1.0.0.rc2 → 1.0.0.rc3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 109bc4e5f8560c3d6775afa7722d6bf0f77bdea1
-  data.tar.gz: ee37d164039e615cbaa2cd32c72e75d7e98eec8f
+  metadata.gz: 50b7e3d9177ef898ea2ee43c2ec789b2cb3293f2
+  data.tar.gz: 26ad4cbb951285caed1570ce28970344fbdf608a
 SHA512:
-  metadata.gz: af19567e42112011c726894551b16b87f85b04176afd4770932a1968988f8717605b6f9b77f901fc467efbed2b60faf292ee41289d0afc3f687a06ee10c400de
-  data.tar.gz: 43c5e86c3bbb2d0a5b65d7a6728ff916b82d8b732dd24fd1177fd7a7ba278f2e329625116f58bf5d9796261716d893f944b438cf5ee1ad66784edeb9be3599ec
+  metadata.gz: 2bee78cb5c5bc10c157d232107bcfedc7225ee23b49987eb7b2eb1161f998731589d40ecaaabd5ca623462f367d83e3efdfe9ff21b6e95b2030ef7a9b0a4e4ce
+  data.tar.gz: 2f877c094484f090dc2d7a418553af0f3303528625d91e5e9c19350e0c4ea7a013f73926f301e5c8632f9e109fd0e36914ac1a973f714f554d17ff140cbed6a5

data/README.md

@@ -1,4 +1,4 @@
-# Amazon S3 output plugin for [Fluentd](http://github.com/fluent/fluentd)
+# Amazon S3 plugin for [Fluentd](http://github.com/fluent/fluentd)
 
 [<img src="https://travis-ci.org/fluent/fluent-plugin-s3.svg?branch=master"
 alt="Build Status" />](https://travis-ci.org/fluent/fluent-plugin-s3) [<img
@@ -20,14 +20,28 @@ the former one is stored in "20110102.gz" file, and latter one in
 SQS queue on the region same as S3 bucket.
 We must setup SQS queue and S3 event notification before use this plugin.
 
+## Requirements
+
+| fluent-plugin-s3  | fluentd | ruby |
+|-------------------|---------|------|
+| >= 1.0.0 | >= v0.14.0 | >= 2.1 |
+|  < 1.0.0 | >= v0.12.0 | >= 1.9 |
+
+NOTE: fluent-plugin-s3 v1.0.0 is now RC. We will release stable v1.0.0 soon.
+
 ## Installation
 
 Simply use RubyGems:
 
-    gem install fluent-plugin-s3
+    $ gem install fluent-plugin-s3 -v "~> 0.8"  --no-document # for fluentd v0.12 or later
+    $ gem install fluent-plugin-s3 -v 1.0.0.rc2 --no-document # for fluentd v0.14 or later
 
 ## Output: Configuration
 
+### v0.14 style
+
+With fluentd v0.14 and fluent-plugin-s3 v1.0.0, use new buffer configuration to dynamic parameters.
+
     <match pattern>
       @type s3
 
@@ -35,15 +49,56 @@ Simply use RubyGems:
       aws_sec_key YOUR_AWS_SECRET_KEY
       s3_bucket YOUR_S3_BUCKET_NAME
       s3_region ap-northeast-1
+
+      path logs/${tag}/%Y/%m/%d/
       s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
+
+      # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
+      # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
+      <buffer tag,time>
+        @type file
+        path /var/log/fluent/s3
+        timekey 3600 # 1 hour partition
+        timekey_wait 10m
+        timekey_use_utc true # use utc
+      </buffer>
+      <format>
+        @type json
+      </format>
+    </match>
+
+For `<buffer>`, you can use any record field in `path` / `s3_object_key_format`.
+
+    path logs/${tag}/${foo}
+    <buffer tag,foo>
+      # parameters...
+    </buffer>
+
+This configuration doesn't work with fluentd v0.12.
+
+### v0.12 style
+
+This configuration works with both fluentd v0.12 and v0.14.
+
+    <match pattern>
+      @type s3
+
+      aws_key_id YOUR_AWS_KEY_ID
+      aws_sec_key YOUR_AWS_SECRET_KEY
+      s3_bucket YOUR_S3_BUCKET_NAME
+      s3_region ap-northeast-1
+
       path logs/
+      s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
       buffer_path /var/log/fluent/s3
-
       time_slice_format %Y%m%d-%H
       time_slice_wait 10m
       utc
+      format json
     </match>
 
+If you want to embed tag in `path` / `s3_object_key_format`, you need to use `fluent-plugin-forest` plugin.
+
 **aws_key_id**
 
 AWS access key id. This parameter is required when your agent is not
@@ -104,7 +159,7 @@ uploaded to S3 in the same time slice.
 * %{file_extention} is always "gz" for
 now.
 * %{uuid_flush} a uuid that is replaced everytime the buffer will be flushed. If you want to use this placeholder, install `uuidtools` gem first.
-* %{hostname} is replaced with `Socket.gethostname` result. This is same as "#{Socket.gethostname}".
+* %{hostname} is replaced with `Socket.gethostname` result.
 * %{hex_random} a random hex string that is replaced for each buffer chunk, not
 assured to be unique. This is used to follow a way of peformance tuning, `Add
 a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance
@@ -136,6 +191,13 @@ You get:
     "log/events/ts=20130111-23/events_1.gz"
     "log/events/ts=20130112-00/events_0.gz"
 
+NOTE: ${hostname} placeholder is deprecated since v0.8. You can get same result by using [configuration's embedded ruby code feature](http://docs.fluentd.org/articles/config-file#embedded-ruby-code).
+
+    s3_object_key_format %{path}%{time_slice}_%{hostname}%{index}.%{file_extension}
+    s3_object_key_format "%{path}%{time_slice}_#{Socket.gethostname}%{index}.%{file_extension}"
+
+Above two configurations are same. The important point is wrapping `""` is needed for `#{Socket.gethostname}`.
+
 **force_path_style**
 
 :force_path_style (Boolean) — default: false — When set to true, the
@@ -157,7 +219,7 @@ archive format on S3. You can use serveral format:
 
 See `Use your compression algorithm` section for adding another format.
 
-**format**
+**`<format>` or format**
 
 Change one line format in the S3 object. Supported formats are "out_file",
 "json", "ltsv" and "single_value". See also [official Formatter article](http://docs.fluentd.org/articles/formatter-plugin-overview).
@@ -472,7 +534,7 @@ Defaults to 'default' or `[ENV]('AWS_PROFILE')`.
 ## Input: Configuration
 
     <source>
-      type s3
+      @type s3
 
       aws_key_id YOUR_AWS_KEY_ID
       aws_sec_key YOUR_AWS_SECRET_KEY
@@ -571,14 +633,15 @@ roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
 with a properly configured IAM policy are preferred over embedding access keys
 on EC2 instances.
 
-## Use your compression algorithm
+## Use your (de)compression algorithm
 
 s3 plugin has plugabble compression mechanizm like Fleuntd's input / output
-plugin. If you set 'store_as xxx', s3 plugin searches
-`fluent/plugin/s3_compressor_xxx.rb`. You can define your compression with
-'S3Output::Compressor' class. Compressor API is here:
+plugin. If you set 'store_as xxx', `out_s3` plugin searches
+`fluent/plugin/s3_compressor_xxx.rb` and `in_s3` plugin searches
+`fluent/plugin/s3_extractor_xxx.rb`. You can define your (de)compression with
+'S3Output::Compressor'/`S3Input::Extractor` classes. Compressor API is here:
 
-    module Fluent
+    module Fluent # Since fluent-plugin-s3 v1.0.0 or later, use Fluent::Plugin instead of Fluent
       class S3Output
         class XXXCompressor < Compressor
           S3Output.register_compressor('xxx', self)
@@ -601,7 +664,8 @@ plugin. If you set 'store_as xxx', s3 plugin searches
       end
     end
 
-See bundled Compressor classes for more detail.
+`Extractor` is similar to `Compressor`
+See bundled `Compressor`/`Extractor` classes for more detail.
 
 ## Website, license, et. al.
 

data/VERSION

@@ -1 +1 @@
-1.0.0.rc2
+1.0.0.rc3

data/lib/fluent/log-ext.rb

@@ -0,0 +1,12 @@
+require 'fluent/log'
+# For Fluentd v0.14.13 or earlier
+# logger for Aws::S3::Client and Aws::SQS::Client required `#<<` method
+module Fluent
+  class Log
+    unless method_defined?(:<<)
+      def <<(message)
+        write(message)
+      end
+    end
+  end
+end

data/lib/fluent/plugin/in_s3.rb

@@ -1,4 +1,5 @@
-require 'fluent/input'
+require 'fluent/plugin/input'
+require 'fluent/log-ext'
 
 require 'aws-sdk-resources'
 require 'zlib'
@@ -6,7 +7,7 @@ require 'time'
 require 'tempfile'
 
 module Fluent::Plugin
-  class S3Input < Fluent::Plugin::Input
+  class S3Input < Input
     Fluent::Plugin.register_input('s3', self)
 
     helpers :compat_parameters, :parser, :thread
@@ -101,6 +102,7 @@ module Fluent::Plugin
       super
 
       s3_client = create_s3_client
+      log.debug("Succeeded to create S3 client")
       @s3 = Aws::S3::Resource.new(client: s3_client)
       @bucket = @s3.bucket(@s3_bucket)
 
@@ -109,8 +111,10 @@ module Fluent::Plugin
       check_apikeys if @check_apikey_on_start
 
       sqs_client = create_sqs_client
+      log.debug("Succeeded to create SQS client")
       response = sqs_client.get_queue_url(queue_name: @sqs.queue_name)
       sqs_queue_url = response.queue_url
+      log.debug("Succeeded to get SQS queue URL")
 
       @poller = Aws::SQS::QueuePoller.new(sqs_queue_url, client: sqs_client)
 
@@ -135,6 +139,7 @@ module Fluent::Plugin
       @poller.poll(options) do |message|
         begin
           body = Yajl.load(message.body)
+          log.debug(body)
           next unless body["Records"] # skip test queue
 
           process(body)
@@ -185,6 +190,10 @@ module Fluent::Plugin
       options = setup_credentials
       options[:region] = @s3_region if @s3_region
       options[:proxy_uri] = @proxy_uri if @proxy_uri
+      log.on_trace do
+        options[:http_wire_trace] = true
+        options[:logger] = log
+      end
 
       Aws::S3::Client.new(options)
     end
@@ -192,12 +201,17 @@ module Fluent::Plugin
     def create_sqs_client
       options = setup_credentials
       options[:region] = @s3_region if @s3_region
+      log.on_trace do
+        options[:http_wire_trace] = true
+        options[:logger] = log
+      end
 
       Aws::SQS::Client.new(options)
     end
 
     def check_apikeys
       @bucket.objects.first
+      log.debug("Succeeded to verify API keys")
     rescue => e
       raise "can't call S3 API. Please check your aws_key_id / aws_sec_key or s3_region configuration. error = #{e.inspect}"
     end

data/lib/fluent/plugin/out_s3.rb

@@ -1,11 +1,12 @@
-require 'fluent/output'
+require 'fluent/plugin/output'
+require 'fluent/log-ext'
 require 'aws-sdk-resources'
 require 'zlib'
 require 'time'
 require 'tempfile'
 
 module Fluent::Plugin
-  class S3Output < Fluent::Plugin::Output
+  class S3Output < Output
     Fluent::Plugin.register_output('s3', self)
 
     helpers :compat_parameters, :formatter, :inject
@@ -113,6 +114,11 @@ module Fluent::Plugin
       config_set_default :@type, DEFAULT_FORMAT_TYPE
     end
 
+    config_section :buffer do
+      config_set_default :chunk_keys, ['time']
+      config_set_default :timekey, (60 * 60 * 24)
+    end
+
     attr_reader :bucket
 
     MAX_HEX_RANDOM_LENGTH = 16
@@ -166,6 +172,10 @@ module Fluent::Plugin
       options[:compute_checksums] = @compute_checksums unless @compute_checksums.nil?
       options[:signature_version] = @signature_version unless @signature_version.nil?
       options[:ssl_verify_peer] = @ssl_verify_peer
+      log.on_trace do
+        options[:http_wire_trace] = true
+        options[:logger] = log
+      end
 
       s3_client = Aws::S3::Client.new(options)
       @s3 = Aws::S3::Resource.new(client: s3_client)
@@ -188,19 +198,22 @@ module Fluent::Plugin
 
     def write(chunk)
       i = 0
+      metadata = chunk.metadata
       previous_path = nil
       time_slice_format = @configured_time_slice_format || timekey_to_timeformat(@buffer_config['timekey'])
-      time_slice = Time.at(chunk.metadata.timekey).utc.strftime(time_slice_format)
+      time_slice = if metadata.timekey.nil?
+                     ''.freeze
+                   else
+                     Time.at(metadata.timekey).utc.strftime(time_slice_format)
+                   end
 
       if @check_object
         begin
-          path = extract_placeholders(@path, chunk.metadata)
-
           @values_for_s3_object_chunk[chunk.unique_id] ||= {
             "%{hex_random}" => hex_random(chunk),
           }
           values_for_s3_object_key = {
-            "%{path}" => path,
+            "%{path}" => @path,
             "%{time_slice}" => time_slice,
             "%{file_extension}" => @compressor.ext,
             "%{index}" => i,
@@ -208,6 +221,7 @@ module Fluent::Plugin
           values_for_s3_object_key["%{uuid_flush}".freeze] = uuid_random if @uuid_flush_enabled
 
           s3path = @s3_object_key_format.gsub(%r(%{[^}]+}), values_for_s3_object_key)
+          s3path = extract_placeholders(s3path, metadata)
           if (i > 0) && (s3path == previous_path)
             if @overwrite
               log.warn "#{s3path} already exists, but will overwrite"
@@ -227,19 +241,18 @@ module Fluent::Plugin
           hms_slicer = Time.now.utc.strftime("%H%M%S")
         end
 
-        path = extract_placeholders(@path, chunk.metadata)
-
         @values_for_s3_object_chunk[chunk.unique_id] ||= {
           "%{hex_random}" => hex_random(chunk),
         }
         values_for_s3_object_key = {
-          "%{path}" => path,
+          "%{path}" => @path,
           "%{time_slice}" => time_slice,
           "%{file_extension}" => @compressor.ext,
         }.merge!(@values_for_s3_object_chunk[chunk.unique_id])
         values_for_s3_object_key["%{uuid_flush}".freeze] = uuid_random if @uuid_flush_enabled
 
         s3path = @s3_object_key_format.gsub(%r(%{[^}]+}), values_for_s3_object_key)
+        s3path = extract_placeholders(s3path, metadata)
       end
 
       tmp = Tempfile.new("s3-")

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-s3
 version: !ruby/object:Gem::Version
-  version: 1.0.0.rc2
+  version: 1.0.0.rc3
 platform: ruby
 authors:
 - Sadayuki Furuhashi
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-01-31 00:00:00.000000000 Z
+date: 2017-03-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -124,6 +124,7 @@ files:
 - VERSION
 - appveyor.yml
 - fluent-plugin-s3.gemspec
+- lib/fluent/log-ext.rb
 - lib/fluent/plugin/in_s3.rb
 - lib/fluent/plugin/out_s3.rb
 - lib/fluent/plugin/s3_compressor_gzip_command.rb
@@ -154,7 +155,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: 1.3.1
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 2.6.8
 signing_key: 
 specification_version: 4
 summary: Amazon S3 output plugin for Fluentd event collector