fluent-plugin-forward-aws 0.1.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+test/awsconfig.yml

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-forward-aws.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2013 - Tomohisa Ota
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,171 @@
+# Fluent::Plugin::Forward-AWS
+
+Forward-AWS plugin forwards log through Amazon Web Service.  
+It uses S3 as log storage, and SNS+SQS for notification.  
+
+Please see [wiki](https://github.com/tomohisaota/fluent-plugin-forward-aws/wiki/Forward-AWS-plugin-concept) to understand the concept.
+
+## Installation
+
+Use ruby gem as :
+
+    $ gem install fluent-plugin-forward-aws
+
+Or, if you're using td-client, you can call td-client's gem
+
+    $ /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-forward-aws
+
+## AWS Configuration
+### IAM
+It is recommended to create IAM user for logging.  
+Create IAM user with credential, and memorize following parameters
++ aws_access_key_id
++ aws_secret_access_key
+
+### S3
+Create bucket, and memorize following parameters
++ aws_s3_endpoint
++ aws_s3_bucketname
+
+### SNS
+Create SNS Topic, and memorize following parameters
++ aws_sns_endpoint
++ aws_sns_topic_arn 
+
+### SQS
+Create SQS Queue, subscribe to SNS, and memorize following parameters
++ aws_sqs_endpoint
++ aws_sqs_queue_url 
+
+#### How to configure SQS as SNS subscriber
+In short, change SQS's access policy to accept "SendMessage" from your SNS ARN, And add SQS ARN to SNS subscribers. 
+You can do the above step in one shot from SQS Management Console.  
+For more detail, check [amazon official document](http://docs.aws.amazon.com/sns/latest/gsg/SendMessageToSQS.html)
+
+## Out Plugin Configuration
+
+###Required AWS permission 
++ s3:PutObject
++ sns:Publish
+
+### Basic configuration
+Use "default" channel for all the log data
+```
+<match **>
+  type forward_aws
+  aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
+  aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+
+  aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
+  aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX
+
+  aws_sns_endpoint      sns.ap-northeast-1.amazonaws.com
+  aws_sns_topic_arn     arn:aws:sns:ap-northeast-1:XXXXXXXXXXXXXXXXXXXX
+
+  buffer_type           file
+  buffer_path           /var/log/td-agent/buffer/forward_aws
+  time_slice_wait       1m
+  utc
+  time_slice_format     %Y/%m/%d/%H/%M
+</match>
+```
+
+### Advanced configuration using forest plugin
+Use tag as channel
+```
+<match **>
+  type forest
+  subtype forward_aws
+  <template>
+    channel               ${tag}
+    aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
+    aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+
+    aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
+    aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX
+
+    aws_sns_endpoint      sns.ap-northeast-1.amazonaws.com
+    aws_sns_topic_arn     arn:aws:sns:ap-northeast-1:XXXXXXXXXXXXXXXXXXXX
+
+    buffer_type           file
+    buffer_path           /var/log/td-agent/buffer/forward_aws-${tag}
+    time_slice_wait       1m
+    utc
+    time_slice_format     %Y/%m/%d/%H/%M
+  </template>
+</match>
+```
+
+## In Plugin Configuration
+
+###Required AWS permission 
++ s3:GetObject
++ sqs:ReceiveMessage
++ sqs:DeleteMessage
+
+### Basic configuration
+Default config is to listen to "default" channel.
+```
+<source>
+  type forward_aws
+  aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
+  aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+
+  aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
+  aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX
+  
+  aws_sqs_endpoint      sqs.ap-northeast-1.amazonaws.com
+  aws_sqs_queue_url     https://sqs.ap-northeast-1.amazonaws.com/XXXXXXXXXXXXXXXXXXXX
+</source>
+```
+
+### Advanced configuration
+Use regex to filter channel
+```
+<source>
+  type forward_aws
+  channel .*
+  channelEnableRegEx true
+  aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
+  aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+
+  aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
+  aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX
+  
+  aws_sqs_endpoint      sqs.ap-northeast-1.amazonaws.com
+  aws_sqs_queue_url     https://sqs.ap-northeast-1.amazonaws.com/XXXXXXXXXXXXXXXXXXXX
+</source>
+```
+
+## Tips
+### How to delete buffer objects on S3
+Forward-AWS plugin do not delete buffer objects on S3.  
+Use [S3's lifecycle management](http://docs.aws.amazon.com/AmazonS3/latest/dev/manage-lifecycle-using-console.html) to automatically archive or delete old buffer objects.
+
+### How to use buffer objects as raw input
+Each buffer object is msgpack stream object with gzip compression.
+
+### Want to forward log per minute, but have archive per day
+You can configure forward per minute, and setup another receiver for archiving.
+
+### Skipping test at startup
+You can control startup test by following optional parameters. Default value is false
++ aws_s3_skiptest
++ aws_sns_skiptest
++ aws_sqs_skiptest
+
+
+## Contributing
+I am newbie for both of Ruby and Fluentd.  
+Feel free to send me pull request.  
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Copyright
+
+Copyright:: Copyright (c) 2013 Tomohisa Ota
+License::   Apache License, Version 2.0

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test

data/fluent-plugin-forward-aws.gemspec

@@ -0,0 +1,22 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |gem|
+  gem.name          = "fluent-plugin-forward-aws"
+  gem.version       = "0.1.1"
+  gem.authors       = ["Tomohisa Ota"]
+  gem.email         = ["tomohisa.ota+github@gmail.com"]
+  gem.description   = "Fluentd Forward Plugin using Amazon Web Service"
+  gem.summary       = gem.description
+  gem.homepage      = ""
+
+  gem.files         = `git ls-files`.split($/)
+  gem.files.reject! { |fn| fn.include? "doc/" }
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+
+  gem.add_runtime_dependency "fluentd"
+  gem.add_runtime_dependency "aws-sdk", "~> 1.8.2"
+end

data/lib/fluent/plugin/in_forward_aws.rb

@@ -0,0 +1,198 @@
+class Fluent::ForwardAWSInput < Fluent::Input
+  Fluent::Plugin.register_input('forward_aws', self)
+
+  # config_param :hoge, :string, :default => 'hoge'
+
+  def initialize
+    super
+    require 'aws-sdk'
+    require 'zlib'
+    require 'tempfile'
+    require 'json'
+    @locker = Mutex::new
+  end
+
+  config_param :channel, :string, :default => "default"
+  config_param :channelEnableRegEx, :bool, :default => false
+
+  config_param :aws_access_key_id, :string, :default => nil
+  config_param :aws_secret_access_key, :string, :default => nil
+  
+  config_param :aws_s3_endpoint, :string, :default => nil
+  config_param :aws_s3_bucketname, :string, :default => nil
+  config_param :aws_s3_skiptest, :bool, :default => false
+  
+  config_param :aws_sqs_endpoint, :string, :default => nil
+  config_param :aws_sqs_queue_url, :string, :default => nil
+  config_param :aws_sqs_skiptest, :bool, :default => false
+  
+  # Not documented parameters. Subject to change in future release
+  config_param :aws_sqs_process_interval, :integer, :default => 1
+  config_param :aws_sqs_monitor_interval, :integer, :default => 10
+  config_param :aws_s3_testobjectname, :string, :default => "Config Check Test Object"
+  config_param :start_thread, :bool, :default => true
+  
+  def configure(conf)
+    super
+    if /^\s*$/ =~ @channel
+      raise Fluent::ConfigError.new("channel is invalid. Exp=[\w]+")
+    end
+    unless @aws_access_key_id
+      raise Fluent::ConfigError.new("aws_access_key_id is required")
+    end
+    unless @aws_secret_access_key
+      raise Fluent::ConfigError.new("aws_secret_access_key is required")
+    end
+    unless @aws_s3_endpoint
+      raise Fluent::ConfigError.new("aws_s3_endpoint is required")
+    end
+    unless @aws_s3_bucketname
+      raise Fluent::ConfigError.new("aws_s3_bucketname is required")
+    end
+    unless @aws_sqs_endpoint
+      raise Fluent::ConfigError.new("aws_sqs_endpoint is required")
+    end
+    unless @aws_sqs_queue_url
+      raise Fluent::ConfigError.new("aws_sqs_queue_url is required")
+    end
+    unless(@aws_s3_skiptest)
+      init_aws_s3_bucket()
+      begin
+        @bucket.objects[@aws_s3_testobjectname].write("TEST", :content_type => 'text/plain')
+      rescue
+        raise Fluent::ConfigError.new("Cannot put object to S3. Need s3:PutObject permission for resource arn:aws:s3:::" + @aws_s3_bucketname+"/*")
+      end
+    end
+    unless(@aws_sqs_skiptest)
+      init_aws_sqs_queue()
+      begin
+        @queue.receive_message() do |msg|
+        end
+      rescue => e
+        raise Fluent::ConfigError.new("Cannot fetch queue from SQS. Need sqs:ReceiveMessage sqs:DeleteMessage permission for resource " + @aws_sqs_queue_url)
+      end
+    end
+  end
+  
+  def start
+    super
+    init_aws_s3_bucket()
+    init_aws_sqs_queue()
+    if(@start_thread)
+      @thread = Thread.new(&method(:run))
+    end
+  end
+
+  def run
+    @running = true
+    while true
+      msg = @queue.receive_message()
+      if msg
+        if(process(JSON.parse(msg.as_sns_message.body)))
+          msg.delete()
+        end
+        sleep @aws_sqs_process_interval
+        @locker.synchronize do
+          return unless @running
+        end
+        next
+      end
+      sleep @aws_sqs_monitor_interval
+      @locker.synchronize do
+        return unless @running
+      end
+    end
+  rescue => e
+    puts e 
+  end
+
+  def process(msg)
+    if msg["type"] == "ping"
+      # Ignore ping message
+      return true
+    end
+    if msg["type"] == "out"
+      # Silently ignore non matching logs
+      if msg["bucketname"] != @aws_s3_bucketname
+        return true
+      end
+      if(@channelEnableRegEx)
+        return true unless Regexp.new(@channel).match(msg["channel"])
+      else
+        return true unless @channel == msg["channel"]
+      end
+      
+      tmpFile = Tempfile.new("forward-aws-")
+      begin
+        #Download log file to temporary file
+        @bucket.objects[msg["path"]].read do |chunk|
+          tmpFile.write(chunk)
+        end
+        tmpFile.close()
+        #gunzip and decode log file
+        streamUnpacker = MessagePack::Unpacker.new()
+        Zlib::GzipReader.open(tmpFile) {|reader|
+          streamUnpacker.feed(reader.read())
+          streamUnpacker.each {|event|
+            (tag, time, record) = event
+            Fluent::Engine.emit(tag,time,record)
+          }
+        }
+        return true
+      ensure
+        tmp.close(true) rescue nil
+      end
+    end
+    # Unknown notification. Do not delete
+    return false
+  end
+
+  def shutdown
+    super
+    # Stop Thread and join
+    @locker.synchronize do
+      @running = false
+    end
+    if(@thread)
+      @thread.run
+      @thread.join
+      @thread = nil
+    end
+  end
+  
+  private
+  
+  def check_aws_credential
+    unless @aws_access_key_id
+      raise Fluent::ConfigError.new("aws_access_key_id is required")
+    end
+    unless @aws_secret_access_key
+      raise Fluent::ConfigError.new("aws_secret_access_key is required")
+    end
+  end
+  
+  private
+  
+  def init_aws_s3_bucket
+    unless @bucket
+      options = {}
+      options[:access_key_id]      = @aws_access_key_id
+      options[:secret_access_key]  = @aws_secret_access_key
+      options[:s3_endpoint]        = @aws_s3_endpoint
+      options[:use_ssl]            = true
+      s3 = AWS::S3.new(options)
+      @bucket = s3.buckets[@aws_s3_bucketname]
+    end
+  end
+  
+  def init_aws_sqs_queue
+    unless @queue
+      options = {}
+      options[:access_key_id]      = @aws_access_key_id
+      options[:secret_access_key]  = @aws_secret_access_key
+      options[:sqs_endpoint]       = @aws_sqs_endpoint
+      sqs = AWS::SQS.new(options)
+      @queue = sqs.queues[@aws_sqs_queue_url]
+    end
+  end
+end

data/lib/fluent/plugin/out_forward_aws.rb

@@ -0,0 +1,147 @@
+class Fluent::ForwardAWSOutput < Fluent::TimeSlicedOutput
+  Fluent::Plugin.register_output('forward_aws', self)
+
+  # config_param :hoge, :string, :default => 'hoge'
+
+  config_param :channel, :string, :default => "default"
+
+  config_param :aws_access_key_id, :string, :default => nil
+  config_param :aws_secret_access_key, :string, :default => nil
+  
+  config_param :aws_s3_endpoint, :string, :default => nil
+  config_param :aws_s3_bucketname, :string, :default => nil
+  config_param :aws_s3_skiptest, :bool, :default => false
+  
+  config_param :aws_sns_endpoint, :string, :default => nil
+  config_param :aws_sns_topic_arn, :string, :default => nil
+  config_param :aws_sns_skiptest, :bool, :default => false
+
+  # Not documented parameters. Subject to change in future release
+  config_param :aws_s3_testobjectname, :string, :default => "Config Check Test Object"
+  config_param :aws_sns_emailsubject, :string, :default => "SNS Message"
+
+  def initialize
+    super
+    require 'aws-sdk'
+    require 'zlib'
+    require 'tempfile'
+    require 'json'
+    require 'securerandom'
+  end
+
+  def configure(conf)
+    super
+    if /^\s*$/ =~ @channel
+      raise Fluent::ConfigError.new("channel is invalid. Exp=[\w]+")
+    end
+    unless @aws_access_key_id
+      raise Fluent::ConfigError.new("aws_access_key_id is required")
+    end
+    unless @aws_secret_access_key
+      raise Fluent::ConfigError.new("aws_secret_access_key is required")
+    end
+    unless @aws_s3_endpoint
+      raise Fluent::ConfigError.new("aws_s3_endpoint is required")
+    end
+    unless @aws_s3_bucketname
+      raise Fluent::ConfigError.new("aws_s3_bucketname is required")
+    end
+    unless @aws_sns_endpoint
+      raise Fluent::ConfigError.new("aws_sns_endpoint is required")
+    end
+    unless @aws_sns_topic_arn
+      raise Fluent::ConfigError.new("aws_sns_topic_arn is required")
+    end
+    unless(@aws_s3_skiptest)
+      init_aws_s3_bucket()
+      begin
+        @bucket.objects[@aws_s3_testobjectname].write("TEST", :content_type => 'text/plain')
+      rescue
+        raise Fluent::ConfigError.new("Cannot put object to S3. Need s3:PutObject permission for resource arn:aws:s3:::" + @aws_s3_bucketname+"/*")
+      end
+    end
+    unless(@aws_sns_skiptest)
+      init_aws_sns_topic()
+      begin
+        notification = {
+          "type" => "ping"
+        }
+        topic = @sns.topics[@aws_sns_topic_arn]
+        topic.publish(JSON.pretty_generate(notification), :subject => @aws_sns_emailsubject)
+      rescue
+        raise Fluent::ConfigError.new("Cannot post notification to SNS. Need sns:Publish permission for resource " + @aws_sns_topic_arn)
+      end
+    end
+  end
+
+  def start
+    super
+    init_aws_s3_bucket()
+    init_aws_sns_topic()
+  end
+
+  def shutdown
+    super
+    # destroy
+  end
+
+  def format(tag, time, record)
+    [tag, time, record].to_msgpack
+  end
+
+  def write(chunk)
+    #Add UUID to avoid name conflict
+    #Current version supports only msgpack + gz, but add suffix for future extentions
+    format = "msgpack"
+    compression = "gzip"
+    compression_suffix = "gz"
+    s3path = "#{@channel}/#{chunk.key}/#{SecureRandom.uuid}.#{format}.#{compression_suffix}"
+    
+    # Create temp gzip file
+    tmpFile = Tempfile.new("forward-aws-")
+    writer = Zlib::GzipWriter.new(tmpFile)
+    begin
+      chunk.write_to(writer)
+      writer.close
+      @bucket.objects[s3path].write(Pathname.new(tmpFile.path), :content_type => 'application/x-gzip')
+      notification = {
+        "type"        => "out",
+        "channel"     => @channel,
+        "bucketname"  => @aws_s3_bucketname,
+        "path"        => s3path,
+        "format"      => format,
+        "compression" => compression
+      }
+      topic = @sns.topics[@aws_sns_topic_arn]
+      topic.publish(JSON.pretty_generate(notification), :subject => @aws_sns_emailsubject)
+    ensure
+      writer.close rescue nil
+      tmp.close(true) rescue nil
+    end
+  end
+  
+  private
+  
+  def init_aws_s3_bucket
+    unless @bucket
+      options = {}
+      options[:access_key_id]      = @aws_access_key_id
+      options[:secret_access_key]  = @aws_secret_access_key
+      options[:s3_endpoint]        = @aws_s3_endpoint
+      options[:use_ssl]            = true
+      s3 = AWS::S3.new(options)
+      @bucket = s3.buckets[@aws_s3_bucketname]
+    end
+  end
+
+  def init_aws_sns_topic
+    unless @sns
+      options = {}
+      options[:access_key_id]     = @aws_access_key_id
+      options[:secret_access_key] = @aws_secret_access_key
+      options[:sns_endpoint]      = @aws_sns_endpoint
+      @sns = AWS::SNS.new(options)
+    end
+  end
+
+end

data/test/awsconfig.yml.sample

@@ -0,0 +1,9 @@
+aws_s3_endpoint: s3-ap-northeast-1.amazonaws.com
+aws_sns_endpoint: sns.ap-northeast-1.amazonaws.com
+aws_sqs_endpoint: sqs.ap-northeast-1.amazonaws.com
+
+aws_access_key_id: 
+aws_secret_access_key: 
+aws_s3_bucketname: 
+aws_sns_topic_arn: 
+aws_sqs_queue_url: 

data/test/helper.rb

@@ -0,0 +1,29 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'test/unit'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'fluent/test'
+unless ENV.has_key?('VERBOSE')
+  nulllogger = Object.new
+  nulllogger.instance_eval {|obj|
+    def method_missing(method, *args)
+      # pass
+    end
+  }
+  $log = nulllogger
+end
+
+require 'fluent/plugin/in_forward_aws'
+require 'fluent/plugin/out_forward_aws'
+
+class Test::Unit::TestCase
+end

data/test/plugin/test_in_foward_aws.rb

@@ -0,0 +1,75 @@
+require 'helper'
+
+class ForwardAWSInputTest < Test::Unit::TestCase
+  DUMMYCONFIG = %[
+    aws_access_key_id     TEST_AWS_ACCESS_KEY_ID
+    aws_secret_access_key TEST_AWS_SECRET_ACCESS_KEY
+
+    aws_s3_endpoint       TEST_AWS_S3_ENDPOINT
+    aws_s3_bucketname     TEST_AWS_S3_BUCKETNAME
+    aws_s3_skiptest       true
+    
+    aws_sqs_endpoint      TEST_AWS_SQS_ENDPOINT
+    aws_sqs_queue_url     TEST_AWS_SQS_QUEUE_URL
+    aws_sqs_skiptest      true
+    
+    start_thread          false
+  ]
+
+  def setup
+    Fluent::Test.setup
+    begin
+      require 'yaml'
+      @AWSTESTCONFIG = YAML.load_file(File.expand_path('../../awsconfig.yml', __FILE__))
+
+      @CONFIG = %[
+        aws_access_key_id     #{@AWSTESTCONFIG["aws_access_key_id"]}
+        aws_secret_access_key #{@AWSTESTCONFIG["aws_secret_access_key"]}
+
+        aws_s3_endpoint       #{@AWSTESTCONFIG["aws_s3_endpoint"]}
+        aws_s3_bucketname     #{@AWSTESTCONFIG["aws_s3_bucketname"]}
+        aws_s3_skiptest       true
+
+        aws_sqs_endpoint      #{@AWSTESTCONFIG["aws_sqs_endpoint"]}
+        aws_sqs_queue_url     #{@AWSTESTCONFIG["aws_sqs_queue_url"]}
+        aws_sqs_skiptest      true
+
+        start_thread          false
+      ]
+    rescue => e
+    end
+  end
+  
+  def create_driver(conf)
+    Fluent::Test::InputTestDriver.new(Fluent::ForwardAWSInput).configure(conf)
+  end
+  
+  def test_configure
+    d = create_driver(DUMMYCONFIG)
+    ### check configurations
+    assert_equal( 'TEST_AWS_ACCESS_KEY_ID',     d.instance.aws_access_key_id)
+    assert_equal( 'TEST_AWS_SECRET_ACCESS_KEY', d.instance.aws_secret_access_key)
+    
+    assert_equal( 'TEST_AWS_S3_ENDPOINT',     d.instance.aws_s3_endpoint)
+    assert_equal( 'TEST_AWS_S3_BUCKETNAME',     d.instance.aws_s3_bucketname)
+
+    assert_equal( 'TEST_AWS_SQS_ENDPOINT',     d.instance.aws_sqs_endpoint)
+    assert_equal( 'TEST_AWS_SQS_QUEUE_URL',     d.instance.aws_sqs_queue_url)
+  end
+  
+  def test_check_aws_s3
+    unless(@CONFIG)
+      # Skip Test
+      return
+    end
+    create_driver(@CONFIG + "aws_s3_skiptest false").run()
+  end
+
+  def test_check_aws_sqs
+    unless(@CONFIG)
+      # Skip Test
+      return
+    end
+    create_driver(@CONFIG + "aws_sqs_skiptest false").run()
+  end
+end

data/test/plugin/test_out_foward_aws.rb

@@ -0,0 +1,91 @@
+require 'helper'
+
+class ForwardAWSOutputTest < Test::Unit::TestCase
+  DUMMYCONFIG = %[
+    aws_access_key_id     TEST_AWS_ACCESS_KEY_ID
+    aws_secret_access_key TEST_AWS_SECRET_ACCESS_KEY
+
+    aws_s3_endpoint       TEST_AWS_S3_ENDPOINT
+    aws_s3_bucketname     TEST_AWS_S3_BUCKETNAME
+    aws_s3_skiptest       true
+    
+    aws_sns_endpoint      TEST_AWS_SNS_ENDPOINT
+    aws_sns_topic_arn     TEST_AWS_SNS_TOPIC_ARN
+    aws_sns_skiptest      true
+    
+    buffer_type memory
+  ]
+  
+  def setup
+    Fluent::Test.setup
+
+    begin
+      require 'yaml'
+      @AWSTESTCONFIG = YAML.load_file(File.expand_path('../../awsconfig.yml', __FILE__))
+
+      @CONFIG = %[
+        aws_access_key_id     #{@AWSTESTCONFIG["aws_access_key_id"]}
+        aws_secret_access_key #{@AWSTESTCONFIG["aws_secret_access_key"]}
+
+        aws_s3_endpoint       #{@AWSTESTCONFIG["aws_s3_endpoint"]}
+        aws_s3_bucketname     #{@AWSTESTCONFIG["aws_s3_bucketname"]}
+        aws_s3_skiptest       true
+
+        aws_sns_endpoint      #{@AWSTESTCONFIG["aws_sns_endpoint"]}
+        aws_sns_topic_arn     #{@AWSTESTCONFIG["aws_sns_topic_arn"]}
+        aws_sns_skiptest      true
+
+        buffer_type memory
+      ]
+    rescue => e
+    end
+  end
+
+  def create_driver(conf)
+    Fluent::Test::BufferedOutputTestDriver.new(Fluent::ForwardAWSOutput) do
+      def write(chunk)
+        chunk.read
+      end
+    end.configure(conf)
+  end
+
+  def test_configure
+    d = create_driver(DUMMYCONFIG)
+    ### check configurations
+    assert_equal( 'TEST_AWS_ACCESS_KEY_ID',     d.instance.aws_access_key_id)
+    assert_equal( 'TEST_AWS_SECRET_ACCESS_KEY', d.instance.aws_secret_access_key)
+    
+    assert_equal( 'TEST_AWS_S3_ENDPOINT',     d.instance.aws_s3_endpoint)
+    assert_equal( 'TEST_AWS_S3_BUCKETNAME',     d.instance.aws_s3_bucketname)
+
+    assert_equal( 'TEST_AWS_SNS_ENDPOINT',     d.instance.aws_sns_endpoint)
+    assert_equal( 'TEST_AWS_SNS_TOPIC_ARN',     d.instance.aws_sns_topic_arn)
+  end
+
+  def test_format
+    d = create_driver(DUMMYCONFIG)
+    time = Time.parse("2011-01-02 13:14:15 UTC").to_i
+    d.emit({"a"=>1}, time)
+    d.emit({"a"=>2}, time)
+
+    d.expect_format ["test",time,{"a"=>1}].to_msgpack
+    d.expect_format ["test",time,{"a"=>2}].to_msgpack
+    d.run
+  end
+
+  def test_check_aws_s3
+    unless(@CONFIG)
+      # Skip Test
+      return
+    end
+    create_driver(@CONFIG + "aws_s3_skiptest false").run()
+  end
+
+  def test_check_aws_sns
+    unless(@CONFIG)
+      # Skip Test
+      return
+    end
+    create_driver(@CONFIG + "aws_sns_skiptest false").run()
+  end
+end

metadata

@@ -0,0 +1,93 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-forward-aws
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+  prerelease: 
+platform: ruby
+authors:
+- Tomohisa Ota
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-02-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.8.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.8.2
+description: Fluentd Forward Plugin using Amazon Web Service
+email:
+- tomohisa.ota+github@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- fluent-plugin-forward-aws.gemspec
+- lib/fluent/plugin/in_forward_aws.rb
+- lib/fluent/plugin/out_forward_aws.rb
+- test/awsconfig.yml.sample
+- test/helper.rb
+- test/plugin/test_in_foward_aws.rb
+- test/plugin/test_out_foward_aws.rb
+homepage: ''
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.25
+signing_key: 
+specification_version: 3
+summary: Fluentd Forward Plugin using Amazon Web Service
+test_files:
+- test/awsconfig.yml.sample
+- test/helper.rb
+- test/plugin/test_in_foward_aws.rb
+- test/plugin/test_out_foward_aws.rb