fluent-plugin-dynamo 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c19597bfc773c66fa330779d66de8749746cbf7cafcf6e2aa4ea88a4d5b82721
+  data.tar.gz: fd2bd597a9f400a4c5a09ac73bdf1f78237e5f4570cada2e1ae54e81b69a1ab2
+SHA512:
+  metadata.gz: ced3fba29310892fb2dc725cd5b4852a89390fc48c1261424aadf2c75bf991a0f53f413e03f5c0d5be61cf7ecaebf01ace8dfe52d074fda69ce2625b24f3037b
+  data.tar.gz: b556811f8a8a65b315bc2455444cb531cf1a19edc34ae2e3327c135982d0fd807d9b15d4f84e9debcd07277f401e9774852362d5b73bcd58bc1c82cd9af5865e

data/AUTHORS

@@ -0,0 +1,3 @@
+Takashi Matsuno
+Sadayuki Furuhashi
+

data/ChangeLog

@@ -0,0 +1,20 @@
+Release 0.1.8 - 2012/07/10
+
+  * Fix gem.homepage url 
+
+Release 0.1.7 - 2012/06/17
+
+  * Inherits DetachMultiProcessMixin
+
+Release 0.1.6 - 2012/06/12
+
+ * Optimized write(chunk) method not to collect all records in memory
+
+Release 0.1.5 - 2012/06/10
+
+  * First release
+
+Release 0.1.0 - 2012/06/09
+
+  * First commit
+

data/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/README.md

@@ -0,0 +1,134 @@
+# Amazon DynamoDB output plugin for [Fluentd](http://fluentd.org) event collector
+
+## Installation
+
+    $ fluent-gem install fluent-plugin-dynamodb
+
+## Configuration
+
+
+### DynamoDB
+
+First of all, you need to create a table in DynamoDB. It's easy to create via Management Console.
+
+Specify table name, hash attribute name and throughput as you like. fluent-plugin-dynamodb will load your table schema and write event-stream out to your table.
+
+
+### Fluentd
+
+    <match dynamodb.**>
+      @type dynamodb
+      aws_key_id AWS_ACCESS_KEY
+      aws_sec_key AWS_SECRET_ACCESS_KEY
+      proxy_uri http://user:password@192.168.0.250:3128/
+      dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
+      dynamo_db_table access_log
+    </match>
+
+ * **aws\_key\_id (optional)** - AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Instance Profile.
+ * **aws\_sec\_key (optional)** - AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Instance Profile.
+ * **proxy_uri (optional)** - your proxy url.
+ * **dynamo\_db\_endpoint (required)** - end point of dynamodb. see  [Regions and Endpoints](http://docs.amazonwebservices.com/general/latest/gr/rande.html#ddb_region)
+ * **dynamo\_db\_table (required)** - table name of dynamodb.
+
+## TIPS
+
+### retrieving data
+
+fluent-plugin-dynamo will add **time** attribute and any other attributes of record automatically.
+For example if you read apache's access log via fluentd, structure of the table will have been like this.
+
+<table>
+  <tr>
+    <th>id (Hash Key)</th>
+    <th>time</th>
+    <th>host</th>
+    <th>path</th>
+    <th>method</th>
+    <th>referer</th>
+    <th>code</th>
+    <th>agent</th>
+    <th>size</th>
+  </tr>
+  <tr>
+    <td>"a937f980-b304-11e1-bc96-c82a14fffef2"</td>
+    <td>"2012-06-10T05:26:46Z"</td>
+    <td>"192.168.0.3"</td>
+    <td>"/index.html"</td>
+    <td>"GET"</td>
+    <td>"-"</td>
+    <td>"200"</td>
+    <td>"Mozilla/5.0"</td>
+    <td>"4286"</td>
+  </tr>
+  <tr>
+    <td>"a87fc51e-b308-11e1-ba0f-5855caf50759"</td>
+    <td>"2012-06-10T05:28:23Z"</td>
+    <td>"192.168.0.4"</td>
+    <td>"/sample.html"</td>
+    <td>"GET"</td>
+    <td>"-"</td>
+    <td>"200"</td>
+    <td>"Mozilla/5.0"</td>
+    <td>"8933"</td>
+  </tr>
+</table>
+
+Item can be retrieved by the key, but fluent-plugin-dynamo uses UUID as a primary key.
+There is no simple way to retrieve logs you want.
+By the way, you can write scan-filter with AWS SDK like [this](https://gist.github.com/2906291), but Hive on EMR is the best practice I think.
+
+### multiprocessing
+
+If you need high throughput and if you have much provisioned throughput and abudant buffer, you can setup multiprocessing. fluent-plugin-dynamodb uses **multi workers**, so you can launch 6 workers as follows.
+
+    <match dynamodb.**>
+      @type dynamodb
+      aws_key_id AWS_ACCESS_KEY
+      aws_sec_key AWS_SECRET_ACCESS_KEY
+      proxy_uri http://user:password@192.168.0.250:3128/
+      dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
+      dynamo_db_table access_log
+    </match>
+    <system>
+      workers 6
+    </system>
+
+### multi-region redundancy
+
+As you know fluentd has **copy** output plugin.
+So you can easily setup multi-region redundancy as follows.
+
+    <match dynamo.**>
+      @type copy
+      <store>
+        @type dynamodb
+        aws_key_id AWS_ACCESS_KEY
+        aws_sec_key AWS_SECRET_ACCESS_KEY
+        dynamo_db_table test
+        dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
+      </store>
+      <store>
+        @type dynamodb
+        aws_key_id AWS_ACCESS_KEY
+        aws_sec_key AWS_SECRET_ACCESS_KEY
+        dynamo_db_table test
+        dynamo_db_endpoint dynamodb.ap-southeast-1.amazonaws.com
+      </store>
+    </match>
+
+## TODO
+
+ * auto-create table
+ * tag_mapped
+
+## Copyright
+
+<table> 
+  <tr>
+    <td>Copyright</td><td>Copyright (c) 2012- Takashi Matsuno</td>
+  </tr>
+  <tr>
+    <td>License</td><td>Apache License, Version 2.0</td>
+  </tr>
+</table>

data/Rakefile

@@ -0,0 +1,14 @@
+
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.test_files = FileList['test/*.rb']
+  test.verbose = true
+end
+
+task :default => [:build]
+

data/VERSION

@@ -0,0 +1 @@
+1.0.0

data/fluent-plugin-dynamo.gemspec

@@ -0,0 +1,25 @@
+# encoding: utf-8
+$:.push File.expand_path('../lib', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.name        = "fluent-plugin-dynamo"
+  gem.description = "Amazon DynamoDB output plugin for Fluent event collector"
+  gem.homepage    = "https://github.com/gonsuke/fluent-plugin-dynamodb"
+  gem.summary     = gem.description
+  gem.license     = "Apache-2.0"
+  gem.version     = File.read("VERSION").strip
+  gem.authors     = ["Takashi Matsuno"]
+  gem.email       = "g0n5uk3@gmail.com"
+  gem.has_rdoc    = false
+  #gem.platform    = Gem::Platform::RUBY
+  gem.files       = `git ls-files`.split("\n")
+  gem.test_files  = `git ls-files -- {test,spec,features}/*`.split("\n")
+  gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  gem.require_paths = ['lib']
+
+  gem.add_dependency "fluentd", [">= 0.14.15", "< 2"]
+  gem.add_dependency "aws-sdk-dynamodb", [">= 1.0.0", "< 2"]
+  gem.add_dependency "uuidtools", "~> 2.1.0"
+  gem.add_development_dependency "rake", ">= 0.9.2"
+  gem.add_development_dependency "test-unit", ">= 3.1.0"
+end

data/lib/fluent/plugin/out_dynamo.rb

@@ -0,0 +1,143 @@
+# -*- coding: utf-8 -*-
+require 'fluent/plugin/output'
+require 'aws-sdk-dynamodb'
+require 'msgpack'
+require 'time'
+require 'uuidtools'
+
+module Fluent::Plugin
+
+
+class DynamoOutput < Fluent::Plugin::Output
+  Fluent::Plugin.register_output('dynamo', self)
+
+  helpers :compat_parameters
+
+  DEFAULT_BUFFER_TYPE = "memory"
+
+  BATCHWRITE_ITEM_LIMIT = 25
+  BATCHWRITE_CONTENT_SIZE_LIMIT = 1024*1024
+
+  config_param :aws_key_id, :string, :default => nil, :secret => true
+  config_param :aws_sec_key, :string, :default => nil, :secret => true
+  config_param :proxy_uri, :string, :default => nil
+  config_param :dynamo_db_region, :string, default: ENV["AWS_REGION"] || "us-east-1"
+  config_param :dynamo_db_table, :string
+  config_param :dynamo_db_endpoint, :string, :default => nil
+  config_param :time_format, :string, :default => nil
+  config_param :add_time_attribute, :bool, :default => true
+  config_param :detach_process, :integer, :default => 2
+
+  config_section :buffer do
+    config_set_default :@type, DEFAULT_BUFFER_TYPE
+  end
+
+  def configure(conf)
+    compat_parameters_convert(conf, :buffer)
+    super
+
+    @timef = Fluent::TimeFormatter.new(@time_format, @localtime)
+  end
+
+  def start
+    options = {}
+    if @aws_key_id && @aws_sec_key
+      options[:access_key_id] = @aws_key_id
+      options[:secret_access_key] = @aws_sec_key
+    end
+    options[:region] = @dynamo_db_region if @dynamo_db_region
+    options[:endpoint] = @dynamo_db_endpoint
+    options[:proxy_uri] = @proxy_uri if @proxy_uri
+
+    super
+
+    begin
+      restart_session(options)
+      valid_table(@dynamo_db_table)
+    rescue Fluent::ConfigError => e
+      log.fatal "ConfigError: Please check your configuration, then restart fluentd. '#{e}'"
+      exit!
+    rescue Exception => e
+      log.fatal "UnknownError: '#{e}'"
+      exit!
+    end
+  end
+
+  def restart_session(options)
+    @dynamo_db = Aws::DynamoDB::Client.new(options)
+    @resource = Aws::DynamoDB::Resource.new(client: @dynamo_db)
+
+  end
+
+  def valid_table(table_name)
+    table = @resource.table(table_name)
+    @hash_key = table.key_schema.select{|e| e.key_type == "HASH" }.first
+    range_key_candidate = table.key_schema.select{|e| e.key_type == "RANGE" }
+    @range_key = range_key_candidate.first if range_key_candidate
+  end
+
+  def match_type!(key, record)
+    if key.key_type == "NUMBER"
+      potential_value = record[key.attribute_name].to_i
+      if potential_value == 0
+        log.fatal "Failed attempt to cast hash_key to Integer."
+      end
+      record[key.attribute_name] = potential_value
+    end
+  end
+
+  def format(tag, time, record)
+    if !record.key?(@hash_key.attribute_name)
+      record[@hash_key.attribute_name] = UUIDTools::UUID.timestamp_create.to_s
+    end
+    match_type!(@hash_key, record)
+
+    formatted_time = @timef.format(time)
+    if @range_key
+      if !record.key?(@range_key.attribute_name)
+        record[@range_key.attribute_name] = formatted_time
+      end
+      match_type!(@range_key, record)
+    end
+    record['time'] = formatted_time if @add_time_attribute
+
+    record.to_msgpack
+  end
+
+  def formatted_to_msgpack_binary?
+    true
+  end
+
+  def multi_workers_ready?
+    true
+  end
+
+  def write(chunk)
+    batch_size = 0
+    batch_records = []
+    chunk.msgpack_each {|record|
+      batch_records << {
+        put_request: {
+          item: record
+        }
+      }
+      batch_size += record.to_json.length # FIXME: heuristic
+      if batch_records.size >= BATCHWRITE_ITEM_LIMIT || batch_size >= BATCHWRITE_CONTENT_SIZE_LIMIT
+        batch_put_records(batch_records)
+        batch_records.clear
+        batch_size = 0
+      end
+    }
+    unless batch_records.empty?
+      batch_put_records(batch_records)
+    end
+  end
+
+  def batch_put_records(records)
+    @dynamo_db.batch_write_item(request_items: { @dynamo_db_table => records })
+  end
+
+end
+
+
+end

data/test/out_dynamo.rb

@@ -0,0 +1,65 @@
+require 'fluent/test'
+require 'fluent/test/helpers'
+require 'fluent/test/driver/output'
+require 'fluent/plugin/out_dynamodb'
+
+class DynamoOutputTest < Test::Unit::TestCase
+  include Fluent::Test::Helpers
+
+  def setup
+    Fluent::Test.setup
+  end
+
+  CONFIG = %[
+    aws_key_id test_key_id
+    aws_sec_key test_sec_key
+    dynamo_db_table test_table
+    dynamo_db_endpoint test.endpoint
+    utc
+    buffer_type memory
+  ]
+
+  def create_driver(conf = CONFIG)
+    Fluent::Test::Driver::Output.new(Fluent::Plugin::DynamoOutput) do
+      def write(chunk)
+        chunk.read
+      end
+    end.configure(conf)
+  end
+
+  def test_configure
+    d = create_driver
+    assert_equal 'test_key_id', d.instance.aws_key_id
+    assert_equal 'test_sec_key', d.instance.aws_sec_key
+    assert_equal 'test_table', d.instance.dynamo_db_table
+    assert_equal 'test.endpoint', d.instance.dynamo_db_endpoint
+  end
+
+  def test_format
+    d = create_driver
+
+    time = event_time("2011-01-02 13:14:15 UTC")
+    d.run(default_tag: 'test') do
+      d.feed(time, {"a"=>1})
+      d.feed(time, {"a"=>2})
+    end
+
+    expected = [{'a' => 1}].to_msgpack + [{'a' => 2}].to_msgpack
+    assert_equal expected, d.formatted
+  end
+
+  def test_write
+    d = create_driver
+
+    time = event_time("2011-01-02 13:14:15 UTC")
+    d.run(default_tag: 'test') do
+      d.feed(time, {"a"=>1})
+      d.feed(time, {"a"=>2})
+    end
+
+    data = d.events
+
+    assert_equal [time, {'a' => 1}].to_msgpack + [time, {'a' => 2}].to_msgpack, data
+  end
+
+end

metadata

@@ -0,0 +1,135 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-dynamo
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Takashi Matsuno
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-09-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.14.15
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.14.15
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+- !ruby/object:Gem::Dependency
+  name: aws-sdk-dynamodb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+- !ruby/object:Gem::Dependency
+  name: uuidtools
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.1.0
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.9.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.9.2
+- !ruby/object:Gem::Dependency
+  name: test-unit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 3.1.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 3.1.0
+description: Amazon DynamoDB output plugin for Fluent event collector
+email: g0n5uk3@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- AUTHORS
+- ChangeLog
+- Gemfile
+- README.md
+- Rakefile
+- VERSION
+- fluent-plugin-dynamo.gemspec
+- lib/fluent/plugin/out_dynamo.rb
+- pkg/fluent-plugin-dynamodb-0.2.0.gem
+- test/out_dynamo.rb
+homepage: https://github.com/gonsuke/fluent-plugin-dynamodb
+licenses:
+- Apache-2.0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: Amazon DynamoDB output plugin for Fluent event collector
+test_files: []