dummer 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: afa2183e1b551764a566ecc80ae1c35eeb5f9459
-  data.tar.gz: 0a4b5914330054a8e70c1f27b7b1e43d2ac13787
+  metadata.gz: e7473e82a0102c5445a4e1be36cf0059e05288fe
+  data.tar.gz: 17f8f33068776fba205d0190aff463798a29c0e7
 SHA512:
-  metadata.gz: 8b5253ad74c8bd4900fc2a060ef67bb8f5bee98a431848279961bc8fd29e25a9f37f2210ef3fe865a8a4f38a67a5602adff0bd83e660641369440143b9434934
-  data.tar.gz: f32cf508096d457d70be676593729dc8893a5d3c91370d387707750200621cb14f0dba8d766dcad8b14ea5cb4eae28f0d4f167c0cf0b7c8b93a58c2bae673618
+  metadata.gz: 67b31312c2289c14eb15d5ff5b1e21405fe1547d8b8be607ab9b259a02a5d312ffd3c0dd244b94611fac2637296738486ac3bc71ea356561d857f3a5995a9e07
+  data.tar.gz: f7498d62588d69aa3ab2489a226eedb6494fc0567b29abde08ea6203dc52f586bc3e0e223cf19eceb687fd0a37bcf7167ce144f70931af47a298785b83ce3de2

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+### 0.3.1 (2014/01/31)
+
+Enhancement:
+
+  * Support posting data to Fluentd process directly
+
 ### 0.3.0 (2014/01/25)
 
 Big Changes:

data/README.md

@@ -1,6 +1,6 @@
 # Dummer
 
-#### NOTE: `dummy_log_generator` was renamed to `dummer` which is easier to type, yay!
+#### NOTE: `dummy_log_generator` was renamed to `dummer`
 
 Dummer is a set of tools to generate dummy log data for Fluentd benchmark. 
 
@@ -38,7 +38,7 @@ Run as
 2. determine a log format, and
 3. generate logs randomly
 
-### Usage
+### Usage (1) - Write to a file
 
 Create a configuration file. A sample configuration is as follows:
 
@@ -73,8 +73,47 @@ id:0423  time:[2013-11-19 02:34:58]  level:DEBUG method:GET  uri:/api/v1/people
 id:0424  time:[2013-11-19 02:34:58]  level:WARN  method:POST uri:/api/v1/textdata  reqtime:2.930590441869852   foobar:XEZ5bQsh
 ```
 
+### Usage (2) - Post to Fluentd process
+
+(experimental)
+
+Create a configuration file. Assume that a fluentd process is running on localhost:20000.
+A sample configuration is as follows:
+
+```ruby
+# dummer.conf
+configure 'sample' do
+  host "localhost" # define `host` and `port` instead of `output`
+  port 20000
+  rate 500
+  tag type: string, any: %w[raw.syslog raw.message raw.nginx] # configure tag
+  field :id, type: :integer, countup: true, format: "%04d"
+  field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
+  field :method, type: :string, any: %w[GET POST PUT]
+  field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata /api/v1/messages]
+  field :reqtime, type: :float, range: 0.1..5.0
+  field :foobar, type: :string, length: 8
+end 
+```
+
+Running
+
+```
+$ dummer -c dummer.conf
+```
+
+Data is posted to fluentd process like (below is the fluentd log generated by out_stdout)
+
+```
+2014-01-31 00:55:32 +0900 raw.message: {"id":"1377","level":"INFO","method":"POST","uri":"/api/v1/people","reqtime":1.678867810409548,"foobar":"paOIWxhQ"}
+2014-01-31 00:55:32 +0900 raw.syslog: {"id":"1378","level":"INFO","method":"GET","uri":"/api/v1/people","reqtime":4.8412816521873445,"foobar":"kUvnC0MK"}
+2014-01-31 00:55:32 +0900 raw.message: {"id":"1379","level":"WARN","method":"GET","uri":"/api/v1/people","reqtime":3.584494903998221,"foobar":"KD78mpjX"}
+```
+
 ### CLI Options
 
+You can specify some configuration parameters on CLI without writing them on a configuration file.
+
 ```
 $ dummer help start
 Usage:
@@ -85,6 +124,8 @@ Options:
                                    # Default: dummer.conf
   -r, [--rate=N]                   # Number of generating messages per second
   -o, [--output=OUTPUT]            # Output file
+  -h, [--host=HOST]                # Host of fluentd process
+  -p, [--port=N]                   # Port of fluentd process
   -m, [--message=MESSAGE]          # Output message
   -d, [--daemonize]                # Daemonize. Stop with `dummer stop`
   -w, [--workers=N]                # Number of parallels
@@ -102,6 +143,14 @@ Following parameters in the configuration file are available:
 
     Specify a filename to output, or IO object (STDOUT, STDERR)
 
+* host
+
+    Post a data to a fluentd process on the specified host. Either of `output` or `host` can be specified.
+
+* port
+
+    Post a data to a fluentd process on the specified post. Default is 24224. 
+
 * rate
 
     Specify how many messages to generate per second. Default: 500 msgs / sec
@@ -118,6 +167,10 @@ Following parameters in the configuration file are available:
 
     Whether add field name as a label or not. Default: true
 
+* tag
+
+    Define tag field to generate. This is effective only for posting data to fluentd process with `host` and `port`.
+
 * field
 
     Define data fields to generate. `message` and `input` options are ignored. 
@@ -132,7 +185,7 @@ Following parameters in the configuration file are available:
 
 ### Field Data Types
 
-You can specify following data types to your `field` parameters:
+You can specify following data types to your `tag` and `field` parameters:
 
 * :datetime
 
@@ -199,6 +252,8 @@ You can specify following data types to your `field` parameters:
 I created a simple version of `dummer` since it can not achieve the maximum system I/O throughputs because of its rich features.
 This simple version, `dummer_simple` could  achieve the system I/O limit in my environment. 
 
+Sorry, but this simple script cannot post data to fluentd process, supports only writing to a file.
+
 ### Usage
 
 ```
@@ -227,6 +282,7 @@ Options:
 ## dummer\_yes
 
 I created a wrapped version of `yes` command, `dummer_yes`, to confrim that `dummer_simple` achieves the maximum system I/O throughputs. 
+
 I do not use `dummer_yes` command anymore because I verified that `dummer_simple` achieves the I/O limit, but I will keep this command so that users can do verification experiments with it. 
 
 ### Usage

data/dummer.gemspec

@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "thor"
   spec.add_dependency "serverengine"
+  spec.add_dependency "fluent-logger"
 
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"

data/example/fluent_logger.conf

@@ -0,0 +1,12 @@
+configure 'sample' do
+  host "localhost"
+  port 20000
+  rate 500
+  tag type: :string, any: %w[raw.syslog raw.message raw.nginx]
+  field :id, type: :integer, countup: true, format: "%04d"
+  field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
+  field :method, type: :string, any: %w[GET POST PUT]
+  field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata /api/v1/messages]
+  field :reqtime, type: :float, range: 0.1..5.0
+  field :foobar, type: :string, length: 8
+end

data/lib/dummer/cli.rb

@@ -17,6 +17,8 @@ module Dummer
     option :config,    :aliases => ["-c"], :type => :string, :default => 'dummer.conf', :desc => 'Config file'
     option :rate,      :aliases => ["-r"], :type => :numeric, :desc => 'Number of generating messages per second'
     option :output,    :aliases => ["-o"], :type => :string, :desc => 'Output file'
+    option :host,      :aliases => ["-h"], :type => :string, :desc => 'Host of fluentd process'
+    option :port,      :aliases => ["-p"], :type => :numeric, :desc => 'Port of fluentd process'
     option :message,   :aliases => ["-m"], :type => :string, :desc => 'Output message'
     # options for serverengine
     option :daemonize, :aliases => ["-d"], :type => :boolean, :desc => 'Daemonize. Stop with `dummer stop`'
@@ -30,10 +32,12 @@ module Dummer
         else
           Dummer::Dsl.new
         end
-      @options[:setting] = dsl.setting
       dsl.setting.rate = options[:rate] if options[:rate]
       dsl.setting.output = options[:output] if options[:output]
+      dsl.setting.host = options[:host] if options[:host]
+      dsl.setting.port = options[:port] if options[:port]
       dsl.setting.message = options[:message] if options[:message]
+      @options[:setting] = dsl.setting
       # options for serverengine
       @options[:workers] ||= dsl.setting.workers
 

data/lib/dummer/generator.rb

@@ -3,79 +3,135 @@ module Dummer
     def initialize(setting)
       @message_proc =
         if fields = setting.fields
-          labeled, delimiter = setting.labeled, setting.delimiter
-          prepare_message_proc_for_fields(fields, labeled, delimiter)
+          Field.message_proc(fields, setting.labeled, setting.delimiter)
         elsif input = setting.input
-          prepare_message_proc_for_input(input)
+          Input.message_proc(input)
         else
-          message = setting.message
-          prepare_message_proc_for_message(message)
+          Message.message_proc(setting.message)
         end
+      @record_proc =
+        if fields = setting.fields
+          Field.record_proc(fields)
+        elsif input = setting.input
+          Input.record_proc(input)
+        else
+          Message.record_proc(setting.message)
+        end
+      @tag_proc = Field.tag_proc(setting.tag)
     end
 
-    def prepare_message_proc_for_input(input)
-      messages = nil
-      begin
-        open(input) do |in_file|
-          messages = in_file.readlines
-        end
-      rescue Errno::ENOENT
-        raise ConfigError.new("Input file `#{input}` is not readable")
-      end
-      idx = -1
-      size = messages.size
-      Proc.new {
-        idx = (idx + 1) % size
-        messages[idx]
-      }
+    # @return [String] message
+    def message
+      @message_proc.call
     end
 
-    def prepare_message_proc_for_message(message)
-      message = "#{message.chomp}\n"
-      Proc.new { message }
+    # @return [String] tag
+    def tag
+      @tag_proc.call
     end
 
-    def prepare_message_proc_for_fields(fields, labeled, delimiter)
-      format_proc = prepare_format_proc(labeled, delimiter)
-      field_procs = prepare_field_procs(fields)
+    # @return [Hash] record
+    def record
+      @record_proc.call
+    end
 
-      prev_data = {}
-      Proc.new {
-        data = {}
-        field_procs.each do |key, proc|
-          prev = prev_data[key] || -1
-          data[key] = proc.call(prev)
-        end
-        prev_data = data
-        format_proc.call(data)
-      }
+    class Message
+      def self.message_proc(message)
+        message = "#{message.chomp}\n"
+        Proc.new { message }
+      end
+
+      def self.record_proc(message)
+        # ToDo: implement parser
+        message_proc = message_proc(message)
+        Proc.new { { "message" => message_proc.call } }
+      end
     end
 
-    def prepare_format_proc(labeled, delimiter)
-      if labeled
-        Proc.new {|fields| "#{fields.map {|key, val| "#{key}:#{val}" }.join(delimiter)}\n" }
-      else
-        Proc.new {|fields| "#{fields.values.join(delimiter)}\n" }
+    class Input
+      def self.message_proc(input)
+        messages = nil
+        begin
+          open(input) do |in_file|
+            messages = in_file.readlines
+          end
+        rescue Errno::ENOENT
+          raise ConfigError.new("Input file `#{input}` is not readable")
+        end
+        idx = -1
+        size = messages.size
+        Proc.new {
+          idx = (idx + 1) % size
+          messages[idx]
+        }
+      end
+
+      def self.record_proc(input)
+        # ToDo: implement parser
+        message_proc = message_proc(input)
+        Proc.new { { "message" => message_proc.call } }
       end
     end
 
-    def prepare_field_procs(fields)
-      rand = ::Dummer::Random.new
-      field_procs = {}
-      fields.each do |key, opts|
-        opts = opts.dup
-        type = opts.delete(:type)
-        if rand.respond_to?(type)
-          field_procs[key] = rand.send(type, opts)
+    class Field
+      def self.message_proc(fields, labeled, delimiter)
+        format_proc = format_proc(labeled, delimiter)
+        record_proc = record_proc(fields)
+
+        Proc.new {
+          hash = record_proc.call
+          format_proc.call(hash)
+        }
+      end
+
+      def self.record_proc(fields)
+        field_procs = field_procs(fields)
+
+        prev_data = {}
+        Proc.new {
+          data = {}
+          field_procs.each do |key, proc|
+            prev = prev_data[key] || -1
+            data[key] = proc.call(prev)
+          end
+          prev_data = data
+        }
+      end
+
+      def self.tag_proc(tag_opts)
+        field_procs({"tag" => tag_opts})["tag"]
+      end
+
+      def self.format_proc(labeled, delimiter)
+        if labeled
+          Proc.new {|fields| "#{fields.map {|key, val| "#{key}:#{val}" }.join(delimiter)}\n" }
         else
-          raise ConfigError.new(type)
+          Proc.new {|fields| "#{fields.values.join(delimiter)}\n" }
         end
       end
-      field_procs
-    end
 
-    def generate
-      @message_proc.call
+      def self.field_procs(fields)
+        rand = ::Dummer::Random.new
+        field_procs = {}
+        fields.each do |key, opts|
+          opts = opts.dup
+          type = opts.delete(:type)
+          field_procs[key] =
+            case type
+            when :string
+              rand.string(opts)
+            when :integer
+              rand.integer(opts)
+            when :float
+              rand.float(opts)
+            when :datetime
+              rand.datetime(opts)
+            else
+              raise ConfigError.new("type: `#{type}` is not defined.")
+            end
+        end
+        field_procs
+      end
     end
   end
 
@@ -85,6 +141,20 @@ module Dummer
       @chars = ('a'..'z').to_a + ('A'..'Z').to_a + ('0'..'9').to_a # no symbols and multi-bytes for now
     end
 
+    def range(range)
+      rand(range)
+    end
+
+    def any(any)
+      any[rand(any.size-1)]
+    end
+
+    def rand(arg = nil)
+      @rand.rand(arg)
+    end
+
+    # belows are data types
+
     def string(length: 8, any: nil, value: nil)
       if value
         string = value.to_s
@@ -162,17 +232,5 @@ module Dummer
         Proc.new { Time.now.strftime(format) }
       end
     end
-
-    def range(range)
-      rand(range)
-    end
-
-    def any(any)
-      any[rand(any.size-1)]
-    end
-
-    def rand(arg = nil)
-      @rand.rand(arg)
-    end
   end
 end

data/lib/dummer/setting.rb

@@ -1,12 +1,16 @@
 module Dummer
   class Setting
     attr_accessor :rate, :output, :labeled, :delimiter, :fields, :workers, :message, :input
+    attr_accessor :host, :port, :tag
 
     def initialize
       @rate = 500
       @output = STDOUT
+      @host = nil
+      @port = 24224
       @labeled = true
       @delimiter = "\t"
+      @tag = {type: :string, value: "dummer"}
       @fields = nil
       @workers = 1
       @message = "time:2013-11-25 00:23:52 +0900\tlevel:ERROR\tmethod:POST\turi:/api/v1/people\treqtime:3.1983877060667103\n"

data/lib/dummer/version.rb

@@ -1,3 +1,3 @@
 module Dummer
-  VERSION = "0.3.0"
+  VERSION = "0.3.1"
 end

data/lib/dummer/worker.rb

@@ -9,16 +9,30 @@ module Dummer
     end
 
     def reload
-      @generator = Generator.new(config[:setting])
-      @rate = config[:setting].rate
+      setting = config[:setting]
+      @generator = Generator.new(setting)
+      @rate = setting.rate
 
-      output = config[:setting].output
-      if output.respond_to?(:write) and output.respond_to?(:close)
-        @output = output
+      if host = setting.host and port = setting.port
+        require 'fluent-logger'
+        @client = Fluent::Logger::FluentLogger.new(nil, :host => host, :port => port)
+      elsif output = setting.output
+        if output.respond_to?(:write) and output.respond_to?(:close)
+          @file = output
+        else
+          @file = open(output, (File::WRONLY | File::APPEND | File::CREAT))
+          @file.sync = true
+        end
       else
-        @output = open(output, (File::WRONLY | File::APPEND | File::CREAT))
-        @output.sync = true
+        raise ConfigError.new("Config parameter `output`, or `host` and `port` do not exist")
       end
+
+      @write_proc =
+        if @client
+          Proc.new {|num| num.times { @client.post(@generator.tag, @generator.record) } }
+        else # @file
+          Proc.new {|num| num.times { @file.write @generator.message } }
+        end
     end
 
     def run
@@ -28,16 +42,16 @@ module Dummer
         current_time = Time.now.to_i
         BIN_NUM.times do
           break unless (!@stop && Time.now.to_i <= current_time)
-          wait(0.1) { write(batch_num) }
+          wait(0.1) { @write_proc.call(batch_num) }
         end
-        write(residual_num)
+        @write_proc.call(residual_num)
         # wait for next second
         while !@stop && Time.now.to_i <= current_time
           sleep 0.01
         end
       end
     ensure
-      @output.close
+      @file.close if @file
     end
 
     def stop
@@ -46,10 +60,6 @@ module Dummer
 
 private
 
-    def write(num)
-      num.times { @output.write @generator.generate }
-    end
-
     def wait(time)
       start_time = Time.now
       yield

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dummer
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - sonots
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-24 00:00:00.000000000 Z
+date: 2014-01-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -38,6 +38,20 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: fluent-logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -129,6 +143,7 @@ files:
 - bin/dummer_simple
 - bin/dummer_yes
 - dummer.gemspec
+- example/fluent_logger.conf
 - example/input.conf
 - example/input.txt
 - example/message.conf