dummy_log_generator 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 6a76a35eb0df876125713ea8afd7d3128368e73b
-  data.tar.gz: e97dc9abc1fff91fcb1e4c6b5b8179b9c25cf596
+  metadata.gz: edaffe0f73cbc9aafc71dc10db3eeb80bfed02d6
+  data.tar.gz: 807e80562f29882db31edee29b8dac692e098caa
 SHA512:
-  metadata.gz: 65200ca5573a8f275ee1dfd1c5869ce30d1e5840737ff55eefc5c45d90c0d9b7c2729976274e6d8a549183c6337d29c3484e2bbf9f4b1aa84e407752f9bd586a
-  data.tar.gz: 15921096adbdd0d1214a7a6afb0b9d29a74b5be39da3af8b4a0b01918597c671d903d044658fa98f7dfeca88289d7ad05e9efc9e007294e2392f4cf0cf2c4cab
+  metadata.gz: a229cbbf0c5863bd2d5a17417d3074d73f6f71c58f53a763a81959f4dd79f4b6e356454ccc7414a2094c90974e950f6ad18203ad6d68174cc51f7e9dfe6c1a4b
+  data.tar.gz: 7b220a87b3e6296d436198e16be03ce7fd2a2a3b391f0f8a925e474c3fb04d007006da2ee627a982751ff43956bfbc7923c5bf4f3e1f9c402133fc3239af14b9

data/.gitignore

@@ -25,4 +25,4 @@ test/tmp
 test/version_tmp
 dummy_log_generator.pid
 dummy_log_generator.log
-
+dummy.log

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+### 0.2.0
+
+Enhancement:
+
+  * Add `message` option to write only a specific message
+  * Add `input` option to write messges by reading lines of an input file in roration
+  * Add `dummy_log_generator_simple` command
+  * Add `dummy_log_generator_yes` command
+
 ### 0.1.0
 
 Enhancement:

data/Gemfile

@@ -1,4 +1,3 @@
 source 'https://rubygems.org'
 
-# Specify your gem's dependencies in dummy_log_generator.gemspec
 gemspec

data/README.md

@@ -1,6 +1,12 @@
 # DummyLogGenerator
 
-Generates dummy log data for Fluentd benchmark 
+Generates dummy log data for Fluentd benchmark. 
+
+This gem includes three executable commands
+
+1. dummy\_log\_generator
+2. dummy\_log\_generator\_simple
+3. dummy\_log\_generator\_yes
 
 ## Installation
 
@@ -19,10 +25,20 @@ Or install it yourself as:
 Run as
 
     $ dummy_log_generator -c dummy_data_generator.conf
+    $ dummy_log_generator_simple [options]
+    $ dummy_log_generator_yes [options]
+
+## dummy\_log\_generator
+
+`dummy_log_generator` allows you to
+
+1. specify a rate of generating messages per second, 
+2. determine a log format, and
+3. generate logs randomly
 
-## Usage
+### Usage
 
-Sample configuration is as follows:
+Create a configuration file. A sample configuration is as follows:
 
 ```ruby
 # dummy_log_generator.conf
@@ -41,7 +57,13 @@ configure 'sample' do
 end 
 ```
 
-Running dummy_log_generator outputs to `dummy.log` like
+Running
+
+```
+$ dummy_log_genrator -c dummy_log_generator.conf
+```
+
+Outputs to the `dummy.log` (specified by `output` parameter) file like: 
 
 ```
 id:422  time:[2013-11-19 02:34:58]  level:INFO  method:POST uri:/api/v1/textdata  reqtime:3.9726677258569842  foobar:LFK6XV1N
@@ -49,9 +71,9 @@ id:423  time:[2013-11-19 02:34:58]  level:DEBUG method:GET  uri:/api/v1/people
 id:424  time:[2013-11-19 02:34:58]  level:WARN  method:POST uri:/api/v1/textdata  reqtime:2.930590441869852   foobar:XEZ5bQsh
 ```
 
-## Configuration Parameters
+### Configuration Parameters
 
-Following parameters for configuration is available
+Following parameters in the configuration file are available:
 
 * output
 
@@ -61,6 +83,10 @@ Following parameters for configuration is available
 
     Specify how many messages to generate per second. Default: 500 msgs / sec
 
+* workers
+
+    Specify number of processes for parallel processing. 
+
 * delimiter
 
     Specify the delimiter between each field. Default: "\t" (Tab)
@@ -73,7 +99,15 @@ Following parameters for configuration is available
 
     Define data fields to generate
 
-## Field Data Types
+* message
+
+    Use this if you want to write only a specific message. `field` is ignored. 
+
+* input
+
+    Use this if you want to write messages by reading lines of an input file in rotation. `field` and `message` are ignored. 
+
+### Field Data Types
 
 You can specify following data types to your `field` parameters:
 
@@ -137,6 +171,64 @@ You can specify following data types to your `field` parameters:
 
     You can specify a fixed float number
 
+## dummy\_log\_generator\_simple
+
+I created a simple version of `dummy_log_generator` since it can not achieve the maximum system I/O throughputs because of its rich features.
+This simple version, `dummy_log_generator_simple` could  achieve the system I/O limit in my environment. 
+
+### Usage
+
+```
+$ dummy_log_genrator_simple [options]
+```
+
+### Options
+
+```
+Usage:
+  dummy_log_generator_simple
+
+Options:
+      [--sync]             # Use fsync(2) or not
+  -s, [--second=N]         # Duration of running in second
+                           # Default: 1
+  -p, [--parallel=N]       # Number of processes to run in parallel
+                           # Default: 1
+  -o, [--output=OUTPUT]    # Output file
+                           # Default: dummy.log
+  -i, [--input=INPUT]      # Input file (Output messages by reading lines of the file in rotation)
+  -m, [--message=MESSAGE]  # Output message
+                           # Default: time:2013-11-20 23:39:42 +0900    level:ERROR     method:POST     uri:/api/v1/people      reqtime:3.1983877060667103
+```
+
+## dummy\_log\_generator\_yes
+
+I created a wrapped version of `yes` command, `dummy_log_generator_yes`, to confrim that `dummy_log_generator_simple` achieves the maximum system I/O throughputs. 
+I do not use `dummy_log_generator_yes` command anymore because I verified that `dummy_log_generator_simple` achieves the I/O limit, but I will keep this command so that users can do verification experiments with it. 
+
+### Usage
+
+```
+$ dummy_log_genrator_yes [options]
+```
+
+### Options
+
+```
+Usage:
+  dummy_log_generator_yes
+
+Options:
+  -s, [--second=N]         # Duration of running in second
+                           # Default: 1
+  -p, [--parallel=N]       # Number of processes to run in parallel
+                           # Default: 1
+  -o, [--output=OUTPUT]    # Output file
+                           # Default: dummy.log
+  -m, [--message=MESSAGE]  # Output message
+                           # Default: time:2013-11-20 23:39:42 +0900  level:ERROR method:POST uri:/api/v1/people  reqtime:3.1983877060667103
+```
+
 ## Relatives
 
 There is a [fluent-plugin-dummydata-producer](https://github.com/tagomoris/fluent-plugin-dummydata-producer), but I wanted to output dummy data to a log file, and I wanted a standalone tool.

data/bin/dummy_log_generator_simple

@@ -0,0 +1,81 @@
+#!/usr/bin/env ruby
+require 'thor'
+
+module DummyLogGeneratorSimple
+  class CLI < Thor
+    default_command :start
+    desc "start", "Start a dummy_log_generator"
+    option :sync,                         :type => :boolean, :desc => 'Use fsync(2) or not'
+    option :second,   :aliases => ["-s"], :type => :numeric, :default => 1, :desc => 'Duration of running in second'
+    option :parallel, :aliases => ["-p"], :type => :numeric, :default => 1, :desc => 'Number of processes to run in parallel'
+    option :output,   :aliases => ["-o"], :type => :string,  :default => 'dummy.log', :desc => 'Output file'
+    option :input,    :aliases => ["-i"], :type => :string,  :desc => 'Input file (Output messages by reading lines of the file in rotation)'
+    option :message,  :aliases => ["-m"], :type => :string,  :desc => 'Output message',
+      :default => "time:2013-11-20 23:39:42 +0900\tlevel:ERROR\tmethod:POST\turi:/api/v1/people\treqtime:3.1983877060667103\n"
+    def start
+      sync     = @options[:sync]
+      parallel = @options[:parallel]
+      output   = @options[:output]
+      message  = "#{@options[:message].chomp}\n"
+      messages = nil
+      if input = @options[:input] and (messages = readlines(input)).nil?
+        STDERR.puts "Input file `#{input}` does not exist or is not readable."
+        exit 1
+      end
+
+      open(output, (File::WRONLY | File::APPEND | File::CREAT)) do |out_file|
+        out_file.sync = true if sync
+        if messages
+          wait_and_kill { write_messages(out_file, messages, parallel) }
+        else
+          wait_and_kill { write_message(out_file, message, parallel) }
+        end
+      end
+    end
+
+    no_commands {
+      def wait_and_kill(&block)
+        second = Time.now + @options[:second]
+        pids = yield # fork
+        while Time.now < second do; sleep 0.01; end
+        pids.each {|pid| Process.kill(:TERM, pid) }
+      end
+
+      def write_message(out_file, message, parallel = 1)
+        parallel.times.map do
+          Process.fork do
+            while true do
+              out_file.write message
+            end
+          end
+        end
+      end
+
+      def write_messages(out_file, messages, parallel = 1)
+        size = messages.size
+        parallel.times.map do
+          Process.fork do
+            idx = -1 # create this in child-process not to be shared
+            while true do
+              idx = (idx + 1) % size
+              out_file.write messages[idx]
+            end
+          end
+        end
+      end
+
+      def readlines(input)
+        messages = nil
+        begin
+          open(input) do |in_file|
+            messages = in_file.readlines
+          end
+        rescue Errno::ENOENT
+        end
+        messages
+      end
+    }
+  end
+end
+
+DummyLogGeneratorSimple::CLI.start(ARGV)

data/bin/dummy_log_generator_yes

@@ -0,0 +1,36 @@
+#!/usr/bin/env ruby
+require 'thor'
+
+module DummyLogGeneratorYes
+  class CLI < Thor
+    default_command :start
+    desc "start", "Start a dummy_log_generator"
+    option :second,   :aliases => ["-s"], :type => :numeric, :default => 1, :desc => 'Duration of running in second'
+    option :parallel, :aliases => ["-p"], :type => :numeric, :default => 1, :desc => 'Number of processes to run in parallel'
+    option :output,   :aliases => ["-o"], :type => :string,  :default => 'dummy.log', :desc => 'Output file'
+    option :message,  :aliases => ["-m"], :type => :string,  :desc => 'Output message',
+      :default => "time:2013-11-20 23:39:42 +0900\tlevel:ERROR\tmethod:POST\turi:/api/v1/people\treqtime:3.1983877060667103\n"
+    def start
+      parallel = @options[:parallel]
+      output   = @options[:output]
+      message  = "#{@options[:message].chomp}\n"
+      wait_and_kill {
+        parallel.times.map do
+          spawn("yes #{message}", :out => [output, (File::WRONLY | File::APPEND | File::CREAT)])
+        end
+      }
+    end
+
+    no_commands {
+      def wait_and_kill(&block)
+        second = Time.now + @options[:second]
+        pids = yield # fork
+        while Time.now < second do; sleep 0.01; end
+        pids.each {|pid| Process.kill(:TERM, pid) }
+      end
+    }
+  end
+end
+
+DummyLogGeneratorYes::CLI.start(ARGV)
+

data/lib/dummy_log_generator/cli.rb

@@ -15,6 +15,7 @@ module DummyLogGenerator
 
     desc "start", "Start a dummy_log_generator"
     option :config,      :aliases => ["-c"], :type => :string, :default => 'dummy_log_generator.conf'
+    option :rate,        :aliases => ["-r"], :type => :numeric
     # options for serverengine
     option :daemonize,   :aliases => ["-d"], :type => :boolean
     option :workers,     :aliases => ["-w"], :type => :numeric
@@ -24,6 +25,7 @@ module DummyLogGenerator
       if options[:config] && File.exists?(options[:config])
         dsl = instance_eval(File.read(options[:config]), options[:config])
         @options[:setting] = dsl.setting
+        @options[:rate]    ||= dsl.setting.rate
         # options for serverengine
         @options[:workers] ||= dsl.setting.workers
       end

data/lib/dummy_log_generator/dsl.rb

@@ -6,29 +6,17 @@ module DummyLogGenerator
       @setting = Setting.new
     end
 
-    def rate(rate)
-      setting.rate = rate
-    end
-
-    def output(output)
-      setting.output = output
+    def method_missing(name, *args)
+      if @setting.respond_to?("#{name}=")
+        @setting.__send__("#{name}=", *args)
+      else
+        raise ConfigError.new("Config parameter `#{name}` does not exist")
+      end
     end
 
     def field(name, opts)
       setting.fields[name] = opts
     end
-
-    def delimiter(delimiter)
-      setting.delimiter = delimiter
-    end
-
-    def labeled(labeled)
-      setting.labeled = labeled
-    end
-
-    def workers(workers)
-      setting.workers = workers
-    end
   end
 end
 

data/lib/dummy_log_generator/generator.rb

@@ -1,44 +1,80 @@
 module DummyLogGenerator
   class Generator
     def initialize(setting)
-      prepare_format_proc(setting.labeled, setting.delimiter)
-      prepare_field_procs(setting.fields)
+      @message_proc =
+        if input = setting.input
+          prepare_message_proc_for_input(input)
+        elsif message = setting.message
+          prepare_message_proc_for_message(message)
+        else
+          fields, labeled, delimiter = setting.fields, setting.labeled, setting.delimiter
+          prepare_message_proc_for_fields(fields, labeled, delimiter)
+        end
     end
 
-    def prepare_format_proc(labeled, delimiter)
-      @format_proc =
-        if labeled
-          Proc.new {|fields| fields.map {|key, val| "#{key}:#{val}" }.join(delimiter) }
-        else
-          Proc.new {|fields| fields.values.join(delimiter) }
+    def prepare_message_proc_for_input(input)
+      messages = nil
+      begin
+        open(input) do |in_file|
+          messages = in_file.readlines
         end
+      rescue Errno::ENOENT
+        raise ConfigError.new("Input file `#{input}` is not readable")
+      end
+      idx = -1
+      size = messages.size
+      Proc.new {
+        idx = (idx + 1) % size
+        messages[idx]
+      }
+    end
+
+    def prepare_message_proc_for_message(message)
+      message = "#{message.chomp}\n"
+      Proc.new { message }
+    end
+
+    def prepare_message_proc_for_fields(fields, labeled, delimiter)
+      format_proc = prepare_format_proc(labeled, delimiter)
+      field_procs = prepare_field_procs(fields)
+
+      prev_data = {}
+      Proc.new {
+        data = {}
+        field_procs.each do |key, proc|
+          prev = prev_data[key] || -1
+          data[key] = proc.call(prev)
+        end
+        prev_data = data
+        format_proc.call(data)
+      }
+    end
+
+    def prepare_format_proc(labeled, delimiter)
+      if labeled
+        Proc.new {|fields| "#{fields.map {|key, val| "#{key}:#{val}" }.join(delimiter)}\n" }
+      else
+        Proc.new {|fields| "#{fields.values.join(delimiter)}\n" }
+      end
     end
 
     def prepare_field_procs(fields)
       rand = ::DummyLogGenerator::Random.new
-      @field_procs = {}
+      field_procs = {}
       fields.each do |key, opts|
         opts = opts.dup
         type = opts.delete(:type)
         if rand.respond_to?(type)
-          @field_procs[key] = rand.send(type, opts)
+          field_procs[key] = rand.send(type, opts)
         else
           raise ConfigError.new(type)
         end
       end
+      field_procs
     end
 
-    def generate(prev_data = {})
-      data = {}
-      @field_procs.each do |key, proc|
-        prev = prev_data[key] || -1
-        data[key] = proc.call(prev)
-      end
-      data
-    end
-
-    def format(fields)
-      @format_proc.call(fields)
+    def generate
+      @message_proc.call
     end
   end
 

data/lib/dummy_log_generator/setting.rb

@@ -1,6 +1,6 @@
 module DummyLogGenerator
   class Setting
-    attr_accessor :rate, :output, :labeled, :delimiter, :fields, :workers
+    attr_accessor :rate, :output, :labeled, :delimiter, :fields, :workers, :message, :input
 
     def initialize
       @rate = 500
@@ -9,6 +9,8 @@ module DummyLogGenerator
       @delimiter = "\t"
       @fields = {}
       @workers = 1
+      @message = nil
+      @input = nil
     end
   end
 end

data/lib/dummy_log_generator/version.rb

@@ -1,3 +1,3 @@
 module DummyLogGenerator
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/dummy_log_generator/worker.rb

@@ -2,13 +2,15 @@ require 'serverengine'
 
 module DummyLogGenerator
   module Worker
+    BIN_NUM = 10
+
     def initialize
       reload
     end
 
     def reload
       @generator = Generator.new(config[:setting])
-      @rate = config[:setting].rate
+      @rate = config[:rate]
 
       output = config[:setting].output
       if output.respond_to?(:write) and output.respond_to?(:close)
@@ -19,27 +21,19 @@ module DummyLogGenerator
       end
     end
 
-    # thanks! ref. https://github.com/tagomoris/fluent-plugin-dummydata-producer/blob/a550fd4424f71cd9227e138c3c89f600ba40a0d5/lib/fluent/plugin/in_dummydata_producer.rb#L63
     def run
-      batch_num = (@rate / 9).to_i + 1
-      prev_data  = {}
+      batch_num    = (@rate / BIN_NUM).to_i
+      residual_num = (@rate % BIN_NUM)
       while !@stop
         current_time = Time.now.to_i
-        rate_count = 0
-
-        while !@stop && rate_count < @rate && Time.now.to_i == current_time
-          batch_num.times do
-            # ToDo: what if generation is slower than I/O?
-            # We may should generate data and output in parallel
-            prev_data = @generator.generate(prev_data)
-            @output.write "#{@generator.format(prev_data)}\n"
-          end
-          rate_count += batch_num
-          sleep 0.1
+        BIN_NUM.times do
+          break unless (!@stop && Time.now.to_i <= current_time)
+          wait(0.1) { write(batch_num) }
         end
+        write(residual_num)
         # wait for next second
-        while !@stop && Time.now.to_i == current_time
-          sleep 0.04
+        while !@stop && Time.now.to_i <= current_time
+          sleep 0.01
         end
       end
     ensure
@@ -49,5 +43,18 @@ module DummyLogGenerator
     def stop
       @stop = true
     end
+
+private
+
+    def write(num)
+      num.times { @output.write @generator.generate }
+    end
+
+    def wait(time)
+      start_time = Time.now
+      yield
+      sleep_time = time - (Time.now - start_time)
+      sleep sleep_time if sleep_time > 0
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dummy_log_generator
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - sonots
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-11-19 00:00:00.000000000 Z
+date: 2013-11-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -113,6 +113,8 @@ email:
 - sonots@gmail.com
 executables:
 - dummy_log_generator
+- dummy_log_generator_simple
+- dummy_log_generator_yes
 extensions: []
 extra_rdoc_files: []
 files:
@@ -124,6 +126,8 @@ files:
 - README.md
 - Rakefile
 - bin/dummy_log_generator
+- bin/dummy_log_generator_simple
+- bin/dummy_log_generator_yes
 - dummy_log_generator.conf
 - dummy_log_generator.gemspec
 - lib/dummy_log_generator.rb