dummer 0.3.0 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: afa2183e1b551764a566ecc80ae1c35eeb5f9459
4
- data.tar.gz: 0a4b5914330054a8e70c1f27b7b1e43d2ac13787
3
+ metadata.gz: e7473e82a0102c5445a4e1be36cf0059e05288fe
4
+ data.tar.gz: 17f8f33068776fba205d0190aff463798a29c0e7
5
5
  SHA512:
6
- metadata.gz: 8b5253ad74c8bd4900fc2a060ef67bb8f5bee98a431848279961bc8fd29e25a9f37f2210ef3fe865a8a4f38a67a5602adff0bd83e660641369440143b9434934
7
- data.tar.gz: f32cf508096d457d70be676593729dc8893a5d3c91370d387707750200621cb14f0dba8d766dcad8b14ea5cb4eae28f0d4f167c0cf0b7c8b93a58c2bae673618
6
+ metadata.gz: 67b31312c2289c14eb15d5ff5b1e21405fe1547d8b8be607ab9b259a02a5d312ffd3c0dd244b94611fac2637296738486ac3bc71ea356561d857f3a5995a9e07
7
+ data.tar.gz: f7498d62588d69aa3ab2489a226eedb6494fc0567b29abde08ea6203dc52f586bc3e0e223cf19eceb687fd0a37bcf7167ce144f70931af47a298785b83ce3de2
@@ -1,3 +1,9 @@
1
+ ### 0.3.1 (2014/01/31)
2
+
3
+ Enhancement:
4
+
5
+ * Support posting data to Fluentd process directly
6
+
1
7
  ### 0.3.0 (2014/01/25)
2
8
 
3
9
  Big Changes:
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Dummer
2
2
 
3
- #### NOTE: `dummy_log_generator` was renamed to `dummer` which is easier to type, yay!
3
+ #### NOTE: `dummy_log_generator` was renamed to `dummer`
4
4
 
5
5
  Dummer is a set of tools to generate dummy log data for Fluentd benchmark.
6
6
 
@@ -38,7 +38,7 @@ Run as
38
38
  2. determine a log format, and
39
39
  3. generate logs randomly
40
40
 
41
- ### Usage
41
+ ### Usage (1) - Write to a file
42
42
 
43
43
  Create a configuration file. A sample configuration is as follows:
44
44
 
@@ -73,8 +73,47 @@ id:0423 time:[2013-11-19 02:34:58] level:DEBUG method:GET uri:/api/v1/people
73
73
  id:0424 time:[2013-11-19 02:34:58] level:WARN method:POST uri:/api/v1/textdata reqtime:2.930590441869852 foobar:XEZ5bQsh
74
74
  ```
75
75
 
76
+ ### Usage (2) - Post to Fluentd process
77
+
78
+ (experimental)
79
+
80
+ Create a configuration file. Assume that a fluentd process is running on localhost:20000.
81
+ A sample configuration is as follows:
82
+
83
+ ```ruby
84
+ # dummer.conf
85
+ configure 'sample' do
86
+ host "localhost" # define `host` and `port` instead of `output`
87
+ port 20000
88
+ rate 500
89
+ tag type: string, any: %w[raw.syslog raw.message raw.nginx] # configure tag
90
+ field :id, type: :integer, countup: true, format: "%04d"
91
+ field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
92
+ field :method, type: :string, any: %w[GET POST PUT]
93
+ field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata /api/v1/messages]
94
+ field :reqtime, type: :float, range: 0.1..5.0
95
+ field :foobar, type: :string, length: 8
96
+ end
97
+ ```
98
+
99
+ Running
100
+
101
+ ```
102
+ $ dummer -c dummer.conf
103
+ ```
104
+
105
+ Data is posted to fluentd process like (below is the fluentd log generated by out_stdout)
106
+
107
+ ```
108
+ 2014-01-31 00:55:32 +0900 raw.message: {"id":"1377","level":"INFO","method":"POST","uri":"/api/v1/people","reqtime":1.678867810409548,"foobar":"paOIWxhQ"}
109
+ 2014-01-31 00:55:32 +0900 raw.syslog: {"id":"1378","level":"INFO","method":"GET","uri":"/api/v1/people","reqtime":4.8412816521873445,"foobar":"kUvnC0MK"}
110
+ 2014-01-31 00:55:32 +0900 raw.message: {"id":"1379","level":"WARN","method":"GET","uri":"/api/v1/people","reqtime":3.584494903998221,"foobar":"KD78mpjX"}
111
+ ```
112
+
76
113
  ### CLI Options
77
114
 
115
+ You can specify some configuration parameters on CLI without writing them on a configuration file.
116
+
78
117
  ```
79
118
  $ dummer help start
80
119
  Usage:
@@ -85,6 +124,8 @@ Options:
85
124
  # Default: dummer.conf
86
125
  -r, [--rate=N] # Number of generating messages per second
87
126
  -o, [--output=OUTPUT] # Output file
127
+ -h, [--host=HOST] # Host of fluentd process
128
+ -p, [--port=N] # Port of fluentd process
88
129
  -m, [--message=MESSAGE] # Output message
89
130
  -d, [--daemonize] # Daemonize. Stop with `dummer stop`
90
131
  -w, [--workers=N] # Number of parallels
@@ -102,6 +143,14 @@ Following parameters in the configuration file are available:
102
143
 
103
144
  Specify a filename to output, or IO object (STDOUT, STDERR)
104
145
 
146
+ * host
147
+
148
+ Post a data to a fluentd process on the specified host. Either of `output` or `host` can be specified.
149
+
150
+ * port
151
+
152
+ Post a data to a fluentd process on the specified post. Default is 24224.
153
+
105
154
  * rate
106
155
 
107
156
  Specify how many messages to generate per second. Default: 500 msgs / sec
@@ -118,6 +167,10 @@ Following parameters in the configuration file are available:
118
167
 
119
168
  Whether add field name as a label or not. Default: true
120
169
 
170
+ * tag
171
+
172
+ Define tag field to generate. This is effective only for posting data to fluentd process with `host` and `port`.
173
+
121
174
  * field
122
175
 
123
176
  Define data fields to generate. `message` and `input` options are ignored.
@@ -132,7 +185,7 @@ Following parameters in the configuration file are available:
132
185
 
133
186
  ### Field Data Types
134
187
 
135
- You can specify following data types to your `field` parameters:
188
+ You can specify following data types to your `tag` and `field` parameters:
136
189
 
137
190
  * :datetime
138
191
 
@@ -199,6 +252,8 @@ You can specify following data types to your `field` parameters:
199
252
  I created a simple version of `dummer` since it can not achieve the maximum system I/O throughputs because of its rich features.
200
253
  This simple version, `dummer_simple` could achieve the system I/O limit in my environment.
201
254
 
255
+ Sorry, but this simple script cannot post data to fluentd process, supports only writing to a file.
256
+
202
257
  ### Usage
203
258
 
204
259
  ```
@@ -227,6 +282,7 @@ Options:
227
282
  ## dummer\_yes
228
283
 
229
284
  I created a wrapped version of `yes` command, `dummer_yes`, to confrim that `dummer_simple` achieves the maximum system I/O throughputs.
285
+
230
286
  I do not use `dummer_yes` command anymore because I verified that `dummer_simple` achieves the I/O limit, but I will keep this command so that users can do verification experiments with it.
231
287
 
232
288
  ### Usage
@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
20
20
 
21
21
  spec.add_dependency "thor"
22
22
  spec.add_dependency "serverengine"
23
+ spec.add_dependency "fluent-logger"
23
24
 
24
25
  spec.add_development_dependency "bundler", "~> 1.3"
25
26
  spec.add_development_dependency "rake"
@@ -0,0 +1,12 @@
1
+ configure 'sample' do
2
+ host "localhost"
3
+ port 20000
4
+ rate 500
5
+ tag type: :string, any: %w[raw.syslog raw.message raw.nginx]
6
+ field :id, type: :integer, countup: true, format: "%04d"
7
+ field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
8
+ field :method, type: :string, any: %w[GET POST PUT]
9
+ field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata /api/v1/messages]
10
+ field :reqtime, type: :float, range: 0.1..5.0
11
+ field :foobar, type: :string, length: 8
12
+ end
@@ -17,6 +17,8 @@ module Dummer
17
17
  option :config, :aliases => ["-c"], :type => :string, :default => 'dummer.conf', :desc => 'Config file'
18
18
  option :rate, :aliases => ["-r"], :type => :numeric, :desc => 'Number of generating messages per second'
19
19
  option :output, :aliases => ["-o"], :type => :string, :desc => 'Output file'
20
+ option :host, :aliases => ["-h"], :type => :string, :desc => 'Host of fluentd process'
21
+ option :port, :aliases => ["-p"], :type => :numeric, :desc => 'Port of fluentd process'
20
22
  option :message, :aliases => ["-m"], :type => :string, :desc => 'Output message'
21
23
  # options for serverengine
22
24
  option :daemonize, :aliases => ["-d"], :type => :boolean, :desc => 'Daemonize. Stop with `dummer stop`'
@@ -30,10 +32,12 @@ module Dummer
30
32
  else
31
33
  Dummer::Dsl.new
32
34
  end
33
- @options[:setting] = dsl.setting
34
35
  dsl.setting.rate = options[:rate] if options[:rate]
35
36
  dsl.setting.output = options[:output] if options[:output]
37
+ dsl.setting.host = options[:host] if options[:host]
38
+ dsl.setting.port = options[:port] if options[:port]
36
39
  dsl.setting.message = options[:message] if options[:message]
40
+ @options[:setting] = dsl.setting
37
41
  # options for serverengine
38
42
  @options[:workers] ||= dsl.setting.workers
39
43
 
@@ -3,79 +3,135 @@ module Dummer
3
3
  def initialize(setting)
4
4
  @message_proc =
5
5
  if fields = setting.fields
6
- labeled, delimiter = setting.labeled, setting.delimiter
7
- prepare_message_proc_for_fields(fields, labeled, delimiter)
6
+ Field.message_proc(fields, setting.labeled, setting.delimiter)
8
7
  elsif input = setting.input
9
- prepare_message_proc_for_input(input)
8
+ Input.message_proc(input)
10
9
  else
11
- message = setting.message
12
- prepare_message_proc_for_message(message)
10
+ Message.message_proc(setting.message)
13
11
  end
12
+ @record_proc =
13
+ if fields = setting.fields
14
+ Field.record_proc(fields)
15
+ elsif input = setting.input
16
+ Input.record_proc(input)
17
+ else
18
+ Message.record_proc(setting.message)
19
+ end
20
+ @tag_proc = Field.tag_proc(setting.tag)
14
21
  end
15
22
 
16
- def prepare_message_proc_for_input(input)
17
- messages = nil
18
- begin
19
- open(input) do |in_file|
20
- messages = in_file.readlines
21
- end
22
- rescue Errno::ENOENT
23
- raise ConfigError.new("Input file `#{input}` is not readable")
24
- end
25
- idx = -1
26
- size = messages.size
27
- Proc.new {
28
- idx = (idx + 1) % size
29
- messages[idx]
30
- }
23
+ # @return [String] message
24
+ def message
25
+ @message_proc.call
31
26
  end
32
27
 
33
- def prepare_message_proc_for_message(message)
34
- message = "#{message.chomp}\n"
35
- Proc.new { message }
28
+ # @return [String] tag
29
+ def tag
30
+ @tag_proc.call
36
31
  end
37
32
 
38
- def prepare_message_proc_for_fields(fields, labeled, delimiter)
39
- format_proc = prepare_format_proc(labeled, delimiter)
40
- field_procs = prepare_field_procs(fields)
33
+ # @return [Hash] record
34
+ def record
35
+ @record_proc.call
36
+ end
41
37
 
42
- prev_data = {}
43
- Proc.new {
44
- data = {}
45
- field_procs.each do |key, proc|
46
- prev = prev_data[key] || -1
47
- data[key] = proc.call(prev)
48
- end
49
- prev_data = data
50
- format_proc.call(data)
51
- }
38
+ class Message
39
+ def self.message_proc(message)
40
+ message = "#{message.chomp}\n"
41
+ Proc.new { message }
42
+ end
43
+
44
+ def self.record_proc(message)
45
+ # ToDo: implement parser
46
+ message_proc = message_proc(message)
47
+ Proc.new { { "message" => message_proc.call } }
48
+ end
52
49
  end
53
50
 
54
- def prepare_format_proc(labeled, delimiter)
55
- if labeled
56
- Proc.new {|fields| "#{fields.map {|key, val| "#{key}:#{val}" }.join(delimiter)}\n" }
57
- else
58
- Proc.new {|fields| "#{fields.values.join(delimiter)}\n" }
51
+ class Input
52
+ def self.message_proc(input)
53
+ messages = nil
54
+ begin
55
+ open(input) do |in_file|
56
+ messages = in_file.readlines
57
+ end
58
+ rescue Errno::ENOENT
59
+ raise ConfigError.new("Input file `#{input}` is not readable")
60
+ end
61
+ idx = -1
62
+ size = messages.size
63
+ Proc.new {
64
+ idx = (idx + 1) % size
65
+ messages[idx]
66
+ }
67
+ end
68
+
69
+ def self.record_proc(input)
70
+ # ToDo: implement parser
71
+ message_proc = message_proc(input)
72
+ Proc.new { { "message" => message_proc.call } }
59
73
  end
60
74
  end
61
75
 
62
- def prepare_field_procs(fields)
63
- rand = ::Dummer::Random.new
64
- field_procs = {}
65
- fields.each do |key, opts|
66
- opts = opts.dup
67
- type = opts.delete(:type)
68
- if rand.respond_to?(type)
69
- field_procs[key] = rand.send(type, opts)
76
+ class Field
77
+ def self.message_proc(fields, labeled, delimiter)
78
+ format_proc = format_proc(labeled, delimiter)
79
+ record_proc = record_proc(fields)
80
+
81
+ Proc.new {
82
+ hash = record_proc.call
83
+ format_proc.call(hash)
84
+ }
85
+ end
86
+
87
+ def self.record_proc(fields)
88
+ field_procs = field_procs(fields)
89
+
90
+ prev_data = {}
91
+ Proc.new {
92
+ data = {}
93
+ field_procs.each do |key, proc|
94
+ prev = prev_data[key] || -1
95
+ data[key] = proc.call(prev)
96
+ end
97
+ prev_data = data
98
+ }
99
+ end
100
+
101
+ def self.tag_proc(tag_opts)
102
+ field_procs({"tag" => tag_opts})["tag"]
103
+ end
104
+
105
+ def self.format_proc(labeled, delimiter)
106
+ if labeled
107
+ Proc.new {|fields| "#{fields.map {|key, val| "#{key}:#{val}" }.join(delimiter)}\n" }
70
108
  else
71
- raise ConfigError.new(type)
109
+ Proc.new {|fields| "#{fields.values.join(delimiter)}\n" }
72
110
  end
73
111
  end
74
- field_procs
75
- end
76
112
 
77
- def generate
78
- @message_proc.call
113
+ def self.field_procs(fields)
114
+ rand = ::Dummer::Random.new
115
+ field_procs = {}
116
+ fields.each do |key, opts|
117
+ opts = opts.dup
118
+ type = opts.delete(:type)
119
+ field_procs[key] =
120
+ case type
121
+ when :string
122
+ rand.string(opts)
123
+ when :integer
124
+ rand.integer(opts)
125
+ when :float
126
+ rand.float(opts)
127
+ when :datetime
128
+ rand.datetime(opts)
129
+ else
130
+ raise ConfigError.new("type: `#{type}` is not defined.")
131
+ end
132
+ end
133
+ field_procs
134
+ end
79
135
  end
80
136
  end
81
137
 
@@ -85,6 +141,20 @@ module Dummer
85
141
  @chars = ('a'..'z').to_a + ('A'..'Z').to_a + ('0'..'9').to_a # no symbols and multi-bytes for now
86
142
  end
87
143
 
144
+ def range(range)
145
+ rand(range)
146
+ end
147
+
148
+ def any(any)
149
+ any[rand(any.size-1)]
150
+ end
151
+
152
+ def rand(arg = nil)
153
+ @rand.rand(arg)
154
+ end
155
+
156
+ # belows are data types
157
+
88
158
  def string(length: 8, any: nil, value: nil)
89
159
  if value
90
160
  string = value.to_s
@@ -162,17 +232,5 @@ module Dummer
162
232
  Proc.new { Time.now.strftime(format) }
163
233
  end
164
234
  end
165
-
166
- def range(range)
167
- rand(range)
168
- end
169
-
170
- def any(any)
171
- any[rand(any.size-1)]
172
- end
173
-
174
- def rand(arg = nil)
175
- @rand.rand(arg)
176
- end
177
235
  end
178
236
  end
@@ -1,12 +1,16 @@
1
1
  module Dummer
2
2
  class Setting
3
3
  attr_accessor :rate, :output, :labeled, :delimiter, :fields, :workers, :message, :input
4
+ attr_accessor :host, :port, :tag
4
5
 
5
6
  def initialize
6
7
  @rate = 500
7
8
  @output = STDOUT
9
+ @host = nil
10
+ @port = 24224
8
11
  @labeled = true
9
12
  @delimiter = "\t"
13
+ @tag = {type: :string, value: "dummer"}
10
14
  @fields = nil
11
15
  @workers = 1
12
16
  @message = "time:2013-11-25 00:23:52 +0900\tlevel:ERROR\tmethod:POST\turi:/api/v1/people\treqtime:3.1983877060667103\n"
@@ -1,3 +1,3 @@
1
1
  module Dummer
2
- VERSION = "0.3.0"
2
+ VERSION = "0.3.1"
3
3
  end
@@ -9,16 +9,30 @@ module Dummer
9
9
  end
10
10
 
11
11
  def reload
12
- @generator = Generator.new(config[:setting])
13
- @rate = config[:setting].rate
12
+ setting = config[:setting]
13
+ @generator = Generator.new(setting)
14
+ @rate = setting.rate
14
15
 
15
- output = config[:setting].output
16
- if output.respond_to?(:write) and output.respond_to?(:close)
17
- @output = output
16
+ if host = setting.host and port = setting.port
17
+ require 'fluent-logger'
18
+ @client = Fluent::Logger::FluentLogger.new(nil, :host => host, :port => port)
19
+ elsif output = setting.output
20
+ if output.respond_to?(:write) and output.respond_to?(:close)
21
+ @file = output
22
+ else
23
+ @file = open(output, (File::WRONLY | File::APPEND | File::CREAT))
24
+ @file.sync = true
25
+ end
18
26
  else
19
- @output = open(output, (File::WRONLY | File::APPEND | File::CREAT))
20
- @output.sync = true
27
+ raise ConfigError.new("Config parameter `output`, or `host` and `port` do not exist")
21
28
  end
29
+
30
+ @write_proc =
31
+ if @client
32
+ Proc.new {|num| num.times { @client.post(@generator.tag, @generator.record) } }
33
+ else # @file
34
+ Proc.new {|num| num.times { @file.write @generator.message } }
35
+ end
22
36
  end
23
37
 
24
38
  def run
@@ -28,16 +42,16 @@ module Dummer
28
42
  current_time = Time.now.to_i
29
43
  BIN_NUM.times do
30
44
  break unless (!@stop && Time.now.to_i <= current_time)
31
- wait(0.1) { write(batch_num) }
45
+ wait(0.1) { @write_proc.call(batch_num) }
32
46
  end
33
- write(residual_num)
47
+ @write_proc.call(residual_num)
34
48
  # wait for next second
35
49
  while !@stop && Time.now.to_i <= current_time
36
50
  sleep 0.01
37
51
  end
38
52
  end
39
53
  ensure
40
- @output.close
54
+ @file.close if @file
41
55
  end
42
56
 
43
57
  def stop
@@ -46,10 +60,6 @@ module Dummer
46
60
 
47
61
  private
48
62
 
49
- def write(num)
50
- num.times { @output.write @generator.generate }
51
- end
52
-
53
63
  def wait(time)
54
64
  start_time = Time.now
55
65
  yield
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: dummer
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.0
4
+ version: 0.3.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - sonots
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2014-01-24 00:00:00.000000000 Z
11
+ date: 2014-01-30 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: thor
@@ -38,6 +38,20 @@ dependencies:
38
38
  - - '>='
39
39
  - !ruby/object:Gem::Version
40
40
  version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: fluent-logger
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - '>='
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - '>='
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
41
55
  - !ruby/object:Gem::Dependency
42
56
  name: bundler
43
57
  requirement: !ruby/object:Gem::Requirement
@@ -129,6 +143,7 @@ files:
129
143
  - bin/dummer_simple
130
144
  - bin/dummer_yes
131
145
  - dummer.gemspec
146
+ - example/fluent_logger.conf
132
147
  - example/input.conf
133
148
  - example/input.txt
134
149
  - example/message.conf