dummer 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: afa2183e1b551764a566ecc80ae1c35eeb5f9459
4
+ data.tar.gz: 0a4b5914330054a8e70c1f27b7b1e43d2ac13787
5
+ SHA512:
6
+ metadata.gz: 8b5253ad74c8bd4900fc2a060ef67bb8f5bee98a431848279961bc8fd29e25a9f37f2210ef3fe865a8a4f38a67a5602adff0bd83e660641369440143b9434934
7
+ data.tar.gz: f32cf508096d457d70be676593729dc8893a5d3c91370d387707750200621cb14f0dba8d766dcad8b14ea5cb4eae28f0d4f167c0cf0b7c8b93a58c2bae673618
@@ -0,0 +1,28 @@
1
+ *.gem
2
+ *.rbc
3
+ ~*
4
+ #*
5
+ *~
6
+ .bundle
7
+ Gemfile.lock
8
+ .rbenv-version
9
+ .ruby-version
10
+ vendor
11
+ doc/*
12
+ tmp/*
13
+ .yardoc
14
+ pkg
15
+ .config
16
+ .yardoc
17
+ InstalledFiles
18
+ _yardoc
19
+ coverage
20
+ doc/
21
+ lib/bundler/man
22
+ rdoc
23
+ spec/reports
24
+ test/tmp
25
+ test/version_tmp
26
+ dummer.pid
27
+ dummer.log
28
+ dummy.log
@@ -0,0 +1,4 @@
1
+ rvm:
2
+ - 2.0.0
3
+ gemfile:
4
+ - Gemfile
@@ -0,0 +1,63 @@
1
+ ### 0.3.0 (2014/01/25)
2
+
3
+ Big Changes:
4
+
5
+ * Rename application name from `dummy_log_generator` to `dummer`. `dummy_log_generator` was too long.
6
+
7
+ ### 0.2.3 (2014/01/22)
8
+
9
+ Fixes:
10
+
11
+ * Fix datetime type format (thanks xcezx)
12
+
13
+ ### 0.2.2 (2014/01/15)
14
+
15
+ Enhancement:
16
+
17
+ * Add CLI options to `dummer` command
18
+
19
+ ### 0.2.1
20
+
21
+ Change:
22
+
23
+ * Change the default behavior to use the default `message` rather than `field`.
24
+
25
+ ### 0.2.0
26
+
27
+ Enhancement:
28
+
29
+ * Add `message` option to write only a specific message
30
+ * Add `input` option to write messges by reading lines of an input file in roration
31
+ * Add `dummer_simple` command
32
+ * Add `dummer_yes` command
33
+
34
+ ### 0.1.0
35
+
36
+ Enhancement:
37
+
38
+ * Support to output into a file (`output` option)
39
+ * No dependency on active_support
40
+ * Speed-up (Proc-based evaluation)
41
+ * Support parallel (`workers` option)
42
+
43
+ ### 0.0.4
44
+
45
+ Enhancement:
46
+
47
+ * Add `format` option to `string`, `integer`, `float` data type
48
+
49
+ ### 0.0.3
50
+
51
+ Enhancement:
52
+
53
+ * Add `value` option to specify fixed value
54
+
55
+ ### 0.0.2
56
+
57
+ Enhancement:
58
+
59
+ * Add `countup: true` suppert
60
+
61
+ ### 0.0.1
62
+
63
+ First version
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+
3
+ gemspec
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2013 sonots
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,279 @@
1
+ # Dummer
2
+
3
+ #### NOTE: `dummy_log_generator` was renamed to `dummer` which is easier to type, yay!
4
+
5
+ Dummer is a set of tools to generate dummy log data for Fluentd benchmark.
6
+
7
+ This gem includes three executable commands
8
+
9
+ 1. dummer
10
+ 2. dummer\_simple
11
+ 3. dummer\_yes
12
+
13
+ ## Installation
14
+
15
+ Add this line to your application's Gemfile:
16
+
17
+ gem 'dummer'
18
+
19
+ And then execute:
20
+
21
+ $ bundle
22
+
23
+ Or install it yourself as:
24
+
25
+ $ gem install dummer
26
+
27
+ Run as
28
+
29
+ $ dummer -c dummer.conf
30
+ $ dummer_simple [options]
31
+ $ dummer_yes [options]
32
+
33
+ ## dummer
34
+
35
+ `dummer` allows you to
36
+
37
+ 1. specify a rate of generating messages per second,
38
+ 2. determine a log format, and
39
+ 3. generate logs randomly
40
+
41
+ ### Usage
42
+
43
+ Create a configuration file. A sample configuration is as follows:
44
+
45
+ ```ruby
46
+ # dummer.conf
47
+ configure 'sample' do
48
+ output "dummy.log"
49
+ rate 500
50
+ delimiter "\t"
51
+ labeled true
52
+ field :id, type: :integer, countup: true, format: "%04d"
53
+ field :time, type: :datetime, format: "[%Y-%m-%d %H:%M:%S]", random: false
54
+ field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
55
+ field :method, type: :string, any: %w[GET POST PUT]
56
+ field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata /api/v1/messages]
57
+ field :reqtime, type: :float, range: 0.1..5.0
58
+ field :foobar, type: :string, length: 8
59
+ end
60
+ ```
61
+
62
+ Running
63
+
64
+ ```
65
+ $ dummer -c dummer.conf
66
+ ```
67
+
68
+ Outputs to the `dummy.log` (specified by `output` parameter) file like:
69
+
70
+ ```
71
+ id:0422 time:[2013-11-19 02:34:58] level:INFO method:POST uri:/api/v1/textdata reqtime:3.9726677258569842 foobar:LFK6XV1N
72
+ id:0423 time:[2013-11-19 02:34:58] level:DEBUG method:GET uri:/api/v1/people reqtime:0.49912949125272277 foobar:DcOYrONH
73
+ id:0424 time:[2013-11-19 02:34:58] level:WARN method:POST uri:/api/v1/textdata reqtime:2.930590441869852 foobar:XEZ5bQsh
74
+ ```
75
+
76
+ ### CLI Options
77
+
78
+ ```
79
+ $ dummer help start
80
+ Usage:
81
+ dummer start
82
+
83
+ Options:
84
+ -c, [--config=CONFIG] # Config file
85
+ # Default: dummer.conf
86
+ -r, [--rate=N] # Number of generating messages per second
87
+ -o, [--output=OUTPUT] # Output file
88
+ -m, [--message=MESSAGE] # Output message
89
+ -d, [--daemonize] # Daemonize. Stop with `dummer stop`
90
+ -w, [--workers=N] # Number of parallels
91
+ [--worker-type=WORKER_TYPE]
92
+ # Default: process
93
+ -p, [--pid-path=PID_PATH]
94
+ # Default: dummer.pid
95
+ ```
96
+
97
+ ### Configuration Parameters
98
+
99
+ Following parameters in the configuration file are available:
100
+
101
+ * output
102
+
103
+ Specify a filename to output, or IO object (STDOUT, STDERR)
104
+
105
+ * rate
106
+
107
+ Specify how many messages to generate per second. Default: 500 msgs / sec
108
+
109
+ * workers
110
+
111
+ Specify number of processes for parallel processing.
112
+
113
+ * delimiter
114
+
115
+ Specify the delimiter between each field. Default: "\t" (Tab)
116
+
117
+ * labeled
118
+
119
+ Whether add field name as a label or not. Default: true
120
+
121
+ * field
122
+
123
+ Define data fields to generate. `message` and `input` options are ignored.
124
+
125
+ * input
126
+
127
+ Use this if you want to write messages by reading lines of an input file in rotation. `message` is ignored.
128
+
129
+ * message
130
+
131
+ Use this if you want to write only a specific message.
132
+
133
+ ### Field Data Types
134
+
135
+ You can specify following data types to your `field` parameters:
136
+
137
+ * :datetime
138
+
139
+ * :format
140
+
141
+ You can specify format of datetime as `%Y-%m-%d %H:%M:%S`. See [Time#strftime](http://www.ruby-doc.org/core-2.0.0/Time.html#method-i-strftime) for details.
142
+
143
+ * :random
144
+
145
+ Generate datetime randomly. Default: false (Time.now)
146
+
147
+ * :value
148
+
149
+ You can specify a fixed Time object.
150
+
151
+ * :string
152
+
153
+ * :any
154
+
155
+ You can specify an array of strings, then the generator picks one from them randomly
156
+
157
+ * :length
158
+
159
+ You can specify the length of string to generate randomly
160
+
161
+ * :value
162
+
163
+ You can specify a fixed string
164
+
165
+ * :integer
166
+
167
+ * :format
168
+
169
+ You can specify a format of string as `%03d`.
170
+
171
+ * :range
172
+
173
+ You can specify a range of integers, then the generator picks one in the range (uniform) randomly
174
+
175
+ * :countup
176
+
177
+ Generate countup data. Default: false
178
+
179
+ * :value
180
+
181
+ You can specify a fixed integer
182
+
183
+ * :float
184
+
185
+ * :format
186
+
187
+ You can specify a format of string as `%03.1f`.
188
+
189
+ * :range
190
+
191
+ You can specify a range of float numbers, then the generator picks one in the range (uniform) randomly
192
+
193
+ * :value
194
+
195
+ You can specify a fixed float number
196
+
197
+ ## dummer\_simple
198
+
199
+ I created a simple version of `dummer` since it can not achieve the maximum system I/O throughputs because of its rich features.
200
+ This simple version, `dummer_simple` could achieve the system I/O limit in my environment.
201
+
202
+ ### Usage
203
+
204
+ ```
205
+ $ dummer_simple [options]
206
+ ```
207
+
208
+ ### Options
209
+
210
+ ```
211
+ Usage:
212
+ dummer_simple
213
+
214
+ Options:
215
+ [--sync] # Set `IO#sync=true`
216
+ -s, [--second=N] # Duration of running in second
217
+ # Default: 1
218
+ -p, [--parallel=N] # Number of processes to run in parallel
219
+ # Default: 1
220
+ -o, [--output=OUTPUT] # Output file
221
+ # Default: dummy.log
222
+ -i, [--input=INPUT] # Input file (Output messages by reading lines of the file in rotation)
223
+ -m, [--message=MESSAGE] # Output message
224
+ # Default: time:2013-11-20 23:39:42 +0900 level:ERROR method:POST uri:/api/v1/people reqtime:3.1983877060667103
225
+ ```
226
+
227
+ ## dummer\_yes
228
+
229
+ I created a wrapped version of `yes` command, `dummer_yes`, to confrim that `dummer_simple` achieves the maximum system I/O throughputs.
230
+ I do not use `dummer_yes` command anymore because I verified that `dummer_simple` achieves the I/O limit, but I will keep this command so that users can do verification experiments with it.
231
+
232
+ ### Usage
233
+
234
+ ```
235
+ $ dummer_yes [options]
236
+ ```
237
+
238
+ ### Options
239
+
240
+ ```
241
+ Usage:
242
+ dummer_yes
243
+
244
+ Options:
245
+ -s, [--second=N] # Duration of running in second
246
+ # Default: 1
247
+ -p, [--parallel=N] # Number of processes to run in parallel
248
+ # Default: 1
249
+ -o, [--output=OUTPUT] # Output file
250
+ # Default: dummy.log
251
+ -m, [--message=MESSAGE] # Output message
252
+ # Default: time:2013-11-20 23:39:42 +0900 level:ERROR method:POST uri:/api/v1/people reqtime:3.1983877060667103
253
+ ```
254
+
255
+ ## Relatives
256
+
257
+ There is a [fluent-plugin-dummydata-producer](https://github.com/tagomoris/fluent-plugin-dummydata-producer), but I wanted to output dummy data to a log file, and I wanted a standalone tool.
258
+
259
+ ## Related Articles
260
+
261
+ * [Fluentd のベンチマークテストに使える dummer (旧称 dummy_log_generator)](http://qiita.com/sonots/items/750da77a18e62852a02f)
262
+
263
+ ## ToDO
264
+
265
+ 1. write tests
266
+ 2. Add an option to send data to fluentd directly
267
+
268
+ ## Contributing
269
+
270
+ 1. Fork it
271
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
272
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
273
+ 4. Push to the branch (`git push origin my-new-feature`)
274
+ 5. Create new Pull Request
275
+
276
+ ## Licenses
277
+
278
+ See [LICENSE.txt](LICENSE.txt)
279
+
@@ -0,0 +1,15 @@
1
+ # encoding: utf-8
2
+ require "bundler/gem_tasks"
3
+
4
+ require 'rspec/core'
5
+ require 'rspec/core/rake_task'
6
+ RSpec::Core::RakeTask.new(:spec) do |spec|
7
+ spec.pattern = FileList['spec/**/*_spec.rb']
8
+ end
9
+ task :default => :spec
10
+
11
+ desc 'Open an irb session preloaded with the gem library'
12
+ task :console do
13
+ sh 'irb -rubygems -I lib -r dummer.rb'
14
+ end
15
+ task :c => :console
@@ -0,0 +1,16 @@
1
+ #!/usr/bin/env ruby
2
+ #
3
+ # This file was generated by Bundler.
4
+ #
5
+ # The application 'yohoushi' is installed as part of a gem, and
6
+ # this file is here to facilitate running it.
7
+ #
8
+
9
+ require 'pathname'
10
+ ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile",
11
+ Pathname.new(__FILE__).realpath)
12
+
13
+ require 'rubygems'
14
+ require 'bundler/setup'
15
+
16
+ load File.expand_path("../../lib/dummer/cli.rb", __FILE__)