dummer 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.gitignore +28 -0
- data/.travis.yml +4 -0
- data/CHANGELOG.md +63 -0
- data/Gemfile +3 -0
- data/LICENSE.txt +22 -0
- data/README.md +279 -0
- data/Rakefile +15 -0
- data/bin/dummer +16 -0
- data/bin/dummer_simple +81 -0
- data/bin/dummer_yes +36 -0
- data/dummer.gemspec +29 -0
- data/example/input.conf +6 -0
- data/example/input.txt +10 -0
- data/example/message.conf +5 -0
- data/example/sample.conf +13 -0
- data/lib/dummer.rb +6 -0
- data/lib/dummer/cli.rb +96 -0
- data/lib/dummer/dsl.rb +28 -0
- data/lib/dummer/error.rb +3 -0
- data/lib/dummer/generator.rb +178 -0
- data/lib/dummer/setting.rb +16 -0
- data/lib/dummer/version.rb +3 -0
- data/lib/dummer/worker.rb +60 -0
- data/lib/ext/hash/except.rb +15 -0
- data/lib/ext/hash/keys.rb +138 -0
- data/lib/ext/hash/slice.rb +42 -0
- metadata +171 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: afa2183e1b551764a566ecc80ae1c35eeb5f9459
|
4
|
+
data.tar.gz: 0a4b5914330054a8e70c1f27b7b1e43d2ac13787
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 8b5253ad74c8bd4900fc2a060ef67bb8f5bee98a431848279961bc8fd29e25a9f37f2210ef3fe865a8a4f38a67a5602adff0bd83e660641369440143b9434934
|
7
|
+
data.tar.gz: f32cf508096d457d70be676593729dc8893a5d3c91370d387707750200621cb14f0dba8d766dcad8b14ea5cb4eae28f0d4f167c0cf0b7c8b93a58c2bae673618
|
data/.gitignore
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
*.gem
|
2
|
+
*.rbc
|
3
|
+
~*
|
4
|
+
#*
|
5
|
+
*~
|
6
|
+
.bundle
|
7
|
+
Gemfile.lock
|
8
|
+
.rbenv-version
|
9
|
+
.ruby-version
|
10
|
+
vendor
|
11
|
+
doc/*
|
12
|
+
tmp/*
|
13
|
+
.yardoc
|
14
|
+
pkg
|
15
|
+
.config
|
16
|
+
.yardoc
|
17
|
+
InstalledFiles
|
18
|
+
_yardoc
|
19
|
+
coverage
|
20
|
+
doc/
|
21
|
+
lib/bundler/man
|
22
|
+
rdoc
|
23
|
+
spec/reports
|
24
|
+
test/tmp
|
25
|
+
test/version_tmp
|
26
|
+
dummer.pid
|
27
|
+
dummer.log
|
28
|
+
dummy.log
|
data/.travis.yml
ADDED
data/CHANGELOG.md
ADDED
@@ -0,0 +1,63 @@
|
|
1
|
+
### 0.3.0 (2014/01/25)
|
2
|
+
|
3
|
+
Big Changes:
|
4
|
+
|
5
|
+
* Rename application name from `dummy_log_generator` to `dummer`. `dummy_log_generator` was too long.
|
6
|
+
|
7
|
+
### 0.2.3 (2014/01/22)
|
8
|
+
|
9
|
+
Fixes:
|
10
|
+
|
11
|
+
* Fix datetime type format (thanks xcezx)
|
12
|
+
|
13
|
+
### 0.2.2 (2014/01/15)
|
14
|
+
|
15
|
+
Enhancement:
|
16
|
+
|
17
|
+
* Add CLI options to `dummer` command
|
18
|
+
|
19
|
+
### 0.2.1
|
20
|
+
|
21
|
+
Change:
|
22
|
+
|
23
|
+
* Change the default behavior to use the default `message` rather than `field`.
|
24
|
+
|
25
|
+
### 0.2.0
|
26
|
+
|
27
|
+
Enhancement:
|
28
|
+
|
29
|
+
* Add `message` option to write only a specific message
|
30
|
+
* Add `input` option to write messges by reading lines of an input file in roration
|
31
|
+
* Add `dummer_simple` command
|
32
|
+
* Add `dummer_yes` command
|
33
|
+
|
34
|
+
### 0.1.0
|
35
|
+
|
36
|
+
Enhancement:
|
37
|
+
|
38
|
+
* Support to output into a file (`output` option)
|
39
|
+
* No dependency on active_support
|
40
|
+
* Speed-up (Proc-based evaluation)
|
41
|
+
* Support parallel (`workers` option)
|
42
|
+
|
43
|
+
### 0.0.4
|
44
|
+
|
45
|
+
Enhancement:
|
46
|
+
|
47
|
+
* Add `format` option to `string`, `integer`, `float` data type
|
48
|
+
|
49
|
+
### 0.0.3
|
50
|
+
|
51
|
+
Enhancement:
|
52
|
+
|
53
|
+
* Add `value` option to specify fixed value
|
54
|
+
|
55
|
+
### 0.0.2
|
56
|
+
|
57
|
+
Enhancement:
|
58
|
+
|
59
|
+
* Add `countup: true` suppert
|
60
|
+
|
61
|
+
### 0.0.1
|
62
|
+
|
63
|
+
First version
|
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2013 sonots
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,279 @@
|
|
1
|
+
# Dummer
|
2
|
+
|
3
|
+
#### NOTE: `dummy_log_generator` was renamed to `dummer` which is easier to type, yay!
|
4
|
+
|
5
|
+
Dummer is a set of tools to generate dummy log data for Fluentd benchmark.
|
6
|
+
|
7
|
+
This gem includes three executable commands
|
8
|
+
|
9
|
+
1. dummer
|
10
|
+
2. dummer\_simple
|
11
|
+
3. dummer\_yes
|
12
|
+
|
13
|
+
## Installation
|
14
|
+
|
15
|
+
Add this line to your application's Gemfile:
|
16
|
+
|
17
|
+
gem 'dummer'
|
18
|
+
|
19
|
+
And then execute:
|
20
|
+
|
21
|
+
$ bundle
|
22
|
+
|
23
|
+
Or install it yourself as:
|
24
|
+
|
25
|
+
$ gem install dummer
|
26
|
+
|
27
|
+
Run as
|
28
|
+
|
29
|
+
$ dummer -c dummer.conf
|
30
|
+
$ dummer_simple [options]
|
31
|
+
$ dummer_yes [options]
|
32
|
+
|
33
|
+
## dummer
|
34
|
+
|
35
|
+
`dummer` allows you to
|
36
|
+
|
37
|
+
1. specify a rate of generating messages per second,
|
38
|
+
2. determine a log format, and
|
39
|
+
3. generate logs randomly
|
40
|
+
|
41
|
+
### Usage
|
42
|
+
|
43
|
+
Create a configuration file. A sample configuration is as follows:
|
44
|
+
|
45
|
+
```ruby
|
46
|
+
# dummer.conf
|
47
|
+
configure 'sample' do
|
48
|
+
output "dummy.log"
|
49
|
+
rate 500
|
50
|
+
delimiter "\t"
|
51
|
+
labeled true
|
52
|
+
field :id, type: :integer, countup: true, format: "%04d"
|
53
|
+
field :time, type: :datetime, format: "[%Y-%m-%d %H:%M:%S]", random: false
|
54
|
+
field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
|
55
|
+
field :method, type: :string, any: %w[GET POST PUT]
|
56
|
+
field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata /api/v1/messages]
|
57
|
+
field :reqtime, type: :float, range: 0.1..5.0
|
58
|
+
field :foobar, type: :string, length: 8
|
59
|
+
end
|
60
|
+
```
|
61
|
+
|
62
|
+
Running
|
63
|
+
|
64
|
+
```
|
65
|
+
$ dummer -c dummer.conf
|
66
|
+
```
|
67
|
+
|
68
|
+
Outputs to the `dummy.log` (specified by `output` parameter) file like:
|
69
|
+
|
70
|
+
```
|
71
|
+
id:0422 time:[2013-11-19 02:34:58] level:INFO method:POST uri:/api/v1/textdata reqtime:3.9726677258569842 foobar:LFK6XV1N
|
72
|
+
id:0423 time:[2013-11-19 02:34:58] level:DEBUG method:GET uri:/api/v1/people reqtime:0.49912949125272277 foobar:DcOYrONH
|
73
|
+
id:0424 time:[2013-11-19 02:34:58] level:WARN method:POST uri:/api/v1/textdata reqtime:2.930590441869852 foobar:XEZ5bQsh
|
74
|
+
```
|
75
|
+
|
76
|
+
### CLI Options
|
77
|
+
|
78
|
+
```
|
79
|
+
$ dummer help start
|
80
|
+
Usage:
|
81
|
+
dummer start
|
82
|
+
|
83
|
+
Options:
|
84
|
+
-c, [--config=CONFIG] # Config file
|
85
|
+
# Default: dummer.conf
|
86
|
+
-r, [--rate=N] # Number of generating messages per second
|
87
|
+
-o, [--output=OUTPUT] # Output file
|
88
|
+
-m, [--message=MESSAGE] # Output message
|
89
|
+
-d, [--daemonize] # Daemonize. Stop with `dummer stop`
|
90
|
+
-w, [--workers=N] # Number of parallels
|
91
|
+
[--worker-type=WORKER_TYPE]
|
92
|
+
# Default: process
|
93
|
+
-p, [--pid-path=PID_PATH]
|
94
|
+
# Default: dummer.pid
|
95
|
+
```
|
96
|
+
|
97
|
+
### Configuration Parameters
|
98
|
+
|
99
|
+
Following parameters in the configuration file are available:
|
100
|
+
|
101
|
+
* output
|
102
|
+
|
103
|
+
Specify a filename to output, or IO object (STDOUT, STDERR)
|
104
|
+
|
105
|
+
* rate
|
106
|
+
|
107
|
+
Specify how many messages to generate per second. Default: 500 msgs / sec
|
108
|
+
|
109
|
+
* workers
|
110
|
+
|
111
|
+
Specify number of processes for parallel processing.
|
112
|
+
|
113
|
+
* delimiter
|
114
|
+
|
115
|
+
Specify the delimiter between each field. Default: "\t" (Tab)
|
116
|
+
|
117
|
+
* labeled
|
118
|
+
|
119
|
+
Whether add field name as a label or not. Default: true
|
120
|
+
|
121
|
+
* field
|
122
|
+
|
123
|
+
Define data fields to generate. `message` and `input` options are ignored.
|
124
|
+
|
125
|
+
* input
|
126
|
+
|
127
|
+
Use this if you want to write messages by reading lines of an input file in rotation. `message` is ignored.
|
128
|
+
|
129
|
+
* message
|
130
|
+
|
131
|
+
Use this if you want to write only a specific message.
|
132
|
+
|
133
|
+
### Field Data Types
|
134
|
+
|
135
|
+
You can specify following data types to your `field` parameters:
|
136
|
+
|
137
|
+
* :datetime
|
138
|
+
|
139
|
+
* :format
|
140
|
+
|
141
|
+
You can specify format of datetime as `%Y-%m-%d %H:%M:%S`. See [Time#strftime](http://www.ruby-doc.org/core-2.0.0/Time.html#method-i-strftime) for details.
|
142
|
+
|
143
|
+
* :random
|
144
|
+
|
145
|
+
Generate datetime randomly. Default: false (Time.now)
|
146
|
+
|
147
|
+
* :value
|
148
|
+
|
149
|
+
You can specify a fixed Time object.
|
150
|
+
|
151
|
+
* :string
|
152
|
+
|
153
|
+
* :any
|
154
|
+
|
155
|
+
You can specify an array of strings, then the generator picks one from them randomly
|
156
|
+
|
157
|
+
* :length
|
158
|
+
|
159
|
+
You can specify the length of string to generate randomly
|
160
|
+
|
161
|
+
* :value
|
162
|
+
|
163
|
+
You can specify a fixed string
|
164
|
+
|
165
|
+
* :integer
|
166
|
+
|
167
|
+
* :format
|
168
|
+
|
169
|
+
You can specify a format of string as `%03d`.
|
170
|
+
|
171
|
+
* :range
|
172
|
+
|
173
|
+
You can specify a range of integers, then the generator picks one in the range (uniform) randomly
|
174
|
+
|
175
|
+
* :countup
|
176
|
+
|
177
|
+
Generate countup data. Default: false
|
178
|
+
|
179
|
+
* :value
|
180
|
+
|
181
|
+
You can specify a fixed integer
|
182
|
+
|
183
|
+
* :float
|
184
|
+
|
185
|
+
* :format
|
186
|
+
|
187
|
+
You can specify a format of string as `%03.1f`.
|
188
|
+
|
189
|
+
* :range
|
190
|
+
|
191
|
+
You can specify a range of float numbers, then the generator picks one in the range (uniform) randomly
|
192
|
+
|
193
|
+
* :value
|
194
|
+
|
195
|
+
You can specify a fixed float number
|
196
|
+
|
197
|
+
## dummer\_simple
|
198
|
+
|
199
|
+
I created a simple version of `dummer` since it can not achieve the maximum system I/O throughputs because of its rich features.
|
200
|
+
This simple version, `dummer_simple` could achieve the system I/O limit in my environment.
|
201
|
+
|
202
|
+
### Usage
|
203
|
+
|
204
|
+
```
|
205
|
+
$ dummer_simple [options]
|
206
|
+
```
|
207
|
+
|
208
|
+
### Options
|
209
|
+
|
210
|
+
```
|
211
|
+
Usage:
|
212
|
+
dummer_simple
|
213
|
+
|
214
|
+
Options:
|
215
|
+
[--sync] # Set `IO#sync=true`
|
216
|
+
-s, [--second=N] # Duration of running in second
|
217
|
+
# Default: 1
|
218
|
+
-p, [--parallel=N] # Number of processes to run in parallel
|
219
|
+
# Default: 1
|
220
|
+
-o, [--output=OUTPUT] # Output file
|
221
|
+
# Default: dummy.log
|
222
|
+
-i, [--input=INPUT] # Input file (Output messages by reading lines of the file in rotation)
|
223
|
+
-m, [--message=MESSAGE] # Output message
|
224
|
+
# Default: time:2013-11-20 23:39:42 +0900 level:ERROR method:POST uri:/api/v1/people reqtime:3.1983877060667103
|
225
|
+
```
|
226
|
+
|
227
|
+
## dummer\_yes
|
228
|
+
|
229
|
+
I created a wrapped version of `yes` command, `dummer_yes`, to confrim that `dummer_simple` achieves the maximum system I/O throughputs.
|
230
|
+
I do not use `dummer_yes` command anymore because I verified that `dummer_simple` achieves the I/O limit, but I will keep this command so that users can do verification experiments with it.
|
231
|
+
|
232
|
+
### Usage
|
233
|
+
|
234
|
+
```
|
235
|
+
$ dummer_yes [options]
|
236
|
+
```
|
237
|
+
|
238
|
+
### Options
|
239
|
+
|
240
|
+
```
|
241
|
+
Usage:
|
242
|
+
dummer_yes
|
243
|
+
|
244
|
+
Options:
|
245
|
+
-s, [--second=N] # Duration of running in second
|
246
|
+
# Default: 1
|
247
|
+
-p, [--parallel=N] # Number of processes to run in parallel
|
248
|
+
# Default: 1
|
249
|
+
-o, [--output=OUTPUT] # Output file
|
250
|
+
# Default: dummy.log
|
251
|
+
-m, [--message=MESSAGE] # Output message
|
252
|
+
# Default: time:2013-11-20 23:39:42 +0900 level:ERROR method:POST uri:/api/v1/people reqtime:3.1983877060667103
|
253
|
+
```
|
254
|
+
|
255
|
+
## Relatives
|
256
|
+
|
257
|
+
There is a [fluent-plugin-dummydata-producer](https://github.com/tagomoris/fluent-plugin-dummydata-producer), but I wanted to output dummy data to a log file, and I wanted a standalone tool.
|
258
|
+
|
259
|
+
## Related Articles
|
260
|
+
|
261
|
+
* [Fluentd のベンチマークテストに使える dummer (旧称 dummy_log_generator)](http://qiita.com/sonots/items/750da77a18e62852a02f)
|
262
|
+
|
263
|
+
## ToDO
|
264
|
+
|
265
|
+
1. write tests
|
266
|
+
2. Add an option to send data to fluentd directly
|
267
|
+
|
268
|
+
## Contributing
|
269
|
+
|
270
|
+
1. Fork it
|
271
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
272
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
273
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
274
|
+
5. Create new Pull Request
|
275
|
+
|
276
|
+
## Licenses
|
277
|
+
|
278
|
+
See [LICENSE.txt](LICENSE.txt)
|
279
|
+
|
data/Rakefile
ADDED
@@ -0,0 +1,15 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
require "bundler/gem_tasks"
|
3
|
+
|
4
|
+
require 'rspec/core'
|
5
|
+
require 'rspec/core/rake_task'
|
6
|
+
RSpec::Core::RakeTask.new(:spec) do |spec|
|
7
|
+
spec.pattern = FileList['spec/**/*_spec.rb']
|
8
|
+
end
|
9
|
+
task :default => :spec
|
10
|
+
|
11
|
+
desc 'Open an irb session preloaded with the gem library'
|
12
|
+
task :console do
|
13
|
+
sh 'irb -rubygems -I lib -r dummer.rb'
|
14
|
+
end
|
15
|
+
task :c => :console
|
data/bin/dummer
ADDED
@@ -0,0 +1,16 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
#
|
3
|
+
# This file was generated by Bundler.
|
4
|
+
#
|
5
|
+
# The application 'yohoushi' is installed as part of a gem, and
|
6
|
+
# this file is here to facilitate running it.
|
7
|
+
#
|
8
|
+
|
9
|
+
require 'pathname'
|
10
|
+
ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile",
|
11
|
+
Pathname.new(__FILE__).realpath)
|
12
|
+
|
13
|
+
require 'rubygems'
|
14
|
+
require 'bundler/setup'
|
15
|
+
|
16
|
+
load File.expand_path("../../lib/dummer/cli.rb", __FILE__)
|