benchmark_driver 0.10.13 → 0.10.14
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -0
- data/README.md +200 -5
- data/images/optcarrot.png +0 -0
- data/lib/benchmark_driver/output/compare.rb +2 -2
- data/lib/benchmark_driver/runner/time.rb +1 -1
- data/lib/benchmark_driver/version.rb +1 -1
- metadata +4 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: e8842fe9609d09ce46061bf5ad452219ec98f1b24ef34020698500160e9823cd
|
4
|
+
data.tar.gz: 69e3714e9bcc55053d198807b17edcc952e6682da5eed337d9a073df85891d79
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b3c190db1fe084198706797c8d22cb7dba2a077d68e9d02d9466f9c5d26c206a4c03d3db6824d8c2ee3c09e90caf32cbcdd8c8067e75f254bf7c2078d45054f9
|
7
|
+
data.tar.gz: 3b50ec42bbe545db63347f5379f79f84b5b35b109728c8600725d642f3841098dfde78d820badcdf3a3836f4f26d2a6490e463e671c1c367dcbd3e76eb48e2f9
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -145,14 +145,16 @@ Comparison:
|
|
145
145
|
2.4.1: 1889805.6 i/s - 1.72x slower
|
146
146
|
```
|
147
147
|
|
148
|
-
## Output
|
148
|
+
## Output Options
|
149
149
|
|
150
150
|
By default, there are following output options.
|
151
151
|
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
152
|
+
| Output | Description |
|
153
|
+
|:-------|:------------|
|
154
|
+
| compare | benchmark-ips's `compare!`-like output (default) |
|
155
|
+
| simple | ruby's original `benchmark/driver.rb`-like simple output |
|
156
|
+
| markdown | output in markdown table |
|
157
|
+
| record | serialize results in `benchmark_driver.record.yml`, to change outputs later as you like |
|
156
158
|
|
157
159
|
With `benchmark-driver` CLI, you can specify it with `-o [output]` or `--output [output]`.
|
158
160
|
|
@@ -166,6 +168,199 @@ Benchmark.driver do |x|
|
|
166
168
|
end
|
167
169
|
```
|
168
170
|
|
171
|
+
### compare
|
172
|
+
|
173
|
+
```
|
174
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --output compare --rbenv '2.4.2;2.5.0'
|
175
|
+
Calculating -------------------------------------
|
176
|
+
2.4.2 2.5.0
|
177
|
+
empty 195.957M 129.970M i/s - 20.000M times in 0.102063s 0.153882s
|
178
|
+
blank 66.554M 55.630M i/s - 20.000M times in 0.300507s 0.359519s
|
179
|
+
|
180
|
+
Comparison:
|
181
|
+
empty
|
182
|
+
2.4.2: 195957398.5 i/s
|
183
|
+
2.5.0: 129969716.9 i/s - 1.51x slower
|
184
|
+
|
185
|
+
blank
|
186
|
+
2.4.2: 66554190.0 i/s
|
187
|
+
2.5.0: 55629883.2 i/s - 1.20x slower
|
188
|
+
```
|
189
|
+
|
190
|
+
### simple
|
191
|
+
|
192
|
+
```
|
193
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --output simple --rbenv '2.4.2;2.5.0'
|
194
|
+
benchmark results (i/s):
|
195
|
+
2.4.2 2.5.0
|
196
|
+
empty 184.084M 117.319M
|
197
|
+
blank 65.843M 62.093M
|
198
|
+
```
|
199
|
+
|
200
|
+
### markdown
|
201
|
+
|
202
|
+
```
|
203
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --output markdown --rbenv '2.4.2;2.5.0'
|
204
|
+
```
|
205
|
+
|
206
|
+
benchmark results (i/s)
|
207
|
+
|
208
|
+
| | 2.4.2| 2.5.0|
|
209
|
+
|:------|:-------|:-------|
|
210
|
+
|empty |187.296M|117.662M|
|
211
|
+
|blank | 58.895M| 58.852M|
|
212
|
+
|
213
|
+
### record
|
214
|
+
|
215
|
+
Measure first, output with various formats later.
|
216
|
+
|
217
|
+
```
|
218
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --output record --rbenv '2.4.2;2.5.0'
|
219
|
+
benchmarking....
|
220
|
+
|
221
|
+
$ benchmark-driver benchmark_driver.record.yml --output compare
|
222
|
+
Calculating -------------------------------------
|
223
|
+
empty 153.380M 114.228M i/s - 20.000M times in 0.130395s 0.175088s
|
224
|
+
blank 67.834M 64.328M i/s - 20.000M times in 0.294836s 0.310906s
|
225
|
+
|
226
|
+
Comparison:
|
227
|
+
empty: 153380113.9 i/s
|
228
|
+
blank: 67834321.4 i/s - 2.26x slower
|
229
|
+
|
230
|
+
$ benchmark-driver benchmark_driver.record.yml --output simple
|
231
|
+
benchmark results (i/s):
|
232
|
+
empty 153.380M 114.228M
|
233
|
+
blank 67.834M 64.328M
|
234
|
+
```
|
235
|
+
|
236
|
+
### gruff
|
237
|
+
|
238
|
+
There is [benchmark\_driver-output-gruff](https://github.com/benchmark-driver/benchmark_driver-output-gruff)
|
239
|
+
plugin that renders a graph using gruff.gem.
|
240
|
+
|
241
|
+
![](./images/optcarrot.png)
|
242
|
+
|
243
|
+
### benchmark-driver.github.io
|
244
|
+
|
245
|
+
Benchmark results for https://benchmark-driver.github.io are also stored by [output plugin](https://github.com/benchmark-driver/skybench/tree/master/plugins/benchmark_driver-output-skybench).
|
246
|
+
You can integrate any system as you like.
|
247
|
+
|
248
|
+
## Runner Options
|
249
|
+
|
250
|
+
Runner decides metrics to be collected. There are following default runner options.
|
251
|
+
|
252
|
+
ips, time, memory, once
|
253
|
+
|
254
|
+
| Runner | Description |
|
255
|
+
|:-------|:------------|
|
256
|
+
| ips | Iteration per second (default) |
|
257
|
+
| time | Elapsed seconds |
|
258
|
+
| memory | Max resident set. This is supported only on Linux for now. |
|
259
|
+
| once | Forces `loop_count` to 1 for testing |
|
260
|
+
| command\_stdout | Special runner to integrate existing benchmarks |
|
261
|
+
|
262
|
+
### ips
|
263
|
+
|
264
|
+
```
|
265
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --runner ips --rbenv '2.4.3;2.5.0'
|
266
|
+
Calculating -------------------------------------
|
267
|
+
2.4.3 2.5.0
|
268
|
+
empty 228.802M 180.125M i/s - 20.000M times in 0.087412s 0.111034s
|
269
|
+
blank 90.012M 88.853M i/s - 20.000M times in 0.222193s 0.225090s
|
270
|
+
|
271
|
+
Comparison:
|
272
|
+
empty
|
273
|
+
2.4.3: 228801720.5 i/s
|
274
|
+
2.5.0: 180124821.8 i/s - 1.27x slower
|
275
|
+
|
276
|
+
blank
|
277
|
+
2.4.3: 90012021.7 i/s
|
278
|
+
2.5.0: 88853269.4 i/s - 1.01x slower
|
279
|
+
```
|
280
|
+
|
281
|
+
### time
|
282
|
+
|
283
|
+
```
|
284
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --runner time --rbenv '2.4.3;2.5.0'
|
285
|
+
Calculating -------------------------------------
|
286
|
+
2.4.3 2.5.0
|
287
|
+
empty 0.087 0.110 s - 20.000M times
|
288
|
+
blank 0.217 0.219 s - 20.000M times
|
289
|
+
|
290
|
+
Comparison:
|
291
|
+
empty
|
292
|
+
2.5.0: 0.1 s
|
293
|
+
2.4.3: 0.1 s - 1.26x slower
|
294
|
+
|
295
|
+
blank
|
296
|
+
2.5.0: 0.2 s
|
297
|
+
2.4.3: 0.2 s - 1.01x slower
|
298
|
+
```
|
299
|
+
|
300
|
+
### memory
|
301
|
+
|
302
|
+
```
|
303
|
+
$ be exe/benchmark-driver examples/yaml/blank_loop.yml --runner memory --rbenv '2.4.3;2.5.0'
|
304
|
+
Calculating -------------------------------------
|
305
|
+
2.4.3 2.5.0
|
306
|
+
empty 9.192M 9.364M bytes - 20.000M times
|
307
|
+
blank 9.080M 9.372M bytes - 20.000M times
|
308
|
+
|
309
|
+
Comparison:
|
310
|
+
empty
|
311
|
+
2.4.3: 9192000.0 bytes
|
312
|
+
2.5.0: 9364000.0 bytes - 1.02x larger
|
313
|
+
|
314
|
+
blank
|
315
|
+
2.4.3: 9080000.0 bytes
|
316
|
+
2.5.0: 9372000.0 bytes - 1.03x larger
|
317
|
+
```
|
318
|
+
|
319
|
+
### once
|
320
|
+
|
321
|
+
Only for testing purpose.
|
322
|
+
|
323
|
+
```
|
324
|
+
$ benchmark-driver examples/yaml/blank_loop.yml --runner once --rbenv '2.4.3;2.5.0'
|
325
|
+
Calculating -------------------------------------
|
326
|
+
2.4.3 2.5.0
|
327
|
+
empty 1.818M 2.681M i/s - 1.000 times in 0.000001s 0.000000s
|
328
|
+
blank 1.531M 2.421M i/s - 1.000 times in 0.000001s 0.000000s
|
329
|
+
|
330
|
+
Comparison:
|
331
|
+
empty
|
332
|
+
2.5.0: 2680965.1 i/s
|
333
|
+
2.4.3: 1818181.8 i/s - 1.47x slower
|
334
|
+
|
335
|
+
blank
|
336
|
+
2.5.0: 2421307.5 i/s
|
337
|
+
2.4.3: 1531393.6 i/s - 1.58x slower
|
338
|
+
```
|
339
|
+
|
340
|
+
### command\_stdout
|
341
|
+
|
342
|
+
See following examples:
|
343
|
+
|
344
|
+
* https://github.com/benchmark-driver/optcarrot
|
345
|
+
* https://github.com/benchmark-driver/fluentd-benchmark
|
346
|
+
|
347
|
+
If you benchmark can run with `ruby foo bar`, specify `foo bar` to `command:`.
|
348
|
+
Then write `stdout_to_metrics:` to convert stdout to metrics. This runner can be used only with YAML interface for now.
|
349
|
+
|
350
|
+
```
|
351
|
+
$ benchmark-driver benchmark.yml --verbose 1 --rbenv '2.6.0-dev;2.6.0-dev,--jit'
|
352
|
+
2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
|
353
|
+
2.6.0-dev,--jit: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
|
354
|
+
Calculating -------------------------------------
|
355
|
+
2.6.0-dev 2.6.0-dev,--jit
|
356
|
+
optcarrot 51.866 67.445 fps
|
357
|
+
|
358
|
+
Comparison:
|
359
|
+
optcarrot
|
360
|
+
2.6.0-dev,--jit: 67.4 fps
|
361
|
+
2.6.0-dev: 51.9 fps - 1.30x slower
|
362
|
+
```
|
363
|
+
|
169
364
|
## Contributing
|
170
365
|
|
171
366
|
Bug reports and pull requests are welcome on GitHub at https://github.com/k0kubun/benchmark_driver.
|
Binary file
|
@@ -104,8 +104,8 @@ class BenchmarkDriver::Output::Compare
|
|
104
104
|
end
|
105
105
|
|
106
106
|
def humanize(value, width = 10)
|
107
|
-
if value
|
108
|
-
raise ArgumentError.new("
|
107
|
+
if value <= 0
|
108
|
+
raise ArgumentError.new("Non positive value: #{value.inspect}")
|
109
109
|
end
|
110
110
|
|
111
111
|
scale = (Math.log10(value) / 3).to_i
|
@@ -6,7 +6,7 @@ class BenchmarkDriver::Runner::Time < BenchmarkDriver::Runner::Ips
|
|
6
6
|
# Dynamically fetched and used by `BenchmarkDriver::JobParser.parse`
|
7
7
|
JobParser = BenchmarkDriver::DefaultJobParser.for(Job)
|
8
8
|
|
9
|
-
METRICS_TYPE = BenchmarkDriver::Metrics::Type.new(unit: 's')
|
9
|
+
METRICS_TYPE = BenchmarkDriver::Metrics::Type.new(unit: 's', larger_better: false)
|
10
10
|
|
11
11
|
# Overriding BenchmarkDriver::Runner::Ips#set_metrics_type
|
12
12
|
def set_metrics_type
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: benchmark_driver
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.10.
|
4
|
+
version: 0.10.14
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Takashi Kokubun
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-
|
11
|
+
date: 2018-05-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -68,6 +68,7 @@ files:
|
|
68
68
|
- examples/yaml/example_multi.yml
|
69
69
|
- examples/yaml/example_single.yml
|
70
70
|
- exe/benchmark-driver
|
71
|
+
- images/optcarrot.png
|
71
72
|
- lib/benchmark_driver.rb
|
72
73
|
- lib/benchmark_driver/config.rb
|
73
74
|
- lib/benchmark_driver/default_job.rb
|
@@ -110,7 +111,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
110
111
|
version: '0'
|
111
112
|
requirements: []
|
112
113
|
rubyforge_project:
|
113
|
-
rubygems_version: 2.7.
|
114
|
+
rubygems_version: 2.7.6
|
114
115
|
signing_key:
|
115
116
|
specification_version: 4
|
116
117
|
summary: Fully-featured accurate benchmark driver for Ruby
|