benchmark_driver 0.10.13 → 0.10.14

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 888668b113284dcb442c1d9bfb4fc37563b1b73b26bd93c21d6b823e0dcace4e
4
- data.tar.gz: 92413919e8a9959fa88137da71cda69a5941bec2988b71fc365d47f08d254b57
3
+ metadata.gz: e8842fe9609d09ce46061bf5ad452219ec98f1b24ef34020698500160e9823cd
4
+ data.tar.gz: 69e3714e9bcc55053d198807b17edcc952e6682da5eed337d9a073df85891d79
5
5
  SHA512:
6
- metadata.gz: e16d5a6c38b9145778e08f045a7022b8b5d648f9f8b66afafa32158fd8bf52ab9c9f2fe7e895ae4fbbbf7ff977fdb7a57b592e6e4e150de1fc256b6cd8fdc446
7
- data.tar.gz: 88055b5b9ef159b75540c62b47eb54bf78316e9174e4f7ef9a3e1768fe0151f6a067814cff4b150cf2fa102d309741793d3c72f9c8e0f7b95c2502052b1f7d30
6
+ metadata.gz: b3c190db1fe084198706797c8d22cb7dba2a077d68e9d02d9466f9c5d26c206a4c03d3db6824d8c2ee3c09e90caf32cbcdd8c8067e75f254bf7c2078d45054f9
7
+ data.tar.gz: 3b50ec42bbe545db63347f5379f79f84b5b35b109728c8600725d642f3841098dfde78d820badcdf3a3836f4f26d2a6490e463e671c1c367dcbd3e76eb48e2f9
data/CHANGELOG.md CHANGED
@@ -1,3 +1,7 @@
1
+ # v0.10.14
2
+
3
+ - Fix a bug that large time is shown as better in time runner
4
+
1
5
  # v0.10.13
2
6
 
3
7
  - Add `x.output` to specify output plugin to Ruby interface
data/README.md CHANGED
@@ -145,14 +145,16 @@ Comparison:
145
145
  2.4.1: 1889805.6 i/s - 1.72x slower
146
146
  ```
147
147
 
148
- ## Output options
148
+ ## Output Options
149
149
 
150
150
  By default, there are following output options.
151
151
 
152
- * compare: benchmark-ips's `compare!`-like output (default)
153
- * simple: ruby's original `benchmark/driver.rb`-like simple output
154
- * markdown: output in markdown table
155
- * record: serialize results in `benchmark_driver.record.yml`, to change outputs later as you like
152
+ | Output | Description |
153
+ |:-------|:------------|
154
+ | compare | benchmark-ips's `compare!`-like output (default) |
155
+ | simple | ruby's original `benchmark/driver.rb`-like simple output |
156
+ | markdown | output in markdown table |
157
+ | record | serialize results in `benchmark_driver.record.yml`, to change outputs later as you like |
156
158
 
157
159
  With `benchmark-driver` CLI, you can specify it with `-o [output]` or `--output [output]`.
158
160
 
@@ -166,6 +168,199 @@ Benchmark.driver do |x|
166
168
  end
167
169
  ```
168
170
 
171
+ ### compare
172
+
173
+ ```
174
+ $ benchmark-driver examples/yaml/blank_loop.yml --output compare --rbenv '2.4.2;2.5.0'
175
+ Calculating -------------------------------------
176
+ 2.4.2 2.5.0
177
+ empty 195.957M 129.970M i/s - 20.000M times in 0.102063s 0.153882s
178
+ blank 66.554M 55.630M i/s - 20.000M times in 0.300507s 0.359519s
179
+
180
+ Comparison:
181
+ empty
182
+ 2.4.2: 195957398.5 i/s
183
+ 2.5.0: 129969716.9 i/s - 1.51x slower
184
+
185
+ blank
186
+ 2.4.2: 66554190.0 i/s
187
+ 2.5.0: 55629883.2 i/s - 1.20x slower
188
+ ```
189
+
190
+ ### simple
191
+
192
+ ```
193
+ $ benchmark-driver examples/yaml/blank_loop.yml --output simple --rbenv '2.4.2;2.5.0'
194
+ benchmark results (i/s):
195
+ 2.4.2 2.5.0
196
+ empty 184.084M 117.319M
197
+ blank 65.843M 62.093M
198
+ ```
199
+
200
+ ### markdown
201
+
202
+ ```
203
+ $ benchmark-driver examples/yaml/blank_loop.yml --output markdown --rbenv '2.4.2;2.5.0'
204
+ ```
205
+
206
+ benchmark results (i/s)
207
+
208
+ | | 2.4.2| 2.5.0|
209
+ |:------|:-------|:-------|
210
+ |empty |187.296M|117.662M|
211
+ |blank | 58.895M| 58.852M|
212
+
213
+ ### record
214
+
215
+ Measure first, output with various formats later.
216
+
217
+ ```
218
+ $ benchmark-driver examples/yaml/blank_loop.yml --output record --rbenv '2.4.2;2.5.0'
219
+ benchmarking....
220
+
221
+ $ benchmark-driver benchmark_driver.record.yml --output compare
222
+ Calculating -------------------------------------
223
+ empty 153.380M 114.228M i/s - 20.000M times in 0.130395s 0.175088s
224
+ blank 67.834M 64.328M i/s - 20.000M times in 0.294836s 0.310906s
225
+
226
+ Comparison:
227
+ empty: 153380113.9 i/s
228
+ blank: 67834321.4 i/s - 2.26x slower
229
+
230
+ $ benchmark-driver benchmark_driver.record.yml --output simple
231
+ benchmark results (i/s):
232
+ empty 153.380M 114.228M
233
+ blank 67.834M 64.328M
234
+ ```
235
+
236
+ ### gruff
237
+
238
+ There is [benchmark\_driver-output-gruff](https://github.com/benchmark-driver/benchmark_driver-output-gruff)
239
+ plugin that renders a graph using gruff.gem.
240
+
241
+ ![](./images/optcarrot.png)
242
+
243
+ ### benchmark-driver.github.io
244
+
245
+ Benchmark results for https://benchmark-driver.github.io are also stored by [output plugin](https://github.com/benchmark-driver/skybench/tree/master/plugins/benchmark_driver-output-skybench).
246
+ You can integrate any system as you like.
247
+
248
+ ## Runner Options
249
+
250
+ Runner decides metrics to be collected. There are following default runner options.
251
+
252
+ ips, time, memory, once
253
+
254
+ | Runner | Description |
255
+ |:-------|:------------|
256
+ | ips | Iteration per second (default) |
257
+ | time | Elapsed seconds |
258
+ | memory | Max resident set. This is supported only on Linux for now. |
259
+ | once | Forces `loop_count` to 1 for testing |
260
+ | command\_stdout | Special runner to integrate existing benchmarks |
261
+
262
+ ### ips
263
+
264
+ ```
265
+ $ benchmark-driver examples/yaml/blank_loop.yml --runner ips --rbenv '2.4.3;2.5.0'
266
+ Calculating -------------------------------------
267
+ 2.4.3 2.5.0
268
+ empty 228.802M 180.125M i/s - 20.000M times in 0.087412s 0.111034s
269
+ blank 90.012M 88.853M i/s - 20.000M times in 0.222193s 0.225090s
270
+
271
+ Comparison:
272
+ empty
273
+ 2.4.3: 228801720.5 i/s
274
+ 2.5.0: 180124821.8 i/s - 1.27x slower
275
+
276
+ blank
277
+ 2.4.3: 90012021.7 i/s
278
+ 2.5.0: 88853269.4 i/s - 1.01x slower
279
+ ```
280
+
281
+ ### time
282
+
283
+ ```
284
+ $ benchmark-driver examples/yaml/blank_loop.yml --runner time --rbenv '2.4.3;2.5.0'
285
+ Calculating -------------------------------------
286
+ 2.4.3 2.5.0
287
+ empty 0.087 0.110 s - 20.000M times
288
+ blank 0.217 0.219 s - 20.000M times
289
+
290
+ Comparison:
291
+ empty
292
+ 2.5.0: 0.1 s
293
+ 2.4.3: 0.1 s - 1.26x slower
294
+
295
+ blank
296
+ 2.5.0: 0.2 s
297
+ 2.4.3: 0.2 s - 1.01x slower
298
+ ```
299
+
300
+ ### memory
301
+
302
+ ```
303
+ $ be exe/benchmark-driver examples/yaml/blank_loop.yml --runner memory --rbenv '2.4.3;2.5.0'
304
+ Calculating -------------------------------------
305
+ 2.4.3 2.5.0
306
+ empty 9.192M 9.364M bytes - 20.000M times
307
+ blank 9.080M 9.372M bytes - 20.000M times
308
+
309
+ Comparison:
310
+ empty
311
+ 2.4.3: 9192000.0 bytes
312
+ 2.5.0: 9364000.0 bytes - 1.02x larger
313
+
314
+ blank
315
+ 2.4.3: 9080000.0 bytes
316
+ 2.5.0: 9372000.0 bytes - 1.03x larger
317
+ ```
318
+
319
+ ### once
320
+
321
+ Only for testing purpose.
322
+
323
+ ```
324
+ $ benchmark-driver examples/yaml/blank_loop.yml --runner once --rbenv '2.4.3;2.5.0'
325
+ Calculating -------------------------------------
326
+ 2.4.3 2.5.0
327
+ empty 1.818M 2.681M i/s - 1.000 times in 0.000001s 0.000000s
328
+ blank 1.531M 2.421M i/s - 1.000 times in 0.000001s 0.000000s
329
+
330
+ Comparison:
331
+ empty
332
+ 2.5.0: 2680965.1 i/s
333
+ 2.4.3: 1818181.8 i/s - 1.47x slower
334
+
335
+ blank
336
+ 2.5.0: 2421307.5 i/s
337
+ 2.4.3: 1531393.6 i/s - 1.58x slower
338
+ ```
339
+
340
+ ### command\_stdout
341
+
342
+ See following examples:
343
+
344
+ * https://github.com/benchmark-driver/optcarrot
345
+ * https://github.com/benchmark-driver/fluentd-benchmark
346
+
347
+ If you benchmark can run with `ruby foo bar`, specify `foo bar` to `command:`.
348
+ Then write `stdout_to_metrics:` to convert stdout to metrics. This runner can be used only with YAML interface for now.
349
+
350
+ ```
351
+ $ benchmark-driver benchmark.yml --verbose 1 --rbenv '2.6.0-dev;2.6.0-dev,--jit'
352
+ 2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
353
+ 2.6.0-dev,--jit: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
354
+ Calculating -------------------------------------
355
+ 2.6.0-dev 2.6.0-dev,--jit
356
+ optcarrot 51.866 67.445 fps
357
+
358
+ Comparison:
359
+ optcarrot
360
+ 2.6.0-dev,--jit: 67.4 fps
361
+ 2.6.0-dev: 51.9 fps - 1.30x slower
362
+ ```
363
+
169
364
  ## Contributing
170
365
 
171
366
  Bug reports and pull requests are welcome on GitHub at https://github.com/k0kubun/benchmark_driver.
Binary file
@@ -104,8 +104,8 @@ class BenchmarkDriver::Output::Compare
104
104
  end
105
105
 
106
106
  def humanize(value, width = 10)
107
- if value < 0
108
- raise ArgumentError.new("Negative value: #{value.inspect}")
107
+ if value <= 0
108
+ raise ArgumentError.new("Non positive value: #{value.inspect}")
109
109
  end
110
110
 
111
111
  scale = (Math.log10(value) / 3).to_i
@@ -6,7 +6,7 @@ class BenchmarkDriver::Runner::Time < BenchmarkDriver::Runner::Ips
6
6
  # Dynamically fetched and used by `BenchmarkDriver::JobParser.parse`
7
7
  JobParser = BenchmarkDriver::DefaultJobParser.for(Job)
8
8
 
9
- METRICS_TYPE = BenchmarkDriver::Metrics::Type.new(unit: 's')
9
+ METRICS_TYPE = BenchmarkDriver::Metrics::Type.new(unit: 's', larger_better: false)
10
10
 
11
11
  # Overriding BenchmarkDriver::Runner::Ips#set_metrics_type
12
12
  def set_metrics_type
@@ -1,3 +1,3 @@
1
1
  module BenchmarkDriver
2
- VERSION = '0.10.13'
2
+ VERSION = '0.10.14'
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: benchmark_driver
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.10.13
4
+ version: 0.10.14
5
5
  platform: ruby
6
6
  authors:
7
7
  - Takashi Kokubun
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2018-03-21 00:00:00.000000000 Z
11
+ date: 2018-05-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -68,6 +68,7 @@ files:
68
68
  - examples/yaml/example_multi.yml
69
69
  - examples/yaml/example_single.yml
70
70
  - exe/benchmark-driver
71
+ - images/optcarrot.png
71
72
  - lib/benchmark_driver.rb
72
73
  - lib/benchmark_driver/config.rb
73
74
  - lib/benchmark_driver/default_job.rb
@@ -110,7 +111,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
110
111
  version: '0'
111
112
  requirements: []
112
113
  rubyforge_project:
113
- rubygems_version: 2.7.3
114
+ rubygems_version: 2.7.6
114
115
  signing_key:
115
116
  specification_version: 4
116
117
  summary: Fully-featured accurate benchmark driver for Ruby