benchmark_driver_monotonic_raw 0.14.13

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 9eae22208b3dcf261733524be5827dbdb4744ec2
+  data.tar.gz: b2cfbd1d7ec3c8f30cc782014057b4e302e2fe55
+SHA512:
+  metadata.gz: 30288dc02ee36d5f7854874a6a2ddb010331c582b712bce6bcb806dcc219e30a5ff952851c629e8d795608e26fa3ff961915e551b660ae8c54f718349ac119a4
+  data.tar.gz: 0b638ec6fbfea4989002a7b94b865d0bdc2fff58f947f22f57c93a7aa505563859c84fa4a42f04bb896d17254d6e39fa6dbd2add51149e0e9d7731b00e08fa84

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/Gemfile.lock
+benchmark_driver.record.yml

data/.rspec

@@ -0,0 +1 @@
+--require spec_helper.rb

data/.travis.yml

@@ -0,0 +1,16 @@
+language: ruby
+rvm:
+  - 2.2.8
+  - 2.3.6
+  - 2.4.3
+  - 2.5.0
+cache: bundler
+branches:
+  only:
+    - master
+before_install:
+  - gem install bundler -v 1.15.4
+  - gem install haml -v 4.0.7
+  - gem install haml -v 5.0.4
+script:
+  - RSPEC_RETRIES=3 VERBOSE=1 bundle exec rake

data/CHANGELOG.md

@@ -0,0 +1,357 @@
+# v0.14.13
+
+- Add macOS support to memory runner
+  [#53](https://github.com/benchmark-driver/benchmark-driver/pull/53)
+
+# v0.14.12
+
+- Reduce the risk of random ZeroDivisionError on warmup
+  [#52](https://github.com/benchmark-driver/benchmark-driver/pull/52)
+
+# v0.14.11
+
+- Add `x.executable` method to Ruby interface for specifying Ruby executable which is not managed by rbenv
+  [#50](https://github.com/benchmark-driver/benchmark-driver/pull/50)
+
+# v0.14.10
+
+- Resolve internal Ruby warnings
+
+# v0.14.9
+
+- Allow recorded runner to run without the recorded Ruby executables
+  [#43](https://github.com/benchmark-driver/benchmark-driver/issues/43) [#44](https://github.com/benchmark-driver/benchmark-driver/issues/44)
+
+# v0.14.8
+
+- Fix bug that "benchmark\_driver.record.yml" created with `-e`/`--rbenv` option
+  has required to specify the same `-e`/`--rbenv` option [#42](https://github.com/benchmark-driver/benchmark-driver/issues/42)
+
+# v0.14.7
+
+- Fix bug on warmup with `BenchmarkDriver::BulkOutput` [#40](https://github.com/benchmark-driver/benchmark-driver/issues/40)
+  - This is currently used by benchmark\_driver-output-gruff
+
+# v0.14.6
+
+- Improve option parser validation
+- Add `--timeout` option only for Linux having timeout(1)
+- Support `--version` option
+- Fix help output for `-e`/`--rbenv` in benchmark-driver command
+
+# v0.14.5
+
+- Fix wrong spacing format in simple / markdown output
+
+# v0.14.4
+
+- Fix runners and outputs to distinguish 0.0 and ERROR value
+
+# v0.14.3
+
+- Support loading runner plugin
+
+# v0.14.2
+
+- Fix definitive error in v0.14.1...
+
+# v0.14.1 (yanked)
+
+- Fix `BenchmarkDriver::BulkOutput` error on contexts with prelude [#38](https://github.com/benchmark-driver/benchmark-driver/issues/38)
+
+# v0.14.0
+
+- `benchmark-driver` command also takes `*.rb` file to run single-execution benchmark
+  - Non-`*.rb` filename is considered as YAML file as before
+- [breaking change] `--verbose` no longer takes LEVEL argument
+  - Added `-v` for `-v`/`-vv` instead of `--verbose 1` and `--verbose 2`.
+  - `--verbose --verbose` can also be used instead of `--verbose 2` if you want.
+- [breaking change] Use ` `(space) to split Ruby arguments in `-e`/`--executables`/`--rbenv`
+  - ex) `-e "/path/to/ruby --jit"` should be used instead of `-e "/path/to/ruby,--jit"`
+- [experimental] Return 0.0 as value when benchmark script fails on ips/time/memory/once runners
+  - Exit status is 0 for that case. `benchmark-driver` exits abnormaly only on its internal bug.
+
+# v0.13.3
+
+- Support `require: false` in contexts to skip automatic requirement with a gem name
+
+# v0.13.2
+
+- Stop generating loop code with `loop_count: 1` on ips runner
+
+# v0.13.1
+
+- Respect a magic comment in `prelude`
+
+# v0.13.0
+
+- Add support for benchmark with different versions of gems and preludes
+  - Only YAML is supported for now, Ruby interface will come later
+- [breaking change] Runner plugin interface is slightly changed
+
+# v0.12.0
+
+- [breaking change] Plugin interface is changed again
+- Fix bugs in a case that multiple YAMLs with different types are specified
+- Output plugin is now ensured to yield the same metrics
+
+# v0.11.1
+
+- Add `--repeat-result` option to return the best, the worst or an average result with `--repeat-count`
+- Add `BenchmarkDriver::BulkOutput` to make an output plugin casually
+
+# v0.11.0
+
+- [breaking change] Plugin interface is completely changed, so all plugins need migration
+  - YAML/Ruby interface is not changed at all
+  - Now the internal model allows to have multiple metrics in the same job,
+    but having multiple metrics is still not respected by built-in plugins
+  - "executable" is renamed to be "context", but still configuring a context
+    other than an executable is not supported yet
+  - A metric can have a name
+- Add `ruby_stdout` runner
+  - This can parse an arbitrary "environment" for the "context"
+  - Metric can have name
+- Metric name is shwon on some outputs like markdown and simple
+- `--run-duration` accepts floating-point number
+- News: Now benchmark-driver.gem can be used as an alias to install benchmark\_driver.gem
+
+# v0.10.16
+
+- `command_stdout` runner uses YAML's dirname as `working_directory` by default
+
+# v0.10.15
+
+- Make `Benchmark.driver` take `repeat_count:` keyword argument
+
+# v0.10.14
+
+- Fix a bug that large time is shown as better in time runner
+
+# v0.10.13
+
+- Add `x.output` to specify output plugin to Ruby interface
+  - You can still use `Benchmark.driver(output: xxx)` form too
+
+# v0.10.12
+
+- Fix some typo in help
+- Add dynamic require for convenience to implement output plugins
+
+# v0.10.11
+
+- Add `:required_ruby_option` parameter in YAML format
+
+# v0.10.10
+
+- Add `BenchmarkDriver::Config::Executable#description` method to return `ruby -v` result for output plugins.
+
+# v0.10.9
+
+- Add `x.rbenv`, `x.loop_count`, `x.verbose` supports to Ruby interface
+
+# v0.10.8
+
+- In `command_stdout`, `$RBENV_VERSION` is no longer passed to --rbenv option because it has no effect for rbenv
+- Instead, now dirname of executable path is prefixed to $PATH in `command_stdout`
+
+# v0.10.7
+
+- Pass `$RBENV_VERSION` for `command_stdout` runner with --rbenv option
+
+# v0.10.6
+
+- Fix TypeError on Ruby <= 2.3
+
+# v0.10.5
+
+- Run runners with Bundler's clean env
+
+# v0.10.4
+
+- Fix frozen string error when parsing multiple jobs
+
+# v0.10.3
+
+- Allow specifying name in `--rbenv`
+- Don't print stderr in "command\_stdout" runner
+
+# v0.10.2
+
+- Optionalize `working_directory` of "command\_stdout" runner
+
+# v0.10.1
+
+- Add "command\_stdout" runner to plug in existing benchmark
+- Explicitly bump supported Ruby version to >= 2.2
+  - v0.10.0 actually does not work with 2.1
+  - You can still benchmark Ruby 2.0, 2.1 by --executable, but you need to use newer Ruby for driver
+
+# v0.10.0
+
+- Add "record" output and "recorded" runner
+  - You can record metrics to yaml and change how to output later
+- Change output interface to set metrics\_type
+  - Now runner needs to call `output#metrics_type=`
+
+# v0.9.2
+
+- Add `--verbose` option
+  - `--verbose 1` shows `ruby -v` for each executable
+  - `--verbose 2` shows executed scripts
+
+# v0.9.1
+
+- Fix memory runner bug that actually doesn't run the benchmarked script
+- Add once runner to test benchmark script in a short time
+
+# v0.9.0
+
+- The concept of runner is renewed
+  - Now it's for specifying metrics like ips, time, memory usage
+  - Old runners (:call and :eval) are no longer supported. :exec only.
+     - So Ruby interface can't take Proc
+  - YAML can have arbitrary format depending on the runner
+- `--compare` option is dropped and changed to `--output compare`
+- `--dir` option is dropped for now
+
+# v0.8.6
+
+- Automatically require `benchmark/output/foo` when `-o foo` is specified
+
+# v0.8.5
+
+- Show time per iteration for ips output
+  - Show clocks/i too when it's < 1000 clocks/i
+- In ips output, 'B' unit (10^9) is changed to 'G'
+
+# v0.8.4
+
+- Add `loop_count` option to Ruby interface
+
+# v0.8.3
+
+- Make benchmark\_driver runnable when Bundler is not installed
+
+# v0.8.2
+
+- Fix bug on showing error message in `benchmark-driver` command
+
+# v0.8.1
+
+- Fix internal implementation of Eval runner
+  - Now this can accept class definition in prelude
+
+# v0.8.0
+
+- Force using :exec runner for YAML interface
+- Fix bug that executables become empty
+
+# v0.7.2
+
+- Respect ignored output option in Ruby interface
+
+# v0.7.1
+
+- Add `x.rbenv` to Ruby interface
+- Add `x.bundler` to Ruby interface
+
+# v0.7.0
+
+- Change Ruby interface for specifying prelude and script
+  - #prelude= is renamed to #prelude
+  - `script:` is no longer a keyword argument
+- Add Eval runner and it's made default for Ruby interface when script is String
+
+# v0.6.2
+
+- Resurrect support of Ruby interface dropped at v0.6.0
+- Invalidate wrong configuration
+- Decide runner type automatically for Ruby interface
+
+# v0.6.1
+
+- Support markdown output
+
+# v0.6.0
+
+- Drop support of Ruby interface
+- Drop support of specifying runner, output options on YAML definition
+- Allow specifying output type on CLI
+- Run multiple benchmark files at once, instead of sequentially executing them
+
+# v0.5.1
+
+- Fix a bug that fails to run multiple Ruby binaries when multiple YAML files are specified
+
+# v0.5.0
+
+- Automatic bundle install for each Ruby executable on `--bundler` or `--bundle`
+- CLI error handling is improved for empty/invalid arguments
+
+# v0.4.5
+
+- Allow specifying arguments for ruby executables
+
+# v0.4.4
+
+- Add `--bundler` option to run benchmark with fixed gems
+
+# v0.4.3
+
+- Add `--filter` option to run only specified benchmarks
+
+# v0.4.2
+
+- Exec runner uses `Bundler.with_clean_env`
+
+# v0.4.1
+
+- Increase the number of pasted script in one loop: 50 -> 1000
+- Add `--dir` option to override `__dir__`
+
+# v0.4.0
+
+- **Full scratch**: v0.1...v0.3 has no relationship with v0.4
+  - Plugin support for runner, output
+  - Add initial limited Ruby interface in addition to previous YAML input
+     - Add new `call` runner for Ruby and `exec` runner for existing functionality
+  - Support repeating multiple times and using the best result
+  - Add `compare!` switching
+  - Gradual loop time estimation for specific duration
+  - Real-time incremental output during benchmark execution
+  - Paste the same script 50 times in one loop
+
+# v0.3.0
+
+- Rename `benchmark_driver` command to `benchmark-driver`
+- Internally use `Benchmark::Driver` namespace instead of `BenchmarkDriver`
+
+# v0.2.4
+
+- Allow specifying multiple rbenv executables at once
+
+# v0.2.3
+
+- Add `--rbenv` option to specify executables in rbenv
+
+# v0.2.2
+
+- Fix loop count option bug
+
+# v0.2.1
+
+- Stop using `i` for wrapper script for loop
+- Fix IPS option bug by @t8m8
+
+# v0.2.0
+
+- Allow specifying loop count
+
+# v0.1.0
+
+- Add `benchmark_driver` command that takes YAML input
+  - Runnable with multiple Ruby binaries
+  - Show actual time
+  - IPS reporter support
+  - Prelude and multiple benchmark scripts in one YAML

data/Gemfile

@@ -0,0 +1,8 @@
+source 'https://rubygems.org'
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in benchmark-driver.gemspec
+gemspec
+
+gem 'pry'

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Takashi Kokubun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,386 @@
+# BenchmarkDriver [![Build Status](https://travis-ci.org/benchmark-driver/benchmark-driver.svg?branch=master)](https://travis-ci.org/benchmark-driver/benchmark-driver)
+
+Fully-featured accurate benchmark driver for Ruby
+
+## Features
+### Accurate Measurement
+
+- Low overhead benchmark by running generated script instead of calling Proc
+- Profiling memory and high-precision real time
+- Running multiple times to minimize measurement errors
+
+### Pluggable & Fully Featured
+
+- Flexible and real-time output format in comparison, markdown table, graph, etc.
+- Measuring various metrics by specifying runners
+- Integrated benchmark support using external libraries
+- Runner and output format are all pluggable
+
+### Flexible Interface
+
+- Ruby interface similar to stdlib benchmark.rb, benchmark-ips
+- YAML input to easily manage structured benchmark set
+- Comparing multiple Ruby binaries, even with miniruby
+
+## Installation
+
+```
+$ gem install benchmark_driver
+```
+
+## Usage
+
+### Ruby Interface
+
+This interface generates code to profile with low overhead and executes it.
+
+```rb
+require 'benchmark_driver'
+
+Benchmark.driver do |x|
+  x.prelude <<~RUBY
+    require 'active_support/all'
+    array = []
+  RUBY
+
+  x.report 'blank?', %{ array.blank? }
+  x.report 'empty?', %{ array.empty? }
+end
+```
+
+or simply:
+
+```rb
+require 'benchmark_driver'
+
+Benchmark.driver do |x|
+  x.prelude <<~RUBY
+    require 'active_support/all'
+    array = []
+  RUBY
+
+  x.report %{ array.blank? }
+  x.report %{ array.empty? }
+end
+```
+
+### Structured YAML Input
+
+With `benchmark-driver` command, you can describe benchmark with YAML input.
+
+```
+$ benchmark-driver -h
+Usage: benchmark-driver [options] [YAML|RUBY]
+    -r, --runner [TYPE]              Specify runner type: ips, time, memory, once (default: ips)
+    -o, --output [TYPE]              Specify output type: compare, simple, markdown, record (default: compare)
+    -e, --executables [EXECS]        Ruby executables (e1::path1 arg1; e2::path2 arg2;...)
+        --rbenv [VERSIONS]           Ruby executables in rbenv (x.x.x arg1;y.y.y arg2;...)
+        --repeat-count [NUM]         Try benchmark NUM times and use the fastest result or the worst memory usage
+        --repeat-result [TYPE]       Yield "best", "average" or "worst" result with --repeat-count (default: best)
+        --bundler                    Install and use gems specified in Gemfile
+        --filter [REGEXP]            Filter out benchmarks with given regexp
+        --run-duration [SECONDS]     Warmup estimates loop_count to run for this duration (default: 3)
+    -v, --verbose                    Verbose mode. Multiple -v options increase visilibity (max: 2)
+```
+
+#### Running single script
+
+With following `example_single.yml`,
+
+```yml
+prelude: |
+  require 'erb'
+  erb = ERB.new(%q[Hello <%= 'World' %>])
+benchmark: erb.result
+```
+
+you can benchmark the script with multiple ruby executables.
+
+```
+$ benchmark-driver example_single.yml --rbenv '2.4.1;2.5.0'
+Warming up --------------------------------------
+          erb.result    71.683k i/s
+Calculating -------------------------------------
+                          2.4.1       2.5.0
+          erb.result    72.387k     75.046k i/s -    215.049k times in 2.970833s 2.865581s
+
+Comparison:
+                       erb.result
+               2.5.0:     75045.5 i/s
+               2.4.1:     72386.8 i/s - 1.04x  slower
+```
+
+#### Running multiple scripts
+
+One YAML file can contain multiple benchmark scripts.
+With following `example_multi.yml`,
+
+```yml
+prelude: |
+  a = 'a' * 100
+  b = 'b' * 100
+benchmark:
+  join: '[a, b].join'
+  str-interp: '"#{a}#{b}"'
+```
+
+you can benchmark the scripts with multiple ruby executables.
+
+```
+$ benchmark-driver example_multi.yml --rbenv '2.4.1;2.5.0'
+Warming up --------------------------------------
+                join     2.509M i/s
+          str-interp     1.772M i/s
+Calculating -------------------------------------
+                          2.4.1       2.5.0
+                join     2.661M      2.863M i/s -      7.527M times in 2.828771s 2.629191s
+          str-interp     1.890M      3.258M i/s -      5.315M times in 2.812240s 1.630997s
+
+Comparison:
+                             join
+               2.5.0:   2862755.1 i/s
+               2.4.1:   2660777.4 i/s - 1.08x  slower
+
+                       str-interp
+               2.5.0:   3258489.7 i/s
+               2.4.1:   1889805.6 i/s - 1.72x  slower
+```
+
+## Output Options
+
+By default, there are following output options.
+
+| Output | Description |
+|:-------|:------------|
+| compare | benchmark-ips's `compare!`-like output (default) |
+| simple | ruby's original `benchmark/driver.rb`-like simple output |
+| markdown | output in markdown table |
+| record | serialize results in `benchmark_driver.record.yml`, to change outputs later as you like |
+
+With `benchmark-driver` CLI, you can specify it with `-o [output]` or `--output [output]`.
+
+With Ruby interface, you can specify it like:
+
+```rb
+Benchmark.driver do |x|
+  x.prelude %{ array = [] }
+  x.report 'Array#empty?', %{ array.empty? }
+  x.output 'markdown'
+end
+```
+
+With following `blank_loop.yml`,
+
+```yml
+loop_count: 20000000
+prelude: |
+  class Array
+    alias_method :blank?, :empty?
+  end
+  array = []
+benchmark:
+  empty: array.empty?
+  blank: array.blank?
+```
+
+you can output results in various ways like:
+
+### compare
+
+```
+$ benchmark-driver blank_loop.yml --output compare --rbenv '2.4.2;2.5.0'
+Calculating -------------------------------------
+                          2.4.2       2.5.0
+               empty   195.957M    129.970M i/s -     20.000M times in 0.102063s 0.153882s
+               blank    66.554M     55.630M i/s -     20.000M times in 0.300507s 0.359519s
+
+Comparison:
+                            empty
+               2.4.2: 195957398.5 i/s
+               2.5.0: 129969716.9 i/s - 1.51x  slower
+
+                            blank
+               2.4.2:  66554190.0 i/s
+               2.5.0:  55629883.2 i/s - 1.20x  slower
+```
+
+### simple
+
+```
+$ benchmark-driver blank_loop.yml --output simple --rbenv '2.4.2;2.5.0'
+benchmark results (i/s):
+          2.4.2     2.5.0
+empty  184.084M  117.319M
+blank   65.843M   62.093M
+```
+
+### markdown
+
+```
+$ benchmark-driver blank_loop.yml --output markdown --rbenv '2.4.2;2.5.0'
+```
+
+benchmark results (i/s)
+
+|       |   2.4.2|   2.5.0|
+|:------|:-------|:-------|
+|empty  |187.296M|117.662M|
+|blank  | 58.895M| 58.852M|
+
+### record
+
+Measure first, output with various formats later.
+
+```
+$ benchmark-driver blank_loop.yml --output record --rbenv '2.4.2;2.5.0'
+benchmarking....
+
+$ benchmark-driver benchmark_driver.record.yml --output compare
+Calculating -------------------------------------
+               empty   153.380M    114.228M i/s -     20.000M times in 0.130395s 0.175088s
+               blank    67.834M     64.328M i/s -     20.000M times in 0.294836s 0.310906s
+
+Comparison:
+               empty: 153380113.9 i/s
+               blank:  67834321.4 i/s - 2.26x  slower
+
+$ benchmark-driver benchmark_driver.record.yml --output simple
+benchmark results (i/s):
+empty  153.380M  114.228M
+blank   67.834M   64.328M
+```
+
+### gruff
+
+There is [benchmark\_driver-output-gruff](https://github.com/benchmark-driver/benchmark_driver-output-gruff)
+plugin that renders a graph using gruff.gem.
+
+![](./images/optcarrot.png)
+
+### benchmark-driver.github.io
+
+Benchmark results for https://benchmark-driver.github.io are also stored by [output plugin](https://github.com/benchmark-driver/skybench/tree/master/plugins/benchmark_driver-output-skybench).
+You can integrate any system as you like.
+
+## Runner Options
+
+Runner decides metrics to be collected. There are following default runner options.
+
+ips, time, memory, once
+
+| Runner | Description |
+|:-------|:------------|
+| ips | Iteration per second (default) |
+| time | Elapsed seconds |
+| memory | Max resident set. This is supported only on Linux and macOS for now. |
+| once | Forces `loop_count` to 1 for testing |
+| ruby\_stdout | Special runner to integrate existing benchmarks |
+
+### ips
+
+```
+$ benchmark-driver blank_loop.yml --runner ips --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+                          2.4.3       2.5.0
+               empty   228.802M    180.125M i/s -     20.000M times in 0.087412s 0.111034s
+               blank    90.012M     88.853M i/s -     20.000M times in 0.222193s 0.225090s
+
+Comparison:
+                            empty
+               2.4.3: 228801720.5 i/s
+               2.5.0: 180124821.8 i/s - 1.27x  slower
+
+                            blank
+               2.4.3:  90012021.7 i/s
+               2.5.0:  88853269.4 i/s - 1.01x  slower
+```
+
+### time
+
+```
+$ benchmark-driver blank_loop.yml --runner time --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+                          2.4.3       2.5.0
+               empty      0.087       0.110 s -     20.000M times
+               blank      0.217       0.219 s -     20.000M times
+
+Comparison:
+                            empty
+               2.5.0:         0.1 s
+               2.4.3:         0.1 s - 1.26x  slower
+
+                            blank
+               2.5.0:         0.2 s
+               2.4.3:         0.2 s - 1.01x  slower
+```
+
+### memory
+
+```
+$ benchmark-driver blank_loop.yml --runner memory --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+                          2.4.3       2.5.0
+               empty     9.192M      9.364M bytes -     20.000M times
+               blank     9.080M      9.372M bytes -     20.000M times
+
+Comparison:
+                            empty
+               2.4.3:   9192000.0 bytes
+               2.5.0:   9364000.0 bytes - 1.02x  larger
+
+                            blank
+               2.4.3:   9080000.0 bytes
+               2.5.0:   9372000.0 bytes - 1.03x  larger
+```
+
+### once
+
+Only for testing purpose.
+
+```
+$ benchmark-driver blank_loop.yml --runner once --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+                          2.4.3       2.5.0
+               empty     1.818M      2.681M i/s -       1.000 times in 0.000001s 0.000000s
+               blank     1.531M      2.421M i/s -       1.000 times in 0.000001s 0.000000s
+
+Comparison:
+                            empty
+               2.5.0:   2680965.1 i/s
+               2.4.3:   1818181.8 i/s - 1.47x  slower
+
+                            blank
+               2.5.0:   2421307.5 i/s
+               2.4.3:   1531393.6 i/s - 1.58x  slower
+```
+
+### ruby\_stdout
+
+See following example:
+
+* https://github.com/benchmark-driver/optcarrot
+
+If your benchmark can run with `ruby foo bar`, specify `foo bar` to `command:`.
+Then write `stdout_to_metrics:` to convert stdout to metrics. This runner can be used only with YAML interface for now.
+
+```
+$ benchmark-driver benchmark.yml --verbose 1 --rbenv '2.6.0-dev;2.6.0-dev --jit'
+2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
+2.6.0-dev --jit: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
+Calculating -------------------------------------
+                      2.6.0-dev  2.6.0-dev --jit
+           optcarrot     51.866           67.445 fps
+
+Comparison:
+                        optcarrot
+     2.6.0-dev --jit:        67.4 fps
+           2.6.0-dev:        51.9 fps - 1.30x  slower
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/benchmark-driver/benchmark-driver.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).