RubyGems - benchmark_driver - Versions diffs - 0.1.0 → 0.2.0 - Mend

benchmark_driver 0.1.0 → 0.2.0

Files changed (15) hide show

checksums.yaml +4 -4
data/README.md +97 -94
data/Rakefile +13 -4
data/benchmark_driver.gemspec +0 -1
data/exe/benchmark_driver +9 -10
data/lib/benchmark_driver.rb +77 -27
data/lib/benchmark_driver/version.rb +1 -1
data/ruby_benchmark_set/core/array.yml +4 -0
data/ruby_benchmark_set/example_multi.yml +10 -0
data/ruby_benchmark_set/example_single.yml +4 -0
data/ruby_benchmark_set/lib/erb.yml +30 -0
metadata +5 -18
data/.rspec +0 -2
data/examples/bm_app_erb.yml +0 -20
data/examples/erb_compile_render.yml +0 -47

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a09ccdd10d22d70ca190e43c4cdfa7bb6606f429
-  data.tar.gz: 6346d84cc347aa2111842e1b07c36138683c9d5b
+  metadata.gz: 027b33623e6d71484da812d92ef6d4acbf154312
+  data.tar.gz: cb8da42411f02760787b766fa0d8a8622750ecd4
 SHA512:
-  metadata.gz: eb5e63d89f0c344189467bde51efdd89a9a0e473eba30c15395d2c7f0e533f170ce164586b8c03a8e96f4cf89f5c804f0ff2f589e92b642e1b129ec00baccb8c
-  data.tar.gz: 87b3a29d42c3b6628e572741902739e7f104a0f00a21e0a24e44dc22edacc623f14dd2524db90083a398a7bb8540fbc9cb9b2ee0c6250c3708290a89a1e49298
+  metadata.gz: a666e41732c84ab63b0b93460b77b01e2b1540b00199f4c71971b7da532f0531bd6daf9813c87a91f55be51fc37fd2ca12723062b294017986d9f98261df4475
+  data.tar.gz: ea81b5f7b48a02c99f5718ec4dff48e25dc07eede56d1194172ca209057a4d170405082907d2f3937a91779ee11a08e1dc443afbc29a355782d540923382c5a0

data/README.md CHANGED Viewed

@@ -1,4 +1,4 @@
-# BenchmarkDriver
+# BenchmarkDriver [![Build Status](https://travis-ci.org/k0kubun/benchmark_driver.svg?branch=master)](https://travis-ci.org/k0kubun/benchmark_driver)
 Benchmark driver for different Ruby executables
@@ -9,155 +9,158 @@ Benchmark driver for different Ruby executables
 ## Usage
 ```
-$ benchmark_driver -h
+$ exe/benchmark_driver -h
 Usage: benchmark_driver [options] [YAML]
-    -d, --duration [SECONDS]         Duration seconds to run each benchmark (default: 1)
     -e, --executables [EXECS]        Ruby executables (e1::path1; e2::path2; e3::path3;...)
-    -r, --result-format=FORMAT       Result output format [time|ips] (default: time)
+    -i, --ips [SECONDS]              Measure IPS in duration seconds (default: 1)
+    -l, --loop-count [COUNT]         Measure execution time with loop count (default: 100000)
     -v, --verbose
 ```
 ### Running single script
-With following `bm_app_erb.yml`,
+With following `example_single.yml`,
 ```yml
 prelude: |
   require 'erb'
-  title = "hello world!"
-  content = "hello world!\n" * 10
-  data = <<EOS
-  <html>
-    <head> <%= title %> </head>
-    <body>
-      <h1> <%= title %> </h1>
-      <p>
-        <%= content %>
-      </p>
-    </body>
-  </html>
-  EOS
-benchmark: |
-  ERB.new(data).result(binding)
+  erb = ERB.new(%q[Hello <%= 'World' %>])
+benchmark: erb.result
 ```
 you can benchmark the script with multiple ruby executables.
 ```
-$ benchmark_driver bm_app_erb.yml -e ruby1::ruby -e ruby2::ruby
+$ exe/benchmark_driver ruby_benchmark_set/example_single.yml -e ruby1::ruby -e ruby2::ruby
 benchmark results:
 Execution time (sec)
 name             ruby1    ruby2
-bm_app_erb       1.028    1.010
+example_single   0.958    0.972
 Speedup ratio: compare with the result of `ruby1' (greater is better)
 name             ruby2
-bm_app_erb       1.018
+example_single   0.986
 ```
-And you can change benchmark output by `-r` option.
+And you can change benchmark output to IPS (iteration per second) by `-i` option.
 ```
-$ benchmark_driver bm_app_erb.yml -e ruby1::ruby -e ruby2::ruby -r ips
+$ exe/benchmark_driver ruby_benchmark_set/example_single.yml -e ruby1::ruby -e ruby2::ruby -i
 Result -------------------------------------------
                          ruby1          ruby2
-      bm_app_erb   16082.8 i/s    15541.7 i/s
+  example_single   99414.1 i/s    99723.3 i/s
-Comparison: bm_app_erb
-           ruby1:      16082.8 i/s
-           ruby2:      15541.7 i/s - 1.03x slower
+Comparison: example_single
+           ruby2:      99723.3 i/s
+           ruby1:      99414.1 i/s - 1.00x slower
 ```
 ### Running multiple scripts
 One YAML file can contain multiple benchmark scripts.
-With following `erb_compile_render.yml`,
+With following `example_multi.yml`,
 ```yml
-- name: erb_compile
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-  benchmark: |
-    ERB.new(data).src
-- name: erb_render
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-    src = "def self.render(title, content); #{ERB.new(data).src}; end"
-    mod = Module.new
-    mod.instance_eval(src, "(ERB)")
-  benchmark: |
-    mod.render(title, content)
+prelude: |
+  a = 'a' * 100
+  b = 'b' * 100
+benchmarks:
+  - name: join
+    benchmark: |
+      [a, b].join
+  - name: interpolation
+    benchmark: |
+      "#{a}#{b}"
 ```
 you can benchmark the scripts with multiple ruby executables.
 ```
-$ benchmark_driver erb_compile_render.yml -e ruby1::ruby -e ruby2::ruby
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -e ruby1::ruby -e ruby2::ruby
 benchmark results:
 Execution time (sec)
 name             ruby1    ruby2
-erb_compile      0.987    0.985
-erb_render       0.834    0.809
+join             0.022    0.022
+interpolation    0.026    0.026
 Speedup ratio: compare with the result of `ruby1' (greater is better)
 name             ruby2
-erb_compile      1.002
-erb_render       1.031
+join             1.045
+interpolation    1.002
 ```
 ```
-$ benchmark_driver erb_compile_render.yml -e ruby1::ruby -e ruby2::ruby -r ips
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -e ruby1::ruby -e ruby2::ruby -i
 Result -------------------------------------------
                          ruby1          ruby2
-     erb_compile   30374.0 i/s    30832.1 i/s
-      erb_render  628403.5 i/s   624588.0 i/s
+            join 4701954.3 i/s  4639520.3 i/s
+   interpolation 4263170.0 i/s  4044083.0 i/s
-Comparison: erb_compile
-           ruby2:      30832.1 i/s
-           ruby1:      30374.0 i/s - 1.02x slower
+Comparison: join
+           ruby1:    4701954.3 i/s
+           ruby2:    4639520.3 i/s - 1.01x slower
-Comparison: erb_render
-           ruby1:     628403.5 i/s
-           ruby2:     624588.0 i/s - 1.01x slower
+Comparison: interpolation
+           ruby1:    4263170.0 i/s
+           ruby2:    4044083.0 i/s - 1.05x slower
 ```
+### Configuring modes
+There are 2 modes:
+- Loop count: Enabled by `-l`. Optionally you can change count to loop by `-l COUNT`.
+- IPS: Enabled by `-i`. Optionally you can change duration by `-i DURATION`.
+Specifying both `-l` and `-i` is nonsense.
+### YAML syntax
+You can specify `benchmark:` or `benchmarks:`.
+#### Single
+```yml
+name: String # optional (default: file name)
+prelude: String # optional
+loop_count: Integer # optional
+benchmark: String # required
+```
+#### Multi
+```yml
+prelude: String # optional (shared)
+loop_count: Integer # optional (shared)
+benchmarks:
+  - name: String # required
+    prelude: String # optional (benchmark specific)
+    loop_count: Integer # optional (benchmark specific)
+    benchmark: String # required
+```
+### Debugging
+If you have a trouble like an unexpectedly fast result, you should check benchmark script by `-v`.
+```
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -v
+--- Running "join" with "ruby" 957780 times ---
+a = 'a' * 100
+b = 'b' * 100
+i = 0
+while i < 957780
+  i += 1
+[a, b].join
+end
+```
+## TODO
+- Measure multiple times and use minimum result
+- Retry and reject negative result in ips mode
+- Change not to take long time for iteration count estimation in ips mode
 ## Contributing

data/Rakefile CHANGED Viewed

@@ -1,6 +1,15 @@
-require "bundler/gem_tasks"
-require "rspec/core/rake_task"
+require 'bundler/gem_tasks'
-RSpec::Core::RakeTask.new(:spec)
+desc 'Run benchmarks in ruby_benchmark_set'
+task :ruby_benchmark_set do
+  require 'bundler'
+  require 'shellwords'
-task :default => :spec
+  Dir.glob(File.expand_path('./ruby_benchmark_set/**/*.yml', __dir__)).sort.each do |path|
+    Bundler.with_clean_env do
+      sh [File.expand_path('./exe/benchmark_driver', __dir__), path].shelljoin
+    end
+  end
+end
+task default: :ruby_benchmark_set

data/benchmark_driver.gemspec CHANGED Viewed

@@ -24,5 +24,4 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'bundler'
   spec.add_development_dependency 'rake'
-  spec.add_development_dependency 'rspec'
 end

data/exe/benchmark_driver CHANGED Viewed

@@ -8,17 +8,19 @@ require 'yaml'
 options = {}
 args = OptionParser.new do |o|
   o.banner = "Usage: #{File.basename($0, '.*')} [options] [YAML]"
-  o.on('-d', '--duration [SECONDS]', 'Duration seconds to run each benchmark (default: 1)') do |n|
-    options[:duration] = n.to_i
-  end
   o.on('-e', '--executables [EXECS]', 'Ruby executables (e1::path1; e2::path2; e3::path3;...)') do |e|
     options[:execs] ||= []
     e.split(/;/).each do |path|
       options[:execs] << path
     end
   end
-  o.on('-r', "--result-format=FORMAT", "Result output format [time|ips] (default: time)", %w[time ips]) do |r|
-    options[:result_format] = r
+  o.on('-i [DURATION]', '--ips [SECONDS]', "Measure IPS in duration seconds (default: #{BenchmarkDriver::DEFAULT_IPS_DURATION})") do |i|
+    options[:measure_type] = 'ips'
+    options[:measure_num]  = i if i
+  end
+  o.on('-l [COUNT]', '--loop-count [COUNT]', "Measure execution time with loop count (default: #{BenchmarkDriver::DEFAULT_LOOP_COUNT})") do |l|
+    options[:measure_type] = 'loop_count'
+    options[:measure_num]  = l if l
   end
   o.on('-v', '--verbose') do |v|
     options[:verbose] = v
@@ -28,9 +30,6 @@ abort "No YAML file is specified" if args.empty?
 driver = BenchmarkDriver.new(options)
 args.each do |yaml|
-  benchmarks = YAML.load(File.read(yaml))
-  if benchmarks.is_a?(Hash)
-    benchmarks[:name] = File.basename(yaml, '.*')
-  end
-  driver.run(benchmarks)
+  default = { name: File.basename(yaml, '.*') }
+  driver.run(default.merge(YAML.load(File.read(yaml))))
 end

data/lib/benchmark_driver.rb CHANGED Viewed

@@ -3,35 +3,39 @@ require 'benchmark'
 require 'tempfile'
 class BenchmarkDriver
-  # @param [Integer] duration - Benchmark duration in seconds
+  MEASURE_TYPES = %w[loop_count ips]
+  DEFAULT_LOOP_COUNT = 100_000
+  DEFAULT_IPS_DURATION = 1
+  # @param [String] measure_type - "loop_count"|"ips"
+  # @param [Integer] measure_num - Loop count for "loop_type", duration seconds for "ips"
   # @param [Array<String>] execs - ["path1", "path2"] or `["ruby1::path1", "ruby2::path2"]`
-  # @param [String] result_format
   # @param [Boolean] verbose
-  def initialize(duration: 1, execs: ['ruby'], result_format: 'time', verbose: false)
-    @duration = duration
+  def initialize(measure_type: 'loop_count', measure_num: nil, execs: ['ruby'], verbose: false)
+    unless MEASURE_TYPES.include?(measure_type)
+      abort "unsupported measure type: #{measure_type.dump}"
+    end
+    @measure_type = measure_type
+    @measure_num = measure_num
     @execs = execs.map do |exec|
       name, path = exec.split('::', 2)
       Executable.new(name, path || name)
     end
-    @result_format = result_format
     @verbose = verbose
   end
-  # @param [Hash,Array<Hash>] hashes
-  def run(hashes)
-    hashes = [hashes] if hashes.is_a?(Hash)
-    benchmarks = hashes.map do |hash|
-      BenchmarkScript.new(Hash[hash.map { |k, v| [k.to_sym, v] }])
-    end
-    if benchmarks.empty?
-      abort 'No benchmark is specified in YAML'
-    end
+  # @param [Hash] root_hash
+  def run(root_hash)
+    root = BenchmarkRoot.new(Hash[root_hash.map { |k, v| [k.to_sym, v] }])
-    results = benchmarks.map do |benchmark|
+    results = root.benchmarks.map do |benchmark|
       metrics_by_exec = {}
       iterations = calc_iterations(@execs.first, benchmark)
       @execs.each do |exec|
-        puts "Running #{benchmark.name.dump} with #{exec.name.dump} #{iterations} times..." if @verbose
+        if @verbose
+          puts "--- Running #{benchmark.name.dump} with #{exec.name.dump} #{iterations} times ---"
+          puts "#{benchmark.benchmark_script(iterations)}\n"
+        end
         elapsed_time = run_benchmark(exec, benchmark, iterations)
         metrics_by_exec[exec] = BenchmarkMetrics.new(iterations, elapsed_time)
       end
@@ -39,13 +43,13 @@ class BenchmarkDriver
     end
     puts if @verbose
-    case @result_format
-    when 'time'
-      ExecutionTimeReporter.report(@execs, results)
+    case @measure_type
+    when 'loop_count'
+      LoopCountReporter.report(@execs, results)
     when 'ips'
       IpsReporter.report(@execs, results)
     else
-      raise "unsupported result format: #{@result_format.dump}"
+      raise "unexpected measure type: #{@measure_type.dump}"
     end
   end
@@ -53,10 +57,18 @@ class BenchmarkDriver
   # Estimate iterations to finish benchmark within `@duration`.
   def calc_iterations(exec, benchmark)
-    # TODO: Change to try from 1, 10, 100 ...
-    base = 1000
-    time = run_benchmark(exec, benchmark, base)
-    (@duration / time * base).to_i
+    case @measure_type
+    when 'loop_count'
+      @measure_num || benchmark.loop_count || DEFAULT_LOOP_COUNT
+    when 'ips'
+      # TODO: Change to try from 1, 10, 100 ...
+      base = 1000
+      time = run_benchmark(exec, benchmark, base)
+      duration = @measure_num || DEFAULT_IPS_DURATION
+      (duration / time * base).to_i
+    else
+      raise "unexpected measure type: #{@measure_type.dump}"
+    end
   end
   def run_benchmark(exec, benchmark, iterations)
@@ -75,17 +87,55 @@ class BenchmarkDriver
     end
   end
+  class BenchmarkRoot
+    # @param [String] name
+    # @param [String] prelude
+    # @param [Integer,nil] loop_count
+    # @param [String,nil]  benchmark  - For running single instant benchmark
+    # @param [Array<Hash>] benchmarks - For running multiple benchmarks
+    def initialize(name:, prelude: '', loop_count: nil, benchmark: nil, benchmarks: [])
+      if benchmark
+        unless benchmarks.empty?
+          raise ArgumentError.new("Only either :benchmark or :benchmarks can be specified")
+        end
+        @benchmarks = [BenchmarkScript.new(name: name, prelude: prelude, benchmark: benchmark)]
+      else
+        @benchmarks = benchmarks.map do |hash|
+          BenchmarkScript.new(Hash[hash.map { |k, v| [k.to_sym, v] }]).tap do |b|
+            b.inherit_root(prelude: prelude, loop_count: loop_count)
+          end
+        end
+      end
+    end
+    # @return [Array<BenchmarkScript>]
+    attr_reader :benchmarks
+  end
   class BenchmarkScript
     # @param [String] name
     # @param [String] prelude
-    # @param [String] script
-    def initialize(name:, prelude: '', benchmark:)
+    # @param [String] benchmark
+    def initialize(name:, prelude: '', loop_count: nil, benchmark:)
       @name = name
       @prelude = prelude
+      @loop_count = loop_count
       @benchmark = benchmark
     end
+    # @return [String]
     attr_reader :name
+    # @return [Integer]
+    attr_reader :loop_count
+    def inherit_root(prelude:, loop_count:)
+      @prelude = "#{prelude}\n#{@prelude}"
+      if @loop_count.nil? && loop_count
+        @loop_count = loop_count
+      end
+    end
     def overhead_script(iterations)
       <<-RUBY
 #{@prelude}
@@ -137,7 +187,7 @@ end
   )
   end
-  module ExecutionTimeReporter
+  module LoopCountReporter
     class << self
       # @param [Array<Executable>] execs
       # @param [Array<BenchmarkResult>] results

data/lib/benchmark_driver/version.rb CHANGED Viewed

@@ -1,3 +1,3 @@
 class BenchmarkDriver
-  VERSION = '0.1.0'
+  VERSION = '0.2.0'
 end

data/ruby_benchmark_set/core/array.yml ADDED Viewed

@@ -0,0 +1,4 @@
+benchmarks:
+  - name: bm_vm2_array
+    benchmark: a = [1,2,3,4,5,6,7,8,9,10]
+    loop_count: 6000000

data/ruby_benchmark_set/example_multi.yml ADDED Viewed

@@ -0,0 +1,10 @@
+prelude: |
+  a = 'a' * 100
+  b = 'b' * 100
+benchmarks:
+  - name: join
+    benchmark: |
+      [a, b].join
+  - name: interpolation
+    benchmark: |
+      "#{a}#{b}"

data/ruby_benchmark_set/example_single.yml ADDED Viewed

@@ -0,0 +1,4 @@
+prelude: |
+  require 'erb'
+  erb = ERB.new(%q[Hello <%= 'World' %>])
+benchmark: erb.result

data/ruby_benchmark_set/lib/erb.yml ADDED Viewed

@@ -0,0 +1,30 @@
+prelude: |
+  require 'erb'
+  template = <<EOS
+  <html>
+    <head> <%= title %> </head>
+    <body>
+      <h1> <%= title %> </h1>
+      <p>
+        <%= content %>
+      </p>
+    </body>
+  </html>
+  EOS
+  title = "hello world!"
+  content = "hello world!\n" * 10
+benchmarks:
+  - name: ERB compiling
+    benchmark: ERB.new(template)
+    loop_count: 10000
+  - name: ERB rendering
+    prelude: |
+      src = "def self.render(title, content); #{ERB.new(template).src}; end"
+      mod = Module.new
+      mod.instance_eval(src, "(ERB)")
+    benchmark: mod.render(title, content)
+    loop_count: 100000

metadata CHANGED Viewed

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: benchmark_driver
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Takashi Kokubun
@@ -38,20 +38,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 description: Benchmark driver for different Ruby executables
 email:
 - takashikkbn@gmail.com
@@ -61,7 +47,6 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".gitignore"
-- ".rspec"
 - ".travis.yml"
 - Gemfile
 - LICENSE.txt
@@ -70,11 +55,13 @@ files:
 - benchmark_driver.gemspec
 - bin/console
 - bin/setup
-- examples/bm_app_erb.yml
-- examples/erb_compile_render.yml
 - exe/benchmark_driver
 - lib/benchmark_driver.rb
 - lib/benchmark_driver/version.rb
+- ruby_benchmark_set/core/array.yml
+- ruby_benchmark_set/example_multi.yml
+- ruby_benchmark_set/example_single.yml
+- ruby_benchmark_set/lib/erb.yml
 homepage: https://github.com/k0kubun/benchmark_driver
 licenses:
 - MIT

data/.rspec DELETED Viewed

	@@ -1,2 +0,0 @@
1	- --format documentation
2	- --color

data/examples/bm_app_erb.yml DELETED Viewed

@@ -1,20 +0,0 @@
-prelude: |
-  require 'erb'
-  title = "hello world!"
-  content = "hello world!\n" * 10
-  data = <<EOS
-  <html>
-    <head> <%= title %> </head>
-    <body>
-      <h1> <%= title %> </h1>
-      <p>
-        <%= content %>
-      </p>
-    </body>
-  </html>
-  EOS
-benchmark: |
-  ERB.new(data).result(binding)

data/examples/erb_compile_render.yml DELETED Viewed

@@ -1,47 +0,0 @@
-- name: erb_compile
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-  benchmark: |
-    ERB.new(data).src
-- name: erb_render
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-    src = "def self.render(title, content); #{ERB.new(data).src}; end"
-    mod = Module.new
-    mod.instance_eval(src, "(ERB)")
-  benchmark: |
-    mod.render(title, content)