RubyGems - benchmark_driver - Versions diffs - 0.1.0 → 0.2.0 - Mend

benchmark_driver 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (15) hide show

checksums.yaml +4 -4
data/README.md +97 -94
data/Rakefile +13 -4
data/benchmark_driver.gemspec +0 -1
data/exe/benchmark_driver +9 -10
data/lib/benchmark_driver.rb +77 -27
data/lib/benchmark_driver/version.rb +1 -1
data/ruby_benchmark_set/core/array.yml +4 -0
data/ruby_benchmark_set/example_multi.yml +10 -0
data/ruby_benchmark_set/example_single.yml +4 -0
data/ruby_benchmark_set/lib/erb.yml +30 -0
metadata +5 -18
data/.rspec +0 -2
data/examples/bm_app_erb.yml +0 -20
data/examples/erb_compile_render.yml +0 -47

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a09ccdd10d22d70ca190e43c4cdfa7bb6606f429
-  data.tar.gz: 6346d84cc347aa2111842e1b07c36138683c9d5b
+  metadata.gz: 027b33623e6d71484da812d92ef6d4acbf154312
+  data.tar.gz: cb8da42411f02760787b766fa0d8a8622750ecd4
 SHA512:
-  metadata.gz: eb5e63d89f0c344189467bde51efdd89a9a0e473eba30c15395d2c7f0e533f170ce164586b8c03a8e96f4cf89f5c804f0ff2f589e92b642e1b129ec00baccb8c
-  data.tar.gz: 87b3a29d42c3b6628e572741902739e7f104a0f00a21e0a24e44dc22edacc623f14dd2524db90083a398a7bb8540fbc9cb9b2ee0c6250c3708290a89a1e49298
+  metadata.gz: a666e41732c84ab63b0b93460b77b01e2b1540b00199f4c71971b7da532f0531bd6daf9813c87a91f55be51fc37fd2ca12723062b294017986d9f98261df4475
+  data.tar.gz: ea81b5f7b48a02c99f5718ec4dff48e25dc07eede56d1194172ca209057a4d170405082907d2f3937a91779ee11a08e1dc443afbc29a355782d540923382c5a0

data/README.md CHANGED Viewed

@@ -1,4 +1,4 @@
-# BenchmarkDriver
+# BenchmarkDriver [![Build Status](https://travis-ci.org/k0kubun/benchmark_driver.svg?branch=master)](https://travis-ci.org/k0kubun/benchmark_driver)
 Benchmark driver for different Ruby executables
@@ -9,155 +9,158 @@ Benchmark driver for different Ruby executables
 ## Usage
 ```
-$ benchmark_driver -h
+$ exe/benchmark_driver -h
 Usage: benchmark_driver [options] [YAML]
-    -d, --duration [SECONDS]         Duration seconds to run each benchmark (default: 1)
     -e, --executables [EXECS]        Ruby executables (e1::path1; e2::path2; e3::path3;...)
-    -r, --result-format=FORMAT       Result output format [time|ips] (default: time)
+    -i, --ips [SECONDS]              Measure IPS in duration seconds (default: 1)
+    -l, --loop-count [COUNT]         Measure execution time with loop count (default: 100000)
     -v, --verbose
 ```
 ### Running single script
-With following `bm_app_erb.yml`,
+With following `example_single.yml`,
 ```yml
 prelude: |
   require 'erb'
-  title = "hello world!"
-  content = "hello world!\n" * 10
-  data = <<EOS
-  <html>
-    <head> <%= title %> </head>
-    <body>
-      <h1> <%= title %> </h1>
-      <p>
-        <%= content %>
-      </p>
-    </body>
-  </html>
-  EOS
-benchmark: |
-  ERB.new(data).result(binding)
+  erb = ERB.new(%q[Hello <%= 'World' %>])
+benchmark: erb.result
 ```
 you can benchmark the script with multiple ruby executables.
 ```
-$ benchmark_driver bm_app_erb.yml -e ruby1::ruby -e ruby2::ruby
+$ exe/benchmark_driver ruby_benchmark_set/example_single.yml -e ruby1::ruby -e ruby2::ruby
 benchmark results:
 Execution time (sec)
 name             ruby1    ruby2
-bm_app_erb       1.028    1.010
+example_single   0.958    0.972
 Speedup ratio: compare with the result of `ruby1' (greater is better)
 name             ruby2
-bm_app_erb       1.018
+example_single   0.986
 ```
-And you can change benchmark output by `-r` option.
+And you can change benchmark output to IPS (iteration per second) by `-i` option.
 ```
-$ benchmark_driver bm_app_erb.yml -e ruby1::ruby -e ruby2::ruby -r ips
+$ exe/benchmark_driver ruby_benchmark_set/example_single.yml -e ruby1::ruby -e ruby2::ruby -i
 Result -------------------------------------------
                          ruby1          ruby2
-      bm_app_erb   16082.8 i/s    15541.7 i/s
+  example_single   99414.1 i/s    99723.3 i/s
-Comparison: bm_app_erb
-           ruby1:      16082.8 i/s
-           ruby2:      15541.7 i/s - 1.03x slower
+Comparison: example_single
+           ruby2:      99723.3 i/s
+           ruby1:      99414.1 i/s - 1.00x slower
 ```
 ### Running multiple scripts
 One YAML file can contain multiple benchmark scripts.
-With following `erb_compile_render.yml`,
+With following `example_multi.yml`,
 ```yml
-- name: erb_compile
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-  benchmark: |
-    ERB.new(data).src
-- name: erb_render
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-    src = "def self.render(title, content); #{ERB.new(data).src}; end"
-    mod = Module.new
-    mod.instance_eval(src, "(ERB)")
-  benchmark: |
-    mod.render(title, content)
+prelude: |
+  a = 'a' * 100
+  b = 'b' * 100
+benchmarks:
+  - name: join
+    benchmark: |
+      [a, b].join
+  - name: interpolation
+    benchmark: |
+      "#{a}#{b}"
 ```
 you can benchmark the scripts with multiple ruby executables.
 ```
-$ benchmark_driver erb_compile_render.yml -e ruby1::ruby -e ruby2::ruby
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -e ruby1::ruby -e ruby2::ruby
 benchmark results:
 Execution time (sec)
 name             ruby1    ruby2
-erb_compile      0.987    0.985
-erb_render       0.834    0.809
+join             0.022    0.022
+interpolation    0.026    0.026
 Speedup ratio: compare with the result of `ruby1' (greater is better)
 name             ruby2
-erb_compile      1.002
-erb_render       1.031
+join             1.045
+interpolation    1.002
 ```
 ```
-$ benchmark_driver erb_compile_render.yml -e ruby1::ruby -e ruby2::ruby -r ips
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -e ruby1::ruby -e ruby2::ruby -i
 Result -------------------------------------------
                          ruby1          ruby2
-     erb_compile   30374.0 i/s    30832.1 i/s
-      erb_render  628403.5 i/s   624588.0 i/s
+            join 4701954.3 i/s  4639520.3 i/s
+   interpolation 4263170.0 i/s  4044083.0 i/s
-Comparison: erb_compile
-           ruby2:      30832.1 i/s
-           ruby1:      30374.0 i/s - 1.02x slower
+Comparison: join
+           ruby1:    4701954.3 i/s
+           ruby2:    4639520.3 i/s - 1.01x slower
-Comparison: erb_render
-           ruby1:     628403.5 i/s
-           ruby2:     624588.0 i/s - 1.01x slower
+Comparison: interpolation
+           ruby1:    4263170.0 i/s
+           ruby2:    4044083.0 i/s - 1.05x slower
 ```
+### Configuring modes
+There are 2 modes:
+- Loop count: Enabled by `-l`. Optionally you can change count to loop by `-l COUNT`.
+- IPS: Enabled by `-i`. Optionally you can change duration by `-i DURATION`.
+Specifying both `-l` and `-i` is nonsense.
+### YAML syntax
+You can specify `benchmark:` or `benchmarks:`.
+#### Single
+```yml
+name: String # optional (default: file name)
+prelude: String # optional
+loop_count: Integer # optional
+benchmark: String # required
+```
+#### Multi
+```yml
+prelude: String # optional (shared)
+loop_count: Integer # optional (shared)
+benchmarks:
+  - name: String # required
+    prelude: String # optional (benchmark specific)
+    loop_count: Integer # optional (benchmark specific)
+    benchmark: String # required
+```
+### Debugging
+If you have a trouble like an unexpectedly fast result, you should check benchmark script by `-v`.
+```
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -v
+--- Running "join" with "ruby" 957780 times ---
+a = 'a' * 100
+b = 'b' * 100
+i = 0
+while i < 957780
+  i += 1
+[a, b].join
+end
+```
+## TODO
+- Measure multiple times and use minimum result
+- Retry and reject negative result in ips mode
+- Change not to take long time for iteration count estimation in ips mode
 ## Contributing

data/Rakefile CHANGED Viewed

@@ -1,6 +1,15 @@
-require "bundler/gem_tasks"
-require "rspec/core/rake_task"
+require 'bundler/gem_tasks'
-RSpec::Core::RakeTask.new(:spec)
+desc 'Run benchmarks in ruby_benchmark_set'
+task :ruby_benchmark_set do
+  require 'bundler'
+  require 'shellwords'
-task :default => :spec
+  Dir.glob(File.expand_path('./ruby_benchmark_set/**/*.yml', __dir__)).sort.each do |path|
+    Bundler.with_clean_env do
+      sh [File.expand_path('./exe/benchmark_driver', __dir__), path].shelljoin
+    end
+  end
+end
+task default: :ruby_benchmark_set

data/benchmark_driver.gemspec CHANGED Viewed

@@ -24,5 +24,4 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'bundler'
   spec.add_development_dependency 'rake'
-  spec.add_development_dependency 'rspec'
 end

data/exe/benchmark_driver CHANGED Viewed

@@ -8,17 +8,19 @@ require 'yaml'
 options = {}
 args = OptionParser.new do |o|
   o.banner = "Usage: #{File.basename($0, '.*')} [options] [YAML]"
-  o.on('-d', '--duration [SECONDS]', 'Duration seconds to run each benchmark (default: 1)') do |n|
-    options[:duration] = n.to_i
-  end
   o.on('-e', '--executables [EXECS]', 'Ruby executables (e1::path1; e2::path2; e3::path3;...)') do |e|
     options[:execs] ||= []
     e.split(/;/).each do |path|
       options[:execs] << path
     end
   end
-  o.on('-r', "--result-format=FORMAT", "Result output format [time|ips] (default: time)", %w[time ips]) do |r|
-    options[:result_format] = r
+  o.on('-i [DURATION]', '--ips [SECONDS]', "Measure IPS in duration seconds (default: #{BenchmarkDriver::DEFAULT_IPS_DURATION})") do |i|
+    options[:measure_type] = 'ips'
+    options[:measure_num]  = i if i
+  end
+  o.on('-l [COUNT]', '--loop-count [COUNT]', "Measure execution time with loop count (default: #{BenchmarkDriver::DEFAULT_LOOP_COUNT})") do |l|
+    options[:measure_type] = 'loop_count'
+    options[:measure_num]  = l if l
   end
   o.on('-v', '--verbose') do |v|
     options[:verbose] = v
@@ -28,9 +30,6 @@ abort "No YAML file is specified" if args.empty?
 driver = BenchmarkDriver.new(options)
 args.each do |yaml|
-  benchmarks = YAML.load(File.read(yaml))
-  if benchmarks.is_a?(Hash)
-    benchmarks[:name] = File.basename(yaml, '.*')
-  end
-  driver.run(benchmarks)
+  default = { name: File.basename(yaml, '.*') }
+  driver.run(default.merge(YAML.load(File.read(yaml))))
 end

data/lib/benchmark_driver.rb CHANGED Viewed

@@ -3,35 +3,39 @@ require 'benchmark'
 require 'tempfile'
 class BenchmarkDriver
-  # @param [Integer] duration - Benchmark duration in seconds
+  MEASURE_TYPES = %w[loop_count ips]
+  DEFAULT_LOOP_COUNT = 100_000
+  DEFAULT_IPS_DURATION = 1
+  # @param [String] measure_type - "loop_count"|"ips"
+  # @param [Integer] measure_num - Loop count for "loop_type", duration seconds for "ips"
   # @param [Array<String>] execs - ["path1", "path2"] or `["ruby1::path1", "ruby2::path2"]`
-  # @param [String] result_format
   # @param [Boolean] verbose
-  def initialize(duration: 1, execs: ['ruby'], result_format: 'time', verbose: false)
-    @duration = duration
+  def initialize(measure_type: 'loop_count', measure_num: nil, execs: ['ruby'], verbose: false)
+    unless MEASURE_TYPES.include?(measure_type)
+      abort "unsupported measure type: #{measure_type.dump}"
+    end
+    @measure_type = measure_type
+    @measure_num = measure_num
     @execs = execs.map do |exec|
       name, path = exec.split('::', 2)
       Executable.new(name, path || name)
     end
-    @result_format = result_format
     @verbose = verbose
   end
-  # @param [Hash,Array<Hash>] hashes
-  def run(hashes)
-    hashes = [hashes] if hashes.is_a?(Hash)
-    benchmarks = hashes.map do |hash|
-      BenchmarkScript.new(Hash[hash.map { |k, v| [k.to_sym, v] }])
-    end
-    if benchmarks.empty?
-      abort 'No benchmark is specified in YAML'
-    end
+  # @param [Hash] root_hash
+  def run(root_hash)
+    root = BenchmarkRoot.new(Hash[root_hash.map { |k, v| [k.to_sym, v] }])
-    results = benchmarks.map do |benchmark|
+    results = root.benchmarks.map do |benchmark|
       metrics_by_exec = {}
       iterations = calc_iterations(@execs.first, benchmark)
       @execs.each do |exec|
-        puts "Running #{benchmark.name.dump} with #{exec.name.dump} #{iterations} times..." if @verbose
+        if @verbose
+          puts "--- Running #{benchmark.name.dump} with #{exec.name.dump} #{iterations} times ---"
+          puts "#{benchmark.benchmark_script(iterations)}\n"
+        end
         elapsed_time = run_benchmark(exec, benchmark, iterations)
         metrics_by_exec[exec] = BenchmarkMetrics.new(iterations, elapsed_time)
       end
@@ -39,13 +43,13 @@ class BenchmarkDriver
     end
     puts if @verbose
-    case @result_format
-    when 'time'
-      ExecutionTimeReporter.report(@execs, results)
+    case @measure_type
+    when 'loop_count'
+      LoopCountReporter.report(@execs, results)
     when 'ips'
       IpsReporter.report(@execs, results)
     else
-      raise "unsupported result format: #{@result_format.dump}"
+      raise "unexpected measure type: #{@measure_type.dump}"
     end
   end
@@ -53,10 +57,18 @@ class BenchmarkDriver
   # Estimate iterations to finish benchmark within `@duration`.
   def calc_iterations(exec, benchmark)
-    # TODO: Change to try from 1, 10, 100 ...
-    base = 1000
-    time = run_benchmark(exec, benchmark, base)
-    (@duration / time * base).to_i
+    case @measure_type
+    when 'loop_count'
+      @measure_num || benchmark.loop_count || DEFAULT_LOOP_COUNT
+    when 'ips'
+      # TODO: Change to try from 1, 10, 100 ...
+      base = 1000
+      time = run_benchmark(exec, benchmark, base)
+      duration = @measure_num || DEFAULT_IPS_DURATION
+      (duration / time * base).to_i
+    else
+      raise "unexpected measure type: #{@measure_type.dump}"
+    end
   end
   def run_benchmark(exec, benchmark, iterations)
@@ -75,17 +87,55 @@ class BenchmarkDriver
     end
   end
+  class BenchmarkRoot
+    # @param [String] name
+    # @param [String] prelude
+    # @param [Integer,nil] loop_count
+    # @param [String,nil]  benchmark  - For running single instant benchmark
+    # @param [Array<Hash>] benchmarks - For running multiple benchmarks
+    def initialize(name:, prelude: '', loop_count: nil, benchmark: nil, benchmarks: [])
+      if benchmark
+        unless benchmarks.empty?
+          raise ArgumentError.new("Only either :benchmark or :benchmarks can be specified")
+        end
+        @benchmarks = [BenchmarkScript.new(name: name, prelude: prelude, benchmark: benchmark)]
+      else
+        @benchmarks = benchmarks.map do |hash|
+          BenchmarkScript.new(Hash[hash.map { |k, v| [k.to_sym, v] }]).tap do |b|
+            b.inherit_root(prelude: prelude, loop_count: loop_count)
+          end
+        end
+      end
+    end
+    # @return [Array<BenchmarkScript>]
+    attr_reader :benchmarks
+  end
   class BenchmarkScript
     # @param [String] name
     # @param [String] prelude
-    # @param [String] script
-    def initialize(name:, prelude: '', benchmark:)
+    # @param [String] benchmark
+    def initialize(name:, prelude: '', loop_count: nil, benchmark:)
       @name = name
       @prelude = prelude
+      @loop_count = loop_count
       @benchmark = benchmark
     end
+    # @return [String]
     attr_reader :name
+    # @return [Integer]
+    attr_reader :loop_count
+    def inherit_root(prelude:, loop_count:)
+      @prelude = "#{prelude}\n#{@prelude}"
+      if @loop_count.nil? && loop_count
+        @loop_count = loop_count
+      end
+    end
     def overhead_script(iterations)
       <<-RUBY
 #{@prelude}
@@ -137,7 +187,7 @@ end
   )
   end
-  module ExecutionTimeReporter
+  module LoopCountReporter
     class << self
       # @param [Array<Executable>] execs
       # @param [Array<BenchmarkResult>] results

data/lib/benchmark_driver/version.rb CHANGED Viewed

@@ -1,3 +1,3 @@
 class BenchmarkDriver
-  VERSION = '0.1.0'
+  VERSION = '0.2.0'
 end

data/ruby_benchmark_set/core/array.yml ADDED Viewed

@@ -0,0 +1,4 @@
+benchmarks:
+  - name: bm_vm2_array
+    benchmark: a = [1,2,3,4,5,6,7,8,9,10]
+    loop_count: 6000000

data/ruby_benchmark_set/example_multi.yml ADDED Viewed

@@ -0,0 +1,10 @@
+prelude: |
+  a = 'a' * 100
+  b = 'b' * 100
+benchmarks:
+  - name: join
+    benchmark: |
+      [a, b].join
+  - name: interpolation
+    benchmark: |
+      "#{a}#{b}"

data/ruby_benchmark_set/example_single.yml ADDED Viewed

@@ -0,0 +1,4 @@
+prelude: |
+  require 'erb'
+  erb = ERB.new(%q[Hello <%= 'World' %>])
+benchmark: erb.result

data/ruby_benchmark_set/lib/erb.yml ADDED Viewed

@@ -0,0 +1,30 @@
+prelude: |
+  require 'erb'
+  template = <<EOS
+  <html>
+    <head> <%= title %> </head>
+    <body>
+      <h1> <%= title %> </h1>
+      <p>
+        <%= content %>
+      </p>
+    </body>
+  </html>
+  EOS
+  title = "hello world!"
+  content = "hello world!\n" * 10
+benchmarks:
+  - name: ERB compiling
+    benchmark: ERB.new(template)
+    loop_count: 10000
+  - name: ERB rendering
+    prelude: |
+      src = "def self.render(title, content); #{ERB.new(template).src}; end"
+      mod = Module.new
+      mod.instance_eval(src, "(ERB)")
+    benchmark: mod.render(title, content)
+    loop_count: 100000

metadata CHANGED Viewed

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: benchmark_driver
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Takashi Kokubun
@@ -38,20 +38,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 description: Benchmark driver for different Ruby executables
 email:
 - takashikkbn@gmail.com
@@ -61,7 +47,6 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".gitignore"
-- ".rspec"
 - ".travis.yml"
 - Gemfile
 - LICENSE.txt
@@ -70,11 +55,13 @@ files:
 - benchmark_driver.gemspec
 - bin/console
 - bin/setup
-- examples/bm_app_erb.yml
-- examples/erb_compile_render.yml
 - exe/benchmark_driver
 - lib/benchmark_driver.rb
 - lib/benchmark_driver/version.rb
+- ruby_benchmark_set/core/array.yml
+- ruby_benchmark_set/example_multi.yml
+- ruby_benchmark_set/example_single.yml
+- ruby_benchmark_set/lib/erb.yml
 homepage: https://github.com/k0kubun/benchmark_driver
 licenses:
 - MIT

data/.rspec DELETED Viewed

	@@ -1,2 +0,0 @@
1	- --format documentation
2	- --color

data/examples/bm_app_erb.yml DELETED Viewed

@@ -1,20 +0,0 @@
-prelude: |
-  require 'erb'
-  title = "hello world!"
-  content = "hello world!\n" * 10
-  data = <<EOS
-  <html>
-    <head> <%= title %> </head>
-    <body>
-      <h1> <%= title %> </h1>
-      <p>
-        <%= content %>
-      </p>
-    </body>
-  </html>
-  EOS
-benchmark: |
-  ERB.new(data).result(binding)

data/examples/erb_compile_render.yml DELETED Viewed

@@ -1,47 +0,0 @@
-- name: erb_compile
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-  benchmark: |
-    ERB.new(data).src
-- name: erb_render
-  prelude: |
-    require 'erb'
-    title = "hello world!"
-    content = "hello world!\n" * 10
-    data = <<EOS
-    <html>
-      <head> <%= title %> </head>
-      <body>
-        <h1> <%= title %> </h1>
-        <p>
-          <%= content %>
-        </p>
-      </body>
-    </html>
-    EOS
-    src = "def self.render(title, content); #{ERB.new(data).src}; end"
-    mod = Module.new
-    mod.instance_eval(src, "(ERB)")
-  benchmark: |
-    mod.render(title, content)