better-benchmark 0.7.0 → 0.8.1

data/README.md

@@ -0,0 +1,115 @@
+# Better Benchmark
+
+Statistically correct benchmarking for Ruby.
+
+## Dependencies
+
+* [The R Project](http://www.r-project.org/)
+* [rsruby](http://github.com/alexgutteridge/rsruby)
+
+## Usage
+
+### Comparing code blocks
+
+    result = Benchmark.compare_realtime {
+      do_something_one_way
+    }.with {
+      do_it_another_way
+    }
+    Benchmark.report_on result
+
+See also example.rb for a more comprehensive example.
+
+### Comparing git revisions
+
+#### With a test script (recommended)
+
+To test two revisions of a library, create a simple runner script:
+
+    # runner.rb
+    require 'mylib'
+
+    class TestQuick
+      def initialize
+        # initialization...
+      end
+
+      def run
+        Benchmark.write_realtime( '/home/pistos/tmp' ) do
+          5000.times do
+            # do something with your lib
+          end
+        end
+      end
+    end
+
+    t = TestQuick.new
+    t.run
+
+Then run the bbench script, passing two git revisions:
+
+    bbench -r 6e84dd5 -r ed1e7c6 -d ~/tmp -- -Ilib runner.rb
+
+#### Without altering or writing new code
+
+You can also test two revisions by running some already-existing script,
+such as a file in your test suite:
+
+    bbench -r 6e84dd5 -r ed1e7c6 -- -Itest -Ilib test/test_something.rb
+
+Be aware, however, that this may produce unnecessarily variant timings due to
+wide variance in the startup time of the Ruby interpreter and script.
+
+### Comparing git working copy
+
+You can also compare the current branch tip to the current (dirty) working copy:
+
+    bbench -w -d ~/tmp -- -Ilib runner.rb
+
+This lets you experiment without committing anything, and then only commit
+when you are confident that your changes result in a performance improvement.
+
+## Interpretation
+
+Considering two "things under test", U1 and U2:
+
+### Example 1
+
+    Set 1 mean: 0.216 s
+    Set 1 std dev: 0.023
+    Set 2 mean: 0.187 s
+    Set 2 std dev: 0.020
+    p.value: 0.00287947346770876
+    W: 88.0
+    The difference (-13.5%) IS statistically significant.
+
+This means that the results permit us to conclude that U2 performed 13.5%
+faster than U1.
+
+### Example 2
+
+    Set 1 mean: 10.968 s
+    Set 1 std dev: 4.294
+    Set 2 mean: 9.036 s
+    Set 2 std dev: 3.581
+    p.value: 0.217562623135379
+    W: 67.0
+    The difference (-17.6%) IS NOT statistically significant.
+
+This means that the results do not permit us to conclude that the performance
+of U1 and U2 differed.
+
+## Not just Ruby
+
+Technically, the bbench script can work with any script or program that writes
+a run time (in seconds) to the file bbench-run-time in the data dir.  Use the
+-e option to specify a different executable than "ruby".  e.g. perl, python,
+java, etc.
+
+## Help, etc.
+
+irc.freenode.net#mathetes or http://webchat.freenode.net?channels=mathetes .
+
+## Repository
+
+git clone git://github.com/Pistos/better-benchmark.git

data/bin/bbench

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'better-benchmark'
+
+b = ::Benchmark::Bencher.new( ARGV.dup )
+b.run

data/example.rb

@@ -1,5 +1,6 @@
 #!/usr/bin/env ruby
 
+require 'rubygems'
 require 'better-benchmark'
 
 # Provide two blocks of code to compare.  For example, two blocks that

data/lib/better-benchmark.rb

@@ -1,59 +1,49 @@
 require 'benchmark'
 require 'rsruby'
 
+require 'better-benchmark/comparison-partial'
+require 'better-benchmark/bencher'
+
 module Benchmark
 
-  BETTER_BENCHMARK_VERSION = '0.7.0'
+  BETTER_BENCHMARK_VERSION = '0.8.0'
+  DEFAULT_REQUIRED_SIGNIFICANCE = 0.01
 
-  class ComparisonPartial
-    def initialize( block, options )
-      @block1 = block
-      @options = options
+  def self.write_realtime( data_dir, &block )
+    t = Benchmark.realtime( &block )
+    File.open( "#{data_dir}/#{Bencher::DATA_FILE}", 'w' ) do |f|
+      f.print t
     end
+  end
 
-    def with( &block2 )
-      times1 = []
-      times2 = []
-
-      (1..@options[ :iterations ]).each do |iteration|
-        if @options[ :verbose ]
-          $stdout.print "."; $stdout.flush
-        end
-
-        times1 << Benchmark.realtime do
-          @options[ :inner_iterations ].times do |i|
-            @block1.call( iteration )
-          end
-        end
-        times2 << Benchmark.realtime do
-          @options[ :inner_iterations ].times do |i|
-            block2.call( iteration )
-          end
-        end
-      end
-
-      r = RSRuby.instance
-      wilcox_result = r.wilcox_test( times1, times2 )
+  # The number of elements in times1 and times2 should be the same.
+  # @param [Array] times1
+  #   An Array of elapsed times in float form, measured in seconds
+  # @param [Array] times2
+  #   An Array of elapsed times in float form, measured in seconds
+  # @param [Fixnum] required_significance
+  #   The maximum p value needed to declare statistical significance
+  def self.compare_times( times1, times2, required_significance = DEFAULT_REQUIRED_SIGNIFICANCE )
+    r = RSRuby.instance
+    wilcox_result = r.wilcox_test( times1, times2 )
 
-      {
-        :results1 => {
-          :times => times1,
-          :mean => r.mean( times1 ),
-          :stddev => r.sd( times1 ),
-        },
-        :results2 => {
-          :times => times2,
-          :mean => r.mean( times2 ),
-          :stddev => r.sd( times2 ),
-        },
-        :p => wilcox_result[ 'p.value' ],
-        :W => wilcox_result[ 'statistic' ][ 'W' ],
-        :significant => (
-          wilcox_result[ 'p.value' ] < @options[ :required_significance ]
-        ),
-      }
-    end
-    alias to with
+    {
+      :results1 => {
+        :times => times1,
+        :mean => r.mean( times1 ),
+        :stddev => r.sd( times1 ),
+      },
+      :results2 => {
+        :times => times2,
+        :mean => r.mean( times2 ),
+        :stddev => r.sd( times2 ),
+      },
+      :p => wilcox_result[ 'p.value' ],
+      :W => wilcox_result[ 'statistic' ][ 'W' ],
+      :significant => (
+        wilcox_result[ 'p.value' ] < ( required_significance || DEFAULT_REQUIRED_SIGNIFICANCE )
+      ),
+    }
   end
 
   # Options:
@@ -85,7 +75,6 @@ module Benchmark
   def self.compare_realtime( options = {}, &block1 )
     options[ :iterations ] ||= 20
     options[ :inner_iterations ] ||= 1
-    options[ :required_significance ] ||= 0.01
 
     if options[ :iterations ] > 30
       warn "The number of iterations is set to #{options[ :iterations ]}.  " +

data/lib/better-benchmark/bencher.rb

@@ -0,0 +1,95 @@
+module Benchmark
+  class Bencher
+    DATA_FILE = 'bbench-run-time'
+
+    def print_usage
+      puts "#{$0} [-i <iterations>] [-w] [-r <revision 1> -r <revision 2>] [-p <max p-value>] [-d <data tmp dir>] [-e <executable/interpreter>] -- <executable's args...>"
+    end
+
+    # @param [Array] argv
+    #   The command line arguments passed to the bencher script
+    def initialize( argv )
+      @iterations = 10
+      @executable = 'ruby'
+
+      while argv.any?
+        arg = argv.shift
+        case arg
+        when '-d'
+          @data_dir = argv.shift
+          begin
+            if ! File.stat( @data_dir ).directory?
+              $stderr.puts "#{@data_dir} is not a directory."
+              exit 3
+            end
+          rescue Errno::ENOENT
+            $stderr.puts "#{@data_dir} does not exist."
+            exit 4
+          end
+        when '-e'
+          @executable = argv.shift
+        when '-i'
+          @iterations = argv.shift.to_i
+        when '-p'
+          @max_p = argv.shift
+        when '-r'
+          if @r1.nil?
+            @r1 = argv.shift
+          else
+            @r2 = argv.shift
+          end
+        when '-w'
+          @test_working_copy = true
+        when '--'
+          @executable_args = argv.dup
+          argv.clear
+        end
+      end
+
+      if ( ! @test_working_copy && ( @r1.nil? || @r2.nil? ) ) || @executable_args.nil?
+        print_usage
+        exit 2
+      end
+    end
+
+    def one_run
+      system "#{@executable} #{ @executable_args.join(' ') }"  or exit $?
+    end
+
+    def time_one_run
+      if @data_dir
+        one_run
+        File.read( "#{@data_dir}/#{DATA_FILE}" ).to_f
+      else
+        t0 = Time.now
+        one_run
+        Time.now.to_f - t0.to_f
+      end
+    end
+
+    def run
+      times1 = []
+      times2 = []
+
+      @iterations.times do
+        if @test_working_copy
+          system "git stash -q"  or exit $?
+        else
+          system "git checkout #{@r1}"  or exit $?
+        end
+        times1 << time_one_run
+
+        if @test_working_copy
+          system "git stash pop -q"  or exit $?
+        else
+          system "git checkout #{@r2}"  or exit $?
+        end
+        times2 << time_one_run
+      end
+
+      ::Benchmark.report_on(
+        ::Benchmark.compare_times( times1, times2, @max_p )
+      )
+    end
+  end
+end

data/lib/better-benchmark/comparison-partial.rb

@@ -0,0 +1,33 @@
+module Benchmark
+  class ComparisonPartial
+    def initialize( block, options )
+      @block1 = block
+      @options = options
+    end
+
+    def with( &block2 )
+      times1 = []
+      times2 = []
+
+      (1..@options[ :iterations ]).each do |iteration|
+        if @options[ :verbose ]
+          $stdout.print "."; $stdout.flush
+        end
+
+        times1 << Benchmark.realtime do
+          @options[ :inner_iterations ].times do |i|
+            @block1.call( iteration )
+          end
+        end
+        times2 << Benchmark.realtime do
+          @options[ :inner_iterations ].times do |i|
+            block2.call( iteration )
+          end
+        end
+      end
+
+      ::Benchmark.compare_times( times1, times2, @options[ :required_significance ] )
+    end
+    alias to with
+  end
+end

metadata

@@ -1,7 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: better-benchmark
 version: !ruby/object:Gem::Version 
-  version: 0.7.0
+  hash: 61
+  prerelease: false
+  segments: 
+  - 0
+  - 8
+  - 1
+  version: 0.8.1
 platform: ruby
 authors: 
 - Pistos
@@ -9,50 +15,74 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-02-11 00:00:00 -05:00
+date: 2010-09-10 00:00:00 -04:00
 default_executable: 
-dependencies: []
-
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rsruby
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :runtime
+  version_requirements: *id001
 description: Statistically correct benchmarking for Ruby.
-email: pistos at purepistos dot net
-executables: []
-
+email: betterbenchmark dot pistos at purepistos dot net
+executables: 
+- bbench
 extensions: []
 
 extra_rdoc_files: 
-- README
+- README.md
 - LICENCE
 files: 
-- README
+- README.md
 - LICENCE
 - example.rb
 - run-example
 - lib/better-benchmark.rb
-has_rdoc: false
-homepage: http://github.com/Pistos/better-benchmark/tree
+- lib/better-benchmark/bencher.rb
+- lib/better-benchmark/comparison-partial.rb
+- bin/bbench
+has_rdoc: true
+homepage: http://github.com/Pistos/better-benchmark
+licenses: []
+
 post_install_message: 
 rdoc_options: []
 
 require_paths: 
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
       version: "0"
-  version: 
 required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
       version: "0"
-  version: 
 requirements: 
 - "The R project: http://www.r-project.org/"
 rubyforge_project: better-benchmark
-rubygems_version: 1.3.1
+rubygems_version: 1.3.7
 signing_key: 
-specification_version: 2
+specification_version: 3
 summary: Statistically correct benchmarking for Ruby.
 test_files: []
 

data/README

@@ -1,34 +0,0 @@
-## Dependencies
-
-The R Project: http://www.r-project.org/
-rsruby: http://web.kuicr.kyoto-u.ac.jp/~alexg/rsruby/
-
-## Usage
-
-result = Benchmark.compare_realtime {
-  do_something_one_way
-}.with {
-  do_it_another_way
-}
-Benchmark.report_on result
-
-See also example.rb for a more comprehensive example.
-
-## Example Output
-
-....................
-Set 1 mean: 0.484 s
-Set 1 std dev: 0.098
-Set 2 mean: 0.469 s
-Set 2 std dev: 0.088
-p.value: 0.601661885634415
-W: 220.0
-The difference (-3.2%) IS NOT statistically significant.
-
-## Help, etc.
-
-irc.freenode.net#mathetes or http://mibbit.com/?server=irc.freenode.net&channel=%23mathetes
-
-## Repository
-
-git clone git://github.com/Pistos/better-benchmark.git