better-benchmark 0.7.0 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md ADDED
@@ -0,0 +1,115 @@
1
+ # Better Benchmark
2
+
3
+ Statistically correct benchmarking for Ruby.
4
+
5
+ ## Dependencies
6
+
7
+ * [The R Project](http://www.r-project.org/)
8
+ * [rsruby](http://github.com/alexgutteridge/rsruby)
9
+
10
+ ## Usage
11
+
12
+ ### Comparing code blocks
13
+
14
+ result = Benchmark.compare_realtime {
15
+ do_something_one_way
16
+ }.with {
17
+ do_it_another_way
18
+ }
19
+ Benchmark.report_on result
20
+
21
+ See also example.rb for a more comprehensive example.
22
+
23
+ ### Comparing git revisions
24
+
25
+ #### With a test script (recommended)
26
+
27
+ To test two revisions of a library, create a simple runner script:
28
+
29
+ # runner.rb
30
+ require 'mylib'
31
+
32
+ class TestQuick
33
+ def initialize
34
+ # initialization...
35
+ end
36
+
37
+ def run
38
+ Benchmark.write_realtime( '/home/pistos/tmp' ) do
39
+ 5000.times do
40
+ # do something with your lib
41
+ end
42
+ end
43
+ end
44
+ end
45
+
46
+ t = TestQuick.new
47
+ t.run
48
+
49
+ Then run the bbench script, passing two git revisions:
50
+
51
+ bbench -r 6e84dd5 -r ed1e7c6 -d ~/tmp -- -Ilib runner.rb
52
+
53
+ #### Without altering or writing new code
54
+
55
+ You can also test two revisions by running some already-existing script,
56
+ such as a file in your test suite:
57
+
58
+ bbench -r 6e84dd5 -r ed1e7c6 -- -Itest -Ilib test/test_something.rb
59
+
60
+ Be aware, however, that this may produce unnecessarily variant timings due to
61
+ wide variance in the startup time of the Ruby interpreter and script.
62
+
63
+ ### Comparing git working copy
64
+
65
+ You can also compare the current branch tip to the current (dirty) working copy:
66
+
67
+ bbench -w -d ~/tmp -- -Ilib runner.rb
68
+
69
+ This lets you experiment without committing anything, and then only commit
70
+ when you are confident that your changes result in a performance improvement.
71
+
72
+ ## Interpretation
73
+
74
+ Considering two "things under test", U1 and U2:
75
+
76
+ ### Example 1
77
+
78
+ Set 1 mean: 0.216 s
79
+ Set 1 std dev: 0.023
80
+ Set 2 mean: 0.187 s
81
+ Set 2 std dev: 0.020
82
+ p.value: 0.00287947346770876
83
+ W: 88.0
84
+ The difference (-13.5%) IS statistically significant.
85
+
86
+ This means that the results permit us to conclude that U2 performed 13.5%
87
+ faster than U1.
88
+
89
+ ### Example 2
90
+
91
+ Set 1 mean: 10.968 s
92
+ Set 1 std dev: 4.294
93
+ Set 2 mean: 9.036 s
94
+ Set 2 std dev: 3.581
95
+ p.value: 0.217562623135379
96
+ W: 67.0
97
+ The difference (-17.6%) IS NOT statistically significant.
98
+
99
+ This means that the results do not permit us to conclude that the performance
100
+ of U1 and U2 differed.
101
+
102
+ ## Not just Ruby
103
+
104
+ Technically, the bbench script can work with any script or program that writes
105
+ a run time (in seconds) to the file bbench-run-time in the data dir. Use the
106
+ -e option to specify a different executable than "ruby". e.g. perl, python,
107
+ java, etc.
108
+
109
+ ## Help, etc.
110
+
111
+ irc.freenode.net#mathetes or http://webchat.freenode.net?channels=mathetes .
112
+
113
+ ## Repository
114
+
115
+ git clone git://github.com/Pistos/better-benchmark.git
data/bin/bbench ADDED
@@ -0,0 +1,6 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'better-benchmark'
4
+
5
+ b = ::Benchmark::Bencher.new( ARGV.dup )
6
+ b.run
data/example.rb CHANGED
@@ -1,5 +1,6 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
+ require 'rubygems'
3
4
  require 'better-benchmark'
4
5
 
5
6
  # Provide two blocks of code to compare. For example, two blocks that
@@ -1,59 +1,49 @@
1
1
  require 'benchmark'
2
2
  require 'rsruby'
3
3
 
4
+ require 'better-benchmark/comparison-partial'
5
+ require 'better-benchmark/bencher'
6
+
4
7
  module Benchmark
5
8
 
6
- BETTER_BENCHMARK_VERSION = '0.7.0'
9
+ BETTER_BENCHMARK_VERSION = '0.8.0'
10
+ DEFAULT_REQUIRED_SIGNIFICANCE = 0.01
7
11
 
8
- class ComparisonPartial
9
- def initialize( block, options )
10
- @block1 = block
11
- @options = options
12
+ def self.write_realtime( data_dir, &block )
13
+ t = Benchmark.realtime( &block )
14
+ File.open( "#{data_dir}/#{Bencher::DATA_FILE}", 'w' ) do |f|
15
+ f.print t
12
16
  end
17
+ end
13
18
 
14
- def with( &block2 )
15
- times1 = []
16
- times2 = []
17
-
18
- (1..@options[ :iterations ]).each do |iteration|
19
- if @options[ :verbose ]
20
- $stdout.print "."; $stdout.flush
21
- end
22
-
23
- times1 << Benchmark.realtime do
24
- @options[ :inner_iterations ].times do |i|
25
- @block1.call( iteration )
26
- end
27
- end
28
- times2 << Benchmark.realtime do
29
- @options[ :inner_iterations ].times do |i|
30
- block2.call( iteration )
31
- end
32
- end
33
- end
34
-
35
- r = RSRuby.instance
36
- wilcox_result = r.wilcox_test( times1, times2 )
19
+ # The number of elements in times1 and times2 should be the same.
20
+ # @param [Array] times1
21
+ # An Array of elapsed times in float form, measured in seconds
22
+ # @param [Array] times2
23
+ # An Array of elapsed times in float form, measured in seconds
24
+ # @param [Fixnum] required_significance
25
+ # The maximum p value needed to declare statistical significance
26
+ def self.compare_times( times1, times2, required_significance = DEFAULT_REQUIRED_SIGNIFICANCE )
27
+ r = RSRuby.instance
28
+ wilcox_result = r.wilcox_test( times1, times2 )
37
29
 
38
- {
39
- :results1 => {
40
- :times => times1,
41
- :mean => r.mean( times1 ),
42
- :stddev => r.sd( times1 ),
43
- },
44
- :results2 => {
45
- :times => times2,
46
- :mean => r.mean( times2 ),
47
- :stddev => r.sd( times2 ),
48
- },
49
- :p => wilcox_result[ 'p.value' ],
50
- :W => wilcox_result[ 'statistic' ][ 'W' ],
51
- :significant => (
52
- wilcox_result[ 'p.value' ] < @options[ :required_significance ]
53
- ),
54
- }
55
- end
56
- alias to with
30
+ {
31
+ :results1 => {
32
+ :times => times1,
33
+ :mean => r.mean( times1 ),
34
+ :stddev => r.sd( times1 ),
35
+ },
36
+ :results2 => {
37
+ :times => times2,
38
+ :mean => r.mean( times2 ),
39
+ :stddev => r.sd( times2 ),
40
+ },
41
+ :p => wilcox_result[ 'p.value' ],
42
+ :W => wilcox_result[ 'statistic' ][ 'W' ],
43
+ :significant => (
44
+ wilcox_result[ 'p.value' ] < ( required_significance || DEFAULT_REQUIRED_SIGNIFICANCE )
45
+ ),
46
+ }
57
47
  end
58
48
 
59
49
  # Options:
@@ -85,7 +75,6 @@ module Benchmark
85
75
  def self.compare_realtime( options = {}, &block1 )
86
76
  options[ :iterations ] ||= 20
87
77
  options[ :inner_iterations ] ||= 1
88
- options[ :required_significance ] ||= 0.01
89
78
 
90
79
  if options[ :iterations ] > 30
91
80
  warn "The number of iterations is set to #{options[ :iterations ]}. " +
@@ -0,0 +1,95 @@
1
+ module Benchmark
2
+ class Bencher
3
+ DATA_FILE = 'bbench-run-time'
4
+
5
+ def print_usage
6
+ puts "#{$0} [-i <iterations>] [-w] [-r <revision 1> -r <revision 2>] [-p <max p-value>] [-d <data tmp dir>] [-e <executable/interpreter>] -- <executable's args...>"
7
+ end
8
+
9
+ # @param [Array] argv
10
+ # The command line arguments passed to the bencher script
11
+ def initialize( argv )
12
+ @iterations = 10
13
+ @executable = 'ruby'
14
+
15
+ while argv.any?
16
+ arg = argv.shift
17
+ case arg
18
+ when '-d'
19
+ @data_dir = argv.shift
20
+ begin
21
+ if ! File.stat( @data_dir ).directory?
22
+ $stderr.puts "#{@data_dir} is not a directory."
23
+ exit 3
24
+ end
25
+ rescue Errno::ENOENT
26
+ $stderr.puts "#{@data_dir} does not exist."
27
+ exit 4
28
+ end
29
+ when '-e'
30
+ @executable = argv.shift
31
+ when '-i'
32
+ @iterations = argv.shift.to_i
33
+ when '-p'
34
+ @max_p = argv.shift
35
+ when '-r'
36
+ if @r1.nil?
37
+ @r1 = argv.shift
38
+ else
39
+ @r2 = argv.shift
40
+ end
41
+ when '-w'
42
+ @test_working_copy = true
43
+ when '--'
44
+ @executable_args = argv.dup
45
+ argv.clear
46
+ end
47
+ end
48
+
49
+ if ( ! @test_working_copy && ( @r1.nil? || @r2.nil? ) ) || @executable_args.nil?
50
+ print_usage
51
+ exit 2
52
+ end
53
+ end
54
+
55
+ def one_run
56
+ system "#{@executable} #{ @executable_args.join(' ') }" or exit $?
57
+ end
58
+
59
+ def time_one_run
60
+ if @data_dir
61
+ one_run
62
+ File.read( "#{@data_dir}/#{DATA_FILE}" ).to_f
63
+ else
64
+ t0 = Time.now
65
+ one_run
66
+ Time.now.to_f - t0.to_f
67
+ end
68
+ end
69
+
70
+ def run
71
+ times1 = []
72
+ times2 = []
73
+
74
+ @iterations.times do
75
+ if @test_working_copy
76
+ system "git stash -q" or exit $?
77
+ else
78
+ system "git checkout #{@r1}" or exit $?
79
+ end
80
+ times1 << time_one_run
81
+
82
+ if @test_working_copy
83
+ system "git stash pop -q" or exit $?
84
+ else
85
+ system "git checkout #{@r2}" or exit $?
86
+ end
87
+ times2 << time_one_run
88
+ end
89
+
90
+ ::Benchmark.report_on(
91
+ ::Benchmark.compare_times( times1, times2, @max_p )
92
+ )
93
+ end
94
+ end
95
+ end
@@ -0,0 +1,33 @@
1
+ module Benchmark
2
+ class ComparisonPartial
3
+ def initialize( block, options )
4
+ @block1 = block
5
+ @options = options
6
+ end
7
+
8
+ def with( &block2 )
9
+ times1 = []
10
+ times2 = []
11
+
12
+ (1..@options[ :iterations ]).each do |iteration|
13
+ if @options[ :verbose ]
14
+ $stdout.print "."; $stdout.flush
15
+ end
16
+
17
+ times1 << Benchmark.realtime do
18
+ @options[ :inner_iterations ].times do |i|
19
+ @block1.call( iteration )
20
+ end
21
+ end
22
+ times2 << Benchmark.realtime do
23
+ @options[ :inner_iterations ].times do |i|
24
+ block2.call( iteration )
25
+ end
26
+ end
27
+ end
28
+
29
+ ::Benchmark.compare_times( times1, times2, @options[ :required_significance ] )
30
+ end
31
+ alias to with
32
+ end
33
+ end
metadata CHANGED
@@ -1,7 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: better-benchmark
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.7.0
4
+ hash: 61
5
+ prerelease: false
6
+ segments:
7
+ - 0
8
+ - 8
9
+ - 1
10
+ version: 0.8.1
5
11
  platform: ruby
6
12
  authors:
7
13
  - Pistos
@@ -9,50 +15,74 @@ autorequire:
9
15
  bindir: bin
10
16
  cert_chain: []
11
17
 
12
- date: 2009-02-11 00:00:00 -05:00
18
+ date: 2010-09-10 00:00:00 -04:00
13
19
  default_executable:
14
- dependencies: []
15
-
20
+ dependencies:
21
+ - !ruby/object:Gem::Dependency
22
+ name: rsruby
23
+ prerelease: false
24
+ requirement: &id001 !ruby/object:Gem::Requirement
25
+ none: false
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ hash: 3
30
+ segments:
31
+ - 0
32
+ version: "0"
33
+ type: :runtime
34
+ version_requirements: *id001
16
35
  description: Statistically correct benchmarking for Ruby.
17
- email: pistos at purepistos dot net
18
- executables: []
19
-
36
+ email: betterbenchmark dot pistos at purepistos dot net
37
+ executables:
38
+ - bbench
20
39
  extensions: []
21
40
 
22
41
  extra_rdoc_files:
23
- - README
42
+ - README.md
24
43
  - LICENCE
25
44
  files:
26
- - README
45
+ - README.md
27
46
  - LICENCE
28
47
  - example.rb
29
48
  - run-example
30
49
  - lib/better-benchmark.rb
31
- has_rdoc: false
32
- homepage: http://github.com/Pistos/better-benchmark/tree
50
+ - lib/better-benchmark/bencher.rb
51
+ - lib/better-benchmark/comparison-partial.rb
52
+ - bin/bbench
53
+ has_rdoc: true
54
+ homepage: http://github.com/Pistos/better-benchmark
55
+ licenses: []
56
+
33
57
  post_install_message:
34
58
  rdoc_options: []
35
59
 
36
60
  require_paths:
37
61
  - lib
38
62
  required_ruby_version: !ruby/object:Gem::Requirement
63
+ none: false
39
64
  requirements:
40
65
  - - ">="
41
66
  - !ruby/object:Gem::Version
67
+ hash: 3
68
+ segments:
69
+ - 0
42
70
  version: "0"
43
- version:
44
71
  required_rubygems_version: !ruby/object:Gem::Requirement
72
+ none: false
45
73
  requirements:
46
74
  - - ">="
47
75
  - !ruby/object:Gem::Version
76
+ hash: 3
77
+ segments:
78
+ - 0
48
79
  version: "0"
49
- version:
50
80
  requirements:
51
81
  - "The R project: http://www.r-project.org/"
52
82
  rubyforge_project: better-benchmark
53
- rubygems_version: 1.3.1
83
+ rubygems_version: 1.3.7
54
84
  signing_key:
55
- specification_version: 2
85
+ specification_version: 3
56
86
  summary: Statistically correct benchmarking for Ruby.
57
87
  test_files: []
58
88
 
data/README DELETED
@@ -1,34 +0,0 @@
1
- ## Dependencies
2
-
3
- The R Project: http://www.r-project.org/
4
- rsruby: http://web.kuicr.kyoto-u.ac.jp/~alexg/rsruby/
5
-
6
- ## Usage
7
-
8
- result = Benchmark.compare_realtime {
9
- do_something_one_way
10
- }.with {
11
- do_it_another_way
12
- }
13
- Benchmark.report_on result
14
-
15
- See also example.rb for a more comprehensive example.
16
-
17
- ## Example Output
18
-
19
- ....................
20
- Set 1 mean: 0.484 s
21
- Set 1 std dev: 0.098
22
- Set 2 mean: 0.469 s
23
- Set 2 std dev: 0.088
24
- p.value: 0.601661885634415
25
- W: 220.0
26
- The difference (-3.2%) IS NOT statistically significant.
27
-
28
- ## Help, etc.
29
-
30
- irc.freenode.net#mathetes or http://mibbit.com/?server=irc.freenode.net&channel=%23mathetes
31
-
32
- ## Repository
33
-
34
- git clone git://github.com/Pistos/better-benchmark.git