better-benchmark 0.7.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +115 -0
- data/bin/bbench +6 -0
- data/example.rb +1 -0
- data/lib/better-benchmark.rb +37 -48
- data/lib/better-benchmark/bencher.rb +95 -0
- data/lib/better-benchmark/comparison-partial.rb +33 -0
- metadata +45 -15
- data/README +0 -34
data/README.md
ADDED
@@ -0,0 +1,115 @@
|
|
1
|
+
# Better Benchmark
|
2
|
+
|
3
|
+
Statistically correct benchmarking for Ruby.
|
4
|
+
|
5
|
+
## Dependencies
|
6
|
+
|
7
|
+
* [The R Project](http://www.r-project.org/)
|
8
|
+
* [rsruby](http://github.com/alexgutteridge/rsruby)
|
9
|
+
|
10
|
+
## Usage
|
11
|
+
|
12
|
+
### Comparing code blocks
|
13
|
+
|
14
|
+
result = Benchmark.compare_realtime {
|
15
|
+
do_something_one_way
|
16
|
+
}.with {
|
17
|
+
do_it_another_way
|
18
|
+
}
|
19
|
+
Benchmark.report_on result
|
20
|
+
|
21
|
+
See also example.rb for a more comprehensive example.
|
22
|
+
|
23
|
+
### Comparing git revisions
|
24
|
+
|
25
|
+
#### With a test script (recommended)
|
26
|
+
|
27
|
+
To test two revisions of a library, create a simple runner script:
|
28
|
+
|
29
|
+
# runner.rb
|
30
|
+
require 'mylib'
|
31
|
+
|
32
|
+
class TestQuick
|
33
|
+
def initialize
|
34
|
+
# initialization...
|
35
|
+
end
|
36
|
+
|
37
|
+
def run
|
38
|
+
Benchmark.write_realtime( '/home/pistos/tmp' ) do
|
39
|
+
5000.times do
|
40
|
+
# do something with your lib
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
t = TestQuick.new
|
47
|
+
t.run
|
48
|
+
|
49
|
+
Then run the bbench script, passing two git revisions:
|
50
|
+
|
51
|
+
bbench -r 6e84dd5 -r ed1e7c6 -d ~/tmp -- -Ilib runner.rb
|
52
|
+
|
53
|
+
#### Without altering or writing new code
|
54
|
+
|
55
|
+
You can also test two revisions by running some already-existing script,
|
56
|
+
such as a file in your test suite:
|
57
|
+
|
58
|
+
bbench -r 6e84dd5 -r ed1e7c6 -- -Itest -Ilib test/test_something.rb
|
59
|
+
|
60
|
+
Be aware, however, that this may produce unnecessarily variant timings due to
|
61
|
+
wide variance in the startup time of the Ruby interpreter and script.
|
62
|
+
|
63
|
+
### Comparing git working copy
|
64
|
+
|
65
|
+
You can also compare the current branch tip to the current (dirty) working copy:
|
66
|
+
|
67
|
+
bbench -w -d ~/tmp -- -Ilib runner.rb
|
68
|
+
|
69
|
+
This lets you experiment without committing anything, and then only commit
|
70
|
+
when you are confident that your changes result in a performance improvement.
|
71
|
+
|
72
|
+
## Interpretation
|
73
|
+
|
74
|
+
Considering two "things under test", U1 and U2:
|
75
|
+
|
76
|
+
### Example 1
|
77
|
+
|
78
|
+
Set 1 mean: 0.216 s
|
79
|
+
Set 1 std dev: 0.023
|
80
|
+
Set 2 mean: 0.187 s
|
81
|
+
Set 2 std dev: 0.020
|
82
|
+
p.value: 0.00287947346770876
|
83
|
+
W: 88.0
|
84
|
+
The difference (-13.5%) IS statistically significant.
|
85
|
+
|
86
|
+
This means that the results permit us to conclude that U2 performed 13.5%
|
87
|
+
faster than U1.
|
88
|
+
|
89
|
+
### Example 2
|
90
|
+
|
91
|
+
Set 1 mean: 10.968 s
|
92
|
+
Set 1 std dev: 4.294
|
93
|
+
Set 2 mean: 9.036 s
|
94
|
+
Set 2 std dev: 3.581
|
95
|
+
p.value: 0.217562623135379
|
96
|
+
W: 67.0
|
97
|
+
The difference (-17.6%) IS NOT statistically significant.
|
98
|
+
|
99
|
+
This means that the results do not permit us to conclude that the performance
|
100
|
+
of U1 and U2 differed.
|
101
|
+
|
102
|
+
## Not just Ruby
|
103
|
+
|
104
|
+
Technically, the bbench script can work with any script or program that writes
|
105
|
+
a run time (in seconds) to the file bbench-run-time in the data dir. Use the
|
106
|
+
-e option to specify a different executable than "ruby". e.g. perl, python,
|
107
|
+
java, etc.
|
108
|
+
|
109
|
+
## Help, etc.
|
110
|
+
|
111
|
+
irc.freenode.net#mathetes or http://webchat.freenode.net?channels=mathetes .
|
112
|
+
|
113
|
+
## Repository
|
114
|
+
|
115
|
+
git clone git://github.com/Pistos/better-benchmark.git
|
data/bin/bbench
ADDED
data/example.rb
CHANGED
data/lib/better-benchmark.rb
CHANGED
@@ -1,59 +1,49 @@
|
|
1
1
|
require 'benchmark'
|
2
2
|
require 'rsruby'
|
3
3
|
|
4
|
+
require 'better-benchmark/comparison-partial'
|
5
|
+
require 'better-benchmark/bencher'
|
6
|
+
|
4
7
|
module Benchmark
|
5
8
|
|
6
|
-
BETTER_BENCHMARK_VERSION = '0.
|
9
|
+
BETTER_BENCHMARK_VERSION = '0.8.0'
|
10
|
+
DEFAULT_REQUIRED_SIGNIFICANCE = 0.01
|
7
11
|
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
+
def self.write_realtime( data_dir, &block )
|
13
|
+
t = Benchmark.realtime( &block )
|
14
|
+
File.open( "#{data_dir}/#{Bencher::DATA_FILE}", 'w' ) do |f|
|
15
|
+
f.print t
|
12
16
|
end
|
17
|
+
end
|
13
18
|
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
@options[ :inner_iterations ].times do |i|
|
25
|
-
@block1.call( iteration )
|
26
|
-
end
|
27
|
-
end
|
28
|
-
times2 << Benchmark.realtime do
|
29
|
-
@options[ :inner_iterations ].times do |i|
|
30
|
-
block2.call( iteration )
|
31
|
-
end
|
32
|
-
end
|
33
|
-
end
|
34
|
-
|
35
|
-
r = RSRuby.instance
|
36
|
-
wilcox_result = r.wilcox_test( times1, times2 )
|
19
|
+
# The number of elements in times1 and times2 should be the same.
|
20
|
+
# @param [Array] times1
|
21
|
+
# An Array of elapsed times in float form, measured in seconds
|
22
|
+
# @param [Array] times2
|
23
|
+
# An Array of elapsed times in float form, measured in seconds
|
24
|
+
# @param [Fixnum] required_significance
|
25
|
+
# The maximum p value needed to declare statistical significance
|
26
|
+
def self.compare_times( times1, times2, required_significance = DEFAULT_REQUIRED_SIGNIFICANCE )
|
27
|
+
r = RSRuby.instance
|
28
|
+
wilcox_result = r.wilcox_test( times1, times2 )
|
37
29
|
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
end
|
56
|
-
alias to with
|
30
|
+
{
|
31
|
+
:results1 => {
|
32
|
+
:times => times1,
|
33
|
+
:mean => r.mean( times1 ),
|
34
|
+
:stddev => r.sd( times1 ),
|
35
|
+
},
|
36
|
+
:results2 => {
|
37
|
+
:times => times2,
|
38
|
+
:mean => r.mean( times2 ),
|
39
|
+
:stddev => r.sd( times2 ),
|
40
|
+
},
|
41
|
+
:p => wilcox_result[ 'p.value' ],
|
42
|
+
:W => wilcox_result[ 'statistic' ][ 'W' ],
|
43
|
+
:significant => (
|
44
|
+
wilcox_result[ 'p.value' ] < ( required_significance || DEFAULT_REQUIRED_SIGNIFICANCE )
|
45
|
+
),
|
46
|
+
}
|
57
47
|
end
|
58
48
|
|
59
49
|
# Options:
|
@@ -85,7 +75,6 @@ module Benchmark
|
|
85
75
|
def self.compare_realtime( options = {}, &block1 )
|
86
76
|
options[ :iterations ] ||= 20
|
87
77
|
options[ :inner_iterations ] ||= 1
|
88
|
-
options[ :required_significance ] ||= 0.01
|
89
78
|
|
90
79
|
if options[ :iterations ] > 30
|
91
80
|
warn "The number of iterations is set to #{options[ :iterations ]}. " +
|
@@ -0,0 +1,95 @@
|
|
1
|
+
module Benchmark
|
2
|
+
class Bencher
|
3
|
+
DATA_FILE = 'bbench-run-time'
|
4
|
+
|
5
|
+
def print_usage
|
6
|
+
puts "#{$0} [-i <iterations>] [-w] [-r <revision 1> -r <revision 2>] [-p <max p-value>] [-d <data tmp dir>] [-e <executable/interpreter>] -- <executable's args...>"
|
7
|
+
end
|
8
|
+
|
9
|
+
# @param [Array] argv
|
10
|
+
# The command line arguments passed to the bencher script
|
11
|
+
def initialize( argv )
|
12
|
+
@iterations = 10
|
13
|
+
@executable = 'ruby'
|
14
|
+
|
15
|
+
while argv.any?
|
16
|
+
arg = argv.shift
|
17
|
+
case arg
|
18
|
+
when '-d'
|
19
|
+
@data_dir = argv.shift
|
20
|
+
begin
|
21
|
+
if ! File.stat( @data_dir ).directory?
|
22
|
+
$stderr.puts "#{@data_dir} is not a directory."
|
23
|
+
exit 3
|
24
|
+
end
|
25
|
+
rescue Errno::ENOENT
|
26
|
+
$stderr.puts "#{@data_dir} does not exist."
|
27
|
+
exit 4
|
28
|
+
end
|
29
|
+
when '-e'
|
30
|
+
@executable = argv.shift
|
31
|
+
when '-i'
|
32
|
+
@iterations = argv.shift.to_i
|
33
|
+
when '-p'
|
34
|
+
@max_p = argv.shift
|
35
|
+
when '-r'
|
36
|
+
if @r1.nil?
|
37
|
+
@r1 = argv.shift
|
38
|
+
else
|
39
|
+
@r2 = argv.shift
|
40
|
+
end
|
41
|
+
when '-w'
|
42
|
+
@test_working_copy = true
|
43
|
+
when '--'
|
44
|
+
@executable_args = argv.dup
|
45
|
+
argv.clear
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
if ( ! @test_working_copy && ( @r1.nil? || @r2.nil? ) ) || @executable_args.nil?
|
50
|
+
print_usage
|
51
|
+
exit 2
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
def one_run
|
56
|
+
system "#{@executable} #{ @executable_args.join(' ') }" or exit $?
|
57
|
+
end
|
58
|
+
|
59
|
+
def time_one_run
|
60
|
+
if @data_dir
|
61
|
+
one_run
|
62
|
+
File.read( "#{@data_dir}/#{DATA_FILE}" ).to_f
|
63
|
+
else
|
64
|
+
t0 = Time.now
|
65
|
+
one_run
|
66
|
+
Time.now.to_f - t0.to_f
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
def run
|
71
|
+
times1 = []
|
72
|
+
times2 = []
|
73
|
+
|
74
|
+
@iterations.times do
|
75
|
+
if @test_working_copy
|
76
|
+
system "git stash -q" or exit $?
|
77
|
+
else
|
78
|
+
system "git checkout #{@r1}" or exit $?
|
79
|
+
end
|
80
|
+
times1 << time_one_run
|
81
|
+
|
82
|
+
if @test_working_copy
|
83
|
+
system "git stash pop -q" or exit $?
|
84
|
+
else
|
85
|
+
system "git checkout #{@r2}" or exit $?
|
86
|
+
end
|
87
|
+
times2 << time_one_run
|
88
|
+
end
|
89
|
+
|
90
|
+
::Benchmark.report_on(
|
91
|
+
::Benchmark.compare_times( times1, times2, @max_p )
|
92
|
+
)
|
93
|
+
end
|
94
|
+
end
|
95
|
+
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module Benchmark
|
2
|
+
class ComparisonPartial
|
3
|
+
def initialize( block, options )
|
4
|
+
@block1 = block
|
5
|
+
@options = options
|
6
|
+
end
|
7
|
+
|
8
|
+
def with( &block2 )
|
9
|
+
times1 = []
|
10
|
+
times2 = []
|
11
|
+
|
12
|
+
(1..@options[ :iterations ]).each do |iteration|
|
13
|
+
if @options[ :verbose ]
|
14
|
+
$stdout.print "."; $stdout.flush
|
15
|
+
end
|
16
|
+
|
17
|
+
times1 << Benchmark.realtime do
|
18
|
+
@options[ :inner_iterations ].times do |i|
|
19
|
+
@block1.call( iteration )
|
20
|
+
end
|
21
|
+
end
|
22
|
+
times2 << Benchmark.realtime do
|
23
|
+
@options[ :inner_iterations ].times do |i|
|
24
|
+
block2.call( iteration )
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
::Benchmark.compare_times( times1, times2, @options[ :required_significance ] )
|
30
|
+
end
|
31
|
+
alias to with
|
32
|
+
end
|
33
|
+
end
|
metadata
CHANGED
@@ -1,7 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: better-benchmark
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
|
4
|
+
hash: 61
|
5
|
+
prerelease: false
|
6
|
+
segments:
|
7
|
+
- 0
|
8
|
+
- 8
|
9
|
+
- 1
|
10
|
+
version: 0.8.1
|
5
11
|
platform: ruby
|
6
12
|
authors:
|
7
13
|
- Pistos
|
@@ -9,50 +15,74 @@ autorequire:
|
|
9
15
|
bindir: bin
|
10
16
|
cert_chain: []
|
11
17
|
|
12
|
-
date:
|
18
|
+
date: 2010-09-10 00:00:00 -04:00
|
13
19
|
default_executable:
|
14
|
-
dependencies:
|
15
|
-
|
20
|
+
dependencies:
|
21
|
+
- !ruby/object:Gem::Dependency
|
22
|
+
name: rsruby
|
23
|
+
prerelease: false
|
24
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ">="
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
hash: 3
|
30
|
+
segments:
|
31
|
+
- 0
|
32
|
+
version: "0"
|
33
|
+
type: :runtime
|
34
|
+
version_requirements: *id001
|
16
35
|
description: Statistically correct benchmarking for Ruby.
|
17
|
-
email: pistos at purepistos dot net
|
18
|
-
executables:
|
19
|
-
|
36
|
+
email: betterbenchmark dot pistos at purepistos dot net
|
37
|
+
executables:
|
38
|
+
- bbench
|
20
39
|
extensions: []
|
21
40
|
|
22
41
|
extra_rdoc_files:
|
23
|
-
- README
|
42
|
+
- README.md
|
24
43
|
- LICENCE
|
25
44
|
files:
|
26
|
-
- README
|
45
|
+
- README.md
|
27
46
|
- LICENCE
|
28
47
|
- example.rb
|
29
48
|
- run-example
|
30
49
|
- lib/better-benchmark.rb
|
31
|
-
|
32
|
-
|
50
|
+
- lib/better-benchmark/bencher.rb
|
51
|
+
- lib/better-benchmark/comparison-partial.rb
|
52
|
+
- bin/bbench
|
53
|
+
has_rdoc: true
|
54
|
+
homepage: http://github.com/Pistos/better-benchmark
|
55
|
+
licenses: []
|
56
|
+
|
33
57
|
post_install_message:
|
34
58
|
rdoc_options: []
|
35
59
|
|
36
60
|
require_paths:
|
37
61
|
- lib
|
38
62
|
required_ruby_version: !ruby/object:Gem::Requirement
|
63
|
+
none: false
|
39
64
|
requirements:
|
40
65
|
- - ">="
|
41
66
|
- !ruby/object:Gem::Version
|
67
|
+
hash: 3
|
68
|
+
segments:
|
69
|
+
- 0
|
42
70
|
version: "0"
|
43
|
-
version:
|
44
71
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
72
|
+
none: false
|
45
73
|
requirements:
|
46
74
|
- - ">="
|
47
75
|
- !ruby/object:Gem::Version
|
76
|
+
hash: 3
|
77
|
+
segments:
|
78
|
+
- 0
|
48
79
|
version: "0"
|
49
|
-
version:
|
50
80
|
requirements:
|
51
81
|
- "The R project: http://www.r-project.org/"
|
52
82
|
rubyforge_project: better-benchmark
|
53
|
-
rubygems_version: 1.3.
|
83
|
+
rubygems_version: 1.3.7
|
54
84
|
signing_key:
|
55
|
-
specification_version:
|
85
|
+
specification_version: 3
|
56
86
|
summary: Statistically correct benchmarking for Ruby.
|
57
87
|
test_files: []
|
58
88
|
|
data/README
DELETED
@@ -1,34 +0,0 @@
|
|
1
|
-
## Dependencies
|
2
|
-
|
3
|
-
The R Project: http://www.r-project.org/
|
4
|
-
rsruby: http://web.kuicr.kyoto-u.ac.jp/~alexg/rsruby/
|
5
|
-
|
6
|
-
## Usage
|
7
|
-
|
8
|
-
result = Benchmark.compare_realtime {
|
9
|
-
do_something_one_way
|
10
|
-
}.with {
|
11
|
-
do_it_another_way
|
12
|
-
}
|
13
|
-
Benchmark.report_on result
|
14
|
-
|
15
|
-
See also example.rb for a more comprehensive example.
|
16
|
-
|
17
|
-
## Example Output
|
18
|
-
|
19
|
-
....................
|
20
|
-
Set 1 mean: 0.484 s
|
21
|
-
Set 1 std dev: 0.098
|
22
|
-
Set 2 mean: 0.469 s
|
23
|
-
Set 2 std dev: 0.088
|
24
|
-
p.value: 0.601661885634415
|
25
|
-
W: 220.0
|
26
|
-
The difference (-3.2%) IS NOT statistically significant.
|
27
|
-
|
28
|
-
## Help, etc.
|
29
|
-
|
30
|
-
irc.freenode.net#mathetes or http://mibbit.com/?server=irc.freenode.net&channel=%23mathetes
|
31
|
-
|
32
|
-
## Repository
|
33
|
-
|
34
|
-
git clone git://github.com/Pistos/better-benchmark.git
|