railsbench 0.9.2 → 0.9.8
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG +1808 -451
- data/GCPATCH +73 -0
- data/INSTALL +5 -0
- data/Manifest.txt +23 -13
- data/PROBLEMS +0 -0
- data/README +23 -7
- data/Rakefile +1 -2
- data/bin/railsbench +7 -1
- data/config/benchmarking.rb +0 -0
- data/config/benchmarks.rb +3 -2
- data/config/benchmarks.yml +0 -0
- data/images/empty.png +0 -0
- data/images/minus.png +0 -0
- data/images/plus.png +0 -0
- data/install.rb +1 -1
- data/latest_changes.txt +18 -0
- data/lib/benchmark.rb +0 -0
- data/lib/railsbench/benchmark.rb +576 -0
- data/lib/railsbench/benchmark_specs.rb +63 -63
- data/lib/railsbench/gc_info.rb +38 -3
- data/lib/railsbench/perf_info.rb +1 -1
- data/lib/railsbench/perf_utils.rb +202 -179
- data/lib/railsbench/railsbenchmark.rb +213 -55
- data/lib/railsbench/version.rb +9 -9
- data/lib/railsbench/write_headers_only.rb +15 -15
- data/postinstall.rb +0 -0
- data/ruby185gc.patch +56 -29
- data/ruby186gc.patch +564 -0
- data/ruby19gc.patch +2425 -0
- data/script/convert_raw_data_files +49 -49
- data/script/generate_benchmarks +14 -4
- data/script/perf_bench +12 -8
- data/script/perf_comp +1 -1
- data/script/perf_comp_gc +9 -1
- data/script/perf_diff +2 -2
- data/script/perf_diff_gc +2 -2
- data/script/perf_html +1 -1
- data/script/perf_plot +192 -75
- data/script/perf_plot_gc +213 -74
- data/script/perf_prof +29 -10
- data/script/perf_run +2 -2
- data/script/perf_run_gc +2 -2
- data/script/perf_table +2 -2
- data/script/perf_tex +1 -1
- data/script/perf_times +6 -6
- data/script/perf_times_gc +14 -2
- data/script/run_urls +16 -10
- data/setup.rb +0 -0
- data/test/railsbench_test.rb +0 -0
- data/test/test_helper.rb +2 -0
- metadata +77 -55
data/GCPATCH
ADDED
@@ -0,0 +1,73 @@
|
|
1
|
+
The garbage collector distributed with ruby tries to adapt memory
|
2
|
+
usage to the amount of memory used by the program, dynamically growing
|
3
|
+
or shrinking the allocated heap as it sees fit.
|
4
|
+
|
5
|
+
For long running server apps this doesn't really work very well. The
|
6
|
+
performance very much depends on the ratio heap size/program size. It
|
7
|
+
behaves somewhat erratic: adding code can actually make your program
|
8
|
+
run faster.
|
9
|
+
|
10
|
+
The supplied patch fixes this problem: it allows one to specify the
|
11
|
+
initial heap size to start with. Heap size will never drop below the
|
12
|
+
initial size. By carefully selecting the initial heap size one can
|
13
|
+
decrease startup time and increase throughput of server apps.
|
14
|
+
|
15
|
+
There are 4 patches for different versions of Ruby:
|
16
|
+
rubygc184.patch Ruby 1.8.4 (but will apply to 1.82 as well)
|
17
|
+
rubygc185.patch Ruby 1.8.5
|
18
|
+
rubygc186.patch Ruby 1.8.6
|
19
|
+
rubygc19.patch Ruby 1.9
|
20
|
+
|
21
|
+
Heap size and other options are controlled through environment
|
22
|
+
variables:
|
23
|
+
|
24
|
+
RUBY_HEAP_MIN_SLOTS
|
25
|
+
- the initial heap size in number of slots used
|
26
|
+
|
27
|
+
RUBY_HEAP_SLOTS_INCREMENT
|
28
|
+
- how many additional slots to allocate when Ruby allocates
|
29
|
+
new heap slots
|
30
|
+
|
31
|
+
RUBY_HEAP_SLOTS_GROWTH_FACTOR
|
32
|
+
- multiplicator used to increase heap block size for the next
|
33
|
+
allocation.
|
34
|
+
|
35
|
+
RUBY_GC_MALLOC_LIMIT
|
36
|
+
- the amount of C data structures which can be allocated
|
37
|
+
without triggering a GC (if this is set too low, GC will be
|
38
|
+
started even if there are empty slots available)
|
39
|
+
|
40
|
+
RUBY_HEAP_FREE_MIN
|
41
|
+
- number of free slots that should be available after GC
|
42
|
+
if fewer slots are available, additional heap will be allocated
|
43
|
+
(Ruby >= 1.8.5 ensures the freelist has at least heapsize*0.2 entries)
|
44
|
+
|
45
|
+
The following values make the patched GC behave like the unpatched GC:
|
46
|
+
|
47
|
+
RUBY_HEAP_MIN_SLOTS=10000
|
48
|
+
RUBY_HEAP_SLOTS_INCREMENT=10000
|
49
|
+
RUBY_HEAP_SLOTS_GROWTH_FACTOR=1.8
|
50
|
+
RUBY_GC_MALLOC_LIMIT=8000000
|
51
|
+
RUBY_HEAP_FREE_MIN=4096
|
52
|
+
|
53
|
+
Try experimenting with these values. You can use perf_run_gc to find
|
54
|
+
out how many slots you need.
|
55
|
+
|
56
|
+
Memory usage of the ruby interpreter can be observed by setting
|
57
|
+
RUBY_GC_STATS=1, before you invoke any of the railsbench commands.
|
58
|
+
|
59
|
+
Per default, GC data gets written to stderr. You can change this
|
60
|
+
behavior by setting environment variable RUBY_GC_DATA_FILE.
|
61
|
+
|
62
|
+
Additionally, the Ruby module GC gets some new methods:
|
63
|
+
|
64
|
+
GC.enable_stats - enable GC statistics collection
|
65
|
+
GC.disable_stats - disable GC statistics collection
|
66
|
+
GC.clear_stats - reset GC statistics
|
67
|
+
GC.collections - number of collections since stats have been enabled
|
68
|
+
GC.time - GC time used since stats have been enabled (miro seconds)
|
69
|
+
GC.dump - dumps current heap topology to the GC log file
|
70
|
+
GC.log - log given string to the GC log file
|
71
|
+
|
72
|
+
railsbench detects whether the patch has been applied and provides
|
73
|
+
GC statistics only in this case.
|
data/INSTALL
CHANGED
@@ -36,6 +36,11 @@ base directory, in order for railsbench to work properly.
|
|
36
36
|
If you obtained railsbench as a svn checkout, add the railsbench
|
37
37
|
script directory to your search path.
|
38
38
|
|
39
|
+
For bash shells you can use railsbench command completion by adding
|
40
|
+
this line code to your .bashrc file:
|
41
|
+
|
42
|
+
complete -W "`railsbench commands`" -o default railsbench
|
43
|
+
|
39
44
|
|
40
45
|
STEP 2: prepare your application for benchmarking
|
41
46
|
-------------------------------------------------
|
data/Manifest.txt
CHANGED
@@ -1,27 +1,37 @@
|
|
1
1
|
BUGS
|
2
2
|
CHANGELOG
|
3
|
+
GCPATCH
|
3
4
|
INSTALL
|
4
|
-
install.rb
|
5
|
-
postinstall.rb
|
6
5
|
LICENSE
|
7
6
|
Manifest.txt
|
8
7
|
PROBLEMS
|
9
8
|
README
|
10
9
|
Rakefile
|
11
|
-
setup.rb
|
12
|
-
ruby184gc.patch
|
13
|
-
ruby185gc.patch
|
14
10
|
bin/railsbench
|
15
|
-
|
11
|
+
config/benchmarking.rb
|
12
|
+
config/benchmarks.rb
|
13
|
+
config/benchmarks.yml
|
14
|
+
images/empty.png
|
15
|
+
images/minus.png
|
16
|
+
images/plus.png
|
17
|
+
install.rb
|
18
|
+
latest_changes.txt
|
19
|
+
lib/benchmark.rb
|
20
|
+
lib/railsbench/benchmark.rb
|
21
|
+
lib/railsbench/benchmark_specs.rb
|
16
22
|
lib/railsbench/gc_info.rb
|
17
23
|
lib/railsbench/perf_info.rb
|
18
24
|
lib/railsbench/perf_utils.rb
|
19
|
-
lib/railsbench/write_headers_only.rb
|
20
25
|
lib/railsbench/railsbenchmark.rb
|
21
|
-
lib/railsbench/
|
22
|
-
lib/
|
23
|
-
|
26
|
+
lib/railsbench/version.rb
|
27
|
+
lib/railsbench/write_headers_only.rb
|
28
|
+
postinstall.rb
|
29
|
+
ruby184gc.patch
|
30
|
+
ruby185gc.patch
|
31
|
+
ruby186gc.patch
|
32
|
+
ruby19gc.patch
|
24
33
|
script/convert_raw_data_files
|
34
|
+
script/generate_benchmarks
|
25
35
|
script/perf_bench
|
26
36
|
script/perf_comp
|
27
37
|
script/perf_comp_gc
|
@@ -38,6 +48,6 @@ script/perf_tex
|
|
38
48
|
script/perf_times
|
39
49
|
script/perf_times_gc
|
40
50
|
script/run_urls
|
41
|
-
|
42
|
-
|
43
|
-
|
51
|
+
setup.rb
|
52
|
+
test/railsbench_test.rb
|
53
|
+
test/test_helper.rb
|
data/PROBLEMS
CHANGED
File without changes
|
data/README
CHANGED
@@ -10,7 +10,7 @@ from the various scripts (and some won't run at all without the
|
|
10
10
|
patch).
|
11
11
|
|
12
12
|
This software was written and conceived by Stefan Kaes. The author can
|
13
|
-
be reached via email: <skaes@
|
13
|
+
be reached via email: <skaes@railsexpress.de>. Please send comments, bug
|
14
14
|
reports, patches or feature requests to this address.
|
15
15
|
|
16
16
|
|
@@ -21,7 +21,7 @@ FILES
|
|
21
21
|
switches:
|
22
22
|
-gcXXX : perform gc after XXX requests
|
23
23
|
-log[=level] : turn on rails logging (at given level)
|
24
|
-
-
|
24
|
+
-nocache : turn off rails caching
|
25
25
|
-path : print $: after loading rails and exit
|
26
26
|
|
27
27
|
config/bechmarks.yml
|
@@ -83,7 +83,7 @@ FILES
|
|
83
83
|
perf_run_gc n [ option-string [ config-name ] ]
|
84
84
|
- run a given benchmark, store performance data in a file
|
85
85
|
in directory $RAILS_PERF_DATA and print results
|
86
|
-
- requires Ruby GC patch
|
86
|
+
- requires Ruby GC patch (or Ruby Enterprise Edition)
|
87
87
|
|
88
88
|
perf_times_gc file
|
89
89
|
- analyse and print garbage collection statistics, which you
|
@@ -103,20 +103,36 @@ FILES
|
|
103
103
|
store profile data in a HTML file in directory $RAILS_PERF_DATA
|
104
104
|
file name is computed from date and benchmark name as described above
|
105
105
|
but has a .html extension instead of .txt
|
106
|
+
- a number of options to steer ruby-prof are available:
|
107
|
+
-ruby_prof=number[/number]
|
108
|
+
sets threshold and min_percent for ruby-prof (defaults to 0.1/1)
|
109
|
+
-profile_type=stack|grind|flat|graph|multi
|
110
|
+
selects the profile format (defaults to stack)
|
111
|
+
-measure_mode=process_time|wall_time|cpu_time|allocations|memory
|
112
|
+
selects what to measure (default to wall_time)
|
106
113
|
|
107
114
|
perf_plot [ options ] file1 file2 ...
|
108
|
-
- plot performance data from raw performance files using gruff
|
115
|
+
- plot performance data from raw performance files using gruff or gnuplot
|
116
|
+
see source for options
|
117
|
+
|
118
|
+
perf_plot_gc [ options ] file1 file2 ...
|
119
|
+
- plot data points from GC performance data stored in raw GC log files using gruff or gnuplot
|
120
|
+
this basically shows object type distribution across garbage collections
|
109
121
|
see source for options
|
110
122
|
|
111
123
|
perf_table [ options ] file1 file2 ...
|
112
124
|
- produces a tabular overview of perf data from given raw data files
|
113
125
|
see source for options
|
114
126
|
|
127
|
+
analyze_heap_dump file
|
128
|
+
- produces a html representation of a ruby heap dump.
|
129
|
+
useful for finding memory leaks.
|
115
130
|
|
116
131
|
ENVIRONMENT
|
117
132
|
|
118
133
|
RAILS_ROOT
|
119
|
-
-
|
134
|
+
- can be set to point to your rails app. if not set, railsbench can only
|
135
|
+
be called from the top level directory of your rails app
|
120
136
|
|
121
137
|
RAILS_PERF_DATA
|
122
138
|
- performance data sets will be stored into this directory
|
@@ -167,7 +183,7 @@ USAGE
|
|
167
183
|
|
168
184
|
will store benchmark data in file
|
169
185
|
|
170
|
-
$RAILS_PERF_DATA
|
186
|
+
$RAILS_PERF_DATA/<current-date>.index.mail.txt.
|
171
187
|
|
172
188
|
You can get nicely formatted output of benchmark data by running
|
173
189
|
|
@@ -306,7 +322,7 @@ CONFIGURATION
|
|
306
322
|
for the following benchmarks:
|
307
323
|
|
308
324
|
a) an entry for each named route
|
309
|
-
b) an entry for each route generated
|
325
|
+
b) an entry for each route generated by unnamed route definitions
|
310
326
|
c) an entry for each controller, combining all actions (named xyz_controller)
|
311
327
|
d) an entry combining all controllers (all_controllers), combining all
|
312
328
|
benchmarks generated by step c.
|
data/Rakefile
CHANGED
@@ -12,14 +12,13 @@ include FileUtils
|
|
12
12
|
require File.join(File.dirname(__FILE__), 'lib', 'railsbench', 'version')
|
13
13
|
|
14
14
|
AUTHOR = "Stefan Kaes" # can also be an array of Authors
|
15
|
-
EMAIL = "skaes@
|
15
|
+
EMAIL = "skaes@railsexpress.de"
|
16
16
|
DESCRIPTION = "rails benchmarking tools"
|
17
17
|
GEM_NAME = "railsbench" # what ppl will type to install your gem
|
18
18
|
RUBYFORGE_PROJECT = "railsbench" # The unix name for your project
|
19
19
|
HOMEPATH = "http://#{RUBYFORGE_PROJECT}.rubyforge.org"
|
20
20
|
RELEASE_TYPES = %w(gem tar zip) # can use: gem, tar, zip
|
21
21
|
|
22
|
-
|
23
22
|
NAME = "railsbench"
|
24
23
|
REV = nil # UNCOMMENT IF REQUIRED: File.read(".svn/entries")[/committed-rev="(d+)"/, 1] rescue nil
|
25
24
|
VERS = (ENV['VERSION'] ||= (Railsbench::VERSION::STRING + (REV ? ".#{REV}" : "")))
|
data/bin/railsbench
CHANGED
@@ -1,8 +1,10 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
3
|
RAILSBENCH_CMDS = %w(
|
4
|
+
analyze_heap_dump
|
4
5
|
base
|
5
6
|
generate_benchmarks
|
7
|
+
commands
|
6
8
|
convert_raw_data_files
|
7
9
|
help
|
8
10
|
install
|
@@ -38,6 +40,8 @@ end
|
|
38
40
|
RAILSBENCH_BASE = File.expand_path(File.dirname(__FILE__) + '/..')
|
39
41
|
|
40
42
|
case real_cmd
|
43
|
+
when 'commands'
|
44
|
+
puts RAILSBENCH_CMDS.keys.uniq.sort
|
41
45
|
when 'base'
|
42
46
|
puts "railsbench is installed in: #{RAILSBENCH_BASE}"
|
43
47
|
exit
|
@@ -49,12 +53,14 @@ when 'path'
|
|
49
53
|
exit
|
50
54
|
when 'install', 'postinstall'
|
51
55
|
load "#{RAILSBENCH_BASE}/#{real_cmd}.rb"
|
56
|
+
when 'analyze_heap_dump', /plot/
|
57
|
+
load "#{RAILSBENCH_BASE}/script/#{real_cmd}"
|
52
58
|
else
|
53
59
|
unless ENV['RAILS_ROOT']
|
54
60
|
if File.exists? 'config/environment.rb'
|
55
61
|
ENV['RAILS_ROOT'] = File.expand_path '.'
|
56
62
|
else
|
57
|
-
$stderr.puts "railsbench: RAILS_ROOT not set"
|
63
|
+
$stderr.puts "railsbench: RAILS_ROOT not set and could not be configured automatically"
|
58
64
|
exit 1
|
59
65
|
end
|
60
66
|
end
|
data/config/benchmarking.rb
CHANGED
File without changes
|
data/config/benchmarks.rb
CHANGED
@@ -13,8 +13,9 @@ RAILS_BENCHMARKER = RailsBenchmark.new
|
|
13
13
|
# especially if you use page caching.
|
14
14
|
# RAILS_BENCHMARKER.relative_url_root = '/blog'
|
15
15
|
|
16
|
-
#
|
17
|
-
#
|
16
|
+
# Customize the code below if your benchmark needs session data
|
17
|
+
# and/or your application specifies a custom session key.
|
18
18
|
|
19
19
|
# require 'user'
|
20
20
|
# RAILS_BENCHMARKER.session_data = {'account' => User.find_first("name='stefan'")}
|
21
|
+
# RAILS_BENCHMARKER.session_key = "my_secret_cookie_name"
|
data/config/benchmarks.yml
CHANGED
File without changes
|
data/images/empty.png
ADDED
Binary file
|
data/images/minus.png
ADDED
Binary file
|
data/images/plus.png
ADDED
Binary file
|
data/install.rb
CHANGED
@@ -53,7 +53,7 @@ __END__
|
|
53
53
|
### mode:ruby ***
|
54
54
|
### End: ***
|
55
55
|
|
56
|
-
# Copyright (C) 2005
|
56
|
+
# Copyright (C) 2005-2008 Stefan Kaes
|
57
57
|
#
|
58
58
|
# This program is free software; you can redistribute it and/or modify
|
59
59
|
# it under the terms of the GNU General Public License as published by
|
data/latest_changes.txt
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
* added support for generating call trees using ruby-prof
|
2
|
+
* show length of freelist in gc plots
|
3
|
+
* make it possible to plot very large gc logs by artificially reducing the data set size
|
4
|
+
* changed logger handling to no longer set logger to nil as most apps don't test for logger being nil
|
5
|
+
* rails 2.2 compatibility
|
6
|
+
* support gnuplot as a plotting engine
|
7
|
+
* allow post and query data to be stored in raw form in benchmarks file
|
8
|
+
* railsbench command completion: complete -W "`railsbench commands`" -o default railsbench
|
9
|
+
* individual scripts now try to autoconfigure RAILS_ROOT too
|
10
|
+
* make it possible to add cookie data in benchmarks
|
11
|
+
* support for memory leak checking under OS X
|
12
|
+
* print detailed comparison of total garbage allocated per object type
|
13
|
+
* provide an interface to summed garbage per object type via :garbage_totals
|
14
|
+
* railsbench now requires ruby-prof version >= 0.6
|
15
|
+
* support for XMLHttpRequest. patch by Vít Ondruch
|
16
|
+
* enable specification of session key. patch by Vít Ondruch
|
17
|
+
* exit benchmarking when an action raises an exception
|
18
|
+
* new GC patches for 1.8.6 and 1.9
|
data/lib/benchmark.rb
CHANGED
File without changes
|
@@ -0,0 +1,576 @@
|
|
1
|
+
=begin
|
2
|
+
#
|
3
|
+
# benchmark.rb - a performance benchmarking library
|
4
|
+
#
|
5
|
+
# $Id$
|
6
|
+
#
|
7
|
+
# Created by Gotoken (gotoken@notwork.org).
|
8
|
+
#
|
9
|
+
# Documentation by Gotoken (original RD), Lyle Johnson (RDoc conversion), and
|
10
|
+
# Gavin Sinclair (editing).
|
11
|
+
#
|
12
|
+
=end
|
13
|
+
|
14
|
+
# == Overview
|
15
|
+
#
|
16
|
+
# The Benchmark module provides methods for benchmarking Ruby code, giving
|
17
|
+
# detailed reports on the time taken for each task.
|
18
|
+
#
|
19
|
+
|
20
|
+
# The Benchmark module provides methods to measure and report the time
|
21
|
+
# used to execute Ruby code.
|
22
|
+
#
|
23
|
+
# * Measure the time to construct the string given by the expression
|
24
|
+
# <tt>"a"*1_000_000</tt>:
|
25
|
+
#
|
26
|
+
# require 'benchmark'
|
27
|
+
#
|
28
|
+
# puts Benchmark.measure { "a"*1_000_000 }
|
29
|
+
#
|
30
|
+
# On my machine (FreeBSD 3.2 on P5, 100MHz) this generates:
|
31
|
+
#
|
32
|
+
# 1.166667 0.050000 1.216667 ( 0.571355)
|
33
|
+
#
|
34
|
+
# This report shows the user CPU time, system CPU time, the sum of
|
35
|
+
# the user and system CPU times, and the elapsed real time. The unit
|
36
|
+
# of time is seconds.
|
37
|
+
#
|
38
|
+
# * Do some experiments sequentially using the #bm method:
|
39
|
+
#
|
40
|
+
# require 'benchmark'
|
41
|
+
#
|
42
|
+
# n = 50000
|
43
|
+
# Benchmark.bm do |x|
|
44
|
+
# x.report { for i in 1..n; a = "1"; end }
|
45
|
+
# x.report { n.times do ; a = "1"; end }
|
46
|
+
# x.report { 1.upto(n) do ; a = "1"; end }
|
47
|
+
# end
|
48
|
+
#
|
49
|
+
# The result:
|
50
|
+
#
|
51
|
+
# user system total real
|
52
|
+
# 1.033333 0.016667 1.016667 ( 0.492106)
|
53
|
+
# 1.483333 0.000000 1.483333 ( 0.694605)
|
54
|
+
# 1.516667 0.000000 1.516667 ( 0.711077)
|
55
|
+
#
|
56
|
+
# * Continuing the previous example, put a label in each report:
|
57
|
+
#
|
58
|
+
# require 'benchmark'
|
59
|
+
#
|
60
|
+
# n = 50000
|
61
|
+
# Benchmark.bm(7) do |x|
|
62
|
+
# x.report("for:") { for i in 1..n; a = "1"; end }
|
63
|
+
# x.report("times:") { n.times do ; a = "1"; end }
|
64
|
+
# x.report("upto:") { 1.upto(n) do ; a = "1"; end }
|
65
|
+
# end
|
66
|
+
#
|
67
|
+
# The result:
|
68
|
+
#
|
69
|
+
# user system total real
|
70
|
+
# for: 1.050000 0.000000 1.050000 ( 0.503462)
|
71
|
+
# times: 1.533333 0.016667 1.550000 ( 0.735473)
|
72
|
+
# upto: 1.500000 0.016667 1.516667 ( 0.711239)
|
73
|
+
#
|
74
|
+
#
|
75
|
+
# * The times for some benchmarks depend on the order in which items
|
76
|
+
# are run. These differences are due to the cost of memory
|
77
|
+
# allocation and garbage collection. To avoid these discrepancies,
|
78
|
+
# the #bmbm method is provided. For example, to compare ways to
|
79
|
+
# sort an array of floats:
|
80
|
+
#
|
81
|
+
# require 'benchmark'
|
82
|
+
#
|
83
|
+
# array = (1..1000000).map { rand }
|
84
|
+
#
|
85
|
+
# Benchmark.bmbm do |x|
|
86
|
+
# x.report("sort!") { array.dup.sort! }
|
87
|
+
# x.report("sort") { array.dup.sort }
|
88
|
+
# end
|
89
|
+
#
|
90
|
+
# The result:
|
91
|
+
#
|
92
|
+
# Rehearsal -----------------------------------------
|
93
|
+
# sort! 11.928000 0.010000 11.938000 ( 12.756000)
|
94
|
+
# sort 13.048000 0.020000 13.068000 ( 13.857000)
|
95
|
+
# ------------------------------- total: 25.006000sec
|
96
|
+
#
|
97
|
+
# user system total real
|
98
|
+
# sort! 12.959000 0.010000 12.969000 ( 13.793000)
|
99
|
+
# sort 12.007000 0.000000 12.007000 ( 12.791000)
|
100
|
+
#
|
101
|
+
#
|
102
|
+
# * Report statistics of sequential experiments with unique labels,
|
103
|
+
# using the #benchmark method:
|
104
|
+
#
|
105
|
+
# require 'benchmark'
|
106
|
+
#
|
107
|
+
# n = 50000
|
108
|
+
# Benchmark.benchmark(" "*7 + CAPTION, 7, FMTSTR, ">total:", ">avg:") do |x|
|
109
|
+
# tf = x.report("for:") { for i in 1..n; a = "1"; end }
|
110
|
+
# tt = x.report("times:") { n.times do ; a = "1"; end }
|
111
|
+
# tu = x.report("upto:") { 1.upto(n) do ; a = "1"; end }
|
112
|
+
# [tf+tt+tu, (tf+tt+tu)/3]
|
113
|
+
# end
|
114
|
+
#
|
115
|
+
# The result:
|
116
|
+
#
|
117
|
+
# user system total real
|
118
|
+
# for: 1.016667 0.016667 1.033333 ( 0.485749)
|
119
|
+
# times: 1.450000 0.016667 1.466667 ( 0.681367)
|
120
|
+
# upto: 1.533333 0.000000 1.533333 ( 0.722166)
|
121
|
+
# >total: 4.000000 0.033333 4.033333 ( 1.889282)
|
122
|
+
# >avg: 1.333333 0.011111 1.344444 ( 0.629761)
|
123
|
+
|
124
|
+
module Benchmark
|
125
|
+
|
126
|
+
BENCHMARK_VERSION = "2002-04-25" #:nodoc"
|
127
|
+
|
128
|
+
OUTPUT = STDOUT unless defined?(OUTPUT)
|
129
|
+
SYNC = true unless defined?(SYNC)
|
130
|
+
|
131
|
+
def Benchmark::times() # :nodoc:
|
132
|
+
Process::times()
|
133
|
+
end
|
134
|
+
|
135
|
+
|
136
|
+
# Invokes the block with a <tt>Benchmark::Report</tt> object, which
|
137
|
+
# may be used to collect and report on the results of individual
|
138
|
+
# benchmark tests. Reserves <i>label_width</i> leading spaces for
|
139
|
+
# labels on each line. Prints _caption_ at the top of the
|
140
|
+
# report, and uses _fmt_ to format each line.
|
141
|
+
# If the block returns an array of
|
142
|
+
# <tt>Benchmark::Tms</tt> objects, these will be used to format
|
143
|
+
# additional lines of output. If _label_ parameters are
|
144
|
+
# given, these are used to label these extra lines.
|
145
|
+
#
|
146
|
+
# _Note_: Other methods provide a simpler interface to this one, and are
|
147
|
+
# suitable for nearly all benchmarking requirements. See the examples in
|
148
|
+
# Benchmark, and the #bm and #bmbm methods.
|
149
|
+
#
|
150
|
+
# Example:
|
151
|
+
#
|
152
|
+
# require 'benchmark'
|
153
|
+
# include Benchmark # we need the CAPTION and FMTSTR constants
|
154
|
+
#
|
155
|
+
# n = 50000
|
156
|
+
# Benchmark.benchmark(" "*7 + CAPTION, 7, FMTSTR, ">total:", ">avg:") do |x|
|
157
|
+
# tf = x.report("for:") { for i in 1..n; a = "1"; end }
|
158
|
+
# tt = x.report("times:") { n.times do ; a = "1"; end }
|
159
|
+
# tu = x.report("upto:") { 1.upto(n) do ; a = "1"; end }
|
160
|
+
# [tf+tt+tu, (tf+tt+tu)/3]
|
161
|
+
# end
|
162
|
+
#
|
163
|
+
# <i>Generates:</i>
|
164
|
+
#
|
165
|
+
# user system total real
|
166
|
+
# for: 1.016667 0.016667 1.033333 ( 0.485749)
|
167
|
+
# times: 1.450000 0.016667 1.466667 ( 0.681367)
|
168
|
+
# upto: 1.533333 0.000000 1.533333 ( 0.722166)
|
169
|
+
# >total: 4.000000 0.033333 4.033333 ( 1.889282)
|
170
|
+
# >avg: 1.333333 0.011111 1.344444 ( 0.629761)
|
171
|
+
#
|
172
|
+
|
173
|
+
def benchmark(caption = "", label_width = nil, fmtstr = nil, *labels) # :yield: report
|
174
|
+
if SYNC
|
175
|
+
sync = OUTPUT.sync
|
176
|
+
OUTPUT.sync = true
|
177
|
+
end
|
178
|
+
label_width ||= 0
|
179
|
+
fmtstr ||= FMTSTR
|
180
|
+
raise ArgumentError, "no block" unless iterator?
|
181
|
+
OUTPUT.print caption
|
182
|
+
results = yield(Report.new(label_width, fmtstr))
|
183
|
+
Array === results and results.grep(Tms).each {|t|
|
184
|
+
OUTPUT.print((labels.shift || t.label || "").ljust(label_width),
|
185
|
+
t.format(fmtstr))
|
186
|
+
}
|
187
|
+
OUTPUT.sync = sync if SYNC
|
188
|
+
end
|
189
|
+
|
190
|
+
|
191
|
+
# A simple interface to the #benchmark method, #bm is generates sequential reports
|
192
|
+
# with labels. The parameters have the same meaning as for #benchmark.
|
193
|
+
#
|
194
|
+
# require 'benchmark'
|
195
|
+
#
|
196
|
+
# n = 50000
|
197
|
+
# Benchmark.bm(7) do |x|
|
198
|
+
# x.report("for:") { for i in 1..n; a = "1"; end }
|
199
|
+
# x.report("times:") { n.times do ; a = "1"; end }
|
200
|
+
# x.report("upto:") { 1.upto(n) do ; a = "1"; end }
|
201
|
+
# end
|
202
|
+
#
|
203
|
+
# <i>Generates:</i>
|
204
|
+
#
|
205
|
+
# user system total real
|
206
|
+
# for: 1.050000 0.000000 1.050000 ( 0.503462)
|
207
|
+
# times: 1.533333 0.016667 1.550000 ( 0.735473)
|
208
|
+
# upto: 1.500000 0.016667 1.516667 ( 0.711239)
|
209
|
+
#
|
210
|
+
|
211
|
+
def bm(label_width = 0, *labels, &blk) # :yield: report
|
212
|
+
benchmark(" "*label_width + CAPTION, label_width, FMTSTR, *labels, &blk)
|
213
|
+
end
|
214
|
+
|
215
|
+
|
216
|
+
# Sometimes benchmark results are skewed because code executed
|
217
|
+
# earlier encounters different garbage collection overheads than
|
218
|
+
# that run later. #bmbm attempts to minimize this effect by running
|
219
|
+
# the tests twice, the first time as a rehearsal in order to get the
|
220
|
+
# runtime environment stable, the second time for
|
221
|
+
# real. <tt>GC.start</tt> is executed before the start of each of
|
222
|
+
# the real timings; the cost of this is not included in the
|
223
|
+
# timings. In reality, though, there's only so much that #bmbm can
|
224
|
+
# do, and the results are not guaranteed to be isolated from garbage
|
225
|
+
# collection and other effects.
|
226
|
+
#
|
227
|
+
# Because #bmbm takes two passes through the tests, it can
|
228
|
+
# calculate the required label width.
|
229
|
+
#
|
230
|
+
# require 'benchmark'
|
231
|
+
#
|
232
|
+
# array = (1..1000000).map { rand }
|
233
|
+
#
|
234
|
+
# Benchmark.bmbm do |x|
|
235
|
+
# x.report("sort!") { array.dup.sort! }
|
236
|
+
# x.report("sort") { array.dup.sort }
|
237
|
+
# end
|
238
|
+
#
|
239
|
+
# <i>Generates:</i>
|
240
|
+
#
|
241
|
+
# Rehearsal -----------------------------------------
|
242
|
+
# sort! 11.928000 0.010000 11.938000 ( 12.756000)
|
243
|
+
# sort 13.048000 0.020000 13.068000 ( 13.857000)
|
244
|
+
# ------------------------------- total: 25.006000sec
|
245
|
+
#
|
246
|
+
# user system total real
|
247
|
+
# sort! 12.959000 0.010000 12.969000 ( 13.793000)
|
248
|
+
# sort 12.007000 0.000000 12.007000 ( 12.791000)
|
249
|
+
#
|
250
|
+
# #bmbm yields a Benchmark::Job object and returns an array of
|
251
|
+
# Benchmark::Tms objects.
|
252
|
+
#
|
253
|
+
def bmbm(width = 0, &blk) # :yield: job
|
254
|
+
job = Job.new(width)
|
255
|
+
yield(job)
|
256
|
+
width = job.width
|
257
|
+
if SYNC
|
258
|
+
sync = OUTPUT.sync
|
259
|
+
OUTPUT.sync = true
|
260
|
+
end
|
261
|
+
|
262
|
+
# rehearsal
|
263
|
+
OUTPUT.print "Rehearsal "
|
264
|
+
puts '-'*(width+CAPTION.length - "Rehearsal ".length)
|
265
|
+
list = []
|
266
|
+
job.list.each{|label,item|
|
267
|
+
OUTPUT.print(label.ljust(width))
|
268
|
+
res = Benchmark::measure(&item)
|
269
|
+
OUTPUT.print res.format()
|
270
|
+
list.push res
|
271
|
+
}
|
272
|
+
sum = Tms.new; list.each{|i| sum += i}
|
273
|
+
ets = sum.format("total: %tsec")
|
274
|
+
OUTPUT.printf("%s %s\n\n",
|
275
|
+
"-"*(width+CAPTION.length-ets.length-1), ets)
|
276
|
+
|
277
|
+
# take
|
278
|
+
OUTPUT.print ' '*width, CAPTION
|
279
|
+
list = []
|
280
|
+
ary = []
|
281
|
+
job.list.each{|label,item|
|
282
|
+
GC::start
|
283
|
+
OUTPUT.print label.ljust(width)
|
284
|
+
res = Benchmark::measure(&item)
|
285
|
+
OUTPUT.print res.format()
|
286
|
+
ary.push res
|
287
|
+
list.push [label, res]
|
288
|
+
}
|
289
|
+
|
290
|
+
OUTPUT.sync = sync if SYNC
|
291
|
+
ary
|
292
|
+
end
|
293
|
+
|
294
|
+
#
|
295
|
+
# Returns the time used to execute the given block as a
|
296
|
+
# Benchmark::Tms object.
|
297
|
+
#
|
298
|
+
def measure(label = "") # :yield:
|
299
|
+
t0, r0 = Benchmark.times, Time.now
|
300
|
+
yield
|
301
|
+
t1, r1 = Benchmark.times, Time.now
|
302
|
+
Benchmark::Tms.new(t1.utime - t0.utime,
|
303
|
+
t1.stime - t0.stime,
|
304
|
+
t1.cutime - t0.cutime,
|
305
|
+
t1.cstime - t0.cstime,
|
306
|
+
r1.to_f - r0.to_f,
|
307
|
+
label)
|
308
|
+
end
|
309
|
+
|
310
|
+
#
|
311
|
+
# Returns the elapsed real time used to execute the given block.
|
312
|
+
#
|
313
|
+
def realtime(&blk) # :yield:
|
314
|
+
Benchmark::measure(&blk).real
|
315
|
+
end
|
316
|
+
|
317
|
+
|
318
|
+
|
319
|
+
#
|
320
|
+
# A Job is a sequence of labelled blocks to be processed by the
|
321
|
+
# Benchmark.bmbm method. It is of little direct interest to the user.
|
322
|
+
#
|
323
|
+
class Job # :nodoc:
|
324
|
+
#
|
325
|
+
# Returns an initialized Job instance.
|
326
|
+
# Usually, one doesn't call this method directly, as new
|
327
|
+
# Job objects are created by the #bmbm method.
|
328
|
+
# _width_ is a initial value for the label offset used in formatting;
|
329
|
+
# the #bmbm method passes its _width_ argument to this constructor.
|
330
|
+
#
|
331
|
+
def initialize(width)
|
332
|
+
@width = width
|
333
|
+
@list = []
|
334
|
+
end
|
335
|
+
|
336
|
+
#
|
337
|
+
# Registers the given label and block pair in the job list.
|
338
|
+
#
|
339
|
+
def item(label = "", &blk) # :yield:
|
340
|
+
raise ArgmentError, "no block" unless block_given?
|
341
|
+
label.concat ' '
|
342
|
+
w = label.length
|
343
|
+
@width = w if @width < w
|
344
|
+
@list.push [label, blk]
|
345
|
+
self
|
346
|
+
end
|
347
|
+
|
348
|
+
alias report item
|
349
|
+
|
350
|
+
# An array of 2-element arrays, consisting of label and block pairs.
|
351
|
+
attr_reader :list
|
352
|
+
|
353
|
+
# Length of the widest label in the #list, plus one.
|
354
|
+
attr_reader :width
|
355
|
+
end
|
356
|
+
|
357
|
+
module_function :benchmark, :measure, :realtime, :bm, :bmbm
|
358
|
+
|
359
|
+
|
360
|
+
|
361
|
+
#
|
362
|
+
# This class is used by the Benchmark.benchmark and Benchmark.bm methods.
|
363
|
+
# It is of little direct interest to the user.
|
364
|
+
#
|
365
|
+
class Report # :nodoc:
|
366
|
+
#
|
367
|
+
# Returns an initialized Report instance.
|
368
|
+
# Usually, one doesn't call this method directly, as new
|
369
|
+
# Report objects are created by the #benchmark and #bm methods.
|
370
|
+
# _width_ and _fmtstr_ are the label offset and
|
371
|
+
# format string used by Tms#format.
|
372
|
+
#
|
373
|
+
def initialize(width = 0, fmtstr = nil)
|
374
|
+
@width, @fmtstr = width, fmtstr
|
375
|
+
end
|
376
|
+
|
377
|
+
#
|
378
|
+
# Prints the _label_ and measured time for the block,
|
379
|
+
# formatted by _fmt_. See Tms#format for the
|
380
|
+
# formatting rules.
|
381
|
+
#
|
382
|
+
def item(label = "", *fmt, &blk) # :yield:
|
383
|
+
OUTPUT.print label.ljust(@width)
|
384
|
+
res = Benchmark::measure(&blk)
|
385
|
+
OUTPUT.print res.format(@fmtstr, *fmt)
|
386
|
+
res
|
387
|
+
end
|
388
|
+
|
389
|
+
alias report item
|
390
|
+
end
|
391
|
+
|
392
|
+
|
393
|
+
|
394
|
+
#
|
395
|
+
# A data object, representing the times associated with a benchmark
|
396
|
+
# measurement.
|
397
|
+
#
|
398
|
+
class Tms
|
399
|
+
CAPTION = " user system total real\n"
|
400
|
+
FMTSTR = "%10.6u %10.6y %10.6t %10.6r\n"
|
401
|
+
|
402
|
+
# User CPU time
|
403
|
+
attr_reader :utime
|
404
|
+
|
405
|
+
# System CPU time
|
406
|
+
attr_reader :stime
|
407
|
+
|
408
|
+
# User CPU time of children
|
409
|
+
attr_reader :cutime
|
410
|
+
|
411
|
+
# System CPU time of children
|
412
|
+
attr_reader :cstime
|
413
|
+
|
414
|
+
# Elapsed real time
|
415
|
+
attr_reader :real
|
416
|
+
|
417
|
+
# Total time, that is _utime_ + _stime_ + _cutime_ + _cstime_
|
418
|
+
attr_reader :total
|
419
|
+
|
420
|
+
# Label
|
421
|
+
attr_reader :label
|
422
|
+
|
423
|
+
#
|
424
|
+
# Returns an initialized Tms object which has
|
425
|
+
# _u_ as the user CPU time, _s_ as the system CPU time,
|
426
|
+
# _cu_ as the children's user CPU time, _cs_ as the children's
|
427
|
+
# system CPU time, _real_ as the elapsed real time and _l_
|
428
|
+
# as the label.
|
429
|
+
#
|
430
|
+
def initialize(u = 0.0, s = 0.0, cu = 0.0, cs = 0.0, real = 0.0, l = nil)
|
431
|
+
@utime, @stime, @cutime, @cstime, @real, @label = u, s, cu, cs, real, l
|
432
|
+
@total = @utime + @stime + @cutime + @cstime
|
433
|
+
end
|
434
|
+
|
435
|
+
#
|
436
|
+
# Returns a new Tms object whose times are the sum of the times for this
|
437
|
+
# Tms object, plus the time required to execute the code block (_blk_).
|
438
|
+
#
|
439
|
+
def add(&blk) # :yield:
|
440
|
+
self + Benchmark::measure(&blk)
|
441
|
+
end
|
442
|
+
|
443
|
+
#
|
444
|
+
# An in-place version of #add.
|
445
|
+
#
|
446
|
+
def add!
|
447
|
+
t = Benchmark::measure(&blk)
|
448
|
+
@utime = utime + t.utime
|
449
|
+
@stime = stime + t.stime
|
450
|
+
@cutime = cutime + t.cutime
|
451
|
+
@cstime = cstime + t.cstime
|
452
|
+
@real = real + t.real
|
453
|
+
self
|
454
|
+
end
|
455
|
+
|
456
|
+
#
|
457
|
+
# Returns a new Tms object obtained by memberwise summation
|
458
|
+
# of the individual times for this Tms object with those of the other
|
459
|
+
# Tms object.
|
460
|
+
# This method and #/() are useful for taking statistics.
|
461
|
+
#
|
462
|
+
def +(other); memberwise(:+, other) end
|
463
|
+
|
464
|
+
#
|
465
|
+
# Returns a new Tms object obtained by memberwise subtraction
|
466
|
+
# of the individual times for the other Tms object from those of this
|
467
|
+
# Tms object.
|
468
|
+
#
|
469
|
+
def -(other); memberwise(:-, other) end
|
470
|
+
|
471
|
+
#
|
472
|
+
# Returns a new Tms object obtained by memberwise multiplication
|
473
|
+
# of the individual times for this Tms object by _x_.
|
474
|
+
#
|
475
|
+
def *(x); memberwise(:*, x) end
|
476
|
+
|
477
|
+
#
|
478
|
+
# Returns a new Tms object obtained by memberwise division
|
479
|
+
# of the individual times for this Tms object by _x_.
|
480
|
+
# This method and #+() are useful for taking statistics.
|
481
|
+
#
|
482
|
+
def /(x); memberwise(:/, x) end
|
483
|
+
|
484
|
+
#
|
485
|
+
# Returns the contents of this Tms object as
|
486
|
+
# a formatted string, according to a format string
|
487
|
+
# like that passed to Kernel.format. In addition, #format
|
488
|
+
# accepts the following extensions:
|
489
|
+
#
|
490
|
+
# <tt>%u</tt>:: Replaced by the user CPU time, as reported by Tms#utime.
|
491
|
+
# <tt>%y</tt>:: Replaced by the system CPU time, as reported by #stime (Mnemonic: y of "s*y*stem")
|
492
|
+
# <tt>%U</tt>:: Replaced by the children's user CPU time, as reported by Tms#cutime
|
493
|
+
# <tt>%Y</tt>:: Replaced by the children's system CPU time, as reported by Tms#cstime
|
494
|
+
# <tt>%t</tt>:: Replaced by the total CPU time, as reported by Tms#total
|
495
|
+
# <tt>%r</tt>:: Replaced by the elapsed real time, as reported by Tms#real
|
496
|
+
# <tt>%n</tt>:: Replaced by the label string, as reported by Tms#label (Mnemonic: n of "*n*ame")
|
497
|
+
#
|
498
|
+
# If _fmtstr_ is not given, FMTSTR is used as default value, detailing the
|
499
|
+
# user, system and real elapsed time.
|
500
|
+
#
|
501
|
+
def format(arg0 = nil, *args)
|
502
|
+
fmtstr = (arg0 || FMTSTR).dup
|
503
|
+
fmtstr.gsub!(/(%[-+\.\d]*)n/){"#{$1}s" % label}
|
504
|
+
fmtstr.gsub!(/(%[-+\.\d]*)u/){"#{$1}f" % utime}
|
505
|
+
fmtstr.gsub!(/(%[-+\.\d]*)y/){"#{$1}f" % stime}
|
506
|
+
fmtstr.gsub!(/(%[-+\.\d]*)U/){"#{$1}f" % cutime}
|
507
|
+
fmtstr.gsub!(/(%[-+\.\d]*)Y/){"#{$1}f" % cstime}
|
508
|
+
fmtstr.gsub!(/(%[-+\.\d]*)t/){"#{$1}f" % total}
|
509
|
+
fmtstr.gsub!(/(%[-+\.\d]*)r/){"(#{$1}f)" % real}
|
510
|
+
arg0 ? Kernel::format(fmtstr, *args) : fmtstr
|
511
|
+
end
|
512
|
+
|
513
|
+
#
|
514
|
+
# Same as #format.
|
515
|
+
#
|
516
|
+
def to_s
|
517
|
+
format
|
518
|
+
end
|
519
|
+
|
520
|
+
#
|
521
|
+
# Returns a new 6-element array, consisting of the
|
522
|
+
# label, user CPU time, system CPU time, children's
|
523
|
+
# user CPU time, children's system CPU time and elapsed
|
524
|
+
# real time.
|
525
|
+
#
|
526
|
+
def to_a
|
527
|
+
[@label, @utime, @stime, @cutime, @cstime, @real]
|
528
|
+
end
|
529
|
+
|
530
|
+
protected
|
531
|
+
def memberwise(op, x)
|
532
|
+
case x
|
533
|
+
when Benchmark::Tms
|
534
|
+
Benchmark::Tms.new(utime.__send__(op, x.utime),
|
535
|
+
stime.__send__(op, x.stime),
|
536
|
+
cutime.__send__(op, x.cutime),
|
537
|
+
cstime.__send__(op, x.cstime),
|
538
|
+
real.__send__(op, x.real)
|
539
|
+
)
|
540
|
+
else
|
541
|
+
Benchmark::Tms.new(utime.__send__(op, x),
|
542
|
+
stime.__send__(op, x),
|
543
|
+
cutime.__send__(op, x),
|
544
|
+
cstime.__send__(op, x),
|
545
|
+
real.__send__(op, x)
|
546
|
+
)
|
547
|
+
end
|
548
|
+
end
|
549
|
+
end
|
550
|
+
|
551
|
+
# The default caption string (heading above the output times).
|
552
|
+
CAPTION = Benchmark::Tms::CAPTION
|
553
|
+
|
554
|
+
# The default format string used to display times. See also Benchmark::Tms#format.
|
555
|
+
FMTSTR = Benchmark::Tms::FMTSTR
|
556
|
+
end
|
557
|
+
|
558
|
+
if __FILE__ == $0
|
559
|
+
include Benchmark
|
560
|
+
|
561
|
+
n = ARGV[0].to_i.nonzero? || 50000
|
562
|
+
puts %Q([#{n} times iterations of `a = "1"'])
|
563
|
+
benchmark(" " + CAPTION, 7, FMTSTR) do |x|
|
564
|
+
x.report("for:") {for i in 1..n; a = "1"; end} # Benchmark::measure
|
565
|
+
x.report("times:") {n.times do ; a = "1"; end}
|
566
|
+
x.report("upto:") {1.upto(n) do ; a = "1"; end}
|
567
|
+
end
|
568
|
+
|
569
|
+
benchmark do
|
570
|
+
[
|
571
|
+
measure{for i in 1..n; a = "1"; end}, # Benchmark::measure
|
572
|
+
measure{n.times do ; a = "1"; end},
|
573
|
+
measure{1.upto(n) do ; a = "1"; end}
|
574
|
+
]
|
575
|
+
end
|
576
|
+
end
|