benchmark-inputs 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: '0487188b1ce28f1da7567a9ac398e3eb9c7a35ea'
4
+ data.tar.gz: 074c9f1dfe4f16ede8603bf591957850272423c2
5
+ SHA512:
6
+ metadata.gz: 2af225315d2406c0a81c0fae7b8043a40259dee1051d3194ea9834614666b1fc1a19d3b1defb2aebcfb6cd74895a485c1b76eb316cc442d9c638140b65de9b8a
7
+ data.tar.gz: e1979aa7693094a20ba26567fb5fda191513b3340e18753c595f97a04bd40a48c9cbbda44e36954e839984d5bf4b323764867396b9207c7ea25678220d2ba730
data/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
10
+ *.gem
data/.travis.yml ADDED
@@ -0,0 +1,5 @@
1
+ sudo: false
2
+ language: ruby
3
+ rvm:
4
+ - 2.2.4
5
+ before_install: gem install bundler -v 1.12.5
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in benchmark-inputs.gemspec
4
+ gemspec
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2016 Jonathan Hefner
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,149 @@
1
+ # benchmark-inputs
2
+
3
+ Input-focused benchmarking for Ruby. Given one or more blocks and a
4
+ list of inputs to yield to them, benchmark-inputs will measure the speed
5
+ (in invocations per second) of each block. Blocks which execute very
6
+ quickly, as in microbenchmarks, are automatically invoked repeatedly to
7
+ provide accurate measurements.
8
+
9
+
10
+ ## Motivation
11
+
12
+ I <3 [Fast Ruby][fast-ruby]. By extension, I <3 [benchmark-ips]. But,
13
+ for certain usages, benchmark-ips doesn't let me write benchmarks the
14
+ way I'd like. Consider the following example, *using benchmark-ips*:
15
+
16
+ ```ruby
17
+ require 'benchmark/ips' ### USING benchmark-ips (NOT benchmark-inputs)
18
+
19
+ STRINGS = ['abc', 'aaa', 'xyz', '']
20
+ Benchmark.ips do |job|
21
+ job.report('String#tr'){ STRINGS.each{|s| s.tr('a', 'A') } }
22
+ job.report('String#gsub'){ STRINGS.each{|s| s.gsub(/a/, 'A') } }
23
+ job.compare!
24
+ end
25
+ ```
26
+
27
+ The calls to `STRINGS.each` introduce performance overhead which skews
28
+ the time measurements. The less time the target function takes, the
29
+ more relative overhead, and thus the more skew. For a microbenchmark
30
+ this can be a problem. A possible workaround is to invoke the function
31
+ on each value individually, but that is more verbose and error-prone:
32
+
33
+ ```ruby
34
+ require 'benchmark/ips' ### USING benchmark-ips (NOT benchmark-inputs)
35
+
36
+ s1 = 'abc'; s2 = 'aaa'; s3 = 'xyz'; s4 = ''
37
+ Benchmark.ips do |job|
38
+ job.report('String#tr') do
39
+ s1.tr('a', 'A'); s2.tr('a', 'A')
40
+ s3.tr('a', 'A'); s4.tr('a', 'A')
41
+ end
42
+ job.report('String#gsub') do
43
+ s1.gsub(/a/, 'A'); s2.gsub(/a/, 'A')
44
+ s3.gsub(/a/, 'A'); s4.gsub(/a/, 'A')
45
+ end
46
+ job.compare!
47
+ end
48
+ ```
49
+
50
+ *Enter benchmark-inputs*. Here is how the same benchmark looks using
51
+ this gem: <a name="example1"></a>
52
+
53
+ ```ruby
54
+ require 'benchmark/inputs' ### USING benchmark-inputs
55
+
56
+ Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
57
+ job.report('String#tr'){|s| s.tr('a', 'A') }
58
+ job.report('String#gsub'){|s| s.gsub(/a/, 'A') }
59
+ job.compare!
60
+ end
61
+ ```
62
+
63
+ Which prints something like the following to `$stdout`:
64
+
65
+ ```
66
+ String#tr
67
+ 1376876.5 i/s (±0.01%)
68
+ String#gsub
69
+ 264340.1 i/s (±0.02%)
70
+
71
+ Comparison:
72
+ String#tr: 1376876.5 i/s
73
+ String#gsub: 264340.1 i/s - 5.21x slower
74
+ ```
75
+
76
+
77
+ ### Benchmarking destructive operations
78
+
79
+ Destructive operations also pose a challenge for microbenchmarks. Each
80
+ invocation needs to operate on the same data, but `dup`ing the data
81
+ introduces too much overhead and skew.
82
+
83
+ benchmark-inputs' solution is to estimate the overhead incurred by
84
+ `dup`, and exclude that from time measurements. Because the benchmark
85
+ job already controls the input data, all of this can be handled with a
86
+ single configuration line:
87
+
88
+ ```ruby
89
+ require 'benchmark/inputs'
90
+
91
+ Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
92
+ job.dup_inputs = true # <--- single configuration line
93
+ job.report('String#tr!'){|s| s.tr!('a', 'A') }
94
+ job.report('String#gsub!'){|s| s.gsub!(/a/, 'A') }
95
+ job.compare!
96
+ end
97
+ ```
98
+
99
+ Which prints out something like:
100
+
101
+ ```
102
+ String#tr!
103
+ 1777274.5 i/s (±0.01%)
104
+ String#gsub!
105
+ 282396.3 i/s (±0.00%)
106
+
107
+ Comparison:
108
+ String#tr!: 1777274.5 i/s
109
+ String#gsub!: 282396.3 i/s - 6.29x slower
110
+ ```
111
+
112
+ That shows a slightly larger performance gap than the previous
113
+ benchmark. This makes sense because the overhead of allocating new
114
+ strings--previously via a non-bang method, but now via `dup`--is now
115
+ excluded from the timings. Thus, the speed of `tr!` relative to `gsub!`
116
+ is further emphasized.
117
+
118
+
119
+ ## Limitations
120
+
121
+ `Benchmark.inputs` generates code based on the array of input values it
122
+ is given. Each input value becomes a local variable. While there is
123
+ theoretically no limit to the number of local variables that can be
124
+ generated, more than a few hundred may slow down the benchmark. But,
125
+ because input values are used to represent different scenarios rather
126
+ than control the number of invocations, this limitation shouldn't pose a
127
+ problem.
128
+
129
+
130
+ ## Installation
131
+
132
+ $ gem install benchmark-inputs
133
+
134
+
135
+ ## Usage
136
+
137
+ See the [example above](#example1), or check the
138
+ [documentation](http://www.rubydoc.info/gems/benchmark-inputs).
139
+
140
+
141
+ ## License
142
+
143
+ [MIT License](http://opensource.org/licenses/MIT)
144
+
145
+
146
+
147
+
148
+ [fast-ruby]: https://github.com/JuanitoFatas/fast-ruby
149
+ [benchmark-ips]: https://github.com/evanphx/benchmark-ips
data/Rakefile ADDED
@@ -0,0 +1,23 @@
1
+ require "bundler/gem_tasks"
2
+ require "rake/testtask"
3
+ require "yard"
4
+
5
+
6
+ desc 'Launch IRB with this gem pre-loaded'
7
+ task :irb do
8
+ require "benchmark/inputs"
9
+ require "irb"
10
+ ARGV.clear
11
+ IRB.start
12
+ end
13
+
14
+ YARD::Rake::YardocTask.new(:doc) do |t|
15
+ end
16
+
17
+ Rake::TestTask.new(:test) do |t|
18
+ t.libs << "test"
19
+ t.libs << "lib"
20
+ t.test_files = FileList['test/**/*_test.rb']
21
+ end
22
+
23
+ task :default => :test
@@ -0,0 +1,25 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'benchmark/inputs/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "benchmark-inputs"
8
+ spec.version = Benchmark::Inputs::VERSION
9
+ spec.authors = ["Jonathan Hefner"]
10
+ spec.email = ["jonathan.hefner@gmail.com"]
11
+
12
+ spec.summary = %q{Input-focused benchmarking}
13
+ spec.homepage = "https://github.com/jonathanhefner/benchmark-inputs"
14
+ spec.license = "MIT"
15
+
16
+ spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
17
+ spec.bindir = "exe"
18
+ spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
19
+ spec.require_paths = ["lib"]
20
+
21
+ spec.add_development_dependency "bundler", "~> 1.12"
22
+ spec.add_development_dependency "rake", "~> 10.0"
23
+ spec.add_development_dependency "minitest", "~> 5.0"
24
+ spec.add_development_dependency "yard", "~> 0.8"
25
+ end
@@ -0,0 +1,197 @@
1
+ require 'benchmark/inputs/version'
2
+
3
+ module Benchmark
4
+
5
+ # Initializes a benchmark job with the given inputs and yields that
6
+ # job to the given block.
7
+ #
8
+ # Example:
9
+ # Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
10
+ # job.report('String#tr'){|s| s.tr('a', 'A') }
11
+ # job.report('String#gsub'){|s| s.gsub(/a/, 'A') }
12
+ # job.compare!
13
+ # end
14
+ #
15
+ # @param [Array] vals input values to yield to each benchmark action
16
+ # @yield [job] configures job and runs benchmarks
17
+ # @yieldparam [Benchmark::Inputs::Job] job benchmark runner
18
+ # @return [Benchmark::Inputs::Job] benchmark runner
19
+ def self.inputs(vals)
20
+ job = Inputs::Job.new(vals)
21
+ yield job
22
+ job
23
+ end
24
+
25
+
26
+ module Inputs
27
+
28
+ NS_PER_S = 1_000_000_000
29
+ NS_PER_MS = NS_PER_S / 1_000
30
+
31
+ class Job
32
+ attr_accessor :sample_n, :sample_dt
33
+ attr_reader :dup_inputs, :reports
34
+
35
+ def initialize(inputs)
36
+ @inputs = inputs
37
+ @dup_inputs = false
38
+ @sample_n = 10
39
+ @sample_dt = NS_PER_MS * 200
40
+ @reports = []
41
+ def_bench!
42
+ end
43
+
44
+ # Sets the +dup_inputs+ flag. If set to true, causes input values
45
+ # to be +dup+'d before they are passed to a +report+ block. This
46
+ # is necessary when +report+ blocks destructively modify their
47
+ # arguments.
48
+ #
49
+ # Example:
50
+ # Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
51
+ # job.dup_inputs = true # <---
52
+ # job.report('String#tr!'){|s| s.tr!('a', 'A') }
53
+ # job.report('String#gsub!'){|s| s.gsub!(/a/, 'A') }
54
+ # job.compare!
55
+ # end
56
+ #
57
+ # @param [Boolean] val value to set
58
+ def dup_inputs=(val)
59
+ @dup_inputs = val
60
+ def_bench!
61
+ @dup_inputs
62
+ end
63
+
64
+ # Benchmarks the given block using the initially provided input
65
+ # values. If +#dup_inputs+ is set to true, each input value is
66
+ # +dup+'d before being passed to the block. Afterwards, the
67
+ # block's invocations per second (i/s) is printed to +$stdout+.
68
+ #
69
+ # @param [String] label label for the benchmark
70
+ # @yield [input] action to benchmark
71
+ # @yieldparam input one of the initially provided input values
72
+ def report(label)
73
+ # estimate repititions
74
+ reps = 1
75
+ reps_time = 0
76
+ while reps_time < @sample_dt
77
+ reps_time = bench(reps){|x| yield(x) }
78
+ reps *= 2
79
+ end
80
+ reps = ((reps / 2) * (reps_time.to_f / @sample_dt)).ceil
81
+
82
+ # benchmark
83
+ r = Report.new(label, reps * @inputs.length)
84
+ i = @sample_n
85
+ GC.start()
86
+ while i > 0
87
+ r.add_sample(bench(reps){|x| yield(x) } - bench(reps){|x| x })
88
+ i -= 1
89
+ end
90
+
91
+ $stdout.puts(r.label)
92
+ $stdout.printf(" %.1f i/s (\u00B1%.2f%%)\n", r.ips, r.stddev / r.ips)
93
+ @reports << r
94
+ r
95
+ end
96
+
97
+ # Prints the relative speeds (from fastest to slowest) of all
98
+ # +#report+ed benchmarks to +$stdout+.
99
+ def compare!
100
+ return $stdout.puts('Nothing to compare!') if @reports.empty?
101
+
102
+ @reports.sort_by!{|r| -r.ips }
103
+ @reports.each{|r| r.slower_than!(@reports.first) }
104
+
105
+ max_label_len = @reports.map{|r| r.label.length }.max
106
+ format = " %#{max_label_len}s: %10.1f i/s"
107
+
108
+ $stdout.puts("\nComparison:")
109
+ @reports.each_with_index do |r, i|
110
+ $stdout.printf(format, r.label, r.ips)
111
+ if r.ratio
112
+ $stdout.printf(' - %.2fx slower', r.ratio)
113
+ elsif i > 0
114
+ $stdout.printf(' - same-ish: difference falls within error')
115
+ end
116
+ $stdout.puts
117
+ end
118
+ $stdout.puts
119
+ end
120
+
121
+ private
122
+
123
+ def def_bench!
124
+ assigns = @inputs.each_index.map do |i|
125
+ "x#{i} = @inputs[#{i}]"
126
+ end.join(';')
127
+
128
+ yields = @inputs.each_with_index.map do |x, i|
129
+ dup = (@dup_inputs && x.respond_to?(:dup)) ? '.dup' : ''
130
+ "yield(x#{i}#{dup})"
131
+ end.join(';')
132
+
133
+ code = <<-CODE
134
+ def bench(reps)
135
+ #{assigns}
136
+ i = reps
137
+ before_time = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)
138
+ while i > 0
139
+ #{yields}
140
+ i -= 1
141
+ end
142
+ after_time = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)
143
+ after_time - before_time
144
+ end
145
+ CODE
146
+
147
+ instance_eval{ undef :bench } if self.respond_to?(:bench)
148
+ instance_eval(code)
149
+ end
150
+ end
151
+
152
+
153
+ class Report
154
+ attr_reader :label, :ratio
155
+
156
+ def initialize(label, invocs_per_sample)
157
+ @label = label.to_s
158
+ @invocs_per_sample = invocs_per_sample.to_f
159
+ @ratio = nil
160
+
161
+ @n = 0
162
+ @mean = 0.0
163
+ @m2 = 0.0
164
+ end
165
+
166
+ def add_sample(time_ns)
167
+ sample_ips = @invocs_per_sample * NS_PER_S / time_ns
168
+
169
+ # see https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm
170
+ # or Knuth's TAOCP vol 2, 3rd edition, page 232
171
+ @n += 1
172
+ delta = sample_ips - @mean
173
+ @mean += delta / @n
174
+ @m2 += delta * (sample_ips - @mean)
175
+ @stddev = nil
176
+ end
177
+
178
+ def ips
179
+ @mean
180
+ end
181
+
182
+ def stddev
183
+ @stddev ||= @n < 2 ? 0.0 : Math.sqrt(@m2 / (@n - 1))
184
+ end
185
+
186
+ def slower_than!(faster)
187
+ @ratio = overlap?(faster) ? nil : (faster.ips / self.ips)
188
+ end
189
+
190
+ def overlap?(faster)
191
+ (faster.ips - faster.stddev) <= (self.ips + self.stddev)
192
+ end
193
+ end
194
+
195
+ end
196
+
197
+ end
@@ -0,0 +1,5 @@
1
+ module Benchmark
2
+ module Inputs
3
+ VERSION = "1.0.0"
4
+ end
5
+ end
metadata ADDED
@@ -0,0 +1,110 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: benchmark-inputs
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0
5
+ platform: ruby
6
+ authors:
7
+ - Jonathan Hefner
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2016-07-05 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: bundler
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '1.12'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '1.12'
27
+ - !ruby/object:Gem::Dependency
28
+ name: rake
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '10.0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '10.0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: minitest
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '5.0'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '5.0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: yard
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '0.8'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '0.8'
69
+ description:
70
+ email:
71
+ - jonathan.hefner@gmail.com
72
+ executables: []
73
+ extensions: []
74
+ extra_rdoc_files: []
75
+ files:
76
+ - ".gitignore"
77
+ - ".travis.yml"
78
+ - Gemfile
79
+ - LICENSE.txt
80
+ - README.md
81
+ - Rakefile
82
+ - benchmark-inputs.gemspec
83
+ - lib/benchmark/inputs.rb
84
+ - lib/benchmark/inputs/version.rb
85
+ homepage: https://github.com/jonathanhefner/benchmark-inputs
86
+ licenses:
87
+ - MIT
88
+ metadata: {}
89
+ post_install_message:
90
+ rdoc_options: []
91
+ require_paths:
92
+ - lib
93
+ required_ruby_version: !ruby/object:Gem::Requirement
94
+ requirements:
95
+ - - ">="
96
+ - !ruby/object:Gem::Version
97
+ version: '0'
98
+ required_rubygems_version: !ruby/object:Gem::Requirement
99
+ requirements:
100
+ - - ">="
101
+ - !ruby/object:Gem::Version
102
+ version: '0'
103
+ requirements: []
104
+ rubyforge_project:
105
+ rubygems_version: 2.4.8
106
+ signing_key:
107
+ specification_version: 4
108
+ summary: Input-focused benchmarking
109
+ test_files: []
110
+ has_rdoc: