benchmark-inputs 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: '0487188b1ce28f1da7567a9ac398e3eb9c7a35ea'
+  data.tar.gz: 074c9f1dfe4f16ede8603bf591957850272423c2
+SHA512:
+  metadata.gz: 2af225315d2406c0a81c0fae7b8043a40259dee1051d3194ea9834614666b1fc1a19d3b1defb2aebcfb6cd74895a485c1b76eb316cc442d9c638140b65de9b8a
+  data.tar.gz: e1979aa7693094a20ba26567fb5fda191513b3340e18753c595f97a04bd40a48c9cbbda44e36954e839984d5bf4b323764867396b9207c7ea25678220d2ba730

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.gem

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.2.4
+before_install: gem install bundler -v 1.12.5

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in benchmark-inputs.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Jonathan Hefner
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,149 @@
+# benchmark-inputs
+
+Input-focused benchmarking for Ruby.  Given one or more blocks and a
+list of inputs to yield to them, benchmark-inputs will measure the speed
+(in invocations per second) of each block.  Blocks which execute very
+quickly, as in microbenchmarks, are automatically invoked repeatedly to
+provide accurate measurements.
+
+
+## Motivation
+
+I <3 [Fast Ruby][fast-ruby].  By extension, I <3 [benchmark-ips].  But,
+for certain usages, benchmark-ips doesn't let me write benchmarks the
+way I'd like. Consider the following example, *using benchmark-ips*:
+
+```ruby
+require 'benchmark/ips' ### USING benchmark-ips (NOT benchmark-inputs)
+
+STRINGS = ['abc', 'aaa', 'xyz', '']
+Benchmark.ips do |job|
+  job.report('String#tr'){ STRINGS.each{|s| s.tr('a', 'A') } }
+  job.report('String#gsub'){ STRINGS.each{|s| s.gsub(/a/, 'A') } }
+  job.compare!
+end
+```
+
+The calls to `STRINGS.each` introduce performance overhead which skews
+the time measurements.  The less time the target function takes, the
+more relative overhead, and thus the more skew.  For a microbenchmark
+this can be a problem.  A possible workaround is to invoke the function
+on each value individually, but that is more verbose and error-prone:
+
+```ruby
+require 'benchmark/ips' ### USING benchmark-ips (NOT benchmark-inputs)
+
+s1 = 'abc'; s2 = 'aaa'; s3 = 'xyz'; s4 = ''
+Benchmark.ips do |job|
+  job.report('String#tr') do
+    s1.tr('a', 'A'); s2.tr('a', 'A')
+    s3.tr('a', 'A'); s4.tr('a', 'A')
+  end
+  job.report('String#gsub') do
+    s1.gsub(/a/, 'A'); s2.gsub(/a/, 'A')
+    s3.gsub(/a/, 'A'); s4.gsub(/a/, 'A')
+  end
+  job.compare!
+end
+```
+
+*Enter benchmark-inputs*.  Here is how the same benchmark looks using
+this gem: <a name="example1"></a>
+
+```ruby
+require 'benchmark/inputs' ### USING benchmark-inputs
+
+Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
+  job.report('String#tr'){|s| s.tr('a', 'A') }
+  job.report('String#gsub'){|s| s.gsub(/a/, 'A') }
+  job.compare!
+end
+```
+
+Which prints something like the following to `$stdout`:
+
+```
+String#tr
+  1376876.5 i/s (±0.01%)
+String#gsub
+  264340.1 i/s (±0.02%)
+
+Comparison:
+    String#tr:   1376876.5 i/s
+  String#gsub:    264340.1 i/s - 5.21x slower
+```
+
+
+### Benchmarking destructive operations
+
+Destructive operations also pose a challenge for microbenchmarks.  Each
+invocation needs to operate on the same data, but `dup`ing the data
+introduces too much overhead and skew.
+
+benchmark-inputs' solution is to estimate the overhead incurred by
+`dup`, and exclude that from time measurements.  Because the benchmark
+job already controls the input data, all of this can be handled with a
+single configuration line:
+
+```ruby
+require 'benchmark/inputs'
+
+Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
+  job.dup_inputs = true  # <--- single configuration line
+  job.report('String#tr!'){|s| s.tr!('a', 'A') }
+  job.report('String#gsub!'){|s| s.gsub!(/a/, 'A') }
+  job.compare!
+end
+```
+
+Which prints out something like:
+
+```
+String#tr!
+  1777274.5 i/s (±0.01%)
+String#gsub!
+  282396.3 i/s (±0.00%)
+
+Comparison:
+    String#tr!:   1777274.5 i/s
+  String#gsub!:    282396.3 i/s - 6.29x slower
+```
+
+That shows a slightly larger performance gap than the previous
+benchmark.  This makes sense because the overhead of allocating new
+strings--previously via a non-bang method, but now via `dup`--is now
+excluded from the timings.  Thus, the speed of `tr!` relative to `gsub!`
+is further emphasized.
+
+
+## Limitations
+
+`Benchmark.inputs` generates code based on the array of input values it
+is given.  Each input value becomes a local variable.  While there is
+theoretically no limit to the number of local variables that can be
+generated, more than a few hundred may slow down the benchmark.  But,
+because input values are used to represent different scenarios rather
+than control the number of invocations, this limitation shouldn't pose a
+problem.
+
+
+## Installation
+
+    $ gem install benchmark-inputs
+
+
+## Usage
+
+See the [example above](#example1), or check the
+[documentation](http://www.rubydoc.info/gems/benchmark-inputs).
+
+
+## License
+
+[MIT License](http://opensource.org/licenses/MIT)
+
+
+
+
+[fast-ruby]: https://github.com/JuanitoFatas/fast-ruby
+[benchmark-ips]: https://github.com/evanphx/benchmark-ips

data/Rakefile

@@ -0,0 +1,23 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+require "yard"
+
+
+desc 'Launch IRB with this gem pre-loaded'
+task :irb do
+  require "benchmark/inputs"
+  require "irb"
+  ARGV.clear
+  IRB.start
+end
+
+YARD::Rake::YardocTask.new(:doc) do |t|
+end
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/benchmark-inputs.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'benchmark/inputs/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "benchmark-inputs"
+  spec.version       = Benchmark::Inputs::VERSION
+  spec.authors       = ["Jonathan Hefner"]
+  spec.email         = ["jonathan.hefner@gmail.com"]
+
+  spec.summary       = %q{Input-focused benchmarking}
+  spec.homepage      = "https://github.com/jonathanhefner/benchmark-inputs"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.12"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "minitest", "~> 5.0"
+  spec.add_development_dependency "yard", "~> 0.8"
+end

data/lib/benchmark/inputs.rb

@@ -0,0 +1,197 @@
+require 'benchmark/inputs/version'
+
+module Benchmark
+
+  # Initializes a benchmark job with the given inputs and yields that
+  # job to the given block.
+  #
+  # Example:
+  #   Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
+  #     job.report('String#tr'){|s| s.tr('a', 'A') }
+  #     job.report('String#gsub'){|s| s.gsub(/a/, 'A') }
+  #     job.compare!
+  #   end
+  #
+  # @param [Array] vals input values to yield to each benchmark action
+  # @yield [job] configures job and runs benchmarks
+  # @yieldparam [Benchmark::Inputs::Job] job benchmark runner
+  # @return [Benchmark::Inputs::Job] benchmark runner
+  def self.inputs(vals)
+    job = Inputs::Job.new(vals)
+    yield job
+    job
+  end
+
+
+  module Inputs
+
+    NS_PER_S = 1_000_000_000
+    NS_PER_MS = NS_PER_S / 1_000
+
+    class Job
+      attr_accessor :sample_n, :sample_dt
+      attr_reader   :dup_inputs, :reports
+
+      def initialize(inputs)
+        @inputs = inputs
+        @dup_inputs = false
+        @sample_n = 10
+        @sample_dt = NS_PER_MS * 200
+        @reports = []
+        def_bench!
+      end
+
+      # Sets the +dup_inputs+ flag.  If set to true, causes input values
+      # to be +dup+'d before they are passed to a +report+ block.  This
+      # is necessary when +report+ blocks destructively modify their
+      # arguments.
+      #
+      # Example:
+      #   Benchmark.inputs(['abc', 'aaa', 'xyz', '']) do |job|
+      #     job.dup_inputs = true  # <---
+      #     job.report('String#tr!'){|s| s.tr!('a', 'A') }
+      #     job.report('String#gsub!'){|s| s.gsub!(/a/, 'A') }
+      #     job.compare!
+      #   end
+      #
+      # @param [Boolean] val value to set
+      def dup_inputs=(val)
+        @dup_inputs = val
+        def_bench!
+        @dup_inputs
+      end
+
+      # Benchmarks the given block using the initially provided input
+      # values.  If +#dup_inputs+ is set to true, each input value is
+      # +dup+'d before being passed to the block.  Afterwards, the
+      # block's invocations per second (i/s) is printed to +$stdout+.
+      #
+      # @param [String] label label for the benchmark
+      # @yield [input] action to benchmark
+      # @yieldparam input one of the initially provided input values
+      def report(label)
+        # estimate repititions
+        reps = 1
+        reps_time = 0
+        while reps_time < @sample_dt
+          reps_time = bench(reps){|x| yield(x) }
+          reps *= 2
+        end
+        reps = ((reps / 2) * (reps_time.to_f / @sample_dt)).ceil
+
+        # benchmark
+        r = Report.new(label, reps * @inputs.length)
+        i = @sample_n
+        GC.start()
+        while i > 0
+          r.add_sample(bench(reps){|x| yield(x) } - bench(reps){|x| x })
+          i -= 1
+        end
+
+        $stdout.puts(r.label)
+        $stdout.printf("  %.1f i/s (\u00B1%.2f%%)\n", r.ips, r.stddev / r.ips)
+        @reports << r
+        r
+      end
+
+      # Prints the relative speeds (from fastest to slowest) of all
+      # +#report+ed benchmarks to +$stdout+.
+      def compare!
+        return $stdout.puts('Nothing to compare!') if @reports.empty?
+
+        @reports.sort_by!{|r| -r.ips }
+        @reports.each{|r| r.slower_than!(@reports.first) }
+
+        max_label_len = @reports.map{|r| r.label.length }.max
+        format = "  %#{max_label_len}s:  %10.1f i/s"
+
+        $stdout.puts("\nComparison:")
+        @reports.each_with_index do |r, i|
+          $stdout.printf(format, r.label, r.ips)
+          if r.ratio
+            $stdout.printf(' - %.2fx slower', r.ratio)
+          elsif i > 0
+            $stdout.printf(' - same-ish: difference falls within error')
+          end
+          $stdout.puts
+        end
+        $stdout.puts
+      end
+
+      private
+
+      def def_bench!
+        assigns = @inputs.each_index.map do |i|
+          "x#{i} = @inputs[#{i}]"
+        end.join(';')
+
+        yields = @inputs.each_with_index.map do |x, i|
+          dup = (@dup_inputs && x.respond_to?(:dup)) ? '.dup' : ''
+          "yield(x#{i}#{dup})"
+        end.join(';')
+
+        code = <<-CODE
+          def bench(reps)
+            #{assigns}
+            i = reps
+            before_time = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)
+            while i > 0
+              #{yields}
+              i -= 1
+            end
+            after_time = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)
+            after_time - before_time
+          end
+        CODE
+
+        instance_eval{ undef :bench } if self.respond_to?(:bench)
+        instance_eval(code)
+      end
+    end
+
+
+    class Report
+      attr_reader :label, :ratio
+
+      def initialize(label, invocs_per_sample)
+        @label = label.to_s
+        @invocs_per_sample = invocs_per_sample.to_f
+        @ratio = nil
+
+        @n = 0
+        @mean = 0.0
+        @m2 = 0.0
+      end
+
+      def add_sample(time_ns)
+        sample_ips = @invocs_per_sample * NS_PER_S / time_ns
+
+        # see https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm
+        # or Knuth's TAOCP vol 2, 3rd edition, page 232
+        @n += 1
+        delta = sample_ips - @mean
+        @mean += delta / @n
+        @m2 += delta * (sample_ips - @mean)
+        @stddev = nil
+      end
+
+      def ips
+        @mean
+      end
+
+      def stddev
+        @stddev ||= @n < 2 ? 0.0 : Math.sqrt(@m2 / (@n - 1))
+      end
+
+      def slower_than!(faster)
+        @ratio = overlap?(faster) ? nil : (faster.ips / self.ips)
+      end
+
+      def overlap?(faster)
+        (faster.ips - faster.stddev) <= (self.ips + self.stddev)
+      end
+    end
+
+  end
+
+end

data/lib/benchmark/inputs/version.rb

@@ -0,0 +1,5 @@
+module Benchmark
+  module Inputs
+    VERSION = "1.0.0"
+  end
+end

metadata

@@ -0,0 +1,110 @@
+--- !ruby/object:Gem::Specification
+name: benchmark-inputs
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Jonathan Hefner
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-07-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.12'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.12'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.8'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.8'
+description: 
+email:
+- jonathan.hefner@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- benchmark-inputs.gemspec
+- lib/benchmark/inputs.rb
+- lib/benchmark/inputs/version.rb
+homepage: https://github.com/jonathanhefner/benchmark-inputs
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.8
+signing_key: 
+specification_version: 4
+summary: Input-focused benchmarking
+test_files: []
+has_rdoc: