rspec-benchmark 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 9e2f1ed591f703a89dbf9a01070a9ba4cf893e82
-  data.tar.gz: f837f60f9864f23ff94e88a0d48adafff16fcd13
+SHA256:
+  metadata.gz: '090d1207708172bb738f301300654c15e9ae2e0a11acf7a256f26137042c97e4'
+  data.tar.gz: 0464e88827f44e6cab4c2e8e269fee7a45b2885713787c2f95b7443b867caf6f
 SHA512:
-  metadata.gz: 63d13b9d25a62d5ad5e3afacb0e538b0d7b5c5bdc0f35488a375401a5e6ad86cc840dddb87b711bdf7f6a303c910d6786271fbe5e56a65ee4cbaa56947c04d56
-  data.tar.gz: df7988107e1e39c7dbfdd2c560ddfba738d9972786671245f5273cd089c50070fabbac8cd2fa9a7f186d0990cd2f18b9a29f681224f27c7bc0ab4ee00024abb9
+  metadata.gz: 5680d4c898961eceb9718800012180918c8e3d1607e9d8a1a51ca2dfba93493abc5b4bc8fa40e6ae7648f4bd01eba6a5cc4aacaa10ab732e897a7a0566b85907
+  data.tar.gz: 30c9323c288c1103872607372c5d015cc4b290df030b64239a0af3e55f4f98925dbea4470178ea552b6684f70dfc15ae36bc801bb245f58e77031b2dbd99377b

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Change log
 
+## [v0.4.0] - 2018-10-01
+
+### Added
+* Add benchmark-trend as a dependency
+* Add ComplexityMatcher with #perform_linear, #perform_constant,
+  #perform_logarithmic, #perform_power and #perform_exponential expectations
+* Add #within and #warmup matchers to IterationMatcher
+* Add #warmup matcher to TimingMatcher
+* Add #within and #warmup matchers to ComparisonMatcher
+
+### Changed
+* Change to require Ruby >= 2.0.0
+* Change to update benchmark-perf dependency
+* Change IterationMatcher to use new Benchmark::Perf::Iteration api
+* Change TimingMatcher to use new Benchmark::Perf::ExecutionTime api
+* Change ComparisonMatcher to use new Benchmark::Perf::Iteration api
+* Change #and_sample matcher to #sample
+* Change TimingMatcher to run only one sample by default
+
 ## [v0.3.0] - 2017-02-05
 
 ### Added
@@ -15,6 +34,7 @@
 
 Initial release
 
+[v0.4.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.3.0...v0.4.0
 [v0.3.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.2.0...v0.3.0
 [v0.2.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.1.0...v0.2.0
 [v0.1.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.1.0

data/README.md

@@ -1,34 +1,39 @@
 # RSpec::Benchmark
+
 [![Gem Version](https://badge.fury.io/rb/rspec-benchmark.svg)][gem]
 [![Build Status](https://secure.travis-ci.org/piotrmurach/rspec-benchmark.svg?branch=master)][travis]
+[![Build status](https://ci.appveyor.com/api/projects/status/nxq3dr8xkafmgiv0?svg=true)][appveyor]
 [![Code Climate](https://codeclimate.com/github/piotrmurach/rspec-benchmark/badges/gpa.svg)][codeclimate]
 [![Coverage Status](https://coveralls.io/repos/github/piotrmurach/rspec-benchmark/badge.svg)][coverage]
 [![Inline docs](http://inch-ci.org/github/piotrmurach/rspec-benchmark.svg?branch=master)][inchpages]
 
 [gem]: http://badge.fury.io/rb/rspec-benchmark
 [travis]: http://travis-ci.org/piotrmurach/rspec-benchmark
+[appveyor]: https://ci.appveyor.com/project/piotrmurach/rspec-benchmark
 [codeclimate]: https://codeclimate.com/github/piotrmurach/rspec-benchmark
 [coverage]: https://coveralls.io/github/piotrmurach/rspec-benchmark
 [inchpages]: http://inch-ci.org/github/piotrmurach/rspec-benchmark
 
 > Performance testing matchers for RSpec
 
-**RSpec::Benchmark** uses [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measurements.
+**RSpec::Benchmark** uses [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measuring execution time and iterations per second and [benchmark-trend](https://github.com/piotrmurach/benchmark-trend) for asymptotic behaviour estimation.
 
 ## Why?
 
 Integration and unit tests ensure that changing code maintains expected functionality. What is not guaranteed is the code changes impact on library performance. It is easy to refactor your way out of fast to slow code.
 
-If you are new to performance testing you may find [Caveats](#3-caveats) section helpful.
+If you are new to performance testing you may find [Caveats](#4-caveats) section helpful.
 
 ## Contents
 
 * [1. Usage](#1-usage)
-  * [1.1 Execution Time](#11-execution-time)
+  * [1.1 Timing](#11-timing)
   * [1.2 Iterations ](#12-iterations)
   * [1.3 Comparison ](#13-comparison)
-* [2. Filtering](#2-filtering)
-* [3. Caveats](#3-caveats)
+  * [1.4 Complexity](#14-complexity)
+* [2. Compounding](#2-compounding)
+* [3. Filtering](#3-filtering)
+* [4. Caveats](#4-caveats)
 
 ## Installation
 
@@ -58,7 +63,14 @@ RSpec.configure do |config|
 end
 ```
 
-This will add the `perform_under`, `perform_at_least`, `perform_faster_than` and `perform_slower_than` matchers to express expected performance benchmark for code executed inside the expectation.
+This will add the following matchers:
+
+* `perform_under` to see how fast your code runs
+* `perform_at_least` to see how many iteration per second your code can do
+* `perform_(faster|slower)_than` to compare implementations
+* `perform_(constant|linear|logarithmic|power|exponential)` to see how your code scales with time
+
+that will help you express expected performance benchmark for an evaluted code.
 
 Alternatively, you can add matchers for particular example:
 
@@ -68,7 +80,15 @@ RSpec.describe "Performance testing" do
 end
 ```
 
-### 1.1 Execution time
+Then you're good to start setting performance expectations:
+
+```ruby
+expect {
+  ...
+}.to perform_under(6).ms
+```
+
+### 1.1 Timing
 
 The `perform_under` matcher answers the question of how long does it take to perform a given block of code on average. The measurements are taken executing the block of code in a child process for accurent cpu times.
 
@@ -80,12 +100,27 @@ All measurements are assumed to be expressed as seconds. However, you can also p
 
 ```ruby
 expect { ... }.to perform_under(10).ms
+expect { ... }.to perform_under(10000).us
+```
+
+by default the above code will be sampled only once but you can change this by using the `sample` matcher like so:
+
+```ruby
+expect { ... }.to perform_under(0.01).sample(10) # repeats measurements 10 times
+```
+
+For extra expressiveness you can use `times`:
+
+```ruby
+expect { ... }.to perform_under(0.01).sample(10).times
 ```
 
-by default the above code will be sampled `30` times but you can change this by using `and_sample` like so:
+You can also use `warmup` matcher that can run your code before the actual samples are taken to reduce erratic execution times.
+
+For example, you can execute code twice before you take 10 actual measurements:
 
 ```ruby
-expect { ... }.to perform_under(0.01).and_sample(10)
+expect { ... }.to perform_under(0.01).sec.warmup(2).times.sample(10).times
 ```
 
 ### 1.2 Iterations
@@ -98,13 +133,15 @@ expect { ... }.to perform_at_least(10000).ips
 
 The `ips` part is optional but its usage clarifies the intent.
 
-The performance timining of this matcher can be tweaked using the `:time` and `:warmup` parameters. These are expressed as seconds. By default `:time` is set to `0.2` and `:warmup` to `0.1` respectively. To change parameters do:
+The performance timining of this matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds.
+
+By default `within` matcher is set to `0.2` second and `warmup` matcher to `0.1` respectively. To change how long measurements are taken, for example, to double the time amount do:
 
 ```ruby
-expect { ... }.to perform_faster_than(time: 0.4, warmup: 0.2) { ... }
+expect { ... }.to perform_at_least(10000).within(0.4).warmup(0.2).ips
 ```
 
-The higher values for `:time` and `:warmup` the more accurate average readings and hence more stable tests at the cost of longer test suite overall time.
+The higher values for `within` and `warmup` the more accurate average readings and more stable tests at the cost of longer test suite overall runtime.
 
 ### 1.3 Comparison
 
@@ -133,33 +170,93 @@ expect { ... }.to perform_slower_than { ... }.exactly(5).times
 
 The `times` part is also optional.
 
-The performance timining of each matcher can be tweaked using the `:time` and `:warmup` parameters. These are expressed as seconds. By default `:time` is set to `0.2` and `:warmup` to `0.1` respectively. To change parameters do:
+The performance timining of each matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds. By default `within` matcher is set to `0.2` and `warmup` matcher to `0.1` second respectively. To change these matchers values do:
+
+```ruby
+expect { ... }.to perform_faster_than.within(0.4).warmup(0.2) { ... }
+```
+
+The higher values for `within` and `warmup` the more accurate average readings and more stable tests at the cost of longer test suite overall runtime.
+
+### 1.4 Complexity
+
+The `perform_constant`, `perform_linear`, `perform_logarithmic`, `perform_power` and `perform_exponential` matchers are useful for estimating the asymptotic behaviour of a given block of code. The most basic way to use the expectations to test how your code scales is to use the matchers:
+
+```ruby
+expect { ... }.to perform_constant
+expect { ... }.to perform_linear
+expect { ... }.to perform_logarithmic
+expect { ... }.to perform_power
+expect { ... }.to perform_exponential
+```
+
+However, for the matchers to be of any use you will need to provide the range of inputs on which they will perform measurements using `in_range` matcher. Each range input together with its corresponding iteration index will be yielded as arguments to the evaluted block.
+
+For example, to create a power range of inputs from `8` to `100_000` do:
 
 ```ruby
-expect { ... }.to perform_faster_than(time: 0.4, warmup: 0.2) { ... }
+expect { |n, i|
+  ...
+}.to perform_linear.in_range(8, 100_000)
 ```
 
-The higher values for `:time` and `:warmup` the more accurate average readings and hence more stable tests at the cost of longer test suite overall time.
+This example will generate and yield input `n` and index `i` pairs `[8, 0]`, `[64, 1]`, `[512, 2]`, `[4K, 3]`, `[32K, 4]` and `[100K, 5]` respectively.
+
+By default the range will be generated using ratio of 8. You can change this using `ratio` matcher:
+
+```ruby
+expect { |n, i|
+  ...
+}.to perform_linear.in_range(8, 100_000).ratio(2)
+```
+
+The performance of code block is measured only once per range input. You can change this value and in doing so increase stability of your performance test using the `sample` matcher. For example, to repeat measurements 100 times for each range data input do:
+
+```ruby
+expect { |n, i|
+  ...
+}.to perform_linear.in_range(8, 100_000).ratio(2).sample(100).times
+```
 
-## 2 Filtering
+## 2. Compounding
 
-Usually performance tests are best left for CI or occasional runs that do not affect TDD/BDD cycle. To achieve isolation you can use RSpec filters. For instance, in `spec_helper`:
+All the matchers can be used in compound expressions via `and/or`. For example, if you wish to check if a computation performs under certain time boundry and iterates at least a given number do:
 
+```ruby
+expect {
+  ...
+}.to perform_under(6).ms and perform_at_least(10000).ips
 ```
-config.filter_run_excluding performance: true
+
+## 3. Filtering
+
+Usually performance tests are best left for CI or occasional runs that do not affect TDD/BDD cycle.
+
+To achieve isolation you can use RSpec filters to exclude performance tests from regular runs. For example, in `spec_helper`:
+
+```ruby
+RSpec.config do |config|
+  config.filter_run_excluding perf: true
+end
 ```
 
 and then in your example group do:
 
 ```ruby
-RSpec.describe ..., performance: true do
+RSpec.describe ..., :perf do
   ...
 end
 ```
 
-Another option is to simply isolate the performance specs in separate directory suc as `spec/performance/...` and add custom rake task.
+Then you can run groups or examples tagged with `perf`:
+
+```
+rspec --tag perf
+```
+
+Another option is to simply isolate the performance specs in separate directory such as `spec/performance/...` and add custom rake task to run them.
 
-## 3 Caveats
+## 4. Caveats
 
 When writing performance tests things to be mindful are:
 
@@ -180,4 +277,4 @@ If you have any other observations please share them!
 
 ## Copyright
 
-Copyright (c) 2016-2017 Piotr Murach. See LICENSE for further details.
+Copyright (c) 2016-2018 Piotr Murach. See LICENSE for further details.

data/lib/rspec/benchmark.rb

@@ -1,7 +1,4 @@
-# encoding: utf-8
+# frozen_string_literal: true
 
-require 'benchmark-perf'
-
-require 'rspec/benchmark/format_time'
-require 'rspec/benchmark/matchers'
-require 'rspec/benchmark/version'
+require_relative 'benchmark/matchers'
+require_relative 'benchmark/version'

data/lib/rspec/benchmark/comparison_matcher.rb

@@ -1,4 +1,6 @@
-# encoding: utf-8
+# frozen_string_literal: true
+
+require 'benchmark-perf'
 
 module RSpec
   module Benchmark
@@ -11,11 +13,11 @@ module RSpec
           check_comparison(comparison_type)
           @expected = expected
           @comparison_type = comparison_type
-          @count = 1
+          @count      = 1
           @count_type = :at_least
-          time   = options.fetch(:time) { 0.2 }
-          warmup = options.fetch(:warmup) { 0.1 }
-          @bench = ::Benchmark::Perf::Iteration.new(time: time, warmup: warmup)
+          @time       = options.fetch(:time) { 0.2 }
+          @warmup     = options.fetch(:warmup) { 0.1 }
+          @bench      = ::Benchmark::Perf::Iteration
         end
 
         # Indicates this matcher matches against a block
@@ -38,8 +40,8 @@ module RSpec
           @actual = block
           return false unless @actual.is_a?(Proc)
 
-          @expected_ips, @expected_stdev, = @bench.run(&@expected)
-          @actual_ips, @actual_stdev, = @bench.run(&@actual)
+          @expected_ips, @expected_stdev, = @bench.run(time: @time, warmup: @warmup, &@expected)
+          @actual_ips, @actual_stdev, = @bench.run(time: @time, warmup: @warmup, &@actual)
 
           @ratio = @actual_ips / @expected_ips.to_f
 
@@ -53,6 +55,28 @@ module RSpec
           end
         end
 
+        # The time before measurements are taken
+        #
+        # @param [Numeric] value
+        #   the time before measurements are taken
+        #
+        # @api public
+        def warmup(value)
+          @warmup = value
+          self
+        end
+
+        # Time to measure iteration for
+        #
+        # @param [Numeric] value
+        #   the time to take measurements for
+        #
+        # @api public
+        def within(value)
+          @time = value
+          self
+        end
+
         # Specify the minimum number of times a block
         # is faster/slower than other.
         # @api public

data/lib/rspec/benchmark/complexity_matcher.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+require 'benchmark-trend'
+
+module RSpec
+  module Benchmark
+    module ComplexityMatcher
+      # Implements the `perform`
+      #
+      # @api public
+      class Matcher
+        def initialize(fit_type, **options)
+          @fit_type  = fit_type
+          @threshold = options.fetch(:threshold) { 0.9 }
+          @repeat    = options.fetch(:repeat) { 1 }
+          @start     = 8
+          @limit     = 8 << 10
+          @ratio     = 8
+        end
+
+        # Indicates this matcher matches against a block
+        #
+        # @return [True]
+        #
+        # @api private
+        def supports_block_expectations?
+          true
+        end
+
+        def matcher_name
+          "perform_#{@fit_type}"
+        end
+
+        # @return [Boolean]
+        #
+        # @api private
+        def matches?(block)
+          range = ::Benchmark::Trend.range(@start, @limit, ratio: @ratio)
+          @trend, trends = ::Benchmark::Trend.infer_trend(range, repeat: @repeat, &block)
+
+          @trend == @fit_type
+        end
+
+        def in_range(start, limit)
+          @start, @limit = start, limit
+          self
+        end
+
+        def ratio(ratio)
+          @ratio = ratio
+          self
+        end
+
+        def sample(repeat)
+          @repeat = repeat
+          self
+        end
+
+        def actual
+          @trend
+        end
+
+        # No-op, syntactic sugar.
+        # @api public
+        def times
+          self
+        end
+
+        # @api private
+        def description
+          "perform #{@fit_type}"
+        end
+
+        # @api private
+        def failure_message
+          "expected block to #{description}, but #{failure_reason}"
+        end
+
+        def failure_message_when_negated
+          "expected block not to #{description}, but #{failure_reason}"
+        end
+
+        def failure_reason
+          "performed #{actual}"
+        end
+      end # Matcher
+    end # ComplexityMatcher
+  end # Benchmark
+end # RSpec

data/lib/rspec/benchmark/format_time.rb

@@ -1,4 +1,4 @@
-# encoding: utf-8
+# frozen_string_literal: true
 
 module RSpec
   module Benchmark