rspec-benchmark 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '090d1207708172bb738f301300654c15e9ae2e0a11acf7a256f26137042c97e4'
-  data.tar.gz: 0464e88827f44e6cab4c2e8e269fee7a45b2885713787c2f95b7443b867caf6f
+  metadata.gz: 798bc5d4e03d43057fe769a612d69c7b9e94c1d6cf00cb119ff343f184c2dcc2
+  data.tar.gz: 437e723c5984d44d2af9e24dc2eab5a952d4be2531f86d1e5a67de6edfcd122c
 SHA512:
-  metadata.gz: 5680d4c898961eceb9718800012180918c8e3d1607e9d8a1a51ca2dfba93493abc5b4bc8fa40e6ae7648f4bd01eba6a5cc4aacaa10ab732e897a7a0566b85907
-  data.tar.gz: 30c9323c288c1103872607372c5d015cc4b290df030b64239a0af3e55f4f98925dbea4470178ea552b6684f70dfc15ae36bc801bb245f58e77031b2dbd99377b
+  metadata.gz: 48a31057edbf78fc8f535110e9a99ccc1bf7a3f7e2ce044eb99cfb3dd1322fe75ce00996acec8b5759b15e9622ef39b6b5cfa4fb678055dda0a8cfd943d1e1cf
+  data.tar.gz: 755534b8baf828aad86a8659bd59c123a2056a3f8b169699db0de5e19468476637f83c093fbd12a1db153c24a29039dacbecd648c67aba7df351b4f15906c492

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Change log
 
+## [v0.5.0] - 2019-04-21
+
+## Added
+* Add benchmark-malloc as a dependency
+* Add AllocationMatcher with  #perform_allocation expectation
+* Add #perform_log, #perform_exp aliases
+* Add threshold matcher for specifying allowed error level when asserting computational complexity
+* Add #configure to allow for global configuration of options
+* Add :run_in_subprocess, :disable_gc & :samples configuration options
+
+## Changed
+* Change to require Ruby >= 2.1.0
+* Change ComplexityMatcher to use threshold when verifying the assertion
+* Change ComplexityMatcher#in_range to accept full range as an input
+
 ## [v0.4.0] - 2018-10-01
 
 ### Added
@@ -34,6 +49,7 @@
 
 Initial release
 
+[v0.5.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.4.0...v0.5.0
 [v0.4.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.3.0...v0.4.0
 [v0.3.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.2.0...v0.3.0
 [v0.2.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.1.0...v0.2.0

data/README.md

@@ -14,15 +14,19 @@
 [coverage]: https://coveralls.io/github/piotrmurach/rspec-benchmark
 [inchpages]: http://inch-ci.org/github/piotrmurach/rspec-benchmark
 
-> Performance testing matchers for RSpec
+> Performance testing matchers for RSpec to set expectations on speed, resources usage and scalability.
 
-**RSpec::Benchmark** uses [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measuring execution time and iterations per second and [benchmark-trend](https://github.com/piotrmurach/benchmark-trend) for asymptotic behaviour estimation.
+**RSpec::Benchmark** is powered by:
+
+* [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measuring execution time and iterations per second.
+* [benchmark-trend](https://github.com/piotrmurach/benchmark-trend) for estimating computation complexity.
+* [benchmark-malloc](https://github.com/piotrmurach/benchmark-malloc) for measuring object and memory allocations.
 
 ## Why?
 
 Integration and unit tests ensure that changing code maintains expected functionality. What is not guaranteed is the code changes impact on library performance. It is easy to refactor your way out of fast to slow code.
 
-If you are new to performance testing you may find [Caveats](#4-caveats) section helpful.
+If you are new to performance testing you may find [Caveats](#5-caveats) section helpful.
 
 ## Contents
 
@@ -31,9 +35,14 @@ If you are new to performance testing you may find [Caveats](#4-caveats) section
   * [1.2 Iterations ](#12-iterations)
   * [1.3 Comparison ](#13-comparison)
   * [1.4 Complexity](#14-complexity)
+  * [1.5 Allocation](#15-allocation)
 * [2. Compounding](#2-compounding)
-* [3. Filtering](#3-filtering)
-* [4. Caveats](#4-caveats)
+* [3. Configuration](#3-configuration)
+  * [3.1 :disable_gc](#31-disable_gc)
+  * [3.2 :run_in_subprocess](#32-run_in_subprocess)
+  * [3.3 :samples](#33-samples)
+* [4. Filtering](#4-filtering)
+* [5. Caveats](#5-caveats)
 
 ## Installation
 
@@ -69,8 +78,9 @@ This will add the following matchers:
 * `perform_at_least` to see how many iteration per second your code can do
 * `perform_(faster|slower)_than` to compare implementations
 * `perform_(constant|linear|logarithmic|power|exponential)` to see how your code scales with time
+* `perform_allocation` to limit object and memory allocations
 
-that will help you express expected performance benchmark for an evaluted code.
+These will help you express expected performance benchmark for an evaluated code.
 
 Alternatively, you can add matchers for particular example:
 
@@ -90,7 +100,7 @@ expect {
 
 ### 1.1 Timing
 
-The `perform_under` matcher answers the question of how long does it take to perform a given block of code on average. The measurements are taken executing the block of code in a child process for accurent cpu times.
+The `perform_under` matcher answers the question of how long does it take to perform a given block of code on average. The measurements are taken executing the block of code in a child process for accurate CPU times.
 
 ```ruby
 expect { ... }.to perform_under(0.01).sec
@@ -103,7 +113,7 @@ expect { ... }.to perform_under(10).ms
 expect { ... }.to perform_under(10000).us
 ```
 
-by default the above code will be sampled only once but you can change this by using the `sample` matcher like so:
+By default the above code will be sampled only once but you can change this by using the `sample` matcher like so:
 
 ```ruby
 expect { ... }.to perform_under(0.01).sample(10) # repeats measurements 10 times
@@ -133,7 +143,7 @@ expect { ... }.to perform_at_least(10000).ips
 
 The `ips` part is optional but its usage clarifies the intent.
 
-The performance timining of this matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds.
+The performance timing of this matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds.
 
 By default `within` matcher is set to `0.2` second and `warmup` matcher to `0.1` respectively. To change how long measurements are taken, for example, to double the time amount do:
 
@@ -170,7 +180,7 @@ expect { ... }.to perform_slower_than { ... }.exactly(5).times
 
 The `times` part is also optional.
 
-The performance timining of each matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds. By default `within` matcher is set to `0.2` and `warmup` matcher to `0.1` second respectively. To change these matchers values do:
+The performance timing of each matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds. By default `within` matcher is set to `0.2` and `warmup` matcher to `0.1` second respectively. To change these matchers values do:
 
 ```ruby
 expect { ... }.to perform_faster_than.within(0.4).warmup(0.2) { ... }
@@ -180,47 +190,124 @@ The higher values for `within` and `warmup` the more accurate average readings a
 
 ### 1.4 Complexity
 
-The `perform_constant`, `perform_linear`, `perform_logarithmic`, `perform_power` and `perform_exponential` matchers are useful for estimating the asymptotic behaviour of a given block of code. The most basic way to use the expectations to test how your code scales is to use the matchers:
+The `perform_constant`, `perform_logarithmic`, `perform_linear`, `perform_power` and `perform_exponential` matchers are useful for estimating the asymptotic behaviour of a given block of code. The most basic way to use the expectations to test how your code scales is to use the matchers:
 
 ```ruby
 expect { ... }.to perform_constant
+expect { ... }.to perform_logarithmic/perform_log
 expect { ... }.to perform_linear
-expect { ... }.to perform_logarithmic
 expect { ... }.to perform_power
-expect { ... }.to perform_exponential
+expect { ... }.to perform_exponential/perform_exp
 ```
 
-However, for the matchers to be of any use you will need to provide the range of inputs on which they will perform measurements using `in_range` matcher. Each range input together with its corresponding iteration index will be yielded as arguments to the evaluted block.
+To test performance in terms of computation complexity you can follow the algorithm:
+
+1. Choose a method to profile.
+2. Choose workloads for the method.
+3. Describe workloads with input features.
+4. Assert the performance in terms of Big-O notation.
+
+Often, before expectation can be set you need to setup some workloads. To create a range of inputs use the `bench_range` helper method.
 
 For example, to create a power range of inputs from `8` to `100_000` do:
 
+```ruby
+sizes = bench_range(8, 100_000) # => [8, 64, 512, 4096, 32768, 100000]
+```
+
+Then you can use the sizes to create test data, for example to check Ruby's `max` performance create array of number arrays.
+
+```ruby
+number_arrays = sizes.map { |n| Array.new(n) { rand(n) } }
+```
+
+Using `in_range` matcher you can inform the expectation about the inputs. Each range value together with its index will be yielded as arguments to the evaluated block.
+
+You can either specify the range limits:
+
 ```ruby
 expect { |n, i|
-  ...
+  number_arrays[i].max
 }.to perform_linear.in_range(8, 100_000)
 ```
 
+Or use previously generated `sizes` array:
+
+```ruby
+expect { |n, i|
+  number_arrays[i].max
+}.to perform_linear.in_range(sizes)
+```
+
 This example will generate and yield input `n` and index `i` pairs `[8, 0]`, `[64, 1]`, `[512, 2]`, `[4K, 3]`, `[32K, 4]` and `[100K, 5]` respectively.
 
-By default the range will be generated using ratio of 8. You can change this using `ratio` matcher:
+By default the range will be generated using ratio of `8`. You can change this using `ratio` matcher:
 
 ```ruby
 expect { |n, i|
-  ...
+  number_arrays[i].max
 }.to perform_linear.in_range(8, 100_000).ratio(2)
 ```
 
-The performance of code block is measured only once per range input. You can change this value and in doing so increase stability of your performance test using the `sample` matcher. For example, to repeat measurements 100 times for each range data input do:
+The performance measurements for a code block are taken only once per range input. You can increase the stability of your performance test by using the `sample` matcher. For example, to repeat measurements 100 times for each range input do:
 
 ```ruby
 expect { |n, i|
-  ...
+  number_arrays[i].max
 }.to perform_linear.in_range(8, 100_000).ratio(2).sample(100).times
 ```
 
+The overall quality of the performance trend is assessed using a threshold value where `0` means a poor fit and `1` a perfect fit. By default this value is configured to `0.9` as a 'good enough' threshold. To change this use `threshold` matcher:
+
+```ruby
+expect { |n, i|
+  number_arrays[i].max
+}.to perform_linear.in_range(8, 100_000).threshold(0.95)
+```
+
+### 1.5 Allocation
+
+The `perform_allocation` matcher checks how much memory or objects have been allocated during a piece of Ruby code execution.
+
+By default the matcher verify the number of object allocations. You can also check for memory allocation using the `bytes` matcher.
+
+To check number of objects allocated do:
+
+```ruby
+expect {
+  ["foo", "bar", "baz"].sort[1]
+}.to perform_allocation(3)
+```
+
+You can also be more granular with your object allocations and specify which object types you're interested in:
+
+```ruby
+expect {
+  _a = [Object.new]
+  _b = {Object.new => 'foo'}
+}.to perform_allocation({Array => 1, Object => 2}).objects
+```
+
+And you can also check how many objects are left when expectation finishes to ensure that `GC` is able to collect them.
+
+```ruby
+expect {
+  ["foo", "bar", "baz"].sort[1]
+}.to perform_allocation(3).and_retain(3)
+```
+
+You can also set expectations on the memory size. In this case the memory size will serve as upper limit for the expectation:
+
+```ruby
+expect {
+  _a = [Object.new]
+  _b = {Object.new => 'foo'}
+}.to perform_allocation({Array => 40, Hash => 384, Object => 80}).bytes
+```
+
 ## 2. Compounding
 
-All the matchers can be used in compound expressions via `and/or`. For example, if you wish to check if a computation performs under certain time boundry and iterates at least a given number do:
+All the matchers can be used in compound expressions via `and/or`. For example, if you wish to check if a computation performs under certain time boundary and iterates at least a given number do:
 
 ```ruby
 expect {
@@ -228,7 +315,48 @@ expect {
 }.to perform_under(6).ms and perform_at_least(10000).ips
 ```
 
-## 3. Filtering
+## 3. Configuration
+
+By default the following configuration is used:
+
+```ruby
+RSpec::Benchmark.configure do |config|
+  config.run_in_subprocess = false
+  config.disable_gc = false
+end
+```
+
+### 3.1. `:disable_gc`
+
+By default all tests are run with `GC` enabled. We want to measure real performance or Ruby code. However, disabling `GC` may lead to much quicker test execution. You can change this setting:
+
+```ruby
+RSpec::Benchmark.configure do |config|
+  config.disable_gc = true
+end
+```
+
+### 3.2 `:run_in_subprocess`
+
+The `perform_under` matcher can run all the measurements in the subprocess. This will increase isolation from other processes activity. However, the `rspec-rails` gem runs all tests in transactions. Unfortunately, when running tests in child process, database connections are used from connection pool and no data can be accessed. This is only a problem when running specs in Rails. Any other Ruby project can run specs using subprocesses. To enable this behaviour do:
+
+```ruby
+RSpec::Benchmark.configure do |config|
+  config.run_in_subprocess = true
+end
+```
+
+### 3.3 `:samples`
+
+The `perform_under` and computational complexity matchers allow to specify how many times to repeat measurements. You configure it globally for all matchers using the `:samples` option which defaults to `1`:
+
+```ruby
+RSpec::Benchmark.configure do |config|
+  config.samples = 10
+end
+```
+
+## 4. Filtering
 
 Usually performance tests are best left for CI or occasional runs that do not affect TDD/BDD cycle.
 
@@ -240,7 +368,7 @@ RSpec.config do |config|
 end
 ```
 
-and then in your example group do:
+And then in your example group do:
 
 ```ruby
 RSpec.describe ..., :perf do
@@ -256,7 +384,7 @@ rspec --tag perf
 
 Another option is to simply isolate the performance specs in separate directory such as `spec/performance/...` and add custom rake task to run them.
 
-## 4. Caveats
+## 5. Caveats
 
 When writing performance tests things to be mindful are:
 
@@ -275,6 +403,10 @@ If you have any other observations please share them!
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create a new Pull Request
 
+## Code of Conduct
+
+Everyone interacting in the Strings project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/piotrmurach/rspec-benchmark/blob/master/CODE_OF_CONDUCT.md).
+
 ## Copyright
 
-Copyright (c) 2016-2018 Piotr Murach. See LICENSE for further details.
+Copyright (c) 2016 Piotr Murach. See LICENSE for further details.

data/lib/rspec-benchmark.rb

@@ -1,3 +1 @@
-# encoding: utf-8
-
-require 'rspec/benchmark'
+require_relative 'rspec/benchmark'

data/lib/rspec/benchmark.rb

@@ -1,4 +1,43 @@
 # frozen_string_literal: true
 
+require_relative 'benchmark/configuration'
 require_relative 'benchmark/matchers'
 require_relative 'benchmark/version'
+
+module RSpec
+  module Benchmark
+    class << self
+      attr_writer :configuration
+    end
+
+    # Current configuration
+    #
+    # @return [RSpec::Benchmark::Configuration]
+    #
+    # @api public
+    def self.configuration
+      @configuration ||= Configuration.new
+    end
+
+    # Reset current configuration to defaults
+    #
+    # @return [RSpec::Benchmark::Configuration]
+    #
+    # @api public
+    def self.reset_configuration
+      @configuration = Configuration.new
+    end
+
+    # Change current configuration
+    #
+    # @example
+    #   RSpec::Benchmark.configure do |config|
+    #     config.run_in_subprocess = false
+    #   end
+    #
+    # @api public
+    def self.configure
+      yield configuration
+    end
+  end
+end

data/lib/rspec/benchmark/allocation_matcher.rb

@@ -0,0 +1,164 @@
+# frozen_string_literal: true
+
+require 'benchmark-malloc'
+
+module RSpec
+  module Benchmark
+    module AllocationMatcher
+      # Implements the `perform_allocation` matcher
+      #
+      # @api private
+      class Matcher
+        def initialize(objects, **options)
+          @objects = objects
+          @retained_objects = nil
+          @warmup = options.fetch(:warmup) { 1 }
+          @bench  = ::Benchmark::Malloc
+          @count_type = :objects
+        end
+
+        # Indicates this matcher matches against a block
+        #
+        # @return [True]
+        #
+        # @api private
+        def supports_block_expectations?
+          true
+        end
+
+        # @return [Boolean]
+        #
+        # @api private
+        def matches?(block)
+          @block = block
+          alloc_stats = @bench.run(&block)
+          @actual = nil
+          @actual_retained = nil
+
+          case @objects
+          when Hash
+            case @count_type
+            when :memory
+              @actual = alloc_stats.allocated.count_memory
+            else
+              @actual = alloc_stats.allocated.count_objects
+            end
+            @objects.all? do |name, count|
+              @actual[name] <= count
+            end
+          when Numeric
+            case @count_type
+            when :memory
+              @actual = alloc_stats.allocated.total_memory
+              @actual_retained = alloc_stats.retained.total_memory
+            else
+              @actual = alloc_stats.allocated.total_objects
+              @actual_retained = alloc_stats.retained.total_objects
+            end
+            result = @actual <= @objects
+            result &= @actual_retained <= @retained_objects if @retained_objects
+            result
+          else
+            raise ArgumentError, "'#{@objects}' is not a recognized argument"
+          end
+        end
+
+        def and_retain(objects)
+          @retained_objects = objects
+          self
+        end
+
+        # The time before measurements are taken
+        #
+        # @param [Numeric] value
+        #   the time before measurements are taken
+        #
+        # @api public
+        def warmup(value)
+          @warmup = value
+          self
+        end
+
+        def objects
+          @count_type = :objects
+          self
+        end
+        alias object objects
+
+        def memory
+          @count_type = :memory
+          self
+        end
+        alias bytes memory
+
+        def failure_message
+          "expected block to #{description}, but #{positive_failure_reason}"
+        end
+
+        def failure_message_when_negated
+          "expected block not to #{description}, but #{negative_failure_reason}"
+        end
+
+        def description
+          desc = ["perform allocation of #{count_objects(@objects)}"]
+          if @retained_objects
+            desc << " and retain #{count_objects(@retained_objects)}"
+          end
+          desc.join
+        end
+
+        def count_objects(objects)
+          if @count_type == :memory
+            "#{objects_to_s(objects)} #{objects == 1 ? "byte" : "bytes"}"
+          else
+            "#{objects_to_s(objects)} #{pluralize_objects(objects)}"
+          end
+        end
+
+        def positive_failure_reason
+          return "was not a block" unless @block.is_a?(Proc)
+          "allocated #{actual}"
+        end
+
+        def negative_failure_reason
+          "allocated #{actual}"
+        end
+
+        def actual
+          if @count_type == :memory
+            "#{objects_to_s(@actual)} bytes"
+          else
+            desc = ["#{objects_to_s(@actual)} #{pluralize_objects(@actual)}"]
+            if @retained_objects
+              desc << " and retained #{objects_to_s(@actual_retained)}"
+            end
+            desc.join
+          end
+        end
+
+        def pluralize_objects(value)
+          if value.respond_to?(:to_hash)
+            if value.keys.size == 1 && value.values.reduce(&:+) == 1
+              "object"
+            else
+              "objects"
+            end
+          else
+            value == 1 ? "object" : "objects"
+          end
+        end
+
+        def objects_to_s(value)
+          if value.respond_to?(:to_hash)
+            value.
+              sort_by { |k,v| k.to_s }.
+              map { |key, val| "#{val} #{key}" if @objects.keys.include?(key) }.
+              compact.join(" and ")
+          else
+            value
+          end
+        end
+      end
+    end # AllocationMatcher
+  end # Benchmark
+end # RSpec

data/lib/rspec/benchmark/complexity_matcher.rb

@@ -11,8 +11,10 @@ module RSpec
       class Matcher
         def initialize(fit_type, **options)
           @fit_type  = fit_type
-          @threshold = options.fetch(:threshold) { 0.9 }
-          @repeat    = options.fetch(:repeat) { 1 }
+          @threshold = options.fetch(:threshold) {
+                        RSpec::Benchmark.configuration.fit_quality }
+          @repeat    = options.fetch(:repeat) {
+                        RSpec::Benchmark.configuration.samples }
           @start     = 8
           @limit     = 8 << 10
           @ratio     = 8
@@ -37,12 +39,30 @@ module RSpec
         def matches?(block)
           range = ::Benchmark::Trend.range(@start, @limit, ratio: @ratio)
           @trend, trends = ::Benchmark::Trend.infer_trend(range, repeat: @repeat, &block)
+          threshold = trends[@trend][:residual]
 
-          @trend == @fit_type
+          @trend == @fit_type && threshold >= @threshold
         end
 
-        def in_range(start, limit)
-          @start, @limit = start, limit
+        # Specify range of inputs
+        #
+        # @api public
+        def in_range(start, limit = (not_set = true))
+          case start
+          when Array
+            @start, *, @limit = *start
+            @ratio = start[1] / start[0]
+          when Numeric
+            @start, @limit = start, limit
+          else
+            raise ArgumentError,
+                "Wrong range argument '#{start}', it expects an array or numeric start value."
+          end
+          self
+        end
+
+        def threshold(threshold)
+          @threshold = threshold
           self
         end
 

data/lib/rspec/benchmark/configuration.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Benchmark
+    class Configuration
+      # Isolate benchmark time measurement in child process
+      # By default false due to Rails loosing DB connections
+      #
+      # @api public
+      attr_accessor :run_in_subprocess
+
+      # GC is enabled to measure real performance
+      #
+      # @api public
+      attr_accessor :disable_gc
+
+      # How many times to repeat measurements
+      #
+      # @return [Integer]
+      #
+      # @api public
+      attr_accessor :samples
+
+      # The fit quality in computational complexity
+      #
+      # @return [Float]
+      #
+      # @api public
+      attr_accessor :fit_quality
+
+      # @api private
+      def initialize
+        @disable_gc  = false
+        @samples     = 1
+        @fit_quality = 0.9
+        @run_in_subprocess = false
+      end
+    end # Configuration
+  end # Benchmark
+end # RSpec

data/lib/rspec/benchmark/matchers.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal
 
+require_relative 'allocation_matcher'
 require_relative 'comparison_matcher'
 require_relative 'complexity_matcher'
 require_relative 'iteration_matcher'
@@ -24,6 +25,19 @@ module RSpec
     #
     # @api public
     module Matchers
+      # Passes if code block performs at least iterations
+      #
+      # @param [Integer] iterations
+      #
+      # @example
+      #   expect { ... }.to perform_allocation(10000)
+      #   expect { ... }.to perform_allocation(10000)
+      #
+      # @api public
+      def perform_allocation(objects, **options)
+        AllocationMatcher::Matcher.new(objects, options)
+      end
+
       # Passes if code block performs at least iterations
       #
       # @param [Integer] iterations
@@ -101,6 +115,7 @@ module RSpec
       def perform_logarithmic(**options)
         ComplexityMatcher::Matcher.new(:logarithmic, options)
       end
+      alias perform_log perform_logarithmic
 
       # Pass if code block performs linear
       #
@@ -137,6 +152,7 @@ module RSpec
       def perform_exponential(**options)
         ComplexityMatcher::Matcher.new(:exponential, options)
       end
+      alias perform_exp perform_exponential
 
       # Generate a geometric progression of inputs
       #

data/lib/rspec/benchmark/timing_matcher.rb

@@ -17,8 +17,11 @@ module RSpec
 
         def initialize(threshold, **options)
           @threshold = threshold
-          @samples   = options.fetch(:samples) { 1 }
+          @samples   = options.fetch(:samples) {
+                         RSpec::Benchmark.configuration.samples }
           @warmup    = options.fetch(:warmup) { 1 }
+          @subprocess = options.fetch(:subprocess) {
+                          RSpec::Benchmark.configuration.run_in_subprocess }
           @scale     = threshold.to_s.split(/\./).last.size
           @block     = nil
           @bench     = ::Benchmark::Perf::ExecutionTime
@@ -39,7 +42,10 @@ module RSpec
         def matches?(block)
           @block = block
           return false unless block.is_a?(Proc)
-          @average, @stddev = @bench.run(repeat: @samples, warmup: @warmup, &block)
+          @average, @stddev = @bench.run(repeat: @samples,
+                                         warmup: @warmup,
+                                         subprocess: @subprocess,
+                                         &block)
           @average <= @threshold
         end
 

data/lib/rspec/benchmark/version.rb

@@ -2,6 +2,6 @@
 
 module RSpec
   module Benchmark
-    VERSION = "0.4.0"
+    VERSION = "0.5.0"
   end # Benchmark
 end # RSpec

data/rspec-benchmark.gemspec

@@ -6,12 +6,18 @@ Gem::Specification.new do |spec|
   spec.name          = "rspec-benchmark"
   spec.version       = RSpec::Benchmark::VERSION
   spec.authors       = ["Piotr Murach"]
-  spec.email         = [""]
+  spec.email         = ["me@piotrmurach.com"]
   spec.summary       = %q{Performance testing matchers for RSpec}
-  spec.description   = %q{Performance testing matchers for RSpec that provide simple way to specify speed and algorithmic complexity benchmark expectations}
-  spec.homepage      = ""
+  spec.description   = %q{Performance testing matchers for RSpec to set expectations on speed, resources usage and scalibility.}
+  spec.homepage      = "https://github.com/piotrmurach/rspec-benchmark"
   spec.license       = "MIT"
 
+  if spec.respond_to?(:metadata)
+    spec.metadata["homepage_uri"] = spec.homepage
+    spec.metadata["source_code_uri"] = "https://github.com/piotrmurach/rspec-benchmark"
+    spec.metadata["changelog_uri"] = "https://github.com/piotrmurach/rspec-benchmark/blob/master/CHANGELOG.md"
+  end
+
   spec.files         = Dir['{lib,spec}/**/*.rb']
   spec.files        += Dir['tasks/*', 'rspec-benchmark.gemspec']
   spec.files        += Dir['README.md', 'CHANGELOG.md', 'LICENSE.txt', 'Rakefile']
@@ -19,12 +25,13 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^spec/})
   spec.require_paths = ["lib"]
 
-  spec.required_ruby_version = '>= 2.0.0'
+  spec.required_ruby_version = '>= 2.1.0'
 
-  spec.add_dependency 'benchmark-perf', '~> 0.4.0'
-  spec.add_dependency 'benchmark-trend', '~> 0.2.0'
+  spec.add_dependency 'benchmark-malloc', '~> 0.1.0'
+  spec.add_dependency 'benchmark-perf',   '~> 0.5.0'
+  spec.add_dependency 'benchmark-trend',  '~> 0.3.0'
   spec.add_dependency 'rspec', '>= 3.0.0', '< 4.0.0'
 
-  spec.add_development_dependency 'bundler', '~> 1.16'
-  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'bundler', '>= 1.5'
+  spec.add_development_dependency 'rake'
 end

data/spec/unit/configuration_spec.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+RSpec.describe RSpec::Benchmark do
+  after(:example) do
+    RSpec::Benchmark.reset_configuration
+  end
+
+  it "defaults :run_in_subprocess option to false" do
+    config = RSpec::Benchmark.configuration
+
+    expect(config.run_in_subprocess).to eq(false)
+  end
+
+  it "defaults :disable_gc option to false" do
+    config = RSpec::Benchmark.configuration
+
+    expect(config.disable_gc).to eq(false)
+  end
+
+  it "defaults :samples option to 1" do
+    config = RSpec::Benchmark.configuration
+
+    expect(config.samples).to eq(1)
+  end
+
+  it "sets :run_in_subprocess option to true" do
+    RSpec::Benchmark.configure do |config|
+      config.run_in_subprocess = true
+    end
+
+    config = RSpec::Benchmark.configuration
+
+    expect(config.run_in_subprocess).to eq(true)
+  end
+
+  it "sets :disable_gc option to true" do
+    RSpec::Benchmark.configure do |config|
+      config.disable_gc = true
+    end
+
+    config = RSpec::Benchmark.configuration
+
+    expect(config.disable_gc).to eq(true)
+  end
+
+  it "sets :samples option to 10" do
+    RSpec::Benchmark.configure do |config|
+      config.samples = 10
+    end
+
+    config = RSpec::Benchmark.configuration
+
+    expect(config.samples).to eq(10)
+  end
+
+  it "uses the :run_in_subprocess option in timing matcher" do
+    RSpec::Benchmark.configure do |config|
+      config.run_in_subprocess = true
+      config.samples = 10
+    end
+
+    bench = [0.005, 0.00001]
+    allow(::Benchmark::Perf::ExecutionTime).to receive(:run).and_return(bench)
+
+    expect { 'x' * 1024 }.to perform_under(0.1)
+
+    expect(::Benchmark::Perf::ExecutionTime).to have_received(:run).with(
+      subprocess: true, warmup: 1, repeat: 10)
+  end
+
+  it "uses the :samples option in complexity matcher" do
+    RSpec::Benchmark.configure do |config|
+      config.samples = 10
+    end
+
+    bench = [:constant, {constant: {residual: 0.9}}]
+    allow(::Benchmark::Trend).to receive(:infer_trend).and_return(bench)
+
+    expect { 'x' * 1024 }.to perform_constant
+
+    expect(::Benchmark::Trend).to have_received(:infer_trend).with(
+      [8, 64, 512, 4096, 8192], repeat: 10)
+  end
+end

data/spec/unit/perform_allocation_spec.rb

@@ -0,0 +1,128 @@
+# frozen_string_literal: true
+
+RSpec.describe "#perform_allocation" do
+  context "expect { ... }.to perform_allocation(...)" do
+    it "passes if the block performs allocations" do
+      expect {
+        _a = [Object.new]
+      }.to perform_allocation(2)
+    end
+
+    it "fails if the block doesn't perform allocation(...)" do
+      expect {
+        expect {
+          _a = [Object.new]
+        }.to perform_allocation(1)
+      }.to raise_error(/expected block to perform allocation of \d object, but allocated \d objects/)
+    end
+  end
+
+  context "expect { ... }.not_to perform_allocation(...)" do
+    it "passes if the block does not perform allocation" do
+      expect {
+        ["foo", "bar", "baz"].sort[1]
+      }.to_not perform_allocation(2)
+    end
+
+    it "fails if the block performs allocation" do
+      expect {
+        expect {
+          _a = [Object.new]
+        }.to_not perform_allocation(2)
+      }.to raise_error(/expected block not to perform allocation of \d objects, but allocated \d objects/)
+    end
+  end
+
+  context "expect { ... }.to perform_allocation(Object => ???, ...).objects" do
+    it "splits object allocations by count" do
+      expect {
+        _a = [Object.new]
+        _b = {Object.new => 'foo'}
+      }.to perform_allocation({Object => 2, Array => 1}).objects
+    end
+
+    it "fails to split object allocations by count" do
+      expect {
+        expect {
+          _a = [Object.new]
+          _b = {Object.new => 'bar'}
+        }.to perform_allocation({Object => 1, Array => 1}).objects
+      }.to raise_error("expected block to perform allocation of 1 Array and 1 Object objects, but allocated 1 Array and 2 Object objects")
+    end
+  end
+
+  context "expect { ... }.not_to perform_allocation(Object => ???, ...).objects" do
+    it "passes if the split of object allocations by count is wrong" do
+      expect {
+        _a = [Object.new]
+        _b = {Object.new => 'foo'}
+      }.to_not perform_allocation({Object => 1, Array => 1}).objects
+    end
+
+    it "fails to split object allocations by count" do
+      expect {
+        expect {
+          _a = [Object.new]
+          _b = {Object.new => 'bar'}
+        }.to_not perform_allocation({Object => 2, Array => 1}).objects
+      }.to raise_error("expected block not to perform allocation of 1 Array and 2 Object objects, but allocated 1 Array and 2 Object objects")
+    end
+  end
+
+  context "expect { ... }.to perform_allocation(...).bytes" do
+    it "passes if the block performs allocations" do
+      expect {
+        _a = [Object.new]
+        _b = {Object.new => 'foo'}
+      }.to perform_allocation(600).bytes
+    end
+
+    it "fails if the block doesn't perform allocation" do
+      expect {
+        expect {
+          _a = {Object.new => 'foo'}
+        }.to perform_allocation(10).bytes
+      }.to raise_error(/expected block to perform allocation of \d+ bytes, but allocated \d+ bytes/)
+    end
+  end
+
+  context "expect { ... }.not_to perform_allocation(...).bytes" do
+    it "passes if the block does not perform allocation" do
+      expect {
+        _a = {Object.new => 'foo'}
+      }.to_not perform_allocation(10).bytes
+    end
+
+    it "fails if the block performs allocation" do
+      expect {
+        expect {
+          _a = [Object.new]
+        }.to_not perform_allocation(200).bytes
+      }.to raise_error(/expected block not to perform allocation of \d+ bytes, but allocated \d+ bytes/)
+    end
+  end
+
+  context "expect { ... }.to perform_allocation(Object => ???, ...).bytes" do
+    it "splits object allocations by memory size" do
+      expect {
+        _a = [Object.new]
+        _b = {Object.new => 'foo'}
+      }.to perform_allocation({Object => 80, Array => 40, Hash => 500}).bytes
+    end
+
+    it "fails to split object allocations by memory size" do
+      expect {
+        expect {
+          _a = [Object.new]
+          _b = {Object.new => 'bar'}
+        }.to perform_allocation({Object => 80, Hash => 0}).bytes
+      }.to raise_error(/expected block to perform allocation of \d+ Hash and \d+ Object bytes, but allocated \d+ Hash and \d+ Object bytes/)
+    end
+  end
+
+  it "fails to recognize expectation type" do
+    expect {
+      expect { Object.new }.to perform_allocation(:error)
+    }.to raise_error(ArgumentError, "'error' is not a recognized argument")
+  end
+end

data/spec/unit/perform_exponential_spec.rb

@@ -16,7 +16,7 @@ RSpec.describe 'RSpec::Benchmark::ComplexityMatcher', '#perform_exponential' do
     it "passes if the block performs exponential" do
       expect { |n, i|
         fibonacci(n)
-      }.to perform_exponential.in_range(1, 15).ratio(2).sample(100)
+      }.to perform_exp.in_range(1, 15).ratio(2).sample(100)
     end
 
     it "fails if the block doesn't perform exponential" do

data/spec/unit/perform_linear_spec.rb

@@ -14,8 +14,9 @@ RSpec.describe 'RSpec::Benchmark::ComplexityMatcher', '#perform_linear' do
 
   it "provides a default range" do
     range = [1,2,3]
+    trend = [:linear, {linear: {residual: 0.95}}]
     allow(::Benchmark::Trend).to receive(:range).and_return(range)
-    allow(::Benchmark::Trend).to receive(:infer_trend).and_return(:linear, {})
+    allow(::Benchmark::Trend).to receive(:infer_trend).and_return(trend)
 
     expect { |n, i| n }.to perform_linear
 
@@ -24,14 +25,21 @@ RSpec.describe 'RSpec::Benchmark::ComplexityMatcher', '#perform_linear' do
 
   it "changes default range using in_range and ratio matchers" do
     range = [1,2,3]
+    trend = [:linear, {linear: {residual: 0.95}}]
     allow(::Benchmark::Trend).to receive(:range).and_return(range)
-    allow(::Benchmark::Trend).to receive(:infer_trend).and_return(:linear, {})
+    allow(::Benchmark::Trend).to receive(:infer_trend).and_return(trend)
 
     expect { |n, i| n }.to perform_linear.in_range(3, 33_000).ratio(2)
 
     expect(::Benchmark::Trend).to have_received(:range).with(3, 33_000, ratio: 2)
   end
 
+  it "fails when wrong argument for range" do
+    expect {
+      expect { |n, i| }.to perform_linear.in_range(1..10)
+    }.to raise_error(ArgumentError, "Wrong range argument '1..10', it expects an array or numeric start value.")
+  end
+
   context "expect { ... }.to perfom_linear" do
     it "passes if the block performs linear" do
       range = bench_range(1, 8 << 10)
@@ -39,7 +47,7 @@ RSpec.describe 'RSpec::Benchmark::ComplexityMatcher', '#perform_linear' do
 
       expect { |n, i|
         numbers[i].max
-      }.to perform_linear.in_range(range[0], range[-1]).sample(100)
+      }.to perform_linear.in_range(range).sample(100).threshold(0.85)
     end
 
     it "fails if the block doesn't perform linear" do

data/spec/unit/perform_logarithmic_spec.rb

@@ -20,7 +20,7 @@ RSpec.describe 'RSpec::Benchmark::ComplexityMatcher', '#perform_logarithmic' do
 
       expect { |n, i|
         numbers[i].bsearch { |x| x == 1 }
-      }.to perform_logarithmic.in_range(range[0], range[-1]).ratio(2).sample(100).times
+      }.to perform_log.in_range(range[0], range[-1]).ratio(2).sample(100).times
     end
 
     it "fails if the block doesn't perform logarithmic" do
@@ -42,7 +42,7 @@ RSpec.describe 'RSpec::Benchmark::ComplexityMatcher', '#perform_logarithmic' do
       expect {
         expect { |n, i|
           numbers[i].bsearch { |x| x == 1 }
-        }.not_to perform_logarithmic.in_range(range[0], range[-1]).ratio(2).sample(100).times
+        }.not_to perform_log.in_range(range[0], range[-1]).ratio(2).sample(100).times
       }.to raise_error("expected block not to perform logarithmic, but performed logarithmic")
     end
   end

data/spec/unit/perform_under_spec.rb

@@ -10,8 +10,11 @@ RSpec.describe 'RSpec::Benchmark::TimingMatcher', '#perform_under' do
   it "allows to configure warmup cycles" do
     bench = [0.005, 0.00001]
     allow(::Benchmark::Perf::ExecutionTime).to receive(:run).and_return(bench)
+
     expect { 'x' * 1024 * 10 }.to perform_under(0.006).sec.warmup(2).times.sample(3)
-    expect(::Benchmark::Perf::ExecutionTime).to have_received(:run).with(repeat: 3, warmup: 2)
+
+    expect(::Benchmark::Perf::ExecutionTime).to have_received(:run).with(
+      subprocess: false, repeat: 3, warmup: 2)
   end
 
   it "doesn't allow sample size less than 1" do

metadata

@@ -1,43 +1,57 @@
 --- !ruby/object:Gem::Specification
 name: rspec-benchmark
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.0
 platform: ruby
 authors:
 - Piotr Murach
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-10-01 00:00:00.000000000 Z
+date: 2019-04-21 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: benchmark-malloc
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.1.0
 - !ruby/object:Gem::Dependency
   name: benchmark-perf
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.4.0
+        version: 0.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.4.0
+        version: 0.5.0
 - !ruby/object:Gem::Dependency
   name: benchmark-trend
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.2.0
+        version: 0.3.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.2.0
+        version: 0.3.0
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -62,34 +76,34 @@ dependencies:
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: '1.5'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: '1.5'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
-description: Performance testing matchers for RSpec that provide simple way to specify
-  speed and algorithmic complexity benchmark expectations
+        version: '0'
+description: Performance testing matchers for RSpec to set expectations on speed,
+  resources usage and scalibility.
 email:
-- ''
+- me@piotrmurach.com
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -100,8 +114,10 @@ files:
 - Rakefile
 - lib/rspec-benchmark.rb
 - lib/rspec/benchmark.rb
+- lib/rspec/benchmark/allocation_matcher.rb
 - lib/rspec/benchmark/comparison_matcher.rb
 - lib/rspec/benchmark/complexity_matcher.rb
+- lib/rspec/benchmark/configuration.rb
 - lib/rspec/benchmark/format_time.rb
 - lib/rspec/benchmark/iteration_matcher.rb
 - lib/rspec/benchmark/matchers.rb
@@ -111,7 +127,9 @@ files:
 - spec/spec_helper.rb
 - spec/unit/comparison_matcher_spec.rb
 - spec/unit/composable_spec.rb
+- spec/unit/configuration_spec.rb
 - spec/unit/format_time_spec.rb
+- spec/unit/perform_allocation_spec.rb
 - spec/unit/perform_at_least_spec.rb
 - spec/unit/perform_constant_spec.rb
 - spec/unit/perform_exponential_spec.rb
@@ -121,10 +139,13 @@ files:
 - spec/unit/perform_under_spec.rb
 - tasks/coverage.rake
 - tasks/spec.rake
-homepage: ''
+homepage: https://github.com/piotrmurach/rspec-benchmark
 licenses:
 - MIT
-metadata: {}
+metadata:
+  homepage_uri: https://github.com/piotrmurach/rspec-benchmark
+  source_code_uri: https://github.com/piotrmurach/rspec-benchmark
+  changelog_uri: https://github.com/piotrmurach/rspec-benchmark/blob/master/CHANGELOG.md
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -133,15 +154,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.0.0
+      version: 2.1.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.3
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: Performance testing matchers for RSpec
@@ -149,7 +169,9 @@ test_files:
 - spec/spec_helper.rb
 - spec/unit/comparison_matcher_spec.rb
 - spec/unit/composable_spec.rb
+- spec/unit/configuration_spec.rb
 - spec/unit/format_time_spec.rb
+- spec/unit/perform_allocation_spec.rb
 - spec/unit/perform_at_least_spec.rb
 - spec/unit/perform_constant_spec.rb
 - spec/unit/perform_exponential_spec.rb