rspec-benchmark 0.3.0 → 0.4.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 9e2f1ed591f703a89dbf9a01070a9ba4cf893e82
4
- data.tar.gz: f837f60f9864f23ff94e88a0d48adafff16fcd13
2
+ SHA256:
3
+ metadata.gz: '090d1207708172bb738f301300654c15e9ae2e0a11acf7a256f26137042c97e4'
4
+ data.tar.gz: 0464e88827f44e6cab4c2e8e269fee7a45b2885713787c2f95b7443b867caf6f
5
5
  SHA512:
6
- metadata.gz: 63d13b9d25a62d5ad5e3afacb0e538b0d7b5c5bdc0f35488a375401a5e6ad86cc840dddb87b711bdf7f6a303c910d6786271fbe5e56a65ee4cbaa56947c04d56
7
- data.tar.gz: df7988107e1e39c7dbfdd2c560ddfba738d9972786671245f5273cd089c50070fabbac8cd2fa9a7f186d0990cd2f18b9a29f681224f27c7bc0ab4ee00024abb9
6
+ metadata.gz: 5680d4c898961eceb9718800012180918c8e3d1607e9d8a1a51ca2dfba93493abc5b4bc8fa40e6ae7648f4bd01eba6a5cc4aacaa10ab732e897a7a0566b85907
7
+ data.tar.gz: 30c9323c288c1103872607372c5d015cc4b290df030b64239a0af3e55f4f98925dbea4470178ea552b6684f70dfc15ae36bc801bb245f58e77031b2dbd99377b
@@ -1,5 +1,24 @@
1
1
  # Change log
2
2
 
3
+ ## [v0.4.0] - 2018-10-01
4
+
5
+ ### Added
6
+ * Add benchmark-trend as a dependency
7
+ * Add ComplexityMatcher with #perform_linear, #perform_constant,
8
+ #perform_logarithmic, #perform_power and #perform_exponential expectations
9
+ * Add #within and #warmup matchers to IterationMatcher
10
+ * Add #warmup matcher to TimingMatcher
11
+ * Add #within and #warmup matchers to ComparisonMatcher
12
+
13
+ ### Changed
14
+ * Change to require Ruby >= 2.0.0
15
+ * Change to update benchmark-perf dependency
16
+ * Change IterationMatcher to use new Benchmark::Perf::Iteration api
17
+ * Change TimingMatcher to use new Benchmark::Perf::ExecutionTime api
18
+ * Change ComparisonMatcher to use new Benchmark::Perf::Iteration api
19
+ * Change #and_sample matcher to #sample
20
+ * Change TimingMatcher to run only one sample by default
21
+
3
22
  ## [v0.3.0] - 2017-02-05
4
23
 
5
24
  ### Added
@@ -15,6 +34,7 @@
15
34
 
16
35
  Initial release
17
36
 
37
+ [v0.4.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.3.0...v0.4.0
18
38
  [v0.3.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.2.0...v0.3.0
19
39
  [v0.2.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.1.0...v0.2.0
20
40
  [v0.1.0]: https://github.com/peter-murach/rspec-benchmark/compare/v0.1.0
data/README.md CHANGED
@@ -1,34 +1,39 @@
1
1
  # RSpec::Benchmark
2
+
2
3
  [![Gem Version](https://badge.fury.io/rb/rspec-benchmark.svg)][gem]
3
4
  [![Build Status](https://secure.travis-ci.org/piotrmurach/rspec-benchmark.svg?branch=master)][travis]
5
+ [![Build status](https://ci.appveyor.com/api/projects/status/nxq3dr8xkafmgiv0?svg=true)][appveyor]
4
6
  [![Code Climate](https://codeclimate.com/github/piotrmurach/rspec-benchmark/badges/gpa.svg)][codeclimate]
5
7
  [![Coverage Status](https://coveralls.io/repos/github/piotrmurach/rspec-benchmark/badge.svg)][coverage]
6
8
  [![Inline docs](http://inch-ci.org/github/piotrmurach/rspec-benchmark.svg?branch=master)][inchpages]
7
9
 
8
10
  [gem]: http://badge.fury.io/rb/rspec-benchmark
9
11
  [travis]: http://travis-ci.org/piotrmurach/rspec-benchmark
12
+ [appveyor]: https://ci.appveyor.com/project/piotrmurach/rspec-benchmark
10
13
  [codeclimate]: https://codeclimate.com/github/piotrmurach/rspec-benchmark
11
14
  [coverage]: https://coveralls.io/github/piotrmurach/rspec-benchmark
12
15
  [inchpages]: http://inch-ci.org/github/piotrmurach/rspec-benchmark
13
16
 
14
17
  > Performance testing matchers for RSpec
15
18
 
16
- **RSpec::Benchmark** uses [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measurements.
19
+ **RSpec::Benchmark** uses [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measuring execution time and iterations per second and [benchmark-trend](https://github.com/piotrmurach/benchmark-trend) for asymptotic behaviour estimation.
17
20
 
18
21
  ## Why?
19
22
 
20
23
  Integration and unit tests ensure that changing code maintains expected functionality. What is not guaranteed is the code changes impact on library performance. It is easy to refactor your way out of fast to slow code.
21
24
 
22
- If you are new to performance testing you may find [Caveats](#3-caveats) section helpful.
25
+ If you are new to performance testing you may find [Caveats](#4-caveats) section helpful.
23
26
 
24
27
  ## Contents
25
28
 
26
29
  * [1. Usage](#1-usage)
27
- * [1.1 Execution Time](#11-execution-time)
30
+ * [1.1 Timing](#11-timing)
28
31
  * [1.2 Iterations ](#12-iterations)
29
32
  * [1.3 Comparison ](#13-comparison)
30
- * [2. Filtering](#2-filtering)
31
- * [3. Caveats](#3-caveats)
33
+ * [1.4 Complexity](#14-complexity)
34
+ * [2. Compounding](#2-compounding)
35
+ * [3. Filtering](#3-filtering)
36
+ * [4. Caveats](#4-caveats)
32
37
 
33
38
  ## Installation
34
39
 
@@ -58,7 +63,14 @@ RSpec.configure do |config|
58
63
  end
59
64
  ```
60
65
 
61
- This will add the `perform_under`, `perform_at_least`, `perform_faster_than` and `perform_slower_than` matchers to express expected performance benchmark for code executed inside the expectation.
66
+ This will add the following matchers:
67
+
68
+ * `perform_under` to see how fast your code runs
69
+ * `perform_at_least` to see how many iteration per second your code can do
70
+ * `perform_(faster|slower)_than` to compare implementations
71
+ * `perform_(constant|linear|logarithmic|power|exponential)` to see how your code scales with time
72
+
73
+ that will help you express expected performance benchmark for an evaluted code.
62
74
 
63
75
  Alternatively, you can add matchers for particular example:
64
76
 
@@ -68,7 +80,15 @@ RSpec.describe "Performance testing" do
68
80
  end
69
81
  ```
70
82
 
71
- ### 1.1 Execution time
83
+ Then you're good to start setting performance expectations:
84
+
85
+ ```ruby
86
+ expect {
87
+ ...
88
+ }.to perform_under(6).ms
89
+ ```
90
+
91
+ ### 1.1 Timing
72
92
 
73
93
  The `perform_under` matcher answers the question of how long does it take to perform a given block of code on average. The measurements are taken executing the block of code in a child process for accurent cpu times.
74
94
 
@@ -80,12 +100,27 @@ All measurements are assumed to be expressed as seconds. However, you can also p
80
100
 
81
101
  ```ruby
82
102
  expect { ... }.to perform_under(10).ms
103
+ expect { ... }.to perform_under(10000).us
104
+ ```
105
+
106
+ by default the above code will be sampled only once but you can change this by using the `sample` matcher like so:
107
+
108
+ ```ruby
109
+ expect { ... }.to perform_under(0.01).sample(10) # repeats measurements 10 times
110
+ ```
111
+
112
+ For extra expressiveness you can use `times`:
113
+
114
+ ```ruby
115
+ expect { ... }.to perform_under(0.01).sample(10).times
83
116
  ```
84
117
 
85
- by default the above code will be sampled `30` times but you can change this by using `and_sample` like so:
118
+ You can also use `warmup` matcher that can run your code before the actual samples are taken to reduce erratic execution times.
119
+
120
+ For example, you can execute code twice before you take 10 actual measurements:
86
121
 
87
122
  ```ruby
88
- expect { ... }.to perform_under(0.01).and_sample(10)
123
+ expect { ... }.to perform_under(0.01).sec.warmup(2).times.sample(10).times
89
124
  ```
90
125
 
91
126
  ### 1.2 Iterations
@@ -98,13 +133,15 @@ expect { ... }.to perform_at_least(10000).ips
98
133
 
99
134
  The `ips` part is optional but its usage clarifies the intent.
100
135
 
101
- The performance timining of this matcher can be tweaked using the `:time` and `:warmup` parameters. These are expressed as seconds. By default `:time` is set to `0.2` and `:warmup` to `0.1` respectively. To change parameters do:
136
+ The performance timining of this matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds.
137
+
138
+ By default `within` matcher is set to `0.2` second and `warmup` matcher to `0.1` respectively. To change how long measurements are taken, for example, to double the time amount do:
102
139
 
103
140
  ```ruby
104
- expect { ... }.to perform_faster_than(time: 0.4, warmup: 0.2) { ... }
141
+ expect { ... }.to perform_at_least(10000).within(0.4).warmup(0.2).ips
105
142
  ```
106
143
 
107
- The higher values for `:time` and `:warmup` the more accurate average readings and hence more stable tests at the cost of longer test suite overall time.
144
+ The higher values for `within` and `warmup` the more accurate average readings and more stable tests at the cost of longer test suite overall runtime.
108
145
 
109
146
  ### 1.3 Comparison
110
147
 
@@ -133,33 +170,93 @@ expect { ... }.to perform_slower_than { ... }.exactly(5).times
133
170
 
134
171
  The `times` part is also optional.
135
172
 
136
- The performance timining of each matcher can be tweaked using the `:time` and `:warmup` parameters. These are expressed as seconds. By default `:time` is set to `0.2` and `:warmup` to `0.1` respectively. To change parameters do:
173
+ The performance timining of each matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds. By default `within` matcher is set to `0.2` and `warmup` matcher to `0.1` second respectively. To change these matchers values do:
174
+
175
+ ```ruby
176
+ expect { ... }.to perform_faster_than.within(0.4).warmup(0.2) { ... }
177
+ ```
178
+
179
+ The higher values for `within` and `warmup` the more accurate average readings and more stable tests at the cost of longer test suite overall runtime.
180
+
181
+ ### 1.4 Complexity
182
+
183
+ The `perform_constant`, `perform_linear`, `perform_logarithmic`, `perform_power` and `perform_exponential` matchers are useful for estimating the asymptotic behaviour of a given block of code. The most basic way to use the expectations to test how your code scales is to use the matchers:
184
+
185
+ ```ruby
186
+ expect { ... }.to perform_constant
187
+ expect { ... }.to perform_linear
188
+ expect { ... }.to perform_logarithmic
189
+ expect { ... }.to perform_power
190
+ expect { ... }.to perform_exponential
191
+ ```
192
+
193
+ However, for the matchers to be of any use you will need to provide the range of inputs on which they will perform measurements using `in_range` matcher. Each range input together with its corresponding iteration index will be yielded as arguments to the evaluted block.
194
+
195
+ For example, to create a power range of inputs from `8` to `100_000` do:
137
196
 
138
197
  ```ruby
139
- expect { ... }.to perform_faster_than(time: 0.4, warmup: 0.2) { ... }
198
+ expect { |n, i|
199
+ ...
200
+ }.to perform_linear.in_range(8, 100_000)
140
201
  ```
141
202
 
142
- The higher values for `:time` and `:warmup` the more accurate average readings and hence more stable tests at the cost of longer test suite overall time.
203
+ This example will generate and yield input `n` and index `i` pairs `[8, 0]`, `[64, 1]`, `[512, 2]`, `[4K, 3]`, `[32K, 4]` and `[100K, 5]` respectively.
204
+
205
+ By default the range will be generated using ratio of 8. You can change this using `ratio` matcher:
206
+
207
+ ```ruby
208
+ expect { |n, i|
209
+ ...
210
+ }.to perform_linear.in_range(8, 100_000).ratio(2)
211
+ ```
212
+
213
+ The performance of code block is measured only once per range input. You can change this value and in doing so increase stability of your performance test using the `sample` matcher. For example, to repeat measurements 100 times for each range data input do:
214
+
215
+ ```ruby
216
+ expect { |n, i|
217
+ ...
218
+ }.to perform_linear.in_range(8, 100_000).ratio(2).sample(100).times
219
+ ```
143
220
 
144
- ## 2 Filtering
221
+ ## 2. Compounding
145
222
 
146
- Usually performance tests are best left for CI or occasional runs that do not affect TDD/BDD cycle. To achieve isolation you can use RSpec filters. For instance, in `spec_helper`:
223
+ All the matchers can be used in compound expressions via `and/or`. For example, if you wish to check if a computation performs under certain time boundry and iterates at least a given number do:
147
224
 
225
+ ```ruby
226
+ expect {
227
+ ...
228
+ }.to perform_under(6).ms and perform_at_least(10000).ips
148
229
  ```
149
- config.filter_run_excluding performance: true
230
+
231
+ ## 3. Filtering
232
+
233
+ Usually performance tests are best left for CI or occasional runs that do not affect TDD/BDD cycle.
234
+
235
+ To achieve isolation you can use RSpec filters to exclude performance tests from regular runs. For example, in `spec_helper`:
236
+
237
+ ```ruby
238
+ RSpec.config do |config|
239
+ config.filter_run_excluding perf: true
240
+ end
150
241
  ```
151
242
 
152
243
  and then in your example group do:
153
244
 
154
245
  ```ruby
155
- RSpec.describe ..., performance: true do
246
+ RSpec.describe ..., :perf do
156
247
  ...
157
248
  end
158
249
  ```
159
250
 
160
- Another option is to simply isolate the performance specs in separate directory suc as `spec/performance/...` and add custom rake task.
251
+ Then you can run groups or examples tagged with `perf`:
252
+
253
+ ```
254
+ rspec --tag perf
255
+ ```
256
+
257
+ Another option is to simply isolate the performance specs in separate directory such as `spec/performance/...` and add custom rake task to run them.
161
258
 
162
- ## 3 Caveats
259
+ ## 4. Caveats
163
260
 
164
261
  When writing performance tests things to be mindful are:
165
262
 
@@ -180,4 +277,4 @@ If you have any other observations please share them!
180
277
 
181
278
  ## Copyright
182
279
 
183
- Copyright (c) 2016-2017 Piotr Murach. See LICENSE for further details.
280
+ Copyright (c) 2016-2018 Piotr Murach. See LICENSE for further details.
@@ -1,7 +1,4 @@
1
- # encoding: utf-8
1
+ # frozen_string_literal: true
2
2
 
3
- require 'benchmark-perf'
4
-
5
- require 'rspec/benchmark/format_time'
6
- require 'rspec/benchmark/matchers'
7
- require 'rspec/benchmark/version'
3
+ require_relative 'benchmark/matchers'
4
+ require_relative 'benchmark/version'
@@ -1,4 +1,6 @@
1
- # encoding: utf-8
1
+ # frozen_string_literal: true
2
+
3
+ require 'benchmark-perf'
2
4
 
3
5
  module RSpec
4
6
  module Benchmark
@@ -11,11 +13,11 @@ module RSpec
11
13
  check_comparison(comparison_type)
12
14
  @expected = expected
13
15
  @comparison_type = comparison_type
14
- @count = 1
16
+ @count = 1
15
17
  @count_type = :at_least
16
- time = options.fetch(:time) { 0.2 }
17
- warmup = options.fetch(:warmup) { 0.1 }
18
- @bench = ::Benchmark::Perf::Iteration.new(time: time, warmup: warmup)
18
+ @time = options.fetch(:time) { 0.2 }
19
+ @warmup = options.fetch(:warmup) { 0.1 }
20
+ @bench = ::Benchmark::Perf::Iteration
19
21
  end
20
22
 
21
23
  # Indicates this matcher matches against a block
@@ -38,8 +40,8 @@ module RSpec
38
40
  @actual = block
39
41
  return false unless @actual.is_a?(Proc)
40
42
 
41
- @expected_ips, @expected_stdev, = @bench.run(&@expected)
42
- @actual_ips, @actual_stdev, = @bench.run(&@actual)
43
+ @expected_ips, @expected_stdev, = @bench.run(time: @time, warmup: @warmup, &@expected)
44
+ @actual_ips, @actual_stdev, = @bench.run(time: @time, warmup: @warmup, &@actual)
43
45
 
44
46
  @ratio = @actual_ips / @expected_ips.to_f
45
47
 
@@ -53,6 +55,28 @@ module RSpec
53
55
  end
54
56
  end
55
57
 
58
+ # The time before measurements are taken
59
+ #
60
+ # @param [Numeric] value
61
+ # the time before measurements are taken
62
+ #
63
+ # @api public
64
+ def warmup(value)
65
+ @warmup = value
66
+ self
67
+ end
68
+
69
+ # Time to measure iteration for
70
+ #
71
+ # @param [Numeric] value
72
+ # the time to take measurements for
73
+ #
74
+ # @api public
75
+ def within(value)
76
+ @time = value
77
+ self
78
+ end
79
+
56
80
  # Specify the minimum number of times a block
57
81
  # is faster/slower than other.
58
82
  # @api public
@@ -0,0 +1,89 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'benchmark-trend'
4
+
5
+ module RSpec
6
+ module Benchmark
7
+ module ComplexityMatcher
8
+ # Implements the `perform`
9
+ #
10
+ # @api public
11
+ class Matcher
12
+ def initialize(fit_type, **options)
13
+ @fit_type = fit_type
14
+ @threshold = options.fetch(:threshold) { 0.9 }
15
+ @repeat = options.fetch(:repeat) { 1 }
16
+ @start = 8
17
+ @limit = 8 << 10
18
+ @ratio = 8
19
+ end
20
+
21
+ # Indicates this matcher matches against a block
22
+ #
23
+ # @return [True]
24
+ #
25
+ # @api private
26
+ def supports_block_expectations?
27
+ true
28
+ end
29
+
30
+ def matcher_name
31
+ "perform_#{@fit_type}"
32
+ end
33
+
34
+ # @return [Boolean]
35
+ #
36
+ # @api private
37
+ def matches?(block)
38
+ range = ::Benchmark::Trend.range(@start, @limit, ratio: @ratio)
39
+ @trend, trends = ::Benchmark::Trend.infer_trend(range, repeat: @repeat, &block)
40
+
41
+ @trend == @fit_type
42
+ end
43
+
44
+ def in_range(start, limit)
45
+ @start, @limit = start, limit
46
+ self
47
+ end
48
+
49
+ def ratio(ratio)
50
+ @ratio = ratio
51
+ self
52
+ end
53
+
54
+ def sample(repeat)
55
+ @repeat = repeat
56
+ self
57
+ end
58
+
59
+ def actual
60
+ @trend
61
+ end
62
+
63
+ # No-op, syntactic sugar.
64
+ # @api public
65
+ def times
66
+ self
67
+ end
68
+
69
+ # @api private
70
+ def description
71
+ "perform #{@fit_type}"
72
+ end
73
+
74
+ # @api private
75
+ def failure_message
76
+ "expected block to #{description}, but #{failure_reason}"
77
+ end
78
+
79
+ def failure_message_when_negated
80
+ "expected block not to #{description}, but #{failure_reason}"
81
+ end
82
+
83
+ def failure_reason
84
+ "performed #{actual}"
85
+ end
86
+ end # Matcher
87
+ end # ComplexityMatcher
88
+ end # Benchmark
89
+ end # RSpec
@@ -1,4 +1,4 @@
1
- # encoding: utf-8
1
+ # frozen_string_literal: true
2
2
 
3
3
  module RSpec
4
4
  module Benchmark