simplecov 0.17.1 → 0.21.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +100 -434
- data/README.md +375 -93
- data/doc/alternate-formatters.md +16 -1
- data/doc/commercial-services.md +5 -0
- data/lib/minitest/simplecov_plugin.rb +15 -0
- data/lib/simplecov.rb +294 -128
- data/lib/simplecov/combine.rb +30 -0
- data/lib/simplecov/combine/branches_combiner.rb +32 -0
- data/lib/simplecov/combine/files_combiner.rb +24 -0
- data/lib/simplecov/combine/lines_combiner.rb +43 -0
- data/lib/simplecov/combine/results_combiner.rb +60 -0
- data/lib/simplecov/command_guesser.rb +6 -3
- data/lib/simplecov/configuration.rb +191 -15
- data/lib/simplecov/coverage_statistics.rb +56 -0
- data/lib/simplecov/default_formatter.rb +20 -0
- data/lib/simplecov/defaults.rb +14 -13
- data/lib/simplecov/exit_codes.rb +5 -0
- data/lib/simplecov/exit_codes/exit_code_handling.rb +29 -0
- data/lib/simplecov/exit_codes/maximum_coverage_drop_check.rb +73 -0
- data/lib/simplecov/exit_codes/minimum_coverage_by_file_check.rb +54 -0
- data/lib/simplecov/exit_codes/minimum_overall_coverage_check.rb +53 -0
- data/lib/simplecov/file_list.rb +72 -13
- data/lib/simplecov/filter.rb +9 -6
- data/lib/simplecov/formatter.rb +2 -2
- data/lib/simplecov/formatter/multi_formatter.rb +5 -7
- data/lib/simplecov/formatter/simple_formatter.rb +4 -4
- data/lib/simplecov/last_run.rb +3 -1
- data/lib/simplecov/lines_classifier.rb +5 -5
- data/lib/simplecov/no_defaults.rb +1 -1
- data/lib/simplecov/process.rb +19 -0
- data/lib/simplecov/profiles.rb +9 -7
- data/lib/simplecov/result.rb +18 -12
- data/lib/simplecov/result_adapter.rb +30 -0
- data/lib/simplecov/result_merger.rb +130 -59
- data/lib/simplecov/simulate_coverage.rb +29 -0
- data/lib/simplecov/source_file.rb +272 -126
- data/lib/simplecov/source_file/branch.rb +84 -0
- data/lib/simplecov/source_file/line.rb +72 -0
- data/lib/simplecov/useless_results_remover.rb +18 -0
- data/lib/simplecov/version.rb +1 -1
- metadata +44 -158
- data/CONTRIBUTING.md +0 -51
- data/ISSUE_TEMPLATE.md +0 -23
- data/lib/simplecov/jruby_fix.rb +0 -44
- data/lib/simplecov/railtie.rb +0 -9
- data/lib/simplecov/railties/tasks.rake +0 -13
- data/lib/simplecov/raw_coverage.rb +0 -41
data/README.md
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
|
-
SimpleCov [](https://badge.fury.io/rb/simplecov) [][Continuous Integration] [](https://codeclimate.com/github/simplecov-ruby/simplecov/maintainability) [](http://inch-ci.org/github/simplecov-ruby/simplecov)
|
|
2
2
|
=========
|
|
3
|
+
|
|
3
4
|
**Code coverage for Ruby**
|
|
4
5
|
|
|
5
6
|
* [Source Code]
|
|
@@ -8,39 +9,42 @@ SimpleCov [][Conti
|
|
|
8
9
|
* [Rubygem]
|
|
9
10
|
* [Continuous Integration]
|
|
10
11
|
|
|
11
|
-
[Coverage]:
|
|
12
|
-
[Source Code]: https://github.com/
|
|
12
|
+
[Coverage]: https://ruby-doc.org/stdlib/libdoc/coverage/rdoc/Coverage.html "API doc for Ruby's Coverage library"
|
|
13
|
+
[Source Code]: https://github.com/simplecov-ruby/simplecov "Source Code @ GitHub"
|
|
13
14
|
[API documentation]: http://rubydoc.info/gems/simplecov/frames "RDoc API Documentation at Rubydoc.info"
|
|
14
15
|
[Configuration]: http://rubydoc.info/gems/simplecov/SimpleCov/Configuration "Configuration options API documentation"
|
|
15
|
-
[Changelog]: https://github.com/
|
|
16
|
+
[Changelog]: https://github.com/simplecov-ruby/simplecov/blob/main/CHANGELOG.md "Project Changelog"
|
|
16
17
|
[Rubygem]: http://rubygems.org/gems/simplecov "SimpleCov @ rubygems.org"
|
|
17
|
-
[Continuous Integration]:
|
|
18
|
-
[Dependencies]: https://gemnasium.com/
|
|
19
|
-
[simplecov-html]: https://github.com/
|
|
18
|
+
[Continuous Integration]: https://github.com/simplecov-ruby/simplecov/actions?query=workflow%3Astable "SimpleCov is built around the clock by github.com"
|
|
19
|
+
[Dependencies]: https://gemnasium.com/simplecov-ruby/simplecov "SimpleCov dependencies on Gemnasium"
|
|
20
|
+
[simplecov-html]: https://github.com/simplecov-ruby/simplecov-html "SimpleCov HTML Formatter Source Code @ GitHub"
|
|
20
21
|
|
|
21
22
|
SimpleCov is a code coverage analysis tool for Ruby. It uses [Ruby's built-in Coverage][Coverage] library to gather code
|
|
22
23
|
coverage data, but makes processing its results much easier by providing a clean API to filter, group, merge, format,
|
|
23
24
|
and display those results, giving you a complete code coverage suite that can be set up with just a couple lines of
|
|
24
25
|
code.
|
|
26
|
+
SimpleCov/Coverage track covered ruby code, gathering coverage for common templating solutions like erb, slim and haml is not supported.
|
|
25
27
|
|
|
26
28
|
In most cases, you'll want overall coverage results for your projects, including all types of tests, Cucumber features,
|
|
27
29
|
etc. SimpleCov automatically takes care of this by caching and merging results when generating reports, so your
|
|
28
30
|
report actually includes coverage across your test suites and thereby gives you a better picture of blank spots.
|
|
29
31
|
|
|
30
|
-
The official formatter of SimpleCov is packaged as a separate gem called [simplecov-html], but will be installed and
|
|
31
|
-
automatically when you launch SimpleCov. If you're curious, you can find it [on GitHub, too][simplecov-html].
|
|
32
|
+
The official formatter of SimpleCov is packaged as a separate gem called [simplecov-html], but will be installed and
|
|
33
|
+
configured automatically when you launch SimpleCov. If you're curious, you can find it [on GitHub, too][simplecov-html].
|
|
32
34
|
|
|
33
35
|
|
|
34
36
|
## Contact
|
|
35
37
|
|
|
36
38
|
*Code and Bug Reports*
|
|
37
39
|
|
|
38
|
-
* [Issue Tracker](https://github.com/
|
|
39
|
-
* See [CONTRIBUTING](https://github.com/
|
|
40
|
+
* [Issue Tracker](https://github.com/simplecov-ruby/simplecov/issues)
|
|
41
|
+
* See [CONTRIBUTING](https://github.com/simplecov-ruby/simplecov/blob/main/CONTRIBUTING.md) for how to contribute along
|
|
42
|
+
with some common problems to check out before creating an issue.
|
|
40
43
|
|
|
41
44
|
*Questions, Problems, Suggestions, etc.*
|
|
42
45
|
|
|
43
|
-
* [Mailing List](https://groups.google.com/forum/#!forum/simplecov) "Open mailing list for discussion and announcements
|
|
46
|
+
* [Mailing List](https://groups.google.com/forum/#!forum/simplecov) "Open mailing list for discussion and announcements
|
|
47
|
+
on Google Groups"
|
|
44
48
|
|
|
45
49
|
Getting started
|
|
46
50
|
---------------
|
|
@@ -69,9 +73,10 @@ Getting started
|
|
|
69
73
|
analysis to happen on. When testing a server process (e.g. a JSON API
|
|
70
74
|
endpoint) via a separate test process (e.g. when using Selenium) where you
|
|
71
75
|
want to see all code executed by the `rails server`, and not just code
|
|
72
|
-
executed in your actual test files, you
|
|
73
|
-
|
|
74
|
-
"shebang" line (`#! /usr/bin/env
|
|
76
|
+
executed in your actual test files, you need to require SimpleCov in the
|
|
77
|
+
server process. For rails for instance, you'll want to add something like this
|
|
78
|
+
to the top of `bin/rails`, but below the "shebang" line (`#! /usr/bin/env
|
|
79
|
+
ruby`) and after config/boot is required:
|
|
75
80
|
|
|
76
81
|
```ruby
|
|
77
82
|
if ENV['RAILS_ENV'] == 'test'
|
|
@@ -81,27 +86,41 @@ Getting started
|
|
|
81
86
|
end
|
|
82
87
|
```
|
|
83
88
|
|
|
84
|
-
3. Run your
|
|
85
|
-
|
|
86
|
-
|
|
89
|
+
3. Run your full test suite to see the percent coverage that your application has.
|
|
90
|
+
4. After running your tests, open `coverage/index.html` in the browser of your choice. For example, in a Mac Terminal,
|
|
91
|
+
run the following command from your application's root directory:
|
|
92
|
+
|
|
93
|
+
```
|
|
94
|
+
open coverage/index.html
|
|
95
|
+
```
|
|
96
|
+
in a debian/ubuntu Terminal,
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
xdg-open coverage/index.html
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
**Note:** [This guide](https://dwheeler.com/essays/open-files-urls.html) can help if you're unsure which command your particular
|
|
103
|
+
operating system requires.
|
|
104
|
+
|
|
105
|
+
5. Add the following to your `.gitignore` file to ensure that coverage results
|
|
87
106
|
are not tracked by Git (optional):
|
|
88
107
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
108
|
+
```
|
|
109
|
+
echo "coverage" >> .gitignore
|
|
110
|
+
```
|
|
111
|
+
Or if you use Windows:
|
|
112
|
+
```
|
|
113
|
+
echo coverage >> .gitignore
|
|
114
|
+
```
|
|
96
115
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
116
|
+
If you're making a Rails application, SimpleCov comes with built-in configurations (see below for information on
|
|
117
|
+
profiles) that will get you started with groups for your Controllers, Models and Helpers. To use it, the
|
|
118
|
+
first two lines of your test_helper should be like this:
|
|
100
119
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
120
|
+
```ruby
|
|
121
|
+
require 'simplecov'
|
|
122
|
+
SimpleCov.start 'rails'
|
|
123
|
+
```
|
|
105
124
|
|
|
106
125
|
## Example output
|
|
107
126
|
|
|
@@ -127,8 +146,9 @@ require 'simplecov'
|
|
|
127
146
|
SimpleCov.start 'rails'
|
|
128
147
|
```
|
|
129
148
|
|
|
130
|
-
You could even track what kind of code your UI testers are touching if you want to go overboard with things. SimpleCov
|
|
131
|
-
care what kind of framework it is running in; it just looks at what code is being executed and generates a
|
|
149
|
+
You could even track what kind of code your UI testers are touching if you want to go overboard with things. SimpleCov
|
|
150
|
+
does not care what kind of framework it is running in; it just looks at what code is being executed and generates a
|
|
151
|
+
report about it.
|
|
132
152
|
|
|
133
153
|
### Notes on specific frameworks and test utilities
|
|
134
154
|
|
|
@@ -148,8 +168,19 @@ to use SimpleCov with them. Here's an overview of the known ones:
|
|
|
148
168
|
race conditions occur when results are merged.
|
|
149
169
|
</td>
|
|
150
170
|
<td>
|
|
151
|
-
<a href="https://github.com/
|
|
152
|
-
<a href="https://github.com/
|
|
171
|
+
<a href="https://github.com/simplecov-ruby/simplecov/issues/64">#64</a> &
|
|
172
|
+
<a href="https://github.com/simplecov-ruby/simplecov/pull/185">#185</a>
|
|
173
|
+
</td>
|
|
174
|
+
</tr>
|
|
175
|
+
<tr>
|
|
176
|
+
<th>
|
|
177
|
+
knapsack_pro
|
|
178
|
+
</th>
|
|
179
|
+
<td>
|
|
180
|
+
To make SimpleCov work with Knapsack Pro Queue Mode to split tests in parallel on CI jobs you need to provide CI node index number to the <code>SimpleCov.command_name</code> in <code>KnapsackPro::Hooks::Queue.before_queue</code> hook.
|
|
181
|
+
</td>
|
|
182
|
+
<td>
|
|
183
|
+
<a href="https://knapsackpro.com/faq/question/how-to-use-simplecov-in-queue-mode">Tip</a>
|
|
153
184
|
</td>
|
|
154
185
|
</tr>
|
|
155
186
|
<tr>
|
|
@@ -162,7 +193,7 @@ to use SimpleCov with them. Here's an overview of the known ones:
|
|
|
162
193
|
to explicitly set the output root using `SimpleCov.root('foo/bar/baz')`
|
|
163
194
|
</td>
|
|
164
195
|
<td>
|
|
165
|
-
<a href="https://github.com/
|
|
196
|
+
<a href="https://github.com/simplecov-ruby/simplecov/issues/95">#95</a>
|
|
166
197
|
</td>
|
|
167
198
|
</tr>
|
|
168
199
|
<tr>
|
|
@@ -173,11 +204,11 @@ to use SimpleCov with them. Here's an overview of the known ones:
|
|
|
173
204
|
Because of how Spork works internally (using preforking), there used to
|
|
174
205
|
be trouble when using SimpleCov with it, but that has apparently been
|
|
175
206
|
resolved with a specific configuration strategy. See <a
|
|
176
|
-
href="https://github.com/
|
|
207
|
+
href="https://github.com/simplecov-ruby/simplecov/issues/42#issuecomment-4440284">this</a>
|
|
177
208
|
comment.
|
|
178
209
|
</td>
|
|
179
210
|
<td>
|
|
180
|
-
<a href="https://github.com/
|
|
211
|
+
<a href="https://github.com/simplecov-ruby/simplecov/issues/42#issuecomment-4440284">#42</a>
|
|
181
212
|
</td>
|
|
182
213
|
</tr>
|
|
183
214
|
<tr>
|
|
@@ -188,7 +219,7 @@ to use SimpleCov with them. Here's an overview of the known ones:
|
|
|
188
219
|
<a href="#want-to-use-spring-with-simplecov">See section below.</a>
|
|
189
220
|
</td>
|
|
190
221
|
<td>
|
|
191
|
-
<a href="https://github.com/
|
|
222
|
+
<a href="https://github.com/simplecov-ruby/simplecov/issues/381">#381</a>
|
|
192
223
|
</td>
|
|
193
224
|
</tr>
|
|
194
225
|
<tr>
|
|
@@ -201,7 +232,7 @@ to use SimpleCov with them. Here's an overview of the known ones:
|
|
|
201
232
|
(Dec 11th, 2011) should have this problem resolved.
|
|
202
233
|
</td>
|
|
203
234
|
<td>
|
|
204
|
-
<a href="https://github.com/
|
|
235
|
+
<a href="https://github.com/simplecov-ruby/simplecov/issues/45">#45</a> &
|
|
205
236
|
<a href="https://github.com/test-unit/test-unit/pull/12">test-unit/test-unit#12</a>
|
|
206
237
|
</td>
|
|
207
238
|
</tr>
|
|
@@ -223,7 +254,8 @@ to use SimpleCov with them. Here's an overview of the known ones:
|
|
|
223
254
|
```ruby
|
|
224
255
|
SimpleCov.some_config_option 'foo'
|
|
225
256
|
```
|
|
226
|
-
* If you do not want to start coverage immediately after launch or want to add additional configuration later on in a
|
|
257
|
+
* If you do not want to start coverage immediately after launch or want to add additional configuration later on in a
|
|
258
|
+
concise way, use:
|
|
227
259
|
|
|
228
260
|
```ruby
|
|
229
261
|
SimpleCov.configure do
|
|
@@ -235,11 +267,12 @@ Please check out the [Configuration] API documentation to find out what you can
|
|
|
235
267
|
|
|
236
268
|
## Using .simplecov for centralized config
|
|
237
269
|
|
|
238
|
-
If you use SimpleCov to merge multiple test suite results (e.g. Test/Unit and Cucumber) into a single report, you'd
|
|
239
|
-
set up all your config options twice, once in `test_helper.rb` and once in `env.rb`.
|
|
270
|
+
If you use SimpleCov to merge multiple test suite results (e.g. Test/Unit and Cucumber) into a single report, you'd
|
|
271
|
+
normally have to set up all your config options twice, once in `test_helper.rb` and once in `env.rb`.
|
|
240
272
|
|
|
241
|
-
To avoid this, you can place a file called `.simplecov` in your project root. You can then just leave the
|
|
242
|
-
test setup helper (**at the top**) and move the `SimpleCov.start` code with all your
|
|
273
|
+
To avoid this, you can place a file called `.simplecov` in your project root. You can then just leave the
|
|
274
|
+
`require 'simplecov'` in each test setup helper (**at the top**) and move the `SimpleCov.start` code with all your
|
|
275
|
+
custom config options into `.simplecov`:
|
|
243
276
|
|
|
244
277
|
```ruby
|
|
245
278
|
# test/test_helper.rb
|
|
@@ -254,21 +287,88 @@ SimpleCov.start 'rails' do
|
|
|
254
287
|
end
|
|
255
288
|
```
|
|
256
289
|
|
|
257
|
-
Using `.simplecov` rather than separately requiring SimpleCov multiple times is recommended if you are merging multiple
|
|
290
|
+
Using `.simplecov` rather than separately requiring SimpleCov multiple times is recommended if you are merging multiple
|
|
291
|
+
test frameworks like Cucumber and RSpec that rely on each other, as invoking SimpleCov multiple times can cause coverage
|
|
292
|
+
information to be lost.
|
|
293
|
+
|
|
294
|
+
## Branch coverage (ruby "~> 2.5")
|
|
295
|
+
Add branch coverage measurement statistics to your results. Supported in CRuby versions 2.5+.
|
|
296
|
+
|
|
297
|
+
```ruby
|
|
298
|
+
SimpleCov.start do
|
|
299
|
+
enable_coverage :branch
|
|
300
|
+
end
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
Branch coverage is a feature introduced in Ruby 2.5 concerning itself with whether a
|
|
304
|
+
particular branch of a condition had been executed. Line coverage on the other hand
|
|
305
|
+
is only interested in whether a line of code has been executed.
|
|
306
|
+
|
|
307
|
+
This comes in handy for instance for one line conditionals:
|
|
308
|
+
|
|
309
|
+
```ruby
|
|
310
|
+
number.odd? ? "odd" : "even"
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
In line coverage this line would always be marked as executed but you'd never know if both
|
|
314
|
+
conditions were met. Guard clauses have a similar story:
|
|
315
|
+
|
|
316
|
+
```ruby
|
|
317
|
+
return if number.odd?
|
|
318
|
+
|
|
319
|
+
# more code
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
If all the code in that method was covered you'd never know if the guard clause was ever
|
|
323
|
+
triggered! With line coverage as just evaluating the condition marks it as covered.
|
|
324
|
+
|
|
325
|
+
In the HTML report the lines of code will be annotated like `branch_type: hit_count`:
|
|
326
|
+
|
|
327
|
+
* `then: 2` - the then branch (of an `if`) was executed twice
|
|
328
|
+
* `else: 0` - the else branch (of an `if` or `case`) was never executed
|
|
329
|
+
|
|
330
|
+
Not that even if you don't declare an `else` branch it will still show up in the coverage
|
|
331
|
+
reports meaning that the condition of the `if` was not hit or that no `when` of `case`
|
|
332
|
+
was hit during the test runs.
|
|
333
|
+
|
|
334
|
+
**Is branch coverage strictly better?** No. Branch coverage really only concerns itself with
|
|
335
|
+
conditionals - meaning coverage of sequential code is of no interest to it. A file without
|
|
336
|
+
conditional logic will have no branch coverage data and SimpleCov will report 0 of 0
|
|
337
|
+
branches covered as 100% (as everything that can be covered was covered).
|
|
338
|
+
|
|
339
|
+
Hence, we recommend looking at both metrics together. Branch coverage might also be a good
|
|
340
|
+
overall metric to look at - while you might be missing only 10% of your lines that might
|
|
341
|
+
account for 50% of your branches for instance.
|
|
342
|
+
|
|
343
|
+
## Primary Coverage
|
|
344
|
+
|
|
345
|
+
By default, the primary coverage type is `line`. To set the primary coverage to something else, use the following:
|
|
346
|
+
|
|
347
|
+
```ruby
|
|
348
|
+
# or in configure SimpleCov.primary_coverage :branch
|
|
349
|
+
SimpleCov.start do
|
|
350
|
+
enable_coverage :branch
|
|
351
|
+
primary_coverage :branch
|
|
352
|
+
end
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
Primary coverage determines what will come in first all output, and the type of coverage to check if you don't specify the type of coverage when customizing exit behavior (`SimpleCov.minimum_coverage 90`).
|
|
356
|
+
|
|
357
|
+
Note that coverage must first be enabled for non-default coverage types.
|
|
258
358
|
|
|
259
359
|
## Filters
|
|
260
360
|
|
|
261
|
-
Filters can be used to remove selected files from your coverage data. By default, a filter is applied that removes all
|
|
262
|
-
OUTSIDE of your project's root directory - otherwise you'd end up with billions of coverage reports for source
|
|
263
|
-
gems you are using.
|
|
361
|
+
Filters can be used to remove selected files from your coverage data. By default, a filter is applied that removes all
|
|
362
|
+
files OUTSIDE of your project's root directory - otherwise you'd end up with billions of coverage reports for source
|
|
363
|
+
files in the gems you are using.
|
|
264
364
|
|
|
265
365
|
You can define your own to remove things like configuration files, tests or whatever you don't need in your coverage
|
|
266
366
|
report.
|
|
267
367
|
|
|
268
368
|
### Defining custom filters
|
|
269
369
|
|
|
270
|
-
You can currently define a filter using either a String or Regexp (that will then be Regexp-matched against each source
|
|
271
|
-
a block or by passing in your own Filter class.
|
|
370
|
+
You can currently define a filter using either a String or Regexp (that will then be Regexp-matched against each source
|
|
371
|
+
file's path), a block or by passing in your own Filter class.
|
|
272
372
|
|
|
273
373
|
#### String filter
|
|
274
374
|
|
|
@@ -300,9 +400,10 @@ SimpleCov.start do
|
|
|
300
400
|
end
|
|
301
401
|
```
|
|
302
402
|
|
|
303
|
-
Block filters receive a SimpleCov::SourceFile instance and expect your block to return either true (if the file is to be
|
|
304
|
-
from the result) or false (if the result should be kept). Please check out the RDoc for SimpleCov::SourceFile to
|
|
305
|
-
methods available to you. In the above example, the filter will remove all files that have less than 5
|
|
403
|
+
Block filters receive a SimpleCov::SourceFile instance and expect your block to return either true (if the file is to be
|
|
404
|
+
removed from the result) or false (if the result should be kept). Please check out the RDoc for SimpleCov::SourceFile to
|
|
405
|
+
learn about the methods available to you. In the above example, the filter will remove all files that have less than 5
|
|
406
|
+
lines of code.
|
|
306
407
|
|
|
307
408
|
#### Custom filter class
|
|
308
409
|
|
|
@@ -316,9 +417,10 @@ end
|
|
|
316
417
|
SimpleCov.add_filter LineFilter.new(5)
|
|
317
418
|
```
|
|
318
419
|
|
|
319
|
-
Defining your own filters is pretty easy: Just inherit from SimpleCov::Filter and define a method
|
|
320
|
-
the filter, a true return value from this method will result in the removal of the
|
|
321
|
-
is being set in the SimpleCov::Filter initialize method and thus is set to
|
|
420
|
+
Defining your own filters is pretty easy: Just inherit from SimpleCov::Filter and define a method
|
|
421
|
+
'matches?(source_file)'. When running the filter, a true return value from this method will result in the removal of the
|
|
422
|
+
given source_file. The filter_argument method is being set in the SimpleCov::Filter initialize method and thus is set to
|
|
423
|
+
5 in this example.
|
|
322
424
|
|
|
323
425
|
#### Array filter
|
|
324
426
|
|
|
@@ -343,16 +445,18 @@ end
|
|
|
343
445
|
# :nocov:
|
|
344
446
|
```
|
|
345
447
|
|
|
346
|
-
The name of the token can be changed to your liking. [Learn more about the nocov feature.]( https://github.com/
|
|
448
|
+
The name of the token can be changed to your liking. [Learn more about the nocov feature.]( https://github.com/simplecov-ruby/simplecov/blob/main/features/config_nocov_token.feature)
|
|
347
449
|
|
|
348
|
-
**Note:** You shouldn't have to use the nocov token to skip private methods that are being included in your coverage. If
|
|
450
|
+
**Note:** You shouldn't have to use the nocov token to skip private methods that are being included in your coverage. If
|
|
451
|
+
you appropriately test the public interface of your classes and objects you should automatically get full coverage of
|
|
452
|
+
your private methods.
|
|
349
453
|
|
|
350
454
|
## Default root filter and coverage for things outside of it
|
|
351
455
|
|
|
352
456
|
By default, SimpleCov filters everything outside of the `SimpleCov.root` directory. However, sometimes you may want
|
|
353
457
|
to include coverage reports for things you include as a gem, for example a Rails Engine.
|
|
354
458
|
|
|
355
|
-
Here's an example by [@lsaffie](https://github.com/lsaffie) from [#221](https://github.com/
|
|
459
|
+
Here's an example by [@lsaffie](https://github.com/lsaffie) from [#221](https://github.com/simplecov-ruby/simplecov/issues/221)
|
|
356
460
|
that shows how you can achieve just that:
|
|
357
461
|
|
|
358
462
|
```ruby
|
|
@@ -368,8 +472,8 @@ end
|
|
|
368
472
|
|
|
369
473
|
You can separate your source files into groups. For example, in a Rails app, you'll want to have separate listings for
|
|
370
474
|
Models, Controllers, Helpers, and Libs. Group definition works similarly to Filters (and also accepts custom
|
|
371
|
-
filter classes), but source files end up in a group when the filter passes (returns true), as opposed to filtering
|
|
372
|
-
which exclude files from results when the filter results in a true value.
|
|
475
|
+
filter classes), but source files end up in a group when the filter passes (returns true), as opposed to filtering
|
|
476
|
+
results, which exclude files from results when the filter results in a true value.
|
|
373
477
|
|
|
374
478
|
Add your groups with:
|
|
375
479
|
|
|
@@ -389,12 +493,11 @@ end
|
|
|
389
493
|
|
|
390
494
|
You normally want to have your coverage analyzed across ALL of your test suites, right?
|
|
391
495
|
|
|
392
|
-
Simplecov automatically caches coverage results in your
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
There are two things to note here though:
|
|
496
|
+
Simplecov automatically caches coverage results in your
|
|
497
|
+
(coverage_path)/.resultset.json, and will merge or override those with
|
|
498
|
+
subsequent runs, depending on whether simplecov considers those subsequent runs
|
|
499
|
+
as different test suites or as the same test suite as the cached results. To
|
|
500
|
+
make this distinction, simplecov has the concept of "test suite names".
|
|
398
501
|
|
|
399
502
|
### Test suite names
|
|
400
503
|
|
|
@@ -448,24 +551,143 @@ SimpleCov.command_name "features" + (ENV['TEST_ENV_NUMBER'] || '')
|
|
|
448
551
|
|
|
449
552
|
[simplecov-html] prints the used test suites in the footer of the generated coverage report.
|
|
450
553
|
|
|
451
|
-
### Timeout for merge
|
|
452
554
|
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
555
|
+
### Merging test runs under the same execution environment
|
|
556
|
+
|
|
557
|
+
Test results are automatically merged with previous runs in the same execution
|
|
558
|
+
environment when generating the result, so when coverage is set up properly for
|
|
559
|
+
Cucumber and your unit / functional / integration tests, all of those test
|
|
560
|
+
suites will be taken into account when building the coverage report.
|
|
561
|
+
|
|
562
|
+
#### Timeout for merge
|
|
563
|
+
|
|
564
|
+
Of course, your cached coverage data is likely to become invalid at some point. Thus, when automatically merging
|
|
565
|
+
subsequent test runs, result sets that are older than `SimpleCov.merge_timeout` will not be used any more. By default,
|
|
566
|
+
the timeout is 600 seconds (10 minutes), and you can raise (or lower) it by specifying `SimpleCov.merge_timeout 3600`
|
|
567
|
+
(1 hour), or, inside a configure/start block, with just `merge_timeout 3600`.
|
|
568
|
+
|
|
569
|
+
You can deactivate this automatic merging altogether with `SimpleCov.use_merging false`.
|
|
570
|
+
|
|
571
|
+
### Merging test runs under different execution environments
|
|
572
|
+
|
|
573
|
+
If your tests are done in parallel across multiple build machines, you can fetch them all and merge them into a single
|
|
574
|
+
result set using the `SimpleCov.collate` method. This can be added to a Rakefile or script file, having downloaded a set of
|
|
575
|
+
`.resultset.json` files from each parallel test run.
|
|
576
|
+
|
|
577
|
+
```ruby
|
|
578
|
+
# lib/tasks/coverage_report.rake
|
|
579
|
+
namespace :coverage do
|
|
580
|
+
desc "Collates all result sets generated by the different test runners"
|
|
581
|
+
task :report do
|
|
582
|
+
require 'simplecov'
|
|
583
|
+
|
|
584
|
+
SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"]
|
|
585
|
+
end
|
|
586
|
+
end
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
`SimpleCov.collate` also takes an optional simplecov profile and an optional
|
|
590
|
+
block for configuration, just the same as `SimpleCov.start` or
|
|
591
|
+
`SimpleCov.configure`. This means you can configure a separate formatter for
|
|
592
|
+
the collated output. For instance, you can make the formatter in
|
|
593
|
+
`SimpleCov.start` the `SimpleCov::Formatter::SimpleFormatter`, and only use more
|
|
594
|
+
complex formatters in the final `SimpleCov.collate` run.
|
|
595
|
+
|
|
596
|
+
```ruby
|
|
597
|
+
# spec/spec_helper.rb
|
|
598
|
+
require 'simplecov'
|
|
599
|
+
|
|
600
|
+
SimpleCov.start 'rails' do
|
|
601
|
+
# Disambiguates individual test runs
|
|
602
|
+
command_name "Job #{ENV["TEST_ENV_NUMBER"]}" if ENV["TEST_ENV_NUMBER"]
|
|
603
|
+
|
|
604
|
+
if ENV['CI']
|
|
605
|
+
formatter SimpleCov::Formatter::SimpleFormatter
|
|
606
|
+
else
|
|
607
|
+
formatter SimpleCov::Formatter::MultiFormatter.new([
|
|
608
|
+
SimpleCov::Formatter::SimpleFormatter,
|
|
609
|
+
SimpleCov::Formatter::HTMLFormatter
|
|
610
|
+
])
|
|
611
|
+
end
|
|
612
|
+
|
|
613
|
+
track_files "**/*.rb"
|
|
614
|
+
end
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
```ruby
|
|
618
|
+
# lib/tasks/coverage_report.rake
|
|
619
|
+
namespace :coverage do
|
|
620
|
+
task :report do
|
|
621
|
+
require 'simplecov'
|
|
622
|
+
|
|
623
|
+
SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do
|
|
624
|
+
formatter SimpleCov::Formatter::MultiFormatter.new([
|
|
625
|
+
SimpleCov::Formatter::SimpleFormatter,
|
|
626
|
+
SimpleCov::Formatter::HTMLFormatter
|
|
627
|
+
])
|
|
628
|
+
end
|
|
629
|
+
end
|
|
630
|
+
end
|
|
631
|
+
```
|
|
632
|
+
|
|
633
|
+
## Running simplecov against subprocesses
|
|
634
|
+
|
|
635
|
+
`SimpleCov.enable_for_subprocesses` will allow SimpleCov to observe subprocesses starting using `Process.fork`.
|
|
636
|
+
This modifies ruby's core Process.fork method so that SimpleCov can see into it, appending `" (subprocess #{pid})"`
|
|
637
|
+
to the `SimpleCov.command_name`, with results that can be merged together using SimpleCov's merging feature.
|
|
638
|
+
|
|
639
|
+
To configure this, use `.at_fork`.
|
|
640
|
+
|
|
641
|
+
```ruby
|
|
642
|
+
SimpleCov.enable_for_subprocesses true
|
|
643
|
+
SimpleCov.at_fork do |pid|
|
|
644
|
+
# This needs a unique name so it won't be ovewritten
|
|
645
|
+
SimpleCov.command_name "#{SimpleCov.command_name} (subprocess: #{pid})"
|
|
646
|
+
# be quiet, the parent process will be in charge of output and checking coverage totals
|
|
647
|
+
SimpleCov.print_error_status = false
|
|
648
|
+
SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter
|
|
649
|
+
SimpleCov.minimum_coverage 0
|
|
650
|
+
# start
|
|
651
|
+
SimpleCov.start
|
|
652
|
+
end
|
|
653
|
+
```
|
|
654
|
+
|
|
655
|
+
NOTE: SimpleCov must have already been started before `Process.fork` was called.
|
|
457
656
|
|
|
458
|
-
|
|
657
|
+
### Running simplecov against spawned subprocesses
|
|
658
|
+
|
|
659
|
+
Perhaps you're testing a ruby script with `PTY.spawn` or `Open3.popen`, or `Process.spawn` or etc.
|
|
660
|
+
SimpleCov can cover this too.
|
|
661
|
+
|
|
662
|
+
Add a .simplecov_spawn.rb file to your project root
|
|
663
|
+
```ruby
|
|
664
|
+
# .simplecov_spawn.rb
|
|
665
|
+
require 'simplecov' # this will also pick up whatever config is in .simplecov
|
|
666
|
+
# so ensure it just contains configuration, and doesn't call SimpleCov.start.
|
|
667
|
+
SimpleCov.command_name 'spawn' # As this is not for a test runner directly, script doesn't have a pre-defined base command_name
|
|
668
|
+
SimpleCov.at_fork.call(Process.pid) # Use the per-process setup described previously
|
|
669
|
+
SimpleCov.start # only now can we start.
|
|
670
|
+
```
|
|
671
|
+
Then, instead of calling your script directly, like:
|
|
672
|
+
```ruby
|
|
673
|
+
PTY.spawn('my_script.rb') do # ...
|
|
674
|
+
```
|
|
675
|
+
Use bin/ruby to require the new .simplecov_spawn file, then your script
|
|
676
|
+
```ruby
|
|
677
|
+
PTY.spawn('ruby -r./.simplecov_spawn my_script.rb') do # ...
|
|
678
|
+
```
|
|
459
679
|
|
|
460
680
|
## Running coverage only on demand
|
|
461
681
|
|
|
462
|
-
The Ruby STDLIB Coverage library that SimpleCov builds upon is *very* fast (on a ~10 min Rails test suite, the speed
|
|
463
|
-
only a couple seconds for me), and therefore it's SimpleCov's policy to just generate coverage every time you
|
|
464
|
-
it doesn't do your test speed any harm and you're always equipped with the latest and greatest
|
|
682
|
+
The Ruby STDLIB Coverage library that SimpleCov builds upon is *very* fast (on a ~10 min Rails test suite, the speed
|
|
683
|
+
drop was only a couple seconds for me), and therefore it's SimpleCov's policy to just generate coverage every time you
|
|
684
|
+
run your tests because it doesn't do your test speed any harm and you're always equipped with the latest and greatest
|
|
685
|
+
coverage results.
|
|
465
686
|
|
|
466
687
|
Because of this, SimpleCov has no explicit built-in mechanism to run coverage only on demand.
|
|
467
688
|
|
|
468
|
-
However, you can still accomplish this very easily by introducing an ENV variable conditional into your SimpleCov setup
|
|
689
|
+
However, you can still accomplish this very easily by introducing an ENV variable conditional into your SimpleCov setup
|
|
690
|
+
block, like this:
|
|
469
691
|
|
|
470
692
|
```ruby
|
|
471
693
|
SimpleCov.start if ENV["COVERAGE"]
|
|
@@ -477,6 +699,21 @@ Then, SimpleCov will only run if you execute your tests like this:
|
|
|
477
699
|
COVERAGE=true rake test
|
|
478
700
|
```
|
|
479
701
|
|
|
702
|
+
## Errors and exit statuses
|
|
703
|
+
|
|
704
|
+
To aid in debugging issues, if an error is raised, SimpleCov will print a message to `STDERR`
|
|
705
|
+
with the exit status of the error, like:
|
|
706
|
+
|
|
707
|
+
```
|
|
708
|
+
SimpleCov failed with exit 1
|
|
709
|
+
```
|
|
710
|
+
|
|
711
|
+
This `STDERR` message can be disabled with:
|
|
712
|
+
|
|
713
|
+
```
|
|
714
|
+
SimpleCov.print_error_status = false
|
|
715
|
+
```
|
|
716
|
+
|
|
480
717
|
## Profiles
|
|
481
718
|
|
|
482
719
|
By default, SimpleCov's only config assumption is that you only want coverage reports for files inside your project
|
|
@@ -514,8 +751,8 @@ end
|
|
|
514
751
|
|
|
515
752
|
### Custom profiles
|
|
516
753
|
|
|
517
|
-
You can load additional profiles with the SimpleCov.load_profile('xyz') method. This allows you to build upon an
|
|
518
|
-
profile and customize it so you can reuse it in unit tests and Cucumber features. For example:
|
|
754
|
+
You can load additional profiles with the SimpleCov.load_profile('xyz') method. This allows you to build upon an
|
|
755
|
+
existing profile and customize it so you can reuse it in unit tests and Cucumber features. For example:
|
|
519
756
|
|
|
520
757
|
```ruby
|
|
521
758
|
# lib/simplecov_custom_profile.rb
|
|
@@ -552,14 +789,23 @@ You can define the minimum coverage percentage expected. SimpleCov will return n
|
|
|
552
789
|
|
|
553
790
|
```ruby
|
|
554
791
|
SimpleCov.minimum_coverage 90
|
|
792
|
+
# same as above (the default is to check line coverage)
|
|
793
|
+
SimpleCov.minimum_coverage line: 90
|
|
794
|
+
# check for a minimum line coverage of 90% and minimum 80% branch coverage
|
|
795
|
+
SimpleCov.minimum_coverage line: 90, branch: 80
|
|
555
796
|
```
|
|
556
797
|
|
|
557
798
|
### Minimum coverage by file
|
|
558
799
|
|
|
559
|
-
You can define the minimum coverage by file percentage expected. SimpleCov will return non-zero if unmet. This is useful
|
|
800
|
+
You can define the minimum coverage by file percentage expected. SimpleCov will return non-zero if unmet. This is useful
|
|
801
|
+
to help ensure coverage is relatively consistent, rather than being skewed by particularly good or bad areas of the code.
|
|
560
802
|
|
|
561
803
|
```ruby
|
|
562
804
|
SimpleCov.minimum_coverage_by_file 80
|
|
805
|
+
# same as above (the default is to check line coverage by file)
|
|
806
|
+
SimpleCov.minimum_coverage_by_file line: 80
|
|
807
|
+
# check for a minimum line coverage by file of 90% and minimum 80% branch coverage
|
|
808
|
+
SimpleCov.minimum_coverage_by_file line: 90, branch: 80
|
|
563
809
|
```
|
|
564
810
|
|
|
565
811
|
### Maximum coverage drop
|
|
@@ -568,6 +814,10 @@ You can define the maximum coverage drop percentage at once. SimpleCov will retu
|
|
|
568
814
|
|
|
569
815
|
```ruby
|
|
570
816
|
SimpleCov.maximum_coverage_drop 5
|
|
817
|
+
# same as above (the default is to check line drop)
|
|
818
|
+
SimpleCov.maximum_coverage_drop line: 5
|
|
819
|
+
# check for a maximum line drop of 5% and maximum 10% branch drop
|
|
820
|
+
SimpleCov.maximum_coverage_drop line: 5, branch: 10
|
|
571
821
|
```
|
|
572
822
|
|
|
573
823
|
### Refuse dropping coverage
|
|
@@ -576,6 +826,10 @@ You can also entirely refuse dropping coverage between test runs:
|
|
|
576
826
|
|
|
577
827
|
```ruby
|
|
578
828
|
SimpleCov.refuse_coverage_drop
|
|
829
|
+
# same as above (the default is to only refuse line drop)
|
|
830
|
+
SimpleCov.refuse_coverage_drop :line
|
|
831
|
+
# refuse drop for line and branch
|
|
832
|
+
SimpleCov.refuse_coverage_drop :line, :branch
|
|
579
833
|
```
|
|
580
834
|
|
|
581
835
|
## Using your own formatter
|
|
@@ -586,8 +840,8 @@ You can use your own formatter with:
|
|
|
586
840
|
SimpleCov.formatter = SimpleCov::Formatter::HTMLFormatter
|
|
587
841
|
```
|
|
588
842
|
|
|
589
|
-
When calling SimpleCov.result.format!, it will be invoked with SimpleCov::Formatter::YourFormatter.new.format(result),
|
|
590
|
-
being an instance of SimpleCov::Result. Do whatever your wish with that!
|
|
843
|
+
When calling SimpleCov.result.format!, it will be invoked with SimpleCov::Formatter::YourFormatter.new.format(result),
|
|
844
|
+
"result" being an instance of SimpleCov::Result. Do whatever your wish with that!
|
|
591
845
|
|
|
592
846
|
|
|
593
847
|
## Using multiple formatters
|
|
@@ -601,6 +855,20 @@ SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
|
|
|
601
855
|
])
|
|
602
856
|
```
|
|
603
857
|
|
|
858
|
+
## JSON formatter
|
|
859
|
+
|
|
860
|
+
SimpleCov is packaged with a separate gem called [simplecov_json_formatter](https://github.com/codeclimate-community/simplecov_json_formatter) that provides you with a JSON formatter, this formatter could be useful for different use cases, such as for CI consumption or for reporting to external services.
|
|
861
|
+
|
|
862
|
+
In order to use it you will need to manually load the installed gem like so:
|
|
863
|
+
|
|
864
|
+
```ruby
|
|
865
|
+
require "simplecov_json_formatter"
|
|
866
|
+
SimpleCov.formatter = SimpleCov::Formatter::JSONFormatter
|
|
867
|
+
```
|
|
868
|
+
|
|
869
|
+
> _Note:_ In case you plan to report your coverage results to CodeClimate services, know that SimpleCov will automatically use the
|
|
870
|
+
> JSON formatter along with the HTML formatter when the `CC_TEST_REPORTER_ID` variable is present in the environment.
|
|
871
|
+
|
|
604
872
|
## Available formatters, editor integrations and hosted services
|
|
605
873
|
|
|
606
874
|
* [Open Source formatter and integration plugins for SimpleCov](doc/alternate-formatters.md)
|
|
@@ -609,11 +877,9 @@ SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
|
|
|
609
877
|
|
|
610
878
|
## Ruby version compatibility
|
|
611
879
|
|
|
612
|
-
|
|
613
|
-
SimpleCov is also built against Ruby 1.8 in [Continuous Integration], but this happens only to ensure that SimpleCov
|
|
614
|
-
does not make your test suite crash right now.
|
|
880
|
+
SimpleCov is built in [Continuous Integration] on Ruby 2.5+ as well as JRuby 9.2+.
|
|
615
881
|
|
|
616
|
-
|
|
882
|
+
Note for JRuby => You need to pass JRUBY_OPTS="--debug" or create .jrubyrc and add debug.fullTrace=true
|
|
617
883
|
|
|
618
884
|
## Want to find dead code in production?
|
|
619
885
|
|
|
@@ -621,10 +887,12 @@ Try [Coverband](https://github.com/danmayer/coverband).
|
|
|
621
887
|
|
|
622
888
|
## Want to use Spring with SimpleCov?
|
|
623
889
|
|
|
624
|
-
If you're using [Spring](https://github.com/rails/spring) to speed up test suite runs and want to run SimpleCov along
|
|
890
|
+
If you're using [Spring](https://github.com/rails/spring) to speed up test suite runs and want to run SimpleCov along
|
|
891
|
+
with them, you'll find that it often misreports coverage with the default config due to some sort of eager loading
|
|
892
|
+
issue. Don't despair!
|
|
625
893
|
|
|
626
894
|
One solution is to [explicitly call eager
|
|
627
|
-
load](https://github.com/
|
|
895
|
+
load](https://github.com/simplecov-ruby/simplecov/issues/381#issuecomment-347651728)
|
|
628
896
|
in your `test_helper.rb` / `spec_helper.rb` after calling `SimpleCov.start`.
|
|
629
897
|
|
|
630
898
|
```ruby
|
|
@@ -633,13 +901,22 @@ SimpleCov.start 'rails'
|
|
|
633
901
|
Rails.application.eager_load!
|
|
634
902
|
```
|
|
635
903
|
|
|
904
|
+
Alternatively, you could disable Spring while running SimpleCov:
|
|
905
|
+
|
|
906
|
+
```
|
|
907
|
+
DISABLE_SPRING=1 rake test
|
|
908
|
+
```
|
|
909
|
+
|
|
636
910
|
Or you could remove `gem 'spring'` from your `Gemfile`.
|
|
637
911
|
|
|
638
912
|
## Troubleshooting
|
|
639
913
|
|
|
640
|
-
The **most common problem is that simplecov isn't required and started before everything else**. In order to track
|
|
914
|
+
The **most common problem is that simplecov isn't required and started before everything else**. In order to track
|
|
915
|
+
coverage for your whole application **simplecov needs to be the first one** so that it (and the underlying coverage
|
|
916
|
+
library) can subsequently track loaded files and their usage.
|
|
641
917
|
|
|
642
|
-
If you are missing coverage for some code a simple trick is to put a puts statement in there and right after
|
|
918
|
+
If you are missing coverage for some code a simple trick is to put a puts statement in there and right after
|
|
919
|
+
`SimpleCov.start` so you can see if the file really was loaded after simplecov was started.
|
|
643
920
|
|
|
644
921
|
```ruby
|
|
645
922
|
# my_code.rb
|
|
@@ -667,9 +944,14 @@ MyCode is being loaded!
|
|
|
667
944
|
|
|
668
945
|
then it's good otherwise you likely have a problem :)
|
|
669
946
|
|
|
947
|
+
## Code of Conduct
|
|
948
|
+
|
|
949
|
+
Everyone participating in this project's development, issue trackers and other channels is expected to follow our
|
|
950
|
+
[Code of Conduct](./CODE_OF_CONDUCT.md)
|
|
951
|
+
|
|
670
952
|
## Contributing
|
|
671
953
|
|
|
672
|
-
See the [contributing guide](https://github.com/
|
|
954
|
+
See the [contributing guide](https://github.com/simplecov-ruby/simplecov/blob/main/CONTRIBUTING.md).
|
|
673
955
|
|
|
674
956
|
## Kudos
|
|
675
957
|
|