rspec-cover_it 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 528de4fe406f539d93d8632624714721d8cded1463a35776c16423d387c13c30
4
+ data.tar.gz: 57fc958451fb2dfeafb94aaaed56aba38befd938a4ea24d2f82d0ac09a46ee4e
5
+ SHA512:
6
+ metadata.gz: 6c636b00bd14fd976d53f9f97cbd2d53334ecb5677ad480ef00e737b51144da2f0ae76fffc1f6d1fb0366381961311e8aec93dfe6453cc2942c10dc652d87a88
7
+ data.tar.gz: 93c2432a8ce2725b2ca3ff9f1c8115a38154dc7c4426f7aa5667d12ed9546b9890d83885bcbe727064ea78972e58ab4e721b6dcef82d93c004d95f8a8ae6609f
data/.gitignore ADDED
@@ -0,0 +1,6 @@
1
+ .ruby-version
2
+ .ruby-gemset
3
+ Gemfile.lock
4
+ *.gem
5
+ coverage/
6
+ tmp/*
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source "https://rubygems.org"
2
+
3
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright 2023 Eric Mueller
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,105 @@
1
+ # RSpec::CoverIt
2
+
3
+ Still a work in progress, but this gem is intended to give us a simpler and
4
+ more _targeted_ way to manage coverage. The usual system for a large project
5
+ is a single overarching SimpleCov target; with good tooling, there will be
6
+ something checking that all of the lines of each PR get covered by the tests
7
+ in that PR, and there are checks that the overall coverage number doesn't drop
8
+ too far.. but it's a _sloppy, messy, lossy_ approach. Code loses coverage,
9
+ code gets added with no tests, tests get removed and other code becomes less
10
+ covered, and a lot of the time, that's because the _coverage is a lie_.
11
+
12
+ Your coverage tool is telling you.. how many times each line of code _got run_
13
+ during your entire test suite. But it's not telling you what is doing the
14
+ running, which means that often large swathes of code are actually only covered
15
+ incidentally, through tests that aren't intended to exercise that code, but
16
+ _happens to_.
17
+
18
+ This approach was moderately inspired by https://github.com/jamesdabbs/rspec-coverage,
19
+ which was itself apparently inspired by a rubyconf 2016 talk by Ryan Davis. That
20
+ talk and library are mostly concerned with making sure that we don't _overcount_
21
+ coverage though, while this one has three goals:
22
+
23
+ 1. Make coverage enforcement simpler
24
+ 1. Enforce coverage _in the test suite_ (or nearly so)
25
+ 1. Make coverage enforcement more local and specific.
26
+
27
+ ## Tentative Usage
28
+
29
+ Note that this bit isn't really implemented, but is more of an outline of how I
30
+ intend the library to work. This is currently a spike, and not a functioning gem.
31
+
32
+ You set up `RSpec::CoverIt` in your `spec_helper.rb` - require `rspec/cover_it`,
33
+ and then `RSpec::CoverIt.setup(**config_options)` (before loading your code, as
34
+ you would with any other coverage library). It's.. _semi_ compatible with other
35
+ coverage systems - it only starts Coverage if it's not already running, and it
36
+ only uses `peek_result`, so it doesn't affect other systems outcomes. But if you
37
+ setup SimpleCov to do branch-coverage or something, that will break this gem,
38
+ for now (we assume basic line-coverage style results).
39
+
40
+ Rough configuration options:
41
+
42
+ * `filter`: Only paths starting with this (matching this regex?) can matter
43
+ * `autoenforce`: off by default - with this disabled, you turn on coverage
44
+ enforcement for a given top-level describe ExampleGroup by adding metadata
45
+ like `cover_it: true` or `cover_it: 95`. If it's enabled though, you instead
46
+ can turn enforcement _off_ for an example group by setting `cover_it: false`.
47
+ * `global_threshold`: 100% by default. This whole "90% coverage is the right
48
+ target" thing is mostly a side-effect of the way we check the entire project
49
+ under one number.. but it's an easy setting to support, and I'm sure people
50
+ will disagree with me.
51
+
52
+ ## Example Group Metadata
53
+
54
+ In example groups, you can use metadata to control the behavior of
55
+ `rspec-cover_it`. These keys have meaning:
56
+
57
+ * `cover_it`: if boolean, it enables or disables the coverage enforcement for
58
+ this ExampleGroup. If numeric, it's enabling, and specifying the coverage
59
+ threshold at the same time (as a percentage - `cover_it: 95` requires 95%
60
+ coverage of the target class by this example group).
61
+ * `covers`: An array of classes and modules that this example groups _thinks
62
+ it is completely testing_. Ideally, you'd have a separate test file for each,
63
+ but sometimes that's hard to do - you can let one spec claim responsibility
64
+ for multiple classes and modules (such as Concerns) using this. Be default
65
+ it is just `[described_class]`. Additionally, if your top-level example
66
+ group _does not describe a Class or Module_, you may use `covers` to let it
67
+ invoke `rspec-cover_it` anyway.
68
+
69
+ ## Implementation
70
+
71
+ We record the coverage information in a `before(:suite)` rspec hook using
72
+ `Coverage.peek_result`, and hold onto that information. Then before and after
73
+ each 'context' (which really amounts to 'each spec file'), we grab the coverage
74
+ information again.
75
+
76
+ We use `Object.const_source_location` to find the file that defines the
77
+ `described_class`, and _that_ is what is being assessed for coverage. This
78
+ means that, if you are reopening the class somewhere else, that coverage won't
79
+ be checked; if you are including 15 Concerns, and don't intend to write separate
80
+ specs for them, be sure to list them as `covers:` metadata on the test. Also,
81
+ shame!.
82
+
83
+ ## Drawbacks and Shortcomings
84
+
85
+ There's nothing in here that stops you from failing to write tests for a class
86
+ at all! If you're using SimpleCov and you've got 100% coverage already, that's
87
+ one of the benefits.. I could pretty reasonably include some kind of
88
+ `after(:suite)` hook that optionally checks net coverage, but.. simplecov does
89
+ that, and the concurrent-testing game makes this a _painful_ topic in reality.
90
+ That's not the goal here, and I'm not going to worry about it.
91
+
92
+ As initially implemented, it fails your tests if you don't run the entire test
93
+ file. `rspec spec/foo_spec.rb:32` will error, because .. running only one of
94
+ your tests _doesn't cover the class_. I have a solution for this, but it uses
95
+ some non-public bits of RSpec, so I'm trying to find a better answer first.
96
+ (Conversation started in their
97
+ [issue tracker](https://github.com/rspec/rspec-core/issues/3037))
98
+
99
+ So far, I haven't found a _great_ way to report that there's a problem. The
100
+ output from having insufficient coverage is _raising an exception from an
101
+ `after(:context)` hook_, which RSpec rescues and formats for itself, and there
102
+ aren't a lot of controls - it works, but it's a bit ugly and doesn't really
103
+ give the right immediate impression. I'm contemplating using an `after(:suite)`
104
+ hook and aggregating them myself, but at the end of the day RSpec is in control
105
+ of the output stream, and we don't entirely fit its metaphor.
@@ -0,0 +1,33 @@
1
+ module RSpec
2
+ module CoverIt
3
+ class Context
4
+ def initialize(scope:, rspec_context:)
5
+ @scope, @rspec_context = scope, rspec_context
6
+ end
7
+
8
+ def cover_it?
9
+ target_class && metadata.fetch(:cover_it, nil)
10
+ end
11
+
12
+ def target_path
13
+ Object.const_source_location(target_class_name).first
14
+ end
15
+
16
+ def target_class
17
+ metadata.fetch(:described_class, nil)
18
+ end
19
+
20
+ def target_class_name
21
+ target_class.name
22
+ end
23
+
24
+ private
25
+
26
+ attr_reader :scope, :rspec_context
27
+
28
+ def metadata
29
+ scope.metadata
30
+ end
31
+ end
32
+ end
33
+ end
@@ -0,0 +1,54 @@
1
+ module RSpec
2
+ module CoverIt
3
+ class ContextCoverage
4
+ def initialize(context:, pretest_results:)
5
+ @context, @pretest_results = context, pretest_results
6
+ @precontext_coverage = @postcontext_coverage = nil
7
+ end
8
+
9
+ attr_accessor :precontext_coverage, :postcontext_coverage
10
+
11
+ def local_coverage
12
+ return nil unless precontext_coverage && postcontext_coverage
13
+ @_local_coverage ||= pretest_coverage
14
+ .zip(precontext_coverage, postcontext_coverage)
15
+ .map { |a, b, c| line_calculation(a, b, c) }
16
+ end
17
+
18
+ def local_coverage_rate
19
+ return nil unless covered_line_count
20
+ covered_line_count.to_f / coverable_line_count.to_f
21
+ end
22
+
23
+ private
24
+
25
+ attr_reader :context, :pretest_results, :precontext_coverage, :postcontext_coverage
26
+
27
+ def target_path
28
+ @_target_path ||= context.target_path
29
+ end
30
+
31
+ def pretest_coverage
32
+ @_pretest_coverage ||= pretest_results[target_path]
33
+ end
34
+
35
+ # Really, we shouldn't see nil for any of these values unless they are all
36
+ # nil. We want the coverage we'd expect to have seen if we ran _just_ this
37
+ # groups of examples, which ought boe the pretest coverage, plus the
38
+ # number of times each line was run _during_ the context.
39
+ def line_calculation(pretest, precontext, postcontext)
40
+ return nil if pretest.nil? || precontext.nil? || postcontext.nil?
41
+ pretest + (postcontext - precontext)
42
+ end
43
+
44
+ def coverable_line_count
45
+ pretest_coverage.compact.count
46
+ end
47
+
48
+ def covered_line_count
49
+ return nil unless local_coverage
50
+ local_coverage.count { |executions| executions && executions > 0 }
51
+ end
52
+ end
53
+ end
54
+ end
@@ -0,0 +1,53 @@
1
+ module RSpec
2
+ module CoverIt
3
+ class CoverageState
4
+ attr_reader :filter
5
+
6
+ def initialize(filter:)
7
+ @filter = filter
8
+ @pretest_results = nil
9
+ @context_coverages = {}
10
+ end
11
+
12
+ # If SimpleCov or something like it is already running, Coverage may be
13
+ # started already. For our purposes, that should be fine.
14
+ def start_tracking
15
+ Coverage.start unless Coverage.running?
16
+ end
17
+
18
+ def finish_load_tracking
19
+ @pretest_results = PretestCoverage.new(filter: filter, results: Coverage.peek_result)
20
+ end
21
+
22
+ def start_tracking_for(scope, rspec_context)
23
+ context = Context.new(scope: scope, rspec_context: rspec_context)
24
+ return unless context.cover_it?
25
+
26
+ context_coverage_for(context).tap do |context_coverage|
27
+ context_coverage.precontext_coverage = Coverage.peek_result[context.target_path]
28
+ end
29
+ end
30
+
31
+ def finish_tracking_for(scope, rspec_context)
32
+ context = Context.new(scope: scope, rspec_context: rspec_context)
33
+ return unless context.cover_it?
34
+
35
+ context_coverage_for(context).tap do |context_coverage|
36
+ context_coverage.postcontext_coverage = Coverage.peek_result[context.target_path]
37
+ fail("missing coverage!") if context_coverage.local_coverage_rate < 1.0
38
+ end
39
+ end
40
+
41
+ private
42
+
43
+ attr_reader :pretest_results
44
+
45
+ def context_coverage_for(context)
46
+ @context_coverages[context.target_class] ||= ContextCoverage.new(
47
+ context: context,
48
+ pretest_results: pretest_results
49
+ )
50
+ end
51
+ end
52
+ end
53
+ end
@@ -0,0 +1,17 @@
1
+ module RSpec
2
+ module CoverIt
3
+ class PretestCoverage
4
+ def initialize(filter:, results:)
5
+ @filter = filter
6
+ @results = results
7
+ .select { |k, _v| k.start_with?(filter) }
8
+ .map { |k, v| [k, v.dup] }
9
+ .to_h
10
+ end
11
+
12
+ def [](path)
13
+ @results[path]
14
+ end
15
+ end
16
+ end
17
+ end
@@ -0,0 +1,5 @@
1
+ module RSpec
2
+ module CoverIt
3
+ VERSION = "0.0.2"
4
+ end
5
+ end
@@ -0,0 +1,30 @@
1
+ require "coverage"
2
+
3
+ module RSpec
4
+ module CoverIt
5
+ Error = Class.new(StandardError)
6
+ MissingCoverage = Class.new(Error)
7
+ end
8
+ end
9
+
10
+ glob = File.expand_path("../cover_it/*.rb", __FILE__)
11
+ Dir.glob(glob).sort.each { |f| require(f) }
12
+
13
+ module RSpec
14
+ module CoverIt
15
+ class << self
16
+ attr_accessor :state
17
+ end
18
+
19
+ def self.setup(filter:)
20
+ RSpec::CoverIt.state = CoverageState.new(filter: filter)
21
+ RSpec::CoverIt.state.start_tracking
22
+
23
+ RSpec.configure do |config|
24
+ config.prepend_before(:suite) { RSpec::CoverIt.state.finish_load_tracking }
25
+ config.prepend_before(:context) { |context| RSpec::CoverIt.state.start_tracking_for(self.class, context) }
26
+ config.append_after(:context) { |context| RSpec::CoverIt.state.finish_tracking_for(self.class, context) }
27
+ end
28
+ end
29
+ end
30
+ end
@@ -0,0 +1,36 @@
1
+ require_relative "lib/rspec/cover_it/version"
2
+
3
+ Gem::Specification.new do |spec|
4
+ spec.name = "rspec-cover_it"
5
+ spec.version = RSpec::CoverIt::VERSION
6
+ spec.authors = ["Eric Mueller"]
7
+ spec.email = ["nevinera@gmail.com"]
8
+
9
+ spec.summary = "A system to enforce test coverage on each class"
10
+ spec.description = <<~DESC
11
+ We're all used to tools that enforce _total_ coverage numbers, but this gem
12
+ tries for something different. Instead of keeping your whole project above
13
+ some threshold, we treat the coverage of _each class_ as a testable quality
14
+ and then enforce that coverage as part of the test suite!
15
+ DESC
16
+ spec.homepage = "https://github.com/nevinera/rspec-cover_it"
17
+ spec.license = "MIT"
18
+ spec.required_ruby_version = Gem::Requirement.new(">= 2.7.0")
19
+
20
+ spec.metadata["homepage_uri"] = spec.homepage
21
+ spec.metadata["source_code_uri"] = spec.homepage
22
+
23
+ spec.require_paths = ["lib"]
24
+ spec.files = Dir.chdir(File.expand_path(__dir__)) do
25
+ `git ls-files -z`
26
+ .split("\x0")
27
+ .reject { |f| f.start_with?("spec") }
28
+ end
29
+
30
+ spec.add_dependency "rspec", "~> 3.10"
31
+
32
+ spec.add_development_dependency "pry", "~> 0.14"
33
+ spec.add_development_dependency "standard", "~> 1.28"
34
+ spec.add_development_dependency "mdl", "~> 0.12"
35
+ spec.add_development_dependency "quiet_quality", "~> 1.2"
36
+ end
metadata ADDED
@@ -0,0 +1,130 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: rspec-cover_it
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.2
5
+ platform: ruby
6
+ authors:
7
+ - Eric Mueller
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2023-06-07 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: rspec
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '3.10'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '3.10'
27
+ - !ruby/object:Gem::Dependency
28
+ name: pry
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '0.14'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '0.14'
41
+ - !ruby/object:Gem::Dependency
42
+ name: standard
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '1.28'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '1.28'
55
+ - !ruby/object:Gem::Dependency
56
+ name: mdl
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '0.12'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '0.12'
69
+ - !ruby/object:Gem::Dependency
70
+ name: quiet_quality
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - "~>"
74
+ - !ruby/object:Gem::Version
75
+ version: '1.2'
76
+ type: :development
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - "~>"
81
+ - !ruby/object:Gem::Version
82
+ version: '1.2'
83
+ description: |
84
+ We're all used to tools that enforce _total_ coverage numbers, but this gem
85
+ tries for something different. Instead of keeping your whole project above
86
+ some threshold, we treat the coverage of _each class_ as a testable quality
87
+ and then enforce that coverage as part of the test suite!
88
+ email:
89
+ - nevinera@gmail.com
90
+ executables: []
91
+ extensions: []
92
+ extra_rdoc_files: []
93
+ files:
94
+ - ".gitignore"
95
+ - Gemfile
96
+ - LICENSE
97
+ - README.md
98
+ - lib/rspec/cover_it.rb
99
+ - lib/rspec/cover_it/context.rb
100
+ - lib/rspec/cover_it/context_coverage.rb
101
+ - lib/rspec/cover_it/coverage_state.rb
102
+ - lib/rspec/cover_it/pretest_coverage.rb
103
+ - lib/rspec/cover_it/version.rb
104
+ - rspec-cover_it.gemspec
105
+ homepage: https://github.com/nevinera/rspec-cover_it
106
+ licenses:
107
+ - MIT
108
+ metadata:
109
+ homepage_uri: https://github.com/nevinera/rspec-cover_it
110
+ source_code_uri: https://github.com/nevinera/rspec-cover_it
111
+ post_install_message:
112
+ rdoc_options: []
113
+ require_paths:
114
+ - lib
115
+ required_ruby_version: !ruby/object:Gem::Requirement
116
+ requirements:
117
+ - - ">="
118
+ - !ruby/object:Gem::Version
119
+ version: 2.7.0
120
+ required_rubygems_version: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - ">="
123
+ - !ruby/object:Gem::Version
124
+ version: '0'
125
+ requirements: []
126
+ rubygems_version: 3.3.26
127
+ signing_key:
128
+ specification_version: 4
129
+ summary: A system to enforce test coverage on each class
130
+ test_files: []