rspec-cover_it 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 528de4fe406f539d93d8632624714721d8cded1463a35776c16423d387c13c30
+  data.tar.gz: 57fc958451fb2dfeafb94aaaed56aba38befd938a4ea24d2f82d0ac09a46ee4e
+SHA512:
+  metadata.gz: 6c636b00bd14fd976d53f9f97cbd2d53334ecb5677ad480ef00e737b51144da2f0ae76fffc1f6d1fb0366381961311e8aec93dfe6453cc2942c10dc652d87a88
+  data.tar.gz: 93c2432a8ce2725b2ca3ff9f1c8115a38154dc7c4426f7aa5667d12ed9546b9890d83885bcbe727064ea78972e58ab4e721b6dcef82d93c004d95f8a8ae6609f

data/.gitignore

@@ -0,0 +1,6 @@
+.ruby-version
+.ruby-gemset
+Gemfile.lock
+*.gem
+coverage/
+tmp/*

data/Gemfile

@@ -0,0 +1,3 @@
+source "https://rubygems.org"
+
+gemspec

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright 2023 Eric Mueller
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,105 @@
+# RSpec::CoverIt
+
+Still a work in progress, but this gem is intended to give us a simpler and
+more _targeted_ way to manage coverage. The usual system for a large project
+is a single overarching SimpleCov target; with good tooling, there will be
+something checking that all of the lines of each PR get covered by the tests
+in that PR, and there are checks that the overall coverage number doesn't drop
+too far.. but it's a _sloppy, messy, lossy_ approach. Code loses coverage,
+code gets added with no tests, tests get removed and other code becomes less
+covered, and a lot of the time, that's because the _coverage is a lie_.
+
+Your coverage tool is telling you.. how many times each line of code _got run_
+during your entire test suite. But it's not telling you what is doing the
+running, which means that often large swathes of code are actually only covered
+incidentally, through tests that aren't intended to exercise that code, but
+_happens to_.
+
+This approach was moderately inspired by https://github.com/jamesdabbs/rspec-coverage,
+which was itself apparently inspired by a rubyconf 2016 talk by Ryan Davis. That
+talk and library are mostly concerned with making sure that we don't _overcount_
+coverage though, while this one has three goals:
+
+1. Make coverage enforcement simpler
+1. Enforce coverage _in the test suite_ (or nearly so)
+1. Make coverage enforcement more local and specific.
+
+## Tentative Usage
+
+Note that this bit isn't really implemented, but is more of an outline of how I
+intend the library to work. This is currently a spike, and not a functioning gem.
+
+You set up `RSpec::CoverIt` in your `spec_helper.rb` - require `rspec/cover_it`,
+and then `RSpec::CoverIt.setup(**config_options)` (before loading your code, as
+you would with any other coverage library). It's.. _semi_ compatible with other
+coverage systems - it only starts Coverage if it's not already running, and it
+only uses `peek_result`, so it doesn't affect other systems outcomes. But if you
+setup SimpleCov to do branch-coverage or something, that will break this gem,
+for now (we assume basic line-coverage style results).
+
+Rough configuration options:
+
+* `filter`: Only paths starting with this (matching this regex?) can matter
+* `autoenforce`: off by default - with this disabled, you turn on coverage
+  enforcement for a given top-level describe ExampleGroup by adding metadata
+  like `cover_it: true` or `cover_it: 95`. If it's enabled though, you instead
+  can turn enforcement _off_ for an example group by setting `cover_it: false`.
+* `global_threshold`: 100% by default. This whole "90% coverage is the right
+  target" thing is mostly a side-effect of the way we check the entire project
+  under one number.. but it's an easy setting to support, and I'm sure people
+  will disagree with me.
+
+## Example Group Metadata
+
+In example groups, you can use metadata to control the behavior of
+`rspec-cover_it`. These keys have meaning:
+
+* `cover_it`: if boolean, it enables or disables the coverage enforcement for
+  this ExampleGroup. If numeric, it's enabling, and specifying the coverage
+  threshold at the same time (as a percentage - `cover_it: 95` requires 95%
+  coverage of the target class by this example group).
+* `covers`: An array of classes and modules that this example groups _thinks
+  it is completely testing_. Ideally, you'd have a separate test file for each,
+  but sometimes that's hard to do - you can let one spec claim responsibility
+  for multiple classes and modules (such as Concerns) using this. Be default
+  it is just `[described_class]`. Additionally, if your top-level example
+  group _does not describe a Class or Module_, you may use `covers` to let it
+  invoke `rspec-cover_it` anyway.
+
+## Implementation
+
+We record the coverage information in a `before(:suite)` rspec hook using
+`Coverage.peek_result`, and hold onto that information. Then before and after
+each 'context' (which really amounts to 'each spec file'), we grab the coverage
+information again.
+
+We use `Object.const_source_location` to find the file that defines the
+`described_class`, and _that_ is what is being assessed for coverage. This
+means that, if you are reopening the class somewhere else, that coverage won't
+be checked; if you are including 15 Concerns, and don't intend to write separate
+specs for them, be sure to list them as `covers:` metadata on the test. Also,
+shame!.
+
+## Drawbacks and Shortcomings
+
+There's nothing in here that stops you from failing to write tests for a class
+at all! If you're using SimpleCov and you've got 100% coverage already, that's
+one of the benefits.. I could pretty reasonably include some kind of
+`after(:suite)` hook that optionally checks net coverage, but.. simplecov does
+that, and the concurrent-testing game makes this a _painful_ topic in reality.
+That's not the goal here, and I'm not going to worry about it.
+
+As initially implemented, it fails your tests if you don't run the entire test
+file. `rspec spec/foo_spec.rb:32` will error, because .. running only one of
+your tests _doesn't cover the class_. I have a solution for this, but it uses
+some non-public bits of RSpec, so I'm trying to find a better answer first.
+(Conversation started in their
+[issue tracker](https://github.com/rspec/rspec-core/issues/3037))
+
+So far, I haven't found a _great_ way to report that there's a problem. The
+output from having insufficient coverage is _raising an exception from an
+`after(:context)` hook_, which RSpec rescues and formats for itself, and there
+aren't a lot of controls - it works, but it's a bit ugly and doesn't really
+give the right immediate impression. I'm contemplating using an `after(:suite)`
+hook and aggregating them myself, but at the end of the day RSpec is in control
+of the output stream, and we don't entirely fit its metaphor.

data/lib/rspec/cover_it/context.rb

@@ -0,0 +1,33 @@
+module RSpec
+  module CoverIt
+    class Context
+      def initialize(scope:, rspec_context:)
+        @scope, @rspec_context = scope, rspec_context
+      end
+
+      def cover_it?
+        target_class && metadata.fetch(:cover_it, nil)
+      end
+
+      def target_path
+        Object.const_source_location(target_class_name).first
+      end
+
+      def target_class
+        metadata.fetch(:described_class, nil)
+      end
+
+      def target_class_name
+        target_class.name
+      end
+
+      private
+
+      attr_reader :scope, :rspec_context
+
+      def metadata
+        scope.metadata
+      end
+    end
+  end
+end

data/lib/rspec/cover_it/context_coverage.rb

@@ -0,0 +1,54 @@
+module RSpec
+  module CoverIt
+    class ContextCoverage
+      def initialize(context:, pretest_results:)
+        @context, @pretest_results = context, pretest_results
+        @precontext_coverage = @postcontext_coverage = nil
+      end
+
+      attr_accessor :precontext_coverage, :postcontext_coverage
+
+      def local_coverage
+        return nil unless precontext_coverage && postcontext_coverage
+        @_local_coverage ||= pretest_coverage
+          .zip(precontext_coverage, postcontext_coverage)
+          .map { |a, b, c| line_calculation(a, b, c) }
+      end
+
+      def local_coverage_rate
+        return nil unless covered_line_count
+        covered_line_count.to_f / coverable_line_count.to_f
+      end
+
+      private
+
+      attr_reader :context, :pretest_results, :precontext_coverage, :postcontext_coverage
+
+      def target_path
+        @_target_path ||= context.target_path
+      end
+
+      def pretest_coverage
+        @_pretest_coverage ||= pretest_results[target_path]
+      end
+
+      # Really, we shouldn't see nil for any of these values unless they are all
+      # nil. We want the coverage we'd expect to have seen if we ran _just_ this
+      # groups of examples, which ought boe the pretest coverage, plus the
+      # number of times each line was run _during_ the context.
+      def line_calculation(pretest, precontext, postcontext)
+        return nil if pretest.nil? || precontext.nil? || postcontext.nil?
+        pretest + (postcontext - precontext)
+      end
+
+      def coverable_line_count
+        pretest_coverage.compact.count
+      end
+
+      def covered_line_count
+        return nil unless local_coverage
+        local_coverage.count { |executions| executions && executions > 0 }
+      end
+    end
+  end
+end

data/lib/rspec/cover_it/coverage_state.rb

@@ -0,0 +1,53 @@
+module RSpec
+  module CoverIt
+    class CoverageState
+      attr_reader :filter
+
+      def initialize(filter:)
+        @filter = filter
+        @pretest_results = nil
+        @context_coverages = {}
+      end
+
+      # If SimpleCov or something like it is already running, Coverage may be
+      # started already. For our purposes, that should be fine.
+      def start_tracking
+        Coverage.start unless Coverage.running?
+      end
+
+      def finish_load_tracking
+        @pretest_results = PretestCoverage.new(filter: filter, results: Coverage.peek_result)
+      end
+
+      def start_tracking_for(scope, rspec_context)
+        context = Context.new(scope: scope, rspec_context: rspec_context)
+        return unless context.cover_it?
+
+        context_coverage_for(context).tap do |context_coverage|
+          context_coverage.precontext_coverage = Coverage.peek_result[context.target_path]
+        end
+      end
+
+      def finish_tracking_for(scope, rspec_context)
+        context = Context.new(scope: scope, rspec_context: rspec_context)
+        return unless context.cover_it?
+
+        context_coverage_for(context).tap do |context_coverage|
+          context_coverage.postcontext_coverage = Coverage.peek_result[context.target_path]
+          fail("missing coverage!") if context_coverage.local_coverage_rate < 1.0
+        end
+      end
+
+      private
+
+      attr_reader :pretest_results
+
+      def context_coverage_for(context)
+        @context_coverages[context.target_class] ||= ContextCoverage.new(
+          context: context,
+          pretest_results: pretest_results
+        )
+      end
+    end
+  end
+end

data/lib/rspec/cover_it/pretest_coverage.rb

@@ -0,0 +1,17 @@
+module RSpec
+  module CoverIt
+    class PretestCoverage
+      def initialize(filter:, results:)
+        @filter = filter
+        @results = results
+          .select { |k, _v| k.start_with?(filter) }
+          .map { |k, v| [k, v.dup] }
+          .to_h
+      end
+
+      def [](path)
+        @results[path]
+      end
+    end
+  end
+end

data/lib/rspec/cover_it/version.rb

@@ -0,0 +1,5 @@
+module RSpec
+  module CoverIt
+    VERSION = "0.0.2"
+  end
+end

data/lib/rspec/cover_it.rb

@@ -0,0 +1,30 @@
+require "coverage"
+
+module RSpec
+  module CoverIt
+    Error = Class.new(StandardError)
+    MissingCoverage = Class.new(Error)
+  end
+end
+
+glob = File.expand_path("../cover_it/*.rb", __FILE__)
+Dir.glob(glob).sort.each { |f| require(f) }
+
+module RSpec
+  module CoverIt
+    class << self
+      attr_accessor :state
+    end
+
+    def self.setup(filter:)
+      RSpec::CoverIt.state = CoverageState.new(filter: filter)
+      RSpec::CoverIt.state.start_tracking
+
+      RSpec.configure do |config|
+        config.prepend_before(:suite) { RSpec::CoverIt.state.finish_load_tracking }
+        config.prepend_before(:context) { |context| RSpec::CoverIt.state.start_tracking_for(self.class, context) }
+        config.append_after(:context) { |context| RSpec::CoverIt.state.finish_tracking_for(self.class, context) }
+      end
+    end
+  end
+end

data/rspec-cover_it.gemspec

@@ -0,0 +1,36 @@
+require_relative "lib/rspec/cover_it/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "rspec-cover_it"
+  spec.version = RSpec::CoverIt::VERSION
+  spec.authors = ["Eric Mueller"]
+  spec.email = ["nevinera@gmail.com"]
+
+  spec.summary = "A system to enforce test coverage on each class"
+  spec.description = <<~DESC
+    We're all used to tools that enforce _total_ coverage numbers, but this gem
+    tries for something different. Instead of keeping your whole project above
+    some threshold, we treat the coverage of _each class_ as a testable quality
+    and then enforce that coverage as part of the test suite!
+  DESC
+  spec.homepage = "https://github.com/nevinera/rspec-cover_it"
+  spec.license = "MIT"
+  spec.required_ruby_version = Gem::Requirement.new(">= 2.7.0")
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+
+  spec.require_paths = ["lib"]
+  spec.files = Dir.chdir(File.expand_path(__dir__)) do
+    `git ls-files -z`
+      .split("\x0")
+      .reject { |f| f.start_with?("spec") }
+  end
+
+  spec.add_dependency "rspec", "~> 3.10"
+
+  spec.add_development_dependency "pry", "~> 0.14"
+  spec.add_development_dependency "standard", "~> 1.28"
+  spec.add_development_dependency "mdl", "~> 0.12"
+  spec.add_development_dependency "quiet_quality", "~> 1.2"
+end

metadata

@@ -0,0 +1,130 @@
+--- !ruby/object:Gem::Specification
+name: rspec-cover_it
+version: !ruby/object:Gem::Version
+  version: 0.0.2
+platform: ruby
+authors:
+- Eric Mueller
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2023-06-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.14'
+- !ruby/object:Gem::Dependency
+  name: standard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.28'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.28'
+- !ruby/object:Gem::Dependency
+  name: mdl
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.12'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.12'
+- !ruby/object:Gem::Dependency
+  name: quiet_quality
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+description: |
+  We're all used to tools that enforce _total_ coverage numbers, but this gem
+  tries for something different. Instead of keeping your whole project above
+  some threshold, we treat the coverage of _each class_ as a testable quality
+  and then enforce that coverage as part of the test suite!
+email:
+- nevinera@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE
+- README.md
+- lib/rspec/cover_it.rb
+- lib/rspec/cover_it/context.rb
+- lib/rspec/cover_it/context_coverage.rb
+- lib/rspec/cover_it/coverage_state.rb
+- lib/rspec/cover_it/pretest_coverage.rb
+- lib/rspec/cover_it/version.rb
+- rspec-cover_it.gemspec
+homepage: https://github.com/nevinera/rspec-cover_it
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/nevinera/rspec-cover_it
+  source_code_uri: https://github.com/nevinera/rspec-cover_it
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.3.26
+signing_key:
+specification_version: 4
+summary: A system to enforce test coverage on each class
+test_files: []