rspec-covers 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f8c722d38cbc69d92194560b7f3fa27b7044042867b9f2fda68a1922a59d00ad
+  data.tar.gz: 1c0e8fba14560d37903695f4f13c970cca371cadc05288d51c84390b40f64582
+SHA512:
+  metadata.gz: 27469f2a735415ff24ad8c21b9ff6517cf4a74a7c7eb948926122dfe7717c5b9e58161e5966e5ab414134d3f12c1067826ac3cfc50e99eb426f44c545bed58ab
+  data.tar.gz: 6f956835839bcb164de4695dde6322adff74db05c5ba80a0dd66229fbe7ee80a520472fb3d7deedb8521529f50dcecee42838ca025abbc7a9b03a8b6d7a9904a

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Yudai Takada
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,202 @@
+# rspec-covers
+
+`rspec-covers` adds coverage intent metadata to RSpec examples.
+
+It lets a spec declare the production code it is responsible for with `covers`,
+declare allowed dependencies with `uses`, and optionally fail examples that
+execute production code outside that declaration.
+
+## Installation
+
+Add the gem to your application's Gemfile:
+
+```ruby
+gem "rspec-covers"
+```
+
+## Usage
+
+Configure the integration from `spec_helper.rb`:
+
+```ruby
+require "rspec/covers"
+
+RSpec.configure do |config|
+  RSpec::Covers.configure(config) do |covers|
+    covers.strict = true
+    covers.risky = :fail
+    covers.mode = :executed
+    covers.undeclared = :warn
+    covers.production_paths = %w[app lib]
+    covers.allowlist = %w[app/models/application_record.rb lib/instrumentation/**]
+    covers.suggest = true
+    covers.validate_declarations = true
+    covers.report_path = ".rspec-covers/result.json"
+  end
+end
+```
+
+Declare intent on example groups or examples:
+
+```ruby
+RSpec.describe UserCreator, covers: UserCreator do
+  it "creates a user", covers: "UserCreator#call" do
+    expect(UserCreator.new(params).call).to be_persisted
+  end
+
+  it "notifies", covers: "UserCreator#call", uses: [Notifier, "User.create!"] do
+    expect(UserCreator.new(params).call).to be_success
+  end
+end
+```
+
+`covers` and `uses` accept:
+
+- constants such as `UserCreator`
+- method strings such as `"UserCreator#call"` or `"User.create!"`
+- file paths or globs such as `"app/services/**/*.rb"`
+- regular expressions matched against production method labels and paths
+
+## Strict Coverage
+
+For each example, `rspec-covers` records per-example line coverage with
+`Coverage.peek_result`.
+
+When `strict` is enabled, an example is risky if it executes production lines
+that are not covered by its `covers` or `uses` declaration. Risky examples raise
+`RSpec::Covers::StrictCoverageError` by default.
+
+Set `covers.risky = :report` to keep the RSpec example green while still
+recording the example as risky in formatter output and JSON reports.
+
+Examples with no declaration follow `covers.undeclared`:
+
+- `:ignore` ignores them
+- `:warn` prints a warning
+- `:risky` fails when production code is executed
+
+## Suggestions
+
+With `covers.suggest = true`, undeclared examples receive ranked `covers`
+suggestions. The scorer combines:
+
+- `NC`: spec file or described class naming convention
+- `NCC`: token overlap between example text and method labels
+- `LCBA`: the production call immediately before `expect(...)`
+- `dynamic`: suite-wide tf-idf over production calls
+
+Run:
+
+```bash
+rake covers:suggest[.rspec-covers/result.json]
+```
+
+With `covers.validate_declarations = true`, examples that already declare
+`covers` are also scored. Weakly supported declarations are written to
+`declaration_validation` in the JSON report and shown by the formatter.
+
+## Checked Mode
+
+`covers.mode = :checked` narrows executed coverage to production methods that
+contributed to expectations. It tracks production method return values passed to
+`expect(...)` and the last production call before each expectation. This is an
+approximation of checked coverage, not a full dynamic slice.
+
+In checked mode, the JSON report includes `unchecked_locations`: production
+lines that ran during the example but were not attributed to expectation
+inputs.
+
+## Rake Tasks
+
+```bash
+rake covers:report[.rspec-covers/result.json]
+rake covers:suggest[.rspec-covers/result.json]
+rake covers:evaluate[ground_truth.json,predictions.json]
+```
+
+The JSON report also includes `orphan_methods`: production methods found from
+loaded constants and static Ripper scans of `production_paths` that were not
+named by any example's `covers` declaration.
+
+`covers:evaluate` reports traceability precision, recall, F1, and
+applicability for suggestions against a JSON ground-truth file. If the ground
+truth examples include `risky`, `seeded`, or `checked_locations`, it also
+reports strict coverage, seeded strict recall, and checked coverage metrics.
+Targets can be declared in the ground truth under `targets` and are reported as
+pass/fail comparisons.
+
+Ground truth can be a simple mapping:
+
+```json
+{
+  "examples": {
+    "spec/example_spec.rb[1:1]": ["UserCreator#call"]
+  }
+}
+```
+
+Or an array with strict and checked labels:
+
+```json
+{
+  "targets": {
+    "traceability.f1": 0.75,
+    "strict.recall": 0.90,
+    "checked.f1": 0.70
+  },
+  "examples": [
+    {
+      "id": "spec/example_spec.rb[1:1]",
+      "covers": ["UserCreator#call"],
+      "risky": true,
+      "seeded": true,
+      "checked_locations": [{ "file": "app/user_creator.rb", "line": 12 }]
+    }
+  ]
+}
+```
+
+Enable JSON formatter events with:
+
+```ruby
+covers.json_events = true
+```
+
+When `RSpec::Covers::Formatter` is active, it emits JSON Lines with
+`type: "rspec_covers.example"` for each recorded example.
+
+## Minitest
+
+The core coverage engine can also be attached to Minitest:
+
+```ruby
+require "minitest/covers"
+
+Minitest::Covers.configure do |covers|
+  covers.strict = true
+end
+
+class UserCreatorTest < Minitest::Test
+  covers "UserCreator#call"
+  uses "Notifier.deliver", target: :test_creates_user
+
+  def test_creates_user
+    assert UserCreator.new(params).call
+  end
+end
+```
+
+Class-level `covers` / `uses` apply to all tests in the class. Pass
+`target: :test_name` for method-level Minitest metadata.
+In checked mode, Minitest `assert(value)` records `value` as the expectation
+actual.
+
+## Notes
+
+Line ranges use `RubyVM::AbstractSyntaxTree` on CRuby and fall back to
+`method_source` when the VM does not expose AST ranges. Coverage from threads
+created inside an example belongs to the same process and may appear in that
+example's evidence.
+
+If SimpleCov or another tool has already started Ruby coverage, `rspec-covers`
+uses `Coverage.peek_result` deltas and does not restart coverage.

data/data/evaluation/ground_truth.example.json

@@ -0,0 +1,26 @@
+{
+  "targets": {
+    "traceability.f1": 0.75,
+    "strict.recall": 0.90,
+    "checked.f1": 0.70
+  },
+  "examples": [
+    {
+      "id": "spec/services/user_creator_spec.rb[1:1]",
+      "covers": ["UserCreator#call"],
+      "risky": false,
+      "checked_locations": [
+        { "file": "app/services/user_creator.rb", "line": 12 }
+      ]
+    },
+    {
+      "id": "spec/services/user_creator_spec.rb[1:2]",
+      "covers": ["UserCreator#call"],
+      "risky": true,
+      "seeded": true,
+      "checked_locations": [
+        { "file": "app/services/user_creator.rb", "line": 18 }
+      ]
+    }
+  ]
+}

data/lib/minitest/covers.rb

@@ -0,0 +1,133 @@
+# frozen_string_literal: true
+
+require "minitest"
+require "rspec/covers"
+
+module Minitest
+  module Covers
+    class ExampleAdapter
+      attr_reader :metadata, :full_description
+
+      def initialize(test)
+        @test = test
+        file, line = source_location
+        @metadata = test.class.rspec_covers_metadata_for(test.name).merge(
+          id: id,
+          file_path: file,
+          line_number: line
+        )
+        @full_description = "#{test.class}##{test.name}"
+      end
+
+      def id
+        "#{@test.class}##{@test.name}"
+      end
+
+      def exception
+        @test.failures.first if @test.respond_to?(:failures)
+      end
+
+      private
+
+      def source_location
+        @test.method(@test.name).source_location || ["(unknown)", 0]
+      rescue NameError
+        ["(unknown)", 0]
+      end
+    end
+
+    module ClassMethods
+      def covers(*items, target: nil)
+        rspec_covers_metadata_store[metadata_key(target)][:covers].concat(items)
+      end
+
+      def uses(*items, target: nil)
+        rspec_covers_metadata_store[metadata_key(target)][:uses].concat(items)
+      end
+
+      def rspec_covers_metadata_for(test_name)
+        metadata = superclass.respond_to?(:rspec_covers_metadata_for) ? superclass.rspec_covers_metadata_for(nil) : base_metadata
+        own = rspec_covers_metadata_store.fetch(:__class__, base_metadata)
+        method_metadata = rspec_covers_metadata_store.fetch(metadata_key(test_name), base_metadata)
+
+        merge_metadata(metadata, own, method_metadata)
+      end
+
+      private
+
+      def rspec_covers_metadata_store
+        @rspec_covers_metadata_store ||= Hash.new { |hash, key| hash[key] = base_metadata }
+      end
+
+      def metadata_key(test_name)
+        test_name&.to_sym || :__class__
+      end
+
+      def base_metadata
+        { covers: [], uses: [] }
+      end
+
+      def merge_metadata(*items)
+        {
+          covers: items.flat_map { |item| item[:covers] },
+          uses: items.flat_map { |item| item[:uses] }
+        }
+      end
+    end
+
+    module InstanceMethods
+      def assert(test, msg = nil)
+        RSpec::Covers.record_expectation_actual(test)
+        super
+      end
+
+      def before_setup
+        Thread.current[:rspec_covers_expectation_object_ids] = []
+        @rspec_covers_before_snapshot = RSpec::Covers.coverage_probe.snapshot
+        RSpec::Covers.call_log_probe.start_example if call_log_enabled?
+        super
+      end
+
+      def after_teardown
+        super
+      ensure
+        begin
+          call_log = call_log_enabled? ? RSpec::Covers.call_log_probe.stop_example : empty_call_log
+          after_snapshot = RSpec::Covers.coverage_probe.snapshot
+          executed = RSpec::Covers.coverage_probe.difference(@rspec_covers_before_snapshot || {}, after_snapshot)
+
+          RSpec::Covers.process_example(ExampleAdapter.new(self), executed, call_log)
+        ensure
+          Thread.current[:rspec_covers_expectation_object_ids] = nil
+        end
+      end
+
+      private
+
+      def call_log_enabled?
+        config = RSpec::Covers.configuration
+        config.suggest || config.mode == :checked || config.validate_declarations
+      end
+
+      def empty_call_log
+        RSpec::Covers::Probe::CallLog.new(calls: [], returns_by_object_id: {})
+      end
+    end
+
+    module_function
+
+    def configure
+      yield RSpec::Covers.configuration if block_given?
+      install!
+      RSpec::Covers.configuration
+    end
+
+    def install!
+      return if @installed
+
+      ::Minitest::Test.extend(ClassMethods)
+      ::Minitest::Test.prepend(InstanceMethods)
+      @installed = true
+    end
+  end
+end

data/lib/rspec/covers/checked_coverage.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require "set"
+
+require_relative "production_inventory"
+
+module RSpec
+  module Covers
+    class CheckedCoverage
+      def initialize(config)
+        @inventory = ProductionInventory.new(config)
+      end
+
+      def filter(executed_locations:, call_log:, expectation_object_ids:)
+        object_ids = (expectation_object_ids + call_log.expectation_object_ids).uniq
+        return Set.new if object_ids.empty? && call_log.labels_before_expectations.empty?
+
+        checked_labels = (call_log.return_labels_for(object_ids) + call_log.labels_before_expectations).uniq
+        checked_locations = @inventory.methods_by_label(checked_labels).each_with_object(Set.new) do |entry, set|
+          set.merge(entry.region.locations)
+        end
+
+        Set.new(executed_locations).select { |location| checked_locations.include?(location) }.to_set
+      end
+    end
+  end
+end

data/lib/rspec/covers/code_location.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require "set"
+
+module RSpec
+  module Covers
+    CodeLocation = Struct.new(:file, :line, keyword_init: true) do
+      def initialize(file:, line:)
+        super(file: File.expand_path(file), line: line.to_i)
+      end
+
+      def to_h
+        { file: file, line: line }
+      end
+    end
+
+    class CodeRegion
+      attr_reader :file, :lines, :label
+
+      def initialize(file:, lines:, label:)
+        @file = File.expand_path(file)
+        @lines = Set.new(lines.map(&:to_i))
+        @label = label
+      end
+
+      def include?(location)
+        file == File.expand_path(location.file) && lines.include?(location.line)
+      end
+
+      def locations
+        lines.map { |line| CodeLocation.new(file: file, line: line) }.to_set
+      end
+
+      def to_h
+        {
+          file: file,
+          lines: lines.to_a.sort,
+          label: label
+        }
+      end
+
+      def self.file(file, label: file)
+        line_count = File.exist?(file) ? File.readlines(file).length : 0
+
+        new(file: file, lines: 1..line_count, label: label)
+      end
+    end
+  end
+end

data/lib/rspec/covers/configuration.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+require "pathname"
+
+module RSpec
+  module Covers
+    class Configuration
+      VALID_MODES = %i[executed checked].freeze
+      VALID_UNDECLARED_POLICIES = %i[ignore warn risky].freeze
+      VALID_RISKY_POLICIES = %i[fail report].freeze
+
+      attr_accessor :strict, :allowlist, :suggest, :root, :production_paths,
+                    :suggestion_weights, :report_path, :suggestion_limit,
+                    :validate_declarations, :validation_threshold,
+                    :json_events
+
+      def initialize
+        @strict = false
+        @mode = :executed
+        @undeclared = :ignore
+        @allowlist = []
+        @suggest = false
+        @root = Dir.pwd
+        @production_paths = %w[app lib]
+        @suggestion_weights = {
+          nc: 1.0,
+          ncc: 1.0,
+          lcba: 1.0,
+          dynamic: 1.0
+        }
+        @suggestion_limit = 3
+        @report_path = nil
+        @validate_declarations = true
+        @validation_threshold = 0.2
+        @risky = :fail
+        @json_events = false
+      end
+
+      attr_reader :mode, :undeclared, :risky
+
+      def mode=(value)
+        normalized = value.to_sym
+        unless VALID_MODES.include?(normalized)
+          raise ArgumentError, "mode must be one of #{VALID_MODES.join(", ")}"
+        end
+
+        @mode = normalized
+      end
+
+      def undeclared=(value)
+        normalized = value.to_sym
+        unless VALID_UNDECLARED_POLICIES.include?(normalized)
+          raise ArgumentError, "undeclared must be one of #{VALID_UNDECLARED_POLICIES.join(", ")}"
+        end
+
+        @undeclared = normalized
+      end
+
+      def risky=(value)
+        normalized = value.to_sym
+        unless VALID_RISKY_POLICIES.include?(normalized)
+          raise ArgumentError, "risky must be one of #{VALID_RISKY_POLICIES.join(", ")}"
+        end
+
+        @risky = normalized
+      end
+
+      def production_file?(path)
+        absolute = File.expand_path(path)
+        return false unless absolute.start_with?("#{expanded_root}#{File::SEPARATOR}")
+        return false unless File.extname(absolute) == ".rb"
+
+        relative = relative_path(absolute)
+        return false if relative.start_with?("spec/", "test/", "tmp/", ".")
+        return false if allowlisted?(absolute)
+
+        production_paths.any? { |pattern| production_path_match?(relative, pattern) }
+      end
+
+      def production_files
+        production_paths.flat_map { |path| files_for_production_path(path) }
+                        .map { |path| File.expand_path(path) }
+                        .select { |path| production_file?(path) }
+                        .uniq
+      end
+
+      def allowlisted?(path)
+        absolute = File.expand_path(path)
+        relative = relative_path(absolute)
+
+        Array(allowlist).any? do |pattern|
+          expanded = File.expand_path(pattern.to_s, expanded_root)
+
+          File.fnmatch?(pattern.to_s, relative, File::FNM_PATHNAME | File::FNM_EXTGLOB) ||
+            File.fnmatch?(expanded, absolute, File::FNM_PATHNAME | File::FNM_EXTGLOB)
+        end
+      end
+
+      def relative_path(path)
+        Pathname.new(File.expand_path(path)).relative_path_from(Pathname.new(expanded_root)).to_s
+      rescue ArgumentError
+        File.expand_path(path)
+      end
+
+      private
+
+      def expanded_root
+        File.expand_path(root)
+      end
+
+      def production_path_match?(relative, pattern)
+        pattern = pattern.to_s.delete_suffix("/")
+
+        relative == pattern ||
+          relative.start_with?("#{pattern}/") ||
+          File.fnmatch?(pattern, relative, File::FNM_PATHNAME | File::FNM_EXTGLOB)
+      end
+
+      def files_for_production_path(path)
+        pattern = path.to_s
+        absolute = File.expand_path(pattern, expanded_root)
+
+        if pattern.match?(/[*?\[\]{}]/)
+          Dir.glob(absolute)
+        elsif File.directory?(absolute)
+          Dir.glob(File.join(absolute, "**", "*.rb"))
+        elsif File.file?(absolute)
+          [absolute]
+        else
+          Dir.glob(File.join(absolute, "**", "*.rb"))
+        end
+      end
+    end
+  end
+end