rspec-tracer 0.8.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 675be2bd6603501deb282508ea234808f0bbdf60c2b626611919f84c32bb4ba9
-  data.tar.gz: 591be21b4e97c50f33385111ac758665a97e83ca82e154cd3b86d32133f7ef52
+  metadata.gz: 1df1d0c90323ae123f24629f74f654bce0fff93bba8e8b052388d582dfb64d8f
+  data.tar.gz: 5caabede1fb71ffcbbd70f0e9bfe8ca355303e843dbb99cacf14f146149afe5a
 SHA512:
-  metadata.gz: 1ab8ca453da65dd7185fa4869633d9b9ab85fb5c73cfb45929928a492d58d0e16d24e3bcef2466ceae4408c7e1ad2e5ab3f10137061ab684b1decf885809607d
-  data.tar.gz: 8dc1d3078f2c16c8ee5142c03d05222dd0bd458c979bca0a35ee1b1e44fead38ed2c1a6d84795a93f2a4d7a15c58da40562f4dae360f22e65f1518fc0ddecf75
+  metadata.gz: 5ad9ec360d867ed745039fc2cf2a4aef314912a5b6a596c8c79e9d6615b00f2b578f5d3a164c2725ed710d9b0aa055855c607eebe52f7b032d1b959d483cb079
+  data.tar.gz: 347236ad3e59372ff1cf33aa3c812ae5a22911ff6b46bfdfced44b96ef80161c4da0798a9afecf96846bc44963713bac4aa625c714266431b975c3152ebd92b1

data/CHANGELOG.md

@@ -1,3 +1,11 @@
+## [0.9.0] - 2021-09-15
+
+### Added
+
+- Handling all examples filtered by RSpec (#34)
+- Warn on incorrect analysis to stop using RSpec Tracer (#35)
+- Run `SimpleCov.at_exit` hook (#36)
+
 ## [0.8.0] - 2021-09-13
 
 ### Fixed

data/README.md

@@ -40,6 +40,7 @@ describing the **intention** and implementation details of **managing dependency
 * [Advanced Configuration](#advanced-configuration)
 * [Filters](#filters)
 * [Environment Variables](#environment-variables)
+* [When Should You Not Use RSpec Tracer](#when-should-you-not-use-rspec-tracer)
 
 ## Demo
 
@@ -93,7 +94,7 @@ These reports provide information on the total number of tests that will run aft
 
 1. Add this line to your `Gemfile` and `bundle install`:
     ```ruby
-    gem 'rspec-tracer', '~> 0.7', group: :test, require: false
+    gem 'rspec-tracer', '~> 0.9', group: :test, require: false
     ```
 
     And, add the followings to your `.gitignore`:
@@ -124,10 +125,8 @@ any of the application code.**
     RSpecTracer.start
     ```
 
-    Currently using RSpec Tracer with SimpleCov has the following two limitations:
-
-    - SimpleCov **won't be able to provide branch coverage report** even when enabled.
-    - RSpec Tracer **nullifies the `SimpleCov.at_exit`** callback.
+    If you use RSpec Tracer with SimpleCov, then **SimpleCov would not report branch
+    coverage results even when enabled**.
 
 3. After running your tests, open `rspec_tracer_report/index.html` in the browser
 of your choice.
@@ -257,52 +256,48 @@ end
 ### Defining Custom Filteres
 
 You can currently define a filter using either a String or Regexp (that will then
-be Regexp-matched against each source file's path), a block or by passing in your
-own Filter class.
-
-#### String Filter
-
-```ruby
-RSpecTracer.start do
-  add_filter '/helpers/'
-end
-```
-
-This simple string filter will remove all files that match "/helpers/" in their path.
+be Regexp-matched against each source file's name relative to the project root),
+a block or by passing in your own Filter class.
 
-#### Regex Filter
-
-```ruby
-RSpecTracer.start do
-  add_filter %r{^/helpers/}
-end
-```
-
-This simple regex filter will remove all files that start with /helper/ in their path.
-
-#### Block Filter
+- **String Filter**: The string filter matches files that have the given string
+in their name. For example, the following string filter will remove all files that
+have `"/helpers/"` in their name.
+  ```ruby
+  RSpecTracer.start do
+    add_filter '/helpers/'
+  end
+  ```
 
-```ruby
-RSpecTracer.start do
-  add_filter do |source_file|
-    source_file[:file_path].include?('/helpers/')
+- **Regex Filter**: The regex filter removes all files that have a successful name
+match against the given regex expression. This simple regex filter will remove
+all files that start with `%r{^/helper/}` in their name:
+  ```ruby
+  RSpecTracer.start do
+    add_filter %r{^/helpers/}
   end
-end
-```
+  ```
 
-Block filters receive a `Hash` object and expect your block to return either true
-(if the file is to be removed from the result) or false (if the result should be kept).
-In the above example, the filter will remove all files that match "/helpers/" in their path.
+- **Block Filter**: Block filters receive a `Hash` object and expect your block
+to return either **true** (if the file is to be removed from the result) or **false**
+(if the result should be kept). In the below example, the filter will remove all
+files that match `"/helpers/"` in their path.
+  ```ruby
+  RSpecTracer.start do
+    add_filter do |source_file|
+      source_file[:file_path].include?('/helpers/')
+    end
+  end
+  ```
 
-#### Array Filter
+  You can also use `source_file[:name]` to define the return value of the block
+  filter for the given source file.
 
-```ruby
-RSpecTracer.start do
-  add_filter ['/helpers/', %r{^/utils/}]
-end
-```
-
-You can pass in an array containing any of the other filter types.
+- **Array Filter**: You can pass in an array containing any of the other filter types:
+  ```ruby
+  RSpecTracer.start do
+    add_filter ['/helpers/', %r{^/utils/}]
+  end
+  ```
 
 ## Environment Variables
 
@@ -340,6 +335,122 @@ specific test suites and not merge them.
   TEST_SUITE_ID=2 bundle exec rspec spec/helpers
   ```
 
+## When Should You Not Use RSpec Tracer
+
+To uniquely identify the examples is one of the requirements for the correctness
+of the RSpec Tracer. Sometimes, it would not be possible to do so depending upon
+how we have written the specs. The following attributes determine the uniqueness:
+
+- The example group
+- The example full description
+- The spec file location, i.e., file name and line number
+- All the shared examples and contexts
+
+Consider the following `Calculator` module:
+```ruby
+module Calculator
+  module_function
+
+  def add(a, b) a + b; end
+  def sub(a, b) a - b; end
+  def mul(a, b) a * b; end
+end
+```
+
+And the corresponding spec file `spec/calculator_spec.rb`:
+```ruby
+RSpec.describe Calculator do
+  describe '#add' do
+    [
+      [1, 2, 3],
+      [0, 0, 0],
+      [5, 32, 37],
+      [-1, -8, -9],
+      [10, -10, 0]
+    ].each { |a, b, r| it { expect(described_class.add(a, b)).to eq(r) } }
+  end
+
+  describe '#sub' do
+    [
+      [1, 2, -1],
+      [10, 0, 10],
+      [37, 5, 32],
+      [-1, -8, 7],
+      [10, 10, 0]
+    ].each do |a, b, r|
+      it 'performs subtraction' do
+        expect(described_class.sub(a, b)).to eq(r)
+      end
+    end
+  end
+
+  describe '#mul' do
+    [
+      [1, 2, -2],
+      [10, 0, 0],
+      [5, 7, 35],
+      [-1, -8, 8],
+      [10, 10, 100]
+    ].each do |a, b, r|
+      it "multiplies #{a} and #{b} to #{r}" do
+        expect(described_class.mul(a, b)).to eq(r)
+      end
+    end
+  end
+end
+```
+
+Running the spec with `bundle exec rspec spec/calculator_spec.rb` generates the
+following output:
+```
+Calculator
+  #mul
+    multiplies 5 and 7 to 35
+    multiplies 10 and 10 to 100
+    multiplies 10 and 0 to 0
+    multiplies 1 and 2 to -2 (FAILED - 1)
+    multiplies -1 and -8 to 8
+  #add
+    example at ./spec/calculator_spec.rb:13
+    example at ./spec/calculator_spec.rb:13
+    example at ./spec/calculator_spec.rb:13
+    example at ./spec/calculator_spec.rb:13
+    example at ./spec/calculator_spec.rb:13
+  #sub
+    performs subtraction
+    performs subtraction
+    performs subtraction
+    performs subtraction
+    performs subtraction
+```
+
+In this scenario, RSpec Tracer cannot determine the `Calculator#add` and
+`Calculator#sub` group examples. Also, it will ask you not to use the gem unless
+you have made some changes to your spec files.
+
+```
+================================================================================
+                IMPORTANT NOTICE -- DO NOT USE RSPEC TRACER
+================================================================================
+    It would be best to make changes so that the RSpec tracer can uniquely
+    identify all the examples, and then you can enable the RSpec tracer back.
+================================================================================
+
+RSpec tracer could not uniquely identify the following 10 examples:
+  - Example ID: eabd51a899db4f64d5839afe96004f03 (5 examples)
+      * Calculator#add (spec/calculator_spec.rb:13)
+      * Calculator#add (spec/calculator_spec.rb:13)
+      * Calculator#add (spec/calculator_spec.rb:13)
+      * Calculator#add (spec/calculator_spec.rb:13)
+      * Calculator#add (spec/calculator_spec.rb:13)
+  - Example ID: 72171b502c5a42b9aa133f165cf09ec2 (5 examples)
+      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
+      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
+      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
+      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
+      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
+```
+
 ## Contributing
 
 Read the [contribution guide](https://github.com/avmnu-sng/rspec-tracer/blob/main/.github/CONTRIBUTING.md).

data/lib/rspec_tracer/reporter.rb

@@ -15,6 +15,7 @@ module RSpecTracer
 
     def register_example(example)
       @all_examples[example[:example_id]] = example
+      @duplicate_examples[example[:example_id]] << example
     end
 
     def on_example_skipped(example_id)
@@ -106,6 +107,17 @@ module RSpecTracer
       file_deleted?(file_name) || file_modified?(file_name)
     end
 
+    def incorrect_analysis?
+      @duplicate_examples.select! { |_, examples| examples.count > 1 }
+
+      return false if @duplicate_examples.empty?
+
+      print_not_use_notice
+      print_duplicate_examples
+
+      true
+    end
+
     def register_dependency(example_id, file_name)
       @dependency[example_id] << file_name
     end
@@ -169,6 +181,7 @@ module RSpecTracer
 
     def initialize_examples
       @all_examples = {}
+      @duplicate_examples = Hash.new { |examples, example_id| examples[example_id] = [] }
       @passed_examples = Set.new
       @possibly_flaky_examples = Set.new
       @flaky_examples = Set.new
@@ -226,6 +239,45 @@ module RSpecTracer
       @reverse_dependency = report.to_h
     end
 
+    def print_not_use_notice
+      justify = ' ' * 4
+      four_justify = justify * 4
+
+      puts '=' * 80
+      puts "#{four_justify}IMPORTANT NOTICE -- DO NOT USE RSPEC TRACER"
+      puts '=' * 80
+      puts "#{justify}It would be best to make changes so that the RSpec tracer can uniquely"
+      puts "#{justify}identify all the examples, and then you can enable the RSpec tracer back."
+      puts '=' * 80
+      puts
+    end
+
+    # rubocop:disable Metrics/AbcSize
+    def print_duplicate_examples
+      total = @duplicate_examples.sum { |_, examples| examples.length }
+
+      puts "RSpec tracer could not uniquely identify the following #{total} examples:"
+
+      justify = ' ' * 2
+      nested_justify = justify * 3
+
+      @duplicate_examples.each_pair do |example_id, examples|
+        puts "#{justify}- Example ID: #{example_id} (#{examples.count} examples)"
+
+        examples.each do |example|
+          description = example[:full_description].strip
+          file_name = example[:rerun_file_name].sub(%r{^/}, '')
+          line_number = example[:rerun_line_number]
+          location = "#{file_name}:#{line_number}"
+
+          puts "#{nested_justify}* #{description} (#{location})"
+        end
+      end
+
+      puts
+    end
+    # rubocop:enable Metrics/AbcSize
+
     def write_all_examples_report
       file_name = File.join(@cache_dir, 'all_examples.json')
 

data/lib/rspec_tracer/rspec_runner.rb

@@ -3,13 +3,23 @@
 module RSpecTracer
   module RSpecRunner
     # rubocop:disable Metrics/AbcSize
-    def run_specs(_example_groups)
+    def run_specs(example_groups)
       actual_count = RSpec.world.example_count
+      RSpecTracer.no_examples = actual_count.zero?
+
+      if RSpecTracer.no_examples
+        RSpecTracer.running = true
+
+        super(example_groups)
+
+        return
+      end
+
       starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      filtered_examples, example_groups = RSpecTracer.filter_examples
+      filtered_examples, filtered_example_groups = RSpecTracer.filter_examples
 
       RSpec.world.instance_variable_set(:@filtered_examples, filtered_examples)
-      RSpec.world.instance_variable_set(:@example_groups, example_groups)
+      RSpec.world.instance_variable_set(:@example_groups, filtered_example_groups)
 
       current_count = RSpec.world.example_count
       ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
@@ -23,7 +33,7 @@ module RSpecTracer
 
       RSpecTracer.running = true
 
-      super(example_groups)
+      super(filtered_example_groups)
     end
     # rubocop:enable Metrics/AbcSize
   end

data/lib/rspec_tracer/runner.rb

@@ -63,6 +63,10 @@ module RSpecTracer
       @reporter.register_deleted_examples(@cache.all_examples)
     end
 
+    def incorrect_analysis?
+      @reporter.incorrect_analysis?
+    end
+
     # rubocop:disable Metrics/AbcSize
     def generate_missed_coverage
       missed_coverage = Hash.new do |files_coverage, file_path|

data/lib/rspec_tracer/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RSpecTracer
-  VERSION = '0.8.0'
+  VERSION = '0.9.0'
 end

data/lib/rspec_tracer.rb

@@ -28,7 +28,7 @@ require_relative 'rspec_tracer/version'
 
 module RSpecTracer
   class << self
-    attr_accessor :running, :pid
+    attr_accessor :running, :pid, :no_examples
 
     def start(&block)
       RSpecTracer.running = false
@@ -74,7 +74,11 @@ module RSpecTracer
     def at_exit_behavior
       return unless RSpecTracer.pid == Process.pid && RSpecTracer.running
 
+      ::Kernel.exit(1) if runner.incorrect_analysis?
+
       run_exit_tasks
+    ensure
+      RSpecTracer.running = false
     end
 
     def start_example_trace
@@ -154,15 +158,11 @@ module RSpecTracer
     def setup_coverage
       @simplecov = defined?(SimpleCov) && SimpleCov.running
 
-      if simplecov?
-        # rubocop:disable Lint/EmptyBlock
-        SimpleCov.at_exit {}
-        # rubocop:enable Lint/EmptyBlock
-      else
-        require 'coverage'
+      return if simplecov?
 
-        ::Coverage.start
-      end
+      require 'coverage'
+
+      ::Coverage.start
     end
 
     def setup_trace_point
@@ -175,11 +175,13 @@ module RSpecTracer
     end
 
     def run_exit_tasks
-      generate_reports
+      if RSpecTracer.no_examples
+        puts 'Skipped reports generation since all examples were filtered out'
+      else
+        generate_reports
+      end
 
       simplecov? ? run_simplecov_exit_task : run_coverage_exit_task
-    ensure
-      RSpecTracer.running = false
     end
 
     def generate_reports
@@ -224,12 +226,13 @@ module RSpecTracer
 
       puts 'SimpleCov will now generate coverage report (<3 RSpec tracer)'
 
-      SimpleCov.result.format!
+      coverage_reporter.record_coverage if RSpecTracer.no_examples
     end
 
     def run_coverage_exit_task
       starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
 
+      coverage_reporter.record_coverage if RSpecTracer.no_examples
       coverage_reporter.generate_final_coverage
 
       file_name = File.join(RSpecTracer.coverage_path, 'coverage.json')

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rspec-tracer
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.9.0
 platform: ruby
 authors:
 - Abhimanyu Singh
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-09-13 00:00:00.000000000 Z
+date: 2021-09-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: docile
@@ -50,9 +50,11 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 3.6.0
-description: RSpec Tracer is a specs dependency analysis tool and a test skipper for
-  RSpec. It maintains a list of files for each test, enabling itself to skip tests
-  in the subsequent runs if none of the dependent files are changed.
+description: RSpec Tracer is a specs dependency analyzer, flaky tests detector, tests
+  accelerator, and coverage reporter tool for RSpec. It maintains a list of files
+  for each test, enabling itself to skip tests in the subsequent runs if none of the
+  dependent files are changed. It uses Ruby's built-in coverage library to keep track
+  of the coverage for each test
 email:
 - abhisinghabhimanyu@gmail.com
 executables: []
@@ -111,7 +113,7 @@ licenses:
 - MIT
 metadata:
   homepage_uri: https://github.com/avmnu-sng/rspec-tracer
-  source_code_uri: https://github.com/avmnu-sng/rspec-tracer/tree/v0.8.0
+  source_code_uri: https://github.com/avmnu-sng/rspec-tracer/tree/v0.9.0
   changelog_uri: https://github.com/avmnu-sng/rspec-tracer/blob/main/CHANGELOG.md
   bug_tracker_uri: https://github.com/avmnu-sng/rspec-tracer/issues
 post_install_message:
@@ -132,5 +134,6 @@ requirements: []
 rubygems_version: 3.2.26
 signing_key:
 specification_version: 4
-summary: RSpec Tracer is a specs dependency analysis tool and a test skipper for RSpec
+summary: RSpec Tracer is a specs dependency analyzer, flaky tests detector, tests
+  accelerator, and coverage reporter tool.
 test_files: []