rspec-tracer 0.4.0 → 0.6.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34f47b6527f2c8f3c1cb3524adf8d81a9fe7e3833c202605a206598dee8758b2
-  data.tar.gz: e647137006f5f8af18999ea8f2665c09d703bf2cd8ea1e84dec85e66f87ba5e2
+  metadata.gz: a86a9027964446fd10fc51379221d718e15699f6f93ba0c56af654273ccf632d
+  data.tar.gz: 1ade4a1b9a14b8cf0a10467889e0631d4b4e30871118734a6d2b4705138b19b2
 SHA512:
-  metadata.gz: 05c204bd00cfa15258a1770790360184c9481d670cc8ef3542b06024976a9e13cf3595e79ecefa0d22c8337d35dee9ef6963b145cf1aa16268f3de467c7d171b
-  data.tar.gz: b2f7e88fbf404793a9e2daaade7448d51bd63150701a6735c9ae8bdcb17eaf25c926bde411cd61bc5dd20133829d1576d16b7588336f05ec968ae4f3d82a5453
+  metadata.gz: c251e47b35e667c0b847c54bb7a7c96ad33c39b8c91f48bcaf96c06cb9b017155ed760398f24e382ee93c94fe8bf7b40289fd9c87518bbd467d42b2bae91daac
+  data.tar.gz: a314d23f2b7998cb64b5183a5a0da4d1d100039accfd28998ccdf15d451f633fb4181688951de400bbaf82095253f56d73e585632e411657ada157d0e285a1b0

data/CHANGELOG.md

@@ -1,3 +1,39 @@
+## [0.6.2] - 2021-09-07
+
+### Added
+
+Improvements towards reducing dependency and coverage processing time (#26)
+
+## [0.6.1] - 2021-09-06
+
+### Fixed
+
+Bug in time formatter (#24)
+
+### Added
+
+Environment variable to control verbose output (#25)
+
+## [0.6.0] - 2021-09-05
+
+### Added
+
+- Improved dependency change detection (#18)
+- Flaky tests detection (#19)
+- Exclude vendor files from analysis (#21)
+- Report elapsed time at various stages (#23)
+
+### Note
+
+The first run on this version will not use any cache on the CI because the number
+of files changed from eight to nine, so there will be no appropriate cache to use.
+
+## [0.5.0] - 2021-09-03
+
+### Fixed
+
+- Limit number of cached files download (#16)
+
 ## [0.4.0] - 2021-09-03
 
 ### Added

data/README.md

@@ -8,7 +8,11 @@ It uses [Ruby's built-in coverage library](https://ruby-doc.org/stdlib/libdoc/co
 to keep track of the coverage for each test. For each test executed, the coverage
 diff provides the desired file list. RSpec Tracer takes care of reporting the
 **correct code coverage when skipping tests** by using the cached reports. Also,
-note that it will **never skip any tests which failed or were pending** in the last runs.
+note that it will **never skip**:
+
+- **Flaky examples**
+- **Failed examples**
+- **Pending examples**
 
 Knowing the examples and files dependency gives us a better insight into the codebase,
 and we have **a clear idea of what to test for when making any changes**. With this data,
@@ -16,9 +20,10 @@ we can also analyze the coupling between different components and much more.
 
 ## Note
 
-You should take some time and go through the [document](./RSPEC_TRACER.md) describing
-the **intention** and implementation details of **skipping tests**, **managing coverage**,
-and **caching on CI**, etc.
+You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing
+the **intention** and implementation details of **managing dependency**, **managing flaky tests**,
+**skipping tests**, and **caching on CI**. You must go through the README file before
+integrating the gem into your project to better understand what is happening.
 
 ## Table of Contents
 
@@ -28,15 +33,18 @@ and **caching on CI**, etc.
   * [Additional Tools](#additional-tools)
 * [Getting Started](#getting-started)
 * [Environment Variables](#environment-variables)
+  * [BUNDLE_PATH](#bundle_path)
   * [CI](#ci)
   * [LOCAL_AWS](#local_aws)
   * [RSPEC_TRACER_NO_SKIP](#rspec_tracer_no_skip)
   * [RSPEC_TRACER_S3_URI](#rspec_tracer_s3_uri)
   * [RSPEC_TRACER_UPLOAD_LOCAL_CACHE](#rspec_tracer_upload_local_cache)
-  * [TEST_SUITE_ID](#test_suite_id)
+  * [RSPEC_TRACER_VERBOSE](#rspec_tracer_verbose)
   * [TEST_SUITES](#test_suites)
+  * [TEST_SUITE_ID](#test_suite_id)
 * [Sample Reports](#sample-reports)
     * [Examples](#examples)
+    * [Flaky Examples](#flaky-examples)
     * [Examples Dependency](#examples-dependency)
     * [Files Dependency](#files-dependency)
 * [Configuring RSpec Tracer](#configuring-rspec-tracer)
@@ -63,7 +71,7 @@ and **caching on CI**, etc.
 
 Add this line to your `Gemfile` and `bundle install`:
 ```ruby
-gem 'rspec-tracer', group: :test, require: false
+gem 'rspec-tracer', version: '~> 0.6', group: :test, require: false
 ```
 
 And, add the followings to your `.gitignore`:
@@ -128,13 +136,13 @@ Rakefile in your project to have the following:
     ```
 3. Before running tests, download the remote cache using the following rake task:
 
-    ```ruby
+    ```sh
     bundle exec rake rspec_tracer:remote_cache:download
     ```
 4. Run the tests with RSpec using `bundle exec rspec`.
 5. After running tests, upload the local cache using the following rake task:
 
-    ```ruby
+    ```sh
     bundle exec rake rspec_tracer:remote_cache:upload
     ```
 6. After running your tests, open `rspec_tracer_report/index.html` in the
@@ -142,7 +150,15 @@ browser of your choice.
 
 ## Environment Variables
 
-To get better control on execution, you can use the following two environment variables:
+To get better control on execution, you can use the following environment variables
+whenever required.
+
+### BUNDLE_PATH
+
+Since the bundler uses a vendor directory inside the project, it might cause slowness
+depending on the vendor size. You can configure the bundle path outside of the project
+using `BUNDLE_PATH` environment variable, for example, `BUNDLE_PATH=$HOME/vendor/bundle`.
+Make sure to cache this directory in the CI configuration.
 
 ### CI
 
@@ -177,11 +193,28 @@ export RSPEC_TRACER_S3_URI=s3://ci-artifacts-bucket/rspec-tracer-cache
 By default, RSpec Tracer does not upload local cache files. You can set this
 environment variable to `true` to upload the local cache to S3.
 
+### RSPEC_TRACER_VERBOSE
+
+To print the intermediate steps and time taken, use this environment variable:
+
+```sh
+export RSPEC_TRACER_VERBOSE=true
+```
+
+### TEST_SUITES
+
+Set this environment variable when using test suite id. It determines the total
+number of different test suites you are running.
+
+```ruby
+export TEST_SUITES=8
+```
+
 ### TEST_SUITE_ID
 
 If you have a large set of tests to run, it is recommended to run them in
 separate groups. This way, RSpec Tracer is not overwhelmed with loading massive
-cached data in the memory. Also, it generate and use cache for specific test suites
+cached data in the memory. Also, it generates and uses cache for specific test suites
 and not merge them.
 
 ```ruby
@@ -189,13 +222,19 @@ TEST_SUITE_ID=1 bundle exec rspec spec/models
 TEST_SUITE_ID=2 bundle exec rspec spec/helpers
 ```
 
-### TEST_SUITES
+If you run parallel builds on the CI, you should specify the test suite ID and
+the total number of test suites when downloading the cache files.
 
-Set this environment variable when using test suite id. It determines the total
-number of different test suites you are running.
+```sh
+$ TEST_SUITES=5 TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:download
+```
 
-```ruby
-export TEST_SUITES=8
+In this case, the appropriate cache should have all the cache files available on
+the S3 for each test suite, not just for the current one. Also, while uploading,
+make sure to provide the test suite id.
+
+```sh
+$ TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:upload
 ```
 
 ## Sample Reports
@@ -214,6 +253,19 @@ These reports provide basic test information:
 
 ![](./readme_files/examples_report_next_run.png)
 
+### Flaky Examples
+
+These reports provide flaky tests information. Assuming **the following two tests
+failed in the first run.**
+
+**Next Run**
+
+![](./readme_files/flaky_examples_report_first_run.png)
+
+**Another Run**
+
+![](./readme_files/flaky_examples_report_next_run.png)
+
 ### Examples Dependency
 
 These reports show a list of dependent files for each test.

data/lib/rspec_tracer/cache.rb

@@ -2,7 +2,8 @@
 
 module RSpecTracer
   class Cache
-    attr_reader :all_examples, :failed_examples, :pending_examples, :all_files, :dependency
+    attr_reader :all_examples, :flaky_examples, :failed_examples, :pending_examples,
+                :all_files, :dependency, :run_id
 
     def initialize
       @run_id = last_run_id
@@ -11,6 +12,7 @@ module RSpecTracer
       @cached = false
 
       @all_examples = {}
+      @flaky_examples = Set.new
       @failed_examples = Set.new
       @pending_examples = Set.new
       @all_files = {}
@@ -20,22 +22,36 @@ module RSpecTracer
     def load_cache_for_run
       return if @run_id.nil? || @cached
 
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       load_all_examples_cache
+      load_flaky_examples_cache
       load_failed_examples_cache
       load_pending_examples_cache
       load_all_files_cache
       load_dependency_cache
 
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       @cached = true
 
-      puts "RSpec tracer loaded cache from #{@cache_dir}" if @run_id
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer loaded cache from #{@cache_dir} (took #{elpased})"
     end
 
     def cached_examples_coverage
       return @examples_coverage if defined?(@examples_coverage)
       return @examples_coverage = {} if @run_id.nil?
 
-      load_examples_coverage_cache
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      coverage = load_examples_coverage_cache
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer loaded cached examples coverage (took #{elpased})" if RSpecTracer.verbose?
+
+      coverage
     end
 
     private
@@ -64,6 +80,14 @@ module RSpecTracer
       end
     end
 
+    def load_flaky_examples_cache
+      file_name = File.join(@cache_dir, 'flaky_examples.json')
+
+      return unless File.file?(file_name)
+
+      @flaky_examples = JSON.parse(File.read(file_name)).to_set
+    end
+
     def load_failed_examples_cache
       file_name = File.join(@cache_dir, 'failed_examples.json')
 

data/lib/rspec_tracer/configuration.rb

@@ -102,6 +102,12 @@ module RSpecTracer
       @coverage_filters ||= []
     end
 
+    def verbose?
+      return @verbose if defined?(@verbose)
+
+      @verbose = ENV.fetch('RSPEC_TRACER_VERBOSE', 'false') == 'true'
+    end
+
     def configure(&block)
       Docile.dsl_eval(self, &block)
     end

data/lib/rspec_tracer/coverage_reporter.rb

@@ -29,11 +29,15 @@ module RSpecTracer
 
     def compute_diff(example_id)
       peek_coverage.each_pair do |file_path, current_stats|
-        if @coverage.key?(file_path)
-          existing_file_diff_coverage(example_id, file_path, current_stats)
-        else
+        unless @coverage.key?(file_path)
           missing_file_diff_coverage(example_id, file_path, current_stats)
+
+          next
         end
+
+        next if current_stats == @coverage[file_path]
+
+        existing_file_diff_coverage(example_id, file_path, current_stats)
       end
     end
 

data/lib/rspec_tracer/defaults.rb

@@ -2,7 +2,13 @@
 
 RSpecTracer.configure do
   add_filter '/vendor/bundle/'
-  add_coverage_filter %w[/autotest/ /features/ /spec/ /test/].freeze
+  add_coverage_filter %w[
+    /autotest/
+    /features/
+    /spec/
+    /test/
+    /vendor/bundle/
+  ].freeze
 end
 
 at_exit do

data/lib/rspec_tracer/html_reporter/reporter.rb

@@ -6,21 +6,22 @@ require 'time'
 module RSpecTracer
   module HTMLReporter
     class Reporter
-      attr_reader :last_run, :examples, :examples_dependency, :files_dependency
+      attr_reader :last_run, :examples, :flaky_examples, :examples_dependency, :files_dependency
 
       def initialize
         @reporter = RSpecTracer.runner.reporter
 
         format_last_run
         format_examples
+        format_flaky_examples
         format_examples_dependency
         format_files_dependency
       end
 
       def generate_report
-        Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
-          FileUtils.cp_r(path, asset_output_path)
-        end
+        starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+        copy_assets
 
         file_name = File.join(RSpecTracer.report_path, 'index.html')
 
@@ -28,11 +29,20 @@ module RSpecTracer
           file.puts(template('layout').result(binding))
         end
 
-        puts "RSpecTracer generated HTML report to #{file_name}"
+        ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+        puts "RSpecTracer generated HTML report to #{file_name} (took #{elpased})"
       end
 
       private
 
+      def copy_assets
+        Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
+          FileUtils.cp_r(path, asset_output_path)
+        end
+      end
+
       def format_last_run
         @last_run = @reporter.last_run.slice(
           :actual_count,
@@ -57,6 +67,10 @@ module RSpecTracer
         end
       end
 
+      def format_flaky_examples
+        @flaky_examples = @examples.slice(*@reporter.flaky_examples).values
+      end
+
       def example_run_local_time(utc_time)
         case utc_time
         when Time
@@ -128,6 +142,14 @@ module RSpecTracer
         template(title_id).result(current_binding)
       end
 
+      def formatted_flaky_examples(title, flaky_examples)
+        title_id = report_container_id(title)
+        current_binding = binding
+
+        current_binding.local_variable_set(:title_id, title_id)
+        template(title_id).result(current_binding)
+      end
+
       def formatted_examples_dependency(title, examples_dependency)
         title_id = report_container_id(title)
         current_binding = binding
@@ -154,7 +176,7 @@ module RSpecTracer
 
       def example_status_css_class(example_status)
         case example_status.split.first
-        when 'Failed'
+        when 'Failed', 'Flaky'
           'red'
         when 'Pending'
           'yellow'

data/lib/rspec_tracer/html_reporter/views/flaky_examples.erb

@@ -0,0 +1,38 @@
+<div class="report_container" id="<%= title_id %>">
+  <h2>
+    <span class="group_name"><%= title %></span>
+    (
+      <span class="blue">
+        <strong><%= flaky_examples.count %></strong>
+      </span> examples
+    )
+  </h2>
+
+  <a name="<%= title_id %>"></a>
+
+  <div class="report-table--responsive">
+    <table class="report-table">
+      <thead>
+        <tr>
+          <th>ID</th>
+          <th>Description</th>
+          <th>Location</th>
+          <th>Result</th>
+          <th>Run At</th>
+        </tr>
+      </thead>
+
+      <tbody>
+        <% flaky_examples.each do |example| %>
+          <tr>
+            <td><%= example[:id] %></td>
+            <td><%= example[:description] %></td>
+            <td><%= example[:location] %></td>
+            <td><strong class="<%= example_result_css_class(example[:result]) %>"><%= example[:result] %></strong></td>
+            <td width="8%"><%= example[:last_run] %></td>
+          </tr>
+        <% end %>
+      </tbody>
+    </table>
+  </div>
+</div>

data/lib/rspec_tracer/html_reporter/views/layout.erb

@@ -19,6 +19,9 @@
 
       <div id="content">
         <%= formatted_examples('Examples', examples.values) %>
+        <% unless flaky_examples.empty? %>
+          <%= formatted_flaky_examples('Flaky Examples', flaky_examples) %>
+        <% end %>
         <%= formatted_examples_dependency('Examples Dependency', examples_dependency) %>
         <%= formatted_files_dependency("Files Dependency", files_dependency) %>
       </div>

data/lib/rspec_tracer/remote_cache/cache.rb

@@ -20,9 +20,6 @@ module RSpecTracer
                   else
                     'aws'
                   end
-        @test_suite_id = ENV['TEST_SUITE_ID'].to_s
-        @test_suites = ENV.fetch('TEST_SUITES', '1').to_i
-        @total_objects = CACHE_FILES_PER_TEST_SUITE * @test_suites
       end
 
       def download
@@ -32,10 +29,7 @@ module RSpecTracer
           return
         end
 
-        @git = RSpecTracer::RemoteCache::Git.new
-        @git.prepare_for_download
-
-        @cache_sha = nearest_cache_sha
+        prepare_for_download
 
         if @cache_sha.nil?
           puts 'Could not find a suitable cache sha to download'
@@ -57,9 +51,7 @@ module RSpecTracer
           return
         end
 
-        @run_id = last_run_id
-        @git = RSpecTracer::RemoteCache::Git.new
-
+        prepare_for_upload
         upload_files
 
         puts "Uploaded cache from #{@upload_path} to #{@upload_prefix}"
@@ -69,20 +61,57 @@ module RSpecTracer
 
       private
 
+      def prepare_for_download
+        @test_suite_id = ENV['TEST_SUITE_ID']
+        @test_suites = ENV['TEST_SUITES']
+
+        if @test_suite_id.nil? ^ @test_suites.nil?
+          raise(
+            CacheDownloadError,
+            'Both the enviornment variables TEST_SUITE_ID and TEST_SUITES are not set'
+          )
+        end
+
+        @git = RSpecTracer::RemoteCache::Git.new
+        @git.prepare_for_download
+
+        generate_cached_files_count_and_regex
+
+        @cache_sha = nearest_cache_sha
+      end
+
+      def generate_cached_files_count_and_regex
+        if @test_suites.nil?
+          @last_run_files_count = 1
+          @last_run_files_regex = '/%<ref>s/last_run.json$'
+          @cached_files_count = CACHE_FILES_PER_TEST_SUITE
+          @cached_files_regex = '/%<ref>s/[0-9a-f]{32}/.+.json'
+        else
+          @test_suites = @test_suites.to_i
+          @test_suites_regex = (1..@test_suites).to_a.join('|')
+
+          @last_run_files_count = @test_suites
+          @last_run_files_regex = "/%<ref>s/(#{@test_suites_regex})/last_run.json$"
+          @cached_files_count = CACHE_FILES_PER_TEST_SUITE * @test_suites.to_i
+          @cached_files_regex = "/%<ref>s/(#{@test_suites_regex})/[0-9a-f]{32}/.+.json$"
+        end
+      end
+
       def nearest_cache_sha
         @git.ref_list.detect do |ref|
-          prefix = "#{@s3_uri}/#{ref}/#{@test_suite_id}/".sub(%r{/+$}, '/')
+          prefix = "#{@s3_uri}/#{ref}/"
 
           puts "Testing prefix #{prefix}"
 
-          command = <<-COMMAND.strip.gsub(/\s+/, ' ')
-            #{@aws_s3} s3 ls #{prefix}
-              --recursive
-              --summarize
-            | grep 'Total Objects'
-          COMMAND
+          objects = `#{@aws_s3} s3 ls #{prefix} --recursive`.chomp.split("\n")
+
+          last_run_regex = Regexp.new(format(@last_run_files_regex, ref: ref))
+
+          next if objects.count { |object| object.match?(last_run_regex) } != @last_run_files_count
 
-          @total_objects == `#{command}`.chomp.split('Total Objects:').last.to_s.strip.to_i
+          cache_regex = Regexp.new(format(@cached_files_regex, ref: ref))
+
+          objects.count { |object| object.match?(cache_regex) } == @cached_files_count
         end
       end
 
@@ -90,10 +119,19 @@ module RSpecTracer
         @download_prefix = "#{@s3_uri}/#{@cache_sha}/#{@test_suite_id}/".sub(%r{/+$}, '/')
         @download_path = RSpecTracer.cache_path
 
-        return if system(
+        raise CacheDownloadError, 'Failed to download cache files' unless system(
           @aws_s3, 's3', 'cp',
-          @download_prefix,
+          File.join(@download_prefix, 'last_run.json'),
           @download_path,
+          out: File::NULL, err: File::NULL
+        )
+
+        @run_id = last_run_id
+
+        return if system(
+          @aws_s3, 's3', 'cp',
+          File.join(@download_prefix, @run_id),
+          File.join(@download_path, @run_id),
           '--recursive',
           out: File::NULL, err: File::NULL
         )
@@ -103,22 +141,20 @@ module RSpecTracer
         raise CacheDownloadError, 'Failed to download cache files'
       end
 
-      def last_run_id
-        file_name = File.join(RSpecTracer.cache_path, 'last_run.json')
-
-        return unless File.file?(file_name)
-
-        run_id = JSON.parse(File.read(file_name))['run_id']
-
-        raise LocalCacheNotFoundError, 'Could not find any local cache to upload' if run_id.nil?
+      def prepare_for_upload
+        @git = RSpecTracer::RemoteCache::Git.new
+        @test_suite_id = ENV['TEST_SUITE_ID']
+        @upload_prefix = if @test_suite_id.nil?
+                           "#{@s3_uri}/#{@git.branch_ref}/"
+                         else
+                           "#{@s3_uri}/#{@git.branch_ref}/#{@test_suite_id}/"
+                         end
 
-        run_id
+        @upload_path = RSpecTracer.cache_path
+        @run_id = last_run_id
       end
 
       def upload_files
-        @upload_prefix = "#{@s3_uri}/#{@git.branch_ref}/#{@test_suite_id}/".sub(%r{/+$}, '/')
-        @upload_path = RSpecTracer.cache_path
-
         return if system(
           @aws_s3, 's3', 'cp',
           File.join(@upload_path, 'last_run.json'),
@@ -127,13 +163,25 @@ module RSpecTracer
         ) && system(
           @aws_s3, 's3', 'cp',
           File.join(@upload_path, @run_id),
-          "#{@upload_prefix}/#{@run_id}",
+          File.join(@upload_prefix, @run_id),
           '--recursive',
           out: File::NULL, err: File::NULL
         )
 
         raise CacheUploadError, 'Failed to upload cache files'
       end
+
+      def last_run_id
+        file_name = File.join(RSpecTracer.cache_path, 'last_run.json')
+
+        return unless File.file?(file_name)
+
+        run_id = JSON.parse(File.read(file_name))['run_id']
+
+        raise LocalCacheNotFoundError, 'Could not find any local cache to upload' if run_id.nil?
+
+        run_id
+      end
     end
   end
 end

data/lib/rspec_tracer/reporter.rb

@@ -2,8 +2,9 @@
 
 module RSpecTracer
   class Reporter
-    attr_reader :all_examples, :pending_examples, :all_files, :dependency,
-                :reverse_dependency, :examples_coverage, :last_run
+    attr_reader :all_examples, :possibly_flaky_examples, :flaky_examples, :pending_examples,
+                :all_files, :modified_files, :deleted_files, :dependency, :reverse_dependency,
+                :examples_coverage, :last_run
 
     def initialize
       initialize_examples
@@ -21,6 +22,7 @@ module RSpecTracer
     end
 
     def on_example_passed(example_id, result)
+      @passed_examples << example_id
       @all_examples[example_id][:execution_result] = formatted_execution_result(result)
     end
 
@@ -44,6 +46,14 @@ module RSpecTracer
       end
     end
 
+    def register_possibly_flaky_example(example_id)
+      @possibly_flaky_examples << example_id
+    end
+
+    def register_flaky_example(example_id)
+      @flaky_examples << example_id
+    end
+
     def register_failed_example(example_id)
       @failed_examples << example_id
     end
@@ -52,6 +62,10 @@ module RSpecTracer
       @pending_examples << example_id
     end
 
+    def example_passed?(example_id)
+      @passed_examples.include?(example_id)
+    end
+
     def example_skipped?(example_id)
       @skipped_examples.include?(example_id)
     end
@@ -126,6 +140,8 @@ module RSpecTracer
     end
 
     def write_reports
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       @run_id = Digest::MD5.hexdigest(@all_examples.keys.sort.to_json)
       @cache_dir = File.join(RSpecTracer.cache_path, @run_id)
 
@@ -133,6 +149,7 @@ module RSpecTracer
 
       %i[
         all_examples
+        flaky_examples
         failed_examples
         pending_examples
         all_files
@@ -142,13 +159,19 @@ module RSpecTracer
         last_run
       ].each { |report_type| send("write_#{report_type}_report") }
 
-      puts "RSpec tracer reports generated to #{@cache_dir}"
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer reports written to #{@cache_dir} (took #{elpased})"
     end
 
     private
 
     def initialize_examples
       @all_examples = {}
+      @passed_examples = Set.new
+      @possibly_flaky_examples = Set.new
+      @flaky_examples = Set.new
       @failed_examples = Set.new
       @skipped_examples = Set.new
       @pending_examples = Set.new
@@ -209,6 +232,12 @@ module RSpecTracer
       File.write(file_name, JSON.pretty_generate(@all_examples))
     end
 
+    def write_flaky_examples_report
+      file_name = File.join(@cache_dir, 'flaky_examples.json')
+
+      File.write(file_name, JSON.pretty_generate(@flaky_examples.to_a))
+    end
+
     def write_failed_examples_report
       file_name = File.join(@cache_dir, 'failed_examples.json')
 

data/lib/rspec_tracer/rspec_runner.rb

@@ -2,24 +2,29 @@
 
 module RSpecTracer
   module RSpecRunner
+    # rubocop:disable Metrics/AbcSize
     def run_specs(_example_groups)
       actual_count = RSpec.world.example_count
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       filtered_examples, example_groups = RSpecTracer.filter_examples
 
       RSpec.world.instance_variable_set(:@filtered_examples, filtered_examples)
       RSpec.world.instance_variable_set(:@example_groups, example_groups)
 
       current_count = RSpec.world.example_count
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
 
       puts
       puts <<-EXAMPLES.strip.gsub(/\s+/, ' ')
         RSpec tracer is running #{current_count} examples (actual: #{actual_count},
-        skipped: #{actual_count - current_count})
+        skipped: #{actual_count - current_count}) (took #{elpased})
       EXAMPLES
 
       RSpecTracer.running = true
 
       super(example_groups)
     end
+    # rubocop:enable Metrics/AbcSize
   end
 end

data/lib/rspec_tracer/runner.rb

@@ -8,6 +8,7 @@ module RSpecTracer
     EXAMPLE_RUN_REASON = {
       explicit_run: 'Explicit run',
       no_cache: 'No cache',
+      flaky_example: 'Flaky example',
       failed_example: 'Failed previously',
       pending_example: 'Pending previously',
       files_changed: 'Files changed'
@@ -20,6 +21,8 @@ module RSpecTracer
       @reporter = RSpecTracer::Reporter.new
       @filtered_examples = {}
 
+      return if @cache.run_id.nil?
+
       @cache.load_cache_for_run
       filter_examples_to_run
     end
@@ -87,16 +90,15 @@ module RSpecTracer
     # rubocop:enable Metrics/AbcSize
 
     def register_dependency(examples_coverage)
+      filtered_files = Set.new
+
       examples_coverage.each_pair do |example_id, example_coverage|
         register_example_files_dependency(example_id)
 
         example_coverage.each_key do |file_path|
-          source_file = RSpecTracer::SourceFile.from_path(file_path)
+          next if filtered_files.include?(file_path)
 
-          next if RSpecTracer.filters.any? { |filter| filter.match?(source_file) }
-
-          @reporter.register_source_file(source_file)
-          @reporter.register_dependency(example_id, source_file[:file_name])
+          filtered_files << file_path unless register_file_dependency(example_id, file_path)
         end
       end
 
@@ -127,15 +129,19 @@ module RSpecTracer
 
     def generate_report
       @reporter.generate_last_run_report
+      generate_examples_status_report
 
-      generate_failed_examples_report
-      generate_pending_examples_report
+      %i[all_files all_examples dependency examples_coverage reverse_dependency].each do |report_type|
+        starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
 
-      %i[all_files all_examples dependency examples_coverage].each do |report_type|
         send("generate_#{report_type}_report")
+
+        ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+        puts "RSpec tracer generated #{report_type.to_s.tr('_', ' ')} report (took #{elpased})" if RSpecTracer.verbose?
       end
 
-      @reporter.generate_reverse_dependency_report
       @reporter.write_reports
     end
 
@@ -146,54 +152,74 @@ module RSpecTracer
     end
 
     def filter_examples_to_run
-      add_previously_failed_examples
-      add_previously_pending_examples
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      @changed_files = fetch_changed_files
+
+      filter_by_example_status
       filter_by_files_changed
-    end
 
-    def add_previously_failed_examples
-      @cache.failed_examples.each do |example_id|
-        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:failed_example]
-      end
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer processed cache (took #{elpased})" if RSpecTracer.verbose?
     end
 
-    def add_previously_pending_examples
-      @cache.pending_examples.each do |example_id|
-        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:pending_example]
-      end
+    def filter_by_example_status
+      add_previously_flaky_examples
+      add_previously_failed_examples
+      add_previously_pending_examples
     end
 
     def filter_by_files_changed
       @cache.dependency.each_pair do |example_id, files|
         next if @filtered_examples.key?(example_id)
+        next if (@changed_files & files).empty?
 
-        files.each do |file_name|
-          break if filtered_by_file_changed?(example_id, file_name)
-        end
+        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:files_changed]
       end
     end
 
-    def filtered_by_file_changed?(example_id, file_name)
-      if @reporter.file_changed?(file_name)
-        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:files_changed]
+    def add_previously_flaky_examples
+      @cache.flaky_examples.each do |example_id|
+        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:flaky_example]
+
+        next unless (@changed_files & @cache.dependency[example_id]).empty?
 
-        return true
+        @reporter.register_possibly_flaky_example(example_id)
       end
+    end
+
+    def add_previously_failed_examples
+      @cache.failed_examples.each do |example_id|
+        next if @filtered_examples.key?(example_id)
+
+        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:failed_example]
+
+        next unless (@changed_files & @cache.dependency[example_id]).empty?
 
-      source_file = registered_source_file(file_name)
+        @reporter.register_possibly_flaky_example(example_id)
+      end
+    end
 
-      return false if source_file &&
-        @cache.all_files[file_name][:digest] == source_file[:digest]
+    def add_previously_pending_examples
+      @cache.pending_examples.each do |example_id|
+        @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:pending_example]
+      end
+    end
 
-      @filtered_examples[example_id] = EXAMPLE_RUN_REASON[:files_changed]
+    def fetch_changed_files
+      @cache.all_files.each_value do |cached_file|
+        file_name = cached_file[:file_name]
+        source_file = RSpecTracer::SourceFile.from_name(file_name)
 
-      if source_file.nil?
-        @reporter.on_file_deleted(file_name)
-      else
-        @reporter.on_file_modified(file_name)
+        if source_file.nil?
+          @reporter.on_file_deleted(file_name)
+        elsif cached_file[:digest] != source_file[:digest]
+          @reporter.on_file_modified(file_name)
+        end
       end
 
-      true
+      @reporter.modified_files | @reporter.deleted_files
     end
 
     def generate_untraced_files(trace_point_files)
@@ -227,14 +253,34 @@ module RSpecTracer
     end
 
     def register_example_file_dependency(example_id, file_name)
-      source_file = registered_source_file(file_name)
+      source_file = RSpecTracer::SourceFile.from_name(file_name)
 
       @reporter.register_source_file(source_file)
       @reporter.register_dependency(example_id, file_name)
     end
 
-    def registered_source_file(file_name)
-      @reporter.all_files[file_name] || RSpecTracer::SourceFile.from_name(file_name)
+    def register_file_dependency(example_id, file_path)
+      source_file = RSpecTracer::SourceFile.from_path(file_path)
+
+      return false if RSpecTracer.filters.any? { |filter| filter.match?(source_file) }
+
+      @reporter.register_source_file(source_file)
+      @reporter.register_dependency(example_id, source_file[:file_name])
+
+      true
+    end
+
+    def generate_examples_status_report
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+      generate_flaky_examples_report
+      generate_failed_examples_report
+      generate_pending_examples_report
+
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer generated flaky, failed, and pending examples report (took #{elpased})" if RSpecTracer.verbose?
     end
 
     def generate_all_files_report
@@ -255,6 +301,16 @@ module RSpecTracer
       end
     end
 
+    def generate_flaky_examples_report
+      @reporter.possibly_flaky_examples.each do |example_id|
+        next if @reporter.example_deleted?(example_id)
+        next unless @cache.flaky_examples.include?(example_id) ||
+          @reporter.example_passed?(example_id)
+
+        @reporter.register_flaky_example(example_id)
+      end
+    end
+
     def generate_failed_examples_report
       @cache.failed_examples.each do |example_id|
         next if @reporter.example_deleted?(example_id) ||
@@ -296,5 +352,9 @@ module RSpecTracer
         end
       end
     end
+
+    def generate_reverse_dependency_report
+      @reporter.generate_reverse_dependency_report
+    end
   end
 end

data/lib/rspec_tracer/time_formatter.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module RSpecTracer
+  module TimeFormatter
+    DEFAULT_PRECISION = 2
+    SECONDS_PRECISION = 5
+
+    UNITS = {
+      second: 60,
+      minute: 60,
+      hour: 24,
+      day: Float::INFINITY
+    }.freeze
+
+    module_function
+
+    def format_time(seconds)
+      return pluralize(format_duration(seconds), 'second') if seconds < 60
+
+      formatted_duration = UNITS.each_pair.with_object([]) do |(unit, count), duration|
+        next unless seconds.positive?
+
+        seconds, remainder = seconds.divmod(count)
+
+        next if remainder.zero?
+
+        duration << pluralize(format_duration(remainder), unit)
+      end
+
+      formatted_duration.reverse.join(' ')
+    end
+
+    def format_duration(duration)
+      return 0 if duration.negative?
+
+      precision = duration < 1 ? SECONDS_PRECISION : DEFAULT_PRECISION
+
+      strip_trailing_zeroes(format("%<duration>0.#{precision}f", duration: duration))
+    end
+
+    def strip_trailing_zeroes(formatted_duration)
+      formatted_duration.sub(/(?:(\..*[^0])0+|\.0+)$/, '\1')
+    end
+
+    def pluralize(duration, unit)
+      if (duration.to_f - 1).abs < Float::EPSILON
+        "#{duration} #{unit}"
+      else
+        "#{duration} #{unit}s"
+      end
+    end
+
+    private_class_method :format_duration, :strip_trailing_zeroes, :pluralize
+  end
+end

data/lib/rspec_tracer/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RSpecTracer
-  VERSION = '0.4.0'
+  VERSION = '0.6.2'
 end

data/lib/rspec_tracer.rb

@@ -23,6 +23,7 @@ require_relative 'rspec_tracer/rspec_runner'
 require_relative 'rspec_tracer/ruby_coverage'
 require_relative 'rspec_tracer/runner'
 require_relative 'rspec_tracer/source_file'
+require_relative 'rspec_tracer/time_formatter'
 require_relative 'rspec_tracer/version'
 
 module RSpecTracer
@@ -184,22 +185,36 @@ module RSpecTracer
     def generate_reports
       puts 'RSpec tracer is generating reports'
 
-      generate_tracer_reports
-      generate_coverage_reports
+      process_dependency
+      process_coverage
       runner.generate_report
       RSpecTracer::HTMLReporter::Reporter.new.generate_report
     end
 
-    def generate_tracer_reports
+    def process_dependency
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       runner.register_deleted_examples
       runner.register_dependency(coverage_reporter.examples_coverage)
       runner.register_untraced_dependency(@traced_files)
+
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer processed dependency (took #{elpased})" if RSpecTracer.verbose?
     end
 
-    def generate_coverage_reports
+    def process_coverage
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       coverage_reporter.generate_final_examples_coverage
       coverage_reporter.merge_coverage(runner.generate_missed_coverage)
       runner.register_examples_coverage(coverage_reporter.examples_coverage)
+
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer processed coverage (took #{elpased})" if RSpecTracer.verbose?
     end
 
     def run_simplecov_exit_task
@@ -213,12 +228,18 @@ module RSpecTracer
     end
 
     def run_coverage_exit_task
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       coverage_reporter.generate_final_coverage
 
       file_name = File.join(RSpecTracer.coverage_path, 'coverage.json')
 
       write_coverage_report(file_name)
-      print_coverage_stats(file_name)
+
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      print_coverage_stats(file_name, elpased)
     end
 
     def write_coverage_report(file_name)
@@ -232,12 +253,13 @@ module RSpecTracer
       File.write(file_name, JSON.pretty_generate(report))
     end
 
-    def print_coverage_stats(file_name)
+    def print_coverage_stats(file_name, elpased)
       stat = coverage_reporter.coverage_stat
 
       puts <<-REPORT.strip.gsub(/\s+/, ' ')
         Coverage report generated for RSpecTracer to #{file_name}. #{stat[:covered_lines]}
         / #{stat[:total_lines]} LOC (#{stat[:covered_percent]}%) covered
+        (took #{elpased})
       REPORT
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rspec-tracer
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.6.2
 platform: ruby
 authors:
 - Abhimanyu Singh
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-09-02 00:00:00.000000000 Z
+date: 2021-09-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: docile
@@ -90,6 +90,7 @@ files:
 - lib/rspec_tracer/html_reporter/views/examples.erb
 - lib/rspec_tracer/html_reporter/views/examples_dependency.erb
 - lib/rspec_tracer/html_reporter/views/files_dependency.erb
+- lib/rspec_tracer/html_reporter/views/flaky_examples.erb
 - lib/rspec_tracer/html_reporter/views/layout.erb
 - lib/rspec_tracer/remote_cache/cache.rb
 - lib/rspec_tracer/remote_cache/git.rb
@@ -99,13 +100,14 @@ files:
 - lib/rspec_tracer/ruby_coverage.rb
 - lib/rspec_tracer/runner.rb
 - lib/rspec_tracer/source_file.rb
+- lib/rspec_tracer/time_formatter.rb
 - lib/rspec_tracer/version.rb
 homepage: https://github.com/avmnu-sng/rspec-tracer
 licenses:
 - MIT
 metadata:
   homepage_uri: https://github.com/avmnu-sng/rspec-tracer
-  source_code_uri: https://github.com/avmnu-sng/rspec-tracer/tree/v0.4.0
+  source_code_uri: https://github.com/avmnu-sng/rspec-tracer/tree/v0.6.2
   changelog_uri: https://github.com/avmnu-sng/rspec-tracer/blob/main/CHANGELOG.md
   bug_tracker_uri: https://github.com/avmnu-sng/rspec-tracer/issues
 post_install_message: