rspec-tracer 0.2.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c4494ed77ebba015c8f3a1bbc95fdfbd815314b5b455a471eb6f20df6dec0d7
-  data.tar.gz: '08e318aec77f76ebf8edbff8e4fc415f7410e5adf42876d2741c8533c7b5ec0d'
+  metadata.gz: 885e868cf847ca1bab4ba2243cd4cd6a4bc3f936a3c8c10f677146433acc2068
+  data.tar.gz: d60fcdeef0067a37faec3ea876ffa519bdd7039a1765ad59e551427bcf2c8e29
 SHA512:
-  metadata.gz: e8ebb4c1534a5a64b8699c35aaa6c99d449e69e4c79fa80212d953ae659a0f2e16f483280b5685c42cfb9f535bfdbc7f7af439d8e2722a482dae38e2205847d8
-  data.tar.gz: 021ed3f090dddd3b0c8bc0768feca11eec52a45c7b280d7e48e085ee1f049660cfbba38c759128d7a46f78732d4534bd36d684feb390aaa014bf78a20e3cb36b
+  metadata.gz: cffa43507448703f4471ed2a463e7ef486f105b06560663d307b21c3303368b0ae0260bb36bbc392f900bfb0e54482fdc8055d231ab75fb33e5a93360be46ac9
+  data.tar.gz: d59157d318beda7c81a65cdd1a38ff3d849521a502e31c6c805916ace8f6f3568a5114f201bc03cdb33e63cae3c0a86ae5e81dd148c1c32f735d7d5d3cea5750

data/CHANGELOG.md

@@ -1,8 +1,36 @@
-## [Unreleased]
+## [0.6.0] - 2021-09-05
 
-**[WIP]** Support for CI
+### Added
+
+- Improved dependency change detection (#18)
+- Flaky tests detection (#19)
+- Exclude vendor files from analysis (#21)
+- Report elapsed time at various stages (#23)
+
+### Note
+
+The first run on this version will not use any cache on the CI because the number
+of files changed from eight to nine, so there will be no appropriate cache to use.
+
+## [0.5.0] - 2021-09-03
+
+### Fixed
+
+- Limit number of cached files download (#16)
+
+## [0.4.0] - 2021-09-03
+
+### Added
+
+- Support for CI
+
+## [0.3.0] - 2021-08-30
+
+### Fixed
+
+- `docile` version compatability with `simplecov`
 
-## [0.1.1] - 2021-08-28
+## [0.2.0] - 2021-08-28
 
 ### Fixed
 

data/README.md

@@ -8,22 +8,63 @@ It uses [Ruby's built-in coverage library](https://ruby-doc.org/stdlib/libdoc/co
 to keep track of the coverage for each test. For each test executed, the coverage
 diff provides the desired file list. RSpec Tracer takes care of reporting the
 **correct code coverage when skipping tests** by using the cached reports. Also,
-note that it will **never skip any tests which failed or were pending** in the last runs.
+note that it will **never skip**:
+
+- **Flaky examples**
+- **Failed examples**
+- **Pending examples**
 
 Knowing the examples and files dependency gives us a better insight into the codebase,
 and we have **a clear idea of what to test for when making any changes**. With this data,
 we can also analyze the coupling between different components and much more.
 
+## Note
+
+You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing
+the **intention** and implementation details of **managing dependency**, **managing flaky tests**,
+**skipping tests**, and **caching on CI**. You must go through the README file before
+integrating the gem into your project to better understand what is happening.
+
+## Table of Contents
+
+* [Demo](#demo)
+* [Installation](#installation)
+  * [Compatibility](#compatibility)
+  * [Additional Tools](#additional-tools)
+* [Getting Started](#getting-started)
+* [Environment Variables](#environment-variables)
+  * [BUNDLE_PATH](#bundle_path)
+  * [CI](#ci)
+  * [LOCAL_AWS](#local_aws)
+  * [RSPEC_TRACER_NO_SKIP](#rspec_tracer_no_skip)
+  * [RSPEC_TRACER_S3_URI](#rspec_tracer_s3_uri)
+  * [RSPEC_TRACER_UPLOAD_LOCAL_CACHE](#rspec_tracer_upload_local_cache)
+  * [TEST_SUITES](#test_suites)
+  * [TEST_SUITE_ID](#test_suite_id)
+* [Sample Reports](#sample-reports)
+    * [Examples](#examples)
+    * [Flaky Examples](#flaky-examples)
+    * [Examples Dependency](#examples-dependency)
+    * [Files Dependency](#files-dependency)
+* [Configuring RSpec Tracer](#configuring-rspec-tracer)
+* [Filters](#filters)
+  * [Defining Custom Filteres](#defining-custom-filteres)
+    * [String Filter](#string-filter)
+    * [Regex Filter](#regex-filter)
+    * [Block Filter](#block-filter)
+    * [Array Filter](#array-filter)
+* [Contributing](#contributing)
+* [License](#license)
+* [Code of Conduct](#code-of-conduct)
+
+## Demo
+
 **First Run**
 ![](./readme_files/first_run.gif)
 
 **Next Run**
 ![](./readme_files/next_run.gif)
 
-## Note
-
-**RSpec Tracer is currently available for use in the local development
-environment only.** Support for CI is work in progress.
 
 ## Installation
 
@@ -45,6 +86,11 @@ RSpec Tracer requires **Ruby 2.5+** and **rspec-core >= 3.6.0**. To use with **R
 make sure to use **rspec-rails >= 4.0.0**. If you are using SimpleCov, it is
 recommended to use **simplecov >= 0.12.0**.
 
+### Additional Tools
+
+To use RSpec Tracer on CI, you need to have an **S3 bucket** and
+**[AWS CLI](https://aws.amazon.com/cli/)** installed.
+
 ## Getting Started
 
 1. **Load and Start RSpec Tracer**
@@ -79,13 +125,50 @@ recommended to use **simplecov >= 0.12.0**.
         RSpecTracer.start
         ```
 
-2. Run the tests with RSpec using `bundle exec rspec`.
-3. After running your tests, open `rspec_tracer_report/index.html` in the
+2. To enable RSpec Tracer to share cache between different builds on CI, update the
+Rakefile in your project to have the following:
+
+    ```ruby
+    spec = Gem::Specification.find_by_name('rspec-tracer')
+
+    load "#{spec.gem_dir}/lib/rspec_tracer/remote_cache/Rakefile"
+    ```
+3. Before running tests, download the remote cache using the following rake task:
+
+    ```sh
+    bundle exec rake rspec_tracer:remote_cache:download
+    ```
+4. Run the tests with RSpec using `bundle exec rspec`.
+5. After running tests, upload the local cache using the following rake task:
+
+    ```sh
+    bundle exec rake rspec_tracer:remote_cache:upload
+    ```
+6. After running your tests, open `rspec_tracer_report/index.html` in the
 browser of your choice.
 
 ## Environment Variables
 
-To get better control on execution, you can use the following two environment variables:
+To get better control on execution, you can use the following environment variables
+whenever required.
+
+### BUNDLE_PATH
+
+Since the bundler uses a vendor directory inside the project, it might cause slowness
+depending on the vendor size. You can configure the bundle path outside of the project
+using `BUNDLE_PATH` environment variable, for example, `BUNDLE_PATH=$HOME/vendor/bundle`.
+Make sure to cache this directory in the CI configuration.
+
+### CI
+
+Mostly all the CI have `CI=true`. If not, you should explicitly set it to `true`.
+
+### LOCAL_AWS
+
+In case you want to test out the caching feature in the local development environment.
+You can install [localstack](https://github.com/localstack/localstack) and
+[awscli-local](https://github.com/localstack/awscli-local) and then invoke the
+rake tasks with `LOCAL_AWS=true`.
 
 ### RSPEC_TRACER_NO_SKIP
 
@@ -96,11 +179,33 @@ any tests. Note that it will continue to maintain cache files and generate repor
 RSPEC_TRACER_NO_SKIP=true bundle exec rspec
 ```
 
+### RSPEC_TRACER_S3_URI
+
+You should provide the S3 bucket path to store the cache files.
+
+```ruby
+export RSPEC_TRACER_S3_URI=s3://ci-artifacts-bucket/rspec-tracer-cache
+```
+
+### RSPEC_TRACER_UPLOAD_LOCAL_CACHE
+
+By default, RSpec Tracer does not upload local cache files. You can set this
+environment variable to `true` to upload the local cache to S3.
+
+### TEST_SUITES
+
+Set this environment variable when using test suite id. It determines the total
+number of different test suites you are running.
+
+```ruby
+export TEST_SUITES=8
+```
+
 ### TEST_SUITE_ID
 
 If you have a large set of tests to run, it is recommended to run them in
 separate groups. This way, RSpec Tracer is not overwhelmed with loading massive
-cached data in the memory. Also, it generate and use cache for specific test suites
+cached data in the memory. Also, it generates and uses cache for specific test suites
 and not merge them.
 
 ```ruby
@@ -108,6 +213,21 @@ TEST_SUITE_ID=1 bundle exec rspec spec/models
 TEST_SUITE_ID=2 bundle exec rspec spec/helpers
 ```
 
+If you run parallel builds on the CI, you should specify the test suite ID and
+the total number of test suites when downloading the cache files.
+
+```sh
+$ TEST_SUITES=5 TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:download
+```
+
+In this case, the appropriate cache should have all the cache files available on
+the S3 for each test suite, not just for the current one. Also, while uploading,
+make sure to provide the test suite id.
+
+```sh
+$ TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:upload
+```
+
 ## Sample Reports
 
 You get the following three reports:
@@ -124,6 +244,19 @@ These reports provide basic test information:
 
 ![](./readme_files/examples_report_next_run.png)
 
+### Flaky Examples
+
+These reports provide flaky tests information. Assuming **the following two tests
+failed in the first run.**
+
+**Next Run**
+
+![](./readme_files/flaky_examples_report_first_run.png)
+
+**Another Run**
+
+![](./readme_files/flaky_examples_report_next_run.png)
+
 ### Examples Dependency
 
 These reports show a list of dependent files for each test.

data/lib/rspec_tracer/cache.rb

@@ -2,7 +2,8 @@
 
 module RSpecTracer
   class Cache
-    attr_reader :all_examples, :failed_examples, :pending_examples, :all_files, :dependency
+    attr_reader :all_examples, :flaky_examples, :failed_examples, :pending_examples,
+                :all_files, :dependency, :run_id
 
     def initialize
       @run_id = last_run_id
@@ -11,6 +12,7 @@ module RSpecTracer
       @cached = false
 
       @all_examples = {}
+      @flaky_examples = Set.new
       @failed_examples = Set.new
       @pending_examples = Set.new
       @all_files = {}
@@ -20,22 +22,36 @@ module RSpecTracer
     def load_cache_for_run
       return if @run_id.nil? || @cached
 
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       load_all_examples_cache
+      load_flaky_examples_cache
       load_failed_examples_cache
       load_pending_examples_cache
       load_all_files_cache
       load_dependency_cache
 
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       @cached = true
 
-      puts "RSpec tracer loaded cache from #{@cache_dir}" if @run_id
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer loaded cache from #{@cache_dir} (took #{elpased})"
     end
 
     def cached_examples_coverage
       return @examples_coverage if defined?(@examples_coverage)
       return @examples_coverage = {} if @run_id.nil?
 
-      load_examples_coverage_cache
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      coverage = load_examples_coverage_cache
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer loaded cached examples coverage (took #{elpased})"
+
+      coverage
     end
 
     private
@@ -64,6 +80,14 @@ module RSpecTracer
       end
     end
 
+    def load_flaky_examples_cache
+      file_name = File.join(@cache_dir, 'flaky_examples.json')
+
+      return unless File.file?(file_name)
+
+      @flaky_examples = JSON.parse(File.read(file_name)).to_set
+    end
+
     def load_failed_examples_cache
       file_name = File.join(@cache_dir, 'failed_examples.json')
 

data/lib/rspec_tracer/defaults.rb

@@ -2,7 +2,13 @@
 
 RSpecTracer.configure do
   add_filter '/vendor/bundle/'
-  add_coverage_filter %w[/autotest/ /features/ /spec/ /test/].freeze
+  add_coverage_filter %w[
+    /autotest/
+    /features/
+    /spec/
+    /test/
+    /vendor/bundle/
+  ].freeze
 end
 
 at_exit do

data/lib/rspec_tracer/html_reporter/reporter.rb

@@ -6,21 +6,22 @@ require 'time'
 module RSpecTracer
   module HTMLReporter
     class Reporter
-      attr_reader :last_run, :examples, :examples_dependency, :files_dependency
+      attr_reader :last_run, :examples, :flaky_examples, :examples_dependency, :files_dependency
 
       def initialize
         @reporter = RSpecTracer.runner.reporter
 
         format_last_run
         format_examples
+        format_flaky_examples
         format_examples_dependency
         format_files_dependency
       end
 
       def generate_report
-        Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
-          FileUtils.cp_r(path, asset_output_path)
-        end
+        starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+        copy_assets
 
         file_name = File.join(RSpecTracer.report_path, 'index.html')
 
@@ -28,11 +29,20 @@ module RSpecTracer
           file.puts(template('layout').result(binding))
         end
 
-        puts "RSpecTracer generated HTML report to #{file_name}"
+        ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+        puts "RSpecTracer generated HTML report to #{file_name} (took #{elpased})"
       end
 
       private
 
+      def copy_assets
+        Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
+          FileUtils.cp_r(path, asset_output_path)
+        end
+      end
+
       def format_last_run
         @last_run = @reporter.last_run.slice(
           :actual_count,
@@ -57,6 +67,10 @@ module RSpecTracer
         end
       end
 
+      def format_flaky_examples
+        @flaky_examples = @examples.slice(*@reporter.flaky_examples).values
+      end
+
       def example_run_local_time(utc_time)
         case utc_time
         when Time
@@ -128,6 +142,14 @@ module RSpecTracer
         template(title_id).result(current_binding)
       end
 
+      def formatted_flaky_examples(title, flaky_examples)
+        title_id = report_container_id(title)
+        current_binding = binding
+
+        current_binding.local_variable_set(:title_id, title_id)
+        template(title_id).result(current_binding)
+      end
+
       def formatted_examples_dependency(title, examples_dependency)
         title_id = report_container_id(title)
         current_binding = binding
@@ -154,7 +176,7 @@ module RSpecTracer
 
       def example_status_css_class(example_status)
         case example_status.split.first
-        when 'Failed'
+        when 'Failed', 'Flaky'
           'red'
         when 'Pending'
           'yellow'

data/lib/rspec_tracer/html_reporter/views/flaky_examples.erb

@@ -0,0 +1,38 @@
+<div class="report_container" id="<%= title_id %>">
+  <h2>
+    <span class="group_name"><%= title %></span>
+    (
+      <span class="blue">
+        <strong><%= flaky_examples.count %></strong>
+      </span> examples
+    )
+  </h2>
+
+  <a name="<%= title_id %>"></a>
+
+  <div class="report-table--responsive">
+    <table class="report-table">
+      <thead>
+        <tr>
+          <th>ID</th>
+          <th>Description</th>
+          <th>Location</th>
+          <th>Result</th>
+          <th>Run At</th>
+        </tr>
+      </thead>
+
+      <tbody>
+        <% flaky_examples.each do |example| %>
+          <tr>
+            <td><%= example[:id] %></td>
+            <td><%= example[:description] %></td>
+            <td><%= example[:location] %></td>
+            <td><strong class="<%= example_result_css_class(example[:result]) %>"><%= example[:result] %></strong></td>
+            <td width="8%"><%= example[:last_run] %></td>
+          </tr>
+        <% end %>
+      </tbody>
+    </table>
+  </div>
+</div>