rspec-tracer 0.3.0 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c266f7b90880f1c3ab8c9e272f5abee4834154c6c9b6541d50fe7701655da6e7
-  data.tar.gz: ce2101a5c35cc334c5e3c061842c0ba27e0c9e7f8ce77e109a047f0eda108a82
+  metadata.gz: 4dff7e48382045aef3821a6f863521c9f74823b5bbd12c16b09c41b47277124e
+  data.tar.gz: cff630db632b7fa96be3d3de675b994325f815c69613b84d891bc8313688ab67
 SHA512:
-  metadata.gz: 5a7152dc475afb4e2ab42aeaf3798108edce0580fcc00a1c2a03b0f319f925633bae59245b7a570938d34372cbb4d134daff5909adf2785325794edcfab36fcd
-  data.tar.gz: 6a30e7b361d877404692229f60bd63d6e19088c4ac627838aae40af357851a0f2757b4561c843b4d91d6e3f41a6a3c3a29989bd94e1fde33efde701b735852fb
+  metadata.gz: 83348356f2a12e930d73cec63e867a13ffab85e428696fc3d91dfeab5a691bac4641d848330d0225e08bc7156d0902faffc8ecd8f4a8871b24c093513c6a8b0d
+  data.tar.gz: 2cc9084dbeb9ca0fa26997b33b486360e3c1fe8b7efbb91f28b3a1786bf0e966b32f75b5d7b98a3488c5ced8d1704005f1c1a5972e876e25d3877a40d8068379

data/CHANGELOG.md

@@ -1,6 +1,38 @@
-## [Unreleased]
+## [0.6.1] - 2021-09-06
 
-**[WIP]** Support for CI
+### Fixed
+
+Bug in time formatter (#24)
+
+### Added
+
+Environment variable to control verbose output (#25)
+
+## [0.6.0] - 2021-09-05
+
+### Added
+
+- Improved dependency change detection (#18)
+- Flaky tests detection (#19)
+- Exclude vendor files from analysis (#21)
+- Report elapsed time at various stages (#23)
+
+### Note
+
+The first run on this version will not use any cache on the CI because the number
+of files changed from eight to nine, so there will be no appropriate cache to use.
+
+## [0.5.0] - 2021-09-03
+
+### Fixed
+
+- Limit number of cached files download (#16)
+
+## [0.4.0] - 2021-09-03
+
+### Added
+
+- Support for CI
 
 ## [0.3.0] - 2021-08-30
 

data/README.md

@@ -8,21 +8,55 @@ It uses [Ruby's built-in coverage library](https://ruby-doc.org/stdlib/libdoc/co
 to keep track of the coverage for each test. For each test executed, the coverage
 diff provides the desired file list. RSpec Tracer takes care of reporting the
 **correct code coverage when skipping tests** by using the cached reports. Also,
-note that it will **never skip any tests which failed or were pending** in the last runs.
+note that it will **never skip**:
+
+- **Flaky examples**
+- **Failed examples**
+- **Pending examples**
 
 Knowing the examples and files dependency gives us a better insight into the codebase,
 and we have **a clear idea of what to test for when making any changes**. With this data,
 we can also analyze the coupling between different components and much more.
 
-Read more on the intention and the implementation idea [here](./RSPEC_TRACER.md).
-
 ## Note
 
-**RSpec Tracer is currently available for use in the local development
-environment only.** Support for CI is work in progress.
-
-**If you find this gem could be a helpful addition to your project, give it a try
-on local and report any issues you encountered.**
+You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing
+the **intention** and implementation details of **managing dependency**, **managing flaky tests**,
+**skipping tests**, and **caching on CI**. You must go through the README file before
+integrating the gem into your project to better understand what is happening.
+
+## Table of Contents
+
+* [Demo](#demo)
+* [Installation](#installation)
+  * [Compatibility](#compatibility)
+  * [Additional Tools](#additional-tools)
+* [Getting Started](#getting-started)
+* [Environment Variables](#environment-variables)
+  * [BUNDLE_PATH](#bundle_path)
+  * [CI](#ci)
+  * [LOCAL_AWS](#local_aws)
+  * [RSPEC_TRACER_NO_SKIP](#rspec_tracer_no_skip)
+  * [RSPEC_TRACER_S3_URI](#rspec_tracer_s3_uri)
+  * [RSPEC_TRACER_UPLOAD_LOCAL_CACHE](#rspec_tracer_upload_local_cache)
+  * [RSPEC_TRACER_VERBOSE](#rspec_tracer_verbose)
+  * [TEST_SUITES](#test_suites)
+  * [TEST_SUITE_ID](#test_suite_id)
+* [Sample Reports](#sample-reports)
+    * [Examples](#examples)
+    * [Flaky Examples](#flaky-examples)
+    * [Examples Dependency](#examples-dependency)
+    * [Files Dependency](#files-dependency)
+* [Configuring RSpec Tracer](#configuring-rspec-tracer)
+* [Filters](#filters)
+  * [Defining Custom Filteres](#defining-custom-filteres)
+    * [String Filter](#string-filter)
+    * [Regex Filter](#regex-filter)
+    * [Block Filter](#block-filter)
+    * [Array Filter](#array-filter)
+* [Contributing](#contributing)
+* [License](#license)
+* [Code of Conduct](#code-of-conduct)
 
 ## Demo
 
@@ -53,6 +87,11 @@ RSpec Tracer requires **Ruby 2.5+** and **rspec-core >= 3.6.0**. To use with **R
 make sure to use **rspec-rails >= 4.0.0**. If you are using SimpleCov, it is
 recommended to use **simplecov >= 0.12.0**.
 
+### Additional Tools
+
+To use RSpec Tracer on CI, you need to have an **S3 bucket** and
+**[AWS CLI](https://aws.amazon.com/cli/)** installed.
+
 ## Getting Started
 
 1. **Load and Start RSpec Tracer**
@@ -87,13 +126,50 @@ recommended to use **simplecov >= 0.12.0**.
         RSpecTracer.start
         ```
 
-2. Run the tests with RSpec using `bundle exec rspec`.
-3. After running your tests, open `rspec_tracer_report/index.html` in the
+2. To enable RSpec Tracer to share cache between different builds on CI, update the
+Rakefile in your project to have the following:
+
+    ```ruby
+    spec = Gem::Specification.find_by_name('rspec-tracer')
+
+    load "#{spec.gem_dir}/lib/rspec_tracer/remote_cache/Rakefile"
+    ```
+3. Before running tests, download the remote cache using the following rake task:
+
+    ```sh
+    bundle exec rake rspec_tracer:remote_cache:download
+    ```
+4. Run the tests with RSpec using `bundle exec rspec`.
+5. After running tests, upload the local cache using the following rake task:
+
+    ```sh
+    bundle exec rake rspec_tracer:remote_cache:upload
+    ```
+6. After running your tests, open `rspec_tracer_report/index.html` in the
 browser of your choice.
 
 ## Environment Variables
 
-To get better control on execution, you can use the following two environment variables:
+To get better control on execution, you can use the following environment variables
+whenever required.
+
+### BUNDLE_PATH
+
+Since the bundler uses a vendor directory inside the project, it might cause slowness
+depending on the vendor size. You can configure the bundle path outside of the project
+using `BUNDLE_PATH` environment variable, for example, `BUNDLE_PATH=$HOME/vendor/bundle`.
+Make sure to cache this directory in the CI configuration.
+
+### CI
+
+Mostly all the CI have `CI=true`. If not, you should explicitly set it to `true`.
+
+### LOCAL_AWS
+
+In case you want to test out the caching feature in the local development environment.
+You can install [localstack](https://github.com/localstack/localstack) and
+[awscli-local](https://github.com/localstack/awscli-local) and then invoke the
+rake tasks with `LOCAL_AWS=true`.
 
 ### RSPEC_TRACER_NO_SKIP
 
@@ -104,11 +180,41 @@ any tests. Note that it will continue to maintain cache files and generate repor
 RSPEC_TRACER_NO_SKIP=true bundle exec rspec
 ```
 
+### RSPEC_TRACER_S3_URI
+
+You should provide the S3 bucket path to store the cache files.
+
+```ruby
+export RSPEC_TRACER_S3_URI=s3://ci-artifacts-bucket/rspec-tracer-cache
+```
+
+### RSPEC_TRACER_UPLOAD_LOCAL_CACHE
+
+By default, RSpec Tracer does not upload local cache files. You can set this
+environment variable to `true` to upload the local cache to S3.
+
+### RSPEC_TRACER_VERBOSE
+
+To print the intermediate steps and time taken, use this environment variable:
+
+```sh
+export RSPEC_TRACER_VERBOSE=true
+```
+
+### TEST_SUITES
+
+Set this environment variable when using test suite id. It determines the total
+number of different test suites you are running.
+
+```ruby
+export TEST_SUITES=8
+```
+
 ### TEST_SUITE_ID
 
 If you have a large set of tests to run, it is recommended to run them in
 separate groups. This way, RSpec Tracer is not overwhelmed with loading massive
-cached data in the memory. Also, it generate and use cache for specific test suites
+cached data in the memory. Also, it generates and uses cache for specific test suites
 and not merge them.
 
 ```ruby
@@ -116,6 +222,21 @@ TEST_SUITE_ID=1 bundle exec rspec spec/models
 TEST_SUITE_ID=2 bundle exec rspec spec/helpers
 ```
 
+If you run parallel builds on the CI, you should specify the test suite ID and
+the total number of test suites when downloading the cache files.
+
+```sh
+$ TEST_SUITES=5 TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:download
+```
+
+In this case, the appropriate cache should have all the cache files available on
+the S3 for each test suite, not just for the current one. Also, while uploading,
+make sure to provide the test suite id.
+
+```sh
+$ TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:upload
+```
+
 ## Sample Reports
 
 You get the following three reports:
@@ -132,6 +253,19 @@ These reports provide basic test information:
 
 ![](./readme_files/examples_report_next_run.png)
 
+### Flaky Examples
+
+These reports provide flaky tests information. Assuming **the following two tests
+failed in the first run.**
+
+**Next Run**
+
+![](./readme_files/flaky_examples_report_first_run.png)
+
+**Another Run**
+
+![](./readme_files/flaky_examples_report_next_run.png)
+
 ### Examples Dependency
 
 These reports show a list of dependent files for each test.

data/lib/rspec_tracer/cache.rb

@@ -2,7 +2,8 @@
 
 module RSpecTracer
   class Cache
-    attr_reader :all_examples, :failed_examples, :pending_examples, :all_files, :dependency
+    attr_reader :all_examples, :flaky_examples, :failed_examples, :pending_examples,
+                :all_files, :dependency, :run_id
 
     def initialize
       @run_id = last_run_id
@@ -11,6 +12,7 @@ module RSpecTracer
       @cached = false
 
       @all_examples = {}
+      @flaky_examples = Set.new
       @failed_examples = Set.new
       @pending_examples = Set.new
       @all_files = {}
@@ -20,22 +22,36 @@ module RSpecTracer
     def load_cache_for_run
       return if @run_id.nil? || @cached
 
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       load_all_examples_cache
+      load_flaky_examples_cache
       load_failed_examples_cache
       load_pending_examples_cache
       load_all_files_cache
       load_dependency_cache
 
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
       @cached = true
 
-      puts "RSpec tracer loaded cache from #{@cache_dir}" if @run_id
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer loaded cache from #{@cache_dir} (took #{elpased})"
     end
 
     def cached_examples_coverage
       return @examples_coverage if defined?(@examples_coverage)
       return @examples_coverage = {} if @run_id.nil?
 
-      load_examples_coverage_cache
+      starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      coverage = load_examples_coverage_cache
+      ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+      puts "RSpec tracer loaded cached examples coverage (took #{elpased})" if RSpecTracer.verbose?
+
+      coverage
     end
 
     private
@@ -64,6 +80,14 @@ module RSpecTracer
       end
     end
 
+    def load_flaky_examples_cache
+      file_name = File.join(@cache_dir, 'flaky_examples.json')
+
+      return unless File.file?(file_name)
+
+      @flaky_examples = JSON.parse(File.read(file_name)).to_set
+    end
+
     def load_failed_examples_cache
       file_name = File.join(@cache_dir, 'failed_examples.json')
 

data/lib/rspec_tracer/configuration.rb

@@ -102,6 +102,12 @@ module RSpecTracer
       @coverage_filters ||= []
     end
 
+    def verbose?
+      return @verbose if defined?(@verbose)
+
+      @verbose = ENV.fetch('RSPEC_TRACER_VERBOSE', 'false') == 'true'
+    end
+
     def configure(&block)
       Docile.dsl_eval(self, &block)
     end

data/lib/rspec_tracer/defaults.rb

@@ -2,7 +2,13 @@
 
 RSpecTracer.configure do
   add_filter '/vendor/bundle/'
-  add_coverage_filter %w[/autotest/ /features/ /spec/ /test/].freeze
+  add_coverage_filter %w[
+    /autotest/
+    /features/
+    /spec/
+    /test/
+    /vendor/bundle/
+  ].freeze
 end
 
 at_exit do

data/lib/rspec_tracer/html_reporter/reporter.rb

@@ -6,21 +6,22 @@ require 'time'
 module RSpecTracer
   module HTMLReporter
     class Reporter
-      attr_reader :last_run, :examples, :examples_dependency, :files_dependency
+      attr_reader :last_run, :examples, :flaky_examples, :examples_dependency, :files_dependency
 
       def initialize
         @reporter = RSpecTracer.runner.reporter
 
         format_last_run
         format_examples
+        format_flaky_examples
         format_examples_dependency
         format_files_dependency
       end
 
       def generate_report
-        Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
-          FileUtils.cp_r(path, asset_output_path)
-        end
+        starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+        copy_assets
 
         file_name = File.join(RSpecTracer.report_path, 'index.html')
 
@@ -28,11 +29,20 @@ module RSpecTracer
           file.puts(template('layout').result(binding))
         end
 
-        puts "RSpecTracer generated HTML report to #{file_name}"
+        ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
+
+        puts "RSpecTracer generated HTML report to #{file_name} (took #{elpased})"
       end
 
       private
 
+      def copy_assets
+        Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
+          FileUtils.cp_r(path, asset_output_path)
+        end
+      end
+
       def format_last_run
         @last_run = @reporter.last_run.slice(
           :actual_count,
@@ -57,6 +67,10 @@ module RSpecTracer
         end
       end
 
+      def format_flaky_examples
+        @flaky_examples = @examples.slice(*@reporter.flaky_examples).values
+      end
+
       def example_run_local_time(utc_time)
         case utc_time
         when Time
@@ -128,6 +142,14 @@ module RSpecTracer
         template(title_id).result(current_binding)
       end
 
+      def formatted_flaky_examples(title, flaky_examples)
+        title_id = report_container_id(title)
+        current_binding = binding
+
+        current_binding.local_variable_set(:title_id, title_id)
+        template(title_id).result(current_binding)
+      end
+
       def formatted_examples_dependency(title, examples_dependency)
         title_id = report_container_id(title)
         current_binding = binding
@@ -154,7 +176,7 @@ module RSpecTracer
 
       def example_status_css_class(example_status)
         case example_status.split.first
-        when 'Failed'
+        when 'Failed', 'Flaky'
           'red'
         when 'Pending'
           'yellow'