rspec-tracer 0.3.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c266f7b90880f1c3ab8c9e272f5abee4834154c6c9b6541d50fe7701655da6e7
4
- data.tar.gz: ce2101a5c35cc334c5e3c061842c0ba27e0c9e7f8ce77e109a047f0eda108a82
3
+ metadata.gz: 4dff7e48382045aef3821a6f863521c9f74823b5bbd12c16b09c41b47277124e
4
+ data.tar.gz: cff630db632b7fa96be3d3de675b994325f815c69613b84d891bc8313688ab67
5
5
  SHA512:
6
- metadata.gz: 5a7152dc475afb4e2ab42aeaf3798108edce0580fcc00a1c2a03b0f319f925633bae59245b7a570938d34372cbb4d134daff5909adf2785325794edcfab36fcd
7
- data.tar.gz: 6a30e7b361d877404692229f60bd63d6e19088c4ac627838aae40af357851a0f2757b4561c843b4d91d6e3f41a6a3c3a29989bd94e1fde33efde701b735852fb
6
+ metadata.gz: 83348356f2a12e930d73cec63e867a13ffab85e428696fc3d91dfeab5a691bac4641d848330d0225e08bc7156d0902faffc8ecd8f4a8871b24c093513c6a8b0d
7
+ data.tar.gz: 2cc9084dbeb9ca0fa26997b33b486360e3c1fe8b7efbb91f28b3a1786bf0e966b32f75b5d7b98a3488c5ced8d1704005f1c1a5972e876e25d3877a40d8068379
data/CHANGELOG.md CHANGED
@@ -1,6 +1,38 @@
1
- ## [Unreleased]
1
+ ## [0.6.1] - 2021-09-06
2
2
 
3
- **[WIP]** Support for CI
3
+ ### Fixed
4
+
5
+ Bug in time formatter (#24)
6
+
7
+ ### Added
8
+
9
+ Environment variable to control verbose output (#25)
10
+
11
+ ## [0.6.0] - 2021-09-05
12
+
13
+ ### Added
14
+
15
+ - Improved dependency change detection (#18)
16
+ - Flaky tests detection (#19)
17
+ - Exclude vendor files from analysis (#21)
18
+ - Report elapsed time at various stages (#23)
19
+
20
+ ### Note
21
+
22
+ The first run on this version will not use any cache on the CI because the number
23
+ of files changed from eight to nine, so there will be no appropriate cache to use.
24
+
25
+ ## [0.5.0] - 2021-09-03
26
+
27
+ ### Fixed
28
+
29
+ - Limit number of cached files download (#16)
30
+
31
+ ## [0.4.0] - 2021-09-03
32
+
33
+ ### Added
34
+
35
+ - Support for CI
4
36
 
5
37
  ## [0.3.0] - 2021-08-30
6
38
 
data/README.md CHANGED
@@ -8,21 +8,55 @@ It uses [Ruby's built-in coverage library](https://ruby-doc.org/stdlib/libdoc/co
8
8
  to keep track of the coverage for each test. For each test executed, the coverage
9
9
  diff provides the desired file list. RSpec Tracer takes care of reporting the
10
10
  **correct code coverage when skipping tests** by using the cached reports. Also,
11
- note that it will **never skip any tests which failed or were pending** in the last runs.
11
+ note that it will **never skip**:
12
+
13
+ - **Flaky examples**
14
+ - **Failed examples**
15
+ - **Pending examples**
12
16
 
13
17
  Knowing the examples and files dependency gives us a better insight into the codebase,
14
18
  and we have **a clear idea of what to test for when making any changes**. With this data,
15
19
  we can also analyze the coupling between different components and much more.
16
20
 
17
- Read more on the intention and the implementation idea [here](./RSPEC_TRACER.md).
18
-
19
21
  ## Note
20
22
 
21
- **RSpec Tracer is currently available for use in the local development
22
- environment only.** Support for CI is work in progress.
23
-
24
- **If you find this gem could be a helpful addition to your project, give it a try
25
- on local and report any issues you encountered.**
23
+ You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing
24
+ the **intention** and implementation details of **managing dependency**, **managing flaky tests**,
25
+ **skipping tests**, and **caching on CI**. You must go through the README file before
26
+ integrating the gem into your project to better understand what is happening.
27
+
28
+ ## Table of Contents
29
+
30
+ * [Demo](#demo)
31
+ * [Installation](#installation)
32
+ * [Compatibility](#compatibility)
33
+ * [Additional Tools](#additional-tools)
34
+ * [Getting Started](#getting-started)
35
+ * [Environment Variables](#environment-variables)
36
+ * [BUNDLE_PATH](#bundle_path)
37
+ * [CI](#ci)
38
+ * [LOCAL_AWS](#local_aws)
39
+ * [RSPEC_TRACER_NO_SKIP](#rspec_tracer_no_skip)
40
+ * [RSPEC_TRACER_S3_URI](#rspec_tracer_s3_uri)
41
+ * [RSPEC_TRACER_UPLOAD_LOCAL_CACHE](#rspec_tracer_upload_local_cache)
42
+ * [RSPEC_TRACER_VERBOSE](#rspec_tracer_verbose)
43
+ * [TEST_SUITES](#test_suites)
44
+ * [TEST_SUITE_ID](#test_suite_id)
45
+ * [Sample Reports](#sample-reports)
46
+ * [Examples](#examples)
47
+ * [Flaky Examples](#flaky-examples)
48
+ * [Examples Dependency](#examples-dependency)
49
+ * [Files Dependency](#files-dependency)
50
+ * [Configuring RSpec Tracer](#configuring-rspec-tracer)
51
+ * [Filters](#filters)
52
+ * [Defining Custom Filteres](#defining-custom-filteres)
53
+ * [String Filter](#string-filter)
54
+ * [Regex Filter](#regex-filter)
55
+ * [Block Filter](#block-filter)
56
+ * [Array Filter](#array-filter)
57
+ * [Contributing](#contributing)
58
+ * [License](#license)
59
+ * [Code of Conduct](#code-of-conduct)
26
60
 
27
61
  ## Demo
28
62
 
@@ -53,6 +87,11 @@ RSpec Tracer requires **Ruby 2.5+** and **rspec-core >= 3.6.0**. To use with **R
53
87
  make sure to use **rspec-rails >= 4.0.0**. If you are using SimpleCov, it is
54
88
  recommended to use **simplecov >= 0.12.0**.
55
89
 
90
+ ### Additional Tools
91
+
92
+ To use RSpec Tracer on CI, you need to have an **S3 bucket** and
93
+ **[AWS CLI](https://aws.amazon.com/cli/)** installed.
94
+
56
95
  ## Getting Started
57
96
 
58
97
  1. **Load and Start RSpec Tracer**
@@ -87,13 +126,50 @@ recommended to use **simplecov >= 0.12.0**.
87
126
  RSpecTracer.start
88
127
  ```
89
128
 
90
- 2. Run the tests with RSpec using `bundle exec rspec`.
91
- 3. After running your tests, open `rspec_tracer_report/index.html` in the
129
+ 2. To enable RSpec Tracer to share cache between different builds on CI, update the
130
+ Rakefile in your project to have the following:
131
+
132
+ ```ruby
133
+ spec = Gem::Specification.find_by_name('rspec-tracer')
134
+
135
+ load "#{spec.gem_dir}/lib/rspec_tracer/remote_cache/Rakefile"
136
+ ```
137
+ 3. Before running tests, download the remote cache using the following rake task:
138
+
139
+ ```sh
140
+ bundle exec rake rspec_tracer:remote_cache:download
141
+ ```
142
+ 4. Run the tests with RSpec using `bundle exec rspec`.
143
+ 5. After running tests, upload the local cache using the following rake task:
144
+
145
+ ```sh
146
+ bundle exec rake rspec_tracer:remote_cache:upload
147
+ ```
148
+ 6. After running your tests, open `rspec_tracer_report/index.html` in the
92
149
  browser of your choice.
93
150
 
94
151
  ## Environment Variables
95
152
 
96
- To get better control on execution, you can use the following two environment variables:
153
+ To get better control on execution, you can use the following environment variables
154
+ whenever required.
155
+
156
+ ### BUNDLE_PATH
157
+
158
+ Since the bundler uses a vendor directory inside the project, it might cause slowness
159
+ depending on the vendor size. You can configure the bundle path outside of the project
160
+ using `BUNDLE_PATH` environment variable, for example, `BUNDLE_PATH=$HOME/vendor/bundle`.
161
+ Make sure to cache this directory in the CI configuration.
162
+
163
+ ### CI
164
+
165
+ Mostly all the CI have `CI=true`. If not, you should explicitly set it to `true`.
166
+
167
+ ### LOCAL_AWS
168
+
169
+ In case you want to test out the caching feature in the local development environment.
170
+ You can install [localstack](https://github.com/localstack/localstack) and
171
+ [awscli-local](https://github.com/localstack/awscli-local) and then invoke the
172
+ rake tasks with `LOCAL_AWS=true`.
97
173
 
98
174
  ### RSPEC_TRACER_NO_SKIP
99
175
 
@@ -104,11 +180,41 @@ any tests. Note that it will continue to maintain cache files and generate repor
104
180
  RSPEC_TRACER_NO_SKIP=true bundle exec rspec
105
181
  ```
106
182
 
183
+ ### RSPEC_TRACER_S3_URI
184
+
185
+ You should provide the S3 bucket path to store the cache files.
186
+
187
+ ```ruby
188
+ export RSPEC_TRACER_S3_URI=s3://ci-artifacts-bucket/rspec-tracer-cache
189
+ ```
190
+
191
+ ### RSPEC_TRACER_UPLOAD_LOCAL_CACHE
192
+
193
+ By default, RSpec Tracer does not upload local cache files. You can set this
194
+ environment variable to `true` to upload the local cache to S3.
195
+
196
+ ### RSPEC_TRACER_VERBOSE
197
+
198
+ To print the intermediate steps and time taken, use this environment variable:
199
+
200
+ ```sh
201
+ export RSPEC_TRACER_VERBOSE=true
202
+ ```
203
+
204
+ ### TEST_SUITES
205
+
206
+ Set this environment variable when using test suite id. It determines the total
207
+ number of different test suites you are running.
208
+
209
+ ```ruby
210
+ export TEST_SUITES=8
211
+ ```
212
+
107
213
  ### TEST_SUITE_ID
108
214
 
109
215
  If you have a large set of tests to run, it is recommended to run them in
110
216
  separate groups. This way, RSpec Tracer is not overwhelmed with loading massive
111
- cached data in the memory. Also, it generate and use cache for specific test suites
217
+ cached data in the memory. Also, it generates and uses cache for specific test suites
112
218
  and not merge them.
113
219
 
114
220
  ```ruby
@@ -116,6 +222,21 @@ TEST_SUITE_ID=1 bundle exec rspec spec/models
116
222
  TEST_SUITE_ID=2 bundle exec rspec spec/helpers
117
223
  ```
118
224
 
225
+ If you run parallel builds on the CI, you should specify the test suite ID and
226
+ the total number of test suites when downloading the cache files.
227
+
228
+ ```sh
229
+ $ TEST_SUITES=5 TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:download
230
+ ```
231
+
232
+ In this case, the appropriate cache should have all the cache files available on
233
+ the S3 for each test suite, not just for the current one. Also, while uploading,
234
+ make sure to provide the test suite id.
235
+
236
+ ```sh
237
+ $ TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:upload
238
+ ```
239
+
119
240
  ## Sample Reports
120
241
 
121
242
  You get the following three reports:
@@ -132,6 +253,19 @@ These reports provide basic test information:
132
253
 
133
254
  ![](./readme_files/examples_report_next_run.png)
134
255
 
256
+ ### Flaky Examples
257
+
258
+ These reports provide flaky tests information. Assuming **the following two tests
259
+ failed in the first run.**
260
+
261
+ **Next Run**
262
+
263
+ ![](./readme_files/flaky_examples_report_first_run.png)
264
+
265
+ **Another Run**
266
+
267
+ ![](./readme_files/flaky_examples_report_next_run.png)
268
+
135
269
  ### Examples Dependency
136
270
 
137
271
  These reports show a list of dependent files for each test.
@@ -2,7 +2,8 @@
2
2
 
3
3
  module RSpecTracer
4
4
  class Cache
5
- attr_reader :all_examples, :failed_examples, :pending_examples, :all_files, :dependency
5
+ attr_reader :all_examples, :flaky_examples, :failed_examples, :pending_examples,
6
+ :all_files, :dependency, :run_id
6
7
 
7
8
  def initialize
8
9
  @run_id = last_run_id
@@ -11,6 +12,7 @@ module RSpecTracer
11
12
  @cached = false
12
13
 
13
14
  @all_examples = {}
15
+ @flaky_examples = Set.new
14
16
  @failed_examples = Set.new
15
17
  @pending_examples = Set.new
16
18
  @all_files = {}
@@ -20,22 +22,36 @@ module RSpecTracer
20
22
  def load_cache_for_run
21
23
  return if @run_id.nil? || @cached
22
24
 
25
+ starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
26
+
23
27
  load_all_examples_cache
28
+ load_flaky_examples_cache
24
29
  load_failed_examples_cache
25
30
  load_pending_examples_cache
26
31
  load_all_files_cache
27
32
  load_dependency_cache
28
33
 
34
+ ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
35
+
29
36
  @cached = true
30
37
 
31
- puts "RSpec tracer loaded cache from #{@cache_dir}" if @run_id
38
+ elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
39
+
40
+ puts "RSpec tracer loaded cache from #{@cache_dir} (took #{elpased})"
32
41
  end
33
42
 
34
43
  def cached_examples_coverage
35
44
  return @examples_coverage if defined?(@examples_coverage)
36
45
  return @examples_coverage = {} if @run_id.nil?
37
46
 
38
- load_examples_coverage_cache
47
+ starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
48
+ coverage = load_examples_coverage_cache
49
+ ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
50
+ elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
51
+
52
+ puts "RSpec tracer loaded cached examples coverage (took #{elpased})" if RSpecTracer.verbose?
53
+
54
+ coverage
39
55
  end
40
56
 
41
57
  private
@@ -64,6 +80,14 @@ module RSpecTracer
64
80
  end
65
81
  end
66
82
 
83
+ def load_flaky_examples_cache
84
+ file_name = File.join(@cache_dir, 'flaky_examples.json')
85
+
86
+ return unless File.file?(file_name)
87
+
88
+ @flaky_examples = JSON.parse(File.read(file_name)).to_set
89
+ end
90
+
67
91
  def load_failed_examples_cache
68
92
  file_name = File.join(@cache_dir, 'failed_examples.json')
69
93
 
@@ -102,6 +102,12 @@ module RSpecTracer
102
102
  @coverage_filters ||= []
103
103
  end
104
104
 
105
+ def verbose?
106
+ return @verbose if defined?(@verbose)
107
+
108
+ @verbose = ENV.fetch('RSPEC_TRACER_VERBOSE', 'false') == 'true'
109
+ end
110
+
105
111
  def configure(&block)
106
112
  Docile.dsl_eval(self, &block)
107
113
  end
@@ -2,7 +2,13 @@
2
2
 
3
3
  RSpecTracer.configure do
4
4
  add_filter '/vendor/bundle/'
5
- add_coverage_filter %w[/autotest/ /features/ /spec/ /test/].freeze
5
+ add_coverage_filter %w[
6
+ /autotest/
7
+ /features/
8
+ /spec/
9
+ /test/
10
+ /vendor/bundle/
11
+ ].freeze
6
12
  end
7
13
 
8
14
  at_exit do
@@ -6,21 +6,22 @@ require 'time'
6
6
  module RSpecTracer
7
7
  module HTMLReporter
8
8
  class Reporter
9
- attr_reader :last_run, :examples, :examples_dependency, :files_dependency
9
+ attr_reader :last_run, :examples, :flaky_examples, :examples_dependency, :files_dependency
10
10
 
11
11
  def initialize
12
12
  @reporter = RSpecTracer.runner.reporter
13
13
 
14
14
  format_last_run
15
15
  format_examples
16
+ format_flaky_examples
16
17
  format_examples_dependency
17
18
  format_files_dependency
18
19
  end
19
20
 
20
21
  def generate_report
21
- Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
22
- FileUtils.cp_r(path, asset_output_path)
23
- end
22
+ starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
23
+
24
+ copy_assets
24
25
 
25
26
  file_name = File.join(RSpecTracer.report_path, 'index.html')
26
27
 
@@ -28,11 +29,20 @@ module RSpecTracer
28
29
  file.puts(template('layout').result(binding))
29
30
  end
30
31
 
31
- puts "RSpecTracer generated HTML report to #{file_name}"
32
+ ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
33
+ elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
34
+
35
+ puts "RSpecTracer generated HTML report to #{file_name} (took #{elpased})"
32
36
  end
33
37
 
34
38
  private
35
39
 
40
+ def copy_assets
41
+ Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
42
+ FileUtils.cp_r(path, asset_output_path)
43
+ end
44
+ end
45
+
36
46
  def format_last_run
37
47
  @last_run = @reporter.last_run.slice(
38
48
  :actual_count,
@@ -57,6 +67,10 @@ module RSpecTracer
57
67
  end
58
68
  end
59
69
 
70
+ def format_flaky_examples
71
+ @flaky_examples = @examples.slice(*@reporter.flaky_examples).values
72
+ end
73
+
60
74
  def example_run_local_time(utc_time)
61
75
  case utc_time
62
76
  when Time
@@ -128,6 +142,14 @@ module RSpecTracer
128
142
  template(title_id).result(current_binding)
129
143
  end
130
144
 
145
+ def formatted_flaky_examples(title, flaky_examples)
146
+ title_id = report_container_id(title)
147
+ current_binding = binding
148
+
149
+ current_binding.local_variable_set(:title_id, title_id)
150
+ template(title_id).result(current_binding)
151
+ end
152
+
131
153
  def formatted_examples_dependency(title, examples_dependency)
132
154
  title_id = report_container_id(title)
133
155
  current_binding = binding
@@ -154,7 +176,7 @@ module RSpecTracer
154
176
 
155
177
  def example_status_css_class(example_status)
156
178
  case example_status.split.first
157
- when 'Failed'
179
+ when 'Failed', 'Flaky'
158
180
  'red'
159
181
  when 'Pending'
160
182
  'yellow'