rspec-tracer 0.3.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +34 -2
- data/README.md +146 -12
- data/lib/rspec_tracer/cache.rb +27 -3
- data/lib/rspec_tracer/configuration.rb +6 -0
- data/lib/rspec_tracer/defaults.rb +7 -1
- data/lib/rspec_tracer/html_reporter/reporter.rb +28 -6
- data/lib/rspec_tracer/html_reporter/views/flaky_examples.erb +38 -0
- data/lib/rspec_tracer/html_reporter/views/layout.erb +3 -0
- data/lib/rspec_tracer/remote_cache/cache.rb +187 -0
- data/lib/rspec_tracer/remote_cache/git.rb +113 -0
- data/lib/rspec_tracer/reporter.rb +40 -11
- data/lib/rspec_tracer/rspec_runner.rb +6 -1
- data/lib/rspec_tracer/runner.rb +84 -34
- data/lib/rspec_tracer/time_formatter.rb +55 -0
- data/lib/rspec_tracer/version.rb +1 -1
- data/lib/rspec_tracer.rb +30 -7
- metadata +7 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4dff7e48382045aef3821a6f863521c9f74823b5bbd12c16b09c41b47277124e
|
4
|
+
data.tar.gz: cff630db632b7fa96be3d3de675b994325f815c69613b84d891bc8313688ab67
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 83348356f2a12e930d73cec63e867a13ffab85e428696fc3d91dfeab5a691bac4641d848330d0225e08bc7156d0902faffc8ecd8f4a8871b24c093513c6a8b0d
|
7
|
+
data.tar.gz: 2cc9084dbeb9ca0fa26997b33b486360e3c1fe8b7efbb91f28b3a1786bf0e966b32f75b5d7b98a3488c5ced8d1704005f1c1a5972e876e25d3877a40d8068379
|
data/CHANGELOG.md
CHANGED
@@ -1,6 +1,38 @@
|
|
1
|
-
## [
|
1
|
+
## [0.6.1] - 2021-09-06
|
2
2
|
|
3
|
-
|
3
|
+
### Fixed
|
4
|
+
|
5
|
+
Bug in time formatter (#24)
|
6
|
+
|
7
|
+
### Added
|
8
|
+
|
9
|
+
Environment variable to control verbose output (#25)
|
10
|
+
|
11
|
+
## [0.6.0] - 2021-09-05
|
12
|
+
|
13
|
+
### Added
|
14
|
+
|
15
|
+
- Improved dependency change detection (#18)
|
16
|
+
- Flaky tests detection (#19)
|
17
|
+
- Exclude vendor files from analysis (#21)
|
18
|
+
- Report elapsed time at various stages (#23)
|
19
|
+
|
20
|
+
### Note
|
21
|
+
|
22
|
+
The first run on this version will not use any cache on the CI because the number
|
23
|
+
of files changed from eight to nine, so there will be no appropriate cache to use.
|
24
|
+
|
25
|
+
## [0.5.0] - 2021-09-03
|
26
|
+
|
27
|
+
### Fixed
|
28
|
+
|
29
|
+
- Limit number of cached files download (#16)
|
30
|
+
|
31
|
+
## [0.4.0] - 2021-09-03
|
32
|
+
|
33
|
+
### Added
|
34
|
+
|
35
|
+
- Support for CI
|
4
36
|
|
5
37
|
## [0.3.0] - 2021-08-30
|
6
38
|
|
data/README.md
CHANGED
@@ -8,21 +8,55 @@ It uses [Ruby's built-in coverage library](https://ruby-doc.org/stdlib/libdoc/co
|
|
8
8
|
to keep track of the coverage for each test. For each test executed, the coverage
|
9
9
|
diff provides the desired file list. RSpec Tracer takes care of reporting the
|
10
10
|
**correct code coverage when skipping tests** by using the cached reports. Also,
|
11
|
-
note that it will **never skip
|
11
|
+
note that it will **never skip**:
|
12
|
+
|
13
|
+
- **Flaky examples**
|
14
|
+
- **Failed examples**
|
15
|
+
- **Pending examples**
|
12
16
|
|
13
17
|
Knowing the examples and files dependency gives us a better insight into the codebase,
|
14
18
|
and we have **a clear idea of what to test for when making any changes**. With this data,
|
15
19
|
we can also analyze the coupling between different components and much more.
|
16
20
|
|
17
|
-
Read more on the intention and the implementation idea [here](./RSPEC_TRACER.md).
|
18
|
-
|
19
21
|
## Note
|
20
22
|
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
23
|
+
You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing
|
24
|
+
the **intention** and implementation details of **managing dependency**, **managing flaky tests**,
|
25
|
+
**skipping tests**, and **caching on CI**. You must go through the README file before
|
26
|
+
integrating the gem into your project to better understand what is happening.
|
27
|
+
|
28
|
+
## Table of Contents
|
29
|
+
|
30
|
+
* [Demo](#demo)
|
31
|
+
* [Installation](#installation)
|
32
|
+
* [Compatibility](#compatibility)
|
33
|
+
* [Additional Tools](#additional-tools)
|
34
|
+
* [Getting Started](#getting-started)
|
35
|
+
* [Environment Variables](#environment-variables)
|
36
|
+
* [BUNDLE_PATH](#bundle_path)
|
37
|
+
* [CI](#ci)
|
38
|
+
* [LOCAL_AWS](#local_aws)
|
39
|
+
* [RSPEC_TRACER_NO_SKIP](#rspec_tracer_no_skip)
|
40
|
+
* [RSPEC_TRACER_S3_URI](#rspec_tracer_s3_uri)
|
41
|
+
* [RSPEC_TRACER_UPLOAD_LOCAL_CACHE](#rspec_tracer_upload_local_cache)
|
42
|
+
* [RSPEC_TRACER_VERBOSE](#rspec_tracer_verbose)
|
43
|
+
* [TEST_SUITES](#test_suites)
|
44
|
+
* [TEST_SUITE_ID](#test_suite_id)
|
45
|
+
* [Sample Reports](#sample-reports)
|
46
|
+
* [Examples](#examples)
|
47
|
+
* [Flaky Examples](#flaky-examples)
|
48
|
+
* [Examples Dependency](#examples-dependency)
|
49
|
+
* [Files Dependency](#files-dependency)
|
50
|
+
* [Configuring RSpec Tracer](#configuring-rspec-tracer)
|
51
|
+
* [Filters](#filters)
|
52
|
+
* [Defining Custom Filteres](#defining-custom-filteres)
|
53
|
+
* [String Filter](#string-filter)
|
54
|
+
* [Regex Filter](#regex-filter)
|
55
|
+
* [Block Filter](#block-filter)
|
56
|
+
* [Array Filter](#array-filter)
|
57
|
+
* [Contributing](#contributing)
|
58
|
+
* [License](#license)
|
59
|
+
* [Code of Conduct](#code-of-conduct)
|
26
60
|
|
27
61
|
## Demo
|
28
62
|
|
@@ -53,6 +87,11 @@ RSpec Tracer requires **Ruby 2.5+** and **rspec-core >= 3.6.0**. To use with **R
|
|
53
87
|
make sure to use **rspec-rails >= 4.0.0**. If you are using SimpleCov, it is
|
54
88
|
recommended to use **simplecov >= 0.12.0**.
|
55
89
|
|
90
|
+
### Additional Tools
|
91
|
+
|
92
|
+
To use RSpec Tracer on CI, you need to have an **S3 bucket** and
|
93
|
+
**[AWS CLI](https://aws.amazon.com/cli/)** installed.
|
94
|
+
|
56
95
|
## Getting Started
|
57
96
|
|
58
97
|
1. **Load and Start RSpec Tracer**
|
@@ -87,13 +126,50 @@ recommended to use **simplecov >= 0.12.0**.
|
|
87
126
|
RSpecTracer.start
|
88
127
|
```
|
89
128
|
|
90
|
-
2.
|
91
|
-
|
129
|
+
2. To enable RSpec Tracer to share cache between different builds on CI, update the
|
130
|
+
Rakefile in your project to have the following:
|
131
|
+
|
132
|
+
```ruby
|
133
|
+
spec = Gem::Specification.find_by_name('rspec-tracer')
|
134
|
+
|
135
|
+
load "#{spec.gem_dir}/lib/rspec_tracer/remote_cache/Rakefile"
|
136
|
+
```
|
137
|
+
3. Before running tests, download the remote cache using the following rake task:
|
138
|
+
|
139
|
+
```sh
|
140
|
+
bundle exec rake rspec_tracer:remote_cache:download
|
141
|
+
```
|
142
|
+
4. Run the tests with RSpec using `bundle exec rspec`.
|
143
|
+
5. After running tests, upload the local cache using the following rake task:
|
144
|
+
|
145
|
+
```sh
|
146
|
+
bundle exec rake rspec_tracer:remote_cache:upload
|
147
|
+
```
|
148
|
+
6. After running your tests, open `rspec_tracer_report/index.html` in the
|
92
149
|
browser of your choice.
|
93
150
|
|
94
151
|
## Environment Variables
|
95
152
|
|
96
|
-
To get better control on execution, you can use the following
|
153
|
+
To get better control on execution, you can use the following environment variables
|
154
|
+
whenever required.
|
155
|
+
|
156
|
+
### BUNDLE_PATH
|
157
|
+
|
158
|
+
Since the bundler uses a vendor directory inside the project, it might cause slowness
|
159
|
+
depending on the vendor size. You can configure the bundle path outside of the project
|
160
|
+
using `BUNDLE_PATH` environment variable, for example, `BUNDLE_PATH=$HOME/vendor/bundle`.
|
161
|
+
Make sure to cache this directory in the CI configuration.
|
162
|
+
|
163
|
+
### CI
|
164
|
+
|
165
|
+
Mostly all the CI have `CI=true`. If not, you should explicitly set it to `true`.
|
166
|
+
|
167
|
+
### LOCAL_AWS
|
168
|
+
|
169
|
+
In case you want to test out the caching feature in the local development environment.
|
170
|
+
You can install [localstack](https://github.com/localstack/localstack) and
|
171
|
+
[awscli-local](https://github.com/localstack/awscli-local) and then invoke the
|
172
|
+
rake tasks with `LOCAL_AWS=true`.
|
97
173
|
|
98
174
|
### RSPEC_TRACER_NO_SKIP
|
99
175
|
|
@@ -104,11 +180,41 @@ any tests. Note that it will continue to maintain cache files and generate repor
|
|
104
180
|
RSPEC_TRACER_NO_SKIP=true bundle exec rspec
|
105
181
|
```
|
106
182
|
|
183
|
+
### RSPEC_TRACER_S3_URI
|
184
|
+
|
185
|
+
You should provide the S3 bucket path to store the cache files.
|
186
|
+
|
187
|
+
```ruby
|
188
|
+
export RSPEC_TRACER_S3_URI=s3://ci-artifacts-bucket/rspec-tracer-cache
|
189
|
+
```
|
190
|
+
|
191
|
+
### RSPEC_TRACER_UPLOAD_LOCAL_CACHE
|
192
|
+
|
193
|
+
By default, RSpec Tracer does not upload local cache files. You can set this
|
194
|
+
environment variable to `true` to upload the local cache to S3.
|
195
|
+
|
196
|
+
### RSPEC_TRACER_VERBOSE
|
197
|
+
|
198
|
+
To print the intermediate steps and time taken, use this environment variable:
|
199
|
+
|
200
|
+
```sh
|
201
|
+
export RSPEC_TRACER_VERBOSE=true
|
202
|
+
```
|
203
|
+
|
204
|
+
### TEST_SUITES
|
205
|
+
|
206
|
+
Set this environment variable when using test suite id. It determines the total
|
207
|
+
number of different test suites you are running.
|
208
|
+
|
209
|
+
```ruby
|
210
|
+
export TEST_SUITES=8
|
211
|
+
```
|
212
|
+
|
107
213
|
### TEST_SUITE_ID
|
108
214
|
|
109
215
|
If you have a large set of tests to run, it is recommended to run them in
|
110
216
|
separate groups. This way, RSpec Tracer is not overwhelmed with loading massive
|
111
|
-
cached data in the memory. Also, it
|
217
|
+
cached data in the memory. Also, it generates and uses cache for specific test suites
|
112
218
|
and not merge them.
|
113
219
|
|
114
220
|
```ruby
|
@@ -116,6 +222,21 @@ TEST_SUITE_ID=1 bundle exec rspec spec/models
|
|
116
222
|
TEST_SUITE_ID=2 bundle exec rspec spec/helpers
|
117
223
|
```
|
118
224
|
|
225
|
+
If you run parallel builds on the CI, you should specify the test suite ID and
|
226
|
+
the total number of test suites when downloading the cache files.
|
227
|
+
|
228
|
+
```sh
|
229
|
+
$ TEST_SUITES=5 TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:download
|
230
|
+
```
|
231
|
+
|
232
|
+
In this case, the appropriate cache should have all the cache files available on
|
233
|
+
the S3 for each test suite, not just for the current one. Also, while uploading,
|
234
|
+
make sure to provide the test suite id.
|
235
|
+
|
236
|
+
```sh
|
237
|
+
$ TEST_SUITE_ID=1 bundle exec rake rspec_tracer:remote_cache:upload
|
238
|
+
```
|
239
|
+
|
119
240
|
## Sample Reports
|
120
241
|
|
121
242
|
You get the following three reports:
|
@@ -132,6 +253,19 @@ These reports provide basic test information:
|
|
132
253
|
|
133
254
|

|
134
255
|
|
256
|
+
### Flaky Examples
|
257
|
+
|
258
|
+
These reports provide flaky tests information. Assuming **the following two tests
|
259
|
+
failed in the first run.**
|
260
|
+
|
261
|
+
**Next Run**
|
262
|
+
|
263
|
+

|
264
|
+
|
265
|
+
**Another Run**
|
266
|
+
|
267
|
+

|
268
|
+
|
135
269
|
### Examples Dependency
|
136
270
|
|
137
271
|
These reports show a list of dependent files for each test.
|
data/lib/rspec_tracer/cache.rb
CHANGED
@@ -2,7 +2,8 @@
|
|
2
2
|
|
3
3
|
module RSpecTracer
|
4
4
|
class Cache
|
5
|
-
attr_reader :all_examples, :
|
5
|
+
attr_reader :all_examples, :flaky_examples, :failed_examples, :pending_examples,
|
6
|
+
:all_files, :dependency, :run_id
|
6
7
|
|
7
8
|
def initialize
|
8
9
|
@run_id = last_run_id
|
@@ -11,6 +12,7 @@ module RSpecTracer
|
|
11
12
|
@cached = false
|
12
13
|
|
13
14
|
@all_examples = {}
|
15
|
+
@flaky_examples = Set.new
|
14
16
|
@failed_examples = Set.new
|
15
17
|
@pending_examples = Set.new
|
16
18
|
@all_files = {}
|
@@ -20,22 +22,36 @@ module RSpecTracer
|
|
20
22
|
def load_cache_for_run
|
21
23
|
return if @run_id.nil? || @cached
|
22
24
|
|
25
|
+
starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
26
|
+
|
23
27
|
load_all_examples_cache
|
28
|
+
load_flaky_examples_cache
|
24
29
|
load_failed_examples_cache
|
25
30
|
load_pending_examples_cache
|
26
31
|
load_all_files_cache
|
27
32
|
load_dependency_cache
|
28
33
|
|
34
|
+
ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
35
|
+
|
29
36
|
@cached = true
|
30
37
|
|
31
|
-
|
38
|
+
elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
|
39
|
+
|
40
|
+
puts "RSpec tracer loaded cache from #{@cache_dir} (took #{elpased})"
|
32
41
|
end
|
33
42
|
|
34
43
|
def cached_examples_coverage
|
35
44
|
return @examples_coverage if defined?(@examples_coverage)
|
36
45
|
return @examples_coverage = {} if @run_id.nil?
|
37
46
|
|
38
|
-
|
47
|
+
starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
48
|
+
coverage = load_examples_coverage_cache
|
49
|
+
ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
50
|
+
elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
|
51
|
+
|
52
|
+
puts "RSpec tracer loaded cached examples coverage (took #{elpased})" if RSpecTracer.verbose?
|
53
|
+
|
54
|
+
coverage
|
39
55
|
end
|
40
56
|
|
41
57
|
private
|
@@ -64,6 +80,14 @@ module RSpecTracer
|
|
64
80
|
end
|
65
81
|
end
|
66
82
|
|
83
|
+
def load_flaky_examples_cache
|
84
|
+
file_name = File.join(@cache_dir, 'flaky_examples.json')
|
85
|
+
|
86
|
+
return unless File.file?(file_name)
|
87
|
+
|
88
|
+
@flaky_examples = JSON.parse(File.read(file_name)).to_set
|
89
|
+
end
|
90
|
+
|
67
91
|
def load_failed_examples_cache
|
68
92
|
file_name = File.join(@cache_dir, 'failed_examples.json')
|
69
93
|
|
@@ -102,6 +102,12 @@ module RSpecTracer
|
|
102
102
|
@coverage_filters ||= []
|
103
103
|
end
|
104
104
|
|
105
|
+
def verbose?
|
106
|
+
return @verbose if defined?(@verbose)
|
107
|
+
|
108
|
+
@verbose = ENV.fetch('RSPEC_TRACER_VERBOSE', 'false') == 'true'
|
109
|
+
end
|
110
|
+
|
105
111
|
def configure(&block)
|
106
112
|
Docile.dsl_eval(self, &block)
|
107
113
|
end
|
@@ -6,21 +6,22 @@ require 'time'
|
|
6
6
|
module RSpecTracer
|
7
7
|
module HTMLReporter
|
8
8
|
class Reporter
|
9
|
-
attr_reader :last_run, :examples, :examples_dependency, :files_dependency
|
9
|
+
attr_reader :last_run, :examples, :flaky_examples, :examples_dependency, :files_dependency
|
10
10
|
|
11
11
|
def initialize
|
12
12
|
@reporter = RSpecTracer.runner.reporter
|
13
13
|
|
14
14
|
format_last_run
|
15
15
|
format_examples
|
16
|
+
format_flaky_examples
|
16
17
|
format_examples_dependency
|
17
18
|
format_files_dependency
|
18
19
|
end
|
19
20
|
|
20
21
|
def generate_report
|
21
|
-
|
22
|
-
|
23
|
-
|
22
|
+
starting = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
23
|
+
|
24
|
+
copy_assets
|
24
25
|
|
25
26
|
file_name = File.join(RSpecTracer.report_path, 'index.html')
|
26
27
|
|
@@ -28,11 +29,20 @@ module RSpecTracer
|
|
28
29
|
file.puts(template('layout').result(binding))
|
29
30
|
end
|
30
31
|
|
31
|
-
|
32
|
+
ending = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
33
|
+
elpased = RSpecTracer::TimeFormatter.format_time(ending - starting)
|
34
|
+
|
35
|
+
puts "RSpecTracer generated HTML report to #{file_name} (took #{elpased})"
|
32
36
|
end
|
33
37
|
|
34
38
|
private
|
35
39
|
|
40
|
+
def copy_assets
|
41
|
+
Dir[File.join(File.dirname(__FILE__), 'public/*')].each do |path|
|
42
|
+
FileUtils.cp_r(path, asset_output_path)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
36
46
|
def format_last_run
|
37
47
|
@last_run = @reporter.last_run.slice(
|
38
48
|
:actual_count,
|
@@ -57,6 +67,10 @@ module RSpecTracer
|
|
57
67
|
end
|
58
68
|
end
|
59
69
|
|
70
|
+
def format_flaky_examples
|
71
|
+
@flaky_examples = @examples.slice(*@reporter.flaky_examples).values
|
72
|
+
end
|
73
|
+
|
60
74
|
def example_run_local_time(utc_time)
|
61
75
|
case utc_time
|
62
76
|
when Time
|
@@ -128,6 +142,14 @@ module RSpecTracer
|
|
128
142
|
template(title_id).result(current_binding)
|
129
143
|
end
|
130
144
|
|
145
|
+
def formatted_flaky_examples(title, flaky_examples)
|
146
|
+
title_id = report_container_id(title)
|
147
|
+
current_binding = binding
|
148
|
+
|
149
|
+
current_binding.local_variable_set(:title_id, title_id)
|
150
|
+
template(title_id).result(current_binding)
|
151
|
+
end
|
152
|
+
|
131
153
|
def formatted_examples_dependency(title, examples_dependency)
|
132
154
|
title_id = report_container_id(title)
|
133
155
|
current_binding = binding
|
@@ -154,7 +176,7 @@ module RSpecTracer
|
|
154
176
|
|
155
177
|
def example_status_css_class(example_status)
|
156
178
|
case example_status.split.first
|
157
|
-
when 'Failed'
|
179
|
+
when 'Failed', 'Flaky'
|
158
180
|
'red'
|
159
181
|
when 'Pending'
|
160
182
|
'yellow'
|