knapsack 1.16.0 → 1.19.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.travis.yml +2 -1
- data/CHANGELOG.md +43 -0
- data/README.md +26 -805
- data/knapsack.gemspec +1 -1
- data/lib/knapsack/config/env.rb +12 -2
- data/lib/knapsack/presenter.rb +6 -2
- data/lib/knapsack/runners/rspec_runner.rb +1 -2
- data/lib/knapsack/version.rb +1 -1
- data/spec/knapsack/config/env_spec.rb +25 -0
- metadata +11 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 6a50a48925cacc27dbadfc763effe1061eb2dc6dec00217e23e304080e74faee
|
|
4
|
+
data.tar.gz: 60b6827c19012025fa3651a6e6017b3351eee25cd76cc5c0795625bd1d6dfe09
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1ec9fe137cf43655aeab8bf89c969b0969485de1d16ea4de287d3eeb08bfa9b486b3aff58b632241792464f4d175b30d433e4901c51f2cb2cb6b71e730d556b5
|
|
7
|
+
data.tar.gz: 94842148dfb9ae8756a127decaabb8bdf0be8d0e3a3b8d708939e46b2ee214c3fef847a82d16562368ff4a0cdcf6cdec7aa7751a4045a69eebaa47c199e5b6e4
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,49 @@
|
|
|
2
2
|
|
|
3
3
|
* TODO
|
|
4
4
|
|
|
5
|
+
### 1.19.0
|
|
6
|
+
|
|
7
|
+
* Add support for Bitbucket Pipelines
|
|
8
|
+
|
|
9
|
+
https://github.com/ArturT/knapsack/pull/97
|
|
10
|
+
|
|
11
|
+
https://github.com/ArturT/knapsack/compare/v1.18.0...v1.19.0
|
|
12
|
+
|
|
13
|
+
### 1.18.0
|
|
14
|
+
|
|
15
|
+
* Add support for Semaphore 2.0
|
|
16
|
+
|
|
17
|
+
https://github.com/ArturT/knapsack/pull/92
|
|
18
|
+
|
|
19
|
+
https://github.com/ArturT/knapsack/compare/v1.17.2...v1.18.0
|
|
20
|
+
|
|
21
|
+
### 1.17.2
|
|
22
|
+
|
|
23
|
+
* Allow for new `bundler` in development
|
|
24
|
+
* Test Ruby 2.6 on CI
|
|
25
|
+
* Add info about Knapsack Pro Queue Mode in knapsack output
|
|
26
|
+
* Update URL to FAQ in knapsack output
|
|
27
|
+
|
|
28
|
+
https://github.com/ArturT/knapsack/pull/90
|
|
29
|
+
|
|
30
|
+
https://github.com/ArturT/knapsack/compare/v1.17.1...v1.17.2
|
|
31
|
+
|
|
32
|
+
### 1.17.1
|
|
33
|
+
|
|
34
|
+
* Fix RSpec signal handling by replacing process
|
|
35
|
+
|
|
36
|
+
https://github.com/ArturT/knapsack/pull/86
|
|
37
|
+
|
|
38
|
+
https://github.com/ArturT/knapsack/compare/v1.17.0...v1.17.1
|
|
39
|
+
|
|
40
|
+
### 1.17.0
|
|
41
|
+
|
|
42
|
+
* Add support for GitLab CI ENV variable `CI_NODE_INDEX` starting from 1.
|
|
43
|
+
|
|
44
|
+
https://github.com/ArturT/knapsack/pull/83
|
|
45
|
+
|
|
46
|
+
https://github.com/ArturT/knapsack/compare/v1.16.0...v1.17.0
|
|
47
|
+
|
|
5
48
|
### 1.16.0
|
|
6
49
|
|
|
7
50
|
* Add support for Ruby >= 1.9.3.
|
data/README.md
CHANGED
|
@@ -1,815 +1,36 @@
|
|
|
1
|
-
|
|
1
|
+
# knapsack gem
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Knapsack splits tests evenly across parallel CI nodes to run fast CI build and save you time.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
[
|
|
8
|
-
|
|
5
|
+
| | knapsack gem | knapsack_pro gem |
|
|
6
|
+
| -------------------------------------------- | ------------ | ---------------- |
|
|
7
|
+
| __Is free__ | ✓ Yes | ✓ Yes, [free plan](https://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=free_plan) |
|
|
8
|
+
| __Regular Mode - a static tests split__ | ✓ Yes | ✓ Yes |
|
|
9
|
+
| __Queue Mode - a dynamic tests split__ <br>([ensures all CI nodes finish work at the same time](https://docs.knapsackpro.com/2020/how-to-speed-up-ruby-and-javascript-tests-with-ci-parallelisation)) | No | ✓ Yes |
|
|
10
|
+
| __Auto [split slow RSpec test file](https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it) between parallel CI nodes__ <br>(a single test file can be auto split by test examples between parallel jobs) | No | ✓ Yes |
|
|
11
|
+
| Tracking tests timing per commit, branch | No | ✓ Yes |
|
|
12
|
+
| Support for other programming languages | No | ✓ Yes |
|
|
13
|
+
| Support for CI providers | limited | ✓ Yes |
|
|
14
|
+
| __Installation README__ | [Install README](http://docs.knapsackpro.com/ruby/knapsack) | [Install README](https://docs.knapsackpro.com/integration/) |
|
|
9
15
|
|
|
10
|
-
[
|
|
11
|
-
[travis]: http://travis-ci.org/ArturT/knapsack
|
|
12
|
-
[codeclimate]: https://codeclimate.com/github/ArturT/knapsack
|
|
13
|
-
[coverage]: https://codeclimate.com/github/ArturT/knapsack
|
|
16
|
+
[Features of knapsack vs knapsack_pro Ruby gem](https://knapsackpro.com/features/ruby_knapsack_pro_vs_knapsack?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=ruby_knapsack_pro_vs_knapsack)
|
|
14
17
|
|
|
15
|
-
|
|
18
|
+
# Do you use Heroku?
|
|
16
19
|
|
|
17
|
-
|
|
20
|
+
Do you know Knapsack Pro Ruby gem is available as Heroku add-on that's currently in beta and it's free to all beta users? It works with your current CI server.
|
|
21
|
+
https://elements.heroku.com/addons/knapsack-pro
|
|
18
22
|
|
|
19
|
-
|
|
23
|
+
Knapsack Pro has Queue Mode that will split Ruby & JS tests in a dynamic way across parallel CI nodes to ensure each parallel job takes a similar time. Thanks to that there is no bottleneck in your CI pipelines.
|
|
20
24
|
|
|
21
|
-
|
|
25
|
+
__See introduction how the Knapsack Pro add-on works__
|
|
26
|
+
https://youtu.be/rmXES2N0_QU
|
|
22
27
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
* [Minitest](http://docs.seattlerb.org/minitest/)
|
|
26
|
-
* [Spinach](https://github.com/codegram/spinach)
|
|
27
|
-
* [Turnip](https://github.com/jnicklas/turnip)
|
|
28
|
+
You may also find useful article how to run parallel dynos on Heroku CI to complete tests faster
|
|
29
|
+
https://docs.knapsackpro.com/2019/how-to-run-tests-faster-on-heroku-ci-with-parallel-dynos
|
|
28
30
|
|
|
29
|
-
|
|
31
|
+
## Do you know
|
|
30
32
|
|
|
31
|
-
!
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
### With Knapsack Pro - optimal test suite split
|
|
38
|
-
|
|
39
|
-
[Watch 1 minute video how Queue Mode works](https://knapsackpro.com/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=video#x-play-how-it-works-video)
|
|
40
|
-
|
|
41
|
-
[](https://knapsackpro.com/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=video#x-play-how-it-works-video)
|
|
42
|
-
|
|
43
|
-
### Features in knapsack vs knapsack_pro
|
|
44
|
-
|
|
45
|
-
| Feature | knapsack gem | knapsack_pro gem
|
|
46
|
-
| --- | :---: | :---:
|
|
47
|
-
| Queue mode designed for optimal dynamic test suite split | ✘ | ✔
|
|
48
|
-
| Test suite split based on tests time execution | ✔ | ✔
|
|
49
|
-
| Automated tests time execution recording | ✘ | ✔
|
|
50
|
-
| Test suite split based on most up to date tests time execution data | ✘ | ✔
|
|
51
|
-
| [Show all features](https://knapsackpro.com/features?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=show_all_features) | |
|
|
52
|
-
|
|
53
|
-
How knapsack_pro makes my life easier as opposed to regular knapsack gem?
|
|
54
|
-
|
|
55
|
-
* The knapsack_pro version has queue mode designed for optimal test suite split thanks to dynamic tests allocation.
|
|
56
|
-
* With the knapsack_pro version the setup and the ongoing work is easier because you don’t have to manually generate knapsack json report with test files time execution for each test suite like rspec, cucumber etc.
|
|
57
|
-
* The knapsack_pro version tracks all you branches so when your test code changes the knapsack_pro tries to provide you as optimal test suite split as possible.
|
|
58
|
-
* When your test codebase changes over time you need to manually generate a new knapsack report in free gem version which is extra overhead and waste of time of the developer.
|
|
59
|
-
|
|
60
|
-
**Only [knapsack_pro gem](https://github.com/KnapsackPro/knapsack_pro-ruby) has Queue Mode feature that saves optimal amount of time? Please [visit Knapsack Pro](http://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=try_knapsack_pro).**
|
|
61
|
-
|
|
62
|
-
### Presentations about knapsack gem
|
|
63
|
-
|
|
64
|
-
* [X 2014 Kraków Ruby User Group](http://slides.com/arturt/parallel-tests-in-comparable-time)
|
|
65
|
-
* [VII 2014 Lunar Logic Dev Meeting](http://slides.com/arturt/knapsack)
|
|
66
|
-
|
|
67
|
-
### Requirements
|
|
68
|
-
|
|
69
|
-
`>= Ruby 1.9.3`
|
|
70
|
-
|
|
71
|
-
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
|
72
|
-
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
|
73
|
-
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
|
|
74
|
-
|
|
75
|
-
- [Update gem](#update-gem)
|
|
76
|
-
- [Installation](#installation)
|
|
77
|
-
- [Usage](#usage)
|
|
78
|
-
- [Step for RSpec](#step-for-rspec)
|
|
79
|
-
- [Step for Cucumber](#step-for-cucumber)
|
|
80
|
-
- [Step for Minitest](#step-for-minitest)
|
|
81
|
-
- [Step for Spinach](#step-for-spinach)
|
|
82
|
-
- [Custom configuration](#custom-configuration)
|
|
83
|
-
- [Common step](#common-step)
|
|
84
|
-
- [Adding or removing tests](#adding-or-removing-tests)
|
|
85
|
-
- [Setup your CI server](#setup-your-ci-server)
|
|
86
|
-
- [Info about ENV variables](#info-about-env-variables)
|
|
87
|
-
- [Passing arguments to rake task](#passing-arguments-to-rake-task)
|
|
88
|
-
- [Passing arguments to rspec](#passing-arguments-to-rspec)
|
|
89
|
-
- [Passing arguments to cucumber](#passing-arguments-to-cucumber)
|
|
90
|
-
- [Passing arguments to minitest](#passing-arguments-to-minitest)
|
|
91
|
-
- [Passing arguments to spinach](#passing-arguments-to-spinach)
|
|
92
|
-
- [Knapsack binary](#knapsack-binary)
|
|
93
|
-
- [Info for CircleCI users](#info-for-circleci-users)
|
|
94
|
-
- [Step 1](#step-1)
|
|
95
|
-
- [Step 2](#step-2)
|
|
96
|
-
- [Info for Travis users](#info-for-travis-users)
|
|
97
|
-
- [Step 1](#step-1-1)
|
|
98
|
-
- [Step 2](#step-2-1)
|
|
99
|
-
- [Info for semaphoreapp.com users](#info-for-semaphoreappcom-users)
|
|
100
|
-
- [Step 1](#step-1-2)
|
|
101
|
-
- [Step 2](#step-2-2)
|
|
102
|
-
- [Info for buildkite.com users](#info-for-buildkitecom-users)
|
|
103
|
-
- [Step 1](#step-1-3)
|
|
104
|
-
- [Step 2](#step-2-3)
|
|
105
|
-
- [Info for snap-ci.com users](#info-for-snap-cicom-users)
|
|
106
|
-
- [Step 1](#step-1-4)
|
|
107
|
-
- [Step 2](#step-2-4)
|
|
108
|
-
- [Info for Jenkins](#info-for-jenkins)
|
|
109
|
-
- [FAQ](#faq)
|
|
110
|
-
- [What time offset warning means?](#what-time-offset-warning-means)
|
|
111
|
-
- [How to generate knapsack report?](#how-to-generate-knapsack-report)
|
|
112
|
-
- [What does "leftover specs" mean?](#what-does-leftover-specs-mean)
|
|
113
|
-
- [Why some of test files are still in "leftover specs" after I generate a new report?](#why-some-of-test-files-are-still-in-leftover-specs-after-i-generate-a-new-report)
|
|
114
|
-
- [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories)
|
|
115
|
-
- [How to update existing knapsack report for a few test files?](#how-to-update-existing-knapsack-report-for-a-few-test-files)
|
|
116
|
-
- [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment)
|
|
117
|
-
- [How can I change log level?](#how-can-i-change-log-level)
|
|
118
|
-
- [Gem tests](#gem-tests)
|
|
119
|
-
- [Spec](#spec)
|
|
120
|
-
- [Spec examples](#spec-examples)
|
|
121
|
-
- [Contributing](#contributing)
|
|
122
|
-
- [Acknowledgements](#acknowledgements)
|
|
123
|
-
- [Mentions](#mentions)
|
|
124
|
-
|
|
125
|
-
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
## Update gem
|
|
129
|
-
|
|
130
|
-
Please check [changelog](CHANGELOG.md) before update gem. Knapsack follows [semantic versioning](http://semver.org).
|
|
131
|
-
|
|
132
|
-
## Installation
|
|
133
|
-
|
|
134
|
-
Add those lines to your application's Gemfile:
|
|
135
|
-
|
|
136
|
-
```ruby
|
|
137
|
-
group :test, :development do
|
|
138
|
-
gem 'knapsack'
|
|
139
|
-
end
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
And then execute:
|
|
143
|
-
|
|
144
|
-
$ bundle
|
|
145
|
-
|
|
146
|
-
Add this line at the bottom of `Rakefile`:
|
|
147
|
-
|
|
148
|
-
```ruby
|
|
149
|
-
Knapsack.load_tasks if defined?(Knapsack)
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
## Usage
|
|
153
|
-
|
|
154
|
-
You can find here example of rails app with already configured knapsack.
|
|
155
|
-
|
|
156
|
-
https://github.com/KnapsackPro/rails-app-with-knapsack
|
|
157
|
-
|
|
158
|
-
### Step for RSpec
|
|
159
|
-
|
|
160
|
-
Add at the beginning of your `spec_helper.rb`:
|
|
161
|
-
|
|
162
|
-
```ruby
|
|
163
|
-
require 'knapsack'
|
|
164
|
-
|
|
165
|
-
# CUSTOM_CONFIG_GOES_HERE
|
|
166
|
-
|
|
167
|
-
Knapsack::Adapters::RSpecAdapter.bind
|
|
168
|
-
```
|
|
169
|
-
|
|
170
|
-
### Step for Cucumber
|
|
171
|
-
|
|
172
|
-
Create file `features/support/knapsack.rb` and add there:
|
|
173
|
-
|
|
174
|
-
```ruby
|
|
175
|
-
require 'knapsack'
|
|
176
|
-
|
|
177
|
-
# CUSTOM_CONFIG_GOES_HERE
|
|
178
|
-
|
|
179
|
-
Knapsack::Adapters::CucumberAdapter.bind
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
### Step for Minitest
|
|
183
|
-
|
|
184
|
-
Add at the beginning of your `test_helper.rb`:
|
|
185
|
-
|
|
186
|
-
```ruby
|
|
187
|
-
require 'knapsack'
|
|
188
|
-
|
|
189
|
-
# CUSTOM_CONFIG_GOES_HERE
|
|
190
|
-
|
|
191
|
-
knapsack_adapter = Knapsack::Adapters::MinitestAdapter.bind
|
|
192
|
-
knapsack_adapter.set_test_helper_path(__FILE__)
|
|
193
|
-
```
|
|
194
|
-
|
|
195
|
-
### Step for Spinach
|
|
196
|
-
|
|
197
|
-
Create file `features/support/env.rb` and add there:
|
|
198
|
-
|
|
199
|
-
```ruby
|
|
200
|
-
require 'knapsack'
|
|
201
|
-
|
|
202
|
-
# CUSTOM_CONFIG_GOES_HERE
|
|
203
|
-
|
|
204
|
-
Knapsack::Adapters::SpinachAdapter.bind
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
### Custom configuration
|
|
208
|
-
|
|
209
|
-
You can change default Knapsack configuration for RSpec, Cucumber, Minitest or Spinach tests. Here are examples what you can do. Put below configuration instead of `CUSTOM_CONFIG_GOES_HERE`.
|
|
210
|
-
|
|
211
|
-
```ruby
|
|
212
|
-
Knapsack.tracker.config({
|
|
213
|
-
enable_time_offset_warning: true,
|
|
214
|
-
time_offset_in_seconds: 30
|
|
215
|
-
})
|
|
216
|
-
|
|
217
|
-
Knapsack.report.config({
|
|
218
|
-
test_file_pattern: 'spec/**{,/*/**}/*_spec.rb', # default value based on adapter
|
|
219
|
-
report_path: 'knapsack_custom_report.json'
|
|
220
|
-
})
|
|
221
|
-
|
|
222
|
-
# you can use your own logger
|
|
223
|
-
require 'logger'
|
|
224
|
-
Knapsack.logger = Logger.new(STDOUT)
|
|
225
|
-
Knapsack.logger.level = Logger::INFO
|
|
226
|
-
```
|
|
227
|
-
|
|
228
|
-
### Common step
|
|
229
|
-
|
|
230
|
-
Generate time execution report for your test files. Run below command on one of your CI nodes.
|
|
231
|
-
|
|
232
|
-
# Step for RSpec
|
|
233
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec
|
|
234
|
-
|
|
235
|
-
# Step for Cucumber
|
|
236
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features
|
|
237
|
-
|
|
238
|
-
# Step for Minitest
|
|
239
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test
|
|
240
|
-
|
|
241
|
-
# If you use rails 5 then run this instead:
|
|
242
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake knapsack:minitest
|
|
243
|
-
|
|
244
|
-
# If you use rails 5.1's SystemTest, run both regular and system tests
|
|
245
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system
|
|
246
|
-
|
|
247
|
-
# Step for Spinach
|
|
248
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec spinach
|
|
249
|
-
|
|
250
|
-
Commit generated report `knapsack_rspec_report.json`, `knapsack_cucumber_report.json`, `knapsack_minitest_report.json` or `knapsack_spinach_report.json` into your repository.
|
|
251
|
-
|
|
252
|
-
This report should be updated only after you add a lot of new slow tests or you change existing ones which causes a big time execution difference between CI nodes. Either way, you will get time offset warning at the end of the rspec/cucumber/minitest results which reminds you when it’s a good time to regenerate the knapsack report.
|
|
253
|
-
|
|
254
|
-
`KNAPSACK_GENERATE_REPORT` is truthy when `"true"` or `0`. All other values are falsy, though
|
|
255
|
-
[`"false"`, and `1` are semantically
|
|
256
|
-
preferrable](https://en.wikipedia.org/wiki/True_and_false_(commands)).
|
|
257
|
-
|
|
258
|
-
#### Adding or removing tests
|
|
259
|
-
|
|
260
|
-
There is no need to regenerate the report every time when you add/remove test file. If you remove a test file then Knapsack will ignore its entry in report. In case when you add a new file and it doesn't already exist in report, the test file will be assigned to one of the CI node.
|
|
261
|
-
|
|
262
|
-
You'll want to regenerate your execution report whenever you remove or add a test file with a long time execution time that would affect one of the CI nodes. You will get Knapsack notification whenever is good time to regenerate report.
|
|
263
|
-
|
|
264
|
-
## Setup your CI server
|
|
265
|
-
|
|
266
|
-
On your CI server run this command for the first CI node. Update `CI_NODE_INDEX` for the next one.
|
|
267
|
-
|
|
268
|
-
# Step for RSpec
|
|
269
|
-
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec
|
|
270
|
-
|
|
271
|
-
# Step for Cucumber
|
|
272
|
-
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:cucumber
|
|
273
|
-
|
|
274
|
-
# Step for Minitest
|
|
275
|
-
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:minitest
|
|
276
|
-
|
|
277
|
-
# Step for Spinach
|
|
278
|
-
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:spinach
|
|
279
|
-
|
|
280
|
-
You can add `KNAPSACK_TEST_FILE_PATTERN` if your tests are not in default directory. For instance:
|
|
281
|
-
|
|
282
|
-
# Step for RSpec
|
|
283
|
-
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_specs/**{,/*/**}/*_spec.rb" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec
|
|
284
|
-
|
|
285
|
-
# Step for Cucumber
|
|
286
|
-
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_features/**/*.feature" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:cucumber
|
|
287
|
-
|
|
288
|
-
# Step for Minitest
|
|
289
|
-
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_spec.rb" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:minitest
|
|
290
|
-
|
|
291
|
-
# Step for Spinach
|
|
292
|
-
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_features/**/*.feature" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:spinach
|
|
293
|
-
|
|
294
|
-
You can set `KNAPSACK_REPORT_PATH` if your knapsack report was saved in non default location. Example:
|
|
295
|
-
|
|
296
|
-
# Step for RSpec
|
|
297
|
-
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec
|
|
298
|
-
|
|
299
|
-
# Step for Cucumber
|
|
300
|
-
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:cucumber
|
|
301
|
-
|
|
302
|
-
# Step for Minitest
|
|
303
|
-
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:minitest
|
|
304
|
-
|
|
305
|
-
# Step for Spinach
|
|
306
|
-
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:spinach
|
|
307
|
-
|
|
308
|
-
### Info about ENV variables
|
|
309
|
-
|
|
310
|
-
`CI_NODE_TOTAL` - total number CI nodes you have.
|
|
311
|
-
|
|
312
|
-
`CI_NODE_INDEX` - index of current CI node starts from 0. Second CI node should have `CI_NODE_INDEX=1`.
|
|
313
|
-
|
|
314
|
-
### Passing arguments to rake task
|
|
315
|
-
|
|
316
|
-
#### Passing arguments to rspec
|
|
317
|
-
|
|
318
|
-
Knapsack allows you to pass arguments through to rspec. For example if you want to run only specs that have the tag `focus`. If you do this with rspec directly it would look like:
|
|
319
|
-
|
|
320
|
-
$ bundle exec rake rspec --tag focus
|
|
321
|
-
|
|
322
|
-
To do this with Knapsack you simply add your rspec arguments as parameters to the knapsack rake task.
|
|
323
|
-
|
|
324
|
-
$ bundle exec rake "knapsack:rspec[--tag focus]"
|
|
325
|
-
|
|
326
|
-
Remember that using tags to limit which specs get run will affect the time each file takes to run. One solution to this is to generate a new `knapsack_rspec_report.json` for the commonly run scenarios.
|
|
327
|
-
|
|
328
|
-
#### Passing arguments to cucumber
|
|
329
|
-
|
|
330
|
-
Add arguments to knapsack cucumber task like this:
|
|
331
|
-
|
|
332
|
-
$ bundle exec rake "knapsack:cucumber[--name feature]"
|
|
333
|
-
|
|
334
|
-
#### Passing arguments to minitest
|
|
335
|
-
|
|
336
|
-
Add arguments to knapsack minitest task like this:
|
|
337
|
-
|
|
338
|
-
$ bundle exec rake "knapsack:minitest[--arg_name value]"
|
|
339
|
-
|
|
340
|
-
For instance to run verbose tests:
|
|
341
|
-
|
|
342
|
-
$ bundle exec rake "knapsack:minitest[--verbose]"
|
|
343
|
-
|
|
344
|
-
#### Passing arguments to spinach
|
|
345
|
-
|
|
346
|
-
Add arguments to knapsack spinach task like this:
|
|
347
|
-
|
|
348
|
-
$ bundle exec rake "knapsack:spinach[--name feature]"
|
|
349
|
-
|
|
350
|
-
### Knapsack binary
|
|
351
|
-
|
|
352
|
-
You can install knapsack globally and use binary. For instance:
|
|
353
|
-
|
|
354
|
-
$ knapsack rspec "--tag custom_tag_name --profile"
|
|
355
|
-
$ knapsack cucumber
|
|
356
|
-
$ knapsack minitest "--verbose --pride"
|
|
357
|
-
$ knapsack spinach "-f spinach_examples"
|
|
358
|
-
|
|
359
|
-
[Here](https://github.com/ArturT/knapsack/pull/21) you will find example when it might be useful.
|
|
360
|
-
|
|
361
|
-
### Info for CircleCI users
|
|
362
|
-
|
|
363
|
-
If you are using circleci.com you can omit `CI_NODE_TOTAL` and `CI_NODE_INDEX`. Knapsack will use `CIRCLE_NODE_TOTAL` and `CIRCLE_NODE_INDEX` provided by CircleCI.
|
|
364
|
-
|
|
365
|
-
Here is an example for test configuration in your `circleci.yml` file.
|
|
366
|
-
|
|
367
|
-
#### Step 1
|
|
368
|
-
|
|
369
|
-
For the first time run all tests on a single CI node with enabled report generator.
|
|
370
|
-
|
|
371
|
-
```yaml
|
|
372
|
-
# CircleCI 1.0
|
|
373
|
-
test:
|
|
374
|
-
override:
|
|
375
|
-
# Step for RSpec
|
|
376
|
-
- KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec
|
|
377
|
-
|
|
378
|
-
# Step for Cucumber
|
|
379
|
-
- KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features
|
|
380
|
-
|
|
381
|
-
# Step for Minitest
|
|
382
|
-
- KNAPSACK_GENERATE_REPORT=true bundle exec rake test
|
|
383
|
-
- KNAPSACK_GENERATE_REPORT=true bundle exec rake test # For rails 5.1 with system tests
|
|
384
|
-
|
|
385
|
-
# Step for Spinach
|
|
386
|
-
- KNAPSACK_GENERATE_REPORT=true bundle exec spinach
|
|
387
|
-
```
|
|
388
|
-
|
|
389
|
-
```YAML
|
|
390
|
-
# CircleCI 2.0
|
|
391
|
-
- run:
|
|
392
|
-
name: Step for RSpec
|
|
393
|
-
command: |
|
|
394
|
-
# export word is important here!
|
|
395
|
-
export KNAPSACK_GENERATE_REPORT=true
|
|
396
|
-
bundle exec rspec spec
|
|
397
|
-
|
|
398
|
-
- run:
|
|
399
|
-
name: Step for Cucumber
|
|
400
|
-
command: |
|
|
401
|
-
# export word is important here!
|
|
402
|
-
export KNAPSACK_GENERATE_REPORT=true
|
|
403
|
-
bundle exec cucumber features
|
|
404
|
-
|
|
405
|
-
- run:
|
|
406
|
-
name: Step for Minitest
|
|
407
|
-
command: |
|
|
408
|
-
# export word is important here!
|
|
409
|
-
export KNAPSACK_GENERATE_REPORT=true
|
|
410
|
-
bundle exec rake test
|
|
411
|
-
|
|
412
|
-
- run:
|
|
413
|
-
name: Step for Spinach
|
|
414
|
-
command: |
|
|
415
|
-
# export word is important here!
|
|
416
|
-
export KNAPSACK_GENERATE_REPORT=true
|
|
417
|
-
bundle exec rspec spinach
|
|
418
|
-
```
|
|
419
|
-
|
|
420
|
-
After tests pass on your CircleCI machine your should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as `knapsack_rspec_report.json`, `knapsack_cucumber_report.json`, `knapsack_minitest_report.json` or `knapsack_spinach_report.json` file and commit.
|
|
421
|
-
|
|
422
|
-
#### Step 2
|
|
423
|
-
|
|
424
|
-
Now you should update test command and enable parallel. Please remember to add additional containers for your project in CircleCI settings.
|
|
425
|
-
|
|
426
|
-
```yaml
|
|
427
|
-
# CircleCI 1.0
|
|
428
|
-
test:
|
|
429
|
-
override:
|
|
430
|
-
# Step for RSpec
|
|
431
|
-
- bundle exec rake knapsack:rspec:
|
|
432
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
433
|
-
|
|
434
|
-
# Step for Cucumber
|
|
435
|
-
- bundle exec rake knapsack:cucumber:
|
|
436
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
437
|
-
|
|
438
|
-
# Step for Minitest
|
|
439
|
-
- bundle exec rake knapsack:minitest:
|
|
440
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
441
|
-
|
|
442
|
-
# Step for Spinach
|
|
443
|
-
- bundle exec rake knapsack:spinach:
|
|
444
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
445
|
-
```
|
|
446
|
-
|
|
447
|
-
```YAML
|
|
448
|
-
# CircleCI 2.0
|
|
449
|
-
- run:
|
|
450
|
-
name: Step for RSpec
|
|
451
|
-
command: bundle exec rake knapsack:rspec
|
|
452
|
-
|
|
453
|
-
- run:
|
|
454
|
-
name: Step for Cucumber
|
|
455
|
-
command: bundle exec rake knapsack:cucumber
|
|
456
|
-
|
|
457
|
-
- run:
|
|
458
|
-
name: Step for Minitest
|
|
459
|
-
command: bundle exec rake knapsack:minitest
|
|
460
|
-
|
|
461
|
-
- run:
|
|
462
|
-
name: Step for Spinach
|
|
463
|
-
command: bundle exec rake knapsack:spinach
|
|
464
|
-
```
|
|
465
|
-
|
|
466
|
-
Now everything should works. You will get warning at the end of rspec/cucumber/minitest results if time execution will take too much.
|
|
467
|
-
|
|
468
|
-
### Info for Travis users
|
|
469
|
-
|
|
470
|
-
#### Step 1
|
|
471
|
-
|
|
472
|
-
For the first time run all tests at once with enabled report generator. Edit `.travis.yml`
|
|
473
|
-
|
|
474
|
-
```yaml
|
|
475
|
-
script:
|
|
476
|
-
# Step for RSpec
|
|
477
|
-
- "KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec"
|
|
478
|
-
|
|
479
|
-
# Step for Cucumber
|
|
480
|
-
- "KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features"
|
|
481
|
-
|
|
482
|
-
# Step for Minitest
|
|
483
|
-
- "KNAPSACK_GENERATE_REPORT=true bundle exec rake test"
|
|
484
|
-
- "KNAPSACK_GENERATE_REPORT=true bundle exec rake test:system" # For rails 5.1 with system tests
|
|
485
|
-
|
|
486
|
-
# Step for Spinach
|
|
487
|
-
- "KNAPSACK_GENERATE_REPORT=true bundle exec spinach"
|
|
488
|
-
```
|
|
489
|
-
|
|
490
|
-
After tests pass your should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as `knapsack_rspec_report.json`, `knapsack_cucumber_report.json`, `knapsack_minitest_report.json` or `knapsack_spinach_report.json` file and commit.
|
|
491
|
-
|
|
492
|
-
#### Step 2
|
|
493
|
-
|
|
494
|
-
You can parallel your builds across virtual machines with [travis matrix feature](http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines). Edit `.travis.yml`
|
|
495
|
-
|
|
496
|
-
```yaml
|
|
497
|
-
script:
|
|
498
|
-
# Step for RSpec
|
|
499
|
-
- "bundle exec rake knapsack:rspec"
|
|
500
|
-
|
|
501
|
-
# Step for Cucumber
|
|
502
|
-
- "bundle exec rake knapsack:cucumber"
|
|
503
|
-
|
|
504
|
-
# Step for Minitest
|
|
505
|
-
- "bundle exec rake knapsack:minitest"
|
|
506
|
-
|
|
507
|
-
# Step for Spinach
|
|
508
|
-
- "bundle exec rake knapsack:spinach"
|
|
509
|
-
|
|
510
|
-
env:
|
|
511
|
-
- CI_NODE_TOTAL=2 CI_NODE_INDEX=0
|
|
512
|
-
- CI_NODE_TOTAL=2 CI_NODE_INDEX=1
|
|
513
|
-
```
|
|
514
|
-
|
|
515
|
-
If you want to have some global ENVs and matrix of ENVs then do it like this:
|
|
516
|
-
|
|
517
|
-
```yaml
|
|
518
|
-
script:
|
|
519
|
-
# Step for RSpec
|
|
520
|
-
- "bundle exec rake knapsack:rspec"
|
|
521
|
-
|
|
522
|
-
# Step for Cucumber
|
|
523
|
-
- "bundle exec rake knapsack:cucumber"
|
|
524
|
-
|
|
525
|
-
# Step for Minitest
|
|
526
|
-
- "bundle exec rake knapsack:minitest"
|
|
527
|
-
|
|
528
|
-
# Step for Spinach
|
|
529
|
-
- "bundle exec rake knapsack:spinach"
|
|
530
|
-
|
|
531
|
-
env:
|
|
532
|
-
global:
|
|
533
|
-
- RAILS_ENV=test
|
|
534
|
-
- MY_GLOBAL_VAR=123
|
|
535
|
-
- CI_NODE_TOTAL=2
|
|
536
|
-
matrix:
|
|
537
|
-
- CI_NODE_INDEX=0
|
|
538
|
-
- CI_NODE_INDEX=1
|
|
539
|
-
```
|
|
540
|
-
|
|
541
|
-
Such configuration will generate matrix with 2 following ENV rows:
|
|
542
|
-
|
|
543
|
-
CI_NODE_TOTAL=2 CI_NODE_INDEX=0 RAILS_ENV=test MY_GLOBAL_VAR=123
|
|
544
|
-
CI_NODE_TOTAL=2 CI_NODE_INDEX=1 RAILS_ENV=test MY_GLOBAL_VAR=123
|
|
545
|
-
|
|
546
|
-
More info about global and matrix ENV configuration in [travis docs](http://docs.travis-ci.com/user/build-configuration/#Environment-variables).
|
|
547
|
-
|
|
548
|
-
### Info for semaphoreapp.com users
|
|
549
|
-
|
|
550
|
-
#### Step 1
|
|
551
|
-
|
|
552
|
-
For the first time run all tests at once with enabled report generator. Set up your build command:
|
|
553
|
-
|
|
554
|
-
# Step for RSpec
|
|
555
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec
|
|
556
|
-
|
|
557
|
-
# Step for Cucumber
|
|
558
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features
|
|
559
|
-
|
|
560
|
-
# Step for Minitest
|
|
561
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
|
|
562
|
-
|
|
563
|
-
# Step for Spinach
|
|
564
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec spinach
|
|
565
|
-
|
|
566
|
-
After tests pass your should copy knapsack json report which is rendered at the end of rspec/cucumber/test results. Save it into your repository as `knapsack_rspec_report.json`, `knapsack_cucumber_report.json`, `knapsack_minitest_report.json` or `knapsack_spinach_report.json` file and commit.
|
|
567
|
-
|
|
568
|
-
#### Step 2
|
|
569
|
-
|
|
570
|
-
Knapsack supports semaphoreapp ENVs `SEMAPHORE_THREAD_COUNT` and `SEMAPHORE_CURRENT_THREAD`. The only thing you need to do is set up knapsack rspec/cucumber/minitest command for as many threads as you need. Here is an example:
|
|
571
|
-
|
|
572
|
-
# Thread 1
|
|
573
|
-
## Step for RSpec
|
|
574
|
-
bundle exec rake knapsack:rspec
|
|
575
|
-
## Step for Cucumber
|
|
576
|
-
bundle exec rake knapsack:cucumber
|
|
577
|
-
## Step for Minitest
|
|
578
|
-
bundle exec rake knapsack:minitest
|
|
579
|
-
## Step for Spinach
|
|
580
|
-
bundle exec rake knapsack:spinach
|
|
581
|
-
|
|
582
|
-
# Thread 2
|
|
583
|
-
## Step for RSpec
|
|
584
|
-
bundle exec rake knapsack:rspec
|
|
585
|
-
## Step for Cucumber
|
|
586
|
-
bundle exec rake knapsack:cucumber
|
|
587
|
-
## Step for Minitest
|
|
588
|
-
bundle exec rake knapsack:minitest
|
|
589
|
-
## Step for Spinach
|
|
590
|
-
bundle exec rake knapsack:spinach
|
|
591
|
-
|
|
592
|
-
Tests will be split across threads.
|
|
593
|
-
|
|
594
|
-
### Info for buildkite.com users
|
|
595
|
-
|
|
596
|
-
#### Step 1
|
|
597
|
-
|
|
598
|
-
For the first time run all tests at once with enabled report generator. Run the following commands locally:
|
|
599
|
-
|
|
600
|
-
# Step for RSpec
|
|
601
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec
|
|
602
|
-
|
|
603
|
-
# Step for Cucumber
|
|
604
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features
|
|
605
|
-
|
|
606
|
-
# Step for Minitest
|
|
607
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
|
|
608
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # rails 5.1 with system tests
|
|
609
|
-
|
|
610
|
-
# Step for Spinach
|
|
611
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec spinach
|
|
612
|
-
|
|
613
|
-
After tests pass your should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as `knapsack_rspec_report.json`, `knapsack_cucumber_report.json`, `knapsack_minitest_report.json` or `knapsack_spinach_report.json` file and commit.
|
|
614
|
-
|
|
615
|
-
#### Step 2
|
|
616
|
-
|
|
617
|
-
Knapsack supports buildkite ENVs `BUILDKITE_PARALLEL_JOB_COUNT` and `BUILDKITE_PARALLEL_JOB`. The only thing you need to do is to configure the parallelism parameter in your build step and run the appropiate command in your build
|
|
618
|
-
|
|
619
|
-
# Step for RSpec
|
|
620
|
-
bundle exec rake knapsack:rspec
|
|
621
|
-
|
|
622
|
-
# Step for Cucumber
|
|
623
|
-
bundle exec rake knapsack:cucumber
|
|
624
|
-
|
|
625
|
-
# Step for Minitest
|
|
626
|
-
bundle exec rake knapsack:minitest
|
|
627
|
-
|
|
628
|
-
# Step for Spinach
|
|
629
|
-
bundle exec rake knapsack:spinach
|
|
630
|
-
|
|
631
|
-
### Info for snap-ci.com users
|
|
632
|
-
|
|
633
|
-
#### Step 1
|
|
634
|
-
|
|
635
|
-
For the first time run all tests at once with enabled report generator. Run the following commands locally:
|
|
636
|
-
|
|
637
|
-
# Step for RSpec
|
|
638
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec
|
|
639
|
-
|
|
640
|
-
# Step for Cucumber
|
|
641
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features
|
|
642
|
-
|
|
643
|
-
# Step for Minitest
|
|
644
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
|
|
645
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # rails 5.1 with system tests
|
|
646
|
-
|
|
647
|
-
# Step for Spinach
|
|
648
|
-
KNAPSACK_GENERATE_REPORT=true bundle exec spinach
|
|
649
|
-
|
|
650
|
-
After tests pass your should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as `knapsack_rspec_report.json`, `knapsack_cucumber_report.json`, `knapsack_minitest_report.json` or `knapsack_spinach_report.json` file and commit.
|
|
651
|
-
|
|
652
|
-
#### Step 2
|
|
653
|
-
|
|
654
|
-
Knapsack supports snap-ci.com ENVs `SNAP_WORKER_TOTAL` and `SNAP_WORKER_INDEX`. The only thing you need to do is to configure number of workers for your project in configuration settings in order to enable parallelism. Next thing is to set below commands to be executed in your stage:
|
|
655
|
-
|
|
656
|
-
# Step for RSpec
|
|
657
|
-
bundle exec rake knapsack:rspec
|
|
658
|
-
|
|
659
|
-
# Step for Cucumber
|
|
660
|
-
bundle exec rake knapsack:cucumber
|
|
661
|
-
|
|
662
|
-
# Step for Minitest
|
|
663
|
-
bundle exec rake knapsack:minitest
|
|
664
|
-
|
|
665
|
-
# Step for Spinach
|
|
666
|
-
bundle exec rake knapsack:spinach
|
|
667
|
-
|
|
668
|
-
### Info for Jenkins
|
|
669
|
-
|
|
670
|
-
In order to run parallel jobs with Jenkins you should use Jenkins Pipeline.
|
|
671
|
-
You can learn basics about it in the article [Parallelism and Distributed Builds with Jenkins](https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins).
|
|
672
|
-
|
|
673
|
-
Here is an example [`Jenkinsfile`](https://github.com/mknapik/jenkins-pipeline-knapsack/blob/master/Jenkinsfile) working with Jenkins Pipeline and knapsack gem.
|
|
674
|
-
|
|
675
|
-
You may want to read article [Knapsack with Jenkins Pipeline](http://blog.knapik.me/knapsack-with-jenkins-pipeline/) from Michał Knapik.
|
|
676
|
-
|
|
677
|
-
More tips can be found in the [issue](https://github.com/ArturT/knapsack/issues/42).
|
|
678
|
-
|
|
679
|
-
## FAQ
|
|
680
|
-
|
|
681
|
-
### What time offset warning means?
|
|
682
|
-
|
|
683
|
-
At the end of tests execution results you can see warning like this:
|
|
684
|
-
|
|
685
|
-
```
|
|
686
|
-
========= Knapsack Time Offset Warning ==========
|
|
687
|
-
Time offset: 30s
|
|
688
|
-
Max allowed node time execution: 02m 30s
|
|
689
|
-
Exceeded time: 37s
|
|
690
|
-
```
|
|
691
|
-
|
|
692
|
-
`Time offset: 30s` - this is the current time offset value, by default it’s 30s. Let’s assume whole test suite takes 4 minutes and you do split across 2 CI nodes so the optimal split is 2 minutes per node. Time offset 30s means when tests on single CI node will take longer than 2 minutes and 30s then you see warning about regenerating report because probably test suite files changed and the knapsack report contains old time execution data about each test file so regenerating knapsack report should help you provide a more optimal test suite split.
|
|
693
|
-
|
|
694
|
-
`Max allowed node time execution: 02m 30s` - it’s average time execution of tests per CI node + time offset. In this case average tests time execution per CI node is 2 minutes.
|
|
695
|
-
|
|
696
|
-
`Exceeded time: 37s` - it means tests on particular CI node took 37s longer than `max allowed node time execution`. Sometimes this value is negative when tests executed faster than `max allowed node time execution`.
|
|
697
|
-
|
|
698
|
-
### How to generate knapsack report?
|
|
699
|
-
|
|
700
|
-
If you want to regenerate report take a look [here](#common-step).
|
|
701
|
-
|
|
702
|
-
`KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec`
|
|
703
|
-
|
|
704
|
-
If you run command like this on your development machine then test suite time execution might be different than if you generate a report on CI machine (for instance tests might be faster on your machine then on CI node) so that might be a reason why you see warning about regenerating report. You can generate the report on single CI node which should give you result specific for your CI node instead of your development machine. In case you don't want to bother about manually regenerating knapsack report please take a look on [knapsack_pro gem](http://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_gem&utm_content=no_manual_report).
|
|
705
|
-
|
|
706
|
-
### What does "leftover specs" mean?
|
|
707
|
-
|
|
708
|
-
When you run your specs with knapsack rake task then you will see in the output something like:
|
|
709
|
-
|
|
710
|
-
```
|
|
711
|
-
Report specs:
|
|
712
|
-
spec/models/user_spec.rb
|
|
713
|
-
spec/controllers/users_controller_spec.rb
|
|
714
|
-
|
|
715
|
-
Leftover specs:
|
|
716
|
-
spec/models/book_spec.rb
|
|
717
|
-
spec/models/author_spec.rb
|
|
718
|
-
```
|
|
719
|
-
|
|
720
|
-
The leftover specs mean we don't have recorded time execution for those test files so the leftover specs were distributed across CI nodes based on file name instead.
|
|
721
|
-
The reason might be that someone added a new test file after knapsack report was generated. Another reason might be an empty spec file.
|
|
722
|
-
If you have a lot of leftover specs then you can [generate knapsack report again](#how-to-generate-knapsack-report) to improve you test distribution across CI nodes.
|
|
723
|
-
|
|
724
|
-
### Why some of test files are still in "leftover specs" after I generate a new report?
|
|
725
|
-
|
|
726
|
-
If test file is empty or has only pending tests then it cannot be recorded so it will end up in leftovers specs list.
|
|
727
|
-
|
|
728
|
-
### How can I run tests from multiple directories?
|
|
729
|
-
|
|
730
|
-
The test file pattern config option supports any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.2.0/Dir.html#method-c-glob) and can be configured to pull test files from multiple directories. An example of this when using RSpec would be `"{spec,engines/**/spec}/**{,/*/**}/*_spec.rb"`. For complex cases like this, the test directory can't be extracted and must be specified manually using the `KNAPSACK_TEST_DIR` environment variable:
|
|
731
|
-
|
|
732
|
-
$ KNAPSACK_TEST_DIR=spec KNAPSACK_TEST_FILE_PATTERN="{spec,engines/**/spec}/**{,/*/**}/*_spec.rb" bundle exec rake knapsack:rspec
|
|
733
|
-
|
|
734
|
-
`KNAPSACK_TEST_DIR` will be your default path for rspec so you should put there your `spec_helper.rb`. Please ensure you will require it in your test files this way:
|
|
735
|
-
|
|
736
|
-
```ruby
|
|
737
|
-
# good
|
|
738
|
-
require_relative 'spec_helper'
|
|
739
|
-
|
|
740
|
-
# bad - won't work
|
|
741
|
-
require 'spec_helper'
|
|
742
|
-
```
|
|
743
|
-
|
|
744
|
-
### How to update existing knapsack report for a few test files?
|
|
745
|
-
|
|
746
|
-
You may want to look at monkey patch in [this issue](https://github.com/ArturT/knapsack/issues/34). Take into account that there are some cons of this approach.
|
|
747
|
-
|
|
748
|
-
### How to run tests for particular CI node in your development environment
|
|
749
|
-
|
|
750
|
-
In your development environment you can debug tests that were run on the particular CI node.
|
|
751
|
-
For instance to run subset of tests for the first CI node with specified seed you can do.
|
|
752
|
-
|
|
753
|
-
CI_NODE_TOTAL=2 \
|
|
754
|
-
CI_NODE_INDEX=0 \
|
|
755
|
-
bundle exec rake "knapsack:rspec[--seed 123]"
|
|
756
|
-
|
|
757
|
-
Above example is for RSpec. You can use respectively rake task name and token environment variable when you want to run tests for minitest, cucumber or spinach.
|
|
758
|
-
|
|
759
|
-
### How can I change log level?
|
|
760
|
-
|
|
761
|
-
You can change log level by specifying the `KNAPSACK_LOG_LEVEL` environment variable.
|
|
762
|
-
|
|
763
|
-
KNAPSACK_LOG_LEVEL=warn bundle exec rake knapsack:rspec
|
|
764
|
-
|
|
765
|
-
Available values are `debug`, `info`, and `warn`. The default log level is `info`.
|
|
766
|
-
|
|
767
|
-
## Gem tests
|
|
768
|
-
|
|
769
|
-
### Spec
|
|
770
|
-
|
|
771
|
-
To run specs for Knapsack gem type:
|
|
772
|
-
|
|
773
|
-
$ bundle exec rspec spec
|
|
774
|
-
|
|
775
|
-
### Spec examples
|
|
776
|
-
|
|
777
|
-
Directory `spec_examples` contains examples of fast and slow specs. There is a `spec_example/spec_helper.rb` with binded Knapsack.
|
|
778
|
-
|
|
779
|
-
To generate a new knapsack report for specs with `focus` tag (only specs in `spec_examples/leftover` directory have no `focus` tag), please type:
|
|
780
|
-
|
|
781
|
-
$ KNAPSACK_GENERATE_REPORT=true bundle exec rspec --default-path spec_examples --tag focus
|
|
782
|
-
|
|
783
|
-
**Warning:** Current `knapsack_rspec_report.json` file was generated for `spec_examples` except `spec_examples/leftover` directory. Just for testing reason to see how leftover specs will be distribute in a dumb way across CI nodes.
|
|
784
|
-
|
|
785
|
-
To see specs distributed for the first CI node type:
|
|
786
|
-
|
|
787
|
-
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 KNAPSACK_SPEC_PATTERN="spec_examples/**{,/*/**}/*_spec.rb" bundle exec rake knapsack:rspec
|
|
788
|
-
|
|
789
|
-
Specs in `spec_examples/leftover` take more than 3 seconds. This should cause a Knapsack time offset warning because we set `time_offset_in_seconds` to 3 in `spec_examples/spec_helper.rb`. Type below to see warning:
|
|
790
|
-
|
|
791
|
-
$ bundle exec rspec --default-path spec_examples
|
|
792
|
-
|
|
793
|
-
## Contributing
|
|
794
|
-
|
|
795
|
-
1. Fork it ( https://github.com/ArturT/knapsack/fork )
|
|
796
|
-
2. Create your feature branch (`git checkout -b my-new-feature`)
|
|
797
|
-
3. Commit your changes (`git commit -am 'Add some feature'`)
|
|
798
|
-
4. Push to the branch (`git push origin my-new-feature`)
|
|
799
|
-
5. You can create example tests in related repository with example of [rails application and knapsack gem usage](https://github.com/KnapsackPro/rails-app-with-knapsack).
|
|
800
|
-
6. Create a new Pull Request
|
|
801
|
-
|
|
802
|
-
## Acknowledgements
|
|
803
|
-
|
|
804
|
-
Many thanks to [Małgorzata Nowak](https://github.com/informatykgosia) for beautiful logo.
|
|
805
|
-
|
|
806
|
-
## Mentions
|
|
807
|
-
|
|
808
|
-
* Lunar Logic Blog | [Parallel your specs and don’t waste time](http://blog.lunarlogic.io/2014/parallel-your-specs-and-dont-waste-time/)
|
|
809
|
-
* Travis CI | [Parallelizing RSpec and Cucumber on multiple VMs](http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-RSpec-and-Cucumber-on-multiple-VMs)
|
|
810
|
-
* Semaphore | [Running RSpec specs in parallel](https://semaphoreapp.com/docs/running-rspec-specs-in-threads.html)
|
|
811
|
-
* Semaphore | [Running Cucumber scenarios in parallel](https://semaphoreapp.com/docs/running-cucumber-scenarios-in-threads.html)
|
|
812
|
-
* Buildkite | [Libraries](https://buildkite.com/docs/guides/parallelizing-builds#libraries)
|
|
813
|
-
* Snap CI | [Knapsack: optimal test suite split based on time execution](https://docs.snap-ci.com/speeding-up-builds/test-parallelism/#parallelism-using-third-party-tools%23knapsack-optimal-test-suite-split-based-on-time-execution)
|
|
814
|
-
* CircleCI | [Test suite split with knapsack gem](https://circleci.com/docs/parallel-manual-setup/#test-suite-split-with-knapsack-gem)
|
|
815
|
-
* Michał Knapik Blog | [Knapsack with Jenkins Pipeline](http://blog.knapik.me/knapsack-with-jenkins-pipeline/)
|
|
33
|
+
* Knapsack Pro is risk-free integration! Knapsack Pro runs tests in Fallback Mode if your CI servers can't reach our API for any reason.
|
|
34
|
+
* We don't need access to your repository. Knapsack Pro is just wrapper around test runner like RSpec, Cucumber, Minitest etc.
|
|
35
|
+
* Hundreds of developers use Knapsack Pro every day to run fast CI builds.
|
|
36
|
+
* It works with other programming languages.
|
data/knapsack.gemspec
CHANGED
|
@@ -22,7 +22,7 @@ Gem::Specification.new do |spec|
|
|
|
22
22
|
|
|
23
23
|
spec.add_dependency 'rake', '>= 0'
|
|
24
24
|
|
|
25
|
-
spec.add_development_dependency 'bundler', '
|
|
25
|
+
spec.add_development_dependency 'bundler', '>= 1.6'
|
|
26
26
|
spec.add_development_dependency 'rspec', '~> 3.0', '>= 2.10.0'
|
|
27
27
|
spec.add_development_dependency 'rspec-its', '~> 1.2'
|
|
28
28
|
spec.add_development_dependency 'cucumber', '>= 0'
|
data/lib/knapsack/config/env.rb
CHANGED
|
@@ -7,11 +7,11 @@ module Knapsack
|
|
|
7
7
|
end
|
|
8
8
|
|
|
9
9
|
def ci_node_total
|
|
10
|
-
ENV['CI_NODE_TOTAL'] || ENV['CIRCLE_NODE_TOTAL'] || ENV['SEMAPHORE_THREAD_COUNT'] || ENV['BUILDKITE_PARALLEL_JOB_COUNT'] || ENV['SNAP_WORKER_TOTAL'] || 1
|
|
10
|
+
ENV['CI_NODE_TOTAL'] || ENV['CIRCLE_NODE_TOTAL'] || ENV['SEMAPHORE_JOB_COUNT'] || ENV['SEMAPHORE_THREAD_COUNT'] || ENV['BUILDKITE_PARALLEL_JOB_COUNT'] || ENV['SNAP_WORKER_TOTAL'] || ENV['BITBUCKET_PARALLEL_STEP_COUNT'] || 1
|
|
11
11
|
end
|
|
12
12
|
|
|
13
13
|
def ci_node_index
|
|
14
|
-
ENV['CI_NODE_INDEX'] || ENV['CIRCLE_NODE_INDEX'] || semaphore_current_thread || ENV['BUILDKITE_PARALLEL_JOB'] || snap_ci_worker_index || 0
|
|
14
|
+
gitlab_ci_node_index || ENV['CI_NODE_INDEX'] || ENV['CIRCLE_NODE_INDEX'] || semaphore_job_index || semaphore_current_thread || ENV['BUILDKITE_PARALLEL_JOB'] || snap_ci_worker_index || ENV['BITBUCKET_PARALLEL_STEP'] || 0
|
|
15
15
|
end
|
|
16
16
|
|
|
17
17
|
def test_file_pattern
|
|
@@ -36,6 +36,10 @@ module Knapsack
|
|
|
36
36
|
index.to_i - 1 if index
|
|
37
37
|
end
|
|
38
38
|
|
|
39
|
+
def semaphore_job_index
|
|
40
|
+
index_starting_from_one(ENV['SEMAPHORE_JOB_INDEX'])
|
|
41
|
+
end
|
|
42
|
+
|
|
39
43
|
def semaphore_current_thread
|
|
40
44
|
index_starting_from_one(ENV['SEMAPHORE_CURRENT_THREAD'])
|
|
41
45
|
end
|
|
@@ -43,6 +47,12 @@ module Knapsack
|
|
|
43
47
|
def snap_ci_worker_index
|
|
44
48
|
index_starting_from_one(ENV['SNAP_WORKER_INDEX'])
|
|
45
49
|
end
|
|
50
|
+
|
|
51
|
+
def gitlab_ci_node_index
|
|
52
|
+
return unless ENV['GITLAB_CI']
|
|
53
|
+
|
|
54
|
+
index_starting_from_one(ENV['CI_NODE_INDEX'])
|
|
55
|
+
end
|
|
46
56
|
end
|
|
47
57
|
end
|
|
48
58
|
end
|
data/lib/knapsack/presenter.rb
CHANGED
|
@@ -61,9 +61,13 @@ Global time execution for this CI node is fine.
|
|
|
61
61
|
Happy testing!}
|
|
62
62
|
end
|
|
63
63
|
str << "\n\nNeed explanation? See FAQ:"
|
|
64
|
-
str << "\nhttps://
|
|
64
|
+
str << "\nhttps://docs.knapsackpro.com/ruby/knapsack#faq"
|
|
65
65
|
str << "\n=================================================\n"
|
|
66
|
-
str << %{
|
|
66
|
+
str << %{See how to split tests in a dynamic way using Queue Mode to ensure all parallel nodes
|
|
67
|
+
finish work at a similar time even when your CI or random test execution is a bottleneck:
|
|
68
|
+
https://youtu.be/hUEB1XDKEFY
|
|
69
|
+
|
|
70
|
+
Sign up for Knapsack Pro here:
|
|
67
71
|
https://knapsackpro.com}
|
|
68
72
|
str << "\n=================================================\n"
|
|
69
73
|
str
|
data/lib/knapsack/version.rb
CHANGED
|
@@ -27,6 +27,11 @@ describe Knapsack::Config::Env do
|
|
|
27
27
|
it { should eql 4 }
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
+
context 'when SEMAPHORE_JOB_COUNT has value' do
|
|
31
|
+
before { stub_const("ENV", { 'SEMAPHORE_JOB_COUNT' => 3 }) }
|
|
32
|
+
it { should eql 3 }
|
|
33
|
+
end
|
|
34
|
+
|
|
30
35
|
context 'when SEMAPHORE_THREAD_COUNT has value' do
|
|
31
36
|
before { stub_const("ENV", { 'SEMAPHORE_THREAD_COUNT' => 3 }) }
|
|
32
37
|
it { should eql 3 }
|
|
@@ -41,6 +46,11 @@ describe Knapsack::Config::Env do
|
|
|
41
46
|
before { stub_const("ENV", { 'SNAP_WORKER_TOTAL' => 6 }) }
|
|
42
47
|
it { should eql 6 }
|
|
43
48
|
end
|
|
49
|
+
|
|
50
|
+
context 'when BITBUCKET_PARALLEL_STEP_COUNT has value' do
|
|
51
|
+
before { stub_const("ENV", { 'BITBUCKET_PARALLEL_STEP_COUNT' => 8 }) }
|
|
52
|
+
it { should eql 8 }
|
|
53
|
+
end
|
|
44
54
|
end
|
|
45
55
|
|
|
46
56
|
context "when ENV doesn't exist" do
|
|
@@ -57,11 +67,21 @@ describe Knapsack::Config::Env do
|
|
|
57
67
|
it { should eql 3 }
|
|
58
68
|
end
|
|
59
69
|
|
|
70
|
+
context 'when CI_NODE_INDEX has value and is in GitLab CI' do
|
|
71
|
+
before { stub_const("ENV", { 'CI_NODE_INDEX' => 3, 'GITLAB_CI' => 'true' }) }
|
|
72
|
+
it { should eql 2 }
|
|
73
|
+
end
|
|
74
|
+
|
|
60
75
|
context 'when CIRCLE_NODE_INDEX has value' do
|
|
61
76
|
before { stub_const("ENV", { 'CIRCLE_NODE_INDEX' => 2 }) }
|
|
62
77
|
it { should eql 2 }
|
|
63
78
|
end
|
|
64
79
|
|
|
80
|
+
context 'when SEMAPHORE_JOB_INDEX has value' do
|
|
81
|
+
before { stub_const("ENV", { 'SEMAPHORE_JOB_INDEX' => 3 }) }
|
|
82
|
+
it { should eql 2 }
|
|
83
|
+
end
|
|
84
|
+
|
|
65
85
|
context 'when SEMAPHORE_CURRENT_THREAD has value' do
|
|
66
86
|
before { stub_const("ENV", { 'SEMAPHORE_CURRENT_THREAD' => 1 }) }
|
|
67
87
|
it { should eql 0 }
|
|
@@ -76,6 +96,11 @@ describe Knapsack::Config::Env do
|
|
|
76
96
|
before { stub_const("ENV", { 'SNAP_WORKER_INDEX' => 4 }) }
|
|
77
97
|
it { should eql 3 }
|
|
78
98
|
end
|
|
99
|
+
|
|
100
|
+
context 'when BITBUCKET_PARALLEL_STEP has value' do
|
|
101
|
+
before { stub_const("ENV", { 'BITBUCKET_PARALLEL_STEP' => 7 }) }
|
|
102
|
+
it { should eql 7 }
|
|
103
|
+
end
|
|
79
104
|
end
|
|
80
105
|
|
|
81
106
|
context "when ENV doesn't exist" do
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: knapsack
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.19.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- ArturT
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2020-09-30 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rake
|
|
@@ -28,36 +28,36 @@ dependencies:
|
|
|
28
28
|
name: bundler
|
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
|
30
30
|
requirements:
|
|
31
|
-
- - "
|
|
31
|
+
- - ">="
|
|
32
32
|
- !ruby/object:Gem::Version
|
|
33
33
|
version: '1.6'
|
|
34
34
|
type: :development
|
|
35
35
|
prerelease: false
|
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
37
|
requirements:
|
|
38
|
-
- - "
|
|
38
|
+
- - ">="
|
|
39
39
|
- !ruby/object:Gem::Version
|
|
40
40
|
version: '1.6'
|
|
41
41
|
- !ruby/object:Gem::Dependency
|
|
42
42
|
name: rspec
|
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
|
44
44
|
requirements:
|
|
45
|
-
- - "~>"
|
|
46
|
-
- !ruby/object:Gem::Version
|
|
47
|
-
version: '3.0'
|
|
48
45
|
- - ">="
|
|
49
46
|
- !ruby/object:Gem::Version
|
|
50
47
|
version: 2.10.0
|
|
48
|
+
- - "~>"
|
|
49
|
+
- !ruby/object:Gem::Version
|
|
50
|
+
version: '3.0'
|
|
51
51
|
type: :development
|
|
52
52
|
prerelease: false
|
|
53
53
|
version_requirements: !ruby/object:Gem::Requirement
|
|
54
54
|
requirements:
|
|
55
|
-
- - "~>"
|
|
56
|
-
- !ruby/object:Gem::Version
|
|
57
|
-
version: '3.0'
|
|
58
55
|
- - ">="
|
|
59
56
|
- !ruby/object:Gem::Version
|
|
60
57
|
version: 2.10.0
|
|
58
|
+
- - "~>"
|
|
59
|
+
- !ruby/object:Gem::Version
|
|
60
|
+
version: '3.0'
|
|
61
61
|
- !ruby/object:Gem::Dependency
|
|
62
62
|
name: rspec-its
|
|
63
63
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -285,8 +285,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
285
285
|
- !ruby/object:Gem::Version
|
|
286
286
|
version: '0'
|
|
287
287
|
requirements: []
|
|
288
|
-
|
|
289
|
-
rubygems_version: 2.6.13
|
|
288
|
+
rubygems_version: 3.0.6
|
|
290
289
|
signing_key:
|
|
291
290
|
specification_version: 4
|
|
292
291
|
summary: Knapsack splits tests across CI nodes and makes sure that tests will run
|