knapsack_pro 3.7.0 → 3.8.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/assets/install-button.png +0 -0
- data/.github/assets/knapsack-diamonds.png +0 -0
- data/CHANGELOG.md +8 -0
- data/README.md +75 -1680
- data/lib/knapsack_pro/adapters/base_adapter.rb +2 -2
- data/lib/knapsack_pro/adapters/rspec_adapter.rb +3 -3
- data/lib/knapsack_pro/adapters/test_unit_adapter.rb +1 -1
- data/lib/knapsack_pro/allocator.rb +4 -4
- data/lib/knapsack_pro/base_allocator_builder.rb +1 -1
- data/lib/knapsack_pro/build_distribution_fetcher.rb +1 -1
- data/lib/knapsack_pro/queue_allocator.rb +4 -4
- data/lib/knapsack_pro/report.rb +1 -1
- data/lib/knapsack_pro/slow_test_file_determiner.rb +1 -1
- data/lib/knapsack_pro/slow_test_file_finder.rb +1 -1
- data/lib/knapsack_pro/tracker.rb +1 -1
- data/lib/knapsack_pro/urls.rb +37 -0
- data/lib/knapsack_pro/version.rb +1 -1
- data/lib/knapsack_pro.rb +1 -0
- data/spec/knapsack_pro/allocator_spec.rb +3 -3
- data/spec/knapsack_pro/base_allocator_builder_spec.rb +1 -1
- data/spec/knapsack_pro/queue_allocator_spec.rb +3 -3
- data/spec/knapsack_pro/report_spec.rb +1 -1
- data/spec/knapsack_pro/slow_test_file_determiner_spec.rb +1 -1
- data/spec/knapsack_pro/slow_test_file_finder_spec.rb +1 -1
- data/spec/knapsack_pro/tracker_spec.rb +1 -0
- metadata +9 -6
data/README.md
CHANGED
@@ -1,1715 +1,110 @@
|
|
1
|
-
# knapsack_pro ruby gem
|
2
|
-
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
1
|
+
# `knapsack_pro` ruby gem
|
2
|
+
|
3
|
+
<p align="center">
|
4
|
+
<a href="https://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=hero_logo">
|
5
|
+
<img alt="Knapsack Pro" src="./.github/assets/knapsack-diamonds.png" width="300" height="300" style="max-width: 100%;" />
|
6
|
+
</a>
|
7
|
+
</p>
|
8
|
+
|
9
|
+
<h3 align="center">Speed up your tests</h3>
|
10
|
+
<p align="center">Run your 1-hour test suite in 2 minutes with optimal parallelisation on your existing CI infrastructure</p>
|
11
|
+
|
12
|
+
---
|
13
|
+
|
14
|
+
<div align="center">
|
15
|
+
<a href="https://circleci.com/gh/KnapsackPro/knapsack_pro-ruby">
|
16
|
+
<img alt="Circle CI" src="https://circleci.com/gh/KnapsackPro/knapsack_pro-ruby.svg" />
|
17
|
+
</a>
|
18
|
+
<a href="https://rubygems.org/gems/knapsack_pro">
|
19
|
+
<img alt="Gem Version" src="https://badge.fury.io/rb/knapsack_pro.svg" />
|
20
|
+
</a>
|
21
|
+
<a href="https://codeclimate.com/github/KnapsackPro/knapsack_pro-ruby">
|
22
|
+
<img alt="Code Climate" src="https://codeclimate.com/github/KnapsackPro/knapsack_pro-ruby/badges/gpa.svg" />
|
23
|
+
</a>
|
24
|
+
</div>
|
25
|
+
|
26
|
+
<br />
|
27
|
+
<br />
|
28
|
+
|
29
|
+
Knapsack Pro wraps your current test runner(s) and works with your existing CI infrastructure to parallelize tests optimally:
|
30
|
+
|
31
|
+
- Dynamically splits your tests based on up-to-date test execution data
|
32
|
+
- Is designed from the ground up for CI and supports all of them
|
33
|
+
- Tracks your CI builds to detect bottlenecks
|
34
|
+
- Does not have access to your source code and collects minimal test data (with opt-in encryption)
|
35
|
+
- Enables you to export historical metrics about your CI builds
|
36
|
+
- Supports out-of-the-box any Ruby test runners, Cypress, Jest (and provides both SDK and API to integrate with any other language)
|
37
|
+
- Replaces local dependencies like Redis with an API and runs your tests regardless of network problems
|
38
|
+
|
39
|
+
The `knapsack_pro` gem supports all CIs and the following test runners:
|
40
|
+
|
41
|
+
- RSpec
|
42
|
+
- Cucumber
|
43
|
+
- Minitest
|
44
|
+
- test-unit
|
45
|
+
- Spinach
|
46
|
+
- Turnip
|
47
|
+
|
48
|
+
## Requirements
|
48
49
|
|
49
50
|
`>= Ruby 2.1.0`
|
50
51
|
|
51
|
-
# Table of Contents
|
52
|
-
|
53
|
-
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
54
|
-
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
55
|
-
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
|
56
|
-
|
57
|
-
- [Update gem](#update-gem)
|
58
|
-
- [Installation](#installation)
|
59
|
-
- [How to set up](#how-to-set-up)
|
60
|
-
- [Usage (How to set up 1 of 3)](#usage-how-to-set-up-1-of-3)
|
61
|
-
- [Step for RSpec](#step-for-rspec)
|
62
|
-
- [Step for Cucumber](#step-for-cucumber)
|
63
|
-
- [Step for Minitest](#step-for-minitest)
|
64
|
-
- [Step for test-unit](#step-for-test-unit)
|
65
|
-
- [Step for Spinach](#step-for-spinach)
|
66
|
-
- [Custom configuration](#custom-configuration)
|
67
|
-
- [Setup your CI server (How to set up 2 of 3)](#setup-your-ci-server-how-to-set-up-2-of-3)
|
68
|
-
- [Set API key token](#set-api-key-token)
|
69
|
-
- [Set knapsack_pro command to execute tests](#set-knapsack_pro-command-to-execute-tests)
|
70
|
-
- [Repository adapter (How to set up 3 of 3)](#repository-adapter-how-to-set-up-3-of-3)
|
71
|
-
- [By default `KNAPSACK_PRO_REPOSITORY_ADAPTER` environment variable is undefined](#by-default-knapsack_pro_repository_adapter-environment-variable-is-undefined)
|
72
|
-
- [When should you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (when CI provider is not supported and you use git)](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-when-ci-provider-is-not-supported-and-you-use-git)
|
73
|
-
- [When you don't use git](#when-you-dont-use-git)
|
74
|
-
- [Queue Mode](#queue-mode)
|
75
|
-
- [How does queue mode work?](#how-does-queue-mode-work)
|
76
|
-
- [How to use queue mode?](#how-to-use-queue-mode)
|
77
|
-
- [Additional info about queue mode](#additional-info-about-queue-mode)
|
78
|
-
- [Extra configuration for Queue Mode](#extra-configuration-for-queue-mode)
|
79
|
-
- [KNAPSACK_PRO_FIXED_QUEUE_SPLIT (remember queue split on retry CI node)](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node)
|
80
|
-
- [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests)
|
81
|
-
- [Supported test runners in queue mode](#supported-test-runners-in-queue-mode)
|
82
|
-
- [Split test files by test cases](#split-test-files-by-test-cases)
|
83
|
-
- [RSpec split test files by test examples (by individual `it`s)](#rspec-split-test-files-by-test-examples-by-individual-its)
|
84
|
-
- [How to manually define a list of slow test files to be split by test cases](#how-to-manually-define-a-list-of-slow-test-files-to-be-split-by-test-cases)
|
85
|
-
- [Extra configuration for CI server](#extra-configuration-for-ci-server)
|
86
|
-
- [Info about ENV variables](#info-about-env-variables)
|
87
|
-
- [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT (test suite split based on seed)](#knapsack_pro_fixed_test_suite_split-test-suite-split-based-on-seed)
|
88
|
-
- [Environment variables for debugging gem](#environment-variables-for-debugging-gem)
|
89
|
-
- [Required CI configuration if you use retry single failed CI node feature on your CI server when KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true (in Queue Mode) or KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true (in Regular Mode)](#required-ci-configuration-if-you-use-retry-single-failed-ci-node-feature-on-your-ci-server-when-knapsack_pro_fixed_queue_splittrue-in-queue-mode-or-knapsack_pro_fixed_test_suite_splittrue-in-regular-mode)
|
90
|
-
- [Passing arguments to rake task](#passing-arguments-to-rake-task)
|
91
|
-
- [Passing arguments to rspec](#passing-arguments-to-rspec)
|
92
|
-
- [Passing arguments to cucumber](#passing-arguments-to-cucumber)
|
93
|
-
- [Passing arguments to minitest](#passing-arguments-to-minitest)
|
94
|
-
- [Passing arguments to test-unit](#passing-arguments-to-test-unit)
|
95
|
-
- [Passing arguments to spinach](#passing-arguments-to-spinach)
|
96
|
-
- [Knapsack Pro binary](#knapsack-pro-binary)
|
97
|
-
- [Test file names encryption](#test-file-names-encryption)
|
98
|
-
- [How to enable test file names encryption?](#how-to-enable-test-file-names-encryption)
|
99
|
-
- [How to debug test file names?](#how-to-debug-test-file-names)
|
100
|
-
- [How to enable branch names encryption?](#how-to-enable-branch-names-encryption)
|
101
|
-
- [How to debug branch names?](#how-to-debug-branch-names)
|
102
|
-
- [Supported CI providers](#supported-ci-providers)
|
103
|
-
- [Info for CircleCI users](#info-for-circleci-users)
|
104
|
-
- [CircleCI and knapsack_pro Queue Mode](#circleci-and-knapsack_pro-queue-mode)
|
105
|
-
- [Info for Travis users](#info-for-travis-users)
|
106
|
-
- [Info for semaphoreci.com users](#info-for-semaphorecicom-users)
|
107
|
-
- [Semaphore 2.0](#semaphore-20)
|
108
|
-
- [Semaphore 1.0](#semaphore-10)
|
109
|
-
- [Info for buildkite.com users](#info-for-buildkitecom-users)
|
110
|
-
- [Info for GitLab CI users](#info-for-gitlab-ci-users)
|
111
|
-
- [GitLab CI `>= 11.5`](#gitlab-ci--115)
|
112
|
-
- [GitLab CI `< 11.5` (old GitLab CI)](#gitlab-ci--115-old-gitlab-ci)
|
113
|
-
- [Info for codeship.com users](#info-for-codeshipcom-users)
|
114
|
-
- [Info for Heroku CI users](#info-for-heroku-ci-users)
|
115
|
-
- [Info for Solano CI users](#info-for-solano-ci-users)
|
116
|
-
- [Info for AppVeyor users](#info-for-appveyor-users)
|
117
|
-
- [Info for snap-ci.com users](#info-for-snap-cicom-users)
|
118
|
-
- [Info for cirrus-ci.org users](#info-for-cirrus-ciorg-users)
|
119
|
-
- [Info for Jenkins users](#info-for-jenkins-users)
|
120
|
-
- [Info for GitHub Actions users](#info-for-github-actions-users)
|
121
|
-
- [Info for Codefresh.io users](#info-for-codefreshio-users)
|
122
|
-
- [Gem tests](#gem-tests)
|
123
|
-
- [Spec](#spec)
|
124
|
-
- [Contributing](#contributing)
|
125
|
-
- [Publishing](#publishing)
|
126
|
-
- [Mentions](#mentions)
|
127
|
-
|
128
|
-
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
129
|
-
|
130
|
-
## Update gem
|
131
|
-
|
132
|
-
Please check [changelog](CHANGELOG.md) before updating gem. Knapsack Pro follows [semantic versioning](http://semver.org).
|
133
|
-
|
134
52
|
## Installation
|
135
53
|
|
136
|
-
|
137
|
-
|
138
|
-
```ruby
|
139
|
-
group :test, :development do
|
140
|
-
gem 'knapsack_pro'
|
141
|
-
end
|
142
|
-
```
|
143
|
-
|
144
|
-
And then execute:
|
145
|
-
|
146
|
-
```bash
|
147
|
-
bundle install
|
148
|
-
```
|
149
|
-
|
150
|
-
If you are not using Rails then add this line at the bottom of `Rakefile`:
|
151
|
-
|
152
|
-
```ruby
|
153
|
-
# Add this only if you are not using Rails.
|
154
|
-
# If you use Rails then knapsack_pro rake tasks are already loaded
|
155
|
-
# so there is no need to explicitly load them.
|
156
|
-
KnapsackPro.load_tasks if defined?(KnapsackPro)
|
157
|
-
```
|
158
|
-
|
159
|
-
__Please check [online installation guide](http://docs.knapsackpro.com/knapsack_pro-ruby/guide/#questions) to get started.__ It will ask you a few questions and generate instruction steps for your project.
|
160
|
-
|
161
|
-
_You only need to read the next section if you want to understand optional gem configuration and features._
|
162
|
-
|
163
|
-
## How to set up
|
164
|
-
|
165
|
-
If you use [VCR](https://github.com/vcr/vcr), [WebMock](https://github.com/bblimke/webmock) or [FakeWeb](https://github.com/chrisk/fakeweb) gems then you need to allow them to make requests to the Knapsack Pro API.
|
166
|
-
|
167
|
-
For VCR add Knapsack Pro API subdomain to [ignore hosts](https://www.relishapp.com/vcr/vcr/v/2-9-3/docs/configuration/ignore-request):
|
168
|
-
|
169
|
-
```ruby
|
170
|
-
# spec/spec_helper.rb or wherever your VCR configuration is
|
171
|
-
|
172
|
-
require 'vcr'
|
173
|
-
VCR.configure do |config|
|
174
|
-
config.hook_into :webmock # or :fakeweb
|
175
|
-
config.ignore_hosts('localhost', '127.0.0.1', '0.0.0.0', 'api.knapsackpro.com')
|
176
|
-
end
|
177
|
-
|
178
|
-
# add below when you hook into webmock
|
179
|
-
require 'webmock/rspec'
|
180
|
-
WebMock.disable_net_connect!(allow_localhost: true, allow: ['api.knapsackpro.com'])
|
181
|
-
|
182
|
-
# add below when you use FakeWeb
|
183
|
-
require 'fakeweb'
|
184
|
-
FakeWeb.allow_net_connect = %r[^https?://api\.knapsackpro\.com]
|
185
|
-
```
|
186
|
-
|
187
|
-
Ensure you have `require false` in your Gemfile for webmock gem (see below) when VCR is hooked into it. That ensures that the webmock configuration in `spec_helper.rb` (above) is loaded properly.
|
188
|
-
|
189
|
-
```ruby
|
190
|
-
# Gemfile
|
191
|
-
group :test do
|
192
|
-
gem 'vcr'
|
193
|
-
gem 'webmock', require: false
|
194
|
-
gem 'fakeweb', require: false # example when you use fakeweb
|
195
|
-
end
|
196
|
-
```
|
197
|
-
|
198
|
-
If you happen to see your tests failing due to WebMock not allowing requests to Knapsack Pro API it means you probably reconfigure WebMock in some of your tests.
|
199
|
-
For instance, you may use `WebMock.reset!` or it's called automatically in `after(:each)` block, if you `require 'webmock/rspec'` ([more about the issue](https://github.com/bblimke/webmock/issues/484#issuecomment-116193449)). It will remove api.knapsackpro.com from whitelisted domains. Please try below:
|
200
|
-
|
201
|
-
```ruby
|
202
|
-
RSpec.configure do |config|
|
203
|
-
config.after(:suite) do
|
204
|
-
WebMock.disable_net_connect!(
|
205
|
-
allow_localhost: true,
|
206
|
-
allow: [
|
207
|
-
'api.knapsackpro.com',
|
208
|
-
],
|
209
|
-
)
|
210
|
-
end
|
211
|
-
end
|
212
|
-
```
|
213
|
-
|
214
|
-
### Usage (How to set up 1 of 3)
|
215
|
-
|
216
|
-
__Tip:__ You can find here an example of a rails app with knapsack_pro already configured.
|
217
|
-
|
218
|
-
https://github.com/KnapsackPro/rails-app-with-knapsack_pro
|
219
|
-
|
220
|
-
#### Step for RSpec
|
221
|
-
|
222
|
-
Add at the beginning of your `spec_helper.rb`:
|
223
|
-
|
224
|
-
```ruby
|
225
|
-
require 'knapsack_pro'
|
226
|
-
|
227
|
-
# CUSTOM_CONFIG_GOES_HERE
|
228
|
-
|
229
|
-
KnapsackPro::Adapters::RSpecAdapter.bind
|
230
|
-
```
|
231
|
-
|
232
|
-
#### Step for Cucumber
|
233
|
-
|
234
|
-
Create file `features/support/knapsack_pro.rb` and add there:
|
235
|
-
|
236
|
-
```ruby
|
237
|
-
require 'knapsack_pro'
|
238
|
-
|
239
|
-
# CUSTOM_CONFIG_GOES_HERE
|
240
|
-
|
241
|
-
KnapsackPro::Adapters::CucumberAdapter.bind
|
242
|
-
```
|
243
|
-
|
244
|
-
#### Step for Minitest
|
245
|
-
|
246
|
-
Add at the beginning of your `test_helper.rb`:
|
247
|
-
|
248
|
-
```ruby
|
249
|
-
require 'knapsack_pro'
|
250
|
-
|
251
|
-
# CUSTOM_CONFIG_GOES_HERE
|
252
|
-
|
253
|
-
knapsack_pro_adapter = KnapsackPro::Adapters::MinitestAdapter.bind
|
254
|
-
knapsack_pro_adapter.set_test_helper_path(__FILE__)
|
255
|
-
```
|
256
|
-
|
257
|
-
#### Step for test-unit
|
258
|
-
|
259
|
-
Add at the beginning of your `test_helper.rb`:
|
260
|
-
|
261
|
-
```ruby
|
262
|
-
require 'knapsack_pro'
|
263
|
-
|
264
|
-
# CUSTOM_CONFIG_GOES_HERE
|
265
|
-
|
266
|
-
knapsack_pro_adapter = KnapsackPro::Adapters::TestUnitAdapter.bind
|
267
|
-
knapsack_pro_adapter.set_test_helper_path(__FILE__)
|
268
|
-
```
|
269
|
-
|
270
|
-
#### Step for Spinach
|
271
|
-
|
272
|
-
Create file `features/support/knapsack_pro.rb` and add there:
|
273
|
-
|
274
|
-
```ruby
|
275
|
-
require 'knapsack_pro'
|
276
|
-
|
277
|
-
# CUSTOM_CONFIG_GOES_HERE
|
278
|
-
|
279
|
-
KnapsackPro::Adapters::SpinachAdapter.bind
|
280
|
-
```
|
281
|
-
|
282
|
-
#### Custom configuration
|
283
|
-
|
284
|
-
You can change the default Knapsack Pro configuration for RSpec, Cucumber, Minitest, test-unit or Spinach tests. Here are examples what you can do. Put the configuration below in place of `CUSTOM_CONFIG_GOES_HERE` (in the configuration samples above).
|
285
|
-
|
286
|
-
```ruby
|
287
|
-
# you can use your own logger
|
288
|
-
require 'logger'
|
289
|
-
KnapsackPro.logger = Logger.new(STDOUT)
|
290
|
-
KnapsackPro.logger.level = Logger::DEBUG
|
291
|
-
```
|
292
|
-
|
293
|
-
Debug is default log level and it is recommended. [Read more](https://knapsackpro.com/faq/question/how-can-i-change-log-level).
|
294
|
-
|
295
|
-
Note your own logger is configured in `spec_helper.rb` or `rails_helper.rb` and it will start working when those files will be loaded.
|
296
|
-
It means the very first request to Knapsack Pro API will be log to `STDOUT` using logger built into knapsack_pro instead of your custom logger.
|
297
|
-
|
298
|
-
If you want to change log level globally than just for your custom log level, please [see this](https://knapsackpro.com/faq/question/how-can-i-change-log-level).
|
299
|
-
|
300
|
-
### Setup your CI server (How to set up 2 of 3)
|
301
|
-
|
302
|
-
#### Set API key token
|
303
|
-
|
304
|
-
Set one or more tokens depending on how many test suites you run on CI server.
|
305
|
-
|
306
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` - as value set token for rspec test suite. Token can be generated when you sign in to [knapsackpro.com](http://www.knapsackpro.com).
|
307
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` - token for cucumber test suite.
|
308
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST` - token for minitest test suite.
|
309
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT` - token for test-unit test suite.
|
310
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH` - token for spinach test suite.
|
311
|
-
|
312
|
-
__Tip:__ In case you have for instance multiple rspec test suites then prepend each of knapsack_pro command which executes tests with `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` variable.
|
313
|
-
|
314
|
-
#### Set knapsack_pro command to execute tests
|
315
|
-
|
316
|
-
On your CI server run this command for the first CI node. Update `KNAPSACK_PRO_CI_NODE_INDEX` for the next one.
|
317
|
-
|
318
|
-
```bash
|
319
|
-
# Step for RSpec
|
320
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec
|
321
|
-
|
322
|
-
# Step for Cucumber
|
323
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber
|
324
|
-
|
325
|
-
# Step for Minitest
|
326
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest
|
327
|
-
|
328
|
-
# Step for test-unit
|
329
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit
|
330
|
-
|
331
|
-
# Step for Spinach
|
332
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach
|
333
|
-
```
|
334
|
-
|
335
|
-
You can add `KNAPSACK_PRO_TEST_FILE_PATTERN` if your tests are not in default directory. For instance:
|
336
|
-
|
337
|
-
```bash
|
338
|
-
# Step for RSpec
|
339
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_specs/**{,/*/**}/*_spec.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec
|
340
|
-
|
341
|
-
# Step for Cucumber
|
342
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber
|
343
|
-
|
344
|
-
# Step for Minitest
|
345
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest
|
346
|
-
|
347
|
-
# Step for test-unit
|
348
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit
|
349
|
-
|
350
|
-
# Step for Spinach
|
351
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach
|
352
|
-
```
|
353
|
-
|
354
|
-
__Tip:__ If you use one of the supported CI providers then instead of the above steps you should [take a look at this](#supported-ci-providers).
|
355
|
-
|
356
|
-
__Tip 2:__ If you use one of unsupported CI providers ([here is list of supported CI providers](#supported-ci-providers)) then you should [set KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-when-ci-provider-is-not-supported-and-you-use-git).
|
357
|
-
|
358
|
-
### Repository adapter (How to set up 3 of 3)
|
359
|
-
|
360
|
-
#### By default `KNAPSACK_PRO_REPOSITORY_ADAPTER` environment variable is undefined
|
361
|
-
|
362
|
-
By default `KNAPSACK_PRO_REPOSITORY_ADAPTER` variable has no value so knapsack_pro will try to get info about branch name and commit hash from [supported CI](#supported-ci-providers) (CI providers have branch, commit, project directory stored as environment variables). In case when you use other CI provider like Jenkins then please set below variables on your own.
|
363
|
-
|
364
|
-
`KNAPSACK_PRO_BRANCH` - It's branch name. You run tests on this branch.
|
365
|
-
|
366
|
-
`KNAPSACK_PRO_COMMIT_HASH` - Commit hash. You run tests for this commit.
|
367
|
-
|
368
|
-
You can also use git as repository adapter to determine branch and commit hash, please see below section.
|
369
|
-
|
370
|
-
#### When should you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (when CI provider is not supported and you use git)
|
371
|
-
|
372
|
-
`KNAPSACK_PRO_REPOSITORY_ADAPTER` - When it has the value `git`, your local version of git on CI server will be used to get the branch name and commit hash. You also need to set `KNAPSACK_PRO_PROJECT_DIR` with the project directory path.
|
373
|
-
|
374
|
-
`KNAPSACK_PRO_PROJECT_DIR` - Path to the project on the CI node, for instance `/home/ubuntu/my-app-repository`. It should be the top-level directory of your repository.
|
375
|
-
|
376
|
-
#### When you don't use git
|
377
|
-
|
378
|
-
If your CI provider does not expose commit hash and branch name through environment variables, then `knapsack_pro` gem does not know these values.
|
379
|
-
You can manually set the values of the current commit hash and branch name in the environment variables:
|
380
|
-
|
381
|
-
* `KNAPSACK_PRO_COMMIT_HASH` - commit hash.
|
382
|
-
* `KNAPSACK_PRO_BRANCH` - branch name.
|
383
|
-
|
384
|
-
## Queue Mode
|
385
|
-
|
386
|
-
knapsack_pro has a built-in queue mode designed to determine the optimal test suite split even when there is an unpredictably longer time execution of test files on one node (e.g. by
|
387
|
-
CI node overload and decrease of performance that may affect how long the tests take on that node, or
|
388
|
-
things like external requests done in individual tests).
|
389
|
-
|
390
|
-
### How does queue mode work?
|
391
|
-
|
392
|
-
On the Knapsack Pro API side, there is test file queue generated for your CI build. Each CI node periodically requests the Knapsack Pro API for test files
|
393
|
-
that should be executed next. Thanks to that each CI node will finish tests at the same time.
|
394
|
-
|
395
|
-
See how it works and what problems can be solved with Queue Mode https://youtu.be/hUEB1XDKEFY
|
54
|
+
The [Installation Guide](http://docs.knapsackpro.com/knapsack_pro-ruby/guide/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=installation_guide) will ask you a few questions and generate instruction steps for your project:
|
396
55
|
|
397
|
-
|
56
|
+
<div align="center">
|
57
|
+
<a href="http://docs.knapsackpro.com/knapsack_pro-ruby/guide/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=installation_guide">
|
58
|
+
<img alt="Install button" src="./.github/assets/install-button.png" width="116" height="50" />
|
59
|
+
</a>
|
60
|
+
</div>
|
398
61
|
|
399
|
-
|
62
|
+
## Upgrade
|
400
63
|
|
401
|
-
|
402
|
-
Thanks to that your first CI build run in Queue Mode will use timing data recorded with Regular Mode to run tests in Queue Mode faster for the very first run.
|
403
|
-
|
404
|
-
Use this command to run Queue Mode:
|
405
|
-
|
406
|
-
```bash
|
407
|
-
# RSpec >= 3.x
|
408
|
-
bundle exec rake knapsack_pro:queue:rspec
|
409
|
-
|
410
|
-
# Minitest
|
411
|
-
bundle exec rake knapsack_pro:queue:minitest
|
412
|
-
|
413
|
-
# Cucumber
|
414
|
-
# If you use spring gem and spring-commands-cucumber gem to start Cucumber tests faster please set
|
415
|
-
# export KNAPSACK_PRO_CUCUMBER_QUEUE_PREFIX=bundle exec spring
|
416
|
-
# or you can use spring binstub
|
417
|
-
# export KNAPSACK_PRO_CUCUMBER_QUEUE_PREFIX=bin/spring
|
418
|
-
# Thanks to that Cucumber will start tests faster for each batch of tests fetched from Knapsack Pro Queue API
|
419
|
-
bundle exec rake knapsack_pro:queue:cucumber
|
420
|
-
```
|
421
|
-
|
422
|
-
If the above command fails for RSpec then you may need to explicitly pass an argument to require the `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files:
|
423
|
-
|
424
|
-
```bash
|
425
|
-
bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"
|
426
|
-
```
|
427
|
-
|
428
|
-
Note: when you run Queue Mode command for the first time without recording tests first in Regular Mode then CI build might be slower (especially for Cucumber).
|
429
|
-
The second CI build should have optimal test suite split with faster tests distribution across CI nodes in Queue Mode.
|
430
|
-
|
431
|
-
__Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes.__
|
432
|
-
|
433
|
-
If you use the capybara-screenshot gem then please [follow this step](https://knapsackpro.com/faq/question/how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec).
|
434
|
-
|
435
|
-
If you use the rspec_junit_formatter gem then please [follow this step](https://knapsackpro.com/faq/question/how-to-use-junit-formatter#how-to-use-junit-formatter-with-knapsack_pro-queue-mode).
|
436
|
-
|
437
|
-
If your test suite is very long and the RSpec output is too long for your CI node then you can set log level `KNAPSACK_PRO_LOG_LEVEL=info` to don't show debug messages in RSpec output. [Read more about log level](https://knapsackpro.com/faq/question/how-can-i-change-log-level).
|
438
|
-
|
439
|
-
### Additional info about queue mode
|
440
|
-
|
441
|
-
* You should use a separate API token for queue mode than for regular mode to avoid problems with test suite split (especially in case you would like to go back to regular mode).
|
442
|
-
There might be some cached test suite splits for git commits you have run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_split-test-suite-split-based-on-seed).
|
443
|
-
|
444
|
-
* If you are not using one of the [supported CI providers](#supported-ci-providers) then please note that the knapsack_pro gem doesn't have a CI build ID in order to generate a queue for each particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit.
|
445
|
-
|
446
|
-
To avoid this you should specify a unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`.
|
447
|
-
|
448
|
-
* Note that in the Queue Mode by default you cannot retry the failed CI node with exactly the same subset of tests that were run on the CI node in the first place. It's possible in regular mode ([read more](#knapsack_pro_fixed_test_suite_split-test-suite-split-based-on-seed)). If you want to have similar behavior in Queue Mode you need to explicitly [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
|
449
|
-
|
450
|
-
By default the Queue Mode works this way:
|
451
|
-
|
452
|
-
* If you retry the failed build and your all CI nodes start this new build then there will be a new dynamic test suite split across CI nodes. The reason is that the most of the CI providers schedule a new CI build with a different ID when you retry CI build. They retry all CI nodes again. In that case you don't have to worry with below edge cases because the CI build ID will be different so a new queue will be initialized on Knapsack Pro API side and all retried CI node will connect to that queue.
|
453
|
-
|
454
|
-
Edge cases:
|
455
|
-
|
456
|
-
* Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes are still running. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The retried CI node will be connected to the queue consumed by still running CI nodes. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
|
457
|
-
|
458
|
-
* Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes already finished work. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The fact is all CI nodes finished work so the queue was consumed.
|
459
|
-
* If you retry CI node in first hour since the CI build started for the first time then the retried CI node won't execute tests because the queue was consumed. There is important reason why it works like that. For instance some CI providers like Buildkite allows to start CI node later than the others so sometimes the particular CI node may start work while all other CI nodes finished work. In that case we don't want to run tests on the CI node because queue was already consumed. We don't know whether the CI node is part of the build or it is retried CI node hence the 1 hour lock on initializing a new queue.
|
460
|
-
* If you retry CI node after 1 hour since the CI build started for the first time then the retried CI node will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
|
461
|
-
|
462
|
-
* When you use unsupported CI provider by knapsack_pro gem or you forget to set unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` per CI build then:
|
463
|
-
* when you retry single CI node then it will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run.
|
464
|
-
* when you retry all CI nodes then a new queue will be initialized and all CI nodes will connect to it.
|
465
|
-
|
466
|
-
### Extra configuration for Queue Mode
|
467
|
-
|
468
|
-
#### KNAPSACK_PRO_FIXED_QUEUE_SPLIT (remember queue split on retry CI node)
|
469
|
-
|
470
|
-
* `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=false` (default)
|
471
|
-
|
472
|
-
By default, the fixed queue split is off. It means when you will run tests for the same commit hash and a total number of nodes and for the same branch, and the CI build ID is different with second tests run then the queue will be generated dynamically and CI nodes will fetch from Knapsack Pro API the test files in a dynamic way. This is default because it gives the optimal test suite split for the whole test build across all CI nodes.
|
473
|
-
|
474
|
-
* `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`
|
475
|
-
|
476
|
-
You can enable fixed queue split in order to remember the test suite split across CI nodes when you used Queue Mode.
|
477
|
-
|
478
|
-
It means when you run test suite or just retry single CI node again for the same commit hash and a total number of nodes and for the same branch
|
479
|
-
then you will get exactly the same test suite split as it was when you run the build for the first time.
|
480
|
-
|
481
|
-
Thanks to that when tests on one of your node failed you can retry the node with exactly the same subset of tests that were run on the node in the first place.
|
482
|
-
|
483
|
-
__IMPORTANT__: [Required CI configuration if you use retry single failed CI node feature on your CI server when KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true (in Queue Mode) or KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true (in Regular Mode)](#required-ci-configuration-if-you-use-retry-single-failed-ci-node-feature-on-your-ci-server-when-knapsack_pro_fixed_queue_splittrue-in-queue-mode-or-knapsack_pro_fixed_test_suite_splittrue-in-regular-mode)
|
484
|
-
|
485
|
-
Other useful info:
|
486
|
-
|
487
|
-
* Note when fixed queue split is enabled then you can run tests in a dynamic way only once for particular commit hash and a total number of nodes and for the same branch.
|
488
|
-
|
489
|
-
* When Knapsack Pro API server has already information about previous queue split then the information will be used. You will see at the beginning of the knapsack command the log with info that queue name is nil because it was not generated this time. You will get the list of all test files that were executed on the particular CI node in the past.
|
490
|
-
|
491
|
-
```
|
492
|
-
[knapsack_pro] {"queue_name"=>nil, "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]}
|
493
|
-
```
|
494
|
-
|
495
|
-
* Knapsack Pro is fault-tolerant and can withstand possible CI instance preemptions (shut down) when you use highly affordable CI nodes like [Google Cloud Preemptible VMs](https://cloud.google.com/preemptible-vms/) or [Amazon EC2 Spot Instances](https://aws.amazon.com/ec2/spot/). When you retry failed CI node or when your CI provider will do auto retry then the knapsack_pro will run tests previosly served to CI node that failed. After that it will try to consume the test files from the Queue if there are remaining test files that were not yet executed. You will see in the logs info that you retry the tests if the `queue_name` has prefix `retry-dead-ci-node`:
|
496
|
-
|
497
|
-
```
|
498
|
-
[knapsack_pro] {"queue_name"=>"retry-dead-ci-node:queue-id", "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]}
|
499
|
-
```
|
500
|
-
|
501
|
-
* To [reproduce tests executed on CI node](https://knapsackpro.com/faq/question/how-to-run-tests-for-particular-ci-node-in-your-development-environment#for-knapsack_pro-queue-mode) in development environment please see FAQ.
|
502
|
-
|
503
|
-
#### KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)
|
504
|
-
|
505
|
-
* `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=true` (default)
|
506
|
-
|
507
|
-
By default, the knapsack_pro will monkey patch [RSpec Formatters](https://www.relishapp.com/rspec/rspec-core/v/2-6/docs/command-line/format-option) in order to
|
508
|
-
hide the summary of pending and failed tests after each intermediate run of tests fetched from the work queue on Knapsack Pro API.
|
509
|
-
knapsack_pro shows summary of all pending and failed tests at the very end when work queue ended. If you use your custom formatter and you have problem with it then you can disable `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false` monkey patching.
|
510
|
-
|
511
|
-
* `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false`
|
512
|
-
|
513
|
-
It causes to show summary of pending and failed tests after each intermediate tests run from the work queue. The summary will grown cumulatively after each intermediate tests run so it means you will see multiple times summary of the same pending/failed tests. It doesn't mean the test files are executed twice. Test files are executed only once. Only summary report grows cumulatively.
|
514
|
-
|
515
|
-
### Supported test runners in queue mode
|
516
|
-
|
517
|
-
At this moment the queue mode works for:
|
518
|
-
|
519
|
-
* RSpec
|
520
|
-
* Minitest
|
521
|
-
* Cucumber
|
522
|
-
|
523
|
-
## Split test files by test cases
|
524
|
-
|
525
|
-
__How it works__: You can split slow test file by test cases. Thanks to that the slow test file can be split across parallel CI nodes because test cases from the test file will run on different CI nodes.
|
526
|
-
|
527
|
-
This is helpful when you have one or a few very slow test files that are a bottleneck for CI build speed and you don't want to manually create a few smaller test files from the slow test files. Instead, you can tell `knapsack_pro` gem to split your slow test files by test cases across parallel CI nodes.
|
528
|
-
|
529
|
-
Knapsack Pro API provides recorded timing of test files from your previously recorded CI builds and `knapsack_pro` gem will use this suggestion to determine slow test files. `knapsack_pro` gem splits only slow test files by test cases. Test files that are fast won't be split by test cases because it is not needed.
|
530
|
-
|
531
|
-
> __Note:__ This feature works for below test runners in Knapsack Pro Regular Mode and Queue Mode.
|
532
|
-
|
533
|
-
### RSpec split test files by test examples (by individual `it`s)
|
534
|
-
|
535
|
-
Read more about [this feature](https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it) and [common problems here](https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it#common-problems).
|
536
|
-
|
537
|
-
> ❗ __RSpec requirement:__ You need `RSpec >= 3.3.0` in order to use this feature.
|
538
|
-
|
539
|
-
In order to split RSpec slow test files by test examples across parallel CI nodes you need to set environment variable:
|
540
|
-
|
541
|
-
```
|
542
|
-
KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true
|
543
|
-
```
|
544
|
-
|
545
|
-
Thanks to that your CI build speed can be faster. We recommend using this feature with [Queue Mode](https://youtu.be/hUEB1XDKEFY) to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.
|
546
|
-
|
547
|
-
### How to manually define a list of slow test files to be split by test cases
|
548
|
-
|
549
|
-
If you don't want to rely on a list of test files from Knapsack Pro API to determine slow test files that should be split by test cases then you can define your own list of slow test files.
|
550
|
-
|
551
|
-
```
|
552
|
-
# enable split by test cases for RSpec
|
553
|
-
KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true
|
554
|
-
|
555
|
-
# example slow test files pattern for RSpec
|
556
|
-
KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN="{spec/models/user_spec.rb,spec/controllers/**/*_spec.rb}"
|
557
|
-
```
|
558
|
-
|
559
|
-
`KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` must be subset of `KNAPSACK_PRO_TEST_FILE_PATTERN` (example default pattern for RSpec is `KNAPSACK_PRO_TEST_FILE_PATTERN="spec/**{,/*/**}/*_spec.rb"`).
|
560
|
-
|
561
|
-
> __Warning:__ `KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` pattern is mostly useful for debugging purposes by developers of `knapsack_pro` gem. If you want to use it then __it is recommended to provide a shortlist of slow test files__ with the pattern.
|
562
|
-
>
|
563
|
-
> If you use a too broad list of slow test files then you may end up slowing your test suite, especially for RSpec it may result in a slow generating list of test examples in your project. The long list of test file example paths won't be accepted by Knapsack Pro API due to API timeout. CI providers like CircleCI may exceed server memory when running too many RSpec test examples.
|
564
|
-
|
565
|
-
## Extra configuration for CI server
|
566
|
-
|
567
|
-
### Info about ENV variables
|
568
|
-
|
569
|
-
By default knapsack_pro gem [supports a few CI providers](#supported-ci-providers) so you don't need to set some environment variables.
|
570
|
-
In case when you use other CI provider for instance [Jenkins](https://jenkins-ci.org) etc then you need to provide configuration via below environment variables.
|
571
|
-
|
572
|
-
`KNAPSACK_PRO_CI_NODE_TOTAL` - total number CI nodes you have.
|
573
|
-
|
574
|
-
`KNAPSACK_PRO_CI_NODE_INDEX` - index of current CI node starts from 0. Second CI node should have `KNAPSACK_PRO_CI_NODE_INDEX=1`.
|
575
|
-
|
576
|
-
#### KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT (test suite split based on seed)
|
577
|
-
|
578
|
-
Note this is for knapsack_pro regular mode only.
|
579
|
-
|
580
|
-
* `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` (default)
|
581
|
-
|
582
|
-
It means when you run test suite again for the same commit hash and total number of nodes and for the same branch
|
583
|
-
then you will get exactly the same test suite split.
|
584
|
-
|
585
|
-
Thanks to that when tests on one of your node failed you can retry the node with exactly the same subset of tests that were run on the node in the first place.
|
586
|
-
|
587
|
-
__IMPORTANT__: [Required CI configuration if you use retry single failed CI node feature on your CI server when KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true (in Queue Mode) or KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true (in Regular Mode)](#required-ci-configuration-if-you-use-retry-single-failed-ci-node-feature-on-your-ci-server-when-knapsack_pro_fixed_queue_splittrue-in-queue-mode-or-knapsack_pro_fixed_test_suite_splittrue-in-regular-mode)
|
588
|
-
|
589
|
-
Other useful info:
|
590
|
-
|
591
|
-
* There is one edge case. When you run tests for the first time and there is no data collected about time execution of your tests then
|
592
|
-
we need to collect data to prepare the first test suite split. The second run of your tests will have fixed test suite split.
|
593
|
-
|
594
|
-
To compare if all your CI nodes are running based on the same test suite split seed you can check the value for seed in knapsack logging message
|
595
|
-
before your test starts. The message looks like:
|
596
|
-
|
597
|
-
```
|
598
|
-
[knapsack_pro] Test suite split seed: 8a606431-02a1-4766-9878-0ea42a07ad21
|
599
|
-
```
|
600
|
-
|
601
|
-
* `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=false`
|
602
|
-
|
603
|
-
When you disable fixed test suite split then your will get test suite split based on most up to date data about your test suite time execution.
|
604
|
-
For instance, when you run tests for the second time for the same commit hash then your will get more optimal test suite split than it was on the first run.
|
605
|
-
|
606
|
-
Don't disable fixed test suite split when:
|
607
|
-
|
608
|
-
* you expect to run the same subset of test suite multiple times for the same node (for instance your would like to retry only single CI node that failed)
|
609
|
-
|
610
|
-
Example of issue:
|
611
|
-
* https://github.com/KnapsackPro/knapsack_pro-ruby/issues/15
|
612
|
-
* https://github.com/KnapsackPro/knapsack_pro-ruby/issues/12
|
613
|
-
|
614
|
-
* you start your tests not at the same time across your CI nodes. For instance, one of the CI node finished faster than the other CI node started. This would change the seed for the second CI node that started later.
|
615
|
-
|
616
|
-
#### Environment variables for debugging gem
|
617
|
-
|
618
|
-
This is only for maintainer of knapsack_pro gem. Not for the end users.
|
619
|
-
|
620
|
-
* `KNAPSACK_PRO_ENDPOINT` - Default value is `https://api.knapsackpro.com` which is endpoint for [Knapsack Pro API](http://docs.knapsackpro.com).
|
621
|
-
|
622
|
-
* `KNAPSACK_PRO_MODE` - Default value is `production` and then endpoint is `https://api.knapsackpro.com`.
|
623
|
-
* When mode is `development` then endpoint is `http://api.knapsackpro.test:3000`.
|
624
|
-
* When mode is `test` then endpoint is `https://api-staging.knapsackpro.com`.
|
625
|
-
|
626
|
-
### Required CI configuration if you use retry single failed CI node feature on your CI server when KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true (in Queue Mode) or KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true (in Regular Mode)
|
627
|
-
|
628
|
-
Read below required configuration step if you use Queue Mode and you set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node) or you use Regular Mode which has by default [`KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true`](#knapsack_pro_fixed_test_suite_split-test-suite-split-based-on-seed).
|
629
|
-
|
630
|
-
* __IMPORTANT:__ If you use __the feature to retry only a single failed CI node__ on your CI server (for instance you use Buildkite and you use [auto-retry](https://buildkite.com/docs/pipelines/command-step#retry-attributes) for the failed job) then you need to be aware of [a race condition that could happen](https://github.com/KnapsackPro/knapsack_pro-ruby/pull/100). knapsack_pro should not allow running tests in Fallback Mode in the case when the failed CI node was retried to prevent running the wrong set of tests.
|
631
|
-
|
632
|
-
knapsack_pro has built-in support for retries of failed parallel CI nodes for listed CI servers:
|
633
|
-
|
634
|
-
* Buildkite (knapsack_pro reads `BUILDKITE_RETRY_COUNT`)
|
635
|
-
|
636
|
-
knapsack_pro reads ENV vars for above CI servers and it disables Fallback Mode when failed parallel CI node can't connect with Knapsack Pro API. This way we prevent running the wrong set of tests by Fallback Mode on retried CI node.
|
637
|
-
|
638
|
-
If you use other CI server you need to manually configure your CI server to set `KNAPSACK_PRO_CI_NODE_RETRY_COUNT=1` only during retry CI node attempt. If `KNAPSACK_PRO_CI_NODE_RETRY_COUNT > 0` then knapsack_pro won't allow starting running tests in Fallback Mode and instead will raise error so a user can manually retry CI node later when a connection to Knapsack Pro API can be established.
|
639
|
-
|
640
|
-
If you cannot set `KNAPSACK_PRO_CI_NODE_RETRY_COUNT` only for retried CI node or it is not possible for your CI server then you can disable Fallback Mode completely `KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false`. When Fallback Mode is disabled then knapsack_pro gem will try to connect to Knapsack Pro API 6 times instead of only 3 times to ensure there is a low chance of failing your CI node due to lost connection with the API.
|
641
|
-
|
642
|
-
### Passing arguments to rake task
|
643
|
-
|
644
|
-
#### Passing arguments to rspec
|
645
|
-
|
646
|
-
Knapsack Pro allows you to pass arguments through to rspec. For example if you want to run only specs that have the tag `focus`. If you do this with rspec directly it would look like:
|
647
|
-
|
648
|
-
```bash
|
649
|
-
bundle exec rake rspec --tag focus
|
650
|
-
```
|
651
|
-
|
652
|
-
To do this with Knapsack Pro you simply add your rspec arguments as parameters to the knapsack_pro rake task.
|
653
|
-
|
654
|
-
```bash
|
655
|
-
bundle exec rake "knapsack_pro:rspec[--tag focus]"
|
656
|
-
```
|
657
|
-
|
658
|
-
#### Passing arguments to cucumber
|
659
|
-
|
660
|
-
Add arguments to knapsack_pro cucumber task like this:
|
64
|
+
Knapsack Pro follows semantic versioning, but make sure to check the [changelog](CHANGELOG.md) before updating gem with:
|
661
65
|
|
662
66
|
```bash
|
663
|
-
bundle
|
67
|
+
bundle update knapsack_pro
|
664
68
|
```
|
665
69
|
|
666
|
-
|
667
|
-
|
668
|
-
Add arguments to knapsack_pro minitest task like this:
|
70
|
+
## Contributing
|
669
71
|
|
670
|
-
|
671
|
-
bundle exec rake "knapsack_pro:minitest[--arg_name value]"
|
672
|
-
```
|
72
|
+
### Testing
|
673
73
|
|
674
|
-
|
74
|
+
RSpec:
|
675
75
|
|
676
76
|
```bash
|
677
|
-
bundle exec
|
77
|
+
bundle exec rspec spec
|
678
78
|
```
|
679
79
|
|
680
|
-
|
80
|
+
Scripted tests can be found in the [Rails App With Knapsack Pro repository](https://github.com/KnapsackPro/rails-app-with-knapsack_pro/blob/master/bin/knapsack_pro_all.rb).
|
681
81
|
|
682
|
-
|
683
|
-
|
684
|
-
```bash
|
685
|
-
bundle exec rake "knapsack_pro:test_unit[--arg_name value]"
|
686
|
-
```
|
82
|
+
### Publishing
|
687
83
|
|
688
|
-
|
84
|
+
Update the version in `lib/knapsack_pro/version.rb` and `CHANGELOG.md`:
|
689
85
|
|
690
86
|
```bash
|
691
|
-
|
87
|
+
git commit -m "Bump version X.X.X"
|
88
|
+
git push origin master
|
692
89
|
```
|
693
90
|
|
694
|
-
|
695
|
-
|
696
|
-
Add arguments to knapsack_pro spinach task like this:
|
91
|
+
Create a git tag for the release:
|
697
92
|
|
698
93
|
```bash
|
699
|
-
|
94
|
+
git tag -a vX.X.X -m "Release vX.X.X"
|
95
|
+
git push --tags
|
700
96
|
```
|
701
97
|
|
702
|
-
|
703
|
-
|
704
|
-
You can install knapsack_pro globally and use binary. For instance:
|
98
|
+
Build the gem and publish it to RubyGems:
|
705
99
|
|
706
100
|
```bash
|
707
|
-
|
708
|
-
|
709
|
-
knapsack_pro cucumber "--name feature"
|
710
|
-
knapsack_pro queue:cucumber "--name feature"
|
711
|
-
knapsack_pro minitest "--verbose --pride"
|
712
|
-
knapsack_pro queue:minitest "--verbose"
|
713
|
-
knapsack_pro test_unit "--verbose"
|
714
|
-
knapsack_pro spinach "--arg_name value"
|
715
|
-
```
|
716
|
-
|
717
|
-
This is optional way of using knapsack_pro when you don't want to add it to `Gemfile`.
|
718
|
-
|
719
|
-
### Test file names encryption
|
720
|
-
|
721
|
-
The knapsack_pro gem collects information about your test file names, branch names, and execution times. The data are stored on the KnapsackPro.com server.
|
722
|
-
If your test file names or branch names are considered sensitive data, then you can encrypt the data before sending it to the KnapsackPro.com API.
|
723
|
-
|
724
|
-
Encryption is disabled by default because the knapsack_pro gem uses your test file names to prepare a better test suite split when the execution time data are not collected on the KnapsackPro.com server yet.
|
725
|
-
When you enable the encryption, then your first test suite split may not be optimal.
|
726
|
-
|
727
|
-
Each test file name is generated with the `Digest::SHA2.hexdigest` method and 64 chars salt.
|
728
|
-
|
729
|
-
Before you enable encryption, please ensure you are using a fresh API token. Do not use the same API token for the encrypted and non-encrypted test suite. You can generate an API token for your test suite in the [user dashboard](https://knapsackpro.com/dashboard).
|
730
|
-
|
731
|
-
The next step is to generate a salt to encrypt test files or branch names with it.
|
732
|
-
|
733
|
-
```bash
|
734
|
-
bundle exec rake knapsack_pro:salt
|
735
|
-
```
|
736
|
-
|
737
|
-
Add the salt to your CI server as the `KNAPSACK_PRO_SALT` environment variable.
|
738
|
-
|
739
|
-
#### How to enable test file names encryption?
|
740
|
-
|
741
|
-
You need to add the `KNAPSACK_PRO_TEST_FILES_ENCRYPTED=true` environment variable to your CI server.
|
742
|
-
|
743
|
-
#### How to debug test file names?
|
744
|
-
|
745
|
-
If you need to check what is the hash value for a particular test file you can check that with the rake task:
|
746
|
-
|
747
|
-
```bash
|
748
|
-
KNAPSACK_PRO_SALT=xxx bundle exec rake "knapsack_pro:encrypted_test_file_names[rspec]"
|
749
|
-
```
|
750
|
-
|
751
|
-
You can pass the name of a test runner like `rspec`, `minitest`, `test_unit`, `cucumber`, `spinach` as an argument to the rake task.
|
752
|
-
|
753
|
-
#### How to enable branch names encryption?
|
754
|
-
|
755
|
-
You need to add the `KNAPSACK_PRO_BRANCH_ENCRYPTED=true` environment variable to your CI server.
|
756
|
-
|
757
|
-
Note a few branch names won't be encrypted because we use them as fallback branches on the Knapsack Pro API side to determine time execution for test files during split for newly created branches.
|
758
|
-
|
759
|
-
* develop
|
760
|
-
* development
|
761
|
-
* dev
|
762
|
-
* master
|
763
|
-
* staging
|
764
|
-
* [see the full list of excluded branch names from encryption](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/crypto/branch_encryptor.rb#L4)
|
765
|
-
|
766
|
-
#### How to debug branch names?
|
767
|
-
|
768
|
-
If you need to check what is the hash for a particular branch, then use the rake task:
|
769
|
-
|
770
|
-
```bash
|
771
|
-
# show all local branches and respective hashes
|
772
|
-
$ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_branch_names
|
773
|
-
|
774
|
-
# show the hash for a branch name passed as an argument to the rake task
|
775
|
-
$ KNAPSACK_PRO_SALT=xxx bundle exec rake "knapsack_pro:encrypted_branch_names[not-encrypted-branch-name]"
|
776
|
-
```
|
777
|
-
|
778
|
-
### Supported CI providers
|
779
|
-
|
780
|
-
#### Info for CircleCI users
|
781
|
-
|
782
|
-
If you are using circleci.com you can omit `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX`. Knapsack Pro will use `CIRCLE_NODE_TOTAL` and `CIRCLE_NODE_INDEX` provided by CircleCI.
|
783
|
-
|
784
|
-
Here is an example for test configuration in your `.circleci/config.yml` file.
|
785
|
-
|
786
|
-
```yaml
|
787
|
-
# CircleCI 2.0
|
788
|
-
|
789
|
-
# some tests that are not balanced and executed only on first CI node
|
790
|
-
- run: case $CIRCLE_NODE_INDEX in 0) npm test ;; esac
|
791
|
-
|
792
|
-
# auto-balancing CI build time execution to be flat and optimal (as fast as possible).
|
793
|
-
# Queue Mode does dynamic tests allocation so the previous not balanced run command won't
|
794
|
-
# create a bottleneck on the CI node
|
795
|
-
- run:
|
796
|
-
name: RSpec via knapsack_pro Queue Mode
|
797
|
-
command: |
|
798
|
-
# export word is important here!
|
799
|
-
export RAILS_ENV=test
|
800
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation]"
|
801
|
-
|
802
|
-
- run:
|
803
|
-
name: Minitest via knapsack_pro Queue Mode
|
804
|
-
command: |
|
805
|
-
# export word is important here!
|
806
|
-
export RAILS_ENV=test
|
807
|
-
bundle exec rake "knapsack_pro:queue:minitest[--verbose]"
|
808
|
-
|
809
|
-
- run:
|
810
|
-
name: Cucumber via knapsack_pro Queue Mode
|
811
|
-
command: |
|
812
|
-
# export word is important here!
|
813
|
-
export RAILS_ENV=test
|
814
|
-
bundle exec rake knapsack_pro:queue:cucumber
|
815
|
-
```
|
816
|
-
|
817
|
-
Please remember to add additional containers for your project in CircleCI settings.
|
818
|
-
|
819
|
-
##### CircleCI and knapsack_pro Queue Mode
|
820
|
-
|
821
|
-
If you use knapsack_pro Queue Mode with CircleCI you may want to [collect metadata](https://circleci.com/docs/2.0/collect-test-data/#metadata-collection-in-custom-test-steps) like junit xml report about your RSpec test suite.
|
822
|
-
|
823
|
-
Here you can read how to configure [junit formatter](https://knapsackpro.com/faq/question/how-to-use-junit-formatter#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). Step for CircleCI is to copy the xml report to `$CIRCLE_TEST_REPORTS` directory. Below is full config for your `spec_helper.rb`:
|
824
|
-
|
825
|
-
```ruby
|
826
|
-
# spec_helper.rb or rails_helper.rb
|
827
|
-
|
828
|
-
# TODO This must be the same path as value for rspec --out argument
|
829
|
-
# Note the path should not contain ~ char, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead.
|
830
|
-
TMP_RSPEC_XML_REPORT = 'tmp/rspec.xml'
|
831
|
-
# move results to FINAL_RSPEC_XML_REPORT so the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT
|
832
|
-
FINAL_RSPEC_XML_REPORT = 'tmp/rspec_final_results.xml'
|
833
|
-
|
834
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
835
|
-
if File.exist?(TMP_RSPEC_XML_REPORT)
|
836
|
-
FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT)
|
837
|
-
end
|
838
|
-
end
|
839
|
-
|
840
|
-
# Here is additional configuration to ensure the xml report will be visible by CircleCI
|
841
|
-
KnapsackPro::Hooks::Queue.after_queue do |queue_id|
|
842
|
-
# Metadata collection
|
843
|
-
# https://circleci.com/docs/2.0/collect-test-data/#metadata-collection-in-custom-test-steps
|
844
|
-
if File.exist?(FINAL_RSPEC_XML_REPORT) && ENV['CIRCLE_TEST_REPORTS']
|
845
|
-
FileUtils.cp(FINAL_RSPEC_XML_REPORT, "#{ENV['CIRCLE_TEST_REPORTS']}/rspec.xml")
|
846
|
-
end
|
847
|
-
end
|
848
|
-
```
|
849
|
-
|
850
|
-
Ensure you have in CircleCI config yml
|
851
|
-
|
852
|
-
```yaml
|
853
|
-
- run:
|
854
|
-
name: RSpec via knapsack_pro Queue Mode
|
855
|
-
command: |
|
856
|
-
export CIRCLE_TEST_REPORTS=/tmp/test-results
|
857
|
-
mkdir $CIRCLE_TEST_REPORTS
|
858
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"
|
859
|
-
|
860
|
-
# collect reports
|
861
|
-
- store_test_results:
|
862
|
-
path: /tmp/test-results
|
863
|
-
- store_artifacts:
|
864
|
-
path: /tmp/test-results
|
865
|
-
destination: test-results
|
866
|
-
```
|
867
|
-
|
868
|
-
You can find a complete CircleCI YML config example in [the article](https://docs.knapsackpro.com/2021/rspec-testing-parallel-jobs-with-circleci-and-junit-xml-report).
|
869
|
-
|
870
|
-
#### Info for Travis users
|
871
|
-
|
872
|
-
You can parallelize your builds across virtual machines with [travis matrix feature](http://docs.travis-ci.com/user/speeding-up-the-build/#parallelizing-your-builds-across-virtual-machines). Edit `.travis.yml`
|
873
|
-
|
874
|
-
```yaml
|
875
|
-
script:
|
876
|
-
# Step for RSpec in Regular Mode
|
877
|
-
- "bundle exec rake knapsack_pro:rspec"
|
878
|
-
|
879
|
-
# Step for RSpec in Queue Mode
|
880
|
-
- "bundle exec rake knapsack_pro:queue:rspec"
|
881
|
-
|
882
|
-
# Step for Cucumber in Regular Mode
|
883
|
-
- "bundle exec rake knapsack_pro:cucumber"
|
884
|
-
|
885
|
-
# Step for Cucumber in Queue Mode
|
886
|
-
- "bundle exec rake knapsack_pro:queue:cucumber"
|
887
|
-
|
888
|
-
# Step for Minitest in Regular Mode
|
889
|
-
- "bundle exec rake knapsack_pro:minitest"
|
890
|
-
|
891
|
-
# Step for Minitest in Queue Mode
|
892
|
-
- "bundle exec rake knapsack_pro:queue:minitest"
|
893
|
-
|
894
|
-
# Step for test-unit in Regular Mode
|
895
|
-
- "bundle exec rake knapsack_pro:test_unit"
|
896
|
-
|
897
|
-
# Step for Spinach in Regular Mode
|
898
|
-
- "bundle exec rake knapsack_pro:spinach"
|
899
|
-
|
900
|
-
env:
|
901
|
-
global:
|
902
|
-
# tokens should be set in travis settings in web interface to avoid expose tokens in build logs
|
903
|
-
- KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token
|
904
|
-
- KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER=cucumber-token
|
905
|
-
- KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=minitest-token
|
906
|
-
- KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token
|
907
|
-
- KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token
|
908
|
-
|
909
|
-
# if you use Knapsack Pro Queue Mode you must set below env variable
|
910
|
-
# to be able to retry single failed parallel job from Travis UI
|
911
|
-
- KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
|
912
|
-
|
913
|
-
- KNAPSACK_PRO_CI_NODE_TOTAL=2
|
914
|
-
jobs:
|
915
|
-
- KNAPSACK_PRO_CI_NODE_INDEX=0
|
916
|
-
- KNAPSACK_PRO_CI_NODE_INDEX=1
|
917
|
-
```
|
918
|
-
|
919
|
-
Such configuration will generate matrix with 2 following ENV rows:
|
920
|
-
|
921
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER=cucumber-token KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=minitest-token KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token
|
922
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER=cucumber-token KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=minitest-token KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token
|
923
|
-
|
924
|
-
More info about global and matrix ENV configuration in [travis docs](https://docs.travis-ci.com/user/customizing-the-build/#build-matrix).
|
925
|
-
|
926
|
-
#### Info for semaphoreci.com users
|
927
|
-
|
928
|
-
##### Semaphore 2.0
|
929
|
-
|
930
|
-
knapsack_pro gem supports environment variables provided by Semaphore CI 2.0 to run your tests. You will have to define a few things in `.semaphore/semaphore.yml` config file.
|
931
|
-
|
932
|
-
* You need to set `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC`. If you don't want to commit secrets in yml file then you can [follow this guide](https://docs.semaphoreci.com/article/66-environment-variables-and-secrets).
|
933
|
-
* You should create as many parallel jobs as you need with `parallelism` property. If your test suite is long you should use more parallel jobs.
|
934
|
-
|
935
|
-
Below you can find full Semaphore CI 2.0 config for Rails project.
|
936
|
-
|
937
|
-
```yaml
|
938
|
-
# .semaphore/semaphore.yml
|
939
|
-
# Use the latest stable version of Semaphore 2.0 YML syntax:
|
940
|
-
version: v1.0
|
941
|
-
|
942
|
-
# Name your pipeline. In case you connect multiple pipelines with promotions,
|
943
|
-
# the name will help you differentiate between, for example, a CI build phase
|
944
|
-
# and delivery phases.
|
945
|
-
name: Demo Rails 5 app
|
946
|
-
|
947
|
-
# An agent defines the environment in which your code runs.
|
948
|
-
# It is a combination of one of available machine types and operating
|
949
|
-
# system images.
|
950
|
-
# See https://docs.semaphoreci.com/article/20-machine-types
|
951
|
-
# and https://docs.semaphoreci.com/article/32-ubuntu-1804-image
|
952
|
-
agent:
|
953
|
-
machine:
|
954
|
-
type: e1-standard-2
|
955
|
-
os_image: ubuntu1804
|
956
|
-
|
957
|
-
# Blocks are the heart of a pipeline and are executed sequentially.
|
958
|
-
# Each block has a task that defines one or more jobs. Jobs define the
|
959
|
-
# commands to execute.
|
960
|
-
# See https://docs.semaphoreci.com/article/62-concepts
|
961
|
-
blocks:
|
962
|
-
- name: Setup
|
963
|
-
task:
|
964
|
-
env_vars:
|
965
|
-
- name: RAILS_ENV
|
966
|
-
value: test
|
967
|
-
jobs:
|
968
|
-
- name: bundle
|
969
|
-
commands:
|
970
|
-
# Checkout code from Git repository. This step is mandatory if the
|
971
|
-
# job is to work with your code.
|
972
|
-
# Optionally you may use --use-cache flag to avoid roundtrip to
|
973
|
-
# remote repository.
|
974
|
-
# See https://docs.semaphoreci.com/article/54-toolbox-reference#libcheckout
|
975
|
-
- checkout
|
976
|
-
# Restore dependencies from cache.
|
977
|
-
# Read about caching: https://docs.semaphoreci.com/article/54-toolbox-reference#cache
|
978
|
-
- cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-master-
|
979
|
-
# Set Ruby version:
|
980
|
-
- sem-version ruby 2.6.1
|
981
|
-
- bundle install --jobs=4 --retry=3 --path vendor/bundle
|
982
|
-
# Store the latest version of dependencies in cache,
|
983
|
-
# to be used in next blocks and future workflows:
|
984
|
-
- cache store gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock) vendor/bundle
|
985
|
-
|
986
|
-
- name: RSpec tests
|
987
|
-
task:
|
988
|
-
env_vars:
|
989
|
-
- name: RAILS_ENV
|
990
|
-
value: test
|
991
|
-
- name: PGHOST
|
992
|
-
value: 127.0.0.1
|
993
|
-
- name: PGUSER
|
994
|
-
value: postgres
|
995
|
-
- name: KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC
|
996
|
-
value: your_api_token_here
|
997
|
-
# This block runs two jobs in parallel and they both share common
|
998
|
-
# setup steps. We can group them in a prologue.
|
999
|
-
# See https://docs.semaphoreci.com/article/50-pipeline-yaml#prologue
|
1000
|
-
prologue:
|
1001
|
-
commands:
|
1002
|
-
- checkout
|
1003
|
-
- cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-master-
|
1004
|
-
# Start Postgres database service.
|
1005
|
-
# See https://docs.semaphoreci.com/article/54-toolbox-reference#sem-service
|
1006
|
-
- sem-service start postgres
|
1007
|
-
- sem-version ruby 2.6.1
|
1008
|
-
- bundle install --jobs=4 --retry=3 --path vendor/bundle
|
1009
|
-
- bundle exec rake db:setup
|
1010
|
-
|
1011
|
-
jobs:
|
1012
|
-
- name: Run tests with Knapsack Pro
|
1013
|
-
parallelism: 2
|
1014
|
-
commands:
|
1015
|
-
# Step for RSpec in Queue Mode
|
1016
|
-
- bundle exec rake knapsack_pro:queue:rspec
|
1017
|
-
# Step for Cucumber in Queue Mode
|
1018
|
-
- bundle exec rake knapsack_pro:queue:cucumber
|
1019
|
-
|
1020
|
-
# Step for RSpec in Regular Mode
|
1021
|
-
- bundle exec rake knapsack_pro:rspec
|
1022
|
-
# Step for Cucumber in Regular Mode
|
1023
|
-
- bundle exec rake knapsack_pro:cucumber
|
1024
|
-
# Step for Minitest in Regular Mode
|
1025
|
-
- bundle exec rake knapsack_pro:minitest
|
1026
|
-
# Step for test-unit in Regular Mode
|
1027
|
-
- bundle exec rake knapsack_pro:test_unit
|
1028
|
-
# Step for Spinach in Regular Mode
|
1029
|
-
- bundle exec rake knapsack_pro:spinach
|
1030
|
-
```
|
1031
|
-
|
1032
|
-
##### Semaphore 1.0
|
1033
|
-
|
1034
|
-
Knapsack Pro supports semaphoreapp ENVs `SEMAPHORE_THREAD_COUNT` and `SEMAPHORE_CURRENT_THREAD`. The only thing you need to do is set up knapsack_pro rspec/cucumber/minitest/test_unit command for as many threads as you need. Here is an example:
|
1035
|
-
|
1036
|
-
```bash
|
1037
|
-
# Thread 1
|
1038
|
-
## Step for RSpec
|
1039
|
-
bundle exec rake knapsack_pro:rspec
|
1040
|
-
## Step for Cucumber
|
1041
|
-
bundle exec rake knapsack_pro:cucumber
|
1042
|
-
## Step for Minitest
|
1043
|
-
bundle exec rake knapsack_pro:minitest
|
1044
|
-
## Step for test-unit
|
1045
|
-
bundle exec rake knapsack_pro:test_unit
|
1046
|
-
## Step for Spinach
|
1047
|
-
bundle exec rake knapsack_pro:spinach
|
1048
|
-
|
1049
|
-
# Thread 2
|
1050
|
-
## Step for RSpec
|
1051
|
-
bundle exec rake knapsack_pro:rspec
|
1052
|
-
## Step for Cucumber
|
1053
|
-
bundle exec rake knapsack_pro:cucumber
|
1054
|
-
## Step for Minitest
|
1055
|
-
bundle exec rake knapsack_pro:minitest
|
1056
|
-
## Step for test-unit
|
1057
|
-
bundle exec rake knapsack_pro:test_unit
|
1058
|
-
## Step for Spinach
|
1059
|
-
bundle exec rake knapsack_pro:spinach
|
1060
|
-
```
|
1061
|
-
|
1062
|
-
Tests will be split across threads.
|
1063
|
-
|
1064
|
-
Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment.
|
1065
|
-
|
1066
|
-
#### Info for buildkite.com users
|
1067
|
-
|
1068
|
-
Knapsack Pro supports buildkite ENVs `BUILDKITE_PARALLEL_JOB_COUNT` and `BUILDKITE_PARALLEL_JOB`. The only thing you need to do is to configure the parallelism parameter in your build step and run the appropiate command in your build
|
1069
|
-
|
1070
|
-
```bash
|
1071
|
-
# Step for RSpec
|
1072
|
-
bundle exec rake knapsack_pro:rspec
|
1073
|
-
|
1074
|
-
# Step for Cucumber
|
1075
|
-
bundle exec rake knapsack_pro:cucumber
|
1076
|
-
|
1077
|
-
# Step for Minitest
|
1078
|
-
bundle exec rake knapsack_pro:minitest
|
1079
|
-
|
1080
|
-
# Step for test-unit
|
1081
|
-
bundle exec rake knapsack_pro:test_unit
|
1082
|
-
|
1083
|
-
# Step for Spinach
|
1084
|
-
bundle exec rake knapsack_pro:spinach
|
1085
|
-
```
|
1086
|
-
|
1087
|
-
Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment.
|
1088
|
-
|
1089
|
-
Here you can find article [how to set up a new pipeline for your project in Buildkite and configure Knapsack Pro](http://docs.knapsackpro.com/2017/auto-balancing-7-hours-tests-between-100-parallel-jobs-on-ci-buildkite-example) and 2 example repositories for Ruby/Rails projects:
|
1090
|
-
|
1091
|
-
* [Buildkite Rails Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro)
|
1092
|
-
* [Buildkite Rails Docker Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-docker-parallel-example-with-knapsack_pro)
|
1093
|
-
|
1094
|
-
If you want to use Buildkite retry single agent feature to retry just failed tests on particular agent (CI node) then you should set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
|
1095
|
-
|
1096
|
-
When using the `docker-compose` plugin on Buildkite, you have to tell it which environment variables to pass to the docker container. Thanks to it knapsack_pro can detect info about CI build like commit, branch name, amount of parallel nodes.
|
1097
|
-
|
1098
|
-
```yaml
|
1099
|
-
steps:
|
1100
|
-
- label: "Test"
|
1101
|
-
parallelism: 2
|
1102
|
-
plugins:
|
1103
|
-
- docker-compose#3.0.3:
|
1104
|
-
run: app
|
1105
|
-
# use here proper knapsack_pro command for your test runner
|
1106
|
-
command: bundle exec rake knapsack_pro:queue:rspec
|
1107
|
-
config: docker-compose.test.yml
|
1108
|
-
env:
|
1109
|
-
- BUILDKITE_PARALLEL_JOB_COUNT
|
1110
|
-
- BUILDKITE_PARALLEL_JOB
|
1111
|
-
- BUILDKITE_BUILD_NUMBER
|
1112
|
-
- BUILDKITE_COMMIT
|
1113
|
-
- BUILDKITE_BRANCH
|
1114
|
-
- BUILDKITE_BUILD_CHECKOUT_PATH
|
1115
|
-
```
|
1116
|
-
|
1117
|
-
#### Info for GitLab CI users
|
1118
|
-
|
1119
|
-
Remember to add API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to [Secret Variables](https://gitlab.com/help/ci/variables/README.md#secret-variables) in `GitLab CI Settings -> CI/CD Pipelines -> Secret Variables`.
|
1120
|
-
|
1121
|
-
##### GitLab CI `>= 11.5`
|
1122
|
-
|
1123
|
-
```yaml
|
1124
|
-
test:
|
1125
|
-
parallel: 2
|
1126
|
-
|
1127
|
-
# Knapsack Pro Regular Mode (deterministic test suite split)
|
1128
|
-
script: bundle exec rake knapsack_pro:rspec
|
1129
|
-
|
1130
|
-
# Other commands you could use:
|
1131
|
-
|
1132
|
-
# Knapsack Pro Regular Mode (deterministic test suite split)
|
1133
|
-
# bundle exec rake knapsack_pro:cucumber
|
1134
|
-
# bundle exec rake knapsack_pro:minitest
|
1135
|
-
# bundle exec rake knapsack_pro:test_unit
|
1136
|
-
# bundle exec rake knapsack_pro:spinach
|
1137
|
-
|
1138
|
-
# Knapsack Pro Queue Mode (dynamic test suite split)
|
1139
|
-
# bundle exec rake knapsack_pro:queue:rspec
|
1140
|
-
# bundle exec rake knapsack_pro:queue:minitest
|
1141
|
-
# bundle exec rake knapsack_pro:queue:cucumber
|
1142
|
-
```
|
1143
|
-
|
1144
|
-
Here you can find info [how to configure the GitLab parallel CI nodes](https://docs.gitlab.com/ee/ci/yaml/#parallel).
|
1145
|
-
|
1146
|
-
##### GitLab CI `< 11.5` (old GitLab CI)
|
1147
|
-
|
1148
|
-
GitLab CI does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same `test` stage. Below is relevant part of `.gitlab-ci.yml` configuration for 2 parallel jobs.
|
1149
|
-
|
1150
|
-
```yaml
|
1151
|
-
# .gitlab-ci.yml
|
1152
|
-
stages:
|
1153
|
-
- test
|
1154
|
-
|
1155
|
-
variables:
|
1156
|
-
KNAPSACK_PRO_CI_NODE_TOTAL: 2
|
1157
|
-
|
1158
|
-
# first CI node running in parallel
|
1159
|
-
test_ci_node_0:
|
1160
|
-
stage: test
|
1161
|
-
script:
|
1162
|
-
- export KNAPSACK_PRO_CI_NODE_INDEX=0
|
1163
|
-
# Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split)
|
1164
|
-
- bundle exec rake knapsack_pro:cucumber
|
1165
|
-
# or use Cucumber tests in Knapsack Pro Queue Mode (dynamic test suite split)
|
1166
|
-
- bundle exec rake knapsack_pro:queue:cucumber
|
1167
|
-
# RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split)
|
1168
|
-
# It will autobalance build because it is executed after Cucumber tests.
|
1169
|
-
- bundle exec rake knapsack_pro:queue:rspec
|
1170
|
-
|
1171
|
-
# second CI node running in parallel
|
1172
|
-
test_ci_node_1:
|
1173
|
-
stage: test
|
1174
|
-
script:
|
1175
|
-
- export KNAPSACK_PRO_CI_NODE_INDEX=1
|
1176
|
-
- bundle exec rake knapsack_pro:cucumber
|
1177
|
-
- bundle exec rake knapsack_pro:queue:cucumber
|
1178
|
-
- bundle exec rake knapsack_pro:queue:rspec
|
1179
|
-
```
|
1180
|
-
|
1181
|
-
#### Info for codeship.com users
|
1182
|
-
|
1183
|
-
Codeship does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each [parallel test pipeline](https://documentation.codeship.com/basic/builds-and-configuration/parallel-tests/#using-parallel-test-pipelines). Below is an example for 2 parallel test pipelines.
|
1184
|
-
|
1185
|
-
Configure test pipelines (1/2 used)
|
1186
|
-
|
1187
|
-
```bash
|
1188
|
-
# first CI node running in parallel
|
1189
|
-
|
1190
|
-
# Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split)
|
1191
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber
|
1192
|
-
|
1193
|
-
# or use Cucumber tests in Knapsack Pro Queue Mode (dynamic test suite split)
|
1194
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:cucumber
|
1195
|
-
|
1196
|
-
# RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split)
|
1197
|
-
# It will autobalance build because it is executed after Cucumber tests.
|
1198
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec
|
1199
|
-
```
|
1200
|
-
|
1201
|
-
Configure test pipelines (2/2 used)
|
1202
|
-
|
1203
|
-
```bash
|
1204
|
-
# second CI node running in parallel
|
1205
|
-
|
1206
|
-
# Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split)
|
1207
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber
|
1208
|
-
|
1209
|
-
# or use Cucumber tests in Knapsack Pro Queue Mode (dynamic test suite split)
|
1210
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:cucumber
|
1211
|
-
|
1212
|
-
# RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split)
|
1213
|
-
# It will autobalance build because it is executed after Cucumber tests.
|
1214
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec
|
1215
|
-
```
|
1216
|
-
|
1217
|
-
Remember to add API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `Environment` page of your project settings in Codeship.
|
1218
|
-
|
1219
|
-
CodeShip uses the same build number if you restart a build. Because of that you need to set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node) in order to be able to restart CI build in Queue Mode.
|
1220
|
-
|
1221
|
-
#### Info for Heroku CI users
|
1222
|
-
|
1223
|
-
You can parallelize your tests on Heroku CI by configuring `app.json`.
|
1224
|
-
|
1225
|
-
You can set how many parallel dynos with tests you want to run with `quantity` value.
|
1226
|
-
Use `test` key to run knapsack_pro gem.
|
1227
|
-
|
1228
|
-
You need to specify also the environment variable with API token for Knapsack Pro.
|
1229
|
-
For any sensitive environment variables (like Knapsack Pro API token) that you do not want in your `app.json` manifest, you can add them to your pipeline’s Heroku CI settings.
|
1230
|
-
|
1231
|
-
Note the [Heroku CI Parallel Test Runs](https://devcenter.heroku.com/articles/heroku-ci-parallel-test-runs) are in Beta and you may need to ask Heroku support to enabled it for your project.
|
1232
|
-
|
1233
|
-
```json
|
1234
|
-
# app.json
|
1235
|
-
{
|
1236
|
-
"environments": {
|
1237
|
-
"test": {
|
1238
|
-
"formation": {
|
1239
|
-
"test": {
|
1240
|
-
"quantity": 2
|
1241
|
-
}
|
1242
|
-
},
|
1243
|
-
"addons": [
|
1244
|
-
"heroku-postgresql"
|
1245
|
-
],
|
1246
|
-
"scripts": {
|
1247
|
-
"test": "bundle exec rake knapsack_pro:rspec"
|
1248
|
-
},
|
1249
|
-
"env": {
|
1250
|
-
"KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC": "rspec-token"
|
1251
|
-
}
|
1252
|
-
}
|
1253
|
-
}
|
1254
|
-
}
|
1255
|
-
```
|
1256
|
-
|
1257
|
-
You can learn more about [Heroku CI](https://devcenter.heroku.com/articles/heroku-ci).
|
1258
|
-
|
1259
|
-
#### Info for Solano CI users
|
1260
|
-
|
1261
|
-
[Solano CI](https://www.solanolabs.com) does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same CI build.
|
1262
|
-
|
1263
|
-
```bash
|
1264
|
-
# Step for RSpec for first CI node
|
1265
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec
|
1266
|
-
# Step for RSpec for second CI node
|
1267
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:rspec
|
1268
|
-
|
1269
|
-
# Step for Cucumber for first CI node
|
1270
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber
|
1271
|
-
# Step for Cucumber for second CI node
|
1272
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber
|
1273
|
-
|
1274
|
-
# Step for Minitest for first CI node
|
1275
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest
|
1276
|
-
# Step for Minitest for second CI node
|
1277
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:minitest
|
1278
|
-
|
1279
|
-
# Step for test-unit for first CI node
|
1280
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit
|
1281
|
-
# Step for test-unit for second CI node
|
1282
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:test_unit
|
1283
|
-
|
1284
|
-
# Step for Spinach for first CI node
|
1285
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach
|
1286
|
-
# Step for Spinach for second CI node
|
1287
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:spinach
|
1288
|
-
```
|
1289
|
-
|
1290
|
-
Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment.
|
1291
|
-
|
1292
|
-
#### Info for AppVeyor users
|
1293
|
-
|
1294
|
-
[AppVeyor](https://www.appveyor.com) does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same CI build.
|
1295
|
-
|
1296
|
-
```bash
|
1297
|
-
# Step for RSpec for first CI node
|
1298
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec
|
1299
|
-
# Step for RSpec for second CI node
|
1300
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:rspec
|
1301
|
-
|
1302
|
-
# Step for Cucumber for first CI node
|
1303
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber
|
1304
|
-
# Step for Cucumber for second CI node
|
1305
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber
|
1306
|
-
|
1307
|
-
# Step for Minitest for first CI node
|
1308
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest
|
1309
|
-
# Step for Minitest for second CI node
|
1310
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:minitest
|
1311
|
-
|
1312
|
-
# Step for test-unit for first CI node
|
1313
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit
|
1314
|
-
# Step for test-unit for second CI node
|
1315
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:test_unit
|
1316
|
-
|
1317
|
-
# Step for Spinach for first CI node
|
1318
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach
|
1319
|
-
# Step for Spinach for second CI node
|
1320
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:spinach
|
1321
|
-
```
|
1322
|
-
|
1323
|
-
Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment.
|
1324
|
-
|
1325
|
-
#### Info for snap-ci.com users
|
1326
|
-
|
1327
|
-
Knapsack Pro supports snap-ci.com ENVs `SNAP_WORKER_TOTAL` and `SNAP_WORKER_INDEX`. The only thing you need to do is to configure number of workers for your project in configuration settings in order to enable parallelism. Next thing is to set below commands to be executed in your stage:
|
1328
|
-
|
1329
|
-
```bash
|
1330
|
-
# Step for RSpec
|
1331
|
-
bundle exec rake knapsack_pro:rspec
|
1332
|
-
|
1333
|
-
# Step for Cucumber
|
1334
|
-
bundle exec rake knapsack_pro:cucumber
|
1335
|
-
|
1336
|
-
# Step for Minitest
|
1337
|
-
bundle exec rake knapsack_pro:minitest
|
1338
|
-
|
1339
|
-
# Step for test-unit
|
1340
|
-
bundle exec rake knapsack_pro:test_unit
|
1341
|
-
|
1342
|
-
# Step for Spinach
|
1343
|
-
bundle exec rake knapsack_pro:spinach
|
1344
|
-
```
|
1345
|
-
|
1346
|
-
Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment.
|
1347
|
-
|
1348
|
-
#### Info for cirrus-ci.org users
|
1349
|
-
|
1350
|
-
Knapsack Pro supports cirrus-ci.org ENVs `CI_NODE_TOTAL` and `CI_NODE_INDEX`. The only thing you need to do is to configure number of parallel CI nodes for your project. Next thing is to set one of below commands to be executed on each parallel CI node:
|
1351
|
-
|
1352
|
-
```bash
|
1353
|
-
# Step for RSpec
|
1354
|
-
bundle exec rake knapsack_pro:rspec
|
1355
|
-
|
1356
|
-
# Step for Cucumber
|
1357
|
-
bundle exec rake knapsack_pro:cucumber
|
1358
|
-
|
1359
|
-
# Step for Minitest
|
1360
|
-
bundle exec rake knapsack_pro:minitest
|
1361
|
-
|
1362
|
-
# Step for test-unit
|
1363
|
-
bundle exec rake knapsack_pro:test_unit
|
1364
|
-
|
1365
|
-
# Step for Spinach
|
1366
|
-
bundle exec rake knapsack_pro:spinach
|
1367
|
-
```
|
1368
|
-
|
1369
|
-
Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment.
|
1370
|
-
|
1371
|
-
Here is example for [`.cirrus.yml` configuration file](https://cirrus-ci.org/examples/#ruby).
|
1372
|
-
|
1373
|
-
#### Info for Jenkins users
|
1374
|
-
|
1375
|
-
In order to run parallel jobs with Jenkins you should use Jenkins Pipeline.
|
1376
|
-
You can learn basics about it in the article [Parallelism and Distributed Builds with Jenkins](https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins).
|
1377
|
-
|
1378
|
-
Here is example `Jenkinsfile` working with Jenkins Pipeline.
|
1379
|
-
|
1380
|
-
```groovy
|
1381
|
-
timeout(time: 60, unit: 'MINUTES') {
|
1382
|
-
node() {
|
1383
|
-
stage('Checkout') {
|
1384
|
-
checkout([/* checkout code from git */])
|
1385
|
-
|
1386
|
-
// determine git commit hash because we need to pass it to knapsack_pro
|
1387
|
-
COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
|
1388
|
-
|
1389
|
-
stash 'source'
|
1390
|
-
}
|
1391
|
-
}
|
1392
|
-
|
1393
|
-
def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
|
1394
|
-
def nodes = [:]
|
1395
|
-
|
1396
|
-
for (int i = 0; i < num_nodes; i++) {
|
1397
|
-
def index = i;
|
1398
|
-
nodes["ci_node_${i}"] = {
|
1399
|
-
node() {
|
1400
|
-
stage('Setup') {
|
1401
|
-
unstash 'source'
|
1402
|
-
// other setup steps
|
1403
|
-
}
|
1404
|
-
|
1405
|
-
def knapsack_options = """\
|
1406
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
|
1407
|
-
KNAPSACK_PRO_CI_NODE_INDEX=${index}\
|
1408
|
-
KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
|
1409
|
-
KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
|
1410
|
-
"""
|
1411
|
-
|
1412
|
-
// example how to run cucumber tests in Knapsack Pro Regular Mode
|
1413
|
-
stage('Run cucumber') {
|
1414
|
-
sh """${knapsack_options} bundle exec rake knapsack_pro:cucumber"""
|
1415
|
-
}
|
1416
|
-
|
1417
|
-
// example how to run rspec tests in Knapsack Pro Queue Mode
|
1418
|
-
// Queue Mode should be as a last stage so it can autobalance build if tests in regular mode were not perfectly distributed
|
1419
|
-
stage('Run rspec') {
|
1420
|
-
sh """KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG} ${knapsack_options} bundle exec rake knapsack_pro:queue:rspec"""
|
1421
|
-
}
|
1422
|
-
}
|
1423
|
-
}
|
1424
|
-
}
|
1425
|
-
|
1426
|
-
parallel nodes // run CI nodes in parallel
|
1427
|
-
}
|
1428
|
-
```
|
1429
|
-
|
1430
|
-
Remember to set environment variables in Jenkins configuration with your API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER`.
|
1431
|
-
Here is [list of environment variables per test runner](#set-api-key-token).
|
1432
|
-
|
1433
|
-
Above example shows how to run cucumber tests in regular mode and later the rspec tests in queue mode to autobalance build.
|
1434
|
-
If you are going to relay on rspec to autobalance build when cucumber tests were not perfectly distributed you should be aware about [possible edge case if your rspec test suite is very short](https://knapsackpro.com/faq/question/why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again).
|
1435
|
-
|
1436
|
-
#### Info for GitHub Actions users
|
1437
|
-
|
1438
|
-
knapsack_pro gem supports environment variables provided by GitHub Actions to run your tests. You will have to define a few things in `.github/workflows/main.yaml` config file.
|
1439
|
-
|
1440
|
-
* You need to set API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in GitHub settings -> Secrets for your repository. [Creating and using secrets in GitHub Actions](https://help.github.com/en/articles/virtual-environments-for-github-actions#creating-and-using-secrets-encrypted-variables).
|
1441
|
-
* You should create as many parallel jobs as you need with [`matrix` property](https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix). If your test suite is slow you should use more parallel jobs. See comment in below config.
|
1442
|
-
|
1443
|
-
Below you can find full GitHub Actions config for Ruby on Rails project.
|
1444
|
-
|
1445
|
-
```yaml
|
1446
|
-
# .github/workflows/main.yaml
|
1447
|
-
name: Main
|
1448
|
-
|
1449
|
-
on: [push]
|
1450
|
-
|
1451
|
-
jobs:
|
1452
|
-
vm-job:
|
1453
|
-
runs-on: ubuntu-latest
|
1454
|
-
|
1455
|
-
# If you need DB like PostgreSQL then define service below.
|
1456
|
-
# Example for Redis can be found here:
|
1457
|
-
# https://github.com/actions/example-services/tree/master/.github/workflows
|
1458
|
-
services:
|
1459
|
-
postgres:
|
1460
|
-
image: postgres:10.8
|
1461
|
-
env:
|
1462
|
-
POSTGRES_USER: postgres
|
1463
|
-
POSTGRES_PASSWORD: ""
|
1464
|
-
POSTGRES_DB: postgres
|
1465
|
-
ports:
|
1466
|
-
# will assign a random free host port
|
1467
|
-
- 5432/tcp
|
1468
|
-
# needed because the postgres container does not provide a healthcheck
|
1469
|
-
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
|
1470
|
-
|
1471
|
-
strategy:
|
1472
|
-
fail-fast: false
|
1473
|
-
matrix:
|
1474
|
-
# [n] - where the n is a number of parallel jobs you want to run your tests on.
|
1475
|
-
# Use a higher number if you have slow tests to split them between more parallel jobs.
|
1476
|
-
# Remember to update the value of the `ci_node_index` below to (0..n-1).
|
1477
|
-
ci_node_total: [2]
|
1478
|
-
# Indexes for parallel jobs (starting from zero).
|
1479
|
-
# E.g. use [0, 1] for 2 parallel jobs, [0, 1, 2] for 3 parallel jobs, etc.
|
1480
|
-
ci_node_index: [0, 1]
|
1481
|
-
|
1482
|
-
steps:
|
1483
|
-
- uses: actions/checkout@v1
|
1484
|
-
|
1485
|
-
- name: Set up Ruby 2.6
|
1486
|
-
uses: actions/setup-ruby@v1
|
1487
|
-
with:
|
1488
|
-
ruby-version: 2.6.5
|
1489
|
-
|
1490
|
-
- uses: actions/cache@v1
|
1491
|
-
with:
|
1492
|
-
path: vendor/bundle
|
1493
|
-
key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
|
1494
|
-
restore-keys: |
|
1495
|
-
${{ runner.os }}-gems-
|
1496
|
-
|
1497
|
-
# required to compile pg ruby gem
|
1498
|
-
- name: install PostgreSQL client
|
1499
|
-
run: sudo apt-get install libpq-dev
|
1500
|
-
|
1501
|
-
- name: Build and create DB
|
1502
|
-
env:
|
1503
|
-
# use localhost for the host here because we have specified a container for the job.
|
1504
|
-
# If we were running the job on the VM this would be postgres
|
1505
|
-
PGHOST: localhost
|
1506
|
-
PGUSER: postgres
|
1507
|
-
PGPORT: ${{ job.services.postgres.ports[5432] }} # get randomly assigned published port
|
1508
|
-
RAILS_ENV: test
|
1509
|
-
run: |
|
1510
|
-
gem install bundler
|
1511
|
-
bundle config path vendor/bundle
|
1512
|
-
bundle install --jobs 4 --retry 3
|
1513
|
-
bin/rails db:setup
|
1514
|
-
|
1515
|
-
- name: Run tests
|
1516
|
-
env:
|
1517
|
-
PGHOST: localhost
|
1518
|
-
PGUSER: postgres
|
1519
|
-
PGPORT: ${{ job.services.postgres.ports[5432] }} # get randomly assigned published port
|
1520
|
-
RAILS_ENV: test
|
1521
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
|
1522
|
-
KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
|
1523
|
-
KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
|
1524
|
-
# if you use Knapsack Pro Queue Mode you must set below env variable
|
1525
|
-
# to be able to retry CI build and run previously recorded tests
|
1526
|
-
KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
|
1527
|
-
run: |
|
1528
|
-
# run tests in Knapsack Pro Regular Mode
|
1529
|
-
bundle exec rake knapsack_pro:rspec
|
1530
|
-
bundle exec rake knapsack_pro:cucumber
|
1531
|
-
bundle exec rake knapsack_pro:minitest
|
1532
|
-
bundle exec rake knapsack_pro:test_unit
|
1533
|
-
bundle exec rake knapsack_pro:spinach
|
1534
|
-
|
1535
|
-
# you can use Knapsack Pro in Queue Mode once recorded first CI build with Regular Mode
|
1536
|
-
bundle exec rake knapsack_pro:queue:rspec
|
1537
|
-
bundle exec rake knapsack_pro:queue:cucumber
|
1538
|
-
bundle exec rake knapsack_pro:queue:minitest
|
1539
|
-
```
|
1540
|
-
|
1541
|
-
#### Info for Codefresh.io users
|
1542
|
-
|
1543
|
-
knapsack_pro gem supports environment variables provided by Codefresh.io to run your tests. You will have to define a few things in `.codefresh/codefresh.yml` config file.
|
1544
|
-
|
1545
|
-
* You need to set an API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to the pipeline) -> Variables tab (see a vertical menu on the right side). Add there new API token depending on the test runner you use:
|
1546
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC`
|
1547
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER`
|
1548
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST`
|
1549
|
-
* `KNAPSACK_PRO_TEST_SUITE_TEST_UNIT`
|
1550
|
-
* `KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH`
|
1551
|
-
* Set where Codefresh YAML file can be found. In Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to pipeline) -> Workflow tab (horizontal menu on the top) -> Path to YAML (set there `./.codefresh/codefresh.yml`).
|
1552
|
-
* Set how many parallel jobs (parallel CI nodes) you want to run with `KNAPSACK_PRO_CI_NODE_TOTAL` environment variable in `.codefresh/codefresh.yml` file.
|
1553
|
-
* Ensure in the `matrix` section you listed all `KNAPSACK_PRO_CI_NODE_INDEX` environment variables with a value from `0` to `KNAPSACK_PRO_CI_NODE_TOTAL-1`. Codefresh will generate a matrix of parallel jobs where each job has a different value for `KNAPSACK_PRO_CI_NODE_INDEX`. Thanks to that Knapsack Pro knows what tests should be run on each parallel job.
|
1554
|
-
|
1555
|
-
Below you can find Codefresh YAML config and `Test.Dockerfile` used by Codefresh to run Ruby on Rails project with PostgreSQL inside of Docker container.
|
1556
|
-
|
1557
|
-
```yaml
|
1558
|
-
# .codefresh/codefresh.yml
|
1559
|
-
version: "1.0"
|
1560
|
-
|
1561
|
-
stages:
|
1562
|
-
- "clone"
|
1563
|
-
- "build"
|
1564
|
-
- "tests"
|
1565
|
-
|
1566
|
-
steps:
|
1567
|
-
main_clone:
|
1568
|
-
type: "git-clone"
|
1569
|
-
description: "Cloning main repository..."
|
1570
|
-
repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
|
1571
|
-
revision: "${{CF_BRANCH}}"
|
1572
|
-
stage: "clone"
|
1573
|
-
BuildTestDockerImage:
|
1574
|
-
title: Building Test Docker image
|
1575
|
-
type: build
|
1576
|
-
arguments:
|
1577
|
-
image_name: '${{CF_ACCOUNT}}/${{CF_REPO_NAME}}-test'
|
1578
|
-
tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'
|
1579
|
-
dockerfile: Test.Dockerfile
|
1580
|
-
stage: "build"
|
1581
|
-
|
1582
|
-
run_tests:
|
1583
|
-
stage: "tests"
|
1584
|
-
image: '${{BuildTestDockerImage}}'
|
1585
|
-
working_directory: /src
|
1586
|
-
fail_fast: false
|
1587
|
-
environment:
|
1588
|
-
- RAILS_ENV=test
|
1589
|
-
# set how many parallel jobs you want to run
|
1590
|
-
- KNAPSACK_PRO_CI_NODE_TOTAL=2
|
1591
|
-
- PGHOST=postgres
|
1592
|
-
- PGUSER=rails-app-with-knapsack_pro
|
1593
|
-
- PGPASSWORD=password
|
1594
|
-
services:
|
1595
|
-
composition:
|
1596
|
-
postgres:
|
1597
|
-
image: postgres:latest
|
1598
|
-
environment:
|
1599
|
-
- POSTGRES_DB=rails-app-with-knapsack_pro_test
|
1600
|
-
- POSTGRES_PASSWORD=password
|
1601
|
-
- POSTGRES_USER=rails-app-with-knapsack_pro
|
1602
|
-
ports:
|
1603
|
-
- 5432
|
1604
|
-
matrix:
|
1605
|
-
environment:
|
1606
|
-
# please ensure you have here listed N-1 indexes
|
1607
|
-
# where N is KNAPSACK_PRO_CI_NODE_TOTAL
|
1608
|
-
- KNAPSACK_PRO_CI_NODE_INDEX=0
|
1609
|
-
- KNAPSACK_PRO_CI_NODE_INDEX=1
|
1610
|
-
commands:
|
1611
|
-
- bin/rails db:prepare
|
1612
|
-
|
1613
|
-
# run tests in Knapsack Pro Regular Mode
|
1614
|
-
- bundle exec rake knapsack_pro:rspec
|
1615
|
-
- bundle exec rake knapsack_pro:cucumber
|
1616
|
-
- bundle exec rake knapsack_pro:minitest
|
1617
|
-
- bundle exec rake knapsack_pro:test_unit
|
1618
|
-
- bundle exec rake knapsack_pro:spinach
|
1619
|
-
|
1620
|
-
# you can use Knapsack Pro in Queue Mode once recorded first CI build with Regular Mode
|
1621
|
-
- bundle exec rake knapsack_pro:queue:rspec
|
1622
|
-
- bundle exec rake knapsack_pro:queue:cucumber
|
1623
|
-
- bundle exec rake knapsack_pro:queue:minitest
|
1624
|
-
```
|
1625
|
-
|
1626
|
-
Add `Test.Dockerfile` to your project repository.
|
1627
|
-
|
1628
|
-
```Dockerfile
|
1629
|
-
# Test.Dockerfile
|
1630
|
-
FROM ruby:2.6.5-alpine3.10
|
1631
|
-
|
1632
|
-
# Prepare Docker image for Nokogiri
|
1633
|
-
RUN apk add --update \
|
1634
|
-
build-base \
|
1635
|
-
libxml2-dev \
|
1636
|
-
libxslt-dev \
|
1637
|
-
jq \
|
1638
|
-
nodejs \
|
1639
|
-
npm \
|
1640
|
-
postgresql-dev \
|
1641
|
-
python3-dev \
|
1642
|
-
sqlite-dev \
|
1643
|
-
git \
|
1644
|
-
&& rm -rf /var/cache/apk/*
|
1645
|
-
|
1646
|
-
# Install AWS CLI
|
1647
|
-
RUN pip3 install awscli
|
1648
|
-
|
1649
|
-
# Use libxml2, libxslt a packages from alpine for building nokogiri
|
1650
|
-
RUN bundle config build.nokogiri --use-system-libraries
|
1651
|
-
|
1652
|
-
# Install Codefresh CLI
|
1653
|
-
RUN wget https://github.com/codefresh-io/cli/releases/download/v0.31.1/codefresh-v0.31.1-alpine-x64.tar.gz
|
1654
|
-
RUN tar -xf codefresh-v0.31.1-alpine-x64.tar.gz -C /usr/local/bin/
|
1655
|
-
|
1656
|
-
COPY . /src
|
1657
|
-
|
1658
|
-
WORKDIR /src
|
1659
|
-
|
1660
|
-
RUN bundle install
|
1661
|
-
```
|
1662
|
-
|
1663
|
-
## Gem tests
|
1664
|
-
|
1665
|
-
### Spec
|
1666
|
-
|
1667
|
-
To run specs for Knapsack Pro gem type:
|
1668
|
-
|
1669
|
-
$ bundle exec rspec spec
|
1670
|
-
|
1671
|
-
## Contributing
|
1672
|
-
|
1673
|
-
1. Fork it ( https://github.com/KnapsackPro/knapsack_pro-ruby )
|
1674
|
-
2. Create your feature branch (`git checkout -b my-new-feature`)
|
1675
|
-
3. Commit your changes (`git commit -am 'Add some feature'`)
|
1676
|
-
4. Push to the branch (`git push origin my-new-feature`)
|
1677
|
-
5. You can create example tests in related repository with example of [rails application and knapsack_pro gem usage](https://github.com/KnapsackPro/rails-app-with-knapsack_pro).
|
1678
|
-
6. Create a new Pull Request
|
1679
|
-
|
1680
|
-
### Publishing
|
1681
|
-
|
1682
|
-
Update version in `lib/knapsack_pro/version.rb` and `CHANGELOG.md`:
|
1683
|
-
|
1684
|
-
```
|
1685
|
-
$ git commit -m "Bump version X.X.X"
|
1686
|
-
$ git push origin master
|
1687
|
-
```
|
1688
|
-
|
1689
|
-
Create git tag for release:
|
1690
|
-
|
1691
|
-
```
|
1692
|
-
$ git tag -a vX.X.X -m "Release vX.X.X"
|
1693
|
-
$ git push --tags
|
1694
|
-
```
|
1695
|
-
|
1696
|
-
Build gem and publish it to RubyGems.org:
|
1697
|
-
|
1698
|
-
```
|
1699
|
-
$ gem build knapsack_pro.gemspec
|
1700
|
-
$ gem push knapsack_pro-X.X.X.gem
|
101
|
+
gem build knapsack_pro.gemspec
|
102
|
+
gem push knapsack_pro-X.X.X.gem
|
1701
103
|
```
|
1702
104
|
|
1703
105
|
Update the latest available gem version in `TestSuiteClientVersionChecker` for the Knapsack Pro API repository.
|
1704
106
|
|
1705
|
-
Update knapsack_pro gem version in:
|
1706
|
-
|
1707
|
-
* https://github.com/KnapsackPro/rails-app-with-knapsack_pro
|
1708
|
-
* our private Knapsack Pro API repository
|
1709
|
-
|
1710
|
-
## Mentions
|
1711
|
-
|
1712
|
-
List of articles where people mentioned Knapsack Pro:
|
107
|
+
Update the `knapsack_pro` gem version in:
|
1713
108
|
|
1714
|
-
|
1715
|
-
|
109
|
+
- [Rails App With Knapsack Pro repository](https://github.com/KnapsackPro/rails-app-with-knapsack_pro)
|
110
|
+
- Knapsack Pro API internal repository
|