knapsack_pro 2.12.0 → 2.17.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.circleci/config.yml +1 -1
- data/CHANGELOG.md +40 -0
- data/README.md +14 -1567
- data/knapsack_pro.gemspec +4 -5
- data/lib/knapsack_pro/adapters/rspec_adapter.rb +12 -2
- data/lib/knapsack_pro/allocator.rb +1 -1
- data/lib/knapsack_pro/config/ci/circle.rb +1 -2
- data/lib/knapsack_pro/queue_allocator.rb +1 -1
- data/lib/knapsack_pro/repository_adapters/git_adapter.rb +2 -1
- data/lib/knapsack_pro/runners/base_runner.rb +4 -0
- data/lib/knapsack_pro/runners/queue/base_runner.rb +4 -0
- data/lib/knapsack_pro/runners/queue/cucumber_runner.rb +3 -5
- data/lib/knapsack_pro/runners/queue/rspec_runner.rb +23 -4
- data/lib/knapsack_pro/runners/spinach_runner.rb +1 -1
- data/lib/knapsack_pro/runners/test_unit_runner.rb +14 -3
- data/lib/knapsack_pro/tracker.rb +12 -3
- data/lib/knapsack_pro/version.rb +1 -1
- data/spec/knapsack_pro/adapters/rspec_adapter_spec.rb +30 -12
- data/spec/knapsack_pro/adapters/test_unit_adapter_spec.rb +8 -1
- data/spec/knapsack_pro/config/ci/circle_spec.rb +3 -3
- data/spec/knapsack_pro/repository_adapters/git_adapter_spec.rb +3 -3
- data/spec/knapsack_pro/runners/queue/cucumber_runner_spec.rb +4 -2
- data/spec/knapsack_pro/runners/queue/rspec_runner_spec.rb +37 -7
- data/spec/knapsack_pro/runners/spinach_runner_spec.rb +5 -2
- data/spec/knapsack_pro/runners/test_unit_runner_spec.rb +1 -3
- data/spec/knapsack_pro/tracker_spec.rb +4 -2
- data/spec/spec_helper.rb +0 -3
- metadata +21 -35
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e22a25e69f6cfda9a42568aed336631f601423b0b930525fc8becbc1b1ee7fe9
|
|
4
|
+
data.tar.gz: ac1df0750000066a933cc4814f185d09a6d38ce09028680c8f20678bb7b0473e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 91687797a4f7c80bcd852f7c8e5ec18f523eb5a9375b3dc069b6abce67bc58ccda010f36b6c003fefa01971e628518bf0b9f9e855d7992c86d9ea7dcb375be4b
|
|
7
|
+
data.tar.gz: c32195b9e6fd4fdee7b5087f61f7d1b28fdb7a6a995930d23e109eda296156026a79fea579e06540e2a64c75ee3a03a6b216f2a8492eb48e422bc9193aad0146
|
data/.circleci/config.yml
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,45 @@
|
|
|
1
1
|
# Change Log
|
|
2
2
|
|
|
3
|
+
### 2.17.0
|
|
4
|
+
|
|
5
|
+
* Use Ruby 3 in development and add small improvements
|
|
6
|
+
|
|
7
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/147
|
|
8
|
+
|
|
9
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v2.16.0...v2.17.0
|
|
10
|
+
|
|
11
|
+
### 2.16.0
|
|
12
|
+
|
|
13
|
+
* Improve test time execution tracking for RSpec
|
|
14
|
+
|
|
15
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/145
|
|
16
|
+
|
|
17
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v2.15.0...v2.16.0
|
|
18
|
+
|
|
19
|
+
### 2.15.0
|
|
20
|
+
|
|
21
|
+
* Do not allow to use the RSpec tag option together with the RSpec split by test examples feature
|
|
22
|
+
|
|
23
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/139
|
|
24
|
+
|
|
25
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v2.14.0...v2.15.0
|
|
26
|
+
|
|
27
|
+
### 2.14.0
|
|
28
|
+
|
|
29
|
+
* Track time spend in RSpec context hook
|
|
30
|
+
|
|
31
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/143
|
|
32
|
+
|
|
33
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v2.13.0...v2.14.0
|
|
34
|
+
|
|
35
|
+
### 2.13.0
|
|
36
|
+
|
|
37
|
+
* Update FAQ links in `knapsack_pro` warnings and remove FAQ from readme
|
|
38
|
+
|
|
39
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/142
|
|
40
|
+
|
|
41
|
+
https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v2.12.0...v2.13.0
|
|
42
|
+
|
|
3
43
|
### 2.12.0
|
|
4
44
|
|
|
5
45
|
* Use 0 seconds as a default test file time execution instead of 0.1s because Knapsack Pro API already accepts 0 seconds value
|
data/README.md
CHANGED
|
@@ -42,11 +42,7 @@ The knapsack_pro has also [queue mode](#queue-mode) to get an optimal test suite
|
|
|
42
42
|
|
|
43
43
|
## FAQ
|
|
44
44
|
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
__OLD:__ This README also contains FAQ questions but we keep adding new info only to our new FAQ page mentioned above.
|
|
48
|
-
|
|
49
|
-
We keep this old FAQ in README to not break old links spread across the web. You can see old FAQ list of questions for common problems and tips in below [Table of Contents](#table-of-contents). Scroll 1 page down and you will see the FAQ in the table of contents.
|
|
45
|
+
[FAQ for knapsack_pro gem can be found here](https://knapsackpro.com/faq/knapsack_pro_client/knapsack_pro_ruby).
|
|
50
46
|
|
|
51
47
|
# Requirements
|
|
52
48
|
|
|
@@ -124,86 +120,6 @@ We keep this old FAQ in README to not break old links spread across the web. You
|
|
|
124
120
|
- [Info for Jenkins users](#info-for-jenkins-users)
|
|
125
121
|
- [Info for GitHub Actions users](#info-for-github-actions-users)
|
|
126
122
|
- [Info for Codefresh.io users](#info-for-codefreshio-users)
|
|
127
|
-
- [FAQ](#faq)
|
|
128
|
-
- [Common problems](#common-problems)
|
|
129
|
-
- [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required)
|
|
130
|
-
- [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper)
|
|
131
|
-
- [Why my CI build fails when I use Test::Unit even when all tests passed?](#why-my-ci-build-fails-when-i-use-testunit-even-when-all-tests-passed)
|
|
132
|
-
- [Why I see HEAD as branch name in user dashboard for Build metrics for my API token?](#why-i-see-head-as-branch-name-in-user-dashboard-for-build-metrics-for-my-api-token)
|
|
133
|
-
- [Why Capybara feature tests randomly fail when using CI parallelisation?](#why-capybara-feature-tests-randomly-fail-when-using-ci-parallelisation)
|
|
134
|
-
- [Why knapsack_pro freezes / hangs my CI (for instance Travis)?](#why-knapsack_pro-freezes--hangs-my-ci-for-instance-travis)
|
|
135
|
-
- [Why tests hitting external API fail?](#why-tests-hitting-external-api-fail)
|
|
136
|
-
- [Why green test suite for Cucumber 2.99 tests always fails with `invalid option: --require`?](#why-green-test-suite-for-cucumber-299-tests-always-fails-with-invalid-option---require)
|
|
137
|
-
- [Queue Mode problems](#queue-mode-problems)
|
|
138
|
-
- [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail)
|
|
139
|
-
- [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail)
|
|
140
|
-
- [Why when I use Queue Mode for RSpec then my rake tasks are run twice?](#why-when-i-use-queue-mode-for-rspec-then-my-rake-tasks-are-run-twice)
|
|
141
|
-
- [Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?](#why-when-i-use-queue-mode-for-rspec-then-i-see-error-superclass-mismatch-for-class)
|
|
142
|
-
- [Why when I use Queue Mode for RSpec then `.rspec` config is ignored?](#why-when-i-use-queue-mode-for-rspec-then-rspec-config-is-ignored)
|
|
143
|
-
- [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard)
|
|
144
|
-
- [Why all test files have 0.1s time execution for my CI build in user dashboard?](#why-all-test-files-have-01s-time-execution-for-my-ci-build-in-user-dashboard)
|
|
145
|
-
- [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result)
|
|
146
|
-
- [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests)
|
|
147
|
-
- [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again)
|
|
148
|
-
- [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again)
|
|
149
|
-
- [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec)
|
|
150
|
-
- [Parallel tests Cucumber and RSpec with Cucumber failures exit CI node early leaving fewer CI nodes to finish RSpec Queue.](#parallel-tests-cucumber-and-rspec-with-cucumber-failures-exit-ci-node-early-leaving-fewer-ci-nodes-to-finish-rspec-queue)
|
|
151
|
-
- [Why when I reran the same build (same commit hash, etc) on Codeship then no tests would get executed in Queue Mode?](#why-when-i-reran-the-same-build-same-commit-hash-etc-on-codeship-then-no-tests-would-get-executed-in-queue-mode)
|
|
152
|
-
- [Why knapsack_pro hangs / freezes / is stale i.e. for Codeship in Queue Mode?](#why-knapsack_pro-hangs--freezes--is-stale-ie-for-codeship-in-queue-mode)
|
|
153
|
-
- [How to find seed in RSpec output when I use Queue Mode for RSpec?](#how-to-find-seed-in-rspec-output-when-i-use-queue-mode-for-rspec)
|
|
154
|
-
- [How to configure puffing-billy gem with Knapsack Pro Queue Mode?](#how-to-configure-puffing-billy-gem-with-knapsack-pro-queue-mode)
|
|
155
|
-
- [General questions](#general-questions)
|
|
156
|
-
- [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment)
|
|
157
|
-
- [for knapsack_pro regular mode](#for-knapsack_pro-regular-mode)
|
|
158
|
-
- [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode)
|
|
159
|
-
- [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily)
|
|
160
|
-
- [for knapsack_pro regular mode](#for-knapsack_pro-regular-mode-1)
|
|
161
|
-
- [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode-1)
|
|
162
|
-
- [How can I change log level?](#how-can-i-change-log-level)
|
|
163
|
-
- [How to write knapsack_pro logs to a file?](#how-to-write-knapsack_pro-logs-to-a-file)
|
|
164
|
-
- [set directory where to write log file (option 1 - recommended)](#set-directory-where-to-write-log-file-option-1---recommended)
|
|
165
|
-
- [set custom logger config (option 2)](#set-custom-logger-config-option-2)
|
|
166
|
-
- [How to preserve logs on my CI after CI build completed?](#how-to-preserve-logs-on-my-ci-after-ci-build-completed)
|
|
167
|
-
- [How to split tests based on test level instead of test file level?](#how-to-split-tests-based-on-test-level-instead-of-test-file-level)
|
|
168
|
-
- [A. Create multiple small test files](#a-create-multiple-small-test-files)
|
|
169
|
-
- [B. Use tags to mark set of tests in particular test file](#b-use-tags-to-mark-set-of-tests-in-particular-test-file)
|
|
170
|
-
- [How to make knapsack_pro works for forked repositories of my project?](#how-to-make-knapsack_pro-works-for-forked-repositories-of-my-project)
|
|
171
|
-
- [How to use junit formatter?](#how-to-use-junit-formatter)
|
|
172
|
-
- [How to use junit formatter with knapsack_pro regular mode?](#how-to-use-junit-formatter-with-knapsack_pro-regular-mode)
|
|
173
|
-
- [How to use junit formatter with knapsack_pro queue mode?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode)
|
|
174
|
-
- [How to use junit formatter with knapsack_pro queue mode when CI nodes use common local drive?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive)
|
|
175
|
-
- [Why `tmp/rspec_final_results.xml` is corrupted when I use junit formatter with knapsack_pro queue mode?](#why-tmprspec_final_resultsxml-is-corrupted-when-i-use-junit-formatter-with-knapsack_pro-queue-mode)
|
|
176
|
-
- [How to use junit formatter with knapsack_pro queue mode in Cucumber?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode-in-cucumber)
|
|
177
|
-
- [How to use JSON formatter for RSpec?](#how-to-use-json-formatter-for-rspec)
|
|
178
|
-
- [How to use RSpec JSON formatter with knapsack_pro Queue Mode?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode)
|
|
179
|
-
- [How to use RSpec JSON formatter with knapsack_pro Queue Mode when CI nodes use common local drive?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive)
|
|
180
|
-
- [How many API keys I need?](#how-many-api-keys-i-need)
|
|
181
|
-
- [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands)
|
|
182
|
-
- [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example)
|
|
183
|
-
- [percy-capybara gem version < 4 (old)](#percy-capybara-gem-version--4-old)
|
|
184
|
-
- [percy-capybara gem version >= 4](#percy-capybara-gem-version--4)
|
|
185
|
-
- [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode)
|
|
186
|
-
- [What hooks are supported in Queue Mode?](#what-hooks-are-supported-in-queue-mode)
|
|
187
|
-
- [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem)
|
|
188
|
-
- [Should I use parallel_tests gem (what are pitfalls)?](#should-i-use-parallel_tests-gem-what-are-pitfalls)
|
|
189
|
-
- [parallel_tests with knapsack_pro on parallel CI nodes](#parallel_tests-with-knapsack_pro-on-parallel-ci-nodes)
|
|
190
|
-
- [parallel_tests with knapsack_pro on single CI machine](#parallel_tests-with-knapsack_pro-on-single-ci-machine)
|
|
191
|
-
- [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests)
|
|
192
|
-
- [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories)
|
|
193
|
-
- [Why I don't see all test files being recorded in user dashboard](#why-i-dont-see-all-test-files-being-recorded-in-user-dashboard)
|
|
194
|
-
- [Why when I use 2 different CI providers then not all test files are executed?](#why-when-i-use-2-different-ci-providers-then-not-all-test-files-are-executed)
|
|
195
|
-
- [How to run only RSpec feature tests or non feature tests?](#how-to-run-only-rspec-feature-tests-or-non-feature-tests)
|
|
196
|
-
- [How to exclude tests from running them?](#how-to-exclude-tests-from-running-them)
|
|
197
|
-
- [How to run a specific list of test files or only some tests from test file?](#how-to-run-a-specific-list-of-test-files-or-only-some-tests-from-test-file)
|
|
198
|
-
- [How to run knapsack_pro only on a few parallel CI nodes instead of all?](#how-to-run-knapsack_pro-only-on-a-few-parallel-ci-nodes-instead-of-all)
|
|
199
|
-
- [How to use CodeClimate with knapsack_pro?](#how-to-use-codeclimate-with-knapsack_pro)
|
|
200
|
-
- [How to use simplecov in Queue Mode?](#how-to-use-simplecov-in-queue-mode)
|
|
201
|
-
- [Do I need to use separate API token for Queue Mode and Regular Mode?](#do-i-need-to-use-separate-api-token-for-queue-mode-and-regular-mode)
|
|
202
|
-
- [How to stop running tests on the first failed test (fail fast tests in RSpec)?](#how-to-stop-running-tests-on-the-first-failed-test-fail-fast-tests-in-rspec)
|
|
203
|
-
- [Questions around data usage and security](#questions-around-data-usage-and-security)
|
|
204
|
-
- [What data is sent to your servers?](#what-data-is-sent-to-your-servers)
|
|
205
|
-
- [How is that data secured?](#how-is-that-data-secured)
|
|
206
|
-
- [Who has access to the data?](#who-has-access-to-the-data)
|
|
207
123
|
- [Gem tests](#gem-tests)
|
|
208
124
|
- [Spec](#spec)
|
|
209
125
|
- [Contributing](#contributing)
|
|
@@ -375,12 +291,12 @@ KnapsackPro.logger = Logger.new(STDOUT)
|
|
|
375
291
|
KnapsackPro.logger.level = Logger::DEBUG
|
|
376
292
|
```
|
|
377
293
|
|
|
378
|
-
Debug is default log level and it is recommended. [Read more](
|
|
294
|
+
Debug is default log level and it is recommended. [Read more](https://knapsackpro.com/faq/question/how-can-i-change-log-level).
|
|
379
295
|
|
|
380
296
|
Note your own logger is configured in `spec_helper.rb` or `rails_helper.rb` and it will start working when those files will be loaded.
|
|
381
297
|
It means the very first request to Knapsack Pro API will be log to `STDOUT` using logger built into knapsack_pro instead of your custom logger.
|
|
382
298
|
|
|
383
|
-
If you want to change log level globally than just for your custom log level, please [see this](
|
|
299
|
+
If you want to change log level globally than just for your custom log level, please [see this](https://knapsackpro.com/faq/question/how-can-i-change-log-level).
|
|
384
300
|
|
|
385
301
|
### Setup your CI server (How to set up 2 of 3)
|
|
386
302
|
|
|
@@ -438,7 +354,7 @@ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KN
|
|
|
438
354
|
|
|
439
355
|
__Tip:__ If you use one of the supported CI providers then instead of the above steps you should [take a look at this](#supported-ci-providers).
|
|
440
356
|
|
|
441
|
-
__Tip 2:__ If you use one of unsupported CI providers ([here is list of supported CI providers](#supported-ci-providers)) then you should [set KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-
|
|
357
|
+
__Tip 2:__ If you use one of unsupported CI providers ([here is list of supported CI providers](#supported-ci-providers)) then you should [set KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-when-ci-provider-is-not-supported-and-you-use-git).
|
|
442
358
|
|
|
443
359
|
### Repository adapter (How to set up 3 of 3)
|
|
444
360
|
|
|
@@ -515,11 +431,11 @@ The second CI build should have optimal test suite split with faster tests distr
|
|
|
515
431
|
|
|
516
432
|
__Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes.__
|
|
517
433
|
|
|
518
|
-
If you use the capybara-screenshot gem then please [follow this step](
|
|
434
|
+
If you use the capybara-screenshot gem then please [follow this step](https://knapsackpro.com/faq/question/how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec).
|
|
519
435
|
|
|
520
|
-
If you use the rspec_junit_formatter gem then please [follow this step](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode).
|
|
436
|
+
If you use the rspec_junit_formatter gem then please [follow this step](https://knapsackpro.com/faq/question/how-to-use-junit-formatter#how-to-use-junit-formatter-with-knapsack_pro-queue-mode).
|
|
521
437
|
|
|
522
|
-
If your test suite is very long and the RSpec output is too long for your CI node then you can set log level `KNAPSACK_PRO_LOG_LEVEL=info` to don't show debug messages in RSpec output. [Read more about log level](
|
|
438
|
+
If your test suite is very long and the RSpec output is too long for your CI node then you can set log level `KNAPSACK_PRO_LOG_LEVEL=info` to don't show debug messages in RSpec output. [Read more about log level](https://knapsackpro.com/faq/question/how-can-i-change-log-level).
|
|
523
439
|
|
|
524
440
|
### Additional info about queue mode
|
|
525
441
|
|
|
@@ -583,7 +499,7 @@ There might be some cached test suite splits for git commits you have run in pas
|
|
|
583
499
|
[knapsack_pro] {"queue_name"=>"retry-dead-ci-node:queue-id", "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]}
|
|
584
500
|
```
|
|
585
501
|
|
|
586
|
-
* To [reproduce tests executed on CI node](#for-knapsack_pro-queue-mode) in development environment please see FAQ.
|
|
502
|
+
* To [reproduce tests executed on CI node](https://knapsackpro.com/faq/question/how-to-run-tests-for-particular-ci-node-in-your-development-environment#for-knapsack_pro-queue-mode) in development environment please see FAQ.
|
|
587
503
|
|
|
588
504
|
#### KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)
|
|
589
505
|
|
|
@@ -877,43 +793,7 @@ $ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_branch_names[not
|
|
|
877
793
|
|
|
878
794
|
If you are using circleci.com you can omit `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX`. Knapsack Pro will use `CIRCLE_NODE_TOTAL` and `CIRCLE_NODE_INDEX` provided by CircleCI.
|
|
879
795
|
|
|
880
|
-
Here is an example for test configuration in your
|
|
881
|
-
|
|
882
|
-
```yaml
|
|
883
|
-
# CircleCI 1.0
|
|
884
|
-
|
|
885
|
-
machine:
|
|
886
|
-
environment:
|
|
887
|
-
# Tokens should be set in CircleCI settings to avoid expose tokens in build logs
|
|
888
|
-
# KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: rspec-token
|
|
889
|
-
# KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: cucumber-token
|
|
890
|
-
# KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST: minitest-token
|
|
891
|
-
# KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT: test-unit-token
|
|
892
|
-
# KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH: spinach-token
|
|
893
|
-
test:
|
|
894
|
-
override:
|
|
895
|
-
# Step for RSpec
|
|
896
|
-
- bundle exec rake knapsack_pro:rspec:
|
|
897
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
898
|
-
|
|
899
|
-
# Step for Cucumber
|
|
900
|
-
- bundle exec rake knapsack_pro:cucumber:
|
|
901
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
902
|
-
|
|
903
|
-
# Step for Minitest
|
|
904
|
-
- bundle exec rake knapsack_pro:minitest:
|
|
905
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
906
|
-
|
|
907
|
-
# Step for test-unit
|
|
908
|
-
- bundle exec rake knapsack_pro:test_unit:
|
|
909
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
910
|
-
|
|
911
|
-
# Step for Spinach
|
|
912
|
-
- bundle exec rake knapsack_pro:spinach:
|
|
913
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
914
|
-
```
|
|
915
|
-
|
|
916
|
-
Here is another example for CircleCI 2.0 platform.
|
|
796
|
+
Here is an example for test configuration in your `.circleci/config.yml` file.
|
|
917
797
|
|
|
918
798
|
```yaml
|
|
919
799
|
# CircleCI 2.0
|
|
@@ -950,9 +830,9 @@ Please remember to add additional containers for your project in CircleCI settin
|
|
|
950
830
|
|
|
951
831
|
##### CircleCI and knapsack_pro Queue Mode
|
|
952
832
|
|
|
953
|
-
If you use knapsack_pro Queue Mode with CircleCI you may want to [collect metadata](https://circleci.com/docs/
|
|
833
|
+
If you use knapsack_pro Queue Mode with CircleCI you may want to [collect metadata](https://circleci.com/docs/2.0/collect-test-data/#metadata-collection-in-custom-test-steps) like junit xml report about your RSpec test suite.
|
|
954
834
|
|
|
955
|
-
Here you can read how to configure [junit formatter](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). Step for CircleCI is to copy the xml report to `$CIRCLE_TEST_REPORTS` directory. Below is full config for your `spec_helper.rb`:
|
|
835
|
+
Here you can read how to configure [junit formatter](https://knapsackpro.com/faq/question/how-to-use-junit-formatter#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). Step for CircleCI is to copy the xml report to `$CIRCLE_TEST_REPORTS` directory. Below is full config for your `spec_helper.rb`:
|
|
956
836
|
|
|
957
837
|
```ruby
|
|
958
838
|
# spec_helper.rb or rails_helper.rb
|
|
@@ -997,6 +877,8 @@ Ensure you have in CircleCI config yml
|
|
|
997
877
|
destination: test-results
|
|
998
878
|
```
|
|
999
879
|
|
|
880
|
+
You can find a complete CircleCI YML config example in [the article](https://docs.knapsackpro.com/2021/rspec-testing-parallel-jobs-with-circleci-and-junit-xml-report).
|
|
881
|
+
|
|
1000
882
|
#### Info for Travis users
|
|
1001
883
|
|
|
1002
884
|
You can parallelize your builds across virtual machines with [travis matrix feature](http://docs.travis-ci.com/user/speeding-up-the-build/#parallelizing-your-builds-across-virtual-machines). Edit `.travis.yml`
|
|
@@ -1561,7 +1443,7 @@ Remember to set environment variables in Jenkins configuration with your API tok
|
|
|
1561
1443
|
Here is [list of environment variables per test runner](#set-api-key-token).
|
|
1562
1444
|
|
|
1563
1445
|
Above example shows how to run cucumber tests in regular mode and later the rspec tests in queue mode to autobalance build.
|
|
1564
|
-
If you are going to relay on rspec to autobalance build when cucumber tests were not perfectly distributed you should be aware about [possible edge case if your rspec test suite is very short](
|
|
1446
|
+
If you are going to relay on rspec to autobalance build when cucumber tests were not perfectly distributed you should be aware about [possible edge case if your rspec test suite is very short](https://knapsackpro.com/faq/question/why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again).
|
|
1565
1447
|
|
|
1566
1448
|
#### Info for GitHub Actions users
|
|
1567
1449
|
|
|
@@ -1790,1441 +1672,6 @@ WORKDIR /src
|
|
|
1790
1672
|
RUN bundle install
|
|
1791
1673
|
```
|
|
1792
1674
|
|
|
1793
|
-
## FAQ
|
|
1794
|
-
|
|
1795
|
-
### Common problems
|
|
1796
|
-
|
|
1797
|
-
#### Why I see API error commit_hash parameter is required?
|
|
1798
|
-
|
|
1799
|
-
ERROR -- : [knapsack_pro] {"errors"=>[{"commit_hash"=>["parameter is required"]}]}
|
|
1800
|
-
|
|
1801
|
-
When Knapsack Pro API returns error like above the problem is because you use CI provider not supported by knapsack_pro which means
|
|
1802
|
-
knapsack_pro gem cannot determine the git commit hash and branch name. To fix this problem you can do:
|
|
1803
|
-
|
|
1804
|
-
* if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported)
|
|
1805
|
-
* if you have no git installed on CI node then you should manually set `KNAPSACK_PRO_BRANCH` and `KNAPSACK_PRO_COMMIT_HASH`. For instance this might be useful when you use Jenkins. [See this](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default)
|
|
1806
|
-
|
|
1807
|
-
#### Why I see `LoadError: cannot load such file -- spec_helper`?
|
|
1808
|
-
|
|
1809
|
-
When your tests fails with:
|
|
1810
|
-
|
|
1811
|
-
LoadError: cannot load such file -- spec_helper
|
|
1812
|
-
|
|
1813
|
-
then problem might be related to the fact you specified complex `KNAPSACK_PRO_TEST_FILE_PATTERN` and knapsack_pro gem cannot detect correct main test directory with spec_helper. You should set `KNAPSACK_PRO_TEST_DIR=spec`. Please [read also example](#how-can-i-run-tests-from-multiple-directories).
|
|
1814
|
-
|
|
1815
|
-
#### Why my CI build fails when I use Test::Unit even when all tests passed?
|
|
1816
|
-
|
|
1817
|
-
Please ensure you are actually using only Test::Unit runner. You may use some hybrid of Test::Unit and Minitest. Ensure you are not loading Minitest.
|
|
1818
|
-
|
|
1819
|
-
#### Why I see HEAD as branch name in user dashboard for Build metrics for my API token?
|
|
1820
|
-
|
|
1821
|
-
knapsack_pro detects your branch name from environment variables of [supported CI providers](#supported-ci-providers). Sometimes the CI provider may expose the `HEAD` instead of branch name (for instance for pull request merge commits).
|
|
1822
|
-
|
|
1823
|
-
The same can happen for CI provider not supported by default by knapsack_pro when you use [KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) to use local git installed on CI node to detect the branch name and git commit.
|
|
1824
|
-
|
|
1825
|
-
knapsack_pro uses git command `git -C /home/user/project_dir rev-parse --abbrev-ref HEAD` to detect branch name. See [source of knapsack_pro](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/repository_adapters/git_adapter.rb). In most of cases it's good way to detect branch name. But if your CI provider during CI build checkouts to specific git commit then git cannot provide the name of the branch. In such scenario you would see `HEAD` as your branch name. It is good enough situation and knapsack_pro will work correctly. The benefit of knowing exactly the branch name allows KnapsackPro API to better track history of test files timing changes across branches in order to better do split of test suite. The difference should be rather very small so it's not a problem that you have `HEAD` as branch name.
|
|
1826
|
-
|
|
1827
|
-
If you would like to see exact branch name instead of `HEAD` in your `build metrics` history in [user dashboard](https://knapsackpro.com/dashboard) then you can explicitly provide the branch name with `KNAPSACK_PRO_BRANCH` for each CI build.
|
|
1828
|
-
|
|
1829
|
-
#### Why Capybara feature tests randomly fail when using CI parallelisation?
|
|
1830
|
-
|
|
1831
|
-
It can happen that when you use CI parallelisation then your CI machine is overloaded and some of Capybara feature specs may randomly fail when tested website loaded slowly.
|
|
1832
|
-
|
|
1833
|
-
You can try to increase default Capybara max wait time from 2 seconds to something bigger like 5 seconds to ensure the Capybara will wait longer till the website is loaded before marking test as failed.
|
|
1834
|
-
|
|
1835
|
-
```ruby
|
|
1836
|
-
# spec/rails_helper.rb
|
|
1837
|
-
Capybara.default_max_wait_time = 5 # in seconds
|
|
1838
|
-
```
|
|
1839
|
-
|
|
1840
|
-
For instance, this tip might be helpful for Heroku CI users who use Heroku dynos with lower performance.
|
|
1841
|
-
|
|
1842
|
-
#### Why knapsack_pro freezes / hangs my CI (for instance Travis)?
|
|
1843
|
-
|
|
1844
|
-
[Freeze error can occur for example on Travis CI](https://docs.travis-ci.com/user/common-build-problems/#ruby-tests-frozen-and-cancelled-after-10-minute-log-silence).
|
|
1845
|
-
The `timecop` gem can result in sporadic freezing due to issues with ordering calls of `Timecop.return`, `Timecop.freeze`, and `Timecop.travel`. For instance, if using RSpec, ensure to have a `Timecop.return` configured to run after all examples:
|
|
1846
|
-
|
|
1847
|
-
```ruby
|
|
1848
|
-
# in, e.g. spec/spec_helper.rb
|
|
1849
|
-
RSpec.configure do |c|
|
|
1850
|
-
c.after(:all) do
|
|
1851
|
-
Timecop.return
|
|
1852
|
-
end
|
|
1853
|
-
end
|
|
1854
|
-
```
|
|
1855
|
-
|
|
1856
|
-
#### Why tests hitting external API fail?
|
|
1857
|
-
|
|
1858
|
-
If you use knapsack_pro and you have tests that do real HTTP requests to external API you need to ensure your tests can be run across parallel CI nodes.
|
|
1859
|
-
|
|
1860
|
-
Let's say you have tests that do requests to Stripe API or any other API. Before running each test you want to make sure Stripe Sandbox is clean up so you have removed all fake subscriptions and customers from Stripe Sandbox.
|
|
1861
|
-
|
|
1862
|
-
```ruby
|
|
1863
|
-
# RSpec hook
|
|
1864
|
-
before(:each) do
|
|
1865
|
-
Stripe::Subscription.all.each { |sub| sub.delete }
|
|
1866
|
-
Stripe::Customer.all.each { |customer| customer.delete }
|
|
1867
|
-
end
|
|
1868
|
-
```
|
|
1869
|
-
|
|
1870
|
-
But this will cause a problem when 2 different test files will run on 2 different CI nodes at the same time and this hook will be called. You will remove subscriptions and customers while another parallel test was running. Simply speaking you have tests that are written in a way that you can't run them in parallel.
|
|
1871
|
-
|
|
1872
|
-
To fix that you can think of:
|
|
1873
|
-
* using [VCR](https://github.com/vcr/vcr) gem to record HTTP requests and then instead of doing real HTTP requests just reply recorded requests.
|
|
1874
|
-
* maybe you could write your tests in a way when you generate some fake customers or subscriptions with fake id and each test has different customer id so there will be no conflict when 2 tests are run at the same time.
|
|
1875
|
-
|
|
1876
|
-
#### Why green test suite for Cucumber 2.99 tests always fails with `invalid option: --require`?
|
|
1877
|
-
|
|
1878
|
-
If you use old Cucumber version 2.99 and `cucumber-rails` gem you could notice bug that knapsack_pro for Cucumber fails with `1` exit status. Error you may see:
|
|
1879
|
-
|
|
1880
|
-
```
|
|
1881
|
-
invalid option: --require
|
|
1882
|
-
|
|
1883
|
-
minitest options:
|
|
1884
|
-
-h, --help Display this help.
|
|
1885
|
-
--no-plugins Bypass minitest plugin auto-loading (or set $MT_NO_PLUGINS).
|
|
1886
|
-
-s, --seed SEED Sets random seed. Also via env. Eg: SEED=n rake
|
|
1887
|
-
-v, --verbose Verbose. Show progress processing files.
|
|
1888
|
-
-n, --name PATTERN Filter run on /regexp/ or string.
|
|
1889
|
-
--exclude PATTERN Exclude /regexp/ or string from run.
|
|
1890
|
-
|
|
1891
|
-
Known extensions: rails, pride
|
|
1892
|
-
-w, --warnings Run with Ruby warnings enabled
|
|
1893
|
-
-e, --environment ENV Run tests in the ENV environment
|
|
1894
|
-
-b, --backtrace Show the complete backtrace
|
|
1895
|
-
-d, --defer-output Output test failures and errors after the test run
|
|
1896
|
-
-f, --fail-fast Abort test run on first failure or error
|
|
1897
|
-
-c, --[no-]color Enable color in the output
|
|
1898
|
-
-p, --pride Pride. Show your testing pride!
|
|
1899
|
-
|
|
1900
|
-
# exit status is 1 - which means failed tests
|
|
1901
|
-
> echo $?
|
|
1902
|
-
1
|
|
1903
|
-
```
|
|
1904
|
-
|
|
1905
|
-
The root problem is that Rails add `minitest` gem and it is started when `cucumber/rails` is loaded. It should not be. You can fix it by adding below in file `features/support/env.rb`:
|
|
1906
|
-
|
|
1907
|
-
```ruby
|
|
1908
|
-
# features/support/env.rb
|
|
1909
|
-
require 'cucumber/rails'
|
|
1910
|
-
|
|
1911
|
-
# this must be after we require cucumber/rails
|
|
1912
|
-
require 'multi_test'
|
|
1913
|
-
MultiTest.disable_autorun
|
|
1914
|
-
```
|
|
1915
|
-
|
|
1916
|
-
The solution comes from: [cucumber/multi_test](https://github.com/cucumber/multi_test/pull/2#issuecomment-21863459)
|
|
1917
|
-
|
|
1918
|
-
#### Queue Mode problems
|
|
1919
|
-
|
|
1920
|
-
##### Why when I use Queue Mode for RSpec then my tests fail?
|
|
1921
|
-
|
|
1922
|
-
knapsack_pro Queue Mode uses `RSpec::Core::Runner` feature that allows [running specs multiple times with different runner options in the same process](https://relishapp.com/rspec/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process).
|
|
1923
|
-
Thanks to that we can run subset of test suite pulled from Knapsack Pro API work queue. This allows dynamic allocation of your tests across CI nodes without reloading whole Ruby/Rails application for each run of test suite subset.
|
|
1924
|
-
|
|
1925
|
-
If you have custom things that are not common in how typical RSpec spec looks like then the RSpec feature won't be able to handle it between test suite subset runs.
|
|
1926
|
-
In that case you need to resolve failed tests in a way that allows RSpec to run the tests. Feel free to [ask me for help](https://knapsackpro.com/contact).
|
|
1927
|
-
|
|
1928
|
-
You can learn more about [recent RSpec team changes](https://github.com/KnapsackPro/knapsack_pro-ruby/pull/42) that was backported into knapsack_pro.
|
|
1929
|
-
|
|
1930
|
-
To solve failing tests in Queue Mode you can check:
|
|
1931
|
-
|
|
1932
|
-
* you use full namespacing. If you see error like `NameError: uninitialized constant MyModule::ModelName` then in some cases a top-level constant would be matched if the code hadn't been loaded for the scoped constant. Try to use full namespacing `::SomeModule::MyModule::ModelName` etc.
|
|
1933
|
-
* you can try to use binary version of knapsack_pro instead of running it via rake task. This helps if your rake tasks mess up with tests and make knapsack_pro Queue Mode fail. [See example](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail):
|
|
1934
|
-
|
|
1935
|
-
```bash
|
|
1936
|
-
# Knapsack Pro Queue Mode run via binary
|
|
1937
|
-
bundle exec knapsack_pro queue:rspec "--profile 10 --format progress"
|
|
1938
|
-
```
|
|
1939
|
-
|
|
1940
|
-
* You can check below questions for common reasons of failing tests in Queue Mode
|
|
1941
|
-
|
|
1942
|
-
##### Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?
|
|
1943
|
-
|
|
1944
|
-
You can use [knapsack_pro binary](#knapsack-pro-binary) instead of rake task version to solve problem:
|
|
1945
|
-
|
|
1946
|
-
```bash
|
|
1947
|
-
# knapsack_pro binary for Queue Mode
|
|
1948
|
-
$ bundle exec knapsack_pro queue:rspec
|
|
1949
|
-
```
|
|
1950
|
-
|
|
1951
|
-
Other solution is to check if your factories for FactoryBot/FactoryGirl use the same methods as Rake DSL and remove problematic part of the code.
|
|
1952
|
-
|
|
1953
|
-
The use of implicit association `task` can cause a problem.
|
|
1954
|
-
|
|
1955
|
-
```ruby
|
|
1956
|
-
# won't work in knapsack_pro Queue Mode
|
|
1957
|
-
FactoryBot.define do
|
|
1958
|
-
factory :assignment do
|
|
1959
|
-
task
|
|
1960
|
-
end
|
|
1961
|
-
end
|
|
1962
|
-
```
|
|
1963
|
-
|
|
1964
|
-
Workaround is to replace `task` with explicit association:
|
|
1965
|
-
|
|
1966
|
-
```ruby
|
|
1967
|
-
# this will work in knapsack_pro Queue Mode
|
|
1968
|
-
FactoryBot.define do
|
|
1969
|
-
factory :assignment do
|
|
1970
|
-
association :task
|
|
1971
|
-
end
|
|
1972
|
-
end
|
|
1973
|
-
```
|
|
1974
|
-
|
|
1975
|
-
##### Why when I use Queue Mode for RSpec then my rake tasks are run twice?
|
|
1976
|
-
|
|
1977
|
-
Why rake tasks are being ran twice in Queue Mode? If you have tests for your rake task then you want to ensure you clear the rake task before loading it inside of test file.
|
|
1978
|
-
In Queue Mode the rake task could be already loaded and loading it again in test file may result in running the task twice.
|
|
1979
|
-
|
|
1980
|
-
```ruby
|
|
1981
|
-
before do
|
|
1982
|
-
# Clear rake task from memory if it was already loaded.
|
|
1983
|
-
# This ensures rake task is loaded only once in knapsack_pro Queue Mode.
|
|
1984
|
-
Rake::Task[task_name].clear if Rake::Task.task_defined?(task_name)
|
|
1985
|
-
|
|
1986
|
-
# loaad rake task only once here
|
|
1987
|
-
Rake.application.rake_require("tasks/dummy")
|
|
1988
|
-
Rake::Task.define_task(:environment)
|
|
1989
|
-
end
|
|
1990
|
-
```
|
|
1991
|
-
|
|
1992
|
-
Here is the full [example how to test rake task along with dummy rake task](https://github.com/KnapsackPro/rails-app-with-knapsack_pro/commit/9f068e900deb3554bd72633e8d61c1cc7f740306) from our example rails project.
|
|
1993
|
-
|
|
1994
|
-
##### Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?
|
|
1995
|
-
|
|
1996
|
-
You may see error like:
|
|
1997
|
-
|
|
1998
|
-
```
|
|
1999
|
-
TypeError:
|
|
2000
|
-
superclass mismatch for class BatchClass
|
|
2001
|
-
```
|
|
2002
|
-
|
|
2003
|
-
when you have 2 test files like this one:
|
|
2004
|
-
|
|
2005
|
-
```ruby
|
|
2006
|
-
# spec/a_spec.rb
|
|
2007
|
-
class BaseBatchClass
|
|
2008
|
-
end
|
|
2009
|
-
|
|
2010
|
-
module Mock
|
|
2011
|
-
module FakeModels
|
|
2012
|
-
class BatchClass < BaseBatchClass
|
|
2013
|
-
def args
|
|
2014
|
-
end
|
|
2015
|
-
end
|
|
2016
|
-
end
|
|
2017
|
-
end
|
|
2018
|
-
|
|
2019
|
-
describe 'A test of something' do
|
|
2020
|
-
it do
|
|
2021
|
-
end
|
|
2022
|
-
end
|
|
2023
|
-
```
|
|
2024
|
-
|
|
2025
|
-
```ruby
|
|
2026
|
-
# spec/b_spec.rb
|
|
2027
|
-
class DifferentBaseBatchClass
|
|
2028
|
-
end
|
|
2029
|
-
|
|
2030
|
-
module Mock
|
|
2031
|
-
module FakeModels
|
|
2032
|
-
# Note the base class is different here!
|
|
2033
|
-
class BatchClass < DifferentBaseBatchClass
|
|
2034
|
-
def args
|
|
2035
|
-
end
|
|
2036
|
-
end
|
|
2037
|
-
end
|
|
2038
|
-
end
|
|
2039
|
-
|
|
2040
|
-
describe 'B test of something' do
|
|
2041
|
-
it do
|
|
2042
|
-
end
|
|
2043
|
-
end
|
|
2044
|
-
```
|
|
2045
|
-
|
|
2046
|
-
Instead of mocking like shown above you could use [RSpec stub_const](https://relishapp.com/rspec/rspec-mocks/docs/mutating-constants) to solve error `superclass mismatch for class BatchClass`.
|
|
2047
|
-
|
|
2048
|
-
##### Why when I use Queue Mode for RSpec then `.rspec` config is ignored?
|
|
2049
|
-
|
|
2050
|
-
The `.rspec` config file is ignored in Queue Mode because knapsack_pro has to pass explicitly arguments to `RSpec::Core::Runner` underhood. You can set your arguments from `.rspec` file in an inline way.
|
|
2051
|
-
|
|
2052
|
-
```
|
|
2053
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --require rails_helper]"
|
|
2054
|
-
```
|
|
2055
|
-
|
|
2056
|
-
See [passing arguments to RSpec](#passing-arguments-to-rspec).
|
|
2057
|
-
|
|
2058
|
-
##### Why I don't see collected time execution data for my build in user dashboard?
|
|
2059
|
-
|
|
2060
|
-
If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open build for your last git commit you should see there info about collected time execution data from all CI nodes. If you don't see collected time execution data for CI nodes then please ensure:
|
|
2061
|
-
|
|
2062
|
-
* you have `Knapsack::Adapters::RspecAdapter.bind` in your `rails_helper.rb` or `spec_helper.rb`
|
|
2063
|
-
* you explicitly set `RAILS_ENV=test` on your CI nodes (for instance you use Docker then please set `RAILS_ENV`)
|
|
2064
|
-
* knapsack_pro Queue Mode saves temporary files with collected time execution data in `your_rails_project/tmp/knapsack_pro/queue/`. Please ensure you don't clean `tmp` directory in your tests so knapsack_pro can publish time execution data to Knapsack Pro API server.
|
|
2065
|
-
|
|
2066
|
-
##### Why all test files have 0.1s time execution for my CI build in user dashboard?
|
|
2067
|
-
|
|
2068
|
-
https://knapsackpro.com/faq/question/why-all-test-files-have-01s-time-execution-for-my-ci-build-in-user-dashboard
|
|
2069
|
-
|
|
2070
|
-
##### Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?
|
|
2071
|
-
|
|
2072
|
-
The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests).
|
|
2073
|
-
|
|
2074
|
-
When you use Queue Mode then knapsack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute.
|
|
2075
|
-
This means RSpec will remember failed tests so far and it will present them at the end of each executed test subset if flag `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false`.
|
|
2076
|
-
You can see the list of all failed test files at the end of knapsack_pro queue mode command.
|
|
2077
|
-
|
|
2078
|
-
##### Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?
|
|
2079
|
-
|
|
2080
|
-
The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests).
|
|
2081
|
-
|
|
2082
|
-
When you use Queue Mode then knapsack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute.
|
|
2083
|
-
This means RSpec will remember pending tests so far and it will present them at the end of each executed test subset if flag `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false`.
|
|
2084
|
-
You can see the list of all pending test files at the end of knapsack_pro queue mode command.
|
|
2085
|
-
|
|
2086
|
-
##### Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?
|
|
2087
|
-
|
|
2088
|
-
No. In Queue Mode the RSpec configuration is updated every time when knapsack_pro gem gets a new set of test files from the Knapsack Pro API and it looks in knapsack_pro output like RSpec was loaded many times but in fact, it loads your project environment only once.
|
|
2089
|
-
|
|
2090
|
-
##### Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?
|
|
2091
|
-
|
|
2092
|
-
This may happen when you use not supported CI provider by knapsack_pro. It's because of missing value of CI build ID. You can set unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` for each CI build. The problem with test suite run again happens when one of your CI node started work later when all other CI nodes already executed whole test suite.
|
|
2093
|
-
The slow CI node that started work late will initialize a new queue hence the tests executed twice.
|
|
2094
|
-
|
|
2095
|
-
To solve this problem you can set `KNAPSACK_PRO_CI_NODE_BUILD_ID` as mentioned above or you can set `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`.
|
|
2096
|
-
Please [read this](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
|
|
2097
|
-
|
|
2098
|
-
##### How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?
|
|
2099
|
-
|
|
2100
|
-
Please use fixed version of capybara-screenshot.
|
|
2101
|
-
|
|
2102
|
-
```ruby
|
|
2103
|
-
# Gemfile
|
|
2104
|
-
group :test do
|
|
2105
|
-
gem 'capybara-screenshot', github: 'mattheworiordan/capybara-screenshot', branch: 'master'
|
|
2106
|
-
end
|
|
2107
|
-
```
|
|
2108
|
-
|
|
2109
|
-
Here is [fix PR](https://github.com/mattheworiordan/capybara-screenshot/pull/205) to official capybara-screenshot repository and the explanation of the problem.
|
|
2110
|
-
|
|
2111
|
-
##### Parallel tests Cucumber and RSpec with Cucumber failures exit CI node early leaving fewer CI nodes to finish RSpec Queue.
|
|
2112
|
-
|
|
2113
|
-
If you run tests in 2 steps like:
|
|
2114
|
-
|
|
2115
|
-
* Step 1. `bundle exec rake knapsack_pro:cucumber` (regular mode)
|
|
2116
|
-
* Step 2. `bundle exec rake knapsack_pro:queue:rspec` (queue mode)
|
|
2117
|
-
|
|
2118
|
-
and your CI provider is configured to fail fast when one of the steps fails then in the case when the first step with Cucumber fails on one of CI nodes then the second step with RSpec in Queue Mode won't start on the CI node that failed fast.
|
|
2119
|
-
|
|
2120
|
-
It means the other CI nodes that will run the second step for RSpec in Queue Mode will consume the whole RSpec Queue so your whole CI build will take more than typical CI build when all Cucumber tests are green.
|
|
2121
|
-
|
|
2122
|
-
You should configure your CI provider to not fail fast the Cucumber step.
|
|
2123
|
-
|
|
2124
|
-
CI providers tips:
|
|
2125
|
-
|
|
2126
|
-
* If you use CircleCI 2.0 you can use `when=always` flag. Read more [here](https://discuss.circleci.com/t/parallel-tests-cuc-rspec-w-failures-exit-early-leaving-less-workers-to-finish/18081).
|
|
2127
|
-
|
|
2128
|
-
##### Why when I reran the same build (same commit hash, etc) on Codeship then no tests would get executed in Queue Mode?
|
|
2129
|
-
|
|
2130
|
-
Codeship uses the same build ID ([`CI_BUILD_NUMBER`](https://documentation.codeship.com/basic/builds-and-configuration/set-environment-variables/#default-environment-variables)) if you re-run a build, so Codeship is not giving enough information to knapsack_pro gem that this is an independent build. Knapsack Pro API assumes you already ran tests for that build ID hence no tests were executed for reran CI build.
|
|
2131
|
-
|
|
2132
|
-
To fix problem you can set `KNAPSACK_PRO_CI_NODE_BUILD_ID=missing-build-id` as empty string.
|
|
2133
|
-
This way knapsack_pro won’t use build ID provided by Codeship and each build will be treated as a unique. This should be good enough solution for most users.
|
|
2134
|
-
|
|
2135
|
-
There is one edge case with that solution. Please note that the knapsack_pro gem doesn't have a CI build ID in order to generate a queue for each particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit.
|
|
2136
|
-
|
|
2137
|
-
To avoid this you should specify a unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`.
|
|
2138
|
-
|
|
2139
|
-
##### Why knapsack_pro hangs / freezes / is stale i.e. for Codeship in Queue Mode?
|
|
2140
|
-
|
|
2141
|
-
Some users with larger projects notice that in Queue Mode knapsack_pro ruby process hangs, for instance for CodeShip CI users.
|
|
2142
|
-
|
|
2143
|
-
It happens due too big RSpec log output in Queue Mode. To produce less logs on output you can change log level to `KNAPSACK_PRO_LOG_LEVEL=warn`.
|
|
2144
|
-
|
|
2145
|
-
Learn more about [log levels](#how-can-i-change-log-level).
|
|
2146
|
-
|
|
2147
|
-
##### How to find seed in RSpec output when I use Queue Mode for RSpec?
|
|
2148
|
-
|
|
2149
|
-
In output for RSpec in knapsack_pro Queue Mode you may see message:
|
|
2150
|
-
|
|
2151
|
-
> INFO -- : [knapsack_pro] To retry in development the subset of tests fetched from API queue please run below command on your machine. If you use --order random then remember to add proper --seed 123 that you will find at the end of rspec command.
|
|
2152
|
-
>
|
|
2153
|
-
> INFO -- : [knapsack_pro] bundle exec rspec --default-path spec "spec/a_spec.rb" "spec/b_spec.rb"
|
|
2154
|
-
|
|
2155
|
-
The seed number is used by RSpec only when you tell it, you need to provide argument `--order random`:
|
|
2156
|
-
|
|
2157
|
-
```bash
|
|
2158
|
-
bundle exec rake "knapsack_pro:queue:rspec[--order random]"
|
|
2159
|
-
```
|
|
2160
|
-
|
|
2161
|
-
then in RSpec output you will see something like:
|
|
2162
|
-
|
|
2163
|
-
```
|
|
2164
|
-
Randomized with seed 11055
|
|
2165
|
-
```
|
|
2166
|
-
|
|
2167
|
-
You can use the seed number to run tests in development:
|
|
2168
|
-
|
|
2169
|
-
```bash
|
|
2170
|
-
bundle exec rspec --seed 11055 --default-path spec "spec/a_spec.rb" "spec/b_spec.rb"
|
|
2171
|
-
```
|
|
2172
|
-
|
|
2173
|
-
If you don't use RSpec argument `--order random` then you don't need to provide `--seed` number when you want to reproduce tests in development.
|
|
2174
|
-
|
|
2175
|
-
##### How to configure puffing-billy gem with Knapsack Pro Queue Mode?
|
|
2176
|
-
|
|
2177
|
-
If you use [puffing-billy](https://github.com/oesmith/puffing-billy) gem you may notice [puffing-billy may crash](https://github.com/oesmith/puffing-billy/issues/253). It happen due to the way how knapsack_pro in Queue Mode uses `RSpec::Core::Runner` ([see](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail)).
|
|
2178
|
-
|
|
2179
|
-
Here is a patch for puffing-billy to make it work in knapsack_pro Queue Mode:
|
|
2180
|
-
|
|
2181
|
-
```ruby
|
|
2182
|
-
# rails_helper.rb or spec_helper.rb
|
|
2183
|
-
|
|
2184
|
-
# A patch to `puffing-billy`'s proxy so that it doesn't try to stop
|
|
2185
|
-
# eventmachine's reactor if it's not running.
|
|
2186
|
-
module BillyProxyPatch
|
|
2187
|
-
def stop
|
|
2188
|
-
return unless EM.reactor_running?
|
|
2189
|
-
super
|
|
2190
|
-
end
|
|
2191
|
-
end
|
|
2192
|
-
Billy::Proxy.prepend(BillyProxyPatch)
|
|
2193
|
-
|
|
2194
|
-
# A patch to `puffing-billy` to start EM if it has been stopped
|
|
2195
|
-
Billy.module_eval do
|
|
2196
|
-
def self.proxy
|
|
2197
|
-
if @billy_proxy.nil? || !(EventMachine.reactor_running? && EventMachine.reactor_thread.alive?)
|
|
2198
|
-
proxy = Billy::Proxy.new
|
|
2199
|
-
proxy.start
|
|
2200
|
-
@billy_proxy = proxy
|
|
2201
|
-
else
|
|
2202
|
-
@billy_proxy
|
|
2203
|
-
end
|
|
2204
|
-
end
|
|
2205
|
-
end
|
|
2206
|
-
|
|
2207
|
-
if ENV["KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC"]
|
|
2208
|
-
KnapsackPro::Hooks::Queue.before_queue do
|
|
2209
|
-
# executes before Queue Mode starts work
|
|
2210
|
-
Billy.proxy.start
|
|
2211
|
-
end
|
|
2212
|
-
|
|
2213
|
-
KnapsackPro::Hooks::Queue.after_queue do
|
|
2214
|
-
# executes after Queue Mode finishes work
|
|
2215
|
-
Billy.proxy.stop
|
|
2216
|
-
end
|
|
2217
|
-
end
|
|
2218
|
-
```
|
|
2219
|
-
|
|
2220
|
-
### General questions
|
|
2221
|
-
|
|
2222
|
-
#### How to run tests for particular CI node in your development environment
|
|
2223
|
-
|
|
2224
|
-
##### for knapsack_pro regular mode
|
|
2225
|
-
|
|
2226
|
-
In your development environment you can debug tests that were run on the particular CI node.
|
|
2227
|
-
For instance to run subset of tests for the first CI node with specified seed you can do.
|
|
2228
|
-
|
|
2229
|
-
```bash
|
|
2230
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
|
|
2231
|
-
KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
|
|
2232
|
-
KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
|
|
2233
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 \
|
|
2234
|
-
KNAPSACK_PRO_CI_NODE_INDEX=0 \
|
|
2235
|
-
bundle exec rake "knapsack_pro:rspec[--seed 123]"
|
|
2236
|
-
```
|
|
2237
|
-
|
|
2238
|
-
Above example is for RSpec. You can use respectively rake task name and token environment variable when you want to run tests for minitest, test_unit, cucumber or spinach.
|
|
2239
|
-
It should work when all CI nodes finished work and sent time execution data to Knapsack Pro API.
|
|
2240
|
-
You can visit [user dashboard](https://knapsackpro.com/dashboard) to preview particular CI build and ensure time execution data were collected from all CI nodes.
|
|
2241
|
-
If at least one CI node has not sent time execution data to the Knapsack Pro API then you should check below solution.
|
|
2242
|
-
|
|
2243
|
-
Check test runner output on particular CI node you would like to retry in development. You should see at the beginning of rspec command an output that can
|
|
2244
|
-
be copied and executed in development.
|
|
2245
|
-
|
|
2246
|
-
```
|
|
2247
|
-
/Users/ubuntu/.rvm/gems/ruby-2.4.0/gems/rspec-core-3.4.4/exe/rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec
|
|
2248
|
-
```
|
|
2249
|
-
|
|
2250
|
-
Command similar to above can be executed in your development this way:
|
|
2251
|
-
|
|
2252
|
-
```bash
|
|
2253
|
-
bundle exec rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec
|
|
2254
|
-
```
|
|
2255
|
-
|
|
2256
|
-
If you were running your tests with `--order random` on your CI then you can additionaly pass seed param with proper value in above command (`--seed 123`).
|
|
2257
|
-
|
|
2258
|
-
##### for knapsack_pro queue mode
|
|
2259
|
-
|
|
2260
|
-
There are a few ways to reproduce tests executed on CI node in your development environment.
|
|
2261
|
-
|
|
2262
|
-
* At the end of `knapsack_pro:queue:rspec` results you will find example of command that you can copy and paste to your development machine. It will run all tests executed on the CI node in a single run. I recommend this approach.
|
|
2263
|
-
|
|
2264
|
-
* For each intermediate request to Knapsack Pro API queue you will also find example of command to run a subset of tests fetched from API. This might be helpful when you use `--order random` for rspec and you would like to reproduce the tests with the same seed.
|
|
2265
|
-
|
|
2266
|
-
* You can also retry tests and record the time execution data for them again for the particular CI node. Note you must be checkout on the same branch and git commit as your CI node was.
|
|
2267
|
-
|
|
2268
|
-
To retry the particular CI node do this on your machine:
|
|
2269
|
-
|
|
2270
|
-
```bash
|
|
2271
|
-
RACK_ENV=test \
|
|
2272
|
-
RAILS_ENV=test \
|
|
2273
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
|
|
2274
|
-
KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
|
|
2275
|
-
KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
|
|
2276
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 \
|
|
2277
|
-
KNAPSACK_PRO_CI_NODE_INDEX=0 \
|
|
2278
|
-
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
|
|
2279
|
-
bundle exec rake "knapsack_pro:queue:rspec"
|
|
2280
|
-
```
|
|
2281
|
-
|
|
2282
|
-
If you were running your tests with `--order random` on your CI like this:
|
|
2283
|
-
|
|
2284
|
-
```bash
|
|
2285
|
-
bundle exec rake "knapsack_pro:queue:rspec[--order random]"
|
|
2286
|
-
```
|
|
2287
|
-
|
|
2288
|
-
Then you can find the seed number visible in rspec output:
|
|
2289
|
-
|
|
2290
|
-
(...)
|
|
2291
|
-
Randomized with seed 123
|
|
2292
|
-
|
|
2293
|
-
You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node:
|
|
2294
|
-
|
|
2295
|
-
```bash
|
|
2296
|
-
RACK_ENV=test \
|
|
2297
|
-
RAILS_ENV=test \
|
|
2298
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
|
|
2299
|
-
KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
|
|
2300
|
-
KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
|
|
2301
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=2 \
|
|
2302
|
-
KNAPSACK_PRO_CI_NODE_INDEX=0 \
|
|
2303
|
-
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
|
|
2304
|
-
bundle exec rake "knapsack_pro:queue:rspec[--seed 123]"
|
|
2305
|
-
```
|
|
2306
|
-
|
|
2307
|
-
#### What happens when Knapsack Pro API is not available/not reachable temporarily?
|
|
2308
|
-
|
|
2309
|
-
##### for knapsack_pro regular mode
|
|
2310
|
-
|
|
2311
|
-
knapsack_pro gem will retry requests to Knapsack Pro API multiple times every few seconds till it switches to fallback behavior (Fallback Mode) and it will split test files across CI nodes based on popular test directory names. When knapsack_pro starts Fallback Mode then you will see a warning in the output.
|
|
2312
|
-
|
|
2313
|
-
Note there is an unlikely scenario when some of the CI nodes may start in Fallback Mode but others don't and then it could happen that some of test files might be skipped. You should [read this to learn more](https://github.com/KnapsackPro/knapsack_pro-ruby/pull/124) and decide if you like to use Fallback Mode when running tests with knapsack_pro Regular Mode.
|
|
2314
|
-
|
|
2315
|
-
If your CI provider allows to retry only one of parallel CI nodes then please [read about this edge case as well](#required-ci-configuration-if-you-use-retry-single-failed-ci-node-feature-on-your-ci-server-when-knapsack_pro_fixed_queue_splittrue-in-queue-mode-or-knapsack_pro_fixed_test_suite_splittrue-in-regular-mode).
|
|
2316
|
-
|
|
2317
|
-
##### for knapsack_pro queue mode
|
|
2318
|
-
|
|
2319
|
-
knapsack_pro gem will retry requests to Knapsack Pro API multiple times every few seconds till it switches to fallback behavior (Fallback Mode) and it will split test files across CI nodes based on popular test directory names.
|
|
2320
|
-
|
|
2321
|
-
Note that if one of the CI nodes loses connection to Knapsack Pro API but others don't, then some of the test files may be executed on multiple CI nodes. **Fallback Mode guarantees each of the test files is run at least once across CI nodes when you use knapsack_pro in Queue Mode.** Thanks to that we know if the whole test suite is green or not. When knapsack_pro starts Fallback Mode then you will see a warning in the output.
|
|
2322
|
-
|
|
2323
|
-
If your CI provider allows to retry only one of parallel CI nodes then please [read about this edge case as well](#required-ci-configuration-if-you-use-retry-single-failed-ci-node-feature-on-your-ci-server-when-knapsack_pro_fixed_queue_splittrue-in-queue-mode-or-knapsack_pro_fixed_test_suite_splittrue-in-regular-mode).
|
|
2324
|
-
|
|
2325
|
-
#### How can I change log level?
|
|
2326
|
-
|
|
2327
|
-
You can change log level by specifying the `KNAPSACK_PRO_LOG_LEVEL` environment variable.
|
|
2328
|
-
|
|
2329
|
-
KNAPSACK_PRO_LOG_LEVEL=info bundle exec rake knapsack_pro:rspec
|
|
2330
|
-
|
|
2331
|
-
Available values are `debug` (default), `info`, `warn`, `error` and `fatal`.
|
|
2332
|
-
|
|
2333
|
-
Recommended log levels you can use:
|
|
2334
|
-
|
|
2335
|
-
* `debug` is default log level and it is recommended to log details about requests to Knapsack Pro API. Thanks to that you can debug things or ensure everything works. For instance in [user dashboard](https://knapsackpro.com/dashboard) you can find tips referring to debug logs.
|
|
2336
|
-
* `info` level shows message like how to retry tests in development or info why something works this way or the other (for instance why tests were not executed on the CI node). You can use `info` level when you really don't want to see all debug messages from default log level.
|
|
2337
|
-
|
|
2338
|
-
#### How to write knapsack_pro logs to a file?
|
|
2339
|
-
|
|
2340
|
-
##### set directory where to write log file (option 1 - recommended)
|
|
2341
|
-
|
|
2342
|
-
Set `KNAPSACK_PRO_LOG_DIR=log` environment variable in order to notify knapsack_pro gem to write logs to `log` directory instead of stdout.
|
|
2343
|
-
If you have Rails project then this should work for you.
|
|
2344
|
-
|
|
2345
|
-
knapsack_pro will create a file with CI node index in name. For instance if you run tests on 2 CI nodes:
|
|
2346
|
-
|
|
2347
|
-
* `log/knapsack_pro_node_0.log`
|
|
2348
|
-
* `log/knapsack_pro_node_1.log`
|
|
2349
|
-
|
|
2350
|
-
`KNAPSACK_PRO_LOG_DIR` has higher priority than custom log set in `rails_helper.rb` as shown below (option 2).
|
|
2351
|
-
|
|
2352
|
-
You can change log level with [KNAPSACK_PRO_LOG_LEVEL environment variable](#how-can-i-change-log-level).
|
|
2353
|
-
|
|
2354
|
-
##### set custom logger config (option 2)
|
|
2355
|
-
|
|
2356
|
-
In your `rails_helper.rb` you can set custom Knapsack Pro logger and write to custom log file.
|
|
2357
|
-
|
|
2358
|
-
```ruby
|
|
2359
|
-
# Ensure you load Rails before using Rails const below.
|
|
2360
|
-
# This line should be already in your rails_helper.rb
|
|
2361
|
-
require File.expand_path('../../config/environment', __FILE__)
|
|
2362
|
-
|
|
2363
|
-
require 'logger'
|
|
2364
|
-
KnapsackPro.logger = Logger.new(Rails.root.join('log', "knapsack_pro_node_#{KnapsackPro::Config::Env.ci_node_index}.log"))
|
|
2365
|
-
KnapsackPro.logger.level = Logger::DEBUG
|
|
2366
|
-
```
|
|
2367
|
-
|
|
2368
|
-
Note if you run knapsack_pro then the very first request to Knapsack Pro API still will be shown to stdout because we need to have set of test files needed to run RSpec before we load `rails_helper.rb` where the configuration of logger actually is loaded for the first time.
|
|
2369
|
-
|
|
2370
|
-
That is why you may prefer to use option 1 instead of this.
|
|
2371
|
-
|
|
2372
|
-
##### How to preserve logs on my CI after CI build completed?
|
|
2373
|
-
|
|
2374
|
-
Follow this tip if you use one of above options to write knapsack_pro log to the file.
|
|
2375
|
-
|
|
2376
|
-
If you would like to keep knapsack_pro logs after your CI build finished then you could use artifacts or some cache mechanize for your CI provider.
|
|
2377
|
-
|
|
2378
|
-
For instance, for [CircleCI 2.0 artifacts](https://circleci.com/docs/2.0/artifacts/) you can specify log directory:
|
|
2379
|
-
|
|
2380
|
-
```yaml
|
|
2381
|
-
- run:
|
|
2382
|
-
name: RSpec via knapsack_pro Queue Mode
|
|
2383
|
-
command: |
|
|
2384
|
-
# export word is important here!
|
|
2385
|
-
export RAILS_ENV=test
|
|
2386
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation]"
|
|
2387
|
-
|
|
2388
|
-
- store_artifacts:
|
|
2389
|
-
path: log
|
|
2390
|
-
```
|
|
2391
|
-
|
|
2392
|
-
Now you can preview logs in `Artifacts` tab in the Circle CI build view.
|
|
2393
|
-
|
|
2394
|
-
#### How to split tests based on test level instead of test file level?
|
|
2395
|
-
|
|
2396
|
-
If you want to split one big test file (test file with long time execution) across multiple CI nodes then you can [check this tip](#split-test-files-by-test-cases) or use other methods like:
|
|
2397
|
-
|
|
2398
|
-
##### A. Create multiple small test files
|
|
2399
|
-
|
|
2400
|
-
Create multiple small test files instead of one long running test file with many test cases.
|
|
2401
|
-
A lot of small test files will give you better test suite split results.
|
|
2402
|
-
|
|
2403
|
-
##### B. Use tags to mark set of tests in particular test file
|
|
2404
|
-
|
|
2405
|
-
Another way is to use tags to mark subset of tests in particular test file and then split tests based on tags.
|
|
2406
|
-
|
|
2407
|
-
This example is for knapsack_pro Regular Mode. You can also use knapsack_pro Queue Mode with tags.
|
|
2408
|
-
|
|
2409
|
-
Here is example of test file with specified tags for describe groups:
|
|
2410
|
-
|
|
2411
|
-
```ruby
|
|
2412
|
-
# spec/features/something_spec.rb
|
|
2413
|
-
describe 'Feature' do
|
|
2414
|
-
describe 'something A', :tagA do
|
|
2415
|
-
it {}
|
|
2416
|
-
it 'another test' {}
|
|
2417
|
-
end
|
|
2418
|
-
|
|
2419
|
-
describe 'something B', :tagB do
|
|
2420
|
-
it {}
|
|
2421
|
-
end
|
|
2422
|
-
|
|
2423
|
-
describe 'something else' do
|
|
2424
|
-
it {}
|
|
2425
|
-
end
|
|
2426
|
-
end
|
|
2427
|
-
```
|
|
2428
|
-
|
|
2429
|
-
You need to create API token per each tag. In this example we need 3 different API tokens.
|
|
2430
|
-
|
|
2431
|
-
You need to run below commands for each CI node.
|
|
2432
|
-
|
|
2433
|
-
```bash
|
|
2434
|
-
# run only tests with tagA
|
|
2435
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagA bundle exec rake "knapsack_pro:rspec[--tag tagA]"
|
|
2436
|
-
|
|
2437
|
-
# run only tests with tagB
|
|
2438
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagB bundle exec rake "knapsack_pro:rspec[--tag tagB]"
|
|
2439
|
-
|
|
2440
|
-
# run other tests without tag A & B
|
|
2441
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tests_without_tags_A_and_B bundle exec rake "knapsack_pro:rspec[--tag ~tagA --tag ~tagB]"
|
|
2442
|
-
```
|
|
2443
|
-
|
|
2444
|
-
#### How to make knapsack_pro works for forked repositories of my project?
|
|
2445
|
-
|
|
2446
|
-
Imagine one of the scenarios, for this example I use the Travis-CI.
|
|
2447
|
-
|
|
2448
|
-
* We don’t want to have secrets like the `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in `.travis.yml` in the codebase, because that code is also distributed to clients.
|
|
2449
|
-
* Adding it as env variables to Travis itself is tricky: It has to work for pull requests from developer’s forks into our main fork; this conflicts with the way Travis handles secrets. We also need a fallback if the token is not provided (when developers do builds within their own fork).
|
|
2450
|
-
|
|
2451
|
-
The solution for this problem is to set `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as env variables in Travis for our main project.
|
|
2452
|
-
This won't be accessible on forked repositories so we will run knapsack_pro in fallback mode there.
|
|
2453
|
-
This way forked repositories have working test suite but without optimal test suite split across CI nodes.
|
|
2454
|
-
|
|
2455
|
-
Create the file `bin/knapsack_pro_rspec` with executable chmod in your main project repository.
|
|
2456
|
-
Below example is for rspec. You can change `$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `$KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` if you use cucumber etc.
|
|
2457
|
-
|
|
2458
|
-
```bash
|
|
2459
|
-
#!/bin/bash
|
|
2460
|
-
if [ "$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC" = "" ]; then
|
|
2461
|
-
KNAPSACK_PRO_ENDPOINT=https://api-disabled-for-fork.knapsackpro.com \
|
|
2462
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=disabled-for-fork \
|
|
2463
|
-
KNAPSACK_PRO_MAX_REQUEST_RETRIES=0 \
|
|
2464
|
-
bundle exec rake knapsack_pro:rspec # use Regular Mode here always
|
|
2465
|
-
else
|
|
2466
|
-
# Regular Mode
|
|
2467
|
-
bundle exec rake knapsack_pro:rspec
|
|
2468
|
-
|
|
2469
|
-
# or you can use Queue Mode instead of Regular Mode if you like
|
|
2470
|
-
# bundle exec rake knapsack_pro:queue:rspec
|
|
2471
|
-
fi
|
|
2472
|
-
```
|
|
2473
|
-
|
|
2474
|
-
Now you can use `bin/knapsack_pro_rspec` command instead of `bundle exec rake knapsack_pro:rspec`.
|
|
2475
|
-
Remember to follow other steps required for your CI provider.
|
|
2476
|
-
|
|
2477
|
-
#### How to use junit formatter?
|
|
2478
|
-
|
|
2479
|
-
##### How to use junit formatter with knapsack_pro regular mode?
|
|
2480
|
-
|
|
2481
|
-
You can use junit formatter for rspec thanks to gem [rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter).
|
|
2482
|
-
Here you can find example how to generate `rspec.xml` file with junit format and at the same time show normal documentation format output for RSpec.
|
|
2483
|
-
|
|
2484
|
-
```bash
|
|
2485
|
-
# Regular Mode
|
|
2486
|
-
bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"
|
|
2487
|
-
```
|
|
2488
|
-
|
|
2489
|
-
##### How to use junit formatter with knapsack_pro queue mode?
|
|
2490
|
-
|
|
2491
|
-
You can use junit formatter for rspec thanks to gem [rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter).
|
|
2492
|
-
|
|
2493
|
-
```bash
|
|
2494
|
-
# Queue Mode
|
|
2495
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"
|
|
2496
|
-
```
|
|
2497
|
-
|
|
2498
|
-
The xml report will contain all tests executed across intermediate test subset runs based on work queue. You need to add after subset queue hook to rename `rspec.xml` to `rspec_final_results.xml` thanks to that the final results file will contain only single xml tag with all tests executed on the CI node. This is related to the way how queue mode works. Detailed explanation is in the [issue](https://github.com/KnapsackPro/knapsack_pro-ruby/issues/40).
|
|
2499
|
-
|
|
2500
|
-
```ruby
|
|
2501
|
-
# spec_helper.rb or rails_helper.rb
|
|
2502
|
-
|
|
2503
|
-
# TODO This must be the same path as value for rspec --out argument
|
|
2504
|
-
# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead.
|
|
2505
|
-
TMP_RSPEC_XML_REPORT = 'tmp/rspec.xml'
|
|
2506
|
-
# move results to FINAL_RSPEC_XML_REPORT so the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT
|
|
2507
|
-
FINAL_RSPEC_XML_REPORT = 'tmp/rspec_final_results.xml'
|
|
2508
|
-
|
|
2509
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2510
|
-
if File.exist?(TMP_RSPEC_XML_REPORT)
|
|
2511
|
-
FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT)
|
|
2512
|
-
end
|
|
2513
|
-
end
|
|
2514
|
-
```
|
|
2515
|
-
|
|
2516
|
-
###### How to use junit formatter with knapsack_pro queue mode when CI nodes use common local drive?
|
|
2517
|
-
|
|
2518
|
-
Note if you use a CI provider or your own CI solution that uses common local drive for all parallel CI nodes then above solution needs to be adjusted to produce report file with CI node index number in the file name to avoid file conflicts. Example file name with CI node index number: `tmp/rspec_final_results_N.xml`.
|
|
2519
|
-
|
|
2520
|
-
```bash
|
|
2521
|
-
# Queue Mode
|
|
2522
|
-
|
|
2523
|
-
# must be exported to read correctly the value in below knapsack_pro command
|
|
2524
|
-
export KNAPSACK_PRO_CI_NODE_INDEX=0
|
|
2525
|
-
# if your CI provider exposes CI node index under other environment variable name then you could use it instead
|
|
2526
|
-
|
|
2527
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.xml]"
|
|
2528
|
-
```
|
|
2529
|
-
|
|
2530
|
-
In below code we use CI node index number in `TMP_RSPEC_XML_REPORT` and `FINAL_RSPEC_XML_REPORT`:
|
|
2531
|
-
|
|
2532
|
-
```ruby
|
|
2533
|
-
# spec_helper.rb or rails_helper.rb
|
|
2534
|
-
|
|
2535
|
-
# TODO This must be the same path as value for rspec --out argument
|
|
2536
|
-
# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead.
|
|
2537
|
-
TMP_RSPEC_XML_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
|
|
2538
|
-
# move results to FINAL_RSPEC_XML_REPORT so the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT
|
|
2539
|
-
FINAL_RSPEC_XML_REPORT = "tmp/rspec_final_results_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
|
|
2540
|
-
|
|
2541
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2542
|
-
if File.exist?(TMP_RSPEC_XML_REPORT)
|
|
2543
|
-
FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT)
|
|
2544
|
-
end
|
|
2545
|
-
end
|
|
2546
|
-
```
|
|
2547
|
-
|
|
2548
|
-
###### Why `tmp/rspec_final_results.xml` is corrupted when I use junit formatter with knapsack_pro queue mode?
|
|
2549
|
-
|
|
2550
|
-
The `tmp/rspec_final_results.xml` might be corrupted due syntax error in your test suite. First check if your test suite is green.
|
|
2551
|
-
Another reason might be that you did not configure the junit formatter as shown in the example for Queue Mode. Please check above 2 questions & answers explaing that.
|
|
2552
|
-
|
|
2553
|
-
###### How to use junit formatter with knapsack_pro queue mode in Cucumber?
|
|
2554
|
-
|
|
2555
|
-
Please provide in `--out` argument directory path where xml files for each test file will be created. It must be a directory in order to work in Queue Mode because in Queue Mode the Cucumber test runner is executed multiple times.
|
|
2556
|
-
Each time for set of tests fetched from Queue so it means multiple xml files will be created in junit format.
|
|
2557
|
-
|
|
2558
|
-
```bash
|
|
2559
|
-
bundle exec rake "knapsack_pro:queue:cucumber[--format junit --out tmp/test-reports/cucumber/queue_mode/]"
|
|
2560
|
-
```
|
|
2561
|
-
|
|
2562
|
-
#### How to use JSON formatter for RSpec?
|
|
2563
|
-
|
|
2564
|
-
##### How to use RSpec JSON formatter with knapsack_pro Queue Mode?
|
|
2565
|
-
|
|
2566
|
-
You need to specify `format` and `out` argument (it's important to provide both).
|
|
2567
|
-
|
|
2568
|
-
```bash
|
|
2569
|
-
# Queue Mode
|
|
2570
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec.json]"
|
|
2571
|
-
```
|
|
2572
|
-
|
|
2573
|
-
The JSON report will contain all tests executed across intermediate test subset runs based on work queue. You need to add after subset queue hook to rename `rspec.json` to `rspec_final_results.json` thanks to that the final results file will contain valid json with all tests executed on the CI node. This is related to the way how Queue Mode works. Detailed explanation is in the [issue](https://github.com/KnapsackPro/knapsack_pro-ruby/issues/40).
|
|
2574
|
-
|
|
2575
|
-
```ruby
|
|
2576
|
-
# spec_helper.rb or rails_helper.rb
|
|
2577
|
-
|
|
2578
|
-
# TODO This must be the same path as value for rspec --out argument
|
|
2579
|
-
# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.json may not work. Please use full path instead.
|
|
2580
|
-
TMP_RSPEC_JSON_REPORT = 'tmp/rspec.json'
|
|
2581
|
-
# move results to FINAL_RSPEC_JSON_REPORT so the results won't accumulate with duplicated JSON in TMP_RSPEC_JSON_REPORT
|
|
2582
|
-
FINAL_RSPEC_JSON_REPORT = 'tmp/rspec_final_results.json'
|
|
2583
|
-
|
|
2584
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2585
|
-
if File.exist?(TMP_RSPEC_JSON_REPORT)
|
|
2586
|
-
FileUtils.mv(TMP_RSPEC_JSON_REPORT, FINAL_RSPEC_JSON_REPORT)
|
|
2587
|
-
end
|
|
2588
|
-
end
|
|
2589
|
-
```
|
|
2590
|
-
|
|
2591
|
-
###### How to use RSpec JSON formatter with knapsack_pro Queue Mode when CI nodes use common local drive?
|
|
2592
|
-
|
|
2593
|
-
Note if you use a CI provider or your own CI solution that uses common local drive for all parallel CI nodes then above solution needs to be adjusted to produce report file with CI node index number in the file name to avoid file conflicts. Example file name with CI node index number: `tmp/rspec_final_results_N.json`.
|
|
2594
|
-
|
|
2595
|
-
```
|
|
2596
|
-
# Queue Mode
|
|
2597
|
-
|
|
2598
|
-
# must be exported to read correctly the value in below knapsack_pro command
|
|
2599
|
-
export KNAPSACK_PRO_CI_NODE_INDEX=0
|
|
2600
|
-
# if your CI provider exposes CI node index under other environment variable name then you could use it instead
|
|
2601
|
-
|
|
2602
|
-
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.json]"
|
|
2603
|
-
```
|
|
2604
|
-
|
|
2605
|
-
In below code we use CI node index number in `TMP_RSPEC_JSON_REPORT` and `FINAL_RSPEC_JSON_REPORT`:
|
|
2606
|
-
|
|
2607
|
-
```ruby
|
|
2608
|
-
# spec_helper.rb or rails_helper.rb
|
|
2609
|
-
|
|
2610
|
-
# TODO This must be the same path as value for rspec --out argument
|
|
2611
|
-
# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.json may not work. Please use full path instead.
|
|
2612
|
-
TMP_RSPEC_JSON_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
|
|
2613
|
-
# move results to FINAL_RSPEC_JSON_REPORT so the results won't accumulate with duplicated JSON in TMP_RSPEC_JSON_REPORT
|
|
2614
|
-
FINAL_RSPEC_JSON_REPORT = "tmp/rspec_final_results_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
|
|
2615
|
-
|
|
2616
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2617
|
-
if File.exist?(TMP_RSPEC_JSON_REPORT)
|
|
2618
|
-
FileUtils.mv(TMP_RSPEC_JSON_REPORT, FINAL_RSPEC_JSON_REPORT)
|
|
2619
|
-
end
|
|
2620
|
-
end
|
|
2621
|
-
```
|
|
2622
|
-
|
|
2623
|
-
#### How many API keys I need?
|
|
2624
|
-
|
|
2625
|
-
Basically you need as many API keys as you have steps in your build.
|
|
2626
|
-
|
|
2627
|
-
Here is example:
|
|
2628
|
-
|
|
2629
|
-
* Step 1. API_KEY_A for `bundle exec rake knapsack_pro:cucumber`
|
|
2630
|
-
* Step 2. API_KEY_B for `bundle exec rake knapsack_pro:rspec`
|
|
2631
|
-
* Step 3. API_KEY_C for `KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/*_spec.rb" bundle exec rake knapsack_pro:rspec`
|
|
2632
|
-
* Step 4. API_KEY_D for `bundle exec rake knapsack_pro:rspec[--tag tagA]`
|
|
2633
|
-
* Step 5. API_KEY_E for `bundle exec rake knapsack_pro:rspec[--tag ~tagA]`
|
|
2634
|
-
* Step 6. API_KEY_F for `bundle exec rake knapsack_pro:queue:rspec`
|
|
2635
|
-
|
|
2636
|
-
Note:
|
|
2637
|
-
|
|
2638
|
-
* If you specified `KNAPSACK_PRO_TEST_FILE_PATTERN` then you run subset of your whole test suite hence you need separate API key because we want to track only tests for this subset.
|
|
2639
|
-
* If you pass `--tag tagA` or `--tag ~tagA` then you run subset of your whole test suite hence you need separate API key.
|
|
2640
|
-
* If you use regular or queue mode then you need separate API key for each mode.
|
|
2641
|
-
|
|
2642
|
-
#### What is optimal order of test commands?
|
|
2643
|
-
|
|
2644
|
-
__Tip 1:__
|
|
2645
|
-
|
|
2646
|
-
I recommend to run first the test commands in the regular mode and later the commands in the queue mode.
|
|
2647
|
-
|
|
2648
|
-
* Step 1. `bundle exec rake knapsack_pro:cucumber` (regular mode)
|
|
2649
|
-
* Step 2. `bundle exec rake knapsack_pro:queue:rspec` (queue mode)
|
|
2650
|
-
|
|
2651
|
-
Thanks to that when for some reason the tests executed for cucumber in regular mode will not be well balanced across CI nodes (for instance when one of CI node has bad performance) then the rspec tests executed later in the queue mode will autobalance your build.
|
|
2652
|
-
|
|
2653
|
-
__Tip 2:__
|
|
2654
|
-
|
|
2655
|
-
When you have short test suite, for instance in javascript then you could distribute tests this way:
|
|
2656
|
-
|
|
2657
|
-
* CI 0
|
|
2658
|
-
* Step 1: `npm test`
|
|
2659
|
-
* Step 2: `bundle exec rake knapsack_pro:queue:rspec`
|
|
2660
|
-
|
|
2661
|
-
* CI 1
|
|
2662
|
-
* Step 1: `bundle exec rake knapsack_pro:queue:rspec`
|
|
2663
|
-
|
|
2664
|
-
You will run your javascript tests on single CI node and the knapsack_pro will auto-balance CI build with Queue Mode. Thanks to that CI build time execution will be flat and optimal (as fast as possible).
|
|
2665
|
-
|
|
2666
|
-
#### How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?
|
|
2667
|
-
|
|
2668
|
-
##### percy-capybara gem version < 4 (old)
|
|
2669
|
-
|
|
2670
|
-
Some tools like [Percy.io](https://percy.io/docs/clients/ruby/capybara-rails) requires to set hooks for RSpec `before(:suite)` and `after(:suite)`.
|
|
2671
|
-
Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will execute multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way:
|
|
2672
|
-
|
|
2673
|
-
```ruby
|
|
2674
|
-
# spec_helper.rb or rails_helper.rb
|
|
2675
|
-
# step for percy-capybara gem version < 4
|
|
2676
|
-
|
|
2677
|
-
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
|
|
2678
|
-
# executes before Queue Mode starts work
|
|
2679
|
-
Percy::Capybara.initialize_build
|
|
2680
|
-
end
|
|
2681
|
-
|
|
2682
|
-
KnapsackPro::Hooks::Queue.after_queue do |queue_id|
|
|
2683
|
-
# executes after Queue Mode finishes work
|
|
2684
|
-
Percy::Capybara.finalize_build
|
|
2685
|
-
end
|
|
2686
|
-
```
|
|
2687
|
-
|
|
2688
|
-
##### percy-capybara gem version >= 4
|
|
2689
|
-
|
|
2690
|
-
If you use [percy-capybara 4.x](https://docs.percy.io/v1/docs/capybara) then you don't need to set RSpec hooks. Insted you need to run knapsack_pro via percy npm command.
|
|
2691
|
-
|
|
2692
|
-
```
|
|
2693
|
-
npx percy exec -- rake knapsack_pro:queue:rspec
|
|
2694
|
-
|
|
2695
|
-
# or you can use knapsack_pro binary version instead of rake task
|
|
2696
|
-
npx percy exec -- knapsack_pro queue:rspec
|
|
2697
|
-
```
|
|
2698
|
-
|
|
2699
|
-
Read more about [knapsack_pro binary version](#knapsack-pro-binary).
|
|
2700
|
-
|
|
2701
|
-
Also you need to follow [Percy step for parallelism](https://docs.percy.io/docs/parallel-test-suites#section-manual-configuration-with-environment-variables).
|
|
2702
|
-
|
|
2703
|
-
* `PERCY_PARALLEL_NONCE` - A unique identifier for this build. This can be anything, but it must be the same across parallel build nodes. Usually, this is just the CI build number or a shared timestamp. You can google environment variables for CI provider you use to check what's the env var for build ID.
|
|
2704
|
-
|
|
2705
|
-
You can also find CI build number for your CI provider in [knapsack_pro source code](https://github.com/KnapsackPro/knapsack_pro-ruby/tree/master/lib/knapsack_pro/config/ci). knapsack_pro has built in environment variables integration for various CI providers. See for example [CircleCI](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/config/ci/circle.rb) - look for method `node_build_id`.
|
|
2706
|
-
|
|
2707
|
-
```bash
|
|
2708
|
-
# example for using CircleCI build ID
|
|
2709
|
-
export PERCY_PARALLEL_NONCE=$CIRCLE_BUILD_NUM
|
|
2710
|
-
```
|
|
2711
|
-
|
|
2712
|
-
* `PERCY_PARALLEL_TOTAL` - The total number of parallel build nodes.
|
|
2713
|
-
|
|
2714
|
-
#### How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?
|
|
2715
|
-
|
|
2716
|
-
Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will be executed multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way:
|
|
2717
|
-
|
|
2718
|
-
```ruby
|
|
2719
|
-
# spec_helper.rb or rails_helper.rb
|
|
2720
|
-
|
|
2721
|
-
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
|
|
2722
|
-
# This will be called only once before the tests started on the CI node.
|
|
2723
|
-
# It will be run inside of the RSpec before(:suite) block only once.
|
|
2724
|
-
# It means you will have access to whatever RSpec provides in the context of the before(:suite) block.
|
|
2725
|
-
end
|
|
2726
|
-
|
|
2727
|
-
KnapsackPro::Hooks::Queue.after_queue do |queue_id|
|
|
2728
|
-
# This will be called only once after test suite is completed.
|
|
2729
|
-
# Note this hook won't be called inside of RSpec after(:suite) block because
|
|
2730
|
-
# we are not able to determine which after(:suite) block will be called as the last one
|
|
2731
|
-
# due to the fact the Knapsack Pro Queue Mode allocates tests in dynamic way.
|
|
2732
|
-
end
|
|
2733
|
-
```
|
|
2734
|
-
|
|
2735
|
-
#### What hooks are supported in Queue Mode?
|
|
2736
|
-
|
|
2737
|
-
Note: Each hook type can be defined multiple times. For instance, if you define `KnapsackPro::Hooks::Queue.before_queue` twice then both block of code will be called when running your tests.
|
|
2738
|
-
|
|
2739
|
-
* RSpec in knapsack_pro Queue Mode supports hooks:
|
|
2740
|
-
|
|
2741
|
-
```ruby
|
|
2742
|
-
# spec_helper.rb or rails_helper.rb
|
|
2743
|
-
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
|
|
2744
|
-
print 'Before Queue Hook - run before test suite'
|
|
2745
|
-
end
|
|
2746
|
-
|
|
2747
|
-
# this will be run after set of tests fetched from Queue has been executed
|
|
2748
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2749
|
-
print 'After Subset Queue Hook - run after subset of test suite'
|
|
2750
|
-
end
|
|
2751
|
-
|
|
2752
|
-
KnapsackPro::Hooks::Queue.after_queue do |queue_id|
|
|
2753
|
-
print 'After Queue Hook - run after test suite'
|
|
2754
|
-
end
|
|
2755
|
-
```
|
|
2756
|
-
|
|
2757
|
-
* Minitest in knapsack_pro Queue Mode supports hooks:
|
|
2758
|
-
|
|
2759
|
-
```ruby
|
|
2760
|
-
# test/test_helper.rb
|
|
2761
|
-
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
|
|
2762
|
-
print 'Before Queue Hook - run before test suite'
|
|
2763
|
-
end
|
|
2764
|
-
|
|
2765
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2766
|
-
print 'After Subset Queue Hook - run after subset of test suite'
|
|
2767
|
-
end
|
|
2768
|
-
|
|
2769
|
-
KnapsackPro::Hooks::Queue.after_queue do |queue_id|
|
|
2770
|
-
print 'After Queue Hook - run after test suite'
|
|
2771
|
-
end
|
|
2772
|
-
```
|
|
2773
|
-
|
|
2774
|
-
* Cucumber in knapsack_pro Queue Mode supports hooks:
|
|
2775
|
-
|
|
2776
|
-
```ruby
|
|
2777
|
-
# features/support/knapsack_pro.rb
|
|
2778
|
-
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
|
|
2779
|
-
print 'Before Queue Hook - run before test suite'
|
|
2780
|
-
end
|
|
2781
|
-
|
|
2782
|
-
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
|
|
2783
|
-
print 'After Subset Queue Hook - run after subset of test suite'
|
|
2784
|
-
end
|
|
2785
|
-
|
|
2786
|
-
# this hook is not supported and won't run
|
|
2787
|
-
KnapsackPro::Hooks::Queue.after_queue do |queue_id|
|
|
2788
|
-
print 'After Queue Hook - run after test suite'
|
|
2789
|
-
end
|
|
2790
|
-
```
|
|
2791
|
-
|
|
2792
|
-
#### How to run knapsack_pro with parallel_tests gem?
|
|
2793
|
-
|
|
2794
|
-
##### Should I use parallel_tests gem (what are pitfalls)?
|
|
2795
|
-
|
|
2796
|
-
If you plan to use parallel_tests please be careful how many parallel processes with running tests you will start on a single CI node.
|
|
2797
|
-
Often it happens that running 2 or more processes with tests using parallel_tests gem on the CI node that has low performance leads to slower execution of test suite. You can accidentally make your whole test suite running slower by using parallel_tests if you have not enough powerful CI server (slow CPU, not enough RAM, slow disk).
|
|
2798
|
-
|
|
2799
|
-
If you use parallel_tests and knapsack_pro you can see recorded tests timing in Knapsack Pro [user dashboard](https://knapsackpro.com/dashboard). See the "Build metrics" link next to your API token and check the last recorded CI build time. You will be able to see there how long each test files took to execute. If you notice that after adding parallel_tests gem your test files started to take more time than before it means you overloaded your CI server.
|
|
2800
|
-
|
|
2801
|
-
You should:
|
|
2802
|
-
|
|
2803
|
-
* reduce the number of parallel processes in parallel_tests gem
|
|
2804
|
-
* or buy a more powerful CI node to allow running more parallel processes (vertical scaling)
|
|
2805
|
-
* or don't use parallel_tests gem at all (recommended)
|
|
2806
|
-
|
|
2807
|
-
In case of tests execution time increase (slower tests) I recommend using more parallel nodes offered by your CI provider to scale your tests horizontally. Basically, adding parallel CI nodes instead of vertically adding more CPU/RAM to CI node is a better option. Parallel_tests gem has mixed output results from the parallel processes so it's easier to just browse tests output from parallel CI nodes when you scale horizontally by using knapsack_pro without parallel_tests.
|
|
2808
|
-
|
|
2809
|
-
If you want to use parallel_tests you can use it with Knapsack Pro Queue Mode to auto-balance tests split across parallel processes started by parallel_tests gem. See below tips on how to do it on [many parallel CI nodes where each node starts many parallel_tests processes](#parallel_tests-with-knapsack_pro-on-parallel-ci-nodes) or [on a single powerful CI server](#parallel_tests-with-knapsack_pro-on-single-ci-machine).
|
|
2810
|
-
|
|
2811
|
-
##### parallel_tests with knapsack_pro on parallel CI nodes
|
|
2812
|
-
|
|
2813
|
-
You can run knapsack_pro with [parallel_tests](https://github.com/grosser/parallel_tests) gem to run multiple concurrent knapsack_pro commands per CI node.
|
|
2814
|
-
|
|
2815
|
-
Let's consider this example. We have 2 CI node. On each CI node we want to run 2 concurrent knapsack_pro commands by parallel_tests gem (`PARALLEL_TESTS_CONCURRENCY=2`).
|
|
2816
|
-
This means we would have 4 parallel knapsack_pro commands in total across all CI nodes. So from knapsack_pro perspective you will have 4 nodes in total.
|
|
2817
|
-
|
|
2818
|
-
Create in your project directory an executable file `bin/parallel_tests`:
|
|
2819
|
-
|
|
2820
|
-
```bash
|
|
2821
|
-
#!/bin/bash
|
|
2822
|
-
# This file should be in bin/parallel_tests
|
|
2823
|
-
|
|
2824
|
-
# updates CI node total based on parallel_tests concurrency
|
|
2825
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=$(( $PARALLEL_TESTS_CONCURRENCY * $KNAPSACK_PRO_CI_NODE_TOTAL ))
|
|
2826
|
-
|
|
2827
|
-
if [ "$TEST_ENV_NUMBER" == "" ]; then
|
|
2828
|
-
PARALLEL_TESTS_CONCURRENCY_INDEX=0
|
|
2829
|
-
else
|
|
2830
|
-
PARALLEL_TESTS_CONCURRENCY_INDEX=$(( $TEST_ENV_NUMBER - 1 ))
|
|
2831
|
-
fi
|
|
2832
|
-
|
|
2833
|
-
KNAPSACK_PRO_CI_NODE_INDEX=$(( $PARALLEL_TESTS_CONCURRENCY_INDEX + ($PARALLEL_TESTS_CONCURRENCY * $KNAPSACK_PRO_CI_NODE_INDEX) ))
|
|
2834
|
-
|
|
2835
|
-
# logs info about ENVs to ensure everything works
|
|
2836
|
-
echo KNAPSACK_PRO_CI_NODE_TOTAL=$KNAPSACK_PRO_CI_NODE_TOTAL KNAPSACK_PRO_CI_NODE_INDEX=$KNAPSACK_PRO_CI_NODE_INDEX PARALLEL_TESTS_CONCURRENCY=$PARALLEL_TESTS_CONCURRENCY
|
|
2837
|
-
|
|
2838
|
-
# you can customize your knapsack_pro command here to use regular or queue mode
|
|
2839
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
2840
|
-
```
|
|
2841
|
-
|
|
2842
|
-
Now you need to set parallel_tests command per CI node:
|
|
2843
|
-
|
|
2844
|
-
* CI node 0 (first CI node):
|
|
2845
|
-
|
|
2846
|
-
```bash
|
|
2847
|
-
export PARALLEL_TESTS_CONCURRENCY=2; # this must be export
|
|
2848
|
-
RAILS_ENV=test \
|
|
2849
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \
|
|
2850
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=$YOUR_CI_NODE_TOTAL \
|
|
2851
|
-
KNAPSACK_PRO_CI_NODE_INDEX=$YOUR_CI_NODE_INDEX \
|
|
2852
|
-
bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests'
|
|
2853
|
-
```
|
|
2854
|
-
|
|
2855
|
-
* CI node 1 (second CI node):
|
|
2856
|
-
|
|
2857
|
-
```bash
|
|
2858
|
-
export PARALLEL_TESTS_CONCURRENCY=2; # this must be export
|
|
2859
|
-
RAILS_ENV=test \
|
|
2860
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \
|
|
2861
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=$YOUR_CI_NODE_TOTAL \
|
|
2862
|
-
KNAPSACK_PRO_CI_NODE_INDEX=$YOUR_CI_NODE_INDEX \
|
|
2863
|
-
bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests'
|
|
2864
|
-
```
|
|
2865
|
-
|
|
2866
|
-
Please note you need to update `$YOUR_CI_NODE_TOTAL` and `$YOUR_CI_NODE_INDEX` to the ENVs provided by your CI provider. For instance in case of CircleCI it would be `$CIRCLE_NODE_TOTAL` and `$CIRCLE_NODE_INDEX`. Below is an example for CircleCI configuration:
|
|
2867
|
-
|
|
2868
|
-
```yaml
|
|
2869
|
-
# circle.yml for CircleCI 1.0
|
|
2870
|
-
# KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx can be set in CircleCI ENV settings
|
|
2871
|
-
test:
|
|
2872
|
-
override:
|
|
2873
|
-
- export PARALLEL_TESTS_CONCURRENCY=2; RAILS_ENV=test KNAPSACK_PRO_CI_NODE_TOTAL=$CIRCLE_NODE_TOTAL KNAPSACK_PRO_CI_NODE_INDEX=$CIRCLE_NODE_INDEX bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests':
|
|
2874
|
-
parallel: true # Caution: there are 8 spaces indentation!
|
|
2875
|
-
```
|
|
2876
|
-
|
|
2877
|
-
In summary, the `bin/parallel_tests` script will calculate a new values for `KNAPSAKC_PRO_*` environment variables and then run knapsack_pro command with them.
|
|
2878
|
-
To ensure everything works you can check output for each CI node.
|
|
2879
|
-
|
|
2880
|
-
* Expected output for CI node 0 (first CI node):
|
|
2881
|
-
|
|
2882
|
-
```
|
|
2883
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=1 PARALLEL_TESTS_CONCURRENCY=2
|
|
2884
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=0 PARALLEL_TESTS_CONCURRENCY=2
|
|
2885
|
-
(tests output here)
|
|
2886
|
-
```
|
|
2887
|
-
|
|
2888
|
-
* Expected output for CI node 1 (second CI node):
|
|
2889
|
-
|
|
2890
|
-
```
|
|
2891
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=2 PARALLEL_TESTS_CONCURRENCY=2
|
|
2892
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=3 PARALLEL_TESTS_CONCURRENCY=2
|
|
2893
|
-
(tests output here)
|
|
2894
|
-
```
|
|
2895
|
-
|
|
2896
|
-
##### parallel_tests with knapsack_pro on single CI machine
|
|
2897
|
-
|
|
2898
|
-
This tip is only relevant to you if you cannot use multiple parallel CI nodes on your CI provider. In such case, you can run your tests on a single CI machine with knapsack_pro Queue Mode in order to auto balance execution of tests and thanks to this better utilize CI machine resources.
|
|
2899
|
-
|
|
2900
|
-
You can run knapsack_pro with [parallel_tests](https://github.com/grosser/parallel_tests) gem to run multiple concurrent knapsack_pro commands on single CI node.
|
|
2901
|
-
|
|
2902
|
-
Create in your project directory an executable file `bin/parallel_tests_knapsack_pro_single_machine`:
|
|
2903
|
-
|
|
2904
|
-
```bash
|
|
2905
|
-
#!/bin/bash
|
|
2906
|
-
# bin/parallel_tests_knapsack_pro_single_machine
|
|
2907
|
-
|
|
2908
|
-
export KNAPSACK_PRO_CI_NODE_TOTAL=$PARALLEL_TESTS_CONCURRENCY
|
|
2909
|
-
|
|
2910
|
-
if [ "$TEST_ENV_NUMBER" == "" ]; then
|
|
2911
|
-
export KNAPSACK_PRO_CI_NODE_INDEX=0
|
|
2912
|
-
else
|
|
2913
|
-
export KNAPSACK_PRO_CI_NODE_INDEX=$(( $TEST_ENV_NUMBER - 1 ))
|
|
2914
|
-
fi
|
|
2915
|
-
|
|
2916
|
-
echo KNAPSACK_PRO_CI_NODE_TOTAL=$KNAPSACK_PRO_CI_NODE_TOTAL KNAPSACK_PRO_CI_NODE_INDEX=$KNAPSACK_PRO_CI_NODE_INDEX PARALLEL_TESTS_CONCURRENCY=$PARALLEL_TESTS_CONCURRENCY
|
|
2917
|
-
|
|
2918
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
2919
|
-
```
|
|
2920
|
-
|
|
2921
|
-
Then you need another script `bin/parallel_tests_knapsack_pro_single_machine_run` to run above script with `parallel_tests`:
|
|
2922
|
-
|
|
2923
|
-
```bash
|
|
2924
|
-
#!/bin/bash
|
|
2925
|
-
# bin/parallel_tests_knapsack_pro_single_machine_run
|
|
2926
|
-
|
|
2927
|
-
export PARALLEL_TESTS_CONCURRENCY=2;
|
|
2928
|
-
|
|
2929
|
-
RAILS_ENV=test \
|
|
2930
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \
|
|
2931
|
-
KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
|
|
2932
|
-
KNAPSACK_PRO_PROJECT_DIR=/home/user/rails-app-repo \
|
|
2933
|
-
PARALLEL_TESTS_CONCURRENCY=$PARALLEL_TESTS_CONCURRENCY \
|
|
2934
|
-
bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests_knapsack_pro_single_machine'
|
|
2935
|
-
```
|
|
2936
|
-
|
|
2937
|
-
Now you can run `bin/parallel_tests_knapsack_pro_single_machine_run` and it will execute 2 parallel processes with `parallel_tests`. Each process will run knapsack_pro Queue Mode to autobalance test files distribution across the processes.
|
|
2938
|
-
|
|
2939
|
-
#### How to retry failed tests (flaky tests)?
|
|
2940
|
-
|
|
2941
|
-
Flaky (nondeterministic) tests, are tests that exhibit both a passing and a failing result with the same code.
|
|
2942
|
-
|
|
2943
|
-
I recommend to use [rspec-retry](https://github.com/NoRedInk/rspec-retry) gem that can retry failing test. It can be useful for randomly failing features specs. For instance you can configure it to retry 3 times features test before marking it as failing.
|
|
2944
|
-
|
|
2945
|
-
Alternative way is to use built into [RSpec only failures option](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rerun failed tests.
|
|
2946
|
-
|
|
2947
|
-
Please add to your RSpec configuration:
|
|
2948
|
-
|
|
2949
|
-
```ruby
|
|
2950
|
-
RSpec.configure do |c|
|
|
2951
|
-
c.example_status_persistence_file_path = "tmp/rspec_examples.txt"
|
|
2952
|
-
end
|
|
2953
|
-
```
|
|
2954
|
-
|
|
2955
|
-
Then you can execute rspec with only failed tests after main knapsack_pro command finish.
|
|
2956
|
-
|
|
2957
|
-
```bash
|
|
2958
|
-
# Run knapsack_pro in Queue Mode and it will save failed tests in tmp/rspec_examples.txt
|
|
2959
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
2960
|
-
|
|
2961
|
-
# run only failed tests from tmp/rspec_examples.txt
|
|
2962
|
-
bundle exec rspec --only-failures
|
|
2963
|
-
```
|
|
2964
|
-
|
|
2965
|
-
#### How can I run tests from multiple directories?
|
|
2966
|
-
|
|
2967
|
-
The test file pattern config option supports any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob) and can be configured to pull test files from multiple directories. An example of this when using RSpec would be `"{spec,engines/*/spec}/**/*_spec.rb"`. For complex cases like this, the test directory can't be extracted and must be specified manually using the `KNAPSACK_PRO_TEST_DIR` environment variable:
|
|
2968
|
-
|
|
2969
|
-
```
|
|
2970
|
-
# This is example where you have in engines directory multiple projects
|
|
2971
|
-
# and each project directory has a spec folder and you would like to run tests for it.
|
|
2972
|
-
# You want to use the spec_helper from the main spec directory.
|
|
2973
|
-
#
|
|
2974
|
-
# Tree:
|
|
2975
|
-
# * spec
|
|
2976
|
-
# * engines
|
|
2977
|
-
# * project_a
|
|
2978
|
-
# * spec
|
|
2979
|
-
# * project_b
|
|
2980
|
-
# * spec
|
|
2981
|
-
$ KNAPSACK_PRO_TEST_DIR=spec KNAPSACK_PRO_TEST_FILE_PATTERN="{spec,engines/*/spec}/**/*_spec.rb" bundle exec rake knapsack_pro:queue:rspec
|
|
2982
|
-
```
|
|
2983
|
-
|
|
2984
|
-
`KNAPSACK_PRO_TEST_DIR` will be your default path for rspec so you should put there your `spec_helper.rb`. Please ensure you will require it in your test files this way if something doesn't work:
|
|
2985
|
-
|
|
2986
|
-
```ruby
|
|
2987
|
-
# good
|
|
2988
|
-
require_relative 'spec_helper'
|
|
2989
|
-
|
|
2990
|
-
# bad - won't work
|
|
2991
|
-
require 'spec_helper'
|
|
2992
|
-
```
|
|
2993
|
-
|
|
2994
|
-
#### Why I don't see all test files being recorded in user dashboard
|
|
2995
|
-
|
|
2996
|
-
If you open `Build metrics` for particular API token at [user dashboard](https://knapsackpro.com/dashboard) and you don't see all time execution data recorded for all test files then you should know that knapsack_pro version older than [`1.0.2`](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md#102) does not track test files with empty content or when the test file contains only pending tests.
|
|
2997
|
-
|
|
2998
|
-
The test files with pending tests are executed so you will see it in RSpec output but just not recorded in Knapsack Pro API because there is nothing to record time for.
|
|
2999
|
-
|
|
3000
|
-
We recommend to update to the latest version of knapsack_pro.
|
|
3001
|
-
|
|
3002
|
-
Please check also this question [why you may don't see time execution data](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) in your dashboard.
|
|
3003
|
-
|
|
3004
|
-
#### Why when I use 2 different CI providers then not all test files are executed?
|
|
3005
|
-
|
|
3006
|
-
Please ensure you use 2 different API token per test suite. If you use 2 CI providers for instance CircleCI and TravisCI at the same time and you run the RSpec test suite then you need to have separate API token for RSpec executed on CircleCI and a separate API token for RSpec test suite executed on the TravisCI.
|
|
3007
|
-
|
|
3008
|
-
#### How to run only RSpec feature tests or non feature tests?
|
|
3009
|
-
|
|
3010
|
-
**Option 1: RSpec tags**
|
|
3011
|
-
|
|
3012
|
-
You can run just feature tests this way. You need to generate a separate API token for it.
|
|
3013
|
-
|
|
3014
|
-
```
|
|
3015
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_FEATURE_TESTS bundle exec rake "knapsack_pro:queue:rspec[--tag type:feature]"
|
|
3016
|
-
```
|
|
3017
|
-
|
|
3018
|
-
If you would like to run only non feature tests then use negation `~type:feature`:
|
|
3019
|
-
|
|
3020
|
-
```
|
|
3021
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_NON_FEATURE_TESTS bundle exec rake "knapsack_pro:queue:rspec[--tag ~type:feature]"
|
|
3022
|
-
```
|
|
3023
|
-
|
|
3024
|
-
Note above examples are for knapsack_pro Queue Mode and when you will run tests you may notice that all test files are run by RSpec but only tests specified by tag like `tag type:feature` will be executed. Basically RSpec will just load all files but run just specified tags.
|
|
3025
|
-
|
|
3026
|
-
**Option 2: specify directory pattern**
|
|
3027
|
-
|
|
3028
|
-
Another approach is to explicitly specify which files should be executed.
|
|
3029
|
-
|
|
3030
|
-
Run all specs from multiple directories except `spec/features` directory which is not listed below.
|
|
3031
|
-
If you would like to run additional directory please add it after comma in `KNAPSACK_PRO_TEST_FILE_PATTERN`.
|
|
3032
|
-
Ensure the list of directories match your spec directory structure.
|
|
3033
|
-
|
|
3034
|
-
```bash
|
|
3035
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_NON_FEATURE_TESTS \
|
|
3036
|
-
KNAPSACK_PRO_TEST_DIR=spec \
|
|
3037
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="{spec/*_spec.rb,spec/controllers/**/*_spec.rb,spec/mailers/**/*_spec.rb,spec/models/**/*_spec.rb,spec/presenters/**/*_spec.rb,spec/requests/**/*_spec.rb,spec/routing/**/*_spec.rb,spec/services/**/*_spec.rb,spec/workers/**/*_spec.rb,spec/jobs/**/*_spec.rb}" \
|
|
3038
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
3039
|
-
```
|
|
3040
|
-
|
|
3041
|
-
When you would like to run tests only from `spec/features` directory then run:
|
|
3042
|
-
|
|
3043
|
-
```bash
|
|
3044
|
-
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_FEATURE_TESTS \
|
|
3045
|
-
KNAPSACK_PRO_TEST_DIR=spec \
|
|
3046
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/**{,/*/**}/*_spec.rb" \
|
|
3047
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
3048
|
-
```
|
|
3049
|
-
|
|
3050
|
-
You can also learn [how to use exclude pattern](#how-to-exclude-tests-from-running-them).
|
|
3051
|
-
|
|
3052
|
-
#### How to exclude tests from running them?
|
|
3053
|
-
|
|
3054
|
-
For instance you would like to run all tests except tests in `features` directory then you could do:
|
|
3055
|
-
|
|
3056
|
-
```bash
|
|
3057
|
-
KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/features/**{,/*/**}/*_spec.rb" \
|
|
3058
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
3059
|
-
```
|
|
3060
|
-
|
|
3061
|
-
You can define at the same time the pattern for tests you would like to run and the exclude pattern. For instance run all controller tests except admin controller tests.
|
|
3062
|
-
|
|
3063
|
-
```bash
|
|
3064
|
-
KNAPSACK_PRO_TEST_FILE_PATTERN="spec/controllers/**{,/*/**}/*_spec.rb" \
|
|
3065
|
-
KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/controllers/admin/**{,/*/**}/*_spec.rb" \
|
|
3066
|
-
bundle exec rake knapsack_pro:queue:rspec
|
|
3067
|
-
```
|
|
3068
|
-
|
|
3069
|
-
The test file pattern and exclude pattern support any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob).
|
|
3070
|
-
|
|
3071
|
-
#### How to run a specific list of test files or only some tests from test file?
|
|
3072
|
-
|
|
3073
|
-
:information_source: If you don't want to use the pattern [`KNAPSACK_PRO_TEST_FILE_PATTERN`](#how-can-i-run-tests-from-multiple-directories) to define a list of tests to run then read below two options.
|
|
3074
|
-
|
|
3075
|
-
**Option 1:**
|
|
3076
|
-
|
|
3077
|
-
If you want to run a specific list of test files that are explicitly defined by you or auto-generated by some kind of script you created then please use:
|
|
3078
|
-
|
|
3079
|
-
`KNAPSACK_PRO_TEST_FILE_LIST=spec/features/dashboard_spec.rb,spec/models/user.rb:10,spec/models/user.rb:29`
|
|
3080
|
-
|
|
3081
|
-
Note `KNAPSACK_PRO_TEST_FILE_LIST` must be a list of test files comma separated. You can provide line number for tests inside of spec file in case of RSpec (this way you can run only one test or a group of tests from RSpec spec file). You can provide the same file a few times with different test line number.
|
|
3082
|
-
|
|
3083
|
-
**Option 2:**
|
|
3084
|
-
|
|
3085
|
-
Similarly, you can also provide a source file containing the test files that you would like to run. For example:
|
|
3086
|
-
`KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE=spec/fixtures/test_file_list_source_file.txt`
|
|
3087
|
-
And the content of the source file can be any of the format below:
|
|
3088
|
-
|
|
3089
|
-
```
|
|
3090
|
-
./spec/test1_spec.rb
|
|
3091
|
-
spec/test2_spec.rb[1]
|
|
3092
|
-
./spec/test3_spec.rb[1:2:3:4]
|
|
3093
|
-
./spec/test4_spec.rb:4
|
|
3094
|
-
./spec/test4_spec.rb:5
|
|
3095
|
-
```
|
|
3096
|
-
|
|
3097
|
-
> Note that each of the line must be ending with `\n` the new line.
|
|
3098
|
-
|
|
3099
|
-
Note when you set `KNAPSACK_PRO_TEST_FILE_LIST` or `KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE` then below environment variables are ignored:
|
|
3100
|
-
|
|
3101
|
-
* `KNAPSACK_PRO_TEST_FILE_PATTERN`
|
|
3102
|
-
* `KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN`
|
|
3103
|
-
|
|
3104
|
-
#### How to run knapsack_pro only on a few parallel CI nodes instead of all?
|
|
3105
|
-
|
|
3106
|
-
You may want to run knapsack_pro only on a few CI nodes when you would like to run a different job on other CI nodes.
|
|
3107
|
-
|
|
3108
|
-
For instance, you have 3 parallel CI nodes. You would like to run knapsack_pro only on two CI nodes. The last CI node you want to use for the different job like running linters etc.
|
|
3109
|
-
|
|
3110
|
-
In such case, you can override the number of total CI nodes available by your CI provider. For instance, Heroku CI provider exposes in ENV variables `CI_NODE_TOTAL=3`.
|
|
3111
|
-
|
|
3112
|
-
You can then run knapsack_pro command this way on the first and the second CI node:
|
|
3113
|
-
|
|
3114
|
-
```
|
|
3115
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=$((CI_NODE_TOTAL-1)) bundle exec rake knapsack_pro:rspec
|
|
3116
|
-
```
|
|
3117
|
-
|
|
3118
|
-
We decrease the number of CI node total by 1 that knapsack_pro can see. This way you can run tests with knapsack_pro only on two CI nodes.
|
|
3119
|
-
On the 3rd CI node, you can run other things like linters etc.
|
|
3120
|
-
|
|
3121
|
-
If you would like to check what is the CI node total ENV variable name exposed by your CI provider you can check that in your CI provider environment variables docs
|
|
3122
|
-
or preview the [ENV variables that knapsack_pro can read](https://github.com/KnapsackPro/knapsack_pro-ruby/tree/master/lib/knapsack_pro/config/ci) for supported CI providers.
|
|
3123
|
-
|
|
3124
|
-
If you use for instance Heroku CI that allows you to provide only one test command you can make a bash script to control what's executed on particular CI node:
|
|
3125
|
-
|
|
3126
|
-
```bash
|
|
3127
|
-
#!/bin/bash
|
|
3128
|
-
# add this file in bin/knapsack_pro_rspec_and_npm_test and change chmod
|
|
3129
|
-
# $ chmod a+x bin/knapsack_pro_rspec_and_npm_test
|
|
3130
|
-
|
|
3131
|
-
# 15 is last CI node (index starts from 0, so in total we have 16 parallel Heroku dynos)
|
|
3132
|
-
if [ "$CI_NODE_INDEX" == "15" ]; then
|
|
3133
|
-
# run npm tests on the last CI node
|
|
3134
|
-
npm test
|
|
3135
|
-
else
|
|
3136
|
-
KNAPSACK_PRO_CI_NODE_TOTAL=$((CI_NODE_TOTAL-1)) bundle exec rake knapsack_pro:queue:rspec
|
|
3137
|
-
fi
|
|
3138
|
-
```
|
|
3139
|
-
|
|
3140
|
-
then in your Heroku CI config `app.json` set:
|
|
3141
|
-
|
|
3142
|
-
```
|
|
3143
|
-
"scripts": {
|
|
3144
|
-
"test": "bin/knapsack_pro_rspec_and_npm_test"
|
|
3145
|
-
}
|
|
3146
|
-
```
|
|
3147
|
-
|
|
3148
|
-
#### How to use CodeClimate with knapsack_pro?
|
|
3149
|
-
|
|
3150
|
-
You can check articles about CodeClimate configuration with knapsack_pro gem:
|
|
3151
|
-
* [CodeClimate and CircleCI 2.0 parallel builds config for RSpec with SimpleCov and JUnit formatter](https://docs.knapsackpro.com/2019/codeclimate-and-circleci-2-0-parallel-builds-config-for-rspec-with-simplecov-and-junit-formatter)
|
|
3152
|
-
* [How to merge CodeClimate reports for parallel jobs (CI nodes) on Semaphore CI 2.0](https://docs.knapsackpro.com/2019/how-to-merge-codeclimate-reports-for-parallel-jobs-ci-nodes).
|
|
3153
|
-
|
|
3154
|
-
#### How to use simplecov in Queue Mode?
|
|
3155
|
-
|
|
3156
|
-
If you would like to make [simplecov](https://github.com/colszowka/simplecov) gem work with knapsack_pro Queue Mode to correctly track code coverage for parallel CI nodes please do:
|
|
3157
|
-
|
|
3158
|
-
```ruby
|
|
3159
|
-
# spec_helper.rb or rails_helper.rb
|
|
3160
|
-
require 'knapsack_pro'
|
|
3161
|
-
|
|
3162
|
-
require 'simplecov'
|
|
3163
|
-
SimpleCov.start
|
|
3164
|
-
|
|
3165
|
-
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
|
|
3166
|
-
SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
|
|
3167
|
-
end
|
|
3168
|
-
```
|
|
3169
|
-
|
|
3170
|
-
This way there should be no conflict between code coverage reports generated per CI node index even when you use the same local drive (for instance you use Jenkins as your CI provider). The simplecov will generate single report at `coverage/index.html` with merged data from parallel CI nodes.
|
|
3171
|
-
|
|
3172
|
-
#### Do I need to use separate API token for Queue Mode and Regular Mode?
|
|
3173
|
-
|
|
3174
|
-
I recommend to record timing of a new test suite with `API token A` and knapsack_pro Regular Mode. After you recorded test suite timing then you should use the `API token A` to run your tests in knapsack_pro Queue Mode. This way Queue Mode will leverage test suite timing recorded in a fast way with Regular Mode so the first run in Queue Mode won't be slow due to recording test files timing for the first time.
|
|
3175
|
-
|
|
3176
|
-
When you want to go back from Queue Mode to Regular Mode then the fact of using the same API token could cause edge cases that some builds might not be well balanced in Regular Mode. That is why I recommend using separate API token for Regular Mode and Queue Mode. If you plan to use only Queue Mode then no worry.
|
|
3177
|
-
|
|
3178
|
-
#### How to stop running tests on the first failed test (fail fast tests in RSpec)?
|
|
3179
|
-
|
|
3180
|
-
If you want to stop running tests as soon as one of it fails then you can pass [--fail-fast](https://relishapp.com/rspec/rspec-core/docs/command-line/fail-fast-option) RSpec option to knapsack_pro:
|
|
3181
|
-
|
|
3182
|
-
```
|
|
3183
|
-
# Regular Mode
|
|
3184
|
-
bundle exec rake "knapsack_pro:rspec[--fail-fast]"
|
|
3185
|
-
|
|
3186
|
-
# Queue Mode
|
|
3187
|
-
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]"
|
|
3188
|
-
```
|
|
3189
|
-
|
|
3190
|
-
You may add a parameter to tell RSpec to stop running the test suite after N failed tests, for example: `--fail-fast=3`.
|
|
3191
|
-
|
|
3192
|
-
```
|
|
3193
|
-
Note there is no = sign on purpose here:
|
|
3194
|
-
|
|
3195
|
-
# Regular Mode
|
|
3196
|
-
bundle exec rake "knapsack_pro:rspec[--fail-fast 3]"
|
|
3197
|
-
|
|
3198
|
-
# Queue Mode
|
|
3199
|
-
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]"
|
|
3200
|
-
```
|
|
3201
|
-
|
|
3202
|
-
There is a downside to it. If you stop running tests then tests that were never run will have no recorded timing of execution and because of that, the future CI build might have tests split across CI nodes in no optimal way.
|
|
3203
|
-
|
|
3204
|
-
### Questions around data usage and security
|
|
3205
|
-
|
|
3206
|
-
#### What data is sent to your servers?
|
|
3207
|
-
|
|
3208
|
-
The knapsack_pro gem sends branch name, commit hash, CI total node number, CI index node number, the test file paths like `spec/models/user_spec.rb` and the time execution of each test file path as a float.
|
|
3209
|
-
|
|
3210
|
-
Here is the [full specification of the API](http://docs.knapsackpro.com/api/v1/) used by knapsack_pro gem.
|
|
3211
|
-
|
|
3212
|
-
#### How is that data secured?
|
|
3213
|
-
|
|
3214
|
-
The test file paths and/or branch names can be [encrypted](#test-file-names-encryption) on your CI node with a salt and later send to knapsackpro.com API.
|
|
3215
|
-
You generate the salt locally and __only you__ can decrypt the test file paths or branch names with the salt. Here you can [see how the data are encrypted](lib/knapsack_pro/crypto/digestor.rb).
|
|
3216
|
-
|
|
3217
|
-
Connection with knapsackpro.com server is via https.
|
|
3218
|
-
|
|
3219
|
-
Regarding payments we use the BraintreePayments.com and they store credit cards and your private information.
|
|
3220
|
-
|
|
3221
|
-
#### Who has access to the data?
|
|
3222
|
-
|
|
3223
|
-
I’m the only admin so I can preview data in case you need help with debugging some problem etc. I’m not able to decrypt them without knowing the salt.
|
|
3224
|
-
|
|
3225
|
-
When you sign in to your user dashboard then you can preview data for recent CI builds. If the test file paths are encrypted then you only see hashes for test file paths.
|
|
3226
|
-
You need to [decrypt](#how-to-debug-test-file-names) them locally on your machine to find out what each test file hash is.
|
|
3227
|
-
|
|
3228
1675
|
## Gem tests
|
|
3229
1676
|
|
|
3230
1677
|
### Spec
|