knapsack_pro 1.10.1 → 1.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fb705b44a624bdd6caa055a2caee16cb3ebcf4f9a15a1cc718b5f6a2f03f89ce
-  data.tar.gz: 6155ff3b7f0bf3d96f12dbad0009499eb02953d0415e48c1bcaf2c3ffbf94027
+  metadata.gz: 640a408f0a698ec1f064a7857699aeaf5ec27c72a3b30e96a0dfc80655473549
+  data.tar.gz: b8a305a9e170cf0f015be62a85c0415bfe3ba76c1255122fdc0c1b0c46a8d934
 SHA512:
-  metadata.gz: 0e02c5b22c2a3a4261dd5cf75dce2a8bdeb086254df40166f38758766fcbfcb4c9c68aa3fb76ef748eab4db668a782ea7af2d4313bbf2e71491ada6af87cf320
-  data.tar.gz: 1d57f7e37be0f0d43c4b13e3cdcbcda612dce17ed9cb4854dca5e1d75d0e310ebc49cb2939476cca1eca275e565db597c3e82f7b3e1c8705c0f41afce887b570
+  metadata.gz: 7e6ecaf953291bf583006846a3511162f7896f5e0ad374dcf861c8aa99dc017f2d102bf0a85b96133b24afc4a3ce99632aaded1384e853a6cd6c2a7d48c77c0d
+  data.tar.gz: 0350dafb658f2d965bd46728f0f18ae1d3dc0e44639eb1d54969c6f203bdc82f2d7653da63ec71fe7de9fe0013b8faf1e5142ef26dac7b702342f012449b7f6c

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Change Log
 
+### 1.11.0
+
+* Add support for `KNAPSACK_PRO_TEST_FILE_LIST` environment variable to run explicitly listed tests
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/86
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.10.1...v1.11.0
+
 ### 1.10.1
 
 * Fix log info when measured time of tests was lost

data/README.md

@@ -120,6 +120,7 @@ You can see list of questions for common problems and tips in below [Table of Co
     - [Why I see HEAD as branch name in user dashboard for Build metrics for my API token?](#why-i-see-head-as-branch-name-in-user-dashboard-for-build-metrics-for-my-api-token)
     - [Why Capybara feature tests randomly fail when using CI parallelisation?](#why-capybara-feature-tests-randomly-fail-when-using-ci-parallelisation)
     - [Why knapsack_pro freezes / hangs my CI (for instance Travis)?](#why-knapsack_pro-freezes--hangs-my-ci-for-instance-travis)
+    - [Why tests hitting external API fail?](#why-tests-hitting-external-api-fail)
     - [Queue Mode problems](#queue-mode-problems)
       - [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail)
       - [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail)
@@ -173,10 +174,12 @@ You can see list of questions for common problems and tips in below [Table of Co
     - [Why when I use 2 different CI providers then not all test files are executed?](#why-when-i-use-2-different-ci-providers-then-not-all-test-files-are-executed)
     - [How to run only RSpec feature tests or non feature tests?](#how-to-run-only-rspec-feature-tests-or-non-feature-tests)
     - [How to exclude tests from running them?](#how-to-exclude-tests-from-running-them)
+    - [How to run a specific list of test files or only some tests from test file?](#how-to-run-a-specific-list-of-test-files-or-only-some-tests-from-test-file)
     - [How to use CodeClimate with knapsack_pro?](#how-to-use-codeclimate-with-knapsack_pro)
     - [How to run knapsack_pro only on a few parallel CI nodes instead of all?](#how-to-run-knapsack_pro-only-on-a-few-parallel-ci-nodes-instead-of-all)
     - [How to use simplecov in Queue Mode?](#how-to-use-simplecov-in-queue-mode)
     - [Do I need to use separate API token for Queue Mode and Regular Mode?](#do-i-need-to-use-separate-api-token-for-queue-mode-and-regular-mode)
+    - [How to stop running tests on the first failed test (fail fast tests in RSpec)?](#how-to-stop-running-tests-on-the-first-failed-test-fail-fast-tests-in-rspec)
   - [Questions around data usage and security](#questions-around-data-usage-and-security)
     - [What data is sent to your servers?](#what-data-is-sent-to-your-servers)
     - [How is that data secured?](#how-is-that-data-secured)
@@ -1409,6 +1412,26 @@ RSpec.configure do |c|
 end
 ```
 
+#### Why tests hitting external API fail?
+
+If you use knapsack_pro and you have tests that do real HTTP requests to external API you need to ensure your tests can be run across parallel CI nodes.
+
+Let's say you have tests that do requests to Stripe API or any other API. Before running each test you want to make sure Stripe Sandbox is clean up so you have removed all fake subscriptions and customers from Stripe Sandbox.
+
+```ruby
+# RSpec hook
+before(:each) do
+  Stripe::Subscription.all.each { |sub| sub.delete }
+  Stripe::Customer.all.each { |customer| customer.delete }
+end
+```
+
+But this will cause a problem when 2 different test files will run on 2 different CI nodes at the same time and this hook will be called. You will remove subscriptions and customers while another parallel test was running. Simply speaking you have tests that are written in a way that you can't run them in parallel.
+
+To fix that you can think of:
+* using [VCR](https://github.com/vcr/vcr) gem to record HTTP requests and then instead of doing real HTTP requests just reply recorded requests.
+* maybe you could write your tests in a way when you generate some fake customers or subscriptions with fake id and each test has different customer id so there will be no conflict when 2 tests are run at the same time.
+
 #### Queue Mode problems
 
 ##### Why when I use Queue Mode for RSpec then my tests fail?
@@ -1421,6 +1444,18 @@ In that case you need to resolve failed tests in a way that allows RSpec to run
 
 You can learn more about [recent RSpec team changes](https://github.com/KnapsackPro/knapsack_pro-ruby/pull/42) that was backported into knapsack_pro.
 
+To solve failing tests in Queue Mode you can check:
+
+* you use full namespacing. If you see error like `NameError: uninitialized constant MyModule::ModelName` then in some cases a top-level constant would be matched if the code hadn't been loaded for the scoped constant. Try to use full namespacing `::SomeModule::MyModule::ModelName` etc.
+* you can try to use binary version of knapsack_pro instead of running it via rake task. This helps if your rake tasks mess up with tests and make knapsack_pro Queue Mode fail. [See example](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail):
+
+    ```bash
+    # Knapsack Pro Queue Mode run via binary
+    bundle exec knapsack_pro queue:rspec "--profile 10 --format progress"
+    ```
+
+* You can check below questions for common reasons of failing tests in Queue Mode
+
 ##### Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?
 
 You can use [knapsack_pro binary](#knapsack-pro-binary) instead of rake task version to solve problem:
@@ -2312,10 +2347,14 @@ require 'spec_helper'
 
 #### Why I don't see all test files being recorded in user dashboard
 
-If you open `Build metrics` for particular API token at [user dashboard](https://knapsackpro.com/dashboard) and you don't see all time execution data recorded for all test files then you should know that knapsack_pro does not track test files with empty content or when the test file contains only pending tests.
+If you open `Build metrics` for particular API token at [user dashboard](https://knapsackpro.com/dashboard) and you don't see all time execution data recorded for all test files then you should know that knapsack_pro version older than [`1.0.2`](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md#102) does not track test files with empty content or when the test file contains only pending tests.
 
 The test files with pending tests are executed so you will see it in RSpec output but just not recorded in Knapsack Pro API because there is nothing to record time for.
 
+We recommend to update to the latest version of knapsack_pro.
+
+Please check also this question [why you may don't see time execution data](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) in your dashboard.
+
 #### Why when I use 2 different CI providers then not all test files are executed?
 
 Please ensure you use 2 different API token per test suite. If you use 2 CI providers for instance CircleCI and TravisCI at the same time and you run the RSpec test suite then you need to have separate API token for RSpec executed on CircleCI and a separate API token for RSpec test suite executed on the TravisCI.
@@ -2383,6 +2422,20 @@ bundle exec rake knapsack_pro:queue:rspec
 
 The test file pattern and exclude pattern support any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob).
 
+#### How to run a specific list of test files or only some tests from test file?
+
+:information_source: If you don't want to use the pattern [`KNAPSACK_PRO_TEST_FILE_PATTERN`](#how-can-i-run-tests-from-multiple-directories) to define a list of tests to run then read below.
+
+If you want to run a specific list of test files that are explicitly defined by you or auto-generated by some kind of script you created then please use:
+
+`KNAPSACK_PRO_TEST_FILE_LIST=spec/features/dashboard_spec.rb,spec/models/user.rb:10,spec/models/user.rb:29`
+
+Note `KNAPSACK_PRO_TEST_FILE_LIST` must be a list of test files comma separated. You can provide line number for tests inside of spec file in case of RSpec (this way you can run only one test or a group of tests from RSpec spec file). You can provide the same file a few times with different test line number.
+
+Note when you set `KNAPSACK_PRO_TEST_FILE_LIST` then below environment variables are ignored:
+* `KNAPSACK_PRO_TEST_FILE_PATTERN`
+* `KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN`
+
 #### How to use CodeClimate with knapsack_pro?
 
 You can check CodeClimate docs about [parallel tests](https://docs.codeclimate.com/docs/configuring-test-coverage#section-parallel-tests) and [multiple test suites](https://docs.codeclimate.com/docs/configuring-test-coverage#section-multiple-test-suites).
@@ -2433,6 +2486,32 @@ I recommend to record timing of a new test suite with `API token A` and knapsack
 
 When you want to go back from Queue Mode to Regular Mode then the fact of using the same API token could cause edge cases that some builds might not be well balanced in Regular Mode. That is why I recommend using separate API token for Regular Mode and Queue Mode. If you plan to use only Queue Mode then no worry.
 
+#### How to stop running tests on the first failed test (fail fast tests in RSpec)?
+
+If you want to stop running tests as soon as one of it fails then you can pass [--fail-fast](https://relishapp.com/rspec/rspec-core/docs/command-line/fail-fast-option) RSpec option to knapsack_pro:
+
+```
+# Regular Mode
+bundle exec rake "knapsack_pro:rspec[--fail-fast]"
+
+# Queue Mode
+bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]"
+```
+
+You may add a parameter to tell RSpec to stop running the test suite after N failed tests, for example: `--fail-fast=3`.
+
+```
+Note there is no = sign on purpose here:
+
+# Regular Mode
+bundle exec rake "knapsack_pro:rspec[--fail-fast 3]"
+
+# Queue Mode
+bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]"
+```
+
+There is a downside to it. If you stop running tests then tests that were never run will have no recorded timing of execution and because of that, the future CI build might have tests split across CI nodes in no optimal way.
+
 ### Questions around data usage and security
 
 #### What data is sent to your servers?

data/lib/knapsack_pro/config/env.rb

@@ -51,6 +51,10 @@ module KnapsackPro
           ENV['KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN']
         end
 
+        def test_file_list
+          ENV['KNAPSACK_PRO_TEST_FILE_LIST']
+        end
+
         def test_dir
           ENV['KNAPSACK_PRO_TEST_DIR']
         end

data/lib/knapsack_pro/test_file_finder.rb

@@ -21,6 +21,10 @@ module KnapsackPro
     attr_reader :test_file_pattern
 
     def test_files
+      if KnapsackPro::Config::Env.test_file_list
+        return KnapsackPro::Config::Env.test_file_list.split(',').map(&:strip)
+      end
+
       test_file_paths = Dir.glob(test_file_pattern).uniq
 
       excluded_test_file_paths =

data/lib/knapsack_pro/version.rb

@@ -1,3 +1,3 @@
 module KnapsackPro
-  VERSION = '1.10.1'
+  VERSION = '1.11.0'
 end

data/spec/knapsack_pro/config/env_spec.rb

@@ -167,6 +167,20 @@ describe KnapsackPro::Config::Env do
     end
   end
 
+  describe '.test_file_list' do
+    subject { described_class.test_file_list }
+
+    context 'when ENV exists' do
+      let(:test_file_list) { 'spec/features/dashboard_spec.rb,spec/models/user.rb:10,spec/models/user.rb:29' }
+      before { stub_const("ENV", { 'KNAPSACK_PRO_TEST_FILE_LIST' => test_file_list }) }
+      it { should eq test_file_list }
+    end
+
+    context "when ENV doesn't exist" do
+      it { should be_nil }
+    end
+  end
+
   describe '.test_dir' do
     subject { described_class.test_dir }
 

data/spec/knapsack_pro/test_file_finder_spec.rb

@@ -52,5 +52,31 @@ describe KnapsackPro::TestFileFinder do
         ])
       end
     end
+
+    context 'when KNAPSACK_PRO_TEST_FILE_LIST is defined' do
+      # added spaces next to comma to check space is removed later
+      let(:test_file_list) { 'spec/bar_spec.rb,spec/foo_spec.rb, spec/time_helpers_spec.rb:10 , spec/time_helpers_spec.rb:38' }
+
+      before do
+        stub_const("ENV", { 'KNAPSACK_PRO_TEST_FILE_LIST' => test_file_list })
+      end
+
+      it do
+        expect(subject).to eq([
+          {
+            'path' => 'spec/bar_spec.rb',
+          },
+          {
+            'path' => 'spec/foo_spec.rb',
+          },
+          {
+            'path' => 'spec/time_helpers_spec.rb:10',
+          },
+          {
+            'path' => 'spec/time_helpers_spec.rb:38',
+          },
+        ])
+      end
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: knapsack_pro
 version: !ruby/object:Gem::Version
-  version: 1.10.1
+  version: 1.11.0
 platform: ruby
 authors:
 - ArturT
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-05-02 00:00:00.000000000 Z
+date: 2019-05-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake