knapsack_pro 0.27.0 → 0.28.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: aeedb148388f2fda48884dbd34b294419fb55711
-  data.tar.gz: 43a1a930c3e5a6703719d38a7dfff0ac3131fed8
+  metadata.gz: 558ebb63c8dd85721ac6b3b0fd3a4ec13a84727f
+  data.tar.gz: d28a11954c0412babd8f6de1f1d1c9171f371d6c
 SHA512:
-  metadata.gz: cf278d4eecf850091463ad6ae7591e91c6993e7e82aa0a0d5b0f21beb6c0b25ef03ec742bb9e116b0b89861e52fa40be64aafd8d6e403459294ffae6cc7aec03
-  data.tar.gz: 7cf8746554c01f34b7a6570f7d952574bcd819cae080244596d99d29df5133468874071310be3018b834a6d052a9683c48bea5d111cef162d682843b0fa2c41c
+  metadata.gz: e31031c889f546e3fa8d19c719cf6c3ddf9cf8e76c5728c915959e05e02e3762577a8aa06bb3d3434e9d308a0c4705b5d395f3eed6368d10fe6cfbc449cefc22
+  data.tar.gz: ce06127801cc30c4d53a53e3566dfebc262b3b7d8339a3ab330b21fae49e26e521b1528c1cc08affa6c295dd2167497f39ea1405e6c187898f9d96a9d3300c94

data/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 * TODO
 
+### 0.28.0
+
+* Show at the end of `knapsack_pro:queue:rspec` command the example how to run all tests executed for the CI node in the development environment.
+* Show for each intermediate request to Knapsack Pro API queue how to run a subset of tests fetched from API.
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v0.27.0...v0.28.0
+
 ### 0.27.0
 
 * Save build subset to API even when no test files were executed on CI node. Add warnings to notify why the test files were not executed on CI node in particular mode: regular or queue mode.

data/README.md

@@ -109,6 +109,7 @@ The knapsack_pro has also [queue mode](#queue-mode) to get most optimal test sui
     - [How is that data secured?](#how-is-that-data-secured)
     - [Who has access to the data?](#who-has-access-to-the-data)
   - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result)
+  - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests)
   - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again)
 - [Gem tests](#gem-tests)
   - [Spec](#spec)
@@ -347,17 +348,25 @@ There might be some cached test suite splits for git commits you run in past for
 
 * If you are not using one of [supported CI providers](#supported-ci-providers) then please note that knapsack_pro gem doesn't know what is CI build ID in order to generated queue for particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit.
 
-  To avoid this you can specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`.
+  To avoid this you should specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`.
 
 * Note that in the Queue Mode by default you cannot retry the failed CI node with exactly the same subset of tests that were run on the CI node in the first place. It's possible in regular mode ([read more](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed)). If you want to have similar behavior in Queue Mode you need to explicitly [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
 
   By default the Queue Mode works this way:
 
-  * If you retry the failed build and your all CI nodes will start a new build then there will be a new dynamic test suite split across CI nodes.
+  * If you retry the failed build and your all CI nodes will start a new build then there will be a new dynamic test suite split across CI nodes. The reason is that the most of the CI providers schedule a new CI build with different ID when you retry CI build. They retry all CI nodes again. In that case you don't have to worry with below edge cases because the CI build ID will be different so a new queue will be initialized on Knapsack Pro API side and all retried CI node will connect to that queue.
 
-  * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes are still running. The retried CI node will be connected to the queue consumed by still running CI nodes.
+  Edge cases:
 
-  * In the case when there are no running CI nodes then your retried CI node will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run.
+  * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes are still running. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The retried CI node will be connected to the queue consumed by still running CI nodes. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
+
+  * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes already finished work. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The fact is all CI nodes finished work so the queue was consumed.
+    * If you retry CI node in first hour since the CI build started for the first time then the retried CI node won't execute tests because the queue was consumed. There is important reason why it works like that. For instance some CI providers like Buildkite allows to start CI node later than the others so sometimes the particular CI node may start work while all other CI nodes finished work. In that case we don't want to run tests on the CI node because queue was already consumed. We don't know whether the CI node is part of the build or it is retried CI node hence the 1 hour lock on initializing a new queue.
+    * If you retry CI node after 1 hour since the CI build started for the first time then the retried CI node will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node).
+
+  * When you use unsupported CI provider by knapack_pro gem or you forget to set unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` per CI build then:
+    * when you retry single CI node then it will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run.
+    * when you retry all CI nodes then a new queue will be initialized and all CI nodes will connect to it.
 
 ### Extra configuration for Queue Mode
 
@@ -382,6 +391,8 @@ There might be some cached test suite splits for git commits you run in past for
 
        [knapsack_pro] {"queue_name"=>nil, "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]}
 
+  To [reproduce tests executed on CI node](#for-knapsack_pro-queue-mode) in development environment please see FAQ.
+
 ### Supported test runners in queue mode
 
 At this moment the queue mode works for:
@@ -688,34 +699,42 @@ If you were running your tests with `--order random` on your CI then you can add
 
 #### for knapsack_pro queue mode
 
-To retry the particular CI node do this on your machine:
+There are a few ways to reproduce tests executed on CI node in your development environment.
 
-    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
-    KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
-    KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
-    KNAPSACK_PRO_CI_NODE_TOTAL=2 \
-    KNAPSACK_PRO_CI_NODE_INDEX=0 \
-    KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
-    bundle exec rake "knapsack_pro:queue:rspec"
+* At the end of `knapsack_pro:queue:rspec` results you will find example of command that you can copy and paste to your development machine. It will run all tests executed on the CI node in a single run.
 
-If you were running your tests with `--order random` on your CI like this:
+* For each intermediate request to Knapsack Pro API queue you will also find example of command to run a subset of tests fetched from API. This might be helpful when you use `--order random` for rspec and you would like to reproduce the tests with the same seed.
 
-    bundle exec rake "knapsack_pro:queue:rspec[--order random]"
+* You can also retry tests and record the time execution data for them again for the particular CI node. Note you must be checkout on the same branch and git commit as your CI node was.
 
-Then you can find the seed number visible in rspec output:
+  To retry the particular CI node do this on your machine:
 
-    (...)
-    Randomized with seed 123
+      KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
+      KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
+      KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
+      KNAPSACK_PRO_CI_NODE_TOTAL=2 \
+      KNAPSACK_PRO_CI_NODE_INDEX=0 \
+      KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
+      bundle exec rake "knapsack_pro:queue:rspec"
 
-You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node:
+  If you were running your tests with `--order random` on your CI like this:
 
-    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
-    KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
-    KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
-    KNAPSACK_PRO_CI_NODE_TOTAL=2 \
-    KNAPSACK_PRO_CI_NODE_INDEX=0 \
-    KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
-    bundle exec rake "knapsack_pro:queue:rspec[--seed 123]"
+      bundle exec rake "knapsack_pro:queue:rspec[--order random]"
+
+  Then you can find the seed number visible in rspec output:
+
+      (...)
+      Randomized with seed 123
+
+  You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node:
+
+      KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \
+      KNAPSACK_PRO_REPOSITORY_ADAPTER=git \
+      KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \
+      KNAPSACK_PRO_CI_NODE_TOTAL=2 \
+      KNAPSACK_PRO_CI_NODE_INDEX=0 \
+      KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
+      bundle exec rake "knapsack_pro:queue:rspec[--seed 123]"
 
 ### What happens when Knapsack Pro API is not available/not reachable temporarily?
 
@@ -834,6 +853,13 @@ When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro
 This means RSpec will remember failed tests so far and it will present them at the end of each executed test subset.
 You can see the list of all failed test files at the end of knapack_pro queue mode command.
 
+### Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?
+
+RSpec collects information about pending tests and presents it at the end of RSpec result.
+When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute.
+This means RSpec will remember pending tests so far and it will present them at the end of each executed test subset.
+You can see the list of all pending test files at the end of knapack_pro queue mode command.
+
 ### Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?
 
 No. In Queue Mode the RSpec configuration is updated every time when knapsack_pro gem gets a new set of test files from the Knapsack Pro API and it looks in knapsack_pro output like RSpec was loaded many times but in fact, it loads your project environment only once.

data/lib/knapsack_pro/runners/queue/rspec_runner.rb

@@ -11,30 +11,52 @@ module KnapsackPro
 
           runner = new(KnapsackPro::Adapters::RSpecAdapter)
 
-          cli_args = (args || '').split
-          run_tests(runner, true, cli_args, 0)
+          cli_args = (args || '').split + [
+            '--default-path', runner.test_dir,
+          ]
+          run_tests(runner, true, cli_args, 0, [])
         end
 
-        def self.run_tests(runner, can_initialize_queue, args, exitstatus)
+        def self.run_tests(runner, can_initialize_queue, args, exitstatus, all_test_file_paths)
           test_file_paths = runner.test_file_paths(can_initialize_queue: can_initialize_queue)
 
           if test_file_paths.empty?
+            unless all_test_file_paths.empty?
+              cli_args = args + all_test_file_paths
+
+              log_rspec_command(cli_args, :end_of_queue)
+            end
+
             KnapsackPro::Report.save_node_queue_to_api
             exit(exitstatus)
           else
             subset_queue_id = KnapsackPro::Config::EnvGenerator.set_subset_queue_id
             ENV['KNAPSACK_PRO_SUBSET_QUEUE_ID'] = subset_queue_id
 
-            cli_args = args + [
-              '--default-path', runner.test_dir,
-            ] + test_file_paths
+            all_test_file_paths += test_file_paths
+            cli_args = args + test_file_paths
+
+            log_rspec_command(cli_args, :subset_queue)
+
             options = RSpec::Core::ConfigurationOptions.new(cli_args)
             exit_code = RSpec::Core::Runner.new(options).run($stderr, $stdout)
             exitstatus = exit_code if exit_code != 0
             RSpec.world.example_groups.clear
 
-            run_tests(runner, false, args, exitstatus)
+            run_tests(runner, false, args, exitstatus, all_test_file_paths)
+          end
+        end
+
+        private
+
+        def self.log_rspec_command(cli_args, type)
+          case type
+          when :subset_queue
+            KnapsackPro.logger.info("To retry in development the subset of tests fetched from API queue please run below command on your machine. If you use --order random then remember to add proper --seed 123 that you will find at the end of rspec command.")
+          when :end_of_queue
+            KnapsackPro.logger.info("To retry in development the tests for this CI node please run below command on your machine. It will run all tests in a single run. If you need to reproduce a particular subset of tests fetched from API queue then above after each request to Knapsack Pro API you will find example rspec command.")
           end
+          KnapsackPro.logger.info("bundle exec rspec " + cli_args.join(' '))
         end
       end
     end

data/lib/knapsack_pro/version.rb

@@ -1,3 +1,3 @@
 module KnapsackPro
-  VERSION = '0.27.0'
+  VERSION = '0.28.0'
 end

data/spec/knapsack_pro/runners/queue/rspec_runner_spec.rb

@@ -2,7 +2,10 @@ describe KnapsackPro::Runners::Queue::RSpecRunner do
   describe '.run' do
     let(:test_suite_token_rspec) { 'fake-token' }
     let(:queue_id) { 'fake-queue-id' }
-    let(:runner) { double }
+    let(:test_dir) { 'fake-test-dir' }
+    let(:runner) do
+      instance_double(described_class, test_dir: test_dir)
+    end
 
     subject { described_class.run(args) }
 
@@ -22,7 +25,7 @@ describe KnapsackPro::Runners::Queue::RSpecRunner do
 
       it do
         result = double
-        expect(described_class).to receive(:run_tests).with(runner, true, ['--example-arg', 'example-value'], 0).and_return(result)
+        expect(described_class).to receive(:run_tests).with(runner, true, ['--example-arg', 'example-value', '--default-path', 'fake-test-dir'], 0, []).and_return(result)
 
         expect(subject).to eq result
       end
@@ -33,7 +36,7 @@ describe KnapsackPro::Runners::Queue::RSpecRunner do
 
       it do
         result = double
-        expect(described_class).to receive(:run_tests).with(runner, true, [], 0).and_return(result)
+        expect(described_class).to receive(:run_tests).with(runner, true, ['--default-path', 'fake-test-dir'], 0, []).and_return(result)
 
         expect(subject).to eq result
       end
@@ -41,15 +44,12 @@ describe KnapsackPro::Runners::Queue::RSpecRunner do
   end
 
   describe '.run_tests' do
-    let(:test_dir) { 'fake-test-dir' }
-    let(:runner) do
-      instance_double(described_class, test_dir: test_dir)
-    end
+    let(:runner) { instance_double(described_class) }
     let(:can_initialize_queue) { double(:can_initialize_queue) }
-    let(:args) { ['--example-arg', 'example-value'] }
+    let(:args) { ['--example-arg', 'example-value', '--default-path', 'fake-test-dir'] }
     let(:exitstatus) { double }
 
-    subject { described_class.run_tests(runner, can_initialize_queue, args, exitstatus) }
+    subject { described_class.run_tests(runner, can_initialize_queue, args, exitstatus, []) }
 
     before do
       expect(runner).to receive(:test_file_paths).with(can_initialize_queue: can_initialize_queue).and_return(test_file_paths)
@@ -67,7 +67,7 @@ describe KnapsackPro::Runners::Queue::RSpecRunner do
         options = double
         expect(RSpec::Core::ConfigurationOptions).to receive(:new).with([
           '--example-arg', 'example-value',
-          '--default-path', test_dir,
+          '--default-path', 'fake-test-dir',
           'a_spec.rb', 'b_spec.rb',
         ]).and_return(options)
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: knapsack_pro
 version: !ruby/object:Gem::Version
-  version: 0.27.0
+  version: 0.28.0
 platform: ruby
 authors:
 - ArturT
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-02-18 00:00:00.000000000 Z
+date: 2017-02-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake