knapsack_pro 0.9.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0a2674eb030549459a86c2b78ad2242a08b708e7
-  data.tar.gz: abc36e6d8db2242652694a2394dc724f878caf2d
+  metadata.gz: 7b9017e6211e870280789b77926d157894d6ea0a
+  data.tar.gz: 72749308b9dcc9737efb3a210853c7c69d17669d
 SHA512:
-  metadata.gz: 32bcf9bf8cf8fbad5f254bd9488b11d2f09cf569f0cc63165afeba51bd1cb85e728e780183ca7844ce8a922f89e12d1c76cb5f48b2763ecef6c77807082f259d
-  data.tar.gz: f1ca9ce31b3f29fedfd175296d3c306588200ca9d7ea9f87e322c7f423c1b041a1cf662b82fcef581a9fe0203bff22ab8325a23484cf22e57ce7d8c98fab59b5
+  metadata.gz: 5a83dd2ef1307dd014a8ec3151e959e2806551a11f238a4f7d06bb80c3b6422a33b88b0f13b1c8e92c4ec86a5b8605a74b5f76c83ad644d7d5eb8f58954c5171
+  data.tar.gz: 2fa61c851548c36e23dd31888d1bffcaa2226eaf56893e92bba7bec4d4547b6a6f47477a219fde05f698b28b524d618f96a7836428b3ce30327dc1681a422f82

data/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 * TODO
 
+### 0.10.0
+
+* Add new environment variable `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT`. The default value is true.
+
+    It means when you run test suite again for the same commit hash and total number of nodes and for the same branch
+    then you will get exactly the same test suite split.
+    This is the new default behavior for the knapsack_pro gem. Thanks to that when tests on one of your node failed
+    you can retry the node with exactly the same subset of tests that were run on the node in the first place.
+
+    There is one edge case. When you run tests for the first time and there is no data collected about time execution of your tests then
+    we need to collect data to prepare the first test suite split. The second run of your tests will have fixed test suite split.
+    To compare if all your CI nodes are running based on the same test suite split seed you can check the value for seed in knapsack logging message
+    before your test starts. The message looks like:
+
+        [knapsack_pro] Test suite split seed: 8a606431-02a1-4766-9878-0ea42a07ad21
+
+  * Show test suite split seed in logger based on `build_distribution_id` from Knapsack Pro API.
+  * Send `fixed_test_suite_split` param to build distribution Knapsack Pro API endpoint.
+
+  Related issues:
+
+  * https://github.com/KnapsackPro/knapsack_pro-ruby/issues/15
+  * https://github.com/KnapsackPro/knapsack_pro-ruby/issues/12
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v0.9.0...v0.10.0
+
 ### 0.9.0
 
 * Add https support for Knapsack Pro API endpoint

data/README.md

@@ -68,6 +68,7 @@ For instance when you will run tests with rake knapsack_pro:rspec then:
     - [When you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported)](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported)
 - [Extra configuration for CI server](#extra-configuration-for-ci-server)
   - [Info about ENV variables](#info-about-env-variables)
+    - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLITE (test suite split based on seed)](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed)
     - [Environment variables for debugging gem](#environment-variables-for-debugging-gem)
   - [Passing arguments to rake task](#passing-arguments-to-rake-task)
     - [Passing arguments to rspec](#passing-arguments-to-rspec)
@@ -268,6 +269,38 @@ In case when you use other CI provider for instance [Jenkins](https://jenkins-ci
 
 `KNAPSACK_PRO_CI_NODE_INDEX` - index of current CI node starts from 0. Second CI node should have `KNAPSACK_PRO_CI_NODE_INDEX=1`.
 
+#### KNAPSACK_PRO_FIXED_TEST_SUITE_SPLITE (test suite split based on seed)
+
+* `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` (default)
+
+    It means when you run test suite again for the same commit hash and total number of nodes and for the same branch
+    then you will get exactly the same test suite split.
+
+    Thanks to that when tests on one of your node failed you can retry the node with exactly the same subset of tests that were run on the node in the first place.
+
+    There is one edge case. When you run tests for the first time and there is no data collected about time execution of your tests then
+    we need to collect data to prepare the first test suite split. The second run of your tests will have fixed test suite split.
+
+    To compare if all your CI nodes are running based on the same test suite split seed you can check the value for seed in knapsack logging message
+    before your test starts. The message looks like:
+
+        [knapsack_pro] Test suite split seed: 8a606431-02a1-4766-9878-0ea42a07ad21
+
+* `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=false`
+
+    When you disable fixed test suite split then your will get test suite split based on most up to date data about your test suite time execution.
+    For instance, when you run tests for the second time for the same commit hash then your will get more optimal test suite split than it was on the first run.
+
+    Don't disable fixed test suite split when:
+
+    * you expect to run the same subset of test suite multiple times for the same node (for instance your would like to retry only single CI node that failed)
+
+        Example of issue: https://github.com/KnapsackPro/knapsack_pro-ruby/issues/15
+
+    * you start your tests not at the same time across your CI nodes. For instance, one of the CI node finished faster than the other CI node started. This would change the seed for the second CI node that started later.
+
+        Example of issue: https://github.com/KnapsackPro/knapsack_pro-ruby/issues/12
+
 #### Environment variables for debugging gem
 
 `KNAPSACK_PRO_ENDPOINT` - Default value is `http://api.knapsackpro.com` which is endpoint for [Knapsack Pro API](http://docs.knapsackpro.com).

data/lib/knapsack_pro/client/api/v1/build_distributions.rb

@@ -9,6 +9,7 @@ module KnapsackPro
                 endpoint_path: '/v1/build_distributions/subset',
                 http_method: :post,
                 request_hash: {
+                  :fixed_test_suite_split => KnapsackPro::Config::Env.fixed_test_suite_split,
                   :commit_hash => args.fetch(:commit_hash),
                   :branch => args.fetch(:branch),
                   :node_total => args.fetch(:node_total),

data/lib/knapsack_pro/client/connection.rb

@@ -65,6 +65,15 @@ module KnapsackPro
         nil
       end
 
+      def seed
+        return if @response.nil? || @response == ''
+        response['build_distribution_id']
+      end
+
+      def has_seed?
+        !seed.nil?
+      end
+
       def post
         uri = URI.parse(endpoint_url)
         http = Net::HTTP.new(uri.host, uri.port)
@@ -78,6 +87,7 @@ module KnapsackPro
         request_uuid = http_response.header['X-Request-Id']
 
         logger.info("API request UUID: #{request_uuid}")
+        logger.info("Test suite split seed: #{seed}") if has_seed?
         logger.info('API response:')
         if errors?
           logger.error(response)

data/lib/knapsack_pro/config/env.rb

@@ -61,6 +61,10 @@ module KnapsackPro
           end
         end
 
+        def fixed_test_suite_split
+          ENV.fetch('KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT', true)
+        end
+
         def test_suite_token
           required_env('KNAPSACK_PRO_TEST_SUITE_TOKEN')
         end

data/lib/knapsack_pro/version.rb

@@ -1,3 +1,3 @@
 module KnapsackPro
-  VERSION = '0.9.0'
+  VERSION = '0.10.0'
 end

data/spec/fixtures/vcr_cassettes/api/v1/build_distributions/subset/success.yml

@@ -29,24 +29,24 @@ http_interactions:
       Content-Type:
       - application/json; charset=utf-8
       Etag:
-      - W/"2348448eb68a360747d21fe1bb304d5b"
+      - W/"bba747f5e635a765e120718b7876d174"
       Cache-Control:
       - max-age=0, private, must-revalidate
       X-Request-Id:
-      - 66e07e14-0dfd-43f9-b331-2a930141dfef
+      - 190e26fb-676b-4b85-9737-8a90d0cce310
       X-Runtime:
-      - '0.158064'
+      - '0.639590'
       Server:
       - WEBrick/1.3.1 (Ruby/2.2.3/2015-08-18)
       Date:
-      - Fri, 23 Oct 2015 20:12:23 GMT
+      - Tue, 28 Jun 2016 20:40:35 GMT
       Content-Length:
-      - '74'
+      - '137'
       Connection:
       - Keep-Alive
     body:
       encoding: UTF-8
-      string: '{"node_index":1,"test_files":[{"path":"b_spec.rb","time_execution":null}]}'
+      string: '{"build_distribution_id":"1f905f45-c203-40f1-9891-d9a91dd11245","node_index":1,"test_files":[{"path":"b_spec.rb","time_execution":null}]}'
     http_version: 
-  recorded_at: Fri, 23 Oct 2015 20:12:23 GMT
+  recorded_at: Tue, 28 Jun 2016 20:40:35 GMT
 recorded_with: VCR 2.9.3

data/spec/knapsack_pro/client/api/v1/build_distributions_spec.rb

@@ -1,5 +1,6 @@
 describe KnapsackPro::Client::API::V1::BuildDistributions do
   describe '.subset' do
+    let(:fixed_test_suite_split) { double }
     let(:commit_hash) { double }
     let(:branch) { double }
     let(:node_total) { double }
@@ -16,12 +17,17 @@ describe KnapsackPro::Client::API::V1::BuildDistributions do
       )
     end
 
+    before do
+      expect(KnapsackPro::Config::Env).to receive(:fixed_test_suite_split).and_return(fixed_test_suite_split)
+    end
+
     it do
       action = double
       expect(KnapsackPro::Client::API::Action).to receive(:new).with({
         endpoint_path: '/v1/build_distributions/subset',
         http_method: :post,
         request_hash: {
+          fixed_test_suite_split: fixed_test_suite_split,
           commit_hash: commit_hash,
           branch: branch,
           node_total: node_total,

data/spec/knapsack_pro/client/connection_spec.rb

@@ -40,15 +40,17 @@ describe KnapsackPro::Client::Connection do
           "{\"fake\":\"hash\",\"test_suite_token\":\"3fa64859337f6e56409d49f865d13fd7\"}",
           { "Content-Type" => "application/json", "Accept" => "application/json" }
         ).and_return(http_response)
-
-        expect(KnapsackPro).to receive(:logger).exactly(3).and_return(logger)
-        expect(logger).to receive(:info).with('API request UUID: fake-uuid')
-        expect(logger).to receive(:info).with('API response:')
       end
 
       context 'when body response is json' do
         let(:body) { '{"errors": "value"}' }
 
+        before do
+          expect(KnapsackPro).to receive(:logger).exactly(3).and_return(logger)
+          expect(logger).to receive(:info).with('API request UUID: fake-uuid')
+          expect(logger).to receive(:info).with('API response:')
+        end
+
         it do
           parsed_response = { 'errors' => 'value' }
 
@@ -60,9 +62,36 @@ describe KnapsackPro::Client::Connection do
         end
       end
 
+      context 'when body response is json with build_distribution_id' do
+        let(:body) { '{"build_distribution_id": "seed-uuid"}' }
+
+        before do
+          expect(KnapsackPro).to receive(:logger).exactly(4).and_return(logger)
+          expect(logger).to receive(:info).with('API request UUID: fake-uuid')
+          expect(logger).to receive(:info).with("Test suite split seed: seed-uuid")
+          expect(logger).to receive(:info).with('API response:')
+        end
+
+        it do
+          parsed_response = { 'build_distribution_id' => 'seed-uuid' }
+
+          expect(logger).to receive(:info).with(parsed_response)
+
+          expect(subject).to eq(parsed_response)
+          expect(connection.success?).to be true
+          expect(connection.errors?).to be false
+        end
+      end
+
       context 'when body response is empty' do
         let(:body) { '' }
 
+        before do
+          expect(KnapsackPro).to receive(:logger).exactly(3).and_return(logger)
+          expect(logger).to receive(:info).with('API request UUID: fake-uuid')
+          expect(logger).to receive(:info).with('API response:')
+        end
+
         it do
           expect(logger).to receive(:info).with('')
 

data/spec/knapsack_pro/config/env_spec.rb

@@ -219,6 +219,20 @@ describe KnapsackPro::Config::Env do
     end
   end
 
+  describe '.fixed_test_suite_split' do
+    subject { described_class.fixed_test_suite_split }
+
+    context 'when ENV exists' do
+      before { stub_const("ENV", { 'KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT' => false }) }
+      it { should eq false }
+    end
+
+    context "when ENV doesn't exist" do
+      before { stub_const("ENV", {}) }
+      it { should be true }
+    end
+  end
+
   describe '.test_suite_token' do
     subject { described_class.test_suite_token }
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: knapsack_pro
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.10.0
 platform: ruby
 authors:
 - ArturT
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-06-17 00:00:00.000000000 Z
+date: 2016-06-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake