knapsack_pro 0.17.0 → 0.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 377525e667c4025224f3d6df9b10f7f474b49ceb
-  data.tar.gz: 7787796cc41143405f662de7dbbe5d46c50489f8
+  metadata.gz: 776a8b23ab2904168e0045473328847f98cf33ba
+  data.tar.gz: de729e204e176211b212fcbcbcd9b1962cc5b753
 SHA512:
-  metadata.gz: e83e3e9032c126c1e9551d195b4858ad176dca3618c8892f19f14ec5fb16a2326748ae8558cb6e7bd5f1748addab180d142af3c63eb8d5f0c9ec2c89f4ac1928
-  data.tar.gz: ce99dbd6384e65ab7163ec8f35b3bc540a72f4fc59482b7b44b4d85e0163a58a837b291611efede2c75b21908f30ecb4994aa00ca834c1a1a8bce26cbec72302
+  metadata.gz: e6e5a91f7d9acd006da5c5b5b37039f3aaf7bc180be673fb764fb4859495cc839d0d78d4f146bfdfd515b0cca8ac8dddb275440909b6046d22e5c1b2dd775fee
+  data.tar.gz: d17eca6592fcaf86c104d42b88aae035fc3ad7b4dfb058f35d984f2452455a1811f2255c46ffe40bfa4a79455eef952d9e58494577a24e6f8e4e392dd64fa6af

data/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 * TODO
 
+### 0.18.0
+
+* Add support for knapsack_pro queue mode
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/20
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v0.17.0...v0.18.0
+
 ### 0.17.0
 
 * Enable fallback mode for SocketError when failed to open TCP connection to http or https API endpoint.

data/README.md

@@ -21,7 +21,7 @@ __Would you like to try knapsack_pro gem?__ You can [get API token here](http://
 
 * If your __project is open source__ then you can use Knapsack Pro for free. Just add a link back to the page http://knapsackpro.com and please let me know via email (arturtrzop@gmail.com). I will mark your account on KnapsackPro.com as open source.
 
-* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to paid plan is a good way to validate that and a way to get support from happy users. Currently, there is no hard limit when the free trial is ending. I'm more interested in getting feedback from users in order to improve the tool. As I saw in the past people want to support the project and are asking for paid plan. Maybe you will be the next one who will join and support the project. Thanks!
+* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to paid plan is a good way to validate that and a way to get support from happy users. Maybe you will be the next one who will join and support the project. Thanks!
 
 # How knapsack_pro works?
 
@@ -33,7 +33,7 @@ Next time when you will run tests you will get proper test files for each CI nod
 
 ## Details
 
-For instance when you will run tests with rake knapsack_pro:rspec then:
+For instance when you will run tests with `rake knapsack_pro:rspec` then:
 
 * information about all your existing test files are sent to API http://docs.knapsackpro.com/api/v1/#build_distributions_subset_post
 * API returns which files should be executed on particular CI node (example KNAPSACK_PRO_CI_NODE_INDEX=0)
@@ -41,6 +41,8 @@ For instance when you will run tests with rake knapsack_pro:rspec then:
 * knapsack_pro will run test files which got from API
 * after tests finished knapsack_pro will send information about time execution of each file to API http://docs.knapsackpro.com/api/v1/#build_subsets_post so data can be used for future test runs
 
+The knapsack_pro has also [queue mode](#queue-mode) to get most optimal test suite split.
+
 # Requirements
 
 * >= Ruby 2.0.0
@@ -66,6 +68,11 @@ For instance when you will run tests with rake knapsack_pro:rspec then:
   - [Repository adapter (How to set up 3 of 3)](#repository-adapter-how-to-set-up-3-of-3)
     - [When you NOT set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER` (default)](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default)
     - [When you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported)](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported)
+- [Queue Mode](#queue-mode)
+  - [How queue mode works?](#how-queue-mode-works)
+  - [How to use queue mode?](#how-to-use-queue-mode)
+  - [Additional info about queue mode](#additional-info-about-queue-mode)
+  - [Supported test runners in queue mode](#supported-test-runners-in-queue-mode)
 - [Extra configuration for CI server](#extra-configuration-for-ci-server)
   - [Info about ENV variables](#info-about-env-variables)
     - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLITE (test suite split based on seed)](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed)
@@ -89,6 +96,14 @@ For instance when you will run tests with rake knapsack_pro:rspec then:
   - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment)
   - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily)
   - [How can I change log level?](#how-can-i-change-log-level)
+  - [How to split tests based on test level instead of test file level?](#how-to-split-tests-based-on-test-level-instead-of-test-file-level)
+    - [A. Create multiple small test files](#a-create-multiple-small-test-files)
+    - [B. Use tags to mark set of tests in particular test file](#b-use-tags-to-mark-set-of-tests-in-particular-test-file)
+  - [How to make knapsack_pro works for forked repositories of my project?](#how-to-make-knapsack_pro-works-for-forked-repositories-of-my-project)
+  - [Questions around data usage and security](#questions-around-data-usage-and-security)
+    - [What data is sent to your servers?](#what-data-is-sent-to-your-servers)
+    - [How is that data secured?](#how-is-that-data-secured)
+    - [Who has access to the data?](#who-has-access-to-the-data)
 - [Gem tests](#gem-tests)
   - [Spec](#spec)
 - [Contributing](#contributing)
@@ -271,6 +286,41 @@ You can also use git as repository adapter to determine branch and commit hash,
 
 `KNAPSACK_PRO_PROJECT_DIR` - Path to the project on CI node for instance `/home/ubuntu/my-app-repository`. It should be main directory of your repository.
 
+## Queue Mode
+
+knapsack_pro has built in queue mode designed to solve problem with optimal test suite split in case of random time execution of test files caused by
+CI node overload and a random decrease of performance that may affect how long the test files are executed.
+The problem with random time execution of test files may be caused by many things like external requests done in tests.
+
+### How queue mode works?
+
+On the Knapsack Pro API side, there is test files queue generated for your CI build. Each of CI node dynamically asks the Knapsack Pro API for test files
+that should be executed. Thanks to that each CI node will finish tests at the same time.
+
+### How to use queue mode?
+
+Please use this command to run queue mode:
+
+    bundle exec rake knapsack_pro:queue:rspec
+
+If above command fails then you may need to explicitly pass an argument to require `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files:
+
+    bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"
+
+Note if you will run queue mode command for the first time it might be slower.
+The second build should have better optimal test suite split.
+
+### Additional info about queue mode
+
+If you are not using one of supported CI providers then please note that knapsack_pro gem doesn't know what is CI build ID in order to generated queue for particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit.
+To avoid this you can specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`.
+
+### Supported test runners in queue mode
+
+At this moment the queue mode works for:
+
+* RSpec
+
 ## Extra configuration for CI server
 
 ### Info about ENV variables
@@ -359,6 +409,7 @@ Add arguments to knapsack_pro spinach task like this:
 You can install knapsack_pro globally and use binary. For instance:
 
     $ knapsack_pro rspec "--tag custom_tag_name --profile"
+    $ knapsack_pro queue:rspec "--tag custom_tag_name --profile"
     $ knapsack_pro cucumber "--name feature"
     $ knapsack_pro minitest "--verbose --pride"
     $ knapsack_pro spinach "--arg_name value"
@@ -564,6 +615,104 @@ You can change log level by specifying the `KNAPSACK_PRO_LOG_LEVEL` environment
 
 Available values are `debug`, `info`, and `warn`. The default log level is `info`.
 
+### How to split tests based on test level instead of test file level?
+
+If you want to split one big test file (test file with long time execution) across multiple CI nodes then you can:
+
+#### A. Create multiple small test files
+
+Create multiple small test files instead of one long running test file with many test cases.
+A lot of small test files will give you better test suite split results.
+
+#### B. Use tags to mark set of tests in particular test file
+
+Another way is to use tags to mark subset of tests in particular test file and then split tests based on tags.
+
+Here is example of test file with specified tags for describe groups:
+
+```ruby
+# spec/features/something_spec.rb
+describe 'Feature' do
+  describe 'something A', :tagA do
+    it {}
+    it 'another test' {}
+  end
+
+  describe 'something B', :tagB do
+    it {}
+  end
+
+  describe 'something else' do
+    it {}
+  end
+end
+```
+
+You need to create multiple API tokens for different tags. In this example we need 3 different API tokens.
+
+You need to run below commands for each CI node.
+
+    # run only tests with tagA
+    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagA bundle exec rake "knapsack_pro:rspec[--tag tagA]"
+
+    # run only tests with tagB
+    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagB bundle exec rake "knapsack_pro:rspec[--tag tagB]"
+
+    # run other tests without tag A & B
+    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tests_without_tags_A_and_B bundle exec rake "knapsack_pro:rspec[--tag ~tagA --tag ~tagB]"
+
+### How to make knapsack_pro works for forked repositories of my project?
+
+Imagine one of the scenarios, for this example I use the Travis-CI.
+
+* We don’t want to have secrets like the `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in `.travis.yml` in the codebase, because that code is also distributed to clients.
+* Adding it as env variables to Travis itself is tricky: It has to work for pull requests from developer’s forks into our main fork; this conflicts with the way Travis handles secrets. We also need a fallback if the token is not provided (when developers do builds within their own fork).
+
+The solution for this problem is to set `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as env variables in Travis for our main project.
+This won't be accessible on forked repositories so we will run knapsack_pro in fallback mode there.
+This way forked repositories have working test suite but without optimal test suite split across CI nodes.
+
+Create the file `bin/knapsack_pro_rspec` with executable chmod in your main project repository.
+Below example is for rspec. You can change `$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `$KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` if you use cucumber etc.
+
+```
+#!/bin/bash
+if [ "$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC" = "" ]; then
+  KNAPSACK_PRO_ENDPOINT=https://api-disabled-for-fork.knapsackpro.com \
+    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=disabled-for-fork \
+    bundle exec rake knapsack_pro:rspec
+else
+    bundle exec rake knapsack_pro:rspec
+fi
+```
+
+Now you can use `bin/knapsack_pro_rspec` command instead of `bundle exec rake knapsack_pro:rspec`.
+Remember to follow other steps required for your CI provider.
+
+### Questions around data usage and security
+
+#### What data is sent to your servers?
+
+The knapsack_pro gem sends branch name, commit hash, CI total node number, CI index node number, the test file paths like `spec/models/user_spec.rb` and the time execution of each test file path as a float.
+
+Here is the [full specification of the API](http://docs.knapsackpro.com/api/v1/) used by knapsack_pro gem.
+
+#### How is that data secured?
+
+The test file paths can be [encrypted](#test-file-names-encryption) on your CI node with your salt and later send to knapsackpro.com API.
+You generate salt locally and only you can decrypt the test file paths.
+
+Connection with knapsackpro.com server is via https.
+
+Regarding payments we use the BraintreePayments.com and they store credit cards and your private information.
+
+#### Who has access to the data?
+
+I’m the only admin so I can preview data in case you need help with debugging some problem etc. I’m not able to decrypt them without knowing the salt.
+
+When you sign in to your user dashboard then you can preview data for recent 100 builds on CI. If the test file paths are encrypted then you only see hashes for test file paths.
+You need to [decrypt](#how-to-debug-test-file-names) them locally on your machine to find out what each test file hash is.
+
 ## Gem tests
 
 ### Spec

data/bin/knapsack_pro

@@ -7,6 +7,7 @@ arguments = ARGV[1]
 
 MAP = {
   'rspec' => KnapsackPro::Runners::RSpecRunner,
+  'queue:rspec' => KnapsackPro::Runners::Queue::RSpecRunner,
   'cucumber' => KnapsackPro::Runners::CucumberRunner,
   'minitest' => KnapsackPro::Runners::MinitestRunner,
   'spinach' => KnapsackPro::Runners::SpinachRunner,

data/knapsack_pro.gemspec

@@ -11,7 +11,7 @@ Gem::Specification.new do |spec|
   spec.summary       = %q{Knapsack Pro splits tests across CI nodes and makes sure that tests will run comparable time on each node.}
   spec.description   = %q{Parallel tests across CI server nodes based on each test file's time execution. It uses KnapsackPro.com API.}
   spec.homepage      = "https://github.com/KnapsackPro/knapsack_pro-ruby"
-  spec.license       = "LGPLv3"
+  spec.license       = "LGPL-3.0"
 
   spec.files         = `git ls-files -z`.split("\x0")
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }

data/lib/knapsack_pro.rb

@@ -6,6 +6,7 @@ require 'uri'
 require 'rake/testtask'
 require 'timecop'
 require 'digest'
+require 'securerandom'
 require_relative 'knapsack_pro/version'
 require_relative 'knapsack_pro/utils'
 require_relative 'knapsack_pro/logger_wrapper'
@@ -16,10 +17,12 @@ require_relative 'knapsack_pro/config/ci/buildkite'
 require_relative 'knapsack_pro/config/ci/travis'
 require_relative 'knapsack_pro/config/ci/snap_ci'
 require_relative 'knapsack_pro/config/env'
+require_relative 'knapsack_pro/config/env_generator'
 require_relative 'knapsack_pro/client/api/action'
 require_relative 'knapsack_pro/client/api/v1/base'
 require_relative 'knapsack_pro/client/api/v1/build_distributions'
 require_relative 'knapsack_pro/client/api/v1/build_subsets'
+require_relative 'knapsack_pro/client/api/v1/queues'
 require_relative 'knapsack_pro/client/connection'
 require_relative 'knapsack_pro/repository_adapters/base_adapter'
 require_relative 'knapsack_pro/repository_adapters/env_adapter'
@@ -35,7 +38,10 @@ require_relative 'knapsack_pro/test_flat_distributor'
 require_relative 'knapsack_pro/task_loader'
 require_relative 'knapsack_pro/tracker'
 require_relative 'knapsack_pro/allocator'
+require_relative 'knapsack_pro/queue_allocator'
+require_relative 'knapsack_pro/base_allocator_builder'
 require_relative 'knapsack_pro/allocator_builder'
+require_relative 'knapsack_pro/queue_allocator_builder'
 require_relative 'knapsack_pro/adapters/base_adapter'
 require_relative 'knapsack_pro/adapters/rspec_adapter'
 require_relative 'knapsack_pro/adapters/cucumber_adapter'
@@ -46,6 +52,8 @@ require_relative 'knapsack_pro/runners/rspec_runner'
 require_relative 'knapsack_pro/runners/cucumber_runner'
 require_relative 'knapsack_pro/runners/minitest_runner'
 require_relative 'knapsack_pro/runners/spinach_runner'
+require_relative 'knapsack_pro/runners/queue/base_runner'
+require_relative 'knapsack_pro/runners/queue/rspec_runner'
 require_relative 'knapsack_pro/crypto/encryptor'
 require_relative 'knapsack_pro/crypto/decryptor'
 require_relative 'knapsack_pro/crypto/digestor'

data/lib/knapsack_pro/adapters/base_adapter.rb

@@ -16,6 +16,12 @@ module KnapsackPro
           bind_time_tracker
           bind_save_report
         end
+
+        if KnapsackPro::Config::Env.queue_recording_enabled?
+          KnapsackPro.logger.info('Test suite time execution queue recording enabled.')
+          bind_time_tracker
+          bind_save_queue_report
+        end
       end
 
       def bind_time_tracker
@@ -25,6 +31,10 @@ module KnapsackPro
       def bind_save_report
         raise NotImplementedError
       end
+
+      def bind_save_queue_report
+        raise NotImplementedError
+      end
     end
   end
 end

data/lib/knapsack_pro/adapters/rspec_adapter.rb

@@ -49,6 +49,14 @@ module KnapsackPro
           end
         end
       end
+
+      def bind_save_queue_report
+        ::RSpec.configure do |config|
+          config.after(:suite) do
+            KnapsackPro::Report.save_subset_queue_to_file
+          end
+        end
+      end
     end
 
     # This is added to provide backwards compatibility

data/lib/knapsack_pro/allocator.rb

@@ -8,24 +8,13 @@ module KnapsackPro
     end
 
     def test_file_paths
-      encrypted_test_files = KnapsackPro::Crypto::Encryptor.call(test_files)
-      action = KnapsackPro::Client::API::V1::BuildDistributions.subset(
-        commit_hash: repository_adapter.commit_hash,
-        branch: repository_adapter.branch,
-        node_total: ci_node_total,
-        node_index: ci_node_index,
-        test_files: encrypted_test_files,
-      )
-      connection = KnapsackPro::Client::Connection.new(action)
+      connection = KnapsackPro::Client::Connection.new(build_action)
       response = connection.call
       if connection.success?
         raise ArgumentError.new(response) if connection.errors?
-        decrypted_test_files = KnapsackPro::Crypto::Decryptor.call(test_files, response['test_files'])
-        KnapsackPro::TestFilePresenter.paths(decrypted_test_files)
+        prepare_test_files(response)
       else
-        test_flat_distributor = KnapsackPro::TestFlatDistributor.new(test_files, ci_node_total)
-        test_files_for_node_index = test_flat_distributor.test_files_for_node(ci_node_index)
-        KnapsackPro::TestFilePresenter.paths(test_files_for_node_index)
+        fallback_test_files
       end
     end
 
@@ -35,5 +24,30 @@ module KnapsackPro
       :ci_node_total,
       :ci_node_index,
       :repository_adapter
+
+    def encrypted_test_files
+      KnapsackPro::Crypto::Encryptor.call(test_files)
+    end
+
+    def build_action
+      KnapsackPro::Client::API::V1::BuildDistributions.subset(
+        commit_hash: repository_adapter.commit_hash,
+        branch: repository_adapter.branch,
+        node_total: ci_node_total,
+        node_index: ci_node_index,
+        test_files: encrypted_test_files,
+      )
+    end
+
+    def prepare_test_files(response)
+      decrypted_test_files = KnapsackPro::Crypto::Decryptor.call(test_files, response['test_files'])
+      KnapsackPro::TestFilePresenter.paths(decrypted_test_files)
+    end
+
+    def fallback_test_files
+      test_flat_distributor = KnapsackPro::TestFlatDistributor.new(test_files, ci_node_total)
+      test_files_for_node_index = test_flat_distributor.test_files_for_node(ci_node_index)
+      KnapsackPro::TestFilePresenter.paths(test_files_for_node_index)
+    end
   end
 end

data/lib/knapsack_pro/allocator_builder.rb

@@ -1,9 +1,5 @@
 module KnapsackPro
-  class AllocatorBuilder
-    def initialize(adapter_class)
-      @adapter_class = adapter_class
-    end
-
+  class AllocatorBuilder < BaseAllocatorBuilder
     def allocator
       KnapsackPro::Allocator.new(
         test_files: test_files,
@@ -12,29 +8,5 @@ module KnapsackPro
         repository_adapter: repository_adapter,
       )
     end
-
-    def test_dir
-      test_file_pattern.split('/').first
-    end
-
-    private
-
-    attr_reader :adapter_class
-
-    def env
-      KnapsackPro::Config::Env
-    end
-
-    def repository_adapter
-      KnapsackPro::RepositoryAdapterInitiator.call
-    end
-
-    def test_file_pattern
-      TestFilePattern.call(adapter_class)
-    end
-
-    def test_files
-      KnapsackPro::TestFileFinder.call(test_file_pattern)
-    end
   end
 end

data/lib/knapsack_pro/base_allocator_builder.rb

@@ -0,0 +1,35 @@
+module KnapsackPro
+  class BaseAllocatorBuilder
+    def initialize(adapter_class)
+      @adapter_class = adapter_class
+    end
+
+    def allocator
+      raise NotImplementedError
+    end
+
+    def test_dir
+      test_file_pattern.split('/').first
+    end
+
+    private
+
+    attr_reader :adapter_class
+
+    def env
+      KnapsackPro::Config::Env
+    end
+
+    def repository_adapter
+      KnapsackPro::RepositoryAdapterInitiator.call
+    end
+
+    def test_file_pattern
+      TestFilePattern.call(adapter_class)
+    end
+
+    def test_files
+      KnapsackPro::TestFileFinder.call(test_file_pattern)
+    end
+  end
+end