knapsack_pro 1.20.2 → 1.21.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6bbc828d338c810094114953894489db0696b70f013beac451b1c45602c323cd
-  data.tar.gz: ae287454ad0658666d1d0cd85c4eaf91eabe009ee95b05b3a76d719a245395ae
+  metadata.gz: 59df1c0c620e9c92fb18e3963029fd02df6f04754ffe1b5c590cd7ca45159f55
+  data.tar.gz: 5cf6b36b8f908d8dbb9788918f7586c3b0334ec91a3a79afd768fbde3a8088b2
 SHA512:
-  metadata.gz: f70aca52cc3808f0028d036dd8b5057130cf6610ede948075a293c3165ddf46e44e8fdf922e7457808dca8f390ded887adb5f4c95146a5b925535621b1de7a93
-  data.tar.gz: 32e5fb5098e39fdb93694184b16c26603ca1edcdc193ab1c513ab9cd55800fa2c54157b28b365260d2e2231bd3bea6ffcc5b2b1921c9076ab75e37095433e5ef
+  metadata.gz: ae228667bbd897ad31512140d6c6382aefe2fce977748f909b3da7cca7a39495731cde4e079cd8a0247bd2d332fe8809aecf69fe773c34f864e74febc0a4dbbc
+  data.tar.gz: fc8255bc114f010a6a85a14aa98dab171767769e8476c3c422fd92e17198588e767f08bdb1072ca36c5038aef23af4396a8b44fa271b720845f26bd16fbae32b

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Change Log
 
+### 1.21.0
+
+* Automatically detect slow test files for RSpec and split them by test examples when `KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true`
+* Add slow test file pattern `KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` to define RSpec slow test files that should be split by test examples
+* Start sending API token in header `KNAPSACK-PRO-TEST-SUITE-TOKEN` instead of a key `test_suite_token` in JSON payload.
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/106
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.20.2...v1.21.0
+
 ### 1.20.2
 
 * Raise an error when running Cucumber in Queue Mode and Cucumber system process doesn't finish execution correctly (for instance Cucumber process was killed by CI server due to lack of memory)

data/README.md

@@ -83,7 +83,8 @@ We keep this old FAQ in README to not break old links spread across the web. You
     - [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests)
   - [Supported test runners in queue mode](#supported-test-runners-in-queue-mode)
 - [Split test files by test cases](#split-test-files-by-test-cases)
-  - [RSpec split test files by test examples (individual `it`)](#rspec-split-test-files-by-test-examples-individual-it)
+  - [RSpec split test files by test examples (by individual `it`s)](#rspec-split-test-files-by-test-examples-by-individual-its)
+  - [How to manually define a list of slow test files to be split by test cases](#how-to-manually-define-a-list-of-slow-test-files-to-be-split-by-test-cases)
 - [Extra configuration for CI server](#extra-configuration-for-ci-server)
   - [Info about ENV variables](#info-about-env-variables)
     - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT (test suite split based on seed)](#knapsack_pro_fixed_test_suite_split-test-suite-split-based-on-seed)
@@ -597,33 +598,44 @@ At this moment the queue mode works for:
 
 ## Split test files by test cases
 
-> __Note:__ this is an experimental feature. It works for Regular Mode and Queue Mode. For large test suite with a few thousand test files, it may generate too many RSpec test example paths that may lead to too large JSON payload in request to Knapsack Pro API and this could trigger the API timeout.
->
-> Please give us feedback so we could improve the feature.
-> https://knapsackpro.com/contact
+__How it works__: You can split slow test file by test cases. Thanks to that the slow test file can be split across parallel CI nodes because test cases from the test file will run on different CI nodes.
+
+This is helpful when you have one or a few very slow test files that are a bottleneck for CI build speed and you don't want to manually create a few smaller test files from the slow test files. Instead, you can tell `knapsack_pro` gem to split your slow test files by test cases across parallel CI nodes.
 
-__How it works__: You can split slow test file by test cases. Thanks to that the test file can be split across parallel CI nodes because test cases from the test file will run on different CI nodes.
+Knapsack Pro API provides recorded timing of test files from your previously recorded CI builds and `knapsack_pro` gem will use this suggestion to determine slow test files. `knapsack_pro` gem splits only slow test files by test cases. Test files that are fast won't be split by test cases because it is not needed.
 
-This is helpful when you have one or a few very slow test files that are a bottleneck for CI build speed and you don't want to manually create a few smaller test files from the slow test files. Instead, you can tell `knapsack_pro` gem to split your test files by test cases across parallel CI nodes.
+> __Note:__ This feature works for below test runners in Knapsack Pro Regular Mode and Queue Mode.
 
 ### RSpec split test files by test examples (by individual `it`s)
 
-> ❗ __WARNING (known issues):__ This is an experimental feature. We detected edge cases for Queue Mode when test examples might be ignored by RSpec and won't run test. Do not use this feature for production yet. We'll appreciate your [feedback](https://knapsackpro.com/contact) how it works while you test it on a separate branch. We work on improvements.
+> ❗ __RSpec requirement:__ You need `RSpec >= 3.3.0` in order to use this feature.
 
-In order to split RSpec test files by test examples across parallel CI nodes you need to set environment variable:
+In order to split RSpec slow test files by test examples across parallel CI nodes you need to set environment variable:
 
 ```
 KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true
 ```
 
-Thanks to that your CI build speed can be faster. We recommend using this feature with Queue Mode to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.
+Thanks to that your CI build speed can be faster. We recommend using this feature with [Queue Mode](https://youtu.be/hUEB1XDKEFY) to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.
 
-Doing tests split by test examples can generate a lot of logs by `knapsack_pro` gem in Queue Mode. We recommend to set log level to:
+### How to manually define a list of slow test files to be split by test cases
+
+If you don't want to rely on a list of test files from Knapsack Pro API to determine slow test files that should be split by test cases then you can define your own list of slow test files.
 
 ```
-KNAPSACK_PRO_LOG_LEVEL=warn
+# enable split by test cases for RSpec
+KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true
+
+# example slow test files pattern for RSpec
+KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN="{spec/models/user_spec.rb,spec/controllers/**/*_spec.rb}"
 ```
 
+`KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` must be subset of `KNAPSACK_PRO_TEST_FILE_PATTERN` (example default pattern for RSpec is `KNAPSACK_PRO_TEST_FILE_PATTERN="spec/**{,/*/**}/*_spec.rb"`).
+
+> __Warning:__ `KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` pattern is mostly useful for debugging purposes by developers of `knapsack_pro` gem. If you want to use it then __it is recommended to provide a shortlist of slow test files__ with the pattern.
+>
+> If you use a too broad list of slow test files then you may end up slowing your test suite, especially for RSpec it may result in a slow generating list of test examples in your project. The long list of test file example paths won't be accepted by Knapsack Pro API due to API timeout. CI providers like CircleCI may exceed server memory when running too many RSpec test examples.
+
 ## Extra configuration for CI server
 
 ### Info about ENV variables

data/lib/knapsack_pro.rb

@@ -55,6 +55,12 @@ require_relative 'knapsack_pro/adapters/test_unit_adapter'
 require_relative 'knapsack_pro/adapters/spinach_adapter'
 require_relative 'knapsack_pro/allocator'
 require_relative 'knapsack_pro/queue_allocator'
+require_relative 'knapsack_pro/test_case_mergers/base_merger'
+require_relative 'knapsack_pro/test_case_mergers/rspec_merger'
+require_relative 'knapsack_pro/build_distribution_fetcher'
+require_relative 'knapsack_pro/slow_test_file_determiner'
+require_relative 'knapsack_pro/slow_test_file_finder'
+require_relative 'knapsack_pro/test_files_with_test_cases_composer'
 require_relative 'knapsack_pro/base_allocator_builder'
 require_relative 'knapsack_pro/allocator_builder'
 require_relative 'knapsack_pro/queue_allocator_builder'

data/lib/knapsack_pro/adapters/base_adapter.rb

@@ -4,6 +4,22 @@ module KnapsackPro
       # Just example, please overwrite constant in subclass
       TEST_DIR_PATTERN = 'test/**{,/*/**}/*_test.rb'
 
+      def self.slow_test_file?(adapter_class, test_file_path)
+        @slow_test_file_paths ||=
+          begin
+            slow_test_files =
+              if KnapsackPro::Config::Env.slow_test_file_pattern
+                KnapsackPro::TestFileFinder.slow_test_files_by_pattern(adapter_class)
+              else
+                # get slow test files from JSON file based on data from API
+                KnapsackPro::SlowTestFileDeterminer.read_from_json_report
+              end
+            KnapsackPro::TestFilePresenter.paths(slow_test_files)
+          end
+        clean_path = KnapsackPro::TestFileCleaner.clean(test_file_path)
+        @slow_test_file_paths.include?(clean_path)
+      end
+
       def self.bind
         adapter = new
         adapter.bind

data/lib/knapsack_pro/adapters/rspec_adapter.rb

@@ -22,18 +22,20 @@ module KnapsackPro
       def bind_time_tracker
         ::RSpec.configure do |config|
           config.around(:each) do |example|
+            current_example_group =
+              if ::RSpec.respond_to?(:current_example)
+                ::RSpec.current_example.metadata[:example_group]
+              else
+                example.metadata
+              end
+
+            current_test_path = KnapsackPro::Adapters::RSpecAdapter.test_path(current_example_group)
+
             KnapsackPro.tracker.current_test_path =
-              if KnapsackPro::Config::Env.rspec_split_by_test_examples?
+              if KnapsackPro::Config::Env.rspec_split_by_test_examples? && KnapsackPro::Adapters::RSpecAdapter.slow_test_file?(RSpecAdapter, current_test_path)
                 example.id
               else
-                current_example_group =
-                  if ::RSpec.respond_to?(:current_example)
-                    ::RSpec.current_example.metadata[:example_group]
-                  else
-                    example.metadata
-                  end
-
-                KnapsackPro::Adapters::RSpecAdapter.test_path(current_example_group)
+                current_test_path
               end
 
             KnapsackPro.tracker.start_timer

data/lib/knapsack_pro/allocator.rb

@@ -1,7 +1,8 @@
 module KnapsackPro
   class Allocator
     def initialize(args)
-      @test_files = args.fetch(:test_files)
+      @fast_and_slow_test_files_to_run = args.fetch(:fast_and_slow_test_files_to_run)
+      @fallback_mode_test_files = args.fetch(:fallback_mode_test_files)
       @ci_node_total = args.fetch(:ci_node_total)
       @ci_node_index = args.fetch(:ci_node_index)
       @repository_adapter = args.fetch(:repository_adapter)
@@ -32,13 +33,14 @@ module KnapsackPro
 
     private
 
-    attr_reader :test_files,
+    attr_reader :fast_and_slow_test_files_to_run,
+      :fallback_mode_test_files,
       :ci_node_total,
       :ci_node_index,
       :repository_adapter
 
     def encrypted_test_files
-      KnapsackPro::Crypto::Encryptor.call(test_files)
+      KnapsackPro::Crypto::Encryptor.call(fast_and_slow_test_files_to_run)
     end
 
     def encrypted_branch
@@ -56,12 +58,12 @@ module KnapsackPro
     end
 
     def prepare_test_files(response)
-      decrypted_test_files = KnapsackPro::Crypto::Decryptor.call(test_files, response['test_files'])
+      decrypted_test_files = KnapsackPro::Crypto::Decryptor.call(fast_and_slow_test_files_to_run, response['test_files'])
       KnapsackPro::TestFilePresenter.paths(decrypted_test_files)
     end
 
     def fallback_test_files
-      test_flat_distributor = KnapsackPro::TestFlatDistributor.new(test_files, ci_node_total)
+      test_flat_distributor = KnapsackPro::TestFlatDistributor.new(fallback_mode_test_files, ci_node_total)
       test_files_for_node_index = test_flat_distributor.test_files_for_node(ci_node_index)
       KnapsackPro::TestFilePresenter.paths(test_files_for_node_index)
     end

data/lib/knapsack_pro/allocator_builder.rb

@@ -2,7 +2,8 @@ module KnapsackPro
   class AllocatorBuilder < BaseAllocatorBuilder
     def allocator
       KnapsackPro::Allocator.new(
-        test_files: test_files,
+        fast_and_slow_test_files_to_run: fast_and_slow_test_files_to_run,
+        fallback_mode_test_files: fallback_mode_test_files,
         ci_node_total: env.ci_node_total,
         ci_node_index: env.ci_node_index,
         repository_adapter: repository_adapter,

data/lib/knapsack_pro/base_allocator_builder.rb

@@ -21,29 +21,40 @@ module KnapsackPro
       KnapsackPro::Config::Env.test_dir || TestFilePattern.test_dir(adapter_class)
     end
 
-    def test_files
-      found_test_files = KnapsackPro::TestFileFinder.call(test_file_pattern)
+    # in fallback mode we always want to run the whole test files
+    # (not split by test cases) to guarantee that each test will be executed
+    # at least once across parallel CI nodes
+    def fallback_mode_test_files
+      all_test_files_to_run
+    end
+
+    # detect test files present on the disk that should be run
+    # this may include some fast test files + slow test files split by test cases
+    def fast_and_slow_test_files_to_run
+      test_files_to_run = all_test_files_to_run
 
       if adapter_class == KnapsackPro::Adapters::RSpecAdapter && KnapsackPro::Config::Env.rspec_split_by_test_examples?
-        test_files_count = found_test_files.size
+        unless Gem::Version.new(RSpec::Core::Version::STRING) >= Gem::Version.new('3.3.0')
+          raise 'RSpec >= 3.3.0 is required to split test files by test examples. Learn more: https://github.com/KnapsackPro/knapsack_pro-ruby#split-test-files-by-test-cases'
+        end
 
-        KnapsackPro.logger.warn("Generating RSpec test examples JSON report to prepare your test suite to be split by test examples (by individual 'it's. Thanks to that a single test file can be split across parallel CI nodes). Analyzing #{test_files_count} test files.")
+        slow_test_files = get_slow_test_files
 
-        if test_files_count > 1000
-          KnapsackPro.logger.warn("You have more than 1000 test files, it may take longer to generate test examples. Please wait...")
-        end
+        KnapsackPro.logger.info("Generating RSpec test examples JSON report for slow test files to prepare it to be split by test examples (by individual 'it's. Thanks to that a single slow test file can be split across parallel CI nodes). Analyzing #{slow_test_files.size} slow test files.")
 
         # generate RSpec JSON report in separate process to not pollute RSpec state
-        cmd = 'bundle exec rake knapsack_pro:rspec_test_example_detector'
+        cmd = 'RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector'
         unless Kernel.system(cmd)
           raise "Could not generate JSON report for RSpec. Rake task failed when running #{cmd}"
         end
 
         # read JSON report
         detector = KnapsackPro::TestCaseDetectors::RSpecTestExampleDetector.new
-        detector.test_file_example_paths
+        test_file_example_paths = detector.test_file_example_paths
+
+        KnapsackPro::TestFilesWithTestCasesComposer.call(test_files_to_run, slow_test_files, test_file_example_paths)
       else
-        found_test_files
+        test_files_to_run
       end
     end
 
@@ -62,5 +73,25 @@ module KnapsackPro
     def test_file_pattern
       TestFilePattern.call(adapter_class)
     end
+
+    def all_test_files_to_run
+      KnapsackPro::TestFileFinder.call(test_file_pattern)
+    end
+
+    def slow_test_file_pattern
+      KnapsackPro::Config::Env.slow_test_file_pattern
+    end
+
+    def get_slow_test_files
+      slow_test_files =
+        if slow_test_file_pattern
+          KnapsackPro::TestFileFinder.slow_test_files_by_pattern(adapter_class)
+        else
+          # get slow test files from API and ensure they exist on disk
+          KnapsackPro::SlowTestFileFinder.call(adapter_class)
+        end
+      KnapsackPro.logger.debug("Detected #{slow_test_files.size} slow test files: #{slow_test_files.inspect}")
+      slow_test_files
+    end
   end
 end

data/lib/knapsack_pro/build_distribution_fetcher.rb

@@ -0,0 +1,57 @@
+module KnapsackPro
+  class BuildDistributionFetcher
+    class BuildDistributionEntity
+      def initialize(response)
+        @response = response
+      end
+
+      def time_execution
+        response.fetch('time_execution')
+      end
+
+      def test_files
+        response.fetch('test_files')
+      end
+
+      private
+
+      attr_reader :response
+    end
+
+    def self.call
+      new.call
+    end
+
+    # get test files and time execution for last build distribution matching:
+    # branch, node_total, node_index
+    def call
+      connection = KnapsackPro::Client::Connection.new(build_action)
+      response = connection.call
+      if connection.success?
+        raise ArgumentError.new(response) if connection.errors?
+        BuildDistributionEntity.new(response)
+      else
+        KnapsackPro.logger.warn("Slow test files fallback behaviour started. We could not connect with Knapsack Pro API to fetch last CI build test files that are needed to determine slow test files. No test files will be split by test cases. It means all test files will be split by the whole test files as if split by test cases would be disabled https://github.com/KnapsackPro/knapsack_pro-ruby#split-test-files-by-test-cases")
+        BuildDistributionEntity.new({
+          'time_execution' => 0.0,
+          'test_files' => [],
+        })
+      end
+    end
+
+    private
+
+    def repository_adapter
+      @repository_adapter ||= KnapsackPro::RepositoryAdapterInitiator.call
+    end
+
+    def build_action
+      KnapsackPro::Client::API::V1::BuildDistributions.last(
+        commit_hash: repository_adapter.commit_hash,
+        branch: repository_adapter.branch,
+        node_total: KnapsackPro::Config::Env.ci_node_total,
+        node_index: KnapsackPro::Config::Env.ci_node_index,
+      )
+    end
+  end
+end

data/lib/knapsack_pro/client/api/v1/build_distributions.rb

@@ -18,6 +18,19 @@ module KnapsackPro
                 }
               )
             end
+
+            def last(args)
+              action_class.new(
+                endpoint_path: '/v1/build_distributions/last',
+                http_method: :get,
+                request_hash: {
+                  :commit_hash => args.fetch(:commit_hash),
+                  :branch => args.fetch(:branch),
+                  :node_total => args.fetch(:node_total),
+                  :node_index => args.fetch(:node_index),
+                }
+              )
+            end
           end
         end
       end

data/lib/knapsack_pro/client/connection.rb

@@ -48,11 +48,7 @@ module KnapsackPro
       end
 
       def request_hash
-        action
-        .request_hash
-        .merge({
-          test_suite_token: test_suite_token
-        })
+        action.request_hash
       end
 
       def request_body
@@ -69,6 +65,7 @@ module KnapsackPro
           'Accept' => 'application/json',
           'KNAPSACK-PRO-CLIENT-NAME' => client_name,
           'KNAPSACK-PRO-CLIENT-VERSION' => KnapsackPro::VERSION,
+          'KNAPSACK-PRO-TEST-SUITE-TOKEN' => test_suite_token,
         }
       end
 
@@ -95,19 +92,15 @@ module KnapsackPro
         !seed.nil?
       end
 
-      def post
+      def make_request(&block)
         retries ||= 0
-        uri = URI.parse(endpoint_url)
-        http = Net::HTTP.new(uri.host, uri.port)
-        http.use_ssl = (uri.scheme == 'https')
-        http.open_timeout = TIMEOUT
-        http.read_timeout = TIMEOUT
 
-        @http_response = http.post(uri.path, request_body, json_headers)
+        @http_response = block.call
         @response_body = parse_response_body(http_response.body)
 
         request_uuid = http_response.header['X-Request-Id'] || 'N/A'
 
+        logger.debug("#{action.http_method.to_s.upcase} #{endpoint_url}")
         logger.debug("API request UUID: #{request_uuid}")
         logger.debug("Test suite split seed: #{seed}") if has_seed?
         logger.debug('API response:')
@@ -134,6 +127,31 @@ module KnapsackPro
           response_body
         end
       end
+
+      def build_http(uri)
+        http = Net::HTTP.new(uri.host, uri.port)
+        http.use_ssl = (uri.scheme == 'https')
+        http.open_timeout = TIMEOUT
+        http.read_timeout = TIMEOUT
+        http
+      end
+
+      def post
+        uri = URI.parse(endpoint_url)
+        http = build_http(uri)
+        make_request do
+          http.post(uri.path, request_body, json_headers)
+        end
+      end
+
+      def get
+        uri = URI.parse(endpoint_url)
+        uri.query = URI.encode_www_form(request_hash)
+        http = build_http(uri)
+        make_request do
+          http.get(uri, json_headers)
+        end
+      end
     end
   end
 end

data/lib/knapsack_pro/config/env.rb

@@ -55,6 +55,10 @@ module KnapsackPro
           ENV['KNAPSACK_PRO_TEST_FILE_PATTERN']
         end
 
+        def slow_test_file_pattern
+          ENV['KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN']
+        end
+
         def test_file_exclude_pattern
           ENV['KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN']
         end