knapsack_pro 1.20.0 → 1.22.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f06c380d215524276faa79157268a701ca58e3e0ab22fdcfb78119786596bf65
-  data.tar.gz: e5c3d4a9258fd43dd869367bd7d1fdaacd786904f67cfe4041f2fc7aa49daf25
+  metadata.gz: 1ff5e0f88a008abb13c1ca010f4b430be290ecf4e349719cac39611eebc575a4
+  data.tar.gz: 83650913e5d6cd21201ebce5ae48c731d61a0bc27bb37f566fcdc3d96a9b0598
 SHA512:
-  metadata.gz: ad9e73fca6cb2b53d8c9d156a012ea43c159cb37380c8c6cbe95f2ee0d94abca8707ea94eff3c59d428e48afdd9f3500f28aff8f5d85d12f7fe22e7f726cf9ca
-  data.tar.gz: 2d1b257cade9a1c82e93c530a9d30cb292c05a3df007ed5e18e809472520d035590ec06b6457790ce4f37190c28beb60801c21a6a646e2ac3674abaf3c299ea1
+  metadata.gz: 3b3d2dc24c6592040f559f77f59168502bb949b135406a06f59c836494902412e202a28bc781760e0862f416ccaa2d2c9a246269fee87e9fd3d9120ba803f4ab
+  data.tar.gz: 9d4005922f963d765b494ab2b35642fa3be85787a6deb8b490ed3e1f64e5055d474d4cdae5b1bba4d4668b4c63fd59d9063102baf7036830e53d409ba8cf016a

data/CHANGELOG.md

@@ -1,5 +1,50 @@
 # Change Log
 
+### 1.22.1
+
+* Fix for an auto split of slow RSpec test files by test examples when using `KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true` and `parallel_tests` gem. Save the JSON reports with unique file names with the CI node index in the name to avoid accidentally overriding the files on the same disk.
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/113
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.22.0...v1.22.1
+
+### 1.22.0
+
+* Increase request retry timebox from 4s to 8s to not flood Knapsack Pro API with too many requests in a short period of time and to give time for API server to autoscale and add additional machines to serve traffic
+* When Fallback Mode is disabled with env `KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false` then retry the request to Knapsack Pro API for 6 times instead of only 3 times.
+
+  Here is related [info why some users want to disable Fallback Mode](https://github.com/KnapsackPro/knapsack_pro-ruby#required-ci-configuration-if-you-use-retry-single-failed-ci-node-feature-on-your-ci-server-when-knapsack_pro_fixed_queue_splittrue-in-queue-mode-or-knapsack_pro_fixed_test_suite_splittrue-in-regular-mode).
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/112
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.21.0...v1.22.0
+
+### 1.21.0
+
+* Automatically detect slow test files for RSpec and split them by test examples when `KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true`
+* Add slow test file pattern `KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` to define RSpec slow test files that should be split by test examples
+* Start sending API token in header `KNAPSACK-PRO-TEST-SUITE-TOKEN` instead of a key `test_suite_token` in JSON payload.
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/106
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.20.2...v1.21.0
+
+### 1.20.2
+
+* Raise an error when running Cucumber in Queue Mode and Cucumber system process doesn't finish execution correctly (for instance Cucumber process was killed by CI server due to lack of memory)
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/111
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.20.1...v1.20.2
+
+### 1.20.1
+
+* Fix bug in RSpec split by test examples in < RSpec 3.6.0 (related to custom json formatter)
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/105
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v1.20.0...v1.20.1
+
 ### 1.20.0
 
 * Add support for tests split by test examples to RSpec older than 3.6.0

data/README.md

@@ -83,7 +83,8 @@ We keep this old FAQ in README to not break old links spread across the web. You
     - [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests)
   - [Supported test runners in queue mode](#supported-test-runners-in-queue-mode)
 - [Split test files by test cases](#split-test-files-by-test-cases)
-  - [RSpec split test files by test examples (individual `it`)](#rspec-split-test-files-by-test-examples-individual-it)
+  - [RSpec split test files by test examples (by individual `it`s)](#rspec-split-test-files-by-test-examples-by-individual-its)
+  - [How to manually define a list of slow test files to be split by test cases](#how-to-manually-define-a-list-of-slow-test-files-to-be-split-by-test-cases)
 - [Extra configuration for CI server](#extra-configuration-for-ci-server)
   - [Info about ENV variables](#info-about-env-variables)
     - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT (test suite split based on seed)](#knapsack_pro_fixed_test_suite_split-test-suite-split-based-on-seed)
@@ -597,33 +598,44 @@ At this moment the queue mode works for:
 
 ## Split test files by test cases
 
-__Note:__ this is an experimental feature. It works for Regular Mode and Queue Mode. For large test suite with a few thousand test files, it may generate too many RSpec test example paths that may lead to too large JSON payload in request to Knapsack Pro API and this could trigger the API timeout.
+__How it works__: You can split slow test file by test cases. Thanks to that the slow test file can be split across parallel CI nodes because test cases from the test file will run on different CI nodes.
 
-Please give us feedback so we could improve the feature.
-https://knapsackpro.com/contact
+This is helpful when you have one or a few very slow test files that are a bottleneck for CI build speed and you don't want to manually create a few smaller test files from the slow test files. Instead, you can tell `knapsack_pro` gem to split your slow test files by test cases across parallel CI nodes.
 
-__How it works__: You can split slow test file by test cases. Thanks to that the test file can be split across parallel CI nodes because test cases from the test file will run on different CI nodes.
+Knapsack Pro API provides recorded timing of test files from your previously recorded CI builds and `knapsack_pro` gem will use this suggestion to determine slow test files. `knapsack_pro` gem splits only slow test files by test cases. Test files that are fast won't be split by test cases because it is not needed.
 
-This is helpful when you have one or a few very slow test files that are a bottleneck for CI build speed and you don't want to manually create a few smaller test files from the slow test files. Instead, you can tell `knapsack_pro` gem to split your test files by test cases across parallel CI nodes.
+> __Note:__ This feature works for below test runners in Knapsack Pro Regular Mode and Queue Mode.
 
 ### RSpec split test files by test examples (by individual `it`s)
 
-__Requirement:__ It works with >= RSpec 3.6.0.
+> ❗ __RSpec requirement:__ You need `RSpec >= 3.3.0` in order to use this feature.
 
-In order to split RSpec test files by test examples across parallel CI nodes you need to set environment variable:
+In order to split RSpec slow test files by test examples across parallel CI nodes you need to set environment variable:
 
 ```
 KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true
 ```
 
-Thanks to that your CI build speed can be faster. We recommend using this feature with Queue Mode to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.
+Thanks to that your CI build speed can be faster. We recommend using this feature with [Queue Mode](https://youtu.be/hUEB1XDKEFY) to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.
 
-Doing tests split by test examples can generate a lot of logs by `knapsack_pro` gem in Queue Mode. We recommend to set log level to:
+### How to manually define a list of slow test files to be split by test cases
+
+If you don't want to rely on a list of test files from Knapsack Pro API to determine slow test files that should be split by test cases then you can define your own list of slow test files.
 
 ```
-KNAPSACK_PRO_LOG_LEVEL=warn
+# enable split by test cases for RSpec
+KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true
+
+# example slow test files pattern for RSpec
+KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN="{spec/models/user_spec.rb,spec/controllers/**/*_spec.rb}"
 ```
 
+`KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` must be subset of `KNAPSACK_PRO_TEST_FILE_PATTERN` (example default pattern for RSpec is `KNAPSACK_PRO_TEST_FILE_PATTERN="spec/**{,/*/**}/*_spec.rb"`).
+
+> __Warning:__ `KNAPSACK_PRO_SLOW_TEST_FILE_PATTERN` pattern is mostly useful for debugging purposes by developers of `knapsack_pro` gem. If you want to use it then __it is recommended to provide a shortlist of slow test files__ with the pattern.
+>
+> If you use a too broad list of slow test files then you may end up slowing your test suite, especially for RSpec it may result in a slow generating list of test examples in your project. The long list of test file example paths won't be accepted by Knapsack Pro API due to API timeout. CI providers like CircleCI may exceed server memory when running too many RSpec test examples.
+
 ## Extra configuration for CI server
 
 ### Info about ENV variables
@@ -699,7 +711,7 @@ Read below required configuration step if you use Queue Mode and you set [`KNAPS
 
   If you use other CI server you need to manually configure your CI server to set `KNAPSACK_PRO_CI_NODE_RETRY_COUNT=1` only during retry CI node attempt. If `KNAPSACK_PRO_CI_NODE_RETRY_COUNT > 0` then knapsack_pro won't allow starting running tests in Fallback Mode and instead will raise error so a user can manually retry CI node later when a connection to Knapsack Pro API can be established.
 
-  If you cannot set `KNAPSACK_PRO_CI_NODE_RETRY_COUNT` only for retried CI node or it is not possible for your CI server then you can disable Fallback Mode completely `KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false`.
+  If you cannot set `KNAPSACK_PRO_CI_NODE_RETRY_COUNT` only for retried CI node or it is not possible for your CI server then you can disable Fallback Mode completely `KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false`. When Fallback Mode is disabled then knapsack_pro gem will try to connect to Knapsack Pro API 6 times instead of only 3 times to ensure there is a low chance of failing your CI node due to lost connection with the API.
 
 ### Passing arguments to rake task
 
@@ -979,19 +991,28 @@ You can parallelize your builds across virtual machines with [travis matrix feat
 
 ```yaml
 script:
-  # Step for RSpec
+  # Step for RSpec in Regular Mode
   - "bundle exec rake knapsack_pro:rspec"
 
-  # Step for Cucumber
+  # Step for RSpec in Queue Mode
+  - "bundle exec rake knapsack_pro:queue:rspec"
+
+  # Step for Cucumber in Regular Mode
   - "bundle exec rake knapsack_pro:cucumber"
 
-  # Step for Minitest
+  # Step for Cucumber in Queue Mode
+  - "bundle exec rake knapsack_pro:queue:cucumber"
+
+  # Step for Minitest in Regular Mode
   - "bundle exec rake knapsack_pro:minitest"
 
-  # Step for test-unit
+  # Step for Minitest in Queue Mode
+  - "bundle exec rake knapsack_pro:queue:minitest"
+
+  # Step for test-unit in Regular Mode
   - "bundle exec rake knapsack_pro:test_unit"
 
-  # Step for Spinach
+  # Step for Spinach in Regular Mode
   - "bundle exec rake knapsack_pro:spinach"
 
 env:
@@ -1003,6 +1024,10 @@ env:
     - KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token
     - KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token
 
+    # if you use Knapsack Pro Queue Mode you must set below env variable
+    # to be able to retry single failed parallel job from Travis UI
+    - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
+
     - KNAPSACK_PRO_CI_NODE_TOTAL=2
   jobs:
     - KNAPSACK_PRO_CI_NODE_INDEX=0

data/lib/knapsack_pro.rb

@@ -55,6 +55,12 @@ require_relative 'knapsack_pro/adapters/test_unit_adapter'
 require_relative 'knapsack_pro/adapters/spinach_adapter'
 require_relative 'knapsack_pro/allocator'
 require_relative 'knapsack_pro/queue_allocator'
+require_relative 'knapsack_pro/test_case_mergers/base_merger'
+require_relative 'knapsack_pro/test_case_mergers/rspec_merger'
+require_relative 'knapsack_pro/build_distribution_fetcher'
+require_relative 'knapsack_pro/slow_test_file_determiner'
+require_relative 'knapsack_pro/slow_test_file_finder'
+require_relative 'knapsack_pro/test_files_with_test_cases_composer'
 require_relative 'knapsack_pro/base_allocator_builder'
 require_relative 'knapsack_pro/allocator_builder'
 require_relative 'knapsack_pro/queue_allocator_builder'

data/lib/knapsack_pro/adapters/base_adapter.rb

@@ -4,6 +4,22 @@ module KnapsackPro
       # Just example, please overwrite constant in subclass
       TEST_DIR_PATTERN = 'test/**{,/*/**}/*_test.rb'
 
+      def self.slow_test_file?(adapter_class, test_file_path)
+        @slow_test_file_paths ||=
+          begin
+            slow_test_files =
+              if KnapsackPro::Config::Env.slow_test_file_pattern
+                KnapsackPro::TestFileFinder.slow_test_files_by_pattern(adapter_class)
+              else
+                # get slow test files from JSON file based on data from API
+                KnapsackPro::SlowTestFileDeterminer.read_from_json_report
+              end
+            KnapsackPro::TestFilePresenter.paths(slow_test_files)
+          end
+        clean_path = KnapsackPro::TestFileCleaner.clean(test_file_path)
+        @slow_test_file_paths.include?(clean_path)
+      end
+
       def self.bind
         adapter = new
         adapter.bind

data/lib/knapsack_pro/adapters/rspec_adapter.rb

@@ -22,18 +22,20 @@ module KnapsackPro
       def bind_time_tracker
         ::RSpec.configure do |config|
           config.around(:each) do |example|
+            current_example_group =
+              if ::RSpec.respond_to?(:current_example)
+                ::RSpec.current_example.metadata[:example_group]
+              else
+                example.metadata
+              end
+
+            current_test_path = KnapsackPro::Adapters::RSpecAdapter.test_path(current_example_group)
+
             KnapsackPro.tracker.current_test_path =
-              if KnapsackPro::Config::Env.rspec_split_by_test_examples?
+              if KnapsackPro::Config::Env.rspec_split_by_test_examples? && KnapsackPro::Adapters::RSpecAdapter.slow_test_file?(RSpecAdapter, current_test_path)
                 example.id
               else
-                current_example_group =
-                  if ::RSpec.respond_to?(:current_example)
-                    ::RSpec.current_example.metadata[:example_group]
-                  else
-                    example.metadata
-                  end
-
-                KnapsackPro::Adapters::RSpecAdapter.test_path(current_example_group)
+                current_test_path
               end
 
             KnapsackPro.tracker.start_timer

data/lib/knapsack_pro/allocator.rb

@@ -1,7 +1,8 @@
 module KnapsackPro
   class Allocator
     def initialize(args)
-      @test_files = args.fetch(:test_files)
+      @fast_and_slow_test_files_to_run = args.fetch(:fast_and_slow_test_files_to_run)
+      @fallback_mode_test_files = args.fetch(:fallback_mode_test_files)
       @ci_node_total = args.fetch(:ci_node_total)
       @ci_node_index = args.fetch(:ci_node_index)
       @repository_adapter = args.fetch(:repository_adapter)
@@ -32,13 +33,14 @@ module KnapsackPro
 
     private
 
-    attr_reader :test_files,
+    attr_reader :fast_and_slow_test_files_to_run,
+      :fallback_mode_test_files,
       :ci_node_total,
       :ci_node_index,
       :repository_adapter
 
     def encrypted_test_files
-      KnapsackPro::Crypto::Encryptor.call(test_files)
+      KnapsackPro::Crypto::Encryptor.call(fast_and_slow_test_files_to_run)
     end
 
     def encrypted_branch
@@ -56,12 +58,12 @@ module KnapsackPro
     end
 
     def prepare_test_files(response)
-      decrypted_test_files = KnapsackPro::Crypto::Decryptor.call(test_files, response['test_files'])
+      decrypted_test_files = KnapsackPro::Crypto::Decryptor.call(fast_and_slow_test_files_to_run, response['test_files'])
       KnapsackPro::TestFilePresenter.paths(decrypted_test_files)
     end
 
     def fallback_test_files
-      test_flat_distributor = KnapsackPro::TestFlatDistributor.new(test_files, ci_node_total)
+      test_flat_distributor = KnapsackPro::TestFlatDistributor.new(fallback_mode_test_files, ci_node_total)
       test_files_for_node_index = test_flat_distributor.test_files_for_node(ci_node_index)
       KnapsackPro::TestFilePresenter.paths(test_files_for_node_index)
     end

data/lib/knapsack_pro/allocator_builder.rb

@@ -2,7 +2,8 @@ module KnapsackPro
   class AllocatorBuilder < BaseAllocatorBuilder
     def allocator
       KnapsackPro::Allocator.new(
-        test_files: test_files,
+        fast_and_slow_test_files_to_run: fast_and_slow_test_files_to_run,
+        fallback_mode_test_files: fallback_mode_test_files,
         ci_node_total: env.ci_node_total,
         ci_node_index: env.ci_node_index,
         repository_adapter: repository_adapter,

data/lib/knapsack_pro/base_allocator_builder.rb

@@ -21,29 +21,40 @@ module KnapsackPro
       KnapsackPro::Config::Env.test_dir || TestFilePattern.test_dir(adapter_class)
     end
 
-    def test_files
-      found_test_files = KnapsackPro::TestFileFinder.call(test_file_pattern)
+    # in fallback mode we always want to run the whole test files
+    # (not split by test cases) to guarantee that each test will be executed
+    # at least once across parallel CI nodes
+    def fallback_mode_test_files
+      all_test_files_to_run
+    end
+
+    # detect test files present on the disk that should be run
+    # this may include some fast test files + slow test files split by test cases
+    def fast_and_slow_test_files_to_run
+      test_files_to_run = all_test_files_to_run
 
       if adapter_class == KnapsackPro::Adapters::RSpecAdapter && KnapsackPro::Config::Env.rspec_split_by_test_examples?
-        test_files_count = found_test_files.size
+        unless Gem::Version.new(RSpec::Core::Version::STRING) >= Gem::Version.new('3.3.0')
+          raise 'RSpec >= 3.3.0 is required to split test files by test examples. Learn more: https://github.com/KnapsackPro/knapsack_pro-ruby#split-test-files-by-test-cases'
+        end
 
-        KnapsackPro.logger.warn("Generating RSpec test examples JSON report to prepare your test suite to be split by test examples (by individual 'it's. Thanks to that a single test file can be split across parallel CI nodes). Analyzing #{test_files_count} test files.")
+        slow_test_files = get_slow_test_files
 
-        if test_files_count > 1000
-          KnapsackPro.logger.warn("You have more than 1000 test files, it may take longer to generate test examples. Please wait...")
-        end
+        KnapsackPro.logger.info("Generating RSpec test examples JSON report for slow test files to prepare it to be split by test examples (by individual 'it's. Thanks to that a single slow test file can be split across parallel CI nodes). Analyzing #{slow_test_files.size} slow test files.")
 
         # generate RSpec JSON report in separate process to not pollute RSpec state
-        cmd = 'bundle exec rake knapsack_pro:rspec_test_example_detector'
+        cmd = 'RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector'
         unless Kernel.system(cmd)
           raise "Could not generate JSON report for RSpec. Rake task failed when running #{cmd}"
         end
 
         # read JSON report
         detector = KnapsackPro::TestCaseDetectors::RSpecTestExampleDetector.new
-        detector.test_file_example_paths
+        test_file_example_paths = detector.test_file_example_paths
+
+        KnapsackPro::TestFilesWithTestCasesComposer.call(test_files_to_run, slow_test_files, test_file_example_paths)
       else
-        found_test_files
+        test_files_to_run
       end
     end
 
@@ -62,5 +73,25 @@ module KnapsackPro
     def test_file_pattern
       TestFilePattern.call(adapter_class)
     end
+
+    def all_test_files_to_run
+      KnapsackPro::TestFileFinder.call(test_file_pattern)
+    end
+
+    def slow_test_file_pattern
+      KnapsackPro::Config::Env.slow_test_file_pattern
+    end
+
+    def get_slow_test_files
+      slow_test_files =
+        if slow_test_file_pattern
+          KnapsackPro::TestFileFinder.slow_test_files_by_pattern(adapter_class)
+        else
+          # get slow test files from API and ensure they exist on disk
+          KnapsackPro::SlowTestFileFinder.call(adapter_class)
+        end
+      KnapsackPro.logger.debug("Detected #{slow_test_files.size} slow test files: #{slow_test_files.inspect}")
+      slow_test_files
+    end
   end
 end

data/lib/knapsack_pro/build_distribution_fetcher.rb

@@ -0,0 +1,57 @@
+module KnapsackPro
+  class BuildDistributionFetcher
+    class BuildDistributionEntity
+      def initialize(response)
+        @response = response
+      end
+
+      def time_execution
+        response.fetch('time_execution')
+      end
+
+      def test_files
+        response.fetch('test_files')
+      end
+
+      private
+
+      attr_reader :response
+    end
+
+    def self.call
+      new.call
+    end
+
+    # get test files and time execution for last build distribution matching:
+    # branch, node_total, node_index
+    def call
+      connection = KnapsackPro::Client::Connection.new(build_action)
+      response = connection.call
+      if connection.success?
+        raise ArgumentError.new(response) if connection.errors?
+        BuildDistributionEntity.new(response)
+      else
+        KnapsackPro.logger.warn("Slow test files fallback behaviour started. We could not connect with Knapsack Pro API to fetch last CI build test files that are needed to determine slow test files. No test files will be split by test cases. It means all test files will be split by the whole test files as if split by test cases would be disabled https://github.com/KnapsackPro/knapsack_pro-ruby#split-test-files-by-test-cases")
+        BuildDistributionEntity.new({
+          'time_execution' => 0.0,
+          'test_files' => [],
+        })
+      end
+    end
+
+    private
+
+    def repository_adapter
+      @repository_adapter ||= KnapsackPro::RepositoryAdapterInitiator.call
+    end
+
+    def build_action
+      KnapsackPro::Client::API::V1::BuildDistributions.last(
+        commit_hash: repository_adapter.commit_hash,
+        branch: repository_adapter.branch,
+        node_total: KnapsackPro::Config::Env.ci_node_total,
+        node_index: KnapsackPro::Config::Env.ci_node_index,
+      )
+    end
+  end
+end

data/lib/knapsack_pro/client/api/v1/build_distributions.rb

@@ -18,6 +18,19 @@ module KnapsackPro
                 }
               )
             end
+
+            def last(args)
+              action_class.new(
+                endpoint_path: '/v1/build_distributions/last',
+                http_method: :get,
+                request_hash: {
+                  :commit_hash => args.fetch(:commit_hash),
+                  :branch => args.fetch(:branch),
+                  :node_total => args.fetch(:node_total),
+                  :node_index => args.fetch(:node_index),
+                }
+              )
+            end
           end
         end
       end