knapsack_pro 3.8.0 → 7.0.0

data/CHANGELOG.md

@@ -1,4 +1,328 @@
-# Change Log
+# Changelog
+
+### 7.0.0
+
+* __(breaking change)__ RSpec in Queue Mode:
+  * The default for `KNAPSACK_PRO_LOG_LEVEL` is `info` instead of `debug`.
+  * The RSpec `before(:suite)` and `after(:suite)` hooks changed:
+
+    __Before:__<br>
+    The `before(:suite)` and `after(:suite)` hooks were executed multiple times. Each time for a batch of tests fetched from Knapsack Pro Queue API.
+
+    __After:__<br>
+    The `before(:suite)` and `after(:suite)` hooks are executed only once: `before(:suite)` is executed before starting tests, `after(:suite)` is executed after all tests are completed. (It is what you would expect from RSpec).
+
+  * The `KnapsackPro::Hooks::Queue.after_queue` hook change:
+
+    __Before:__<br>
+    The `KnapsackPro::Hooks::Queue.after_queue` hook is executed outside of the `after(:suite)` hook.
+
+    __After:__<br>
+    The `KnapsackPro::Hooks::Queue.after_queue` hook is executed __inside__ of the `after(:suite)` hook.
+
+* Recommended RSpec changes in your project:
+  * Remove the following code if you use Queue Mode and the `rspec_junit_formatter` gem to generate JUnit XML or JSON reports:
+
+    ```ruby
+    # REMOVE THE FOLLOWING CODE
+
+    # spec_helper.rb or rails_helper.rb
+    TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
+    FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
+
+    KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
+      if File.exist?(TMP_REPORT)
+        FileUtils.mv(TMP_REPORT, FINAL_REPORT)
+      end
+    end
+    ```
+
+    Learn more about [using Knapsack Pro with RSpec formatters](https://docs.knapsackpro.com/ruby/rspec/#formatters-rspec_junit_formatter-json) and [using Knapsack Pro with CircleCI](https://docs.knapsackpro.com/ruby/circleci/) in the docs.
+
+  * Replace the following code if you are using Queue Mode and the `percy-capybara` gem on a version older than 4:
+
+    Before:
+
+    ```ruby
+    KnapsackPro::Hooks::Queue.before_queue { |queue_id| Percy::Capybara.initialize_build }
+    KnapsackPro::Hooks::Queue.after_queue { |queue_id| Percy::Capybara.finalize_build }
+    ```
+
+    After:
+
+    ```ruby
+    # recommended
+    before(:suite) { Percy::Capybara.initialize_build }
+    after(:suite) { Percy::Capybara.finalize_build }
+    ```
+
+    Learn more about [using Knapsack Pro with Percy](https://docs.knapsackpro.com/ruby/hooks/#percy-capybara) in the docs.
+
+  * We are no longer modifying the default RSpec formatters in Queue Mode. You can remove the [`KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS`](https://docs.knapsackpro.com/ruby/reference/#knapsack_pro_modify_default_rspec_formatters-removed-rspec) environment variable from your CI config if you are using it.
+
+* RSpec improvements in Queue Mode:
+  * Termination signals (`HUP`, `INT`, `TERM`, `ABRT`, `QUIT`, `USR1`, and `USR2`) are handled earlier: the process will terminate before the next top-level example group (`describe` or `context`) instead of waiting for the next Knapsack Pro batch of tests.
+
+  * Respect the `--error-exit-code` option. It sets a custom exit code (instead of `1`) when RSpec fails outside an example (e.g. lack of memory, termination signal).
+
+    ```bash
+    bundle exec rake "knapsack_pro:queue:rspec[--error-exit-code 3]"
+    ```
+
+  * Respect the `--failure-exit-code` option. It sets a custom exit code for when any examples fail.
+
+    ```bash
+    bundle exec rake "knapsack_pro:queue:rspec[--failure-exit-code 2]"
+    ```
+
+  * Respect the `--fail-fast` option and show a warning in the Knapsack Pro log.
+
+  * Ignore the `fail_if_no_examples` option in Queue Mode:
+    * A late CI node, started after all tests were executed by other nodes, is expected to receive an empty batch.
+    * A batch could contain tests with no examples (e.g. commented out)
+
+  * Raise an exception if the [deprecated `run_all_when_everything_filtered`](https://docs.knapsackpro.com/ruby/rspec/#some-of-my-test-files-are-not-executed) option is detected.
+
+PR with the above changes: https://github.com/KnapsackPro/knapsack_pro-ruby/pull/237
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v6.0.4...v7.0.0
+
+### 6.0.4
+
+* fix(minitest): avoid installing `at_exit` (that would result in an empty run of Minitest after Knapsack Pro is finished in Queue Mode)
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/236
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v6.0.3...v6.0.4
+
+### 6.0.3
+
+* fix(Turnip): make sure `.feature` files are recorded
+* fix(RSpec): stop recording `UNKNOWN_PATH` that would generate an error in case of a CI node retry
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/233
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v6.0.2...v6.0.3
+
+### 6.0.2
+
+* fix(RSpec): allow using `TimeTracker` in RSpec < 3.10.2 when formatters were required to expose `#output`
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/232
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v6.0.1...v6.0.2
+
+### 6.0.1
+
+* fix(RSpec): allow using Queue Mode in RSpec <= 3.10 when the `rspec_is_quitting` method is not present for RSpec World object
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/231
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v6.0.0...v6.0.1
+
+### 6.0.0
+
+* __(breaking change)__ Dropped support for Turnip < 2.0.0
+* Use an RSpec Formatter to track tests' execution times more accurately
+* Removed `Time.raw_now`
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/229
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.7.0...v6.0.0
+
+### 5.7.0
+
+* Performance improvement: don't run `rake knapsack_pro:rspec_test_example_detector` when no slow test files are detected for RSpec.
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/225
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.6.0...v5.7.0
+
+### 5.6.0
+
+* Use `frozen_string_literal: true` to reduce memory usage
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/220
+
+* Enforce `frozen_string_literal: true` in the gem files with Rubocop
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/222
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.5.0...v5.6.0
+
+### 5.5.0
+
+* Detect user seats for AppVeyor, Codefresh, Codeship
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/221
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.4.1...v5.5.0
+
+### 5.4.1
+
+* Fixes RSpec conflict (see https://github.com/KnapsackPro/knapsack_pro-ruby/issues/217)
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/218
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.4.0...v5.4.1
+
+### 5.4.0
+
+* Send to the API the CI provider with a header
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/216
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.3.5...v5.4.0
+
+### 5.3.5
+
+* Handle RSpec exceptions when running RSpec in Queue Mode
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/214
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/215
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.3.4...v5.3.5
+
+### 5.3.4
+
+* fix(Queue Mode): handle OS signals and RSpec internal `wants_to_quit` and `rspec_is_quitting` states to stop consuming tests from the Queue API when the CI node is terminated
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/207
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.3.3...v5.3.4
+
+### 5.3.3
+
+* Fix hanging CI when `git fetch --shallow-since` takes too long
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/213
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.3.2...v5.3.3
+
+### 5.3.2
+
+* On top of 5.3.1, avoid noise to stderr when git is not available when collecting the build author
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.3.1...v5.3.2
+
+### 5.3.1
+
+* Avoid noise to stderr when git is not available when collecting authors
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/211
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.3.0...v5.3.1
+
+### 5.3.0
+
+* Perf: Send authors to the API only on the first request (for Queue Mode)
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.2.1...v5.3.0
+
+### 5.2.1
+
+* Shallow fetch the last month of commits only on CI
+* Ensure input to `git shortlog`
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/209
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.2.0...v5.2.1
+
+### 5.2.0
+
+* Send authors to the API
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/208
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.1.2...v5.2.0
+
+### 5.1.2
+
+* Fix broken RSpec split by test examples feature when `SPEC_OPTS` is set in Queue Mode. Ignore `SPEC_OPTS` when generating test examples report for slow test files.
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/191
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.1.1...v5.1.2
+
+### 5.1.1
+
+* Use `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true` as default value in Queue Mode for GitLab CI
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/206
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.1.0...v5.1.1
+
+### 5.1.0
+
+* Mask user seats data instead of hashing it
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/202
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v5.0.0...v5.1.0
+
+
+### 5.0.0
+
+* __(breaking change)__ Use `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true` as default value in Queue Mode and use `false` for proper CI providers
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/198
+
+* Detect CI from environment and get the correct ENVs instead of trying all of them and risk conflicts
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/201
+
+    __(breaking change)__ for Buildkite. You need to pass the `BUILDKITE` environment variable to Docker Compose.
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/issues/204
+
+* Set `RAILS_ENV=test` / `RACK_ENV=test` in Queue Mode
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/199
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v4.1.0...v5.0.0
+
+### 4.1.0
+
+* Add support for CI node retry count on GitHub Actions
+
+    __(breaking change)__ for open-source forked repositories using GitHub Actions. See a fix in PR description:
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/197
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v4.0.0...v4.1.0
+
+### 4.0.0
+
+* __(breaking change)__  Raise when `KNAPSACK_PRO_CI_NODE_BUILD_ID` is missing
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/195
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v3.11.0...v4.0.0
+
+### 3.11.0
+
+* Send distinguishable user seat info over to the API
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/192
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v3.10.0...v3.11.0
+
+### 3.10.0
+
+* Remove Solano CI and Snap CI support because they do not exist anymore
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/194
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v3.9.0...v3.10.0
+
+### 3.9.0
+
+* Suppress all RSpec spec file names displayed in stdout at the beginning of running tests in Regular Mode only when the log level is >= `warn`
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/190
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v3.8.0...v3.9.0
 
 ### 3.8.0
 

data/Gemfile

@@ -2,3 +2,12 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in knapsack.gemspec
 gemspec
+
+group :test do
+  gem 'rspec_junit_formatter', require: false
+  gem 'nokogiri', require: false
+  gem 'simplecov', require: false
+
+  # This line is going to be replaced on CI to test different RSpec versions.
+  # gem 'rspec-core', 'x.x.x'
+end

data/README.md

@@ -18,9 +18,6 @@
   <a href="https://rubygems.org/gems/knapsack_pro">
     <img alt="Gem Version" src="https://badge.fury.io/rb/knapsack_pro.svg" />
   </a>
-  <a href="https://codeclimate.com/github/KnapsackPro/knapsack_pro-ruby">
-    <img alt="Code Climate" src="https://codeclimate.com/github/KnapsackPro/knapsack_pro-ruby/badges/gpa.svg" />
-  </a>
 </div>
 
 <br />
@@ -45,16 +42,12 @@ The `knapsack_pro` gem supports all CIs and the following test runners:
 - Spinach
 - Turnip
 
-## Requirements
-
-`>= Ruby 2.1.0`
-
 ## Installation
 
-The [Installation Guide](http://docs.knapsackpro.com/knapsack_pro-ruby/guide/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=installation_guide) will ask you a few questions and generate instruction steps for your project:
+The [Installation Guide](https://docs.knapsackpro.com/knapsack_pro-ruby/guide/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=installation_guide) will ask you a few questions and generate instruction steps for your project:
 
 <div align="center">
-  <a href="http://docs.knapsackpro.com/knapsack_pro-ruby/guide/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=installation_guide">
+  <a href="https://docs.knapsackpro.com/knapsack_pro-ruby/guide/?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=installation_guide">
     <img alt="Install button" src="./.github/assets/install-button.png" width="116" height="50" />
   </a>
 </div>
@@ -74,7 +67,7 @@ bundle update knapsack_pro
 RSpec:
 
 ```bash
-bundle exec rspec spec
+bin/test
 ```
 
 Scripted tests can be found in the [Rails App With Knapsack Pro repository](https://github.com/KnapsackPro/rails-app-with-knapsack_pro/blob/master/bin/knapsack_pro_all.rb).

data/bin/test

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+bundle exec ruby spec/knapsack_pro/formatters/time_tracker_specs.rb
+FORMATTERS_EXIT_CODE=$?
+
+bundle exec rspec spec
+RSPEC_EXIT_CODE=$?
+
+if [ "$FORMATTERS_EXIT_CODE" -ne "0" ]; then
+  exit $FORMATTERS_EXIT_CODE
+fi
+
+if [ "$RSPEC_EXIT_CODE" -ne "0" ]; then
+  exit $RSPEC_EXIT_CODE
+fi

data/knapsack_pro.gemspec

@@ -4,26 +4,27 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'knapsack_pro/version'
 
 Gem::Specification.new do |spec|
-  spec.name          = "knapsack_pro"
+  spec.name          = 'knapsack_pro'
   spec.version       = KnapsackPro::VERSION
+  spec.required_ruby_version = '>= 2.7.0'
   spec.authors       = ['ArturT']
-  spec.email         = ['arturtrzop@gmail.com']
+  spec.email         = ['support@knapsackpro.com']
   spec.summary       = %q{Knapsack Pro splits tests across parallel CI nodes and ensures each parallel job finish work at a similar time.}
-  spec.description   = %q{Run tests in parallel across CI server nodes based on tests execution time. Split tests in a dynamic way to ensure parallel jobs are done at a similar time. Thanks to that your CI build time is as fast as possible. It works with many CI providers.}
+  spec.description   = %q{Knapsack Pro wraps your current test runner(s) and works with your existing CI infrastructure to parallelize tests optimally. It dynamically splits your tests based on up-to-date test execution data. It's designed from the ground up for CI and supports all of them.}
   spec.homepage      = 'https://knapsackpro.com'
   spec.license       = 'MIT'
   spec.metadata    = {
     'bug_tracker_uri' => 'https://github.com/KnapsackPro/knapsack_pro-ruby/issues',
     'changelog_uri' => 'https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md',
-    'documentation_uri' => 'https://docs.knapsackpro.com/integration/',
+    'documentation_uri' => 'https://docs.knapsackpro.com/knapsack_pro-ruby/guide/',
     'homepage_uri' => 'https://knapsackpro.com',
     'source_code_uri' => 'https://github.com/KnapsackPro/knapsack_pro-ruby'
   }
 
   spec.files         = `git ls-files -z`.split("\x0")
-  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.executables   = ['knapsack_pro']
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
-  spec.require_paths = ["lib"]
+  spec.require_paths = ['lib']
 
   spec.add_dependency 'rake', '>= 0'
 

data/lib/knapsack_pro/adapters/base_adapter.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module KnapsackPro
   module Adapters
     class BaseAdapter
@@ -9,6 +11,14 @@ module KnapsackPro
         "#{KnapsackPro::Config::TempFiles::TEMP_DIRECTORY_PATH}/#{adapter_name}-bind_method_called_for_node_#{KnapsackPro::Config::Env.ci_node_index}.txt"
       end
 
+      def self.split_by_test_cases_enabled?
+        false
+      end
+
+      def self.test_file_cases_for(slow_test_files)
+        raise NotImplementedError
+      end
+
       def self.slow_test_file?(adapter_class, test_file_path)
         @slow_test_file_paths ||=
           begin
@@ -50,13 +60,13 @@ module KnapsackPro
         File.write(self.class.adapter_bind_method_called_file, nil)
 
         if KnapsackPro::Config::Env.recording_enabled?
-          KnapsackPro.logger.debug('Test suite time execution recording enabled.')
+          KnapsackPro.logger.debug('Regular Mode enabled.')
           bind_time_tracker
           bind_save_report
         end
 
         if KnapsackPro::Config::Env.queue_recording_enabled?
-          KnapsackPro.logger.debug('Test suite time execution queue recording enabled.')
+          KnapsackPro.logger.debug('Queue Mode enabled.')
           bind_queue_mode
         end
       end
@@ -73,8 +83,13 @@ module KnapsackPro
         raise NotImplementedError
       end
 
+      def bind_after_queue_hook
+        raise NotImplementedError
+      end
+
       def bind_queue_mode
         bind_before_queue_hook
+        bind_after_queue_hook
         bind_time_tracker
       end
     end

data/lib/knapsack_pro/adapters/cucumber_adapter.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module KnapsackPro
   module Adapters
     class CucumberAdapter < BaseAdapter
@@ -63,9 +65,7 @@ module KnapsackPro
         end
       end
 
-      def bind_queue_mode
-        super
-
+      def bind_after_queue_hook
         ::Kernel.at_exit do
           KnapsackPro::Hooks::Queue.call_after_subset_queue
           KnapsackPro::Report.save_subset_queue_to_file

data/lib/knapsack_pro/adapters/minitest_adapter.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module KnapsackPro
   module Adapters
     class MinitestAdapter < BaseAdapter