knapsack_pro 0.41.0 → 0.42.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 13b0f54f8e0ffa8fa8a1dafac285ccdad9ee4559
-  data.tar.gz: e7208a544178784df5dcfc4d00032f4333e22408
+  metadata.gz: 57d4426354fb25838c3bb77b4fca90389abe875b
+  data.tar.gz: b1099760fde424a458a77f6c405638c0658279b8
 SHA512:
-  metadata.gz: 91ceb897da7ec79de2c86a8dc76298d4f9d3a091fe90a66657c3cb6a81b9188e613fcc65e3376309742922f3f0513e0b5f7a9d3b7ac0055febec6c31fb500994
-  data.tar.gz: 0a7f9b2160f50e6423f2a58afb4fdf774e779524ab5f2a8efb6a6a8b0cee6ff9e97b859916ce27427eda5de6faf806bce2294a58f7ad19d57ebae7f166cdb1a4
+  metadata.gz: 623a862646a5e14aa0423a51b095569485d014a68c2b135756890465e94d3ec72aa5650a64133c143488d0bf7739dc169b902eff661bc3d59023062f9c7426b8
+  data.tar.gz: 37e03af1ea495e3d42eb0d5f75d0fdcc53d77d689fc317d7931c4692f0348b2fcf6e5a76dbfb21dd3469808b1576a23c8fb3376730841c4d6e2641a8324ef0b6

data/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 * TODO
 
+### 0.42.0
+
+* Clear RSpec examples without shared examples in similar way as in RSpec 3.6.0
+
+    https://github.com/KnapsackPro/knapsack_pro-ruby/pull/42
+
+https://github.com/KnapsackPro/knapsack_pro-ruby/compare/v0.41.0...v0.42.0
+
 ### 0.41.0
 
 * Add after subset queue hook and example how to use JUnit formatter in Queue Mode.

data/README.md

@@ -103,6 +103,7 @@ The knapsack_pro has also [queue mode](#queue-mode) to get optimal test suite sp
   - [Common problems](#common-problems)
     - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required)
     - [Queue Mode problems](#queue-mode-problems)
+      - [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard)
       - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result)
       - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests)
       - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again)
@@ -128,6 +129,7 @@ The knapsack_pro has also [queue mode](#queue-mode) to get optimal test suite sp
     - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example)
     - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode)
     - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem)
+    - [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests)
   - [Questions around data usage and security](#questions-around-data-usage-and-security)
     - [What data is sent to your servers?](#what-data-is-sent-to-your-servers)
     - [How is that data secured?](#how-is-that-data-secured)
@@ -362,8 +364,12 @@ If above command fails then you may need to explicitly pass an argument to requi
 Note if you will run queue mode command for the first time it might be slower.
 The second build should have better optimal test suite split.
 
+Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes.
+
 If you use capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec).
 
+If you use rspec_junit_formatter gem then please [follow this step](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode).
+
 ### Additional info about queue mode
 
 * You should use different API token for queue mode than for regular mode to avoid problem with test suite split in case you would like to go back to regular mode.
@@ -856,6 +862,14 @@ knapack_pro gem cannot determine the git commit hash and branch name. To fix thi
 
 #### Queue Mode problems
 
+##### Why I don't see collected time execution data for my build in user dashboard?
+
+If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open build for your last git commit you should see there info about collected time execution data from all CI nodes. If you don't see collected time execution data for CI nodes then please ensure:
+
+* you have `Knapsack::Adapters::RspecAdapter.bind` in your `rails_helper.rb` or `spec_helper.rb`
+* you explicitly set `RAILS_ENV=test` on your CI nodes (for instance you use Docker then please set `RAILS_ENV`)
+* knapsack_pro Queue Mode saves temporary files with collected time execution data in `your_rails_project/tmp/knapsack_pro/queue/`. Please ensure you don't clean `tmp` directory in your tests so knapsack_pro can publish time execution data to Knapsack Pro API server.
+
 ##### Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?
 
 The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests).
@@ -1268,6 +1282,30 @@ To ensure everything works you can check output for each CI node.
     (tests output here)
     ```
 
+#### How to retry failed tests (flaky tests)?
+
+Flaky (nondeterministic) tests, are tests that exhibit both a passing and a failing result with the same code.
+
+You can use [only failures option in RSpec](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rerun failed tests.
+
+Please add to your RSpec configuration:
+
+```
+RSpec.configure do |c|
+  c.example_status_persistence_file_path = "tmp/rspec_examples.txt"
+end
+```
+
+Then you can execute rspec with only failed tests after main knapsack_pro command finish.
+
+```
+# Run knapack_pro in Queue Mode and it will save failed tests in tmp/rspec_examples.txt
+bundle exec rake knapsack_pro:queue:rspec
+
+# run only failed tests from tmp/rspec_examples.txt
+bundle exec rspec --only-failures
+```
+
 ### Questions around data usage and security
 
 #### What data is sent to your servers?

data/lib/knapsack_pro/runners/queue/rspec_runner.rb

@@ -49,7 +49,8 @@ module KnapsackPro
             options = RSpec::Core::ConfigurationOptions.new(cli_args)
             exit_code = RSpec::Core::Runner.new(options).run($stderr, $stdout)
             exitstatus = exit_code if exit_code != 0
-            RSpec.world.example_groups.clear
+
+            rspec_clear_examples
 
             KnapsackPro::Hooks::Queue.call_after_subset_queue
 
@@ -75,6 +76,23 @@ module KnapsackPro
             KnapsackPro::TestFilePresenter.stringify_paths(test_file_paths)
           )
         end
+
+        # Clear rspec examples without the shared examples:
+        # https://github.com/rspec/rspec-core/pull/2379
+        #
+        # Keep formatters and report to accumulate info about failed/pending tests
+        def self.rspec_clear_examples
+          if RSpec::ExampleGroups.respond_to?(:remove_all_constants)
+            RSpec::ExampleGroups.remove_all_constants
+          else
+            RSpec::ExampleGroups.constants.each do |constant|
+              RSpec::ExampleGroups.__send__(:remove_const, constant)
+            end
+          end
+          RSpec.world.example_groups.clear
+          RSpec.configuration.start_time = ::RSpec::Core::Time.now
+          RSpec.configuration.reset_filters
+        end
       end
     end
   end

data/lib/knapsack_pro/version.rb

@@ -1,3 +1,3 @@
 module KnapsackPro
-  VERSION = '0.41.0'
+  VERSION = '0.42.0'
 end

data/spec/knapsack_pro/runners/queue/rspec_runner_spec.rb

@@ -97,7 +97,7 @@ describe KnapsackPro::Runners::Queue::RSpecRunner do
         expect(RSpec::Core::Runner).to receive(:new).with(options).and_return(rspec_core_runner)
         expect(rspec_core_runner).to receive(:run).with($stderr, $stdout).and_return(exit_code)
 
-        expect(RSpec).to receive_message_chain(:world, :example_groups, :clear)
+        expect(described_class).to receive(:rspec_clear_examples)
 
         expect(KnapsackPro::Hooks::Queue).to receive(:call_after_subset_queue)
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: knapsack_pro
 version: !ruby/object:Gem::Version
-  version: 0.41.0
+  version: 0.42.0
 platform: ruby
 authors:
 - ArturT
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-05-26 00:00:00.000000000 Z
+date: 2017-06-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake