flakey_spec_catcher 0.7.1 → 0.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bbde8c00cdc9c8c24d9d81572fe3217d73114d7d360a87ff4f08adea746aae55
-  data.tar.gz: 8142ae2b56440f6a53db1b03c3e736a5556aed9c31945e0ccfbfc3a5d9df5b3b
+  metadata.gz: 765b9037e2fff584368572be0d907ce9d3dba0ce04b283615526947cd3fe2ab2
+  data.tar.gz: 18a09d27cf2e5850d0ddd65e8fe6006ffdd365f2fb164f624187aeea17baf841
 SHA512:
-  metadata.gz: 69f5f46b6dc7a8cf31c030b16a3a129dc5768b60214abc9cf603facc4ad3c94d2ee2df7a08838c9b033d91941dc7af4b388861f94b0f539466a107b2dfd10a7c
-  data.tar.gz: 72325c48ca9c58ab7f65f569905b3a4782b9678cf608f86c9b8982f4b6cc70c797e5a6ee6ab04c7d35c1c9275c35d48d9b1f38b1e9d60e08bf8caf357db08db2
+  metadata.gz: 8ec0601d6583dd288459019b2c686ed83323290d10053be001683e40bcd478850aa3f50b27f88b6c3df5f88cad3cf55ebaf37b7d070833236cb4edb01716af4a
+  data.tar.gz: a97d0fbe2bb48992a938105556cba87fa25ed48e8dff0404b7c0ad4d9fc59c4242bf74c26550418d1e1385dc5ec144de9dca3d1b0bce03af13e6091b247d5bc5

data/README.md

@@ -1,12 +1,27 @@
-# Flakey Spec Catcher (FSC)
+# Flakey Spec Catcher
 
 ### About:
 
-This gem is intended to catch and prevent the merging of flakey ruby specs.
-Currently, it will run the equivalent of a git diff between the current branch's
-commit and origin/master to detect any changes to files that contain `_spec.rb`.
-For these files, it will re-run either the changed tests or the whole spec
-file's test suite a specified number of times.
+This gem is intended to catch and prevent the merging of flakey RSpec tests.
+
+There are two primary usecases for flakey_spec_catcher (FSC):
+
+1. Git Detection - Make changes to specs, add them to a git commit and then
+   allow FSC to detect changes and re-run only the new/edited tests in your
+   commit many times.
+
+2. Manual Re-runs - Specify a test to re-run many times regardless of whether it has
+   corresponding changes in yuor commit.
+
+FSC detects changes by running the equivalent of a git diff
+between the current branch's commit and the HEAD of Source Control Management (SCM)
+to detect any changes to files that contain `_spec.rb`. For these files, it will
+re-run either the changed tests or the whole spec file's test suite a specified
+number of times based on user configurations.
+
+If all re-runs that FSC triggers pass, the result of running it
+will be a status code of 0. If a single re-run fails, a non-zero exit status
+will result.
 
 Note that `flakey_spec_catcher` will only detect committed changes, so make sure
 to add any changed spec files to the commit and amend it if you're testing
@@ -15,27 +30,70 @@ locally.
 All that's required is to install the gem and then call its executable,
 `flakey_spec_catcher`.
 
+### Why use FSC?
+
+FSC allows you to specify a usage with which to re-run
+tests. The default usage is inserting all test cases manually into
+RSpec::Core::Runner API and then clearing all example and test data between
+each re-run. So long as RSpec is configured to not reset browser+server
+configurations between test runs or test suites, re-running tests this way
+will be the fastest way to ensure that tests are not flakey.
+If providing a custom usage, FSC will run each re-run
+as a separate process and will likely incur more significant overhead, however,
+it may require less tweaking of your existing RSpec configuration.
+
+If you have setup your environment to use the default usage
+(RSpec::Core::Runner), FSC will use an RSpec listener to
+combine your test results and give you a consolidated summary of how many re-runs
+of each test case failed with a specific assertion.
+
+FSC is ideal for testing in continuous integration since it will
+run only created/edited tests and can run them many times in an efficient manner.
+So long as silent mode is not enabled, FSC will return an exit
+status that can be used to fail a build if flakey specs are detected.
+
+### What do I need to run FSC?
+
+You'll need to be running in a git repository for FSC to detect
+changes. You'll also need to have at least one commit merged in SCM so that
+FSC has a baseline for comparison.
+
+Ensure that your gem dependencies meet the base requirements specified by
+flakey_spec_catcher.gemspec
+
+### Tips for running FSC efficiently
+
+In order to run FSC in the most efficient way via RSpec::Core::Runner,
+you'll need to ensure that your browser+server connections do not reset before/after
+each example or example group if using Selenium. If RSpec has additional dependencies,
+you'll also need to ensure that FSC has access to any needed gems.
+It's best to do this by adding flakey_speec_catcher to your Gemfile via bundler
+and then running it with `bundle exec flakey_spec_catcher`
+
+### In what cases if FSC not suitable?
+
+Since FSC by default re-runs tests at the smallest possible level of
+change (a single test case or example), it is not suitable to allow it to run
+on non-idempotent tests, or any kind of test that cannot be re-run on its own repeatedly.
+For any such tests, applicable test files may be excluded or specific examples may
+be tagged and those tags can then be excluded in runtime.
+
 ### Install:
 
 ```sh
 gem install flakey_spec_catcher
 ```
 
-Or add it to your Gemfile
-
-```ruby
-# Gemfile
-gem 'flakey_spec_catcher'
-```
-
-and install with Bundler:
+Or add it using bundler
 
 ```sh
+bundle add flakey_spec_catcher
 bundle install
 ```
 
 ### Configuration:
 
+#### Environment Variables
 In some cases, certain environment variables can be configured:
 
   - `FSC_REPEAT_FACTOR`: Number of times the examples in each file will be run.
@@ -44,8 +102,8 @@ In some cases, certain environment variables can be configured:
   - `FSC_IGNORE_FILES`: Specify changed files that will be exempt from FSC
   re-runs. Regex matching is allowed.
 
-  - `FSC_IGNORE_BRANCHES`: Ignore any committed changes related to the specified
-  branch(es).
+  - `FSC_IGNORE_BRANCHES`: Disable running flakey_spec_catcher in git detection
+  mode on any commits occurring on a specified branch.
 
   - `FSC_SILENT_MODE`: If 'true', FSC will return back with a exit status 0
   regardless of the result. If 'false', FSC will return a non-zero exit status
@@ -60,6 +118,20 @@ In some cases, certain environment variables can be configured:
   'spec/ui' with `bundle exec rspec` and want your 'spec/api' changes to be
   re-run with `parallel_rspec`, you could specify the following value:
   `FSC_USAGE_PATTERNS='{ spec/ui/** => bundle exec rspec }, { spec/api => parallel_rspec }'`
+  Note that by specifying a usage pattern, you will be running a separate process
+  for each re-run and runtimes will suffer. This should only be used when re-running
+  via RSpec::Core::Runner (default usage) is not suitable
+
+  - `FSC_EXCLUDED_TAGS`: Specify tags to exclude from re-runs. If a given example/testcase
+  matches any of the specified tags, then that testcase will be excluded from the
+  re-run queue. Tags may contain a corresponding value that is either a symbol or string and
+  multiple tag name value pairs may be specified in a comma separated list.
+  For example, if a testcase contains a description such as
+  `it 'tests controller functionality', :tag1 => 'enabled' do`
+  then this test could be excluded using
+  `FSC_EXCLUDED_TAGS=':tag1 => enabled'.
+  Note: examples do not inherit the tags from their parent example groups or parent contexts.
+
 
 Any of these environment variables can be overriden in the commit message by adding the environment variable key and
 usage/value according to the accepted values specified above.
@@ -75,20 +147,127 @@ FSC_USAGE_PATTERNS = '{ spec/ui => bundle exec rspec }, { spec/api => parallel_r
 
 ```
 
+#### Command Line Arguments
+```
+  -t, --test=TEST_NAME             Specify one or more specs in comma separated list
+  -u, --usage=USAGE                Specify a re-run usage for the manual re-run
+  -r, --repeat=REPEAT_FACTOR       Specify a repeat factor for the manual re-run(s)
+  -e  --excluded-tags=EXCLUDED     Specify tags to exclude in a comma separated list
+  -v, --version                    Prints current flakey_spec_catcher_version
+  -h, --help                       Displays available flakey_spec_catcher cli overrides
+```
+
+Examples:
+
+```
+# Re-run spec/test_spec.rb:4 50 times using RSpec::Core::Runner
+flakey_spec_catcher --test='spec/test_spec.rb:4' --repeat='50'
+
+# Re-run all tests in spec/test_spec.rb 50 times each using 'bundle exec rspc' for
+#  each re-run (at a file level since no line number is specified)
+flakey_spec_catcher --test='spec/test_spec.rb' --repeat='50' --usage='bundle exec rspec'
+
+# Re-run all tests found by git detection and exclude tests that
+#  have the tags :flakey or :unsuccessful => true
+flakey_spec_catcher --excluded-tags=':flakey, :unsuccessful => true'
+
+```
+
 ### Manually re-running a test:
 
 Once the gem has been installed, you may manually specify a test to run along with a custom usage
 (if none is provided, the specified test will run with RSpec::Core::Runner) and a repeat_factor.
 If no repeat_factor is provided, FSC will check ENV['FSC_REPEAT_FACTOR'] to see if a value
-has been configured. If none was provided, it will fall back on the default of  20 re-runs
+has been configured. If none was provided, it will fall back on the default of 20 re-runs
 
 Usage Examples:
 ```
+# Re-run api/spec/user_spec.rb:3 using RSpec::Core::Runner (default usage) 15 times
 flakey_spec_catcher --test='api/spec/user_spec.rb:3' --repeat='15'
+
+# Re-run all files matching the glob api/spec/*_spec.rb in a separate process using
+# 'bundle exec parallel_rspec' FSC_REPEAT_FACTOR times each
 flakey_spec_catcher --test='api/spec/*_spec.rb' --usage='bundle exec parallel_rspec'
+
+# Re-run api/spec/admin_spec.rb in a separate process using rspec, 10 times each
 flakey_spec_catcher --test='api/spec/admin_spec.rb' --repeat='10' --usage='rspec'
 ```
 
+### Additional Git Detection Examples
+
+```sh
+# Run git detection mode and exclude tests that have the tags :flakey or :unsuccessful => true
+#  run each test 20 times (default) each
+bundle exec flakey_spec_catcher --excluded-tags=':flakey, :unsuccessful => true'
+```
+
+```sh
+export FSC_REPEAT_FACTOR='10'
+export FSC_RERUN_FILE_ONLY='true'
+export FSC_IGNORE_FILES='test_spec.rb'
+
+# Run git detection mode, exclude any tests that match /test_spec.rb/ 10 times each
+#  if a change is found in any other *_spec.rb file, all tests in that file will be re-run
+bundle exec flakey_spec_catcher
+
+```
+
+### Full example
+  1. In a repo, changes are made to spec files that are then added to a git commit.
+  2. Run flakey_spec_catcher with any desired user configurations
+
+  Example Output:
+  ```Flakey Spec Catcher Settings:
+  Current Branch: master
+  Remote: origin
+  Current Sha: 12341234123412341234123412341234
+  Base Sha: 56785678567856785678567856785678
+  Repeat factor: 20
+  Changed Specs Detected: ["api/spec/changed_spec1.rb:2", "api/spec/changed_spec.rb:3",
+   "ui/spec/new_spec.rb:3", "ui/spec/new_spec.rb: 5",
+  ]
+
+  ********************************************
+  Re-run Preview
+  Running api/spec/changed_spec1.rb:2 20 times
+  Running api/spec/changed_spec1.rb:3 20 times
+  Running ui/spec/new_spec.rb:3 20 times
+  Running ui/spec/new_spec.rb:5 20 times
+  ********************************************
+
+  ********** SUMMARY **********
+  2 example(s) ran 20 times without any failures
+
+  *************************************************
+              No Flakiness Detected!
+  *************************************************
+  ```
+
+  If any of the tests were flakey, output would resemble something like
+
+  ```********** SUMMARY **********
+  1 example(s) ran 20 times without any failures
+
+  Test Description (./api/spec/test_spec.rb:2)
+
+  FAILED 15 / 20 times
+    5 times with exception message:
+      expected: 1
+          got: 0
+
+      (compared using ==)
+    5 times with exception message:
+      expected: 1
+          got: 2
+
+      (compared using ==)
+    5 times with exception message:
+      expected: 1
+          got: 3
+
+      (compared using ==)
+  ```
+
 ### How FSC works:
 
   1. GitController runs a git diff and creates a ChangeSummary object to
@@ -98,15 +277,18 @@ flakey_spec_catcher --test='api/spec/admin_spec.rb' --repeat='10' --usage='rspec
   3. After representing all changed code blocks as ChangeContext objects within
   a ChangeCapsule (one per file), ChangeCapsules are stored in the
   CapsuleManager.
-  4. User Configurations are parsed using the UserConfig class and an instance
-  is stored as an attribute in the Runner class.
-  5. UserConfig and CapsuleManager objects are used to initialize a RerunManager
+  4. Command line interface overrides are retrieved using the CLIOverride class
+  and a referece to a CLIOverride instance is set in the UserConfig instance.
+  5. User Configurations are parsed using the UserConfig class for environment variables
+  and are set for any user configurations that don't have an existing CLIOverride
+  provided value.
+  6. UserConfig and CapsuleManager objects are used to initialize a RerunManager
   object which creates a RerunCapsule object for each ChangeContext and pairs it
   with an rspec usage based on user configurations.
-  6. RerunManager flattens the number of re-runs based on user settings and
+  7. RerunManager determines thee re-runs based on user settings and
   passes back the identified file/test changes to the Runner along with their
   specified usages.
-  7. Runner will re-run each of the changes at a file level or the specific
+  8. Runner will re-run each of the changes at a file level or the specific
   context-level of each test and will exit with a 0 if all changes passed all of
   their re-runs or a non-zero exit status (passed from RSpec).
 
@@ -155,13 +337,22 @@ code health and code coverage are both assessed in SonarQube.
 See: <https://sonarqube.core.inseng.net/dashboard?id=tab%3Aflakey_spec_catcher>
 
 To test your FSC changes, we've setup some scripts that will build docker images
-and run tests in the latest ruby versions.
+and run the FSC specs in the latest ruby versions.
 
 ```sh
 bash bin/build
 bash bin/test
 ```
 
+To build the gem locally and test out FSC in another repository:
+
+```sh
+gem build flakey_spec_catcher.gemspec
+gem install flakey_spc_catcher-<MAJOR-MINOR-PATCH>.gem
+<cd to repo for testing>
+flakey_spec_catcher <ARGS>
+```
+
 ### License:
 
 MIT. See LICENSE.txt for details.

data/lib/flakey_spec_catcher/user_config.rb

@@ -14,7 +14,8 @@ module FlakeySpecCatcher
     attr_reader :enable_runs
 
     USER_CONFIG_ENV_VARS = %w[FSC_REPEAT_FACTOR FSC_IGNORE_FILES FSC_IGNORE_BRANCHES
-                              FSC_SILENT_MODE FSC_RERUN_FILE_ONLY FSC_USAGE_PATTERNS].freeze
+                              FSC_SILENT_MODE FSC_RERUN_FILE_ONLY FSC_USAGE_PATTERNS
+                              FSC_EXCLUDED_TAGS].freeze
 
     def initialize(cli_override: CliOverride.new)
       apply_env_var_settings
@@ -150,6 +151,8 @@ module FlakeySpecCatcher
         @rerun_file_only = env_var_string_to_bool(env_value)
       when 'FSC_USAGE_PATTERNS'
         @rspec_usage_patterns = env_var_string_to_pairs(env_value)
+      when 'FSC_EXCLUDED_TAGS'
+        @excluded_tags = env_var_string_to_tags(env_value)
       end
     end
     # rubocop:enable Metrics/CyclomaticComplexity

data/lib/flakey_spec_catcher/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module FlakeySpecCatcher
-  VERSION = '0.7.1'
+  VERSION = '0.7.2'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: flakey_spec_catcher
 version: !ruby/object:Gem::Version
-  version: 0.7.1
+  version: 0.7.2
 platform: ruby
 authors:
 - Brian Watson
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-01-10 00:00:00.000000000 Z
+date: 2020-01-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec