rspec_n 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: 779f827fe4de372504a3fa7c5d4168914b4a0c3d51c05f7e441cf94ecd6953ba
-  data.tar.gz: 928f1f99ec404b2f3a65d07542521658dfb1296e0ed5e9e5623f256b22789db8
+SHA1:
+  metadata.gz: 02e086ee01e65f0ac9e1352f77e8ed372a999284
+  data.tar.gz: 9a2b8f3caaf24bb2812dd8f90520df1edf5ab6e3
 SHA512:
-  metadata.gz: b5f002fde7c4a9ad096a17abfe21a836efa53a9ce42c3d8ad94de3df68213455555f2fe5e21bd808f3a883cfa8591f9401bb925adc3e3afe5e084ccd290b2280
-  data.tar.gz: 48060eb007f625e03ab00e498b2c5cdf2fd365b59fb658978549a527b438bc000f988c875ff960e4f5618dd4804627b06bf1cf9852fc5d4fe4e31b10c5dc4133
+  metadata.gz: 7f896ea775b492f458322bbffc031346e2cb5406e47d6d46103b1ddc251e9b632b4b733ba4ed8d44501da9501834869c7929fc1dccb9a230c1321d2dc2d501a5
+  data.tar.gz: eb1ecd8d5a36d39572fec169a4997e347f3328496aab16960b3db7aab41f472d26ee7c02c91ecf8e7432b0effcb1b27c75276b89b24f4ab2728fd64a0efa404a

data/CHANGELOG.md

@@ -1,8 +1,14 @@
 # CHANGELOG
 
-## Next Release
+## 1.1.0 (Mar 03, 2019)
 
-1. (New) Add initial core logic. (Issue #1)
-1. (New) Add initial CLI. (Issue #4)
-1. (New) Write results of each iteration to a separate file.  (Issue #5)
-1. (New) Add `-s` option to stop on first failure.  (Issue #6)
+1. (New) Add **Result Counts** column which shows the result counts for each run (e.g. "400 examples, 2 failures, 3 pending" ).  ([Issue #19](https://github.com/roberts1000/rspec_n/issues/19))
+1. (New) Allow an optional path argument, that is passed to RSpec, to target specs (e.g. `rspec_n 3 spec/features/example_spec.rb`, `rspec_n spec/features/example_spec.rb:5`, `rspec_n spec/features 4`).  ([Issue #20](https://github.com/roberts1000/rspec_n/issues/20))
+1. (New) Rename the `Result Counts` column to `Results` and remove the existing `Results` column.  ([Issue #23](https://github.com/roberts1000/rspec_n/issues/23))
+
+## 1.0.0 (Feb 18, 2019)
+
+1. (New) Add initial core logic. ([Issue #1](https://github.com/roberts1000/rspec_n/issues/1))
+1. (New) Add initial CLI. ([Issue #4](https://github.com/roberts1000/rspec_n/issues/4))
+1. (New) Write results of each iteration to a separate file.  ([Issue #5](https://github.com/roberts1000/rspec_n/issues/5))
+1. (New) Add `-s` option to stop on first failure.  ([Issue #6](https://github.com/roberts1000/rspec_n/issues/6))

data/Gemfile

@@ -1,5 +1,7 @@
 source "https://rubygems.org"
 
+ruby '2.3.7'
+
 # Specify your gem's dependencies in rspec_n.gemspec
 gemspec
 

data/LICENSE.md

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2019 Roberts
+Copyright (c) 2019 roberts1000
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -2,12 +2,16 @@
 
 rspec_n is a Ruby gem that makes it easy to run a project's RSpec test suite N times.  You can customize the command that is used to start RSpec, or let rspec_n guess the best command (based on the files in your project).  rspec_n is useful for finding repeatability or flakiness issues in automated test suites.
 
-![sample](https://user-images.githubusercontent.com/2053901/52986788-d0bb7c80-33c6-11e9-9f13-0e191bdd2bb3.png)
+![example](https://user-images.githubusercontent.com/2053901/53691471-c6956880-3d4c-11e9-8248-68bbb4c24786.png)
 
 ## Version Policy
 
 Releases are versioned using [semver 2.0.0](https://semver.org/spec/v2.0.0.html).
 
+## Supported Rubies
+
+Ruby 2.3.7+ is supported.
+
 ## Installation
 
 Install by executing
@@ -18,43 +22,63 @@ The gem will install an exectuable called `rspec_n` on your system.  You may als
 
 ## Usage
 
-To run RSpec N times
+The simplest way to run rspec_n is to give it a positive integer which tells it how many times to run RSpec:
 
-    $ rspec_n N
+    $ rspec_n 5
 
-If your project has a `bin/start_rspec` file, rspec_n will invoke it to start RSpec (rspec_n assumes you have placed the necessary commands to cleanly start your test suite in that file). Otherwise, it will attempt to identify your project type and use best practices to start RSpec for your given project type.  The following is a list of project types supported by rspec_n, and the commands used when that project type is selected:
+As each iteration completes, output will be sent to the screen and dumped to a file.  If you need to examine the detailed output of a run, it is available in the output files.
 
-1. Ruby on Rails - `DISABLE_DATABASE_ENVIRONMENT_CHECK=1 RAILS_ENV=test bundle exec rake db:drop db:create db:migrate && bundle exec rspec`.
+You can also list one or more paths to target specs.  You can do anything you would normally do when giving RSpec paths:
 
-If it cannot guess a project type, rspec_n will invoke `bundle exec rspec`.  You can override this by specifying a command directly on the CLI.
+    $ rspec_n 5 spec/path/to/something_spec.rb
+    $ rspec_n 5 spec/path/to/folder spec/path/to/some/other/file_spec.rb
+    $ rspec_n 5 spec/path/to/folder
+    $ rspec_n 5 spec/path/to/something_spec.rb:5
 
-#### Use Custom Command to Start RSpec
+By default, `--order rand` is sent to RSpec to force it to run specs in random order.  You can use a `defined` order if you don't want randomness:
 
-By default, rspec_n will inspect the files in your project and pick the best way to start RSpec.  You can take control by using the `-c` option which lets you specify a custom command.  The following example deletes the `tmp` folder before starting RSpec:
+    $ rspec_n 5 --order defined
 
-    $ rspec_n 5 -c 'rm -rf tmp && bundle exec rspec'
+Or, let the configuration files in the project determine the order:
+
+    $ rspec_n 5 --order project
+
+#### Automatic Command Selection
+
+rspec_n will inspect the files in your project and pick the best way to start RSpec.  If it can't make an educated guess, it will use `bundle exec rspec` as the base command and add any extra information you've entered on the command line (like the order or paths).  The following is a list of project types that rspec_n can identify and the associated commands it will try to execute:
 
-You must wrap your entire command string in single or double quotes if it contains spaces (which it probably will). This command will be invoked each time rspec_n attempts to start RSpec.
+1. Ruby on Rails Applications: `DISABLE_DATABASE_ENVIRONMENT_CHECK=1 RAILS_ENV=test bundle exec rake db:drop db:create db:migrate && bundle exec rspec`.
+2. Everything else: `bundle exec rspec`.
 
-#### Control spec Order
+#### Use Custom Command to Start RSpec
+
+If you don't want rspec_n to automatically pick the best command, you can use the `-c` option and specify a command.  The following example deletes the `tmp` folder before starting RSpec:
 
-rspec_n provides three command line options for controlling the execution order of your specs.
+    $ rspec_n 5 -c 'rm -rf tmp && bundle exec rspec'
 
-1. `--order rand`    - Passes `--order rand` to RSpec which causes RSpec to run your specs in a random order.
-1. `--order defined` - Passes `--order defined` to RSpec which causes RSpec to run your specs in the order that they are loaded by RSpec.
-1. `--order project` - Passes nothing to RSpec, which means the project configuration files will determine the order.
+There are couple points to consider:
 
-If you do nothing, rspec_n will use `--order rand`.  This ensures the specs in your project are executed in random order for each iteration of RSpec.  rspec_n doesn't supply a seed when it start RSpec using a random strategy; it lets RSpec choose the seed and reports the values in the results.
+1. Wrap your entire command in a single or double quoted string so rspec_n can acturately determine the command.
+1. You can use the `&&` operator to join commands.
+1. rspec_n was partially created to help discover flaky test suites so it will add `--order rand` to a custom command even if don't specify the order.  You must explicitly use `--order defined` or `--order project` if you want something else.
 
-#### Control File Ouput
+#### Control File Output
 
 rspec_n writes output for each iteration in a sequence of files `rspec_n_iteration.1`, `rspec_n_iteration.2`, etc...  This saves you from having to rerun your test suite when you find a particular seed that causes your test suite to fail.  If you want to disable this, add the `--no-file` option to the command.
 
-**Note:** rspec_n deletes all files matching `rspec_n_iteration.*` when it starts so you must move those files to another location if you want to save them.
+    $ rspec_n 5 --no-file
+
+**Note:** rspec_n deletes all files matching `rspec_n_iteration.*` when it starts so be sure tomove those files to another location if you want to save them.
 
 #### Stop on First Failure
 
-You can tell rspec_n to abort when an iteration fails by using the `-s` flag.  Any remaining iterations will be skipped.
+You can tell rspec_n to abort the first time an iteration fails by using the `-s` flag.  Any remaining iterations will be skipped.
+
+## Understanding the Results
+
+rspec_n uses the STDOUT, STDERR and EXIT STATUS of the `rspec` command  to figure out what to show in the **Results** column.  The general results are determined by finding the line in RSpec's STDOUT that says `xyz examples, xyz failures, xyz pending`.  rspec_n considers the run to be successful if RSpec returns an EXIT STATUS of 0 **regardless of any content in the STDERR stream**; it considers the run to be a failure if RSpec's EXIT STATUS > 0.
+
+There are times when RSpec's STDERR might have content, even though it returns an EXIT STATUS of 0. This frequently happens with deprecation notices and RSpec itself will pass the test suite in this situation. Unfortunately, it's not uncommon for code to write messages that look like errors to STDERR, but not actually raise an error and cause RSpec to fail a spec example.  rspec_n reports this situation by adding a `(Warning)` label to the results to indicate there's something extra in the STDERR that you might want to investigate.
 
 ## Development
 

data/exe/rspec_n

@@ -16,7 +16,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 require "cri"
 require "rspec_n/constants"
-require "rspec_n/errors"
+require "rspec_n/errors/bad_option"
 
 # rubocop:disable Metrics/BlockLength
 command = Cri::Command.define do
@@ -64,10 +64,6 @@ end
 begin
   command.run(ARGV)
   exit 0
-# Raised when the user specifies an argument incorrectly.
-rescue RspecN::BadArgument => e
-  warn e.message.colorize(:red)
-  exit 1
 rescue RspecN::BadOption => e
   warn e.message.colorize(:red)
   exit 1

data/lib/rspec_n.rb

@@ -8,7 +8,7 @@ require "rspec_n/helpers/time_helpers"
 require "rspec_n/helpers/core_ext/array"
 require "rspec_n/helpers/core_ext/string"
 
-require "rspec_n/errors"
+require "rspec_n/errors/bad_option"
 
 require "rspec_n/constants"
 require "rspec_n/input"

data/lib/rspec_n/formatters/file_formatter.rb

@@ -3,7 +3,7 @@ module RspecN
     class FileFormatter
       include RspecN::TimeHelpers
 
-      BASE_FILE_NAME = "rspec_n_iteration"
+      BASE_FILE_NAME = "rspec_n_iteration".freeze
 
       def initialize(runner:)
         @runner = runner
@@ -12,7 +12,7 @@ module RspecN
       end
 
       def delete_all_files
-        Dir.glob("#{BASE_FILE_NAME}.**").each { |file| File.delete(file)}
+        Dir.glob("#{BASE_FILE_NAME}.**").each { |file| File.delete(file) }
       end
 
       def write(run)

data/lib/rspec_n/formatters/table_formatter.rb

@@ -7,7 +7,14 @@ module RspecN
 
       def initialize(runner:)
         @runner = runner
-        @columns = { "Run" => 8, "Start Time" => 22, "Finish Time" => 22, "Duration" => 13, "Seed" => 11, "Result" => 18 }
+        @columns = {
+          "Run" => 7,
+          "Start Time" => 21,
+          "Finish Time" => 21,
+          "Duration" => 12,
+          "Seed" => 9,
+          "Results" => 50
+        }
         @header_columns_string = padded_header_column_labels
         @table_width = @header_columns_string.size
         @format = "%m/%d %l:%M:%S %p"
@@ -28,7 +35,7 @@ module RspecN
         print pad_field("Finish Time", run.formatted_finish_time(@format))
         print duration_field(run)
         print seed_field(run)
-        puts result_field(run)
+        puts result_count_field(run)
       end
 
       private
@@ -41,12 +48,11 @@ module RspecN
       def write_conclusion
         puts "-" * @table_width
         puts ""
-        puts "Total Duration:              #{convert_seconds_to_hms(@runner.total_duration_seconds)}"
-        puts "Average Duration:            #{convert_seconds_to_hms(@runner.avg_duration_seconds)}"
-        puts "Total Passed:                #{@runner.total_passed.to_s.colorize(:green)}"
-        puts "Total Passed with Warnings:  #{@runner.total_passed_with_warnings.to_s.colorize(:green)}"
-        puts "Total Failed:                #{@runner.total_failed.to_s.colorize(:red)}"
-        puts "Total Skipped:               #{@runner.total_skipped}"
+        puts "Total Duration:      #{convert_seconds_to_hms(@runner.total_duration_seconds)}"
+        puts "Avg Run Duration:    #{convert_seconds_to_hms(@runner.avg_duration_seconds)}"
+        puts "Runs Passed:         #{@runner.total_passed.to_s.colorize(:green)}"
+        puts "Runs Failed:         #{@runner.total_failed.to_s.colorize(:red)}"
+        puts "Runs Skipped:        #{@runner.total_skipped}"
         puts ""
       end
 
@@ -65,7 +71,7 @@ module RspecN
 
       def pad_field(column_name, value)
         max_width = max_column_width_for(column_name)
-        value_size = value.to_s.size
+        value_size = value.to_s.remove_color.size
         pad_count = max_width - value_size
         value.to_s + (" " * pad_count)
       end
@@ -79,13 +85,18 @@ module RspecN
         run.seed.nil? ? pad_field("Seed", "None") : pad_field("Seed", run.seed)
       end
 
-      def result_field(run)
+      def result_count_field(run)
+        warning_part = run.has_warnings? ? " (Warnings)".colorize(:yellow) : ""
+        run.result_count_string.colorize(result_color_symbol(run)) + warning_part
+      end
+
+      def result_color_symbol(run)
         case run.status_string
-        when "Pass with Warnings" then "Pass with Warnings".colorize(:yellow)
-        when "Pass" then "Pass".colorize(:green)
-        when "Fail" then "Fail".colorize(:red)
-        when "Skip" then "Skip".colorize(:yellow)
-        else "Unknown".colorize(:yellow)
+        when "Pass (Warnings)" then :green
+        when "Pass" then :green
+        when "Fail" then :red
+        when "Skip" then :yellow
+        else :yellow
         end
       end
     end

data/lib/rspec_n/helpers/core_ext/string.rb

@@ -3,4 +3,8 @@ class String
   def all_digits?
     self =~ /\A\d+\z/ ? true : false
   end
+
+  def remove_color
+    gsub(/\e\[(\d|;)+m/, "")
+  end
 end

data/lib/rspec_n/input.rb

@@ -3,11 +3,12 @@ module RspecN
     attr_accessor :iterations, :command, :stop_fast, :write_files
     def initialize(options, args)
       @args = args
+      @unprocessed_args_array = args.entries
       @options = options
-      validate_args
-      validate_order
       @iterations = determine_iterations
-      @order = options.fetch(:order, "rand")
+      @spec_path = determine_spec_path
+      validate_order
+      @order = @options.fetch(:order, "rand")
       @command = determine_command
       @stop_fast = options.fetch(:"stop-fast", false)
       @write_files = !options.fetch(:'no-file', false)
@@ -19,12 +20,6 @@ module RspecN
 
     private
 
-    def validate_args
-      return if @args.size.zero?
-      raise BadArgument, @args.join(', ') if @args.empty? || !@args.first.all_digits?
-      raise BadArgument, @args.first if @args.first.to_i < 1
-    end
-
     def validate_order
       return unless (order = @options.fetch(:order, nil))
 
@@ -32,13 +27,24 @@ module RspecN
     end
 
     def determine_iterations
-      @args.empty? ? RspecN::DEFAULT_ITERATIONS : @args.first.to_i
+      value = @unprocessed_args_array.detect(&:all_digits?)
+
+      if value
+        @unprocessed_args_array.delete(value)
+        value.to_i
+      else
+        RspecN::DEFAULT_ITERATIONS
+      end
     end
 
-    def determine_command
-      return @options[:command] if @options.fetch(:command, nil)
+    def determine_spec_path
+      @unprocessed_args_array.empty? ? nil : @unprocessed_args_array.join(" ")
+    end
 
-      guessed_command + " --order " + @order
+    def determine_command
+      command = @options.fetch(:command, guessed_command)
+      command += " " + @spec_path if @spec_path
+      command + " --order " + @order
     end
 
     def guessed_command

data/lib/rspec_n/run.rb

@@ -1,18 +1,21 @@
 module RspecN
   class Run
-    attr_accessor :iteration, :start_time, :finish_time, :seed, :rspec_stdout, :rspec_stderr, :rspec_status, :duration_seconds, :status_string
+    attr_accessor :duration_seconds, :finish_time, :iteration, :result_count_string, :rspec_stdout, :rspec_stderr, :rspec_status,
+      :seed, :status_string, :start_time
 
     def initialize(iteration:)
-      @iteration = iteration
-      @start_time = nil
-      @finish_time = nil
       @duration_seconds = nil
-      @seed = nil
+      @finish_time = nil
+      @has_warnings = nil
+      @iteration = iteration
+      @result_count_string = nil
       @rspec_stdout = nil
       @rspec_stderr = nil
       @rspec_status = nil
-      @status_string = nil
+      @seed = nil
       @skipped = false
+      @status_string = nil
+      @start_time = nil
     end
 
     def start_clock
@@ -24,6 +27,8 @@ module RspecN
       finalize_duration_seconds
       finalize_seed
       finalize_status_string
+      finalize_result_count_string
+      finalize_has_warnings
     end
 
     def go(command)
@@ -38,12 +43,12 @@ module RspecN
       finish_time.strftime(format)
     end
 
-    def passed?
-      @status_string == "Pass"
+    def has_warnings?
+      @has_warnings
     end
 
-    def passed_with_warnings?
-      @status_string == "Pass with Warnings"
+    def passed?
+      @status_string == "Pass"
     end
 
     def skip
@@ -67,17 +72,28 @@ module RspecN
 
     def finalize_seed
       result = @rspec_stdout.match(/^Randomized with seed (\d*)/)
-      return if result.nil?  # A seed wasn't used
+      return if result.nil? # A seed wasn't used
 
       @seed = result.captures.first&.strip
     end
 
+    # rubocop:disable Style/NegatedIf
     def finalize_status_string
       return @status_string = "Skip" if skipped?
-      return @status_string = "Pass with Warnings" if @rspec_status.exitstatus.zero? && !@rspec_stderr.empty?
       return @status_string = "Pass" if @rspec_status.exitstatus.zero?
       return @status_string = "Fail" if !@rspec_status.exitstatus.zero?
+
       @status_string = "Undetermined"
     end
+    # rubocop:enable Style/NegatedIf
+
+    def finalize_result_count_string
+      result = @rspec_stdout.match(/\d*\s+examples?,\s+\d*\s+failures?(,\s+\d*\s+pending)*/)
+      @result_count_string = result ? result.to_s : ""
+    end
+
+    def finalize_has_warnings
+      @has_warnings = @rspec_status.exitstatus.zero? && !@rspec_stderr.empty?
+    end
   end
 end

data/lib/rspec_n/runner.rb

@@ -25,19 +25,15 @@ module RspecN
     end
 
     def total_passed
-      @runs.values.select { |run| run.passed? }.size
-    end
-
-    def total_passed_with_warnings
-      @runs.values.select { |run| run.passed_with_warnings? }.size
+      @runs.values.select(&:passed?).size
     end
 
     def total_failed
-      @runs.values.select { |run| run.failed? }.size
+      @runs.values.select(&:failed?).size
     end
 
     def total_skipped
-      @runs.values.select { |run| run.skipped? }.size
+      @runs.values.select(&:skipped?).size
     end
 
     private

data/lib/rspec_n/version.rb

@@ -1,3 +1,3 @@
 module RspecN
-  VERSION = "1.0.0".freeze
+  VERSION = "1.1.0".freeze
 end

data/rspec_n.gemspec

@@ -33,6 +33,7 @@ Gem::Specification.new do |spec|
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
+  spec.required_ruby_version = '>= 2.3.7'
 
   spec.add_development_dependency "bundler", "~> 2.0"
   spec.add_development_dependency "pry", "~> 0.11.2"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rspec_n
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - roberts1000
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-02-19 00:00:00.000000000 Z
+date: 2019-03-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -116,8 +116,6 @@ files:
 - exe/rspec_n
 - lib/rspec_n.rb
 - lib/rspec_n/constants.rb
-- lib/rspec_n/errors.rb
-- lib/rspec_n/errors/bad_argument.rb
 - lib/rspec_n/errors/bad_option.rb
 - lib/rspec_n/formatters/file_formatter.rb
 - lib/rspec_n/formatters/table_formatter.rb
@@ -144,7 +142,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 2.3.7
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
@@ -152,7 +150,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.7.8
+rubygems_version: 2.6.14
 signing_key: 
 specification_version: 4
 summary: A ruby gem that runs RSpec N times.

data/lib/rspec_n/errors.rb

@@ -1,2 +0,0 @@
-require_relative "errors/bad_argument"
-require_relative "errors/bad_option"

data/lib/rspec_n/errors/bad_argument.rb

@@ -1,18 +0,0 @@
-module RspecN
-  class BadArgument < StandardError
-    def initialize(msg="")
-      @details = msg
-      super
-    end
-
-    def message
-      "There was an error with the argument.  rspec_n only accepts a single argument, a number greater than 0, which \n" \
-      "specifies how times RSpec should run.  You entered:\n\n"\
-      "  #{@details}\n\n" \
-      "Here are some example ways to use rspec_n (some of these may not be valid for your particular test suite):\n\n" \
-      "  rspec_n 5\n" \
-      "  rspec_n 3 --command 'bundle exec rspec'\n" \
-      "  rspec_n --command 'bin/rails db:test:prepare && bundle exec rspec'"
-    end
-  end
-end