rspec_n 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +13 -0
- data/.rspec +3 -0
- data/.rubocop.yml +5 -0
- data/.travis.yml +7 -0
- data/CHANGELOG.md +8 -0
- data/Gemfile +8 -0
- data/LICENSE.md +21 -0
- data/README.md +71 -0
- data/Rakefile +11 -0
- data/bin/console +13 -0
- data/bin/setup +8 -0
- data/exe/rspec_n +74 -0
- data/lib/rspec_n/constants.rb +9 -0
- data/lib/rspec_n/errors/bad_argument.rb +18 -0
- data/lib/rspec_n/errors/bad_option.rb +13 -0
- data/lib/rspec_n/errors.rb +2 -0
- data/lib/rspec_n/formatters/file_formatter.rb +35 -0
- data/lib/rspec_n/formatters/table_formatter.rb +93 -0
- data/lib/rspec_n/helpers/core_ext/array.rb +14 -0
- data/lib/rspec_n/helpers/core_ext/string.rb +6 -0
- data/lib/rspec_n/helpers/time_helpers.rb +14 -0
- data/lib/rspec_n/input.rb +61 -0
- data/lib/rspec_n/run.rb +83 -0
- data/lib/rspec_n/runner.rb +71 -0
- data/lib/rspec_n/version.rb +3 -0
- data/lib/rspec_n.rb +28 -0
- data/rspec_n.gemspec +45 -0
- metadata +159 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 779f827fe4de372504a3fa7c5d4168914b4a0c3d51c05f7e441cf94ecd6953ba
|
4
|
+
data.tar.gz: 928f1f99ec404b2f3a65d07542521658dfb1296e0ed5e9e5623f256b22789db8
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: b5f002fde7c4a9ad096a17abfe21a836efa53a9ce42c3d8ad94de3df68213455555f2fe5e21bd808f3a883cfa8591f9401bb925adc3e3afe5e084ccd290b2280
|
7
|
+
data.tar.gz: 48060eb007f625e03ab00e498b2c5cdf2fd365b59fb658978549a527b438bc000f988c875ff960e4f5618dd4804627b06bf1cf9852fc5d4fe4e31b10c5dc4133
|
data/.gitignore
ADDED
data/.rspec
ADDED
data/.rubocop.yml
ADDED
data/.travis.yml
ADDED
data/CHANGELOG.md
ADDED
data/Gemfile
ADDED
data/LICENSE.md
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2019 Roberts
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,71 @@
|
|
1
|
+
# rspec_n
|
2
|
+
|
3
|
+
rspec_n is a Ruby gem that makes it easy to run a project's RSpec test suite N times. You can customize the command that is used to start RSpec, or let rspec_n guess the best command (based on the files in your project). rspec_n is useful for finding repeatability or flakiness issues in automated test suites.
|
4
|
+
|
5
|
+
![sample](https://user-images.githubusercontent.com/2053901/52986788-d0bb7c80-33c6-11e9-9f13-0e191bdd2bb3.png)
|
6
|
+
|
7
|
+
## Version Policy
|
8
|
+
|
9
|
+
Releases are versioned using [semver 2.0.0](https://semver.org/spec/v2.0.0.html).
|
10
|
+
|
11
|
+
## Installation
|
12
|
+
|
13
|
+
Install by executing
|
14
|
+
|
15
|
+
$ gem install rspec_n
|
16
|
+
|
17
|
+
The gem will install an exectuable called `rspec_n` on your system. You may also want to add `rspec_n_iteration.*` to your `.gitignore` to exclude the output generated by rspec_n from your project's repo.
|
18
|
+
|
19
|
+
## Usage
|
20
|
+
|
21
|
+
To run RSpec N times
|
22
|
+
|
23
|
+
$ rspec_n N
|
24
|
+
|
25
|
+
If your project has a `bin/start_rspec` file, rspec_n will invoke it to start RSpec (rspec_n assumes you have placed the necessary commands to cleanly start your test suite in that file). Otherwise, it will attempt to identify your project type and use best practices to start RSpec for your given project type. The following is a list of project types supported by rspec_n, and the commands used when that project type is selected:
|
26
|
+
|
27
|
+
1. Ruby on Rails - `DISABLE_DATABASE_ENVIRONMENT_CHECK=1 RAILS_ENV=test bundle exec rake db:drop db:create db:migrate && bundle exec rspec`.
|
28
|
+
|
29
|
+
If it cannot guess a project type, rspec_n will invoke `bundle exec rspec`. You can override this by specifying a command directly on the CLI.
|
30
|
+
|
31
|
+
#### Use Custom Command to Start RSpec
|
32
|
+
|
33
|
+
By default, rspec_n will inspect the files in your project and pick the best way to start RSpec. You can take control by using the `-c` option which lets you specify a custom command. The following example deletes the `tmp` folder before starting RSpec:
|
34
|
+
|
35
|
+
$ rspec_n 5 -c 'rm -rf tmp && bundle exec rspec'
|
36
|
+
|
37
|
+
You must wrap your entire command string in single or double quotes if it contains spaces (which it probably will). This command will be invoked each time rspec_n attempts to start RSpec.
|
38
|
+
|
39
|
+
#### Control spec Order
|
40
|
+
|
41
|
+
rspec_n provides three command line options for controlling the execution order of your specs.
|
42
|
+
|
43
|
+
1. `--order rand` - Passes `--order rand` to RSpec which causes RSpec to run your specs in a random order.
|
44
|
+
1. `--order defined` - Passes `--order defined` to RSpec which causes RSpec to run your specs in the order that they are loaded by RSpec.
|
45
|
+
1. `--order project` - Passes nothing to RSpec, which means the project configuration files will determine the order.
|
46
|
+
|
47
|
+
If you do nothing, rspec_n will use `--order rand`. This ensures the specs in your project are executed in random order for each iteration of RSpec. rspec_n doesn't supply a seed when it start RSpec using a random strategy; it lets RSpec choose the seed and reports the values in the results.
|
48
|
+
|
49
|
+
#### Control File Ouput
|
50
|
+
|
51
|
+
rspec_n writes output for each iteration in a sequence of files `rspec_n_iteration.1`, `rspec_n_iteration.2`, etc... This saves you from having to rerun your test suite when you find a particular seed that causes your test suite to fail. If you want to disable this, add the `--no-file` option to the command.
|
52
|
+
|
53
|
+
**Note:** rspec_n deletes all files matching `rspec_n_iteration.*` when it starts so you must move those files to another location if you want to save them.
|
54
|
+
|
55
|
+
#### Stop on First Failure
|
56
|
+
|
57
|
+
You can tell rspec_n to abort when an iteration fails by using the `-s` flag. Any remaining iterations will be skipped.
|
58
|
+
|
59
|
+
## Development
|
60
|
+
|
61
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for a Pry console, or `rake console` for an IRB console, that will allow you to experiment.
|
62
|
+
|
63
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
64
|
+
|
65
|
+
## Contributing
|
66
|
+
|
67
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/roberts1000/rspec_n.
|
68
|
+
|
69
|
+
## License
|
70
|
+
|
71
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "rspec_n"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
require "pry"
|
10
|
+
Pry.start
|
11
|
+
|
12
|
+
# require "irb"
|
13
|
+
# IRB.start(__FILE__)
|
data/bin/setup
ADDED
data/exe/rspec_n
ADDED
@@ -0,0 +1,74 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
# This script runs the rspec_n executable.
|
4
|
+
#
|
5
|
+
# EXECUTION
|
6
|
+
#
|
7
|
+
# rspec_n # Run the project's RSpec test suite 10 times.
|
8
|
+
# rspec_n N # Run the projects's test suite N times.
|
9
|
+
#
|
10
|
+
# EXIT STATUS
|
11
|
+
#
|
12
|
+
# See below for exit status codes and their meaning.
|
13
|
+
|
14
|
+
lib = File.expand_path('../lib', __dir__)
|
15
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
16
|
+
|
17
|
+
require "cri"
|
18
|
+
require "rspec_n/constants"
|
19
|
+
require "rspec_n/errors"
|
20
|
+
|
21
|
+
# rubocop:disable Metrics/BlockLength
|
22
|
+
command = Cri::Command.define do
|
23
|
+
usage 'rspec_n [iterations] [options]'
|
24
|
+
description "rspec_n is an executable installed by the rspec_n Ruby gem. It provides a " \
|
25
|
+
"way to re-run an RSpec test suite 'N' times, which is useful when determining " \
|
26
|
+
"if a test suite is consistent.\n\n" \
|
27
|
+
"STATUS CODES\n\n" \
|
28
|
+
"0 - rspec_n ran successfully \u00A0\n" \
|
29
|
+
"1 - problem with the commandline options \u00A0"
|
30
|
+
|
31
|
+
flag :h, :help, 'show help' do |_value, cmd|
|
32
|
+
puts cmd.help
|
33
|
+
exit 0
|
34
|
+
end
|
35
|
+
|
36
|
+
command_description = "By default, rspec_n will guess the best command to run RSpec; " \
|
37
|
+
"override it by setting a custom command; wrap the entire command in single or double quotes; " \
|
38
|
+
"join multiple commands with '&&'"
|
39
|
+
option :c, :command, command_description, argument: :required
|
40
|
+
|
41
|
+
flag nil, "no-file", "Do not write iteration output to files"
|
42
|
+
|
43
|
+
flag :s, "stop-fast", "Stop when an iteration reports a failure."
|
44
|
+
|
45
|
+
command_description =
|
46
|
+
"The order RSpec uses when running specs. Can be one of three values: \u00A0\n" \
|
47
|
+
"\u00A0\u00A0defined - Run specs in same order \u00A0\n" \
|
48
|
+
"\u00A0\u00A0project - Let project decide \u00A0\n" \
|
49
|
+
"\u00A0\u00A0rand - Randomize specs (default)"
|
50
|
+
option :o, :order, command_description, argument: :required
|
51
|
+
|
52
|
+
flag :v, :version, 'show the current version' do |_value, _cmd|
|
53
|
+
puts RspecN::VERSION
|
54
|
+
exit 0
|
55
|
+
end
|
56
|
+
|
57
|
+
run do |options, args, _cmd|
|
58
|
+
require "rspec_n"
|
59
|
+
RspecN::Runner.new(options, args).start
|
60
|
+
end
|
61
|
+
end
|
62
|
+
# rubocop:enable Metrics/BlockLength
|
63
|
+
|
64
|
+
begin
|
65
|
+
command.run(ARGV)
|
66
|
+
exit 0
|
67
|
+
# Raised when the user specifies an argument incorrectly.
|
68
|
+
rescue RspecN::BadArgument => e
|
69
|
+
warn e.message.colorize(:red)
|
70
|
+
exit 1
|
71
|
+
rescue RspecN::BadOption => e
|
72
|
+
warn e.message.colorize(:red)
|
73
|
+
exit 1
|
74
|
+
end
|
@@ -0,0 +1,9 @@
|
|
1
|
+
module RspecN
|
2
|
+
ALLOWED_ORDER_OPTIONS = %w[defined project rand].freeze
|
3
|
+
DEFAULT_ITERATIONS = 10
|
4
|
+
DEFAULT_COMMAND = 'bundle exec rspec'.freeze
|
5
|
+
DEFAULT_RSPEC_STARTER_COMMAND = 'bin/start_rspec'.freeze
|
6
|
+
# rubocop:disable Metrics/LineLength
|
7
|
+
DEFAULT_RAILS_COMMAND = 'DISABLE_DATABASE_ENVIRONMENT_CHECK=1 RAILS_ENV=test bundle exec rake db:drop db:create db:migrate && bundle exec rspec'.freeze
|
8
|
+
# rubocop:enable Metrics/LineLength
|
9
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
module RspecN
|
2
|
+
class BadArgument < StandardError
|
3
|
+
def initialize(msg="")
|
4
|
+
@details = msg
|
5
|
+
super
|
6
|
+
end
|
7
|
+
|
8
|
+
def message
|
9
|
+
"There was an error with the argument. rspec_n only accepts a single argument, a number greater than 0, which \n" \
|
10
|
+
"specifies how times RSpec should run. You entered:\n\n"\
|
11
|
+
" #{@details}\n\n" \
|
12
|
+
"Here are some example ways to use rspec_n (some of these may not be valid for your particular test suite):\n\n" \
|
13
|
+
" rspec_n 5\n" \
|
14
|
+
" rspec_n 3 --command 'bundle exec rspec'\n" \
|
15
|
+
" rspec_n --command 'bin/rails db:test:prepare && bundle exec rspec'"
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
module RspecN
|
2
|
+
class BadOption < StandardError
|
3
|
+
def initialize(msg="")
|
4
|
+
@details = msg
|
5
|
+
super
|
6
|
+
end
|
7
|
+
|
8
|
+
def message
|
9
|
+
allowed = RspecN::ALLOWED_ORDER_OPTIONS.collect { |val| "'" + val + "'" }
|
10
|
+
"Order must be #{allowed.to_sentence}. '#{@details}' is not allowed.\n"
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
module RspecN
|
2
|
+
module Formatters
|
3
|
+
class FileFormatter
|
4
|
+
include RspecN::TimeHelpers
|
5
|
+
|
6
|
+
BASE_FILE_NAME = "rspec_n_iteration"
|
7
|
+
|
8
|
+
def initialize(runner:)
|
9
|
+
@runner = runner
|
10
|
+
@format = "%m/%d %l:%M:%S %p"
|
11
|
+
delete_all_files
|
12
|
+
end
|
13
|
+
|
14
|
+
def delete_all_files
|
15
|
+
Dir.glob("#{BASE_FILE_NAME}.**").each { |file| File.delete(file)}
|
16
|
+
end
|
17
|
+
|
18
|
+
def write(run)
|
19
|
+
return if run.skipped?
|
20
|
+
return unless @runner.input.write_files?
|
21
|
+
|
22
|
+
file_name = "#{BASE_FILE_NAME}.#{run.iteration}"
|
23
|
+
|
24
|
+
File.open(file_name, "w") do |f|
|
25
|
+
f.write("Iteration: #{run.iteration}\n")
|
26
|
+
f.write("Start Time: #{run.formatted_start_time(@format)}\n")
|
27
|
+
f.write("Finish Time: #{run.formatted_finish_time(@format)}\n")
|
28
|
+
f.write("Duration: #{convert_seconds_to_hms(run.duration_seconds)}\n\n")
|
29
|
+
f.write(run.rspec_stdout)
|
30
|
+
f.write(run.rspec_stderr)
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
@@ -0,0 +1,93 @@
|
|
1
|
+
module RspecN
|
2
|
+
module Formatters
|
3
|
+
class TableFormatter
|
4
|
+
include RspecN::TimeHelpers
|
5
|
+
|
6
|
+
attr_accessor :columns
|
7
|
+
|
8
|
+
def initialize(runner:)
|
9
|
+
@runner = runner
|
10
|
+
@columns = { "Run" => 8, "Start Time" => 22, "Finish Time" => 22, "Duration" => 13, "Seed" => 11, "Result" => 18 }
|
11
|
+
@header_columns_string = padded_header_column_labels
|
12
|
+
@table_width = @header_columns_string.size
|
13
|
+
@format = "%m/%d %l:%M:%S %p"
|
14
|
+
end
|
15
|
+
|
16
|
+
def observe
|
17
|
+
write_table_header
|
18
|
+
yield
|
19
|
+
write_conclusion
|
20
|
+
end
|
21
|
+
|
22
|
+
def show_pre_run_info(run)
|
23
|
+
print pad_field("Run", run.iteration)
|
24
|
+
print pad_field("Start Time", run.formatted_start_time(@format))
|
25
|
+
end
|
26
|
+
|
27
|
+
def show_post_run_info(run)
|
28
|
+
print pad_field("Finish Time", run.formatted_finish_time(@format))
|
29
|
+
print duration_field(run)
|
30
|
+
print seed_field(run)
|
31
|
+
puts result_field(run)
|
32
|
+
end
|
33
|
+
|
34
|
+
private
|
35
|
+
|
36
|
+
def write_table_header
|
37
|
+
puts @header_columns_string
|
38
|
+
puts "-" * @table_width
|
39
|
+
end
|
40
|
+
|
41
|
+
def write_conclusion
|
42
|
+
puts "-" * @table_width
|
43
|
+
puts ""
|
44
|
+
puts "Total Duration: #{convert_seconds_to_hms(@runner.total_duration_seconds)}"
|
45
|
+
puts "Average Duration: #{convert_seconds_to_hms(@runner.avg_duration_seconds)}"
|
46
|
+
puts "Total Passed: #{@runner.total_passed.to_s.colorize(:green)}"
|
47
|
+
puts "Total Passed with Warnings: #{@runner.total_passed_with_warnings.to_s.colorize(:green)}"
|
48
|
+
puts "Total Failed: #{@runner.total_failed.to_s.colorize(:red)}"
|
49
|
+
puts "Total Skipped: #{@runner.total_skipped}"
|
50
|
+
puts ""
|
51
|
+
end
|
52
|
+
|
53
|
+
def padded_header_column_label(name, width)
|
54
|
+
right_padding = width - name.size
|
55
|
+
name + (" " * right_padding)
|
56
|
+
end
|
57
|
+
|
58
|
+
def padded_header_column_labels
|
59
|
+
columns.collect { |name, max_width| padded_header_column_label(name, max_width) }.join("")
|
60
|
+
end
|
61
|
+
|
62
|
+
def max_column_width_for(name)
|
63
|
+
columns[name]
|
64
|
+
end
|
65
|
+
|
66
|
+
def pad_field(column_name, value)
|
67
|
+
max_width = max_column_width_for(column_name)
|
68
|
+
value_size = value.to_s.size
|
69
|
+
pad_count = max_width - value_size
|
70
|
+
value.to_s + (" " * pad_count)
|
71
|
+
end
|
72
|
+
|
73
|
+
def duration_field(run)
|
74
|
+
hms = convert_seconds_to_hms(run.duration_seconds)
|
75
|
+
pad_field("Duration", hms)
|
76
|
+
end
|
77
|
+
|
78
|
+
def seed_field(run)
|
79
|
+
run.seed.nil? ? pad_field("Seed", "None") : pad_field("Seed", run.seed)
|
80
|
+
end
|
81
|
+
|
82
|
+
def result_field(run)
|
83
|
+
case run.status_string
|
84
|
+
when "Pass with Warnings" then "Pass with Warnings".colorize(:yellow)
|
85
|
+
when "Pass" then "Pass".colorize(:green)
|
86
|
+
when "Fail" then "Fail".colorize(:red)
|
87
|
+
when "Skip" then "Skip".colorize(:yellow)
|
88
|
+
else "Unknown".colorize(:yellow)
|
89
|
+
end
|
90
|
+
end
|
91
|
+
end
|
92
|
+
end
|
93
|
+
end
|
@@ -0,0 +1,14 @@
|
|
1
|
+
class Array
|
2
|
+
def to_sentence(word_connector: ", ", two_word_connector: " or ", last_word_connector: " or ")
|
3
|
+
case length
|
4
|
+
when 0
|
5
|
+
""
|
6
|
+
when 1
|
7
|
+
self[0].to_s.dup
|
8
|
+
when 2
|
9
|
+
"#{self[0]}#{two_word_connector}#{self[1]}"
|
10
|
+
else
|
11
|
+
"#{self[0...-1].join(word_connector)}#{last_word_connector}#{self[-1]}"
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
@@ -0,0 +1,14 @@
|
|
1
|
+
module RspecN
|
2
|
+
module TimeHelpers
|
3
|
+
def convert_seconds_to_hms(total_seconds)
|
4
|
+
hours = total_seconds / (60 * 60)
|
5
|
+
minutes = (total_seconds / 60) % 60
|
6
|
+
seconds = total_seconds % 60
|
7
|
+
[hours, minutes, seconds].map do |t|
|
8
|
+
# Right justify and pad with 0 until length is 2.
|
9
|
+
# So if the duration of any of the time components is 0, then it will display as 00
|
10
|
+
t.round.to_s.rjust(2, '0')
|
11
|
+
end.join(':')
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
@@ -0,0 +1,61 @@
|
|
1
|
+
module RspecN
|
2
|
+
class Input
|
3
|
+
attr_accessor :iterations, :command, :stop_fast, :write_files
|
4
|
+
def initialize(options, args)
|
5
|
+
@args = args
|
6
|
+
@options = options
|
7
|
+
validate_args
|
8
|
+
validate_order
|
9
|
+
@iterations = determine_iterations
|
10
|
+
@order = options.fetch(:order, "rand")
|
11
|
+
@command = determine_command
|
12
|
+
@stop_fast = options.fetch(:"stop-fast", false)
|
13
|
+
@write_files = !options.fetch(:'no-file', false)
|
14
|
+
end
|
15
|
+
|
16
|
+
def write_files?
|
17
|
+
@write_files
|
18
|
+
end
|
19
|
+
|
20
|
+
private
|
21
|
+
|
22
|
+
def validate_args
|
23
|
+
return if @args.size.zero?
|
24
|
+
raise BadArgument, @args.join(', ') if @args.empty? || !@args.first.all_digits?
|
25
|
+
raise BadArgument, @args.first if @args.first.to_i < 1
|
26
|
+
end
|
27
|
+
|
28
|
+
def validate_order
|
29
|
+
return unless (order = @options.fetch(:order, nil))
|
30
|
+
|
31
|
+
raise BadOption, order unless RspecN::ALLOWED_ORDER_OPTIONS.include?(order)
|
32
|
+
end
|
33
|
+
|
34
|
+
def determine_iterations
|
35
|
+
@args.empty? ? RspecN::DEFAULT_ITERATIONS : @args.first.to_i
|
36
|
+
end
|
37
|
+
|
38
|
+
def determine_command
|
39
|
+
return @options[:command] if @options.fetch(:command, nil)
|
40
|
+
|
41
|
+
guessed_command + " --order " + @order
|
42
|
+
end
|
43
|
+
|
44
|
+
def guessed_command
|
45
|
+
return RspecN::DEFAULT_RSPEC_STARTER_COMMAND if project_uses_rspec_starter?
|
46
|
+
return RspecN::DEFAULT_RAILS_COMMAND if project_is_rails_based?
|
47
|
+
|
48
|
+
RspecN::DEFAULT_COMMAND
|
49
|
+
end
|
50
|
+
|
51
|
+
def project_uses_rspec_starter?
|
52
|
+
app_file_name = "bin/start_rspec"
|
53
|
+
File.file?(app_file_name)
|
54
|
+
end
|
55
|
+
|
56
|
+
def project_is_rails_based?
|
57
|
+
app_file_name = "config/application.rb"
|
58
|
+
File.file?(app_file_name) && File.readlines(app_file_name).grep(/Rails::Application/).any?
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
data/lib/rspec_n/run.rb
ADDED
@@ -0,0 +1,83 @@
|
|
1
|
+
module RspecN
|
2
|
+
class Run
|
3
|
+
attr_accessor :iteration, :start_time, :finish_time, :seed, :rspec_stdout, :rspec_stderr, :rspec_status, :duration_seconds, :status_string
|
4
|
+
|
5
|
+
def initialize(iteration:)
|
6
|
+
@iteration = iteration
|
7
|
+
@start_time = nil
|
8
|
+
@finish_time = nil
|
9
|
+
@duration_seconds = nil
|
10
|
+
@seed = nil
|
11
|
+
@rspec_stdout = nil
|
12
|
+
@rspec_stderr = nil
|
13
|
+
@rspec_status = nil
|
14
|
+
@status_string = nil
|
15
|
+
@skipped = false
|
16
|
+
end
|
17
|
+
|
18
|
+
def start_clock
|
19
|
+
@start_time = Time.now
|
20
|
+
end
|
21
|
+
|
22
|
+
def stop_clock
|
23
|
+
@finish_time = Time.now
|
24
|
+
finalize_duration_seconds
|
25
|
+
finalize_seed
|
26
|
+
finalize_status_string
|
27
|
+
end
|
28
|
+
|
29
|
+
def go(command)
|
30
|
+
@rspec_stdout, @rspec_stderr, @rspec_status = Open3.capture3(command)
|
31
|
+
end
|
32
|
+
|
33
|
+
def formatted_start_time(format)
|
34
|
+
start_time.strftime(format)
|
35
|
+
end
|
36
|
+
|
37
|
+
def formatted_finish_time(format)
|
38
|
+
finish_time.strftime(format)
|
39
|
+
end
|
40
|
+
|
41
|
+
def passed?
|
42
|
+
@status_string == "Pass"
|
43
|
+
end
|
44
|
+
|
45
|
+
def passed_with_warnings?
|
46
|
+
@status_string == "Pass with Warnings"
|
47
|
+
end
|
48
|
+
|
49
|
+
def skip
|
50
|
+
@skipped = true
|
51
|
+
@duration_seconds = 0
|
52
|
+
end
|
53
|
+
|
54
|
+
def skipped?
|
55
|
+
@skipped
|
56
|
+
end
|
57
|
+
|
58
|
+
def failed?
|
59
|
+
@status_string == "Fail"
|
60
|
+
end
|
61
|
+
|
62
|
+
private
|
63
|
+
|
64
|
+
def finalize_duration_seconds
|
65
|
+
@duration_seconds = (@finish_time - @start_time).round
|
66
|
+
end
|
67
|
+
|
68
|
+
def finalize_seed
|
69
|
+
result = @rspec_stdout.match(/^Randomized with seed (\d*)/)
|
70
|
+
return if result.nil? # A seed wasn't used
|
71
|
+
|
72
|
+
@seed = result.captures.first&.strip
|
73
|
+
end
|
74
|
+
|
75
|
+
def finalize_status_string
|
76
|
+
return @status_string = "Skip" if skipped?
|
77
|
+
return @status_string = "Pass with Warnings" if @rspec_status.exitstatus.zero? && !@rspec_stderr.empty?
|
78
|
+
return @status_string = "Pass" if @rspec_status.exitstatus.zero?
|
79
|
+
return @status_string = "Fail" if !@rspec_status.exitstatus.zero?
|
80
|
+
@status_string = "Undetermined"
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
@@ -0,0 +1,71 @@
|
|
1
|
+
module RspecN
|
2
|
+
class Runner < Cri::CommandRunner
|
3
|
+
attr_reader :command, :input, :iterations, :runs
|
4
|
+
|
5
|
+
def initialize(options, args)
|
6
|
+
@input = Input.new(options, args)
|
7
|
+
@iterations = @input.iterations
|
8
|
+
@command = @input.command
|
9
|
+
@display_formatter = Formatters::TableFormatter.new(runner: self)
|
10
|
+
@file_formatter = Formatters::FileFormatter.new(runner: self)
|
11
|
+
initialize_runs
|
12
|
+
end
|
13
|
+
|
14
|
+
def start
|
15
|
+
display_intro
|
16
|
+
@display_formatter.observe { run_tests }
|
17
|
+
end
|
18
|
+
|
19
|
+
def total_duration_seconds
|
20
|
+
@runs.values.inject(0) { |sum, run| sum + run.duration_seconds }
|
21
|
+
end
|
22
|
+
|
23
|
+
def avg_duration_seconds
|
24
|
+
(total_duration_seconds / @iterations).floor
|
25
|
+
end
|
26
|
+
|
27
|
+
def total_passed
|
28
|
+
@runs.values.select { |run| run.passed? }.size
|
29
|
+
end
|
30
|
+
|
31
|
+
def total_passed_with_warnings
|
32
|
+
@runs.values.select { |run| run.passed_with_warnings? }.size
|
33
|
+
end
|
34
|
+
|
35
|
+
def total_failed
|
36
|
+
@runs.values.select { |run| run.failed? }.size
|
37
|
+
end
|
38
|
+
|
39
|
+
def total_skipped
|
40
|
+
@runs.values.select { |run| run.skipped? }.size
|
41
|
+
end
|
42
|
+
|
43
|
+
private
|
44
|
+
|
45
|
+
def initialize_runs
|
46
|
+
@runs = {}
|
47
|
+
@iterations.times { |i| @runs[i + 1] = Run.new(iteration: i + 1) }
|
48
|
+
end
|
49
|
+
|
50
|
+
def run_tests
|
51
|
+
found_failure = false
|
52
|
+
|
53
|
+
@runs.each do |_iteration, run|
|
54
|
+
next run.skip if @input.stop_fast && found_failure
|
55
|
+
|
56
|
+
run.start_clock
|
57
|
+
@display_formatter.show_pre_run_info(run)
|
58
|
+
run.go(@command)
|
59
|
+
run.stop_clock
|
60
|
+
@display_formatter.show_post_run_info(run)
|
61
|
+
found_failure ||= run.failed?
|
62
|
+
@file_formatter.write(run)
|
63
|
+
end
|
64
|
+
end
|
65
|
+
|
66
|
+
def display_intro
|
67
|
+
iteration_part = @iterations > 1 ? "#{@iterations} times" : "1 time"
|
68
|
+
puts "\nRSpec will execute #{iteration_part.colorize(:yellow)} using #{command.to_s.colorize(:yellow)}\n\n"
|
69
|
+
end
|
70
|
+
end
|
71
|
+
end
|
data/lib/rspec_n.rb
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
require "rspec_n/version"
|
2
|
+
|
3
|
+
require "colorize"
|
4
|
+
require "cri"
|
5
|
+
require "open3"
|
6
|
+
|
7
|
+
require "rspec_n/helpers/time_helpers"
|
8
|
+
require "rspec_n/helpers/core_ext/array"
|
9
|
+
require "rspec_n/helpers/core_ext/string"
|
10
|
+
|
11
|
+
require "rspec_n/errors"
|
12
|
+
|
13
|
+
require "rspec_n/constants"
|
14
|
+
require "rspec_n/input"
|
15
|
+
require "rspec_n/runner"
|
16
|
+
require "rspec_n/run"
|
17
|
+
require "rspec_n/formatters/file_formatter"
|
18
|
+
require "rspec_n/formatters/table_formatter"
|
19
|
+
|
20
|
+
# Setup pry for development when running "rake console". Guard against load
|
21
|
+
# errors in production (since pry is only loaded as a DEVELOPMENT dependency
|
22
|
+
# in the .gemspec)
|
23
|
+
# rubocop:disable Lint/HandleExceptions
|
24
|
+
begin
|
25
|
+
require "pry"
|
26
|
+
rescue LoadError
|
27
|
+
end
|
28
|
+
# rubocop:enable Lint/HandleExceptions
|
data/rspec_n.gemspec
ADDED
@@ -0,0 +1,45 @@
|
|
1
|
+
lib = File.expand_path("lib", __dir__)
|
2
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
3
|
+
require "rspec_n/version"
|
4
|
+
|
5
|
+
# rubocop:disable Metrics/BlockLength
|
6
|
+
Gem::Specification.new do |spec|
|
7
|
+
spec.name = "rspec_n"
|
8
|
+
spec.version = RspecN::VERSION
|
9
|
+
spec.authors = ["roberts1000"]
|
10
|
+
spec.email = ["roberts@corlewsolutions.com"]
|
11
|
+
|
12
|
+
spec.summary = "A ruby gem that runs RSpec N times."
|
13
|
+
spec.description = "A ruby gem that runs RSpec N times."
|
14
|
+
spec.homepage = "https://github.com/roberts1000/rspec_n"
|
15
|
+
spec.license = "MIT"
|
16
|
+
|
17
|
+
# Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
|
18
|
+
# to allow pushing to a single host or delete this section to allow pushing to any host.
|
19
|
+
if spec.respond_to?(:metadata)
|
20
|
+
spec.metadata["homepage_uri"] = spec.homepage
|
21
|
+
spec.metadata["source_code_uri"] = "https://github.com/roberts1000/rspec_n"
|
22
|
+
spec.metadata["changelog_uri"] = "https://github.com/roberts1000/rspec_n/blob/master/CHANGELOG.md"
|
23
|
+
else
|
24
|
+
raise "RubyGems 2.0 or newer is required to protect against " \
|
25
|
+
"public gem pushes."
|
26
|
+
end
|
27
|
+
|
28
|
+
# Specify which files should be added to the gem when it is released.
|
29
|
+
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
30
|
+
spec.files = Dir.chdir(File.expand_path(__dir__)) do
|
31
|
+
`git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
32
|
+
end
|
33
|
+
spec.bindir = "exe"
|
34
|
+
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
35
|
+
spec.require_paths = ["lib"]
|
36
|
+
|
37
|
+
spec.add_development_dependency "bundler", "~> 2.0"
|
38
|
+
spec.add_development_dependency "pry", "~> 0.11.2"
|
39
|
+
spec.add_development_dependency "rake", "~> 12.0"
|
40
|
+
spec.add_development_dependency "rspec", "~> 3.0"
|
41
|
+
|
42
|
+
spec.add_dependency "colorize", "~> 0.8.0"
|
43
|
+
spec.add_dependency "cri", "~> 2.15.3"
|
44
|
+
end
|
45
|
+
# rubocop:enable Metrics/BlockLength
|
metadata
ADDED
@@ -0,0 +1,159 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: rspec_n
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- roberts1000
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2019-02-19 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: bundler
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '2.0'
|
20
|
+
type: :development
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '2.0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: pry
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 0.11.2
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: 0.11.2
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: rake
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '12.0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '12.0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rspec
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '3.0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '3.0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: colorize
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - "~>"
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: 0.8.0
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - "~>"
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: 0.8.0
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: cri
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - "~>"
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: 2.15.3
|
90
|
+
type: :runtime
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - "~>"
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: 2.15.3
|
97
|
+
description: A ruby gem that runs RSpec N times.
|
98
|
+
email:
|
99
|
+
- roberts@corlewsolutions.com
|
100
|
+
executables:
|
101
|
+
- rspec_n
|
102
|
+
extensions: []
|
103
|
+
extra_rdoc_files: []
|
104
|
+
files:
|
105
|
+
- ".gitignore"
|
106
|
+
- ".rspec"
|
107
|
+
- ".rubocop.yml"
|
108
|
+
- ".travis.yml"
|
109
|
+
- CHANGELOG.md
|
110
|
+
- Gemfile
|
111
|
+
- LICENSE.md
|
112
|
+
- README.md
|
113
|
+
- Rakefile
|
114
|
+
- bin/console
|
115
|
+
- bin/setup
|
116
|
+
- exe/rspec_n
|
117
|
+
- lib/rspec_n.rb
|
118
|
+
- lib/rspec_n/constants.rb
|
119
|
+
- lib/rspec_n/errors.rb
|
120
|
+
- lib/rspec_n/errors/bad_argument.rb
|
121
|
+
- lib/rspec_n/errors/bad_option.rb
|
122
|
+
- lib/rspec_n/formatters/file_formatter.rb
|
123
|
+
- lib/rspec_n/formatters/table_formatter.rb
|
124
|
+
- lib/rspec_n/helpers/core_ext/array.rb
|
125
|
+
- lib/rspec_n/helpers/core_ext/string.rb
|
126
|
+
- lib/rspec_n/helpers/time_helpers.rb
|
127
|
+
- lib/rspec_n/input.rb
|
128
|
+
- lib/rspec_n/run.rb
|
129
|
+
- lib/rspec_n/runner.rb
|
130
|
+
- lib/rspec_n/version.rb
|
131
|
+
- rspec_n.gemspec
|
132
|
+
homepage: https://github.com/roberts1000/rspec_n
|
133
|
+
licenses:
|
134
|
+
- MIT
|
135
|
+
metadata:
|
136
|
+
homepage_uri: https://github.com/roberts1000/rspec_n
|
137
|
+
source_code_uri: https://github.com/roberts1000/rspec_n
|
138
|
+
changelog_uri: https://github.com/roberts1000/rspec_n/blob/master/CHANGELOG.md
|
139
|
+
post_install_message:
|
140
|
+
rdoc_options: []
|
141
|
+
require_paths:
|
142
|
+
- lib
|
143
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
144
|
+
requirements:
|
145
|
+
- - ">="
|
146
|
+
- !ruby/object:Gem::Version
|
147
|
+
version: '0'
|
148
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
149
|
+
requirements:
|
150
|
+
- - ">="
|
151
|
+
- !ruby/object:Gem::Version
|
152
|
+
version: '0'
|
153
|
+
requirements: []
|
154
|
+
rubyforge_project:
|
155
|
+
rubygems_version: 2.7.8
|
156
|
+
signing_key:
|
157
|
+
specification_version: 4
|
158
|
+
summary: A ruby gem that runs RSpec N times.
|
159
|
+
test_files: []
|