minitest-suite 0.0.1 → 0.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +5 -0
- data/Gemfile.lock +1 -1
- data/README.md +62 -8
- data/example/Gemfile.lock +1 -1
- data/example/test/suite_filter_test.rb +65 -0
- data/example/test/suite_fixed_order_test.rb +113 -0
- data/example/test/{sweet_test.rb → suite_shuffle_test.rb} +4 -24
- data/lib/minitest/suite.rb +54 -15
- data/lib/minitest/suite/version.rb +1 -1
- data/script/test +19 -1
- metadata +5 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f4c6431daa7d510ea5146c3b8b417c77847f7f3f5d5a64b331248b8e3299e8e5
|
4
|
+
data.tar.gz: e8c90ad9b02cda9467c7eee661ce47d539e2b055be68219e09965b23890238ae
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 9554fb2541a0be0b30c454841a667640194ee5c0f1ce9e53f1c4a5c5bc0ea0c37fb6d39e9ccc9803c8161aed0651b4833ce23c26f39c1ca6916cf7a101f220ed
|
7
|
+
data.tar.gz: 419c7a9f46e85b059dffcefea7170527c274fdcf38533f0d41b197480284253390239ee9d7d076223bdb46f763281e70a25c9daa2e5331455b666726adf8ca12
|
data/CHANGELOG.md
CHANGED
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -5,8 +5,8 @@ The minitest-suite gem modifies the way
|
|
5
5
|
`Minitest::Test` subclasses are run by allowing you to organize them into
|
6
6
|
sub-suites or groups.
|
7
7
|
|
8
|
-
This can be handy when you want to (usually for performance reasons)
|
9
|
-
one logical grouping of tests at a time before moving onto the next.
|
8
|
+
This can be handy when you want to (usually for performance reasons) run
|
9
|
+
one logical grouping of tests at a time before moving onto the next group.
|
10
10
|
|
11
11
|
## Using minitest-suite
|
12
12
|
|
@@ -26,8 +26,8 @@ require "minitest/suite"
|
|
26
26
|
|
27
27
|
### Declaring suites
|
28
28
|
|
29
|
-
|
30
|
-
test
|
29
|
+
Just pass a suite name to `suite` in the class body of each
|
30
|
+
test to group that test with others of the same named suite:
|
31
31
|
|
32
32
|
```ruby
|
33
33
|
class SweetTest < Minitest::Test
|
@@ -47,9 +47,9 @@ of those suites will be shuffled.
|
|
47
47
|
For example, suppose you have 4 test classes, two fruits and two vegetables:
|
48
48
|
|
49
49
|
```ruby
|
50
|
-
class
|
50
|
+
class Broccoli < Minitest::Test
|
51
51
|
suite :veggie
|
52
|
-
def test_it() = puts("
|
52
|
+
def test_it() = puts("🥦")
|
53
53
|
end
|
54
54
|
|
55
55
|
class Pear < Minitest::Test
|
@@ -77,17 +77,71 @@ the order will still be randomized):
|
|
77
77
|
|
78
78
|
🍐
|
79
79
|
.🍎
|
80
|
-
|
80
|
+
.🥦
|
81
81
|
.🌶
|
82
82
|
.
|
83
83
|
```
|
84
84
|
|
85
85
|
To wit, the above strategy will ensure you'd never see this test order:
|
86
|
-
|
86
|
+
🍐,🌶,🍎,🥦.
|
87
87
|
|
88
88
|
Looking for more? Check out this repo's [example
|
89
89
|
test](/example/test/sweet_test.rb).
|
90
90
|
|
91
|
+
### Configuration
|
92
|
+
|
93
|
+
Since you're going to the trouble of organizing your tests into logical suites,
|
94
|
+
you may as well have a little additional control over which suites run and in
|
95
|
+
what order. Below are a few handy things you can do once you're set up.
|
96
|
+
|
97
|
+
#### Fix the ordering that suites run in
|
98
|
+
|
99
|
+
A very sensible strategy is to optimize the speed of your feedback loop by run
|
100
|
+
your fastest tests first so your tests can fail fast.
|
101
|
+
|
102
|
+
As a typical example, suppose you want to run your pure Ruby unit tests, then
|
103
|
+
your Rails model tests, then your other (presumably slower) integration tests.
|
104
|
+
Near the top of your test helper, before your tests have started running, set
|
105
|
+
the order like this:
|
106
|
+
|
107
|
+
```ruby
|
108
|
+
Minitest::Suite.order = [:unit, :model]
|
109
|
+
```
|
110
|
+
|
111
|
+
With this set, any test classes that call `suite :unit` will be shuffled and run
|
112
|
+
first, then any tests with `suite :model`, and then the rest of your suites and
|
113
|
+
tests.
|
114
|
+
|
115
|
+
(Fail-fast behavior is available for Minitest via the [minitest-fail-fast
|
116
|
+
gem](https://github.com/teoljungberg/minitest-fail-fast) or the [Rails test
|
117
|
+
runner](https://guides.rubyonrails.org/testing.html#the-rails-test-runner)'s
|
118
|
+
`-f` flag.)
|
119
|
+
|
120
|
+
#### Filter to run only certain test suites
|
121
|
+
|
122
|
+
If you want to run only tests belonging to a certain suite or set of suites,
|
123
|
+
just set the environment variable `MINITEST_SUITE_ONLY` to a comma-delimited
|
124
|
+
string of suite names to run:
|
125
|
+
|
126
|
+
```
|
127
|
+
$ MINITEST_SUITE_ONLY="unit,model" bin/rake test
|
128
|
+
```
|
129
|
+
|
130
|
+
When using this option, note that test classes that _don't_ call `suite` **will
|
131
|
+
not be run**.
|
132
|
+
|
133
|
+
#### Run all tests except certain suites
|
134
|
+
|
135
|
+
If there's a suite of tests you don't want to run, set the environment variable
|
136
|
+
`MINITEST_SUITE_EXCEPT` to a comma-delimited string of suite names to skip:
|
137
|
+
|
138
|
+
```
|
139
|
+
$ MINITEST_SUITE_EXCEPT="integration,browser" bin/rake test
|
140
|
+
```
|
141
|
+
|
142
|
+
When using this option, note that test classes that _don't_ call `suite` **will
|
143
|
+
be run**.
|
144
|
+
|
91
145
|
## Code of Conduct
|
92
146
|
|
93
147
|
This project follows Test Double's [code of
|
data/example/Gemfile.lock
CHANGED
@@ -0,0 +1,65 @@
|
|
1
|
+
def assert_good_only_run(suites, bad)
|
2
|
+
unless bad.empty?
|
3
|
+
raise <<~MSG
|
4
|
+
Expected to only run #{suites.inspect}, but these suites ran and shouldn't have:
|
5
|
+
|
6
|
+
#{bad.join("\n")}
|
7
|
+
MSG
|
8
|
+
end
|
9
|
+
end
|
10
|
+
|
11
|
+
module Exe
|
12
|
+
def self.cution(exe)
|
13
|
+
@executions ||= []
|
14
|
+
@executions << exe
|
15
|
+
end
|
16
|
+
|
17
|
+
def self.cutions
|
18
|
+
@executions || []
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
at_exit do
|
23
|
+
if (setting = ENV["MINITEST_SUITE_ONLY"])
|
24
|
+
suites = setting.gsub(/\s+/, "").split(",").map(&:to_sym)
|
25
|
+
bad = Exe.cutions.reject { |exe| suites.include?(exe) }
|
26
|
+
assert_good_only_run(suites, bad)
|
27
|
+
elsif (setting = ENV["MINITEST_SUITE_EXCEPT"])
|
28
|
+
suites = setting.gsub(/\s+/, "").split(",").map(&:to_sym)
|
29
|
+
bad = Exe.cutions.select { |exe| suites.include?(exe) }
|
30
|
+
assert_good_only_run(suites, bad)
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
require "minitest/suite"
|
35
|
+
require "minitest/autorun"
|
36
|
+
|
37
|
+
class Test1 < Minitest::Test
|
38
|
+
suite :suite1
|
39
|
+
|
40
|
+
def test_it
|
41
|
+
Exe.cution :suite1
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
class Test2 < Minitest::Test
|
46
|
+
suite :suite2
|
47
|
+
|
48
|
+
def test_it
|
49
|
+
Exe.cution :suite2
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
class Test3 < Minitest::Test
|
54
|
+
suite :suite3
|
55
|
+
|
56
|
+
def test_it
|
57
|
+
Exe.cution :suite3
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
class Test4 < Minitest::Test
|
62
|
+
def test_it
|
63
|
+
Exe.cution :unsuitened
|
64
|
+
end
|
65
|
+
end
|
@@ -0,0 +1,113 @@
|
|
1
|
+
module Exe
|
2
|
+
def self.cution(exe)
|
3
|
+
@executions ||= []
|
4
|
+
@executions << exe
|
5
|
+
end
|
6
|
+
|
7
|
+
def self.cutions
|
8
|
+
@executions || []
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
at_exit do
|
13
|
+
expected = Minitest::Suite.order
|
14
|
+
expected_index = 0
|
15
|
+
|
16
|
+
Exe.cutions.each.with_index do |suite, actual_index|
|
17
|
+
finished_suites = expected_index == 0 ? [] : expected[0..(expected_index - 1)]
|
18
|
+
|
19
|
+
if finished_suites.include?(suite)
|
20
|
+
raise <<~MSG
|
21
|
+
Ensuring order #{expected.inspect} but #{suite.inspect} test was just run
|
22
|
+
out of order. Actual execution order:
|
23
|
+
|
24
|
+
#{Exe.cutions.map(&:inspect).join("\n")}
|
25
|
+
MSG
|
26
|
+
elsif suite == expected[expected_index]
|
27
|
+
# cool
|
28
|
+
else
|
29
|
+
expected_index += 1
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
require "minitest/suite"
|
35
|
+
require "minitest/autorun"
|
36
|
+
|
37
|
+
Minitest::Suite.order = ENV["ORDER"].split(",").map(&:to_sym)
|
38
|
+
|
39
|
+
class Test1 < Minitest::Test
|
40
|
+
suite :suite1
|
41
|
+
|
42
|
+
def test_it
|
43
|
+
Exe.cution :suite1
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
47
|
+
class Test2 < Minitest::Test
|
48
|
+
suite :suite2
|
49
|
+
|
50
|
+
def test_it
|
51
|
+
Exe.cution :suite2
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
class Test3 < Minitest::Test
|
56
|
+
suite :suite3
|
57
|
+
|
58
|
+
def test_it
|
59
|
+
Exe.cution :suite3
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
class Test4 < Minitest::Test
|
64
|
+
def test_it
|
65
|
+
Exe.cution :unsuitened
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
class Test5 < Minitest::Test
|
70
|
+
suite :suite1
|
71
|
+
|
72
|
+
def test_it
|
73
|
+
Exe.cution :suite1
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
class Test6 < Minitest::Test
|
78
|
+
suite :suite2
|
79
|
+
|
80
|
+
def test_it
|
81
|
+
Exe.cution :suite2
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
class Test7 < Minitest::Test
|
86
|
+
suite :suite3
|
87
|
+
|
88
|
+
def test_it
|
89
|
+
Exe.cution :suite3
|
90
|
+
end
|
91
|
+
end
|
92
|
+
|
93
|
+
class Test8 < Minitest::Test
|
94
|
+
def test_it
|
95
|
+
Exe.cution :unsuitened
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
class Test9 < Minitest::Test
|
100
|
+
suite :suite1
|
101
|
+
|
102
|
+
def test_it
|
103
|
+
Exe.cution :suite1
|
104
|
+
end
|
105
|
+
end
|
106
|
+
|
107
|
+
class Test10 < Minitest::Test
|
108
|
+
suite :suite2
|
109
|
+
|
110
|
+
def test_it
|
111
|
+
Exe.cution :suite2
|
112
|
+
end
|
113
|
+
end
|
@@ -15,10 +15,10 @@ at_exit do
|
|
15
15
|
raise "Expected each suite to be run contiguously, but was not. Actual suite execution order was: #{suite_order}"
|
16
16
|
end
|
17
17
|
|
18
|
-
puts "Run order:"
|
19
|
-
executions.each.with_index do |execution, i|
|
20
|
-
|
21
|
-
end
|
18
|
+
# puts "Run order:"
|
19
|
+
# executions.each.with_index do |execution, i|
|
20
|
+
# puts "#{i + 2}. #{execution[:suite]}: #{execution[:class]}##{execution[:method]}"
|
21
|
+
# end
|
22
22
|
end
|
23
23
|
|
24
24
|
require "minitest/suite"
|
@@ -52,23 +52,3 @@ CLASS_COUNT.times do |i|
|
|
52
52
|
end
|
53
53
|
})
|
54
54
|
end
|
55
|
-
|
56
|
-
class Pumpkin < Minitest::Test
|
57
|
-
suite :veggie
|
58
|
-
def test_it() = puts("🎃")
|
59
|
-
end
|
60
|
-
|
61
|
-
class Pear < Minitest::Test
|
62
|
-
suite :fruit
|
63
|
-
def test_it() = puts("🍐")
|
64
|
-
end
|
65
|
-
|
66
|
-
class Pepper < Minitest::Test
|
67
|
-
suite :veggie
|
68
|
-
def test_it() = puts("🌶")
|
69
|
-
end
|
70
|
-
|
71
|
-
class Apple < Minitest::Test
|
72
|
-
suite :fruit
|
73
|
-
def test_it() = puts("🍎")
|
74
|
-
end
|
data/lib/minitest/suite.rb
CHANGED
@@ -1,35 +1,69 @@
|
|
1
|
-
require_relative "suite/version"
|
2
1
|
require "minitest"
|
2
|
+
require_relative "suite/version"
|
3
3
|
|
4
4
|
module Minitest
|
5
5
|
module Suite
|
6
6
|
class Error < StandardError; end
|
7
|
+
Registration = Struct.new(:suite, :test, keyword_init: true)
|
7
8
|
|
8
9
|
def self.register(suite_name:, test_class:)
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
raise Error.new("
|
10
|
+
if !suite_name.is_a?(Symbol)
|
11
|
+
raise Error.new("suite_name must be a Symbol")
|
12
|
+
elsif !test_class.ancestors.include?(Minitest::Test)
|
13
|
+
raise Error.new("test_class must be a Minitest::Test")
|
14
|
+
elsif (conflict = registrations.find { |r| r.test == test_class && r.suite != suite_name })
|
15
|
+
raise Error.new("#{conflict.test.name || "Class"} is already registered to the #{conflict.suite.inspect} suite")
|
16
|
+
elsif registrations.none? { |r| r.test == test_class && r.suite == suite_name }
|
17
|
+
registrations << Registration.new(suite: suite_name, test: test_class)
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def self.order=(suite_order)
|
22
|
+
if !suite_order.is_a?(Array) ||
|
23
|
+
suite_order.any? { |suite| !suite.is_a?(Symbol) }
|
24
|
+
raise Error.new("Minitest::Suite.order must be an array of Symbol suite names")
|
25
|
+
else
|
26
|
+
@@suite_order = suite_order.uniq
|
13
27
|
end
|
14
|
-
suites[suite_name] = (suites[suite_name] + [test_class]).uniq
|
15
28
|
end
|
16
29
|
|
17
|
-
def self.
|
18
|
-
|
30
|
+
def self.order
|
31
|
+
@@suite_order ||= []
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.registrations
|
35
|
+
@@registrations ||= reset
|
19
36
|
end
|
20
37
|
|
21
38
|
def self.reset
|
22
|
-
|
39
|
+
@@registrations = []
|
40
|
+
end
|
41
|
+
|
42
|
+
def self.filter_runnables(runnables)
|
43
|
+
if (only = ENV["MINITEST_SUITE_ONLY"])
|
44
|
+
suites = only.gsub(/\s+/, "").split(",").map(&:to_sym)
|
45
|
+
selected_tests = registrations.select { |r| suites.include?(r.suite) }
|
46
|
+
.map(&:test)
|
47
|
+
runnables.select { |r| selected_tests.include?(r) }
|
48
|
+
elsif (except = ENV["MINITEST_SUITE_EXCEPT"])
|
49
|
+
suites = except.gsub(/\s+/, "").split(",").map(&:to_sym)
|
50
|
+
excepted_tests = registrations.select { |r| suites.include?(r.suite) }
|
51
|
+
.map(&:test)
|
52
|
+
runnables.reject { |r| excepted_tests.include?(r) }
|
53
|
+
else
|
54
|
+
runnables
|
55
|
+
end
|
23
56
|
end
|
24
57
|
|
25
58
|
class PartialArrayProxy < Array
|
26
59
|
def shuffle
|
27
|
-
|
60
|
+
filtered = Suite.registrations.select { |r| include?(r.test) }
|
61
|
+
suites = Suite.order | (filtered.map(&:suite).uniq + [:__unsuitened]).shuffle
|
28
62
|
suites.flat_map { |suite|
|
29
|
-
if suite == :
|
30
|
-
(self -
|
63
|
+
if suite == :__unsuitened
|
64
|
+
(self - filtered.map(&:test)).shuffle
|
31
65
|
else
|
32
|
-
|
66
|
+
filtered.select { |r| r.suite == suite }.map(&:test).shuffle
|
33
67
|
end
|
34
68
|
}
|
35
69
|
end
|
@@ -52,8 +86,13 @@ module Minitest
|
|
52
86
|
class << self
|
53
87
|
undef_method :runnables
|
54
88
|
define_method :runnables do
|
55
|
-
|
56
|
-
@@runnables
|
89
|
+
filtered = Minitest::Suite.filter_runnables(@@runnables)
|
90
|
+
if @@runnables.is_a?(Minitest::Suite::PartialArrayProxy) &&
|
91
|
+
filtered == @runnables
|
92
|
+
@@runnables
|
93
|
+
else
|
94
|
+
@@runnables = Minitest::Suite::PartialArrayProxy.new(filtered)
|
95
|
+
end
|
57
96
|
end
|
58
97
|
end
|
59
98
|
end
|
data/script/test
CHANGED
@@ -3,5 +3,23 @@
|
|
3
3
|
set -ex
|
4
4
|
|
5
5
|
bundle exec rake
|
6
|
+
|
6
7
|
cd example
|
7
|
-
|
8
|
+
|
9
|
+
bundle exec ruby test/suite_shuffle_test.rb
|
10
|
+
|
11
|
+
bundle exec ruby test/suite_filter_test.rb
|
12
|
+
MINITEST_SUITE_ONLY="suite1" bundle exec ruby test/suite_filter_test.rb
|
13
|
+
MINITEST_SUITE_ONLY="suite1, suite2" bundle exec ruby test/suite_filter_test.rb
|
14
|
+
MINITEST_SUITE_ONLY="suite1,suite2,suite3" bundle exec ruby test/suite_filter_test.rb
|
15
|
+
MINITEST_SUITE_ONLY="nonsense" bundle exec ruby test/suite_filter_test.rb
|
16
|
+
MINITEST_SUITE_EXCEPT="suite1" bundle exec ruby test/suite_filter_test.rb
|
17
|
+
MINITEST_SUITE_EXCEPT="suite1,suite3" bundle exec ruby test/suite_filter_test.rb
|
18
|
+
MINITEST_SUITE_EXCEPT="suite1,suite2,suite3" bundle exec ruby test/suite_filter_test.rb
|
19
|
+
MINITEST_SUITE_EXCEPT="nonsense" bundle exec ruby test/suite_filter_test.rb
|
20
|
+
|
21
|
+
ORDER="" bundle exec ruby test/suite_fixed_order_test.rb
|
22
|
+
ORDER="suite1" bundle exec ruby test/suite_fixed_order_test.rb
|
23
|
+
ORDER="suite2,suite3" bundle exec ruby test/suite_fixed_order_test.rb
|
24
|
+
ORDER="suite3,suite2,suite1" bundle exec ruby test/suite_fixed_order_test.rb
|
25
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: minitest-suite
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Justin Searls
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-08-01 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: minitest
|
@@ -44,7 +44,9 @@ files:
|
|
44
44
|
- example/Gemfile
|
45
45
|
- example/Gemfile.lock
|
46
46
|
- example/Rakefile
|
47
|
-
- example/test/
|
47
|
+
- example/test/suite_filter_test.rb
|
48
|
+
- example/test/suite_fixed_order_test.rb
|
49
|
+
- example/test/suite_shuffle_test.rb
|
48
50
|
- lib/minitest/suite.rb
|
49
51
|
- lib/minitest/suite/version.rb
|
50
52
|
- minitest-suite.gemspec
|