minitest-bisect 1.4.1 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: cff92ce857a2c5a2a5ec16125c68b77513490096eeb225d88c9a386e01c7970e
4
- data.tar.gz: e9683eb9d3f1960aa90f2558aa860eb372cfd1f12053536e651917e7a02fb256
3
+ metadata.gz: 895bdb11c49734d1219e519a88532a03221bd0b2ae25431cc3e35d552716f841
4
+ data.tar.gz: 661d13d6f64de3a029fec5b68d7b89a3843e5e628fe9f53191e20981ed5bd806
5
5
  SHA512:
6
- metadata.gz: 5b4867ee6ac8d70e7fc85d7a1e5982db15d161f8f5deac85d1def14bf0a8ffdfad921924a6d7fe5137ca0445a65908c4d00668456bcfcad6a965cd13e1437459
7
- data.tar.gz: a5548ed9cd15fb5174e2e8f8a51998818431603410aa95015357651b6394e9333e71cb41a278ae4f854c992bec7aa3a218a91227b7d6883d6d68d3e78e71615b
6
+ metadata.gz: a2d962388ce5dafc12591a3e5362d3e0be98b2e46f1bb6ce46a0b7897836ec517c4ad9f4ed07f159311cbfbd792f148fd6e8acfb88b8fec95ac770b47cce0fd5
7
+ data.tar.gz: 5b896b2f11f3a01e99106abde712ad9784d55e4b974a66d453331acfb09f0e68d2c41973dd643d3827866d5a649d756c2d7388226b5f669164a544ca7b52d74c
checksums.yaml.gz.sig CHANGED
Binary file
data.tar.gz.sig CHANGED
Binary file
data/History.rdoc CHANGED
@@ -1,3 +1,14 @@
1
+ === 1.5.0 / 2019-06-06
2
+
3
+ * 2 minor enhancements:
4
+
5
+ * Print out the culprit methods once done.
6
+ * Default to using -Itest:lib like minitest-sprint.
7
+
8
+ * 1 bug fix:
9
+
10
+ * Remove the server arguments from the final repro output.
11
+
1
12
  === 1.4.1 / 2019-05-26
2
13
 
3
14
  * 2 bug fixes:
data/README.rdoc CHANGED
@@ -30,13 +30,37 @@ Let's say you have a bunch of test files and they fail sometimes, but
30
30
  not consistently. You get a run that fails, so you record the
31
31
  randomization seed that minitest displays at the top of every run.
32
32
 
33
+ === Normally, it passes:
34
+
35
+ % minitest example
36
+ Run options: --seed 42
37
+
38
+ # Running:
39
+
40
+ ..............................................................................
41
+ ..............................................................................
42
+ ..............................................................................
43
+ ..............................................................................
44
+ ..............................................................................
45
+ ..............................................................................
46
+ ..............................................................................
47
+ ..............................................................................
48
+ ..............................................................................
49
+ ..............................................................................
50
+ ....................
51
+
52
+ Finished in 200.341744s, 3.6371 runs/s, 3.4230 assertions/s.
53
+
54
+ 800 runs, 798 assertions, 0 failures, 0 errors, 0 skips
55
+
33
56
  === Original Failure:
34
57
 
35
- Here's an example run that fails randomly:
58
+ But someone sees the failure either locally or on the CI. They record
59
+ the output and get the randomization seed that causes the test
60
+ ordering bug:
36
61
 
37
- $ rake
38
- ruby -I.:lib -e 'require "example/test_bad1.rb";require "example/test_bad2.rb";require "example/test_bad3.rb";require "example/test_bad4.rb";require "example/test_bad5.rb";require "example/test_bad6.rb";require "example/test_bad7.rb";require "example/test_bad8.rb"'
39
- Run options: --seed 3911
62
+ $ minitest example --seed 314
63
+ Run options: --seed 314
40
64
 
41
65
  # Running:
42
66
 
@@ -73,58 +97,50 @@ same order and reproduce every time.
73
97
  minitest_bisect will first minimize the number of files, then it will
74
98
  turn around and minimize the number of methods.
75
99
 
76
- $ minitest_bisect --seed 3911 example/test*.rb
77
- reproducing...
78
- reproduced
79
- # of culprit files: 4
80
- # of culprit files: 2
81
- # of culprit files: 2
82
- # of culprit files: 2
83
- # of culprit files: 2
100
+ % minitest_bisect example --seed 314
101
+ reproducing... in 203.83 sec
102
+ verifying... in 0.37 sec
103
+ # of culprit methods: 64 in 16.65 sec
104
+ # of culprit methods: 32 in 8.53 sec
105
+ # of culprit methods: 32 in 8.52 sec
106
+ # of culprit methods: 16 in 4.46 sec
107
+ # of culprit methods: 16 in 4.44 sec
108
+ # of culprit methods: 8 in 2.41 sec
109
+ # of culprit methods: 4 in 1.40 sec
110
+ # of culprit methods: 2 in 0.89 sec
111
+ # of culprit methods: 2 in 0.89 sec
112
+ # of culprit methods: 1 in 0.62 sec
113
+ # of culprit methods: 1 in 0.63 sec
84
114
 
85
- Minimal files found in 5 steps:
115
+ Minimal methods found in 11 steps:
86
116
 
87
- ruby -Ilib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad4.rb"' -- --seed 3911 -s 48222
117
+ Culprit methods: ["TestBad1#test_bad1_1"]
88
118
 
89
- reproducing...
90
- reproduced
91
- # of culprit methods: 64
92
- # of culprit methods: 64
93
- # of culprit methods: 32
94
- # of culprit methods: 16
95
- # of culprit methods: 8
96
- # of culprit methods: 8
97
- # of culprit methods: 4
98
- # of culprit methods: 2
99
- # of culprit methods: 2
100
- # of culprit methods: 1
101
-
102
- Minimal methods found in 10 steps:
103
-
104
- ruby -Ilib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad4.rb"' -- --seed 3911 -s 48222 -n '/^(?:TestBad1\#test_bad1_1|TestBad4\#test_bad4_4)$/'
119
+ ruby -Itest:lib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad2.rb" ; require "./example/test_bad3.rb" ; require "./example/test_bad4.rb" ; require "./example/test_bad5.rb" ; require "./example/test_bad6.rb" ; require "./example/test_bad7.rb" ; require "./example/test_bad8.rb"' -- --seed 314 -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
105
120
 
106
121
  Final reproduction:
107
122
 
108
- Run options: --seed 3911 -s 48222 -n "/^(?:TestBad1\\#test_bad1_1|TestBad4\\#test_bad4_4)$/"
123
+ Run options: --seed 314 -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
109
124
 
110
125
  # Running:
111
126
 
112
127
  .F
113
128
 
114
- Finished in 0.505776s, 3.9543 runs/s, 1.9772 assertions/s.
129
+ Finished in 0.512999s, 3.8986 runs/s, 1.9493 assertions/s.
115
130
 
116
131
  1) Failure:
117
132
  TestBad4#test_bad4_4 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example/helper.rb:16]:
118
133
  muahahaha order dependency bug!
119
134
 
120
- 2 runs, 1 assertions, 1 failures, 0 errors, 0 skips
121
-
122
135
  Voila! This reduced it from 800 tests across 8 files down to 2 tests
123
- across 2 files. Note how we went from a 200 second test run to a 0.5
124
- second test run. Debugging that will be much easier.
136
+ across 2 files. Note how we went from a ~200 second test run to a ~0.5
137
+ second test run. Debugging that will be 400x faster and that much
138
+ easier.
139
+
140
+ === What Now?
125
141
 
126
- It is now up to you to look at the source of both of those tests to
127
- determine what side-effects (or lack thereof) are causing your test
142
+ Now, it is now up to you to look at the source of both of those tests
143
+ to determine what side-effects (or lack thereof) are causing your test
128
144
  failure when run in this specific order.
129
145
 
130
146
  This happens in a single run. Depending on how many files / tests you
@@ -14,7 +14,7 @@ def create_test suffix, n_methods, bad_methods = {}
14
14
  case bad_methods[n]
15
15
  when true then
16
16
  $hosed += 1
17
- when Fixnum then
17
+ when Integer then
18
18
  flunk "muahahaha order dependency bug!" if $hosed >= bad_methods[n]
19
19
  else
20
20
  assert true
@@ -5,7 +5,7 @@ require "rbconfig"
5
5
  require "path_expander"
6
6
 
7
7
  class Minitest::Bisect
8
- VERSION = "1.4.1"
8
+ VERSION = "1.5.0"
9
9
 
10
10
  class PathExpander < ::PathExpander
11
11
  TEST_GLOB = "**/{test_*,*_test,spec_*,*_spec}.rb" # :nodoc:
@@ -14,7 +14,7 @@ class Minitest::Bisect
14
14
 
15
15
  def initialize args = ARGV # :nodoc:
16
16
  super args, TEST_GLOB
17
- self.rb_flags = []
17
+ self.rb_flags = %w[-Itest:lib]
18
18
  end
19
19
 
20
20
  ##
@@ -162,8 +162,10 @@ class Minitest::Bisect
162
162
  puts
163
163
  puts "Minimal methods found in #{count} steps:"
164
164
  puts
165
+ puts "Culprit methods: %p" % [found]
166
+ puts
165
167
  cmd = build_methods_cmd cmd, found, bad
166
- puts cmd
168
+ puts cmd.sub(/--server \d+/, "")
167
169
  puts
168
170
  cmd
169
171
  end
@@ -220,7 +220,7 @@ class TestMinitest::TestBisect::TestPathExpander < Minitest::Test
220
220
  expander = mtbpe.new args
221
221
 
222
222
  assert_kind_of PathExpander, expander
223
- assert_equal [], expander.rb_flags
223
+ assert_equal %w[-Itest:lib], expander.rb_flags
224
224
  assert_same mtbpe::TEST_GLOB, expander.glob
225
225
  end
226
226
 
@@ -230,7 +230,7 @@ class TestMinitest::TestBisect::TestPathExpander < Minitest::Test
230
230
  expander = Minitest::Bisect::PathExpander.new args
231
231
 
232
232
  exp_files = %w[1 2 3 4 5 6]
233
- exp_flags = %w[-Iblah -d -w]
233
+ exp_flags = %w[-Itest:lib -Iblah -d -w]
234
234
 
235
235
  files = expander.process_flags(args)
236
236
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: minitest-bisect
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.4.1
4
+ version: 1.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ryan Davis
@@ -29,7 +29,7 @@ cert_chain:
29
29
  Em82dBUFsipwMLCYj39kcyHWAxyl6Ae1Cn9r/ItVBCxoeFdrHjfavnrIEoXUt4bU
30
30
  UfBugfLD19bu3nvL+zTAGx/U
31
31
  -----END CERTIFICATE-----
32
- date: 2019-05-27 00:00:00.000000000 Z
32
+ date: 2019-06-06 00:00:00.000000000 Z
33
33
  dependencies:
34
34
  - !ruby/object:Gem::Dependency
35
35
  name: minitest-server
@@ -99,14 +99,14 @@ dependencies:
99
99
  requirements:
100
100
  - - "~>"
101
101
  - !ruby/object:Gem::Version
102
- version: '3.17'
102
+ version: '3.18'
103
103
  type: :development
104
104
  prerelease: false
105
105
  version_requirements: !ruby/object:Gem::Requirement
106
106
  requirements:
107
107
  - - "~>"
108
108
  - !ruby/object:Gem::Version
109
- version: '3.17'
109
+ version: '3.18'
110
110
  description: |-
111
111
  Hunting down random test failures can be very very difficult,
112
112
  sometimes impossible, but minitest-bisect makes it easy.
@@ -178,7 +178,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
178
178
  - !ruby/object:Gem::Version
179
179
  version: '0'
180
180
  requirements: []
181
- rubygems_version: 3.0.2
181
+ rubygems_version: 3.0.3
182
182
  signing_key:
183
183
  specification_version: 4
184
184
  summary: Hunting down random test failures can be very very difficult, sometimes impossible,
metadata.gz.sig CHANGED
Binary file