minitest-bisect 1.5.1 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '039c173b7cc6976666b776aaa73002f584a372dcc797362bfed5e7afcf9fe8c1'
-  data.tar.gz: f806ca88ff544455cd14e2ec0f3a7885d6ecb06d57e5b56e3a782a9fab69ad76
+  metadata.gz: 52bae214fa2f36196af3ec54139c2bec839de200e8dee1015f916b56a666a69b
+  data.tar.gz: 15a204ca83b0da8a8773e565f3bc40c439713edacef4a1d66435f2f9938edf6e
 SHA512:
-  metadata.gz: 72fd7368da5f8f48118d3d6392e2df08ac6a8fe471c55a902606c157a7b8f27bc517f5d6e98f8a1a9c616c1fec6720b8293e06300baad88c22336dcc579f12c9
-  data.tar.gz: f8f6c77ed11d3322871856fbf89da78dbfa2d730eb386a06450aff6fef36abedb55ee9b4eb89e8d6a75c3d12f298db0c35a38771aa6fc6ab75577e8596aac9c9
+  metadata.gz: 12c4227163ccd825b06f3c745bafa5a59725327e8b313c68a99cc732cea488c1cdda723c1e229d6ac50683e669594820a643c6d2bd0c5ac17867e75397969d10
+  data.tar.gz: '09994a51e53f704d642d27da154c835ba99b39bfbc2ddefa98e77a4357abfe062084628b6fb8c5efd9b596a641e562e595b3b0f80e196997d8bba5002c1448e5'

data/History.rdoc

@@ -1,3 +1,29 @@
+=== 1.7.0 / 2023-07-06
+
+* 1 major enhancement:
+
+  * Extend bisect_methods to do "inverse" run (eg false positives) via -n=/RE/ argument.
+
+* 4 minor enhancements:
+
+  * Added example_inverse.rb
+  * Collapsed examples from directories to individual files.
+  * Refactor: push command generation from #run down to #bisect_methods.
+  * build_files_cmd no longer calls reset (zero effect change).
+
+* 4 bug fixes:
+
+  * 100% documentation coverage.
+  * Fix server process ID to be a string to match rest of args.
+  * Fix shebang cmd on bin/minitest_bisect.
+  * Print known bad methods with found culprit methods in final run.
+
+=== 1.6.0 / 2022-05-23
+
+* 1 minor enhancement:
+
+  * Removed #bisect_files and mode accessor. Just deal with methods only for now.
+
 === 1.5.1 / 2019-09-22
 
 * 1 bug fix:

data/Manifest.txt

@@ -4,24 +4,10 @@ Manifest.txt
 README.rdoc
 Rakefile
 bin/minitest_bisect
-example-many/helper.rb
-example-many/test_bad1.rb
-example-many/test_bad2.rb
-example-many/test_bad3.rb
-example-many/test_bad4.rb
-example-many/test_bad5.rb
-example-many/test_bad6.rb
-example-many/test_bad7.rb
-example-many/test_bad8.rb
-example/helper.rb
-example/test_bad1.rb
-example/test_bad2.rb
-example/test_bad3.rb
-example/test_bad4.rb
-example/test_bad5.rb
-example/test_bad6.rb
-example/test_bad7.rb
-example/test_bad8.rb
+example.rb
+example_helper.rb
+example_inverse.rb
+example_many.rb
 lib/minitest/bisect.rb
 lib/minitest/find_minimal_combination.rb
 test/minitest/test_bisect.rb

data/README.rdoc

@@ -19,10 +19,13 @@ what minitest-bisect does best.
 
 == FEATURES/PROBLEMS:
 
-* minitest_bisect first runs your tests on a per-file basis to
-  minimize the number of tests you need to sift through.
-* minitest_bisect next runs the minimized files and figures out your
-  exact failure reproduction.
+* normal mode: runs your tests with a fixed seed and minimizes the
+  tests to find a minimal set to reproduce the failure.
+* inverted mode: runs your tests with a fixed seed and a starting
+  failure and minimizes the tests to find a minimal set to reproduce
+  the false positive.
+* Includes Enumerable#find_minimal_combination and
+  #find_minimal_combination_and_count.
 
 == SYNOPSIS:
 
@@ -94,8 +97,7 @@ So, you run the tests again, but this time with minitest_bisect.
 Provide the seed given in the failure so the tests always run in the
 same order and reproduce every time.
 
-minitest_bisect will first minimize the number of files, then it will
-turn around and minimize the number of methods.
+minitest_bisect will minimize the number of methods.
 
   % minitest_bisect example --seed 314
   reproducing... in 203.83 sec
@@ -114,7 +116,7 @@ turn around and minimize the number of methods.
 
   Minimal methods found in 11 steps:
 
-  Culprit methods: ["TestBad1#test_bad1_1"]
+  Culprit methods: ["TestBad1#test_bad1_1", "TestBad4#test_bad4_4"]
 
   ruby -Itest:lib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad2.rb" ; require "./example/test_bad3.rb" ; require "./example/test_bad4.rb" ; require "./example/test_bad5.rb" ; require "./example/test_bad6.rb" ; require "./example/test_bad7.rb" ; require "./example/test_bad8.rb"' -- --seed 314 -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
 
@@ -148,9 +150,47 @@ have and how long they take, the minimization might take a long time,
 but each iteration can reduce the number of tests so it should get
 quicker with time.
 
+=== But sometimes, it fails:
+
+Sometimes a test fails by itself consistently but passes most of the
+time in CI. This is a false-positive in that the test SHOULD be
+failing all the time but some other test is causing a side-effect that
+it making it pass. This is sometimes harder to track down and reason
+about. minitest-bisect now has an "inverted" mode, where you run as
+normal but point it at a failing test by name. It will then validate
+that the test fails by itself but passes when run in full with the
+given seed. Then it tries to isolate and answer "what is the minimal
+combination of tests to run to # make this test pass?". For example:
+
+  % minitest_bisect example --seed=3 -n="/failing_test_name_regexp/"
+
+  reproducing w/ scoped failure (inverted run!)... in 0.06 sec
+  reproducing false positive... in 0.06 sec
+  # of culprit methods: 2 in 0.06 sec
+  # of culprit methods: 1 in 0.06 sec
+  # of culprit methods: 1 in 0.06 sec
+
+  Minimal methods found in 3 steps:
+
+  Culprit methods: ["TestBad1#test_make_it_go", "TestBad1#test_fail_without"]
+
+  /Users/ryan/.rubies/ruby-3.2.2/bin/ruby -Itest:lib -Ilib -e 'require "./example"' -- --seed 3  -n "/^(?:TestBad1#(?:test_make_it_go|test_fail_without))$/"
+
+  Final reproduction:
+
+  Run options: --seed 3 -n "/^(?:TestBad1#(?:test_make_it_go|test_fail_without))$/"
+
+  # Running:
+
+  ..
+
+  Finished in 0.000369s, 5420.0544 runs/s, 0.0000 assertions/s.
+
+  2 runs, 0 assertions, 0 failures, 0 errors, 0 skips
+
 == REQUIREMENTS:
 
-* minitest 5
+* minitest 5+
 
 == INSTALL:
 

data/Rakefile

@@ -40,11 +40,7 @@ def banner text
 end
 
 def run cmd
-  sh cmd do end
-end
-
-def req glob
-  Dir["#{glob}.rb"].map { |s| "require #{s.inspect}" }.join ";"
+  sh cmd do end # block form lets it fail w/o halting rake
 end
 
 task :repro => :isolate do
@@ -56,10 +52,10 @@ task :repro => :isolate do
   ruby = "ruby -I.:lib"
 
   banner "Original run that causes the test order dependency bug"
-  run "#{ruby} -e '#{req "example/test*"}' -- --seed 3911"
+  run "#{ruby} example.rb --seed 1"
 
   banner "Reduce the problem down to the minimal reproduction"
-  run "#{ruby} bin/minitest_bisect -Ilib --seed 3911 example/test*.rb"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 1 example.rb"
 end
 
 task :many => :isolate do
@@ -71,10 +67,31 @@ task :many => :isolate do
   ruby = "ruby -I.:lib"
 
   banner "Original run that causes the test order dependency bug"
-  run "#{ruby} -e '#{req "example-many/test*"}' -- --seed 27083"
+  run "#{ruby} ./example_many.rb --seed 2"
 
   banner "Reduce the problem down to the minimal reproduction"
-  run "#{ruby} bin/minitest_bisect -Ilib --seed 27083 example-many/test*.rb"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 2 example_many.rb"
+end
+
+task :inverse => :isolate do
+  unless ENV.key? "SLEEP" then
+    warn "NOTE: Defaulting to sleeping 0.01 seconds per test."
+    warn "NOTE: Use SLEEP=0 to disable or any other value to simulate your tests."
+  end
+
+  ruby = "ruby -I.:lib"
+
+  banner "Original run that passes (seed 1)"
+  run "#{ruby} example_inverse.rb --seed 1"
+
+  banner "Original run that *looks like* a test order dependency bug (seed 3)"
+  run "#{ruby} example_inverse.rb --seed 3"
+
+  banner "BAD bisection (tests fail by themselves) (seed 3)"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 3 example_inverse.rb"
+
+  banner "Reduce the passing run down to the minimal reproduction (seed 1)"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 1 example_inverse.rb -n=/TestBad4#test_bad4_4$/"
 end
 
 # vim: syntax=ruby

data/bin/minitest_bisect

@@ -1,4 +1,4 @@
-#!/usr/bin/ruby -w
+#!/usr/bin/env ruby -w
 
 require "minitest/bisect"
 

data/example.rb

@@ -0,0 +1,59 @@
+#!/usr/bin/env ruby -w
+
+require_relative "example_helper"
+
+# 800 tests, test test_bad4_4 fails if test_bad1_1 runs before it
+TestBad1 = create_test 1, 100, 1 => :infect
+TestBad2 = create_test 2, 100
+TestBad3 = create_test 3, 100
+TestBad4 = create_test 4, 100, 4 => :flunk
+TestBad5 = create_test 5, 100
+TestBad6 = create_test 6, 100
+TestBad7 = create_test 7, 100
+TestBad8 = create_test 8, 100
+
+# seed 1 == one fail
+# seed 3 == all pass
+
+# % SLEEP=0 ruby ./example.rb --seed 1
+#
+# and it fails, as expected. So we run it through minitest_bisect and see:
+#
+#   reproducing... in 0.14 sec
+#   verifying... in 0.06 sec
+#   # of culprit methods: 128 in 0.08 sec
+#   # of culprit methods: 64 in 0.07 sec
+#   # of culprit methods: 64 in 0.07 sec
+#   # of culprit methods: 32 in 0.06 sec
+#   # of culprit methods: 16 in 0.06 sec
+#   # of culprit methods: 8 in 0.06 sec
+#   # of culprit methods: 4 in 0.06 sec
+#   # of culprit methods: 4 in 0.06 sec
+#   # of culprit methods: 2 in 0.06 sec
+#   # of culprit methods: 1 in 0.06 sec
+#
+#   Minimal methods found in 10 steps:
+#
+#   Culprit methods: ["TestBad1#test_bad1_1", "TestBad4#test_bad4_4"]
+#
+#   /Users/ryan/.rubies/ruby-3.2.2/bin/ruby -Itest:lib -e 'require "././example.rb"' -- --seed 1  -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
+#
+#   Final reproduction:
+#
+#   Run options: --seed 1 -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
+#
+#   # Running:
+#
+#   .F
+#
+#   Finished in 0.001349s, 1482.5797 runs/s, 741.2898 assertions/s.
+#
+#     1) Failure:
+#   TestBad4#test_bad4_4 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example.rb:20]:
+#   muahahaha order dependency bug!
+#
+#   2 runs, 1 assertions, 1 failures, 0 errors, 0 skips
+#
+# and that's all there is to it! You now know that test_bad4_4 fails
+# only when paired with test_bad1_1 before it. You can now debug the
+# two methods and see what they're both modifying/dependent on.

data/{example-many/helper.rb → example_helper.rb}

@@ -1,4 +1,7 @@
-$hosed ||= 0
+require "minitest/autorun"
+
+$good = true
+$bomb = 0
 
 def create_test suffix, n_methods, bad_methods = {}
   raise ArgumentError, "Bad args" if Hash === n_methods
@@ -12,10 +15,16 @@ def create_test suffix, n_methods, bad_methods = {}
         sleep delay if delay > 0
 
         case bad_methods[n]
-        when true then
-          $hosed += 1
+        when :flunk then
+          flunk "muahahaha order dependency bug!" unless $good
+        when :infect then
+          $good = false
+        when :fix then
+          $good = true
+        when :tick then
+          $bomb += 1
         when Integer then
-          flunk "muahahaha order dependency bug!" if $hosed >= bad_methods[n]
+          flunk "muahahaha order dependency bug!" if $bomb >= bad_methods[n]
         else
           assert true
         end

data/example_inverse.rb

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby -w
+
+require_relative "example_helper"
+
+$good = false
+
+# 800 tests, test test_bad4_4 passes if test_bad1_1 runs before it
+TestBad1 = create_test 1, 100, 1 => :fix
+TestBad2 = create_test 2, 100
+TestBad3 = create_test 3, 100
+TestBad4 = create_test 4, 100, 4 => :flunk
+TestBad5 = create_test 5, 100
+TestBad6 = create_test 6, 100
+TestBad7 = create_test 7, 100
+TestBad8 = create_test 8, 100
+
+# seed 1 == all pass
+# seed 3 == one fail
+
+# UNLIKE the scenario spelled out in example.rb...
+#
+# % SLEEP=0 ruby ./example_inverse.rb --seed 1
+#
+# passes, but
+#
+# % SLEEP=0 ruby ./example_inverse.rb --seed 3
+#
+# has 1 failure! So we run that through minitest_bisect:
+#
+# % SLEEP=0 ruby -Ilib bin/minitest_bisect ./example_inverse.rb --seed 3
+#
+# and see:
+#
+#   Tests fail by themselves. This may not be an ordering issue.
+#
+# followed by a result that doesn't actually lead to the failure.
+#
+# Doing a normal run of minitest_bisect in this scenario doesn't
+# detect anything actually relevant. This is because we're not dealing
+# with a false *negative*, we're dealing with a false *positive*. The
+# question is "what makes this test pass?". So, we run it again with
+# the passing seed but this time point it at the failing test by name:
+#
+# % SLEEP=0 ruby -Ilib bin/minitest_bisect ./example_inverse.rb --seed 1 -n=/TestBad4#test_bad4_4$/
+#
+# and it outputs:
+#
+#   Culprit methods: ["TestBad1#test_bad1_1", "TestBad4#test_bad4_4"]
+
+# and shows a minimized run with the 2 passing tests. You now know
+# that test_bad4_4 passes only when paired with test_bad1_1 before it.
+# You can now debug the two methods and see what they're both
+# modifying/dependent on.

data/example_many.rb

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby -w
+
+require_relative "example_helper"
+
+TestBad1 = create_test 1, 100,  1 => :tick
+TestBad2 = create_test 2, 100
+TestBad3 = create_test 3, 100, 72 => :tick
+TestBad4 = create_test 4, 100
+TestBad5 = create_test 5, 100, 17 => :tick
+TestBad6 = create_test 6, 100
+TestBad7 = create_test 7, 100
+TestBad8 = create_test 8, 100, 43 => 3
+
+# seed 1 == all pass
+# seed 2 == one fail
+
+# % ruby example_many.rb --seed 2
+
+# and it fails, as expected. So we run it through minitest_bisect and see:
+
+# % ruby -Ilib bin/minitest_bisect example_many.rb --seed 2
+
+#   reproducing... in 0.15 sec
+#   verifying... in 0.06 sec
+#   # of culprit methods: 256 in 0.09 sec
+#   ... 82 more bisections ...
+#   # of culprit methods: 3 in 0.06 sec
+#
+#   Minimal methods found in 84 steps:
+#
+#   Culprit methods: ["TestBad3#test_bad3_72", "TestBad5#test_bad5_17", "TestBad1#test_bad1_1", "TestBad8#test_bad8_43"]
+#
+#   /Users/ryan/.rubies/ruby-3.2.2/bin/ruby -Itest:lib -e 'require "./example_many.rb"' -- --seed 2  -n "/^(?:TestBad3#(?:test_bad3_72)|TestBad5#(?:test_bad5_17)|TestBad1#(?:test_bad1_1)|TestBad8#(?:test_bad8_43))$/"
+#
+#   Final reproduction:
+#
+#   Run options: --seed 2 -n "/^(?:TestBad3#(?:test_bad3_72)|TestBad5#(?:test_bad5_17)|TestBad1#(?:test_bad1_1)|TestBad8#(?:test_bad8_43))$/"
+#
+#   # Running:
+#
+#   ...F
+#
+#   Finished in 0.001404s, 2849.0028 runs/s, 712.2507 assertions/s.
+#
+#     1) Failure:
+#   TestBad8#test_bad8_43 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example_many.rb:20]:
+#   muahahaha order dependency bug!
+#
+#   4 runs, 1 assertions, 1 failures, 0 errors, 0 skips
+#
+# and that's all there is to it! You now know that the failing test
+# fails only when paired with the other 3 tests before it. You can now
+# debug the 4 methods and see what they're all modifying/dependent on.

data/lib/minitest/bisect.rb

@@ -4,10 +4,15 @@ require "shellwords"
 require "rbconfig"
 require "path_expander"
 
+module Minitest; end # :nodoc:
+
+##
+# Minitest::Bisect helps you isolate and debug random test failures.
+
 class Minitest::Bisect
-  VERSION = "1.5.1"
+  VERSION = "1.7.0" # :nodoc:
 
-  class PathExpander < ::PathExpander
+  class PathExpander < ::PathExpander # :nodoc:
     TEST_GLOB = "**/{test_*,*_test,spec_*,*_spec}.rb" # :nodoc:
 
     attr_accessor :rb_flags
@@ -39,7 +44,7 @@ class Minitest::Bisect
   end
 
   mtbv = ENV["MTB_VERBOSE"].to_i
-  SHH = case
+  SHH = case # :nodoc:
         when mtbv == 1 then " > /dev/null"
         when mtbv >= 2 then nil
         else " > /dev/null 2>&1"
@@ -51,21 +56,48 @@ class Minitest::Bisect
               RbConfig::CONFIG['ruby_install_name'] +
                 RbConfig::CONFIG['EXEEXT']).sub(/.*\s.*/m, '"\&"')
 
-  attr_accessor :tainted, :failures, :culprits, :mode, :seen_bad
+  ##
+  # True if this run has seen a failure.
+
+  attr_accessor :tainted
   alias :tainted? :tainted
 
+  ##
+  # Failures seen in this run. Shape:
+  #
+  #   {"file.rb"=>{"Class"=>["test_method1", "test_method2"] ...} ...}
+
+  attr_accessor :failures
+
+  ##
+  # An array of tests seen so far. NOT cleared by #reset.
+
+  attr_accessor :culprits
+
+  attr_accessor :seen_bad # :nodoc:
+
+  ##
+  # Top-level runner. Instantiate and call +run+, handling exceptions.
+
   def self.run files
     new.run files
   rescue => e
     warn e.message
+    warn "Try running with MTB_VERBOSE=2 to verify."
     exit 1
   end
 
+  ##
+  # Instantiate a new Bisect.
+
   def initialize
     self.culprits = []
     self.failures = Hash.new { |h, k| h[k] = Hash.new { |h2, k2| h2[k2] = [] } }
   end
 
+  ##
+  # Reset per-bisect-run variables.
+
   def reset
     self.seen_bad = false
     self.tainted  = false
@@ -73,23 +105,23 @@ class Minitest::Bisect
     # not clearing culprits on purpose
   end
 
+  ##
+  # Instance-level runner. Handles Minitest::Server, argument
+  # processing, and invoking +bisect_methods+.
+
   def run args
     Minitest::Server.run self
 
     cmd = nil
 
-    if :until_I_have_negative_filtering_in_minitest != 0 then
-      mt_flags = args.dup
-      expander = Minitest::Bisect::PathExpander.new mt_flags
+    mt_flags = args.dup
+    expander = Minitest::Bisect::PathExpander.new mt_flags
 
-      files = expander.process
-      rb_flags = expander.rb_flags
-      mt_flags += ["--server", $$]
+    files = expander.process
+    rb_flags = expander.rb_flags
+    mt_flags += ["--server", $$.to_s]
 
-      cmd = bisect_methods build_files_cmd(files, rb_flags, mt_flags)
-    else
-      cmd = bisect_methods bisect_files args
-    end
+    cmd = bisect_methods files, rb_flags, mt_flags
 
     puts "Final reproduction:"
     puts
@@ -99,70 +131,87 @@ class Minitest::Bisect
     Minitest::Server.stop
   end
 
-  def bisect_files files
-    self.mode = :files
-
-    files, flags = files.partition { |arg| File.file? arg }
-    rb_flags, mt_flags = flags.partition { |arg| arg =~ /^-I/ }
-    mt_flags += ["--server", $$]
-
-    puts "reproducing..."
-    system "#{build_files_cmd files, rb_flags, mt_flags} #{SHH}"
-    abort "Reproduction run passed? Aborting. Try running with MTB_VERBOSE=2 to verify." unless tainted?
-    puts "reproduced"
-
-    found, count = files.find_minimal_combination_and_count do |test|
-      puts "# of culprit files: #{test.size}"
-
-      system "#{build_files_cmd test, rb_flags, mt_flags} #{SHH}"
-
-      self.tainted?
+  ##
+  # Normal: find "what is the minimal combination of tests to run to
+  #         make X fail?"
+  #
+  # Run with: minitest_bisect ... --seed=N
+  #
+  # 1. Verify the failure running normally with the seed.
+  #    2. If no failure, punt.
+  #    3. If no passing tests before failure, punt. (No culprits == no debug)
+  # 4. Verify the failure doesn't fail in isolation.
+  #    5. If it still fails by itself, warn that it might not be an ordering
+  #       issue.
+  # 6. Cull all tests after the failure, they're not involved.
+  # 7. Bisect the culprits + bad until you find a minimal combo that fails.
+  # 8. Display minimal combo by running one last time.
+  #
+  # Inverted: find "what is the minimal combination of tests to run to
+  #           make this test pass?"
+  #
+  # Run with: minitest_bisect ... --seed=N -n="/failing_test_name_regexp/"
+  #
+  # 1. Verify the failure by running normally w/ the seed and -n=/.../
+  #    2. If no failure, punt.
+  # 3. Verify the passing case by running everything.
+  #    4. If failure, punt. This is not a false positive.
+  # 5. Cull all tests after the bad test from #1, they're not involved.
+  # 6. Bisect the culprits + bad until you find a minimal combo that passes.
+  # 7. Display minimal combo by running one last time.
+
+  def bisect_methods files, rb_flags, mt_flags
+    bad_names, mt_flags = mt_flags.partition { |s| s =~ /^(?:-n|--name)/ }
+    normal   = bad_names.empty?
+    inverted = !normal
+
+    if inverted then
+      time_it "reproducing w/ scoped failure (inverted run!)...", build_methods_cmd(build_files_cmd(files, rb_flags, mt_flags + bad_names))
+      raise "No failures. Probably not a false positive. Aborting." if failures.empty?
+      bad = map_failures
     end
 
-    puts
-    puts "Minimal files found in #{count} steps:"
-    puts
-    cmd = build_files_cmd found, rb_flags, mt_flags
-    puts cmd
-    cmd
-  end
-
-  def bisect_methods cmd
-    self.mode = :methods
+    cmd = build_files_cmd(files, rb_flags, mt_flags)
 
-    time_it "reproducing...", build_methods_cmd(cmd)
+    msg = normal ? "reproducing..." : "reproducing false positive..."
+    time_it msg, build_methods_cmd(cmd)
 
-    unless tainted? then
-      $stderr.puts "Reproduction run passed? Aborting."
-      abort "Try running with MTB_VERBOSE=2 to verify."
+    if normal then
+      raise "Reproduction run passed? Aborting." unless tainted?
+      raise "Verification failed. No culprits? Aborting." if culprits.empty? && seen_bad
+    else
+      raise "Reproduction failed? Not false positive. Aborting." if tainted?
+      raise "Verification failed. No culprits? Aborting." if culprits.empty? || seen_bad
     end
 
-    bad = map_failures
+    if normal then
+      bad = map_failures
 
-    raise "Nothing to verify against because every test failed. Aborting." if
-      culprits.empty? && seen_bad
+      time_it "verifying...", build_methods_cmd(cmd, [], bad)
 
-    time_it "verifying...", build_methods_cmd(cmd, [], bad)
+      new_bad = map_failures
 
-    new_bad = map_failures
-
-    if bad == new_bad then
-      warn "Tests fail by themselves. This may not be an ordering issue."
+      if bad == new_bad then
+        warn "Tests fail by themselves. This may not be an ordering issue."
+      end
     end
 
+    idx = culprits.index bad.first
+    self.culprits = culprits.take idx+1 if idx # cull tests after bad
+
     # culprits populated by initial reproduction via minitest/server
     found, count = culprits.find_minimal_combination_and_count do |test|
       prompt = "# of culprit methods: #{test.size}"
 
       time_it prompt, build_methods_cmd(cmd, test, bad)
 
-      self.tainted?
+      normal == tainted? # either normal and failed, or inverse and passed
     end
 
     puts
     puts "Minimal methods found in #{count} steps:"
     puts
-    puts "Culprit methods: %p" % [found]
+    puts "Culprit methods: %p" % [found + bad]
     puts
     cmd = build_methods_cmd cmd, found, bad
     puts cmd.sub(/--server \d+/, "")
@@ -170,14 +219,14 @@ class Minitest::Bisect
     cmd
   end
 
-  def time_it prompt, cmd
+  def time_it prompt, cmd # :nodoc:
     print prompt
     t0 = Time.now
     system "#{cmd} #{SHH}"
     puts " in %.2f sec" % (Time.now - t0)
   end
 
-  def map_failures
+  def map_failures # :nodoc:
     # from: {"file.rb"=>{"Class"=>["test_method1", "test_method2"]}}
     #   to: ["Class#test_method1", "Class#test_method2"]
     failures.values.map { |h|
@@ -185,15 +234,31 @@ class Minitest::Bisect
     }.flatten.sort
   end
 
-  def build_files_cmd culprits, rb, mt
-    reset
-
+  def build_files_cmd culprits, rb, mt # :nodoc:
     tests = culprits.flatten.compact.map { |f| %(require "./#{f}") }.join " ; "
 
     %(#{RUBY} #{rb.shelljoin} -e '#{tests}' -- #{mt.map(&:to_s).shelljoin})
   end
 
-  def build_re bad
+  def build_methods_cmd cmd, culprits = [], bad = nil # :nodoc:
+    reset
+
+    if bad then
+      re = build_re culprits + bad
+
+      cmd += " -n \"#{re}\"" if bad
+    end
+
+    if ENV["MTB_VERBOSE"].to_i >= 1 then
+      puts
+      puts cmd
+      puts
+    end
+
+    cmd
+  end
+
+  def build_re bad # :nodoc:
     re = []
 
     # bad by class, you perv
@@ -213,44 +278,24 @@ class Minitest::Bisect
     "/^(?:#{re})$/"
   end
 
-  def re_escape str
+  def re_escape str # :nodoc:
     str.gsub(/([`'"!?&\[\]\(\)\{\}\|\+])/, '\\\\\1')
   end
 
-  def build_methods_cmd cmd, culprits = [], bad = nil
-    reset
-
-    if bad then
-      re = build_re culprits + bad
-
-      cmd += " -n \"#{re}\"" if bad
-    end
-
-    if ENV["MTB_VERBOSE"].to_i >= 1 then
-      puts
-      puts cmd
-      puts
-    end
-
-    cmd
-  end
-
   ############################################################
   # Server Methods:
 
-  def minitest_start
+  def minitest_start # :nodoc:
     self.failures.clear
   end
 
-  def minitest_result file, klass, method, fails, assertions, time
+  def minitest_result file, klass, method, fails, assertions, time # :nodoc:
     fails.reject! { |fail| Minitest::Skip === fail }
 
-    if mode == :methods then
-      if fails.empty? then
-        culprits << "#{klass}##{method}" unless seen_bad # UGH
-      else
-        self.seen_bad = true
-      end
+    if fails.empty? then
+      culprits << "#{klass}##{method}" unless seen_bad # UGH
+    else
+      self.seen_bad = true
     end
 
     return if fails.empty?

data/lib/minitest/find_minimal_combination.rb

@@ -1,10 +1,13 @@
 #!/usr/bin/ruby -w
 
-module Minitest; end
+##
+# Finds the minimal combination of a collection of items that satisfy
+# +test+.
 
 class ComboFinder
   ##
-  # Find the minimal combination of a collection of items that satisfy +test+.
+  # Find the minimal combination of a collection of items that satisfy
+  # +test+.
   #
   # If you think of the collection as a binary tree, this algorithm
   # does a breadth first search of the combinations that satisfy
@@ -79,11 +82,11 @@ class ComboFinder
     ary
   end
 
-  def d s = ""
+  def d s = "" # :nodoc:
     warn s if ENV["MTB_DEBUG"]
   end
 
-  def cache_result result, data, cache
+  def cache_result result, data, cache # :nodoc:
     cache[data] = true
 
     return result if result
@@ -103,7 +106,7 @@ class ComboFinder
   end
 end
 
-class Array
+class Array # :nodoc:
   ##
   # Find the minimal combination of a collection of items that satisfy +test+.
 

data/test/minitest/test_bisect.rb

@@ -8,6 +8,7 @@ class TestMinitest::TestBisect < Minitest::Test
 
   def setup
     self.bisect = Minitest::Bisect.new
+    bisect.reset
   end
 
   def test_class_run
@@ -124,9 +125,6 @@ class TestMinitest::TestBisect < Minitest::Test
   end
 
   def test_minitest_result
-    bisect.mode = :methods
-    bisect.reset
-
     bisect.minitest_result "file.rb", "TestClass", "test_method", [], 1, 1
 
     assert_equal false, bisect.tainted
@@ -135,9 +133,6 @@ class TestMinitest::TestBisect < Minitest::Test
   end
 
   def test_minitest_result_skip
-    bisect.mode = :methods
-    bisect.reset
-
     fail = Minitest::Skip.new("woot")
 
     bisect.minitest_result "file.rb", "TestClass", "test_method", [fail], 1, 1
@@ -148,9 +143,6 @@ class TestMinitest::TestBisect < Minitest::Test
   end
 
   def test_minitest_result_fail
-    bisect.mode = :methods
-    bisect.reset
-
     fail = Minitest::Assertion.new "msg"
 
     bisect.minitest_result "file.rb", "TestClass", "test_method", [fail], 1, 1
@@ -163,9 +155,6 @@ class TestMinitest::TestBisect < Minitest::Test
   end
 
   def test_minitest_result_error
-    bisect.mode = :methods
-    bisect.reset
-
     fail = Minitest::UnexpectedError.new RuntimeError.new("woot")
 
     bisect.minitest_result "file.rb", "TestClass", "test_method", [fail], 1, 1

data/test/minitest/test_find_minimal_combination.rb

@@ -10,93 +10,99 @@ describe Array, :find_minimal_combination do
     lambda { |sample| bad & sample == bad }
   end
 
+  def record_and_check(tests, *bad)
+    lambda { |test| tests << test.join; bad & test == bad }
+  end
+
+  def parse_trials s
+    s.lines.map { |s| s.chomp.sub(/#.*/, '').delete " " }.reject(&:empty?)
+  end
+
   def assert_steps input, bad, exp
     tests = []
 
-    found = input.find_minimal_combination do |test|
-      tests << test
-      bad & test == bad
-    end
+    found = input.find_minimal_combination(&record_and_check(tests, *bad))
 
     assert_equal bad, found, "algorithm is bad"
 
-    assert_equal exp, tests
+    assert_equal parse_trials(exp), tests
   end
 
-  def test_ordering_best_case
-    a   = (0..15).to_a
-    bad = [0, 1]
-    exp = [[0, 1, 2, 3, 4, 5, 6, 7],
-           [0, 1, 2, 3],
-           [0, 1],
-           [0],
-           [1],
-           [0, 1]]
-
-    assert_steps a, bad, exp
+  HEX = "0123456789ABCDEF".chars.to_a
+
+  # lvl      collection
+  #
+  #  0 |         A
+  #  1 |     B       C
+  #  2 |   D   E   F   G
+  #  3 |  H I J K L M N O
+  #    |
+  #  4 |  0123456789ABCDEF
+
+  def test_ordering_best_case_1
+    ary = HEX
+    bad = %w[0]
+    exp = <<~EOT
+      #123456789ABCDEF
+      01234567         # HIT! -- level 1 = B, C
+      0123             # HIT! -- level 2 = D, E
+      01               # HIT! -- level 3 = H, I
+      0                # HIT!
+    EOT
+
+    assert_steps ary, bad, exp
   end
 
-  def test_ordering
-    a     = (0..15).to_a
-    bad   = [0, 15]
-
-    # lvl              collection
-    #
-    #  0 |                 A
-    #  1 |         B               C
-    #  2 |     D       E       F       G
-    #  3 |   H   I   J   K   L   M   N   O
-    #  4 |  0 1 2 3 4 5 6 7 8 9 A B C D E F
-    #
-    #  0    +++++++++++++++++++++++++++++++
-    #  1    ---------------
-    #  1                    ---------------
-    #  2    -------         -------
-    #  2    +++++++                 +++++++
-    #  3    xxxxxxx
-    #  3                            xxxxxxx
-    #  3    ---                     ---
-    #  3    +++                         +++
-    #  4    xxx
-    #  4                                xxx
-    #  4    -                           -
-    #  4    +                             +
-    #
-    #  - = miss
-    #  + = hit
-    #  x = unwanted test
-
-    exp = [
-           # level 1 = B, C
-           [0, 1, 2, 3, 4, 5, 6, 7],
-           [8, 9, 10, 11, 12, 13, 14, 15],
-
-           # level 2 = DF, DG, EF, EG
-           [0, 1, 2, 3, 8, 9, 10, 11],
-           [0, 1, 2, 3, 12, 13, 14, 15],
-
-           # level 3
-           # [0, 1, 2, 3],     # I think this is bad, we've tested B
-           # [12, 13, 14, 15], # again, bad, we've tested C
-           [0, 1, 12, 13],
-           [0, 1, 14, 15],
-
-           # level 4
-           # [0, 1],
-           # [14, 15],
-           [0, 14],
-           [0, 15],
-          ]
-
-    assert_steps a, bad, exp
+  def test_ordering_best_case_2
+    ary = HEX
+    bad = %w[0 1]
+    exp = <<~EOT
+      01234567         # HIT! -- level 1 = B, C
+      0123             # HIT! -- level 2 = D, E
+      01               # HIT! -- level 3 = H, I
+      0                # miss -- level 4 = 0, 1, n_combos = 1
+       1               # miss
+      01               # HIT! -- level 3 = H,    n_combos = 2
+    EOT
+
+    assert_steps ary, bad, exp
   end
 
-  make_my_diffs_pretty!
+  def test_ordering
+    ary = HEX
+    bad = %w[1 F]
+    exp = <<~EOT
+      01234567         # miss -- level 1 = B, C
+              89ABCDEF # miss
+      0123    89AB     # miss -- level 2 = DF, DG, EF, EG
+      0123        CDEF # HIT!
+      01          CD   # miss -- level 3 = HN, HO
+      01            EF # HIT!
+      0             E  # miss -- level 4 = 0E, 0F, 1E, 1F
+      0              F # miss
+       1            E  # miss
+       1             F # HIT!
+    EOT
+
+    assert_steps ary, bad, exp
+  end
 
   def self.test_find_minimal_combination max, *bad
-    define_method "test_find_minimal_combination_#{max}_#{bad.join "_"}" do
+    define_method "%s_%s_%s" % [__method__, max, bad.join("_")] do
       a = (1..max).to_a
-      _(a.find_minimal_combination(&check(*bad))).must_equal bad
+
+      assert_equal bad, a.find_minimal_combination(&check(*bad))
+    end
+  end
+
+  def self.test_find_minimal_combination_and_count max, nsteps, *bad
+    define_method "%s_%s_%s_%s" % [__method__, max, nsteps, bad.join("_")] do
+      a = (1..max).to_a
+
+      found, count = a.find_minimal_combination_and_count(&check(*bad))
+
+      assert_equal bad, found
+      assert_equal nsteps, count
     end
   end
 
@@ -119,7 +125,14 @@ describe Array, :find_minimal_combination do
   test_find_minimal_combination 1023, 7, 15, 166, 1001
   test_find_minimal_combination 1023, 1000, 1001, 1002
   test_find_minimal_combination 1023, 1001, 1003, 1005, 1007
+  test_find_minimal_combination 1024, 1001, 1003, 1005, 1007
+  test_find_minimal_combination 1024, 1, 1024
+
+  test_find_minimal_combination_and_count 1024, 12, 1, 2
+  test_find_minimal_combination_and_count 1024, 23, 1, 1023
+  test_find_minimal_combination_and_count 1024, 24, 1, 1024
+  test_find_minimal_combination_and_count 1023, 26, 1, 1023
 
-  # test_find_minimal_combination 1024, 1001, 1003, 1005, 1007
-  # test_find_minimal_combination 1023, 1001, 1003, 1005, 1007
+  test_find_minimal_combination_and_count 1024, 93, 1001, 1003, 1005, 1007
+  test_find_minimal_combination_and_count 1023, 93, 1001, 1003, 1005, 1007
 end

data.tar.gz.sig

@@ -1,3 +1 @@
-h���
-��U$�P،�3��]�1j*�&��d��7ta��
�]��ޫ��D��Q��-���-��SnT������U2��ӗ�����6y
-
+�;+�K�>]@+��^c�T�Zɫ;�Mag��R[�0q*q钮�O������Gge�|�NC�5~͚**�4�!'���w9����?SV<��wq�=?,��Obx��05{�5�r�ThG:�ﶉ�6{��'�T�VJ͌��;ꉂ�J4>r��~���
�Ű�]�G<J�f����G���¯.>	�.vg�=���2�_�֝�#WI
���|�A�L((l] B������F�׼�9⻛u

metadata

@@ -1,18 +1,18 @@
 --- !ruby/object:Gem::Specification
 name: minitest-bisect
 version: !ruby/object:Gem::Version
-  version: 1.5.1
+  version: 1.7.0
 platform: ruby
 authors:
 - Ryan Davis
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain:
 - |
   -----BEGIN CERTIFICATE-----
-  MIIDPjCCAiagAwIBAgIBAzANBgkqhkiG9w0BAQsFADBFMRMwEQYDVQQDDApyeWFu
+  MIIDPjCCAiagAwIBAgIBBzANBgkqhkiG9w0BAQsFADBFMRMwEQYDVQQDDApyeWFu
   ZC1ydWJ5MRkwFwYKCZImiZPyLGQBGRYJemVuc3BpZGVyMRMwEQYKCZImiZPyLGQB
-  GRYDY29tMB4XDTE4MTIwNDIxMzAxNFoXDTE5MTIwNDIxMzAxNFowRTETMBEGA1UE
+  GRYDY29tMB4XDTIzMDEwMTA3NTExN1oXDTI0MDEwMTA3NTExN1owRTETMBEGA1UE
   AwwKcnlhbmQtcnVieTEZMBcGCgmSJomT8ixkARkWCXplbnNwaWRlcjETMBEGCgmS
   JomT8ixkARkWA2NvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALda
   b9DCgK+627gPJkB6XfjZ1itoOQvpqH1EXScSaba9/S2VF22VYQbXU1xQXL/WzCkx
@@ -22,14 +22,14 @@ cert_chain:
   qhtV7HJxNKuPj/JFH0D2cswvzznE/a5FOYO68g+YCuFi5L8wZuuM8zzdwjrWHqSV
   gBEfoTEGr7Zii72cx+sCAwEAAaM5MDcwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAw
   HQYDVR0OBBYEFEfFe9md/r/tj/Wmwpy+MI8d9k/hMA0GCSqGSIb3DQEBCwUAA4IB
-  AQCbJwLmpJR2PomLU+Zzw3KRzH/hbyUWc/ftru71AopZ1fy4iY9J/BW5QYKVYwbP
-  V0FSBWtvfI/RdwfKGtuGhPKECZgmLieGuZ3XCc09qPu1bdg7i/tu1p0t0c6163ku
-  nDMDIC/t/DAFK0TY9I3HswuyZGbLW7rgF0DmiuZdN/RPhHq2pOLMLXJmFclCb/im
-  9yToml/06TJdUJ5p64mkBs0TzaK66DIB1Smd3PdtfZqoRV+EwaXMdx0Hb3zdR1JR
-  Em82dBUFsipwMLCYj39kcyHWAxyl6Ae1Cn9r/ItVBCxoeFdrHjfavnrIEoXUt4bU
-  UfBugfLD19bu3nvL+zTAGx/U
+  AQAkg3y+PBnBAPWdxxITm5sPHqdWQgSyCpRA20o4LTuWr8BWhSXBkfQNa7cY6fOn
+  xyM34VPzBFbExv6XOGDfOMFBVaYTHuN9peC/5/umL7kLl+nflXzL2QA7K6LYj5Bg
+  sM574Onr0dZDM6Vn69bzQ7rBIFDfK/OhlPzqKZad4nsdcsVH8ODCiT+ATMIZyz5K
+  WCnNtqlyiWXI8tdTpahDgcUwfcN/oN7v4K8iU5IbLJX6HQ5DKgmKjfb6XyMth16k
+  ROfWo9Uyp8ba/j9eVG14KkYRaLydAY1MNQk2yd3R5CGfeOpD1kttxjoypoUJ2dOG
+  nsNBRuQJ1UfiCG97a6DNm+Fr
   -----END CERTIFICATE-----
-date: 2019-09-23 00:00:00.000000000 Z
+date: 2023-07-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest-server
@@ -99,14 +99,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.18'
+        version: '4.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.18'
+        version: '4.0'
 description: |-
   Hunting down random test failures can be very very difficult,
   sometimes impossible, but minitest-bisect makes it easy.
@@ -135,24 +135,10 @@ files:
 - README.rdoc
 - Rakefile
 - bin/minitest_bisect
-- example-many/helper.rb
-- example-many/test_bad1.rb
-- example-many/test_bad2.rb
-- example-many/test_bad3.rb
-- example-many/test_bad4.rb
-- example-many/test_bad5.rb
-- example-many/test_bad6.rb
-- example-many/test_bad7.rb
-- example-many/test_bad8.rb
-- example/helper.rb
-- example/test_bad1.rb
-- example/test_bad2.rb
-- example/test_bad3.rb
-- example/test_bad4.rb
-- example/test_bad5.rb
-- example/test_bad6.rb
-- example/test_bad7.rb
-- example/test_bad8.rb
+- example.rb
+- example_helper.rb
+- example_inverse.rb
+- example_many.rb
 - lib/minitest/bisect.rb
 - lib/minitest/find_minimal_combination.rb
 - test/minitest/test_bisect.rb
@@ -160,8 +146,9 @@ files:
 homepage: https://github.com/seattlerb/minitest-bisect
 licenses:
 - MIT
-metadata: {}
-post_install_message: 
+metadata:
+  homepage_uri: https://github.com/seattlerb/minitest-bisect
+post_install_message:
 rdoc_options:
 - "--main"
 - README.rdoc
@@ -178,8 +165,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
-signing_key: 
+rubygems_version: 3.4.10
+signing_key:
 specification_version: 4
 summary: Hunting down random test failures can be very very difficult, sometimes impossible,
   but minitest-bisect makes it easy

data/example/helper.rb

@@ -1,25 +0,0 @@
-$hosed ||= false
-
-def create_test suffix, n_methods, bad_methods = {}
-  raise ArgumentError, "Bad args" if Hash === n_methods
-
-  delay = (ENV["SLEEP"] || 0.01).to_f
-
-  Class.new(Minitest::Test) do
-    n_methods.times do |n|
-      n += 1
-      define_method "test_bad#{suffix}_#{n}" do
-        sleep delay if delay > 0
-
-        case bad_methods[n]
-        when true then
-          flunk "muahahaha order dependency bug!" if $hosed
-        when false then
-          $hosed = true
-        else
-          assert true
-        end
-      end
-    end
-  end
-end

data/example/test_bad1.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad1 = create_test 1, 100, 1 => false

data/example/test_bad2.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad2 = create_test 2, 100

data/example/test_bad3.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad3 = create_test 3, 100

data/example/test_bad4.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad4 = create_test 4, 100, 4 => true

data/example/test_bad5.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad5 = create_test 5, 100

data/example/test_bad6.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad6 = create_test 6, 100

data/example/test_bad7.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad7 = create_test 7, 100

data/example/test_bad8.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad8 = create_test 8, 100

data/example-many/test_bad1.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad1 = create_test 1, 100, 1 => true

data/example-many/test_bad2.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad2 = create_test 2, 100

data/example-many/test_bad3.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad3 = create_test 3, 100, 72 => true

data/example-many/test_bad4.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad4 = create_test 4, 100

data/example-many/test_bad5.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad5 = create_test 5, 100, 17 => true

data/example-many/test_bad6.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad6 = create_test 6, 100

data/example-many/test_bad7.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad7 = create_test 7, 100

data/example-many/test_bad8.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad8 = create_test 8, 100, 43 => 3