minitest-bisect 1.6.0 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0adf4dd9767c961c0030a23e4e200245d74505bb0c5bd79dc0ff4201adeadc63
-  data.tar.gz: 501678cbbf5567c3408f1e44721dcbb11bc917b8e2d77c7ca063652e5ffc133f
+  metadata.gz: 52bae214fa2f36196af3ec54139c2bec839de200e8dee1015f916b56a666a69b
+  data.tar.gz: 15a204ca83b0da8a8773e565f3bc40c439713edacef4a1d66435f2f9938edf6e
 SHA512:
-  metadata.gz: 3c9f20afc3dcd8cb7e782caaa456e8f9dded619624226c9a7123290cc8cd47abe1ec32afdc1acd61dba41b1d1d1797389296e45f7abc234c27eca3206a8f5473
-  data.tar.gz: 3db4728af1051d31167814e277af68ee39fc0451a3c8df5c45f417a64b7d5fac8e72a9f2b5a590fa816566f1397f91429c1f16d608cba811d361faad246e7f6b
+  metadata.gz: 12c4227163ccd825b06f3c745bafa5a59725327e8b313c68a99cc732cea488c1cdda723c1e229d6ac50683e669594820a643c6d2bd0c5ac17867e75397969d10
+  data.tar.gz: '09994a51e53f704d642d27da154c835ba99b39bfbc2ddefa98e77a4357abfe062084628b6fb8c5efd9b596a641e562e595b3b0f80e196997d8bba5002c1448e5'

data/History.rdoc

@@ -1,3 +1,23 @@
+=== 1.7.0 / 2023-07-06
+
+* 1 major enhancement:
+
+  * Extend bisect_methods to do "inverse" run (eg false positives) via -n=/RE/ argument.
+
+* 4 minor enhancements:
+
+  * Added example_inverse.rb
+  * Collapsed examples from directories to individual files.
+  * Refactor: push command generation from #run down to #bisect_methods.
+  * build_files_cmd no longer calls reset (zero effect change).
+
+* 4 bug fixes:
+
+  * 100% documentation coverage.
+  * Fix server process ID to be a string to match rest of args.
+  * Fix shebang cmd on bin/minitest_bisect.
+  * Print known bad methods with found culprit methods in final run.
+
 === 1.6.0 / 2022-05-23
 
 * 1 minor enhancement:

data/Manifest.txt

@@ -4,24 +4,10 @@ Manifest.txt
 README.rdoc
 Rakefile
 bin/minitest_bisect
-example-many/helper.rb
-example-many/test_bad1.rb
-example-many/test_bad2.rb
-example-many/test_bad3.rb
-example-many/test_bad4.rb
-example-many/test_bad5.rb
-example-many/test_bad6.rb
-example-many/test_bad7.rb
-example-many/test_bad8.rb
-example/helper.rb
-example/test_bad1.rb
-example/test_bad2.rb
-example/test_bad3.rb
-example/test_bad4.rb
-example/test_bad5.rb
-example/test_bad6.rb
-example/test_bad7.rb
-example/test_bad8.rb
+example.rb
+example_helper.rb
+example_inverse.rb
+example_many.rb
 lib/minitest/bisect.rb
 lib/minitest/find_minimal_combination.rb
 test/minitest/test_bisect.rb

data/README.rdoc

@@ -19,10 +19,13 @@ what minitest-bisect does best.
 
 == FEATURES/PROBLEMS:
 
-* minitest_bisect first runs your tests on a per-file basis to
-  minimize the number of tests you need to sift through.
-* minitest_bisect next runs the minimized files and figures out your
-  exact failure reproduction.
+* normal mode: runs your tests with a fixed seed and minimizes the
+  tests to find a minimal set to reproduce the failure.
+* inverted mode: runs your tests with a fixed seed and a starting
+  failure and minimizes the tests to find a minimal set to reproduce
+  the false positive.
+* Includes Enumerable#find_minimal_combination and
+  #find_minimal_combination_and_count.
 
 == SYNOPSIS:
 
@@ -94,8 +97,7 @@ So, you run the tests again, but this time with minitest_bisect.
 Provide the seed given in the failure so the tests always run in the
 same order and reproduce every time.
 
-minitest_bisect will first minimize the number of files, then it will
-turn around and minimize the number of methods.
+minitest_bisect will minimize the number of methods.
 
   % minitest_bisect example --seed 314
   reproducing... in 203.83 sec
@@ -114,7 +116,7 @@ turn around and minimize the number of methods.
 
   Minimal methods found in 11 steps:
 
-  Culprit methods: ["TestBad1#test_bad1_1"]
+  Culprit methods: ["TestBad1#test_bad1_1", "TestBad4#test_bad4_4"]
 
   ruby -Itest:lib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad2.rb" ; require "./example/test_bad3.rb" ; require "./example/test_bad4.rb" ; require "./example/test_bad5.rb" ; require "./example/test_bad6.rb" ; require "./example/test_bad7.rb" ; require "./example/test_bad8.rb"' -- --seed 314 -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
 
@@ -148,9 +150,47 @@ have and how long they take, the minimization might take a long time,
 but each iteration can reduce the number of tests so it should get
 quicker with time.
 
+=== But sometimes, it fails:
+
+Sometimes a test fails by itself consistently but passes most of the
+time in CI. This is a false-positive in that the test SHOULD be
+failing all the time but some other test is causing a side-effect that
+it making it pass. This is sometimes harder to track down and reason
+about. minitest-bisect now has an "inverted" mode, where you run as
+normal but point it at a failing test by name. It will then validate
+that the test fails by itself but passes when run in full with the
+given seed. Then it tries to isolate and answer "what is the minimal
+combination of tests to run to # make this test pass?". For example:
+
+  % minitest_bisect example --seed=3 -n="/failing_test_name_regexp/"
+
+  reproducing w/ scoped failure (inverted run!)... in 0.06 sec
+  reproducing false positive... in 0.06 sec
+  # of culprit methods: 2 in 0.06 sec
+  # of culprit methods: 1 in 0.06 sec
+  # of culprit methods: 1 in 0.06 sec
+
+  Minimal methods found in 3 steps:
+
+  Culprit methods: ["TestBad1#test_make_it_go", "TestBad1#test_fail_without"]
+
+  /Users/ryan/.rubies/ruby-3.2.2/bin/ruby -Itest:lib -Ilib -e 'require "./example"' -- --seed 3  -n "/^(?:TestBad1#(?:test_make_it_go|test_fail_without))$/"
+
+  Final reproduction:
+
+  Run options: --seed 3 -n "/^(?:TestBad1#(?:test_make_it_go|test_fail_without))$/"
+
+  # Running:
+
+  ..
+
+  Finished in 0.000369s, 5420.0544 runs/s, 0.0000 assertions/s.
+
+  2 runs, 0 assertions, 0 failures, 0 errors, 0 skips
+
 == REQUIREMENTS:
 
-* minitest 5
+* minitest 5+
 
 == INSTALL:
 

data/Rakefile

@@ -40,11 +40,7 @@ def banner text
 end
 
 def run cmd
-  sh cmd do end
-end
-
-def req glob
-  Dir["#{glob}.rb"].map { |s| "require #{s.inspect}" }.join ";"
+  sh cmd do end # block form lets it fail w/o halting rake
 end
 
 task :repro => :isolate do
@@ -56,10 +52,10 @@ task :repro => :isolate do
   ruby = "ruby -I.:lib"
 
   banner "Original run that causes the test order dependency bug"
-  run "#{ruby} -e '#{req "example/test*"}' -- --seed 3911"
+  run "#{ruby} example.rb --seed 1"
 
   banner "Reduce the problem down to the minimal reproduction"
-  run "#{ruby} bin/minitest_bisect -Ilib --seed 3911 example/test*.rb"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 1 example.rb"
 end
 
 task :many => :isolate do
@@ -71,10 +67,31 @@ task :many => :isolate do
   ruby = "ruby -I.:lib"
 
   banner "Original run that causes the test order dependency bug"
-  run "#{ruby} -e '#{req "example-many/test*"}' -- --seed 27083"
+  run "#{ruby} ./example_many.rb --seed 2"
 
   banner "Reduce the problem down to the minimal reproduction"
-  run "#{ruby} bin/minitest_bisect -Ilib --seed 27083 example-many/test*.rb"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 2 example_many.rb"
+end
+
+task :inverse => :isolate do
+  unless ENV.key? "SLEEP" then
+    warn "NOTE: Defaulting to sleeping 0.01 seconds per test."
+    warn "NOTE: Use SLEEP=0 to disable or any other value to simulate your tests."
+  end
+
+  ruby = "ruby -I.:lib"
+
+  banner "Original run that passes (seed 1)"
+  run "#{ruby} example_inverse.rb --seed 1"
+
+  banner "Original run that *looks like* a test order dependency bug (seed 3)"
+  run "#{ruby} example_inverse.rb --seed 3"
+
+  banner "BAD bisection (tests fail by themselves) (seed 3)"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 3 example_inverse.rb"
+
+  banner "Reduce the passing run down to the minimal reproduction (seed 1)"
+  run "#{ruby} bin/minitest_bisect -Ilib --seed 1 example_inverse.rb -n=/TestBad4#test_bad4_4$/"
 end
 
 # vim: syntax=ruby

data/bin/minitest_bisect

@@ -1,4 +1,4 @@
-#!/usr/bin/ruby -w
+#!/usr/bin/env ruby -w
 
 require "minitest/bisect"
 

data/example.rb

@@ -0,0 +1,59 @@
+#!/usr/bin/env ruby -w
+
+require_relative "example_helper"
+
+# 800 tests, test test_bad4_4 fails if test_bad1_1 runs before it
+TestBad1 = create_test 1, 100, 1 => :infect
+TestBad2 = create_test 2, 100
+TestBad3 = create_test 3, 100
+TestBad4 = create_test 4, 100, 4 => :flunk
+TestBad5 = create_test 5, 100
+TestBad6 = create_test 6, 100
+TestBad7 = create_test 7, 100
+TestBad8 = create_test 8, 100
+
+# seed 1 == one fail
+# seed 3 == all pass
+
+# % SLEEP=0 ruby ./example.rb --seed 1
+#
+# and it fails, as expected. So we run it through minitest_bisect and see:
+#
+#   reproducing... in 0.14 sec
+#   verifying... in 0.06 sec
+#   # of culprit methods: 128 in 0.08 sec
+#   # of culprit methods: 64 in 0.07 sec
+#   # of culprit methods: 64 in 0.07 sec
+#   # of culprit methods: 32 in 0.06 sec
+#   # of culprit methods: 16 in 0.06 sec
+#   # of culprit methods: 8 in 0.06 sec
+#   # of culprit methods: 4 in 0.06 sec
+#   # of culprit methods: 4 in 0.06 sec
+#   # of culprit methods: 2 in 0.06 sec
+#   # of culprit methods: 1 in 0.06 sec
+#
+#   Minimal methods found in 10 steps:
+#
+#   Culprit methods: ["TestBad1#test_bad1_1", "TestBad4#test_bad4_4"]
+#
+#   /Users/ryan/.rubies/ruby-3.2.2/bin/ruby -Itest:lib -e 'require "././example.rb"' -- --seed 1  -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
+#
+#   Final reproduction:
+#
+#   Run options: --seed 1 -n "/^(?:TestBad1#(?:test_bad1_1)|TestBad4#(?:test_bad4_4))$/"
+#
+#   # Running:
+#
+#   .F
+#
+#   Finished in 0.001349s, 1482.5797 runs/s, 741.2898 assertions/s.
+#
+#     1) Failure:
+#   TestBad4#test_bad4_4 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example.rb:20]:
+#   muahahaha order dependency bug!
+#
+#   2 runs, 1 assertions, 1 failures, 0 errors, 0 skips
+#
+# and that's all there is to it! You now know that test_bad4_4 fails
+# only when paired with test_bad1_1 before it. You can now debug the
+# two methods and see what they're both modifying/dependent on.

data/{example-many/helper.rb → example_helper.rb}

@@ -1,4 +1,7 @@
-$hosed ||= 0
+require "minitest/autorun"
+
+$good = true
+$bomb = 0
 
 def create_test suffix, n_methods, bad_methods = {}
   raise ArgumentError, "Bad args" if Hash === n_methods
@@ -12,10 +15,16 @@ def create_test suffix, n_methods, bad_methods = {}
         sleep delay if delay > 0
 
         case bad_methods[n]
-        when true then
-          $hosed += 1
+        when :flunk then
+          flunk "muahahaha order dependency bug!" unless $good
+        when :infect then
+          $good = false
+        when :fix then
+          $good = true
+        when :tick then
+          $bomb += 1
         when Integer then
-          flunk "muahahaha order dependency bug!" if $hosed >= bad_methods[n]
+          flunk "muahahaha order dependency bug!" if $bomb >= bad_methods[n]
         else
           assert true
         end

data/example_inverse.rb

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby -w
+
+require_relative "example_helper"
+
+$good = false
+
+# 800 tests, test test_bad4_4 passes if test_bad1_1 runs before it
+TestBad1 = create_test 1, 100, 1 => :fix
+TestBad2 = create_test 2, 100
+TestBad3 = create_test 3, 100
+TestBad4 = create_test 4, 100, 4 => :flunk
+TestBad5 = create_test 5, 100
+TestBad6 = create_test 6, 100
+TestBad7 = create_test 7, 100
+TestBad8 = create_test 8, 100
+
+# seed 1 == all pass
+# seed 3 == one fail
+
+# UNLIKE the scenario spelled out in example.rb...
+#
+# % SLEEP=0 ruby ./example_inverse.rb --seed 1
+#
+# passes, but
+#
+# % SLEEP=0 ruby ./example_inverse.rb --seed 3
+#
+# has 1 failure! So we run that through minitest_bisect:
+#
+# % SLEEP=0 ruby -Ilib bin/minitest_bisect ./example_inverse.rb --seed 3
+#
+# and see:
+#
+#   Tests fail by themselves. This may not be an ordering issue.
+#
+# followed by a result that doesn't actually lead to the failure.
+#
+# Doing a normal run of minitest_bisect in this scenario doesn't
+# detect anything actually relevant. This is because we're not dealing
+# with a false *negative*, we're dealing with a false *positive*. The
+# question is "what makes this test pass?". So, we run it again with
+# the passing seed but this time point it at the failing test by name:
+#
+# % SLEEP=0 ruby -Ilib bin/minitest_bisect ./example_inverse.rb --seed 1 -n=/TestBad4#test_bad4_4$/
+#
+# and it outputs:
+#
+#   Culprit methods: ["TestBad1#test_bad1_1", "TestBad4#test_bad4_4"]
+
+# and shows a minimized run with the 2 passing tests. You now know
+# that test_bad4_4 passes only when paired with test_bad1_1 before it.
+# You can now debug the two methods and see what they're both
+# modifying/dependent on.

data/example_many.rb

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby -w
+
+require_relative "example_helper"
+
+TestBad1 = create_test 1, 100,  1 => :tick
+TestBad2 = create_test 2, 100
+TestBad3 = create_test 3, 100, 72 => :tick
+TestBad4 = create_test 4, 100
+TestBad5 = create_test 5, 100, 17 => :tick
+TestBad6 = create_test 6, 100
+TestBad7 = create_test 7, 100
+TestBad8 = create_test 8, 100, 43 => 3
+
+# seed 1 == all pass
+# seed 2 == one fail
+
+# % ruby example_many.rb --seed 2
+
+# and it fails, as expected. So we run it through minitest_bisect and see:
+
+# % ruby -Ilib bin/minitest_bisect example_many.rb --seed 2
+
+#   reproducing... in 0.15 sec
+#   verifying... in 0.06 sec
+#   # of culprit methods: 256 in 0.09 sec
+#   ... 82 more bisections ...
+#   # of culprit methods: 3 in 0.06 sec
+#
+#   Minimal methods found in 84 steps:
+#
+#   Culprit methods: ["TestBad3#test_bad3_72", "TestBad5#test_bad5_17", "TestBad1#test_bad1_1", "TestBad8#test_bad8_43"]
+#
+#   /Users/ryan/.rubies/ruby-3.2.2/bin/ruby -Itest:lib -e 'require "./example_many.rb"' -- --seed 2  -n "/^(?:TestBad3#(?:test_bad3_72)|TestBad5#(?:test_bad5_17)|TestBad1#(?:test_bad1_1)|TestBad8#(?:test_bad8_43))$/"
+#
+#   Final reproduction:
+#
+#   Run options: --seed 2 -n "/^(?:TestBad3#(?:test_bad3_72)|TestBad5#(?:test_bad5_17)|TestBad1#(?:test_bad1_1)|TestBad8#(?:test_bad8_43))$/"
+#
+#   # Running:
+#
+#   ...F
+#
+#   Finished in 0.001404s, 2849.0028 runs/s, 712.2507 assertions/s.
+#
+#     1) Failure:
+#   TestBad8#test_bad8_43 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example_many.rb:20]:
+#   muahahaha order dependency bug!
+#
+#   4 runs, 1 assertions, 1 failures, 0 errors, 0 skips
+#
+# and that's all there is to it! You now know that the failing test
+# fails only when paired with the other 3 tests before it. You can now
+# debug the 4 methods and see what they're all modifying/dependent on.

data/lib/minitest/bisect.rb

@@ -4,10 +4,15 @@ require "shellwords"
 require "rbconfig"
 require "path_expander"
 
+module Minitest; end # :nodoc:
+
+##
+# Minitest::Bisect helps you isolate and debug random test failures.
+
 class Minitest::Bisect
-  VERSION = "1.6.0"
+  VERSION = "1.7.0" # :nodoc:
 
-  class PathExpander < ::PathExpander
+  class PathExpander < ::PathExpander # :nodoc:
     TEST_GLOB = "**/{test_*,*_test,spec_*,*_spec}.rb" # :nodoc:
 
     attr_accessor :rb_flags
@@ -39,7 +44,7 @@ class Minitest::Bisect
   end
 
   mtbv = ENV["MTB_VERBOSE"].to_i
-  SHH = case
+  SHH = case # :nodoc:
         when mtbv == 1 then " > /dev/null"
         when mtbv >= 2 then nil
         else " > /dev/null 2>&1"
@@ -51,21 +56,48 @@ class Minitest::Bisect
               RbConfig::CONFIG['ruby_install_name'] +
                 RbConfig::CONFIG['EXEEXT']).sub(/.*\s.*/m, '"\&"')
 
-  attr_accessor :tainted, :failures, :culprits, :seen_bad
+  ##
+  # True if this run has seen a failure.
+
+  attr_accessor :tainted
   alias :tainted? :tainted
 
+  ##
+  # Failures seen in this run. Shape:
+  #
+  #   {"file.rb"=>{"Class"=>["test_method1", "test_method2"] ...} ...}
+
+  attr_accessor :failures
+
+  ##
+  # An array of tests seen so far. NOT cleared by #reset.
+
+  attr_accessor :culprits
+
+  attr_accessor :seen_bad # :nodoc:
+
+  ##
+  # Top-level runner. Instantiate and call +run+, handling exceptions.
+
   def self.run files
     new.run files
   rescue => e
     warn e.message
+    warn "Try running with MTB_VERBOSE=2 to verify."
     exit 1
   end
 
+  ##
+  # Instantiate a new Bisect.
+
   def initialize
     self.culprits = []
     self.failures = Hash.new { |h, k| h[k] = Hash.new { |h2, k2| h2[k2] = [] } }
   end
 
+  ##
+  # Reset per-bisect-run variables.
+
   def reset
     self.seen_bad = false
     self.tainted  = false
@@ -73,6 +105,10 @@ class Minitest::Bisect
     # not clearing culprits on purpose
   end
 
+  ##
+  # Instance-level runner. Handles Minitest::Server, argument
+  # processing, and invoking +bisect_methods+.
+
   def run args
     Minitest::Server.run self
 
@@ -83,9 +119,9 @@ class Minitest::Bisect
 
     files = expander.process
     rb_flags = expander.rb_flags
-    mt_flags += ["--server", $$]
+    mt_flags += ["--server", $$.to_s]
 
-    cmd = bisect_methods build_files_cmd(files, rb_flags, mt_flags)
+    cmd = bisect_methods files, rb_flags, mt_flags
 
     puts "Final reproduction:"
     puts
@@ -95,40 +131,87 @@ class Minitest::Bisect
     Minitest::Server.stop
   end
 
-  def bisect_methods cmd
-    time_it "reproducing...", build_methods_cmd(cmd)
-
-    unless tainted? then
-      $stderr.puts "Reproduction run passed? Aborting."
-      abort "Try running with MTB_VERBOSE=2 to verify."
+  ##
+  # Normal: find "what is the minimal combination of tests to run to
+  #         make X fail?"
+  #
+  # Run with: minitest_bisect ... --seed=N
+  #
+  # 1. Verify the failure running normally with the seed.
+  #    2. If no failure, punt.
+  #    3. If no passing tests before failure, punt. (No culprits == no debug)
+  # 4. Verify the failure doesn't fail in isolation.
+  #    5. If it still fails by itself, warn that it might not be an ordering
+  #       issue.
+  # 6. Cull all tests after the failure, they're not involved.
+  # 7. Bisect the culprits + bad until you find a minimal combo that fails.
+  # 8. Display minimal combo by running one last time.
+  #
+  # Inverted: find "what is the minimal combination of tests to run to
+  #           make this test pass?"
+  #
+  # Run with: minitest_bisect ... --seed=N -n="/failing_test_name_regexp/"
+  #
+  # 1. Verify the failure by running normally w/ the seed and -n=/.../
+  #    2. If no failure, punt.
+  # 3. Verify the passing case by running everything.
+  #    4. If failure, punt. This is not a false positive.
+  # 5. Cull all tests after the bad test from #1, they're not involved.
+  # 6. Bisect the culprits + bad until you find a minimal combo that passes.
+  # 7. Display minimal combo by running one last time.
+
+  def bisect_methods files, rb_flags, mt_flags
+    bad_names, mt_flags = mt_flags.partition { |s| s =~ /^(?:-n|--name)/ }
+    normal   = bad_names.empty?
+    inverted = !normal
+
+    if inverted then
+      time_it "reproducing w/ scoped failure (inverted run!)...", build_methods_cmd(build_files_cmd(files, rb_flags, mt_flags + bad_names))
+      raise "No failures. Probably not a false positive. Aborting." if failures.empty?
+      bad = map_failures
     end
 
-    bad = map_failures
+    cmd = build_files_cmd(files, rb_flags, mt_flags)
 
-    raise "Nothing to verify against because every test failed. Aborting." if
-      culprits.empty? && seen_bad
+    msg = normal ? "reproducing..." : "reproducing false positive..."
+    time_it msg, build_methods_cmd(cmd)
 
-    time_it "verifying...", build_methods_cmd(cmd, [], bad)
+    if normal then
+      raise "Reproduction run passed? Aborting." unless tainted?
+      raise "Verification failed. No culprits? Aborting." if culprits.empty? && seen_bad
+    else
+      raise "Reproduction failed? Not false positive. Aborting." if tainted?
+      raise "Verification failed. No culprits? Aborting." if culprits.empty? || seen_bad
+    end
+
+    if normal then
+      bad = map_failures
 
-    new_bad = map_failures
+      time_it "verifying...", build_methods_cmd(cmd, [], bad)
 
-    if bad == new_bad then
-      warn "Tests fail by themselves. This may not be an ordering issue."
+      new_bad = map_failures
+
+      if bad == new_bad then
+        warn "Tests fail by themselves. This may not be an ordering issue."
+      end
     end
 
+    idx = culprits.index bad.first
+    self.culprits = culprits.take idx+1 if idx # cull tests after bad
+
     # culprits populated by initial reproduction via minitest/server
     found, count = culprits.find_minimal_combination_and_count do |test|
       prompt = "# of culprit methods: #{test.size}"
 
       time_it prompt, build_methods_cmd(cmd, test, bad)
 
-      self.tainted?
+      normal == tainted? # either normal and failed, or inverse and passed
     end
 
     puts
     puts "Minimal methods found in #{count} steps:"
     puts
-    puts "Culprit methods: %p" % [found]
+    puts "Culprit methods: %p" % [found + bad]
     puts
     cmd = build_methods_cmd cmd, found, bad
     puts cmd.sub(/--server \d+/, "")
@@ -136,14 +219,14 @@ class Minitest::Bisect
     cmd
   end
 
-  def time_it prompt, cmd
+  def time_it prompt, cmd # :nodoc:
     print prompt
     t0 = Time.now
     system "#{cmd} #{SHH}"
     puts " in %.2f sec" % (Time.now - t0)
   end
 
-  def map_failures
+  def map_failures # :nodoc:
     # from: {"file.rb"=>{"Class"=>["test_method1", "test_method2"]}}
     #   to: ["Class#test_method1", "Class#test_method2"]
     failures.values.map { |h|
@@ -151,15 +234,31 @@ class Minitest::Bisect
     }.flatten.sort
   end
 
-  def build_files_cmd culprits, rb, mt
-    reset
-
+  def build_files_cmd culprits, rb, mt # :nodoc:
     tests = culprits.flatten.compact.map { |f| %(require "./#{f}") }.join " ; "
 
     %(#{RUBY} #{rb.shelljoin} -e '#{tests}' -- #{mt.map(&:to_s).shelljoin})
   end
 
-  def build_re bad
+  def build_methods_cmd cmd, culprits = [], bad = nil # :nodoc:
+    reset
+
+    if bad then
+      re = build_re culprits + bad
+
+      cmd += " -n \"#{re}\"" if bad
+    end
+
+    if ENV["MTB_VERBOSE"].to_i >= 1 then
+      puts
+      puts cmd
+      puts
+    end
+
+    cmd
+  end
+
+  def build_re bad # :nodoc:
     re = []
 
     # bad by class, you perv
@@ -179,36 +278,18 @@ class Minitest::Bisect
     "/^(?:#{re})$/"
   end
 
-  def re_escape str
+  def re_escape str # :nodoc:
     str.gsub(/([`'"!?&\[\]\(\)\{\}\|\+])/, '\\\\\1')
   end
 
-  def build_methods_cmd cmd, culprits = [], bad = nil
-    reset
-
-    if bad then
-      re = build_re culprits + bad
-
-      cmd += " -n \"#{re}\"" if bad
-    end
-
-    if ENV["MTB_VERBOSE"].to_i >= 1 then
-      puts
-      puts cmd
-      puts
-    end
-
-    cmd
-  end
-
   ############################################################
   # Server Methods:
 
-  def minitest_start
+  def minitest_start # :nodoc:
     self.failures.clear
   end
 
-  def minitest_result file, klass, method, fails, assertions, time
+  def minitest_result file, klass, method, fails, assertions, time # :nodoc:
     fails.reject! { |fail| Minitest::Skip === fail }
 
     if fails.empty? then

data/lib/minitest/find_minimal_combination.rb

@@ -1,10 +1,13 @@
 #!/usr/bin/ruby -w
 
-module Minitest; end
+##
+# Finds the minimal combination of a collection of items that satisfy
+# +test+.
 
 class ComboFinder
   ##
-  # Find the minimal combination of a collection of items that satisfy +test+.
+  # Find the minimal combination of a collection of items that satisfy
+  # +test+.
   #
   # If you think of the collection as a binary tree, this algorithm
   # does a breadth first search of the combinations that satisfy
@@ -79,11 +82,11 @@ class ComboFinder
     ary
   end
 
-  def d s = ""
+  def d s = "" # :nodoc:
     warn s if ENV["MTB_DEBUG"]
   end
 
-  def cache_result result, data, cache
+  def cache_result result, data, cache # :nodoc:
     cache[data] = true
 
     return result if result
@@ -103,7 +106,7 @@ class ComboFinder
   end
 end
 
-class Array
+class Array # :nodoc:
   ##
   # Find the minimal combination of a collection of items that satisfy +test+.
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: minitest-bisect
 version: !ruby/object:Gem::Version
-  version: 1.6.0
+  version: 1.7.0
 platform: ruby
 authors:
 - Ryan Davis
@@ -10,9 +10,9 @@ bindir: bin
 cert_chain:
 - |
   -----BEGIN CERTIFICATE-----
-  MIIDPjCCAiagAwIBAgIBBjANBgkqhkiG9w0BAQsFADBFMRMwEQYDVQQDDApyeWFu
+  MIIDPjCCAiagAwIBAgIBBzANBgkqhkiG9w0BAQsFADBFMRMwEQYDVQQDDApyeWFu
   ZC1ydWJ5MRkwFwYKCZImiZPyLGQBGRYJemVuc3BpZGVyMRMwEQYKCZImiZPyLGQB
-  GRYDY29tMB4XDTIxMTIyMzIzMTkwNFoXDTIyMTIyMzIzMTkwNFowRTETMBEGA1UE
+  GRYDY29tMB4XDTIzMDEwMTA3NTExN1oXDTI0MDEwMTA3NTExN1owRTETMBEGA1UE
   AwwKcnlhbmQtcnVieTEZMBcGCgmSJomT8ixkARkWCXplbnNwaWRlcjETMBEGCgmS
   JomT8ixkARkWA2NvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALda
   b9DCgK+627gPJkB6XfjZ1itoOQvpqH1EXScSaba9/S2VF22VYQbXU1xQXL/WzCkx
@@ -22,14 +22,14 @@ cert_chain:
   qhtV7HJxNKuPj/JFH0D2cswvzznE/a5FOYO68g+YCuFi5L8wZuuM8zzdwjrWHqSV
   gBEfoTEGr7Zii72cx+sCAwEAAaM5MDcwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAw
   HQYDVR0OBBYEFEfFe9md/r/tj/Wmwpy+MI8d9k/hMA0GCSqGSIb3DQEBCwUAA4IB
-  AQCKB5jfsuSnKb+t/Wrh3UpdkmX7TrEsjVmERC0pPqzQ5GQJgmEXDD7oMgaKXaAq
-  x2m+KSZDrqk7c8uho5OX6YMqg4KdxehfSLqqTZGoeV78qwf/jpPQZKTf+W9gUSJh
-  zsWpo4K50MP+QtdSbKXZwjAafpQ8hK0MnnZ/aeCsW9ov5vdXpYbf3dpg6ADXRGE7
-  lQY2y1tJ5/chqu6h7dQmnm2ABUqx9O+JcN9hbCYoA5i/EeubUEtFIh2w3SpO6YfB
-  JFmxn4h9YO/pVdB962BdBNNDia0kgIjI3ENnkLq0dKpYU3+F3KhEuTksLO0L6X/V
-  YsuyUzsMz6GQA4khyaMgKNSD
+  AQAkg3y+PBnBAPWdxxITm5sPHqdWQgSyCpRA20o4LTuWr8BWhSXBkfQNa7cY6fOn
+  xyM34VPzBFbExv6XOGDfOMFBVaYTHuN9peC/5/umL7kLl+nflXzL2QA7K6LYj5Bg
+  sM574Onr0dZDM6Vn69bzQ7rBIFDfK/OhlPzqKZad4nsdcsVH8ODCiT+ATMIZyz5K
+  WCnNtqlyiWXI8tdTpahDgcUwfcN/oN7v4K8iU5IbLJX6HQ5DKgmKjfb6XyMth16k
+  ROfWo9Uyp8ba/j9eVG14KkYRaLydAY1MNQk2yd3R5CGfeOpD1kttxjoypoUJ2dOG
+  nsNBRuQJ1UfiCG97a6DNm+Fr
   -----END CERTIFICATE-----
-date: 2022-05-23 00:00:00.000000000 Z
+date: 2023-07-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest-server
@@ -99,14 +99,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.23'
+        version: '4.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.23'
+        version: '4.0'
 description: |-
   Hunting down random test failures can be very very difficult,
   sometimes impossible, but minitest-bisect makes it easy.
@@ -135,24 +135,10 @@ files:
 - README.rdoc
 - Rakefile
 - bin/minitest_bisect
-- example-many/helper.rb
-- example-many/test_bad1.rb
-- example-many/test_bad2.rb
-- example-many/test_bad3.rb
-- example-many/test_bad4.rb
-- example-many/test_bad5.rb
-- example-many/test_bad6.rb
-- example-many/test_bad7.rb
-- example-many/test_bad8.rb
-- example/helper.rb
-- example/test_bad1.rb
-- example/test_bad2.rb
-- example/test_bad3.rb
-- example/test_bad4.rb
-- example/test_bad5.rb
-- example/test_bad6.rb
-- example/test_bad7.rb
-- example/test_bad8.rb
+- example.rb
+- example_helper.rb
+- example_inverse.rb
+- example_many.rb
 - lib/minitest/bisect.rb
 - lib/minitest/find_minimal_combination.rb
 - test/minitest/test_bisect.rb
@@ -179,7 +165,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.12
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: Hunting down random test failures can be very very difficult, sometimes impossible,

metadata.gz.sig

@@ -1,4 +1,3 @@
-8=o���PLv8�0�p�
-��dI�{%5���!ggr=@�<�\�Fe��M�_�:	vs�Zk\,�
-L�V0��~���gщ�,��a/N�yR�,����>U]�N�L"VT������"c�
?;�D�����],��H��™����r�L쇊�9��`��_nI�:�v��'y4�
-�ݸ:1��
+�-���J�y��R�sT�G���t�
Q�l�%s���ꢅ��	���9�����L)T������E���.����1��+�]�Ήۙ�L\
+oG1

+�^3�P�ˢ|�LԒ�΍Z-�|&u�ݽ�=�aHK�e�1,�z}�\Y�;�y	�5�~X�<˳� <sAX{�S�|�IȪ X
���S*��&\���z�Z�lJ^<>߬4��%�.�F{dI(3���P��u��S2��S;�{���L!4O

data/example/helper.rb

@@ -1,25 +0,0 @@
-$hosed ||= false
-
-def create_test suffix, n_methods, bad_methods = {}
-  raise ArgumentError, "Bad args" if Hash === n_methods
-
-  delay = (ENV["SLEEP"] || 0.01).to_f
-
-  Class.new(Minitest::Test) do
-    n_methods.times do |n|
-      n += 1
-      define_method "test_bad#{suffix}_#{n}" do
-        sleep delay if delay > 0
-
-        case bad_methods[n]
-        when true then
-          flunk "muahahaha order dependency bug!" if $hosed
-        when false then
-          $hosed = true
-        else
-          assert true
-        end
-      end
-    end
-  end
-end

data/example/test_bad1.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad1 = create_test 1, 100, 1 => false

data/example/test_bad2.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad2 = create_test 2, 100

data/example/test_bad3.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad3 = create_test 3, 100

data/example/test_bad4.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad4 = create_test 4, 100, 4 => true

data/example/test_bad5.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad5 = create_test 5, 100

data/example/test_bad6.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad6 = create_test 6, 100

data/example/test_bad7.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad7 = create_test 7, 100

data/example/test_bad8.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad8 = create_test 8, 100

data/example-many/test_bad1.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad1 = create_test 1, 100, 1 => true

data/example-many/test_bad2.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad2 = create_test 2, 100

data/example-many/test_bad3.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad3 = create_test 3, 100, 72 => true

data/example-many/test_bad4.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad4 = create_test 4, 100

data/example-many/test_bad5.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad5 = create_test 5, 100, 17 => true

data/example-many/test_bad6.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad6 = create_test 6, 100

data/example-many/test_bad7.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad7 = create_test 7, 100

data/example-many/test_bad8.rb

@@ -1,4 +0,0 @@
-require "minitest/autorun"
-require_relative "helper"
-
-TestBad8 = create_test 8, 100, 43 => 3