deflake 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bb6b0172e3743811b5c1d8cc3fa64bb8c4970c490615c3faef9e7b21156cdc3d
+  data.tar.gz: 992063be61424751f86dcdf6ad37e7acf895e10d2365785189a8bd691656d7d0
+SHA512:
+  metadata.gz: 3aece386132f23ef51f9c2fd828dbc7c123d03103affae67836252fce76f173f310bc30f794188eab4430936344865b9ec2542a322c9ed2c52b6cdcc8a5b9ef5
+  data.tar.gz: 42363d3e5b5a486acc2a230e7c499c4f0ff4bc729fda1654c9e5db44361180dd93da0afbfa590a31e85d008a086f126184704787d2ef164855ca4cfe34d6e6ed

data/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.1.0] - 2026-11-08
+
+First public release.
+
+- `deflake TEST_PATH[:LINE]` runs a Minitest or RSpec file many times, each in a fresh process with a new seed, and reports how often every test failed, with errors grouped by message and the seeds that reproduce them.
+- Every failing test is run again on its own. A test that passes alone is reported as order-dependent, and deflake finds the tests that break it with its own bisect, for Minitest 5 and 6 and for RSpec. A failure must repeat to count, so a test that fails at random is reported as such, without culprits.
+- Errors outside of any test, such as in an RSpec `after(:context)` hook, are reported too, and so are leaks of class-level hooks.
+- Leak detection: every test is checked for global state it leaves behind, namely ENV, the clock (`travel_to` or `freeze_time` without `travel_back`), global variables, `Rails.cache`, `ActiveSupport::CurrentAttributes`, database rows and the ActiveJob test queue. Projects can add their own detectors.
+- `--jobs N` runs test processes in parallel, each with its own `TEST_ENV_NUMBER`.
+- `--timeout SECONDS` stops a test process that runs longer, for tests that hang.
+- `--format json` for CI and other tools.
+- Exit code 0 when every test passed and nothing leaked, 1 otherwise, 2 on usage errors, for a file without tests, or when the tests could not run. Ctrl-C stops the test processes and ends deflake as interrupted.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Роман Беляев
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,140 @@
+# deflake
+
+Find flaky tests in Minitest and RSpec and explain why they fail.
+
+A flaky test fails once in a while, someone adds a retry, and the cause stays.
+deflake runs a suspicious test many times and answers three questions:
+
+1. **How often does it fail**, and with which errors and seeds?
+2. **Does it fail on its own**, or only after some other test? If the latter, which one?
+3. **What global state does a test leave behind**: ENV, the clock, globals, `Rails.cache`, `Current` attributes, database rows, enqueued jobs?
+
+## Installation
+
+Add it to the test group of your Gemfile:
+
+```ruby
+group :test do
+  gem "deflake", require: false
+end
+```
+
+deflake needs Ruby 3.3 or newer and works with Minitest 5.11 or newer (6 included) and RSpec 3.3 or newer. It has no dependencies of its own.
+
+## Usage
+
+Run deflake from the root of your project:
+
+```bash
+bundle exec deflake test/models/user_test.rb --runs 50
+bundle exec deflake spec/models/user_spec.rb:42 --runs 50 --jobs 4
+```
+
+```
+OrderTest#test_pollutes: failed 0 of 10 runs (0%)
+  leaks: $polluted was changed (10 of 10 runs)
+OrderTest#test_victim: failed 6 of 10 runs (60%)
+  verdict: fails only after other tests (alone: failed 0 of 10 runs (0%))
+  culprit: OrderTest#test_pollutes
+  reproduce: PARALLEL_WORKERS=1 MT_CPU=1 ruby -Ilib -Itest test/models/order_test.rb --seed 16166
+  seeds 16166, 35999, 58287, 35391, 30777, 1131:
+    polluted by another test
+```
+
+What happens:
+
+1. **The whole file runs `--runs` times** (10 by default), every time in a fresh `ruby` process with a new random seed. Running the whole file, not just one test, is what makes order-dependent failures show up. A test that takes its process down (`exit`, a crash, a signal) counts as failed in that run.
+2. **Every test that failed runs again on its own**, as many times. The verdict is *fails on its own* if it still fails, which points at time, randomness or a race inside the test, or *fails only after other tests* if it does not, or if it skips itself every time it runs alone. When it cannot run alone at all, as with errors outside of tests or a file that breaks before any test whenever one test runs on its own, the verdict is *failed, could not run it alone*. The errors of the runs alone are shown too when the runs of the file did not have them.
+3. **Order-dependent failures are bisected** with one of the failing seeds, down to a smallest set of tests that break the victim; when two tests break it independently, one set is named. deflake has the test framework run just those tests in the recorded order, so hooks around a whole class still run, and watches only the victim, so other failing tests do not get in the way. A failure counts only when it happens twice in a row, and the set found must break the victim again at the end, so a test that fails at random gets no culprits: when the failure does not happen again with the same seed and order, the verdict is *fails at random, not after particular tests*, with the command that runs that seed. Tests in a class without a name cannot be put in an order, so when the file has some and the failure does not happen again without them, the verdict stays *fails only after other tests*, and deflake says it could not look among them. When the victim fails with that seed with no test before it, the verdict is *fails on its own*, with the command that shows it. Up to three failing seeds are tried, the first seed of every error first. A test that fails on its own is bisected too when the runs of the file have an error it never had alone. A culprit that two other tests of its group replace, because the state comes from a hook of the group such as `before(:context)` or `before_all`, is named as such. A victim stopped by `--timeout` does not count as failing in a bisect, as fewer tests before it would not reach the limit. This works on Minitest 5 and 6 alike, `Minitest::Spec` included, and on RSpec, which keeps the relative order of any subset of examples for a seed.
+4. **Every test is checked for leaks**, tests that skip themselves included: a snapshot of global state before the test, setup included, and after it, teardown, transaction rollback and whatever Rails does around a test (such as the executor clearing `Current`) included. Whatever changed is reported. So is what a whole Minitest class or top-level RSpec group leaves behind beyond the leaks of its tests, such as the state a `before_all` or `before(:context)` hook sets up and never puts back: it shows up under *errors outside of tests*.
+
+A process that fails while every test in it passed is reported as *errors outside of tests*, with the end of its output. An error RSpec reports outside of examples, such as in an `after(:context)` hook, is reported with its class and message, even when an example failed in the same run. So is a run that breaks before its first test, such as a `before(:suite)` hook that fails now and then: the other runs still count. A failure deflake cannot run again by name, such as a class-level hook (minitest-hooks' `before_all`) that reports its error as a result, shows up there too. Only when no run gets to a test does deflake give up with exit code 2, and say how the test process ended, for example that it timed out.
+
+With `TEST_PATH:LINE` deflake still runs the whole file, but reports only the tests on that line, and errors outside of tests, which may be what stopped them. In Minitest that is the test declared closest above the line, whatever its style: `def test_...`, Rails' `test "..."`, `it "..."` or a macro of the project that defines a test; a line in a test of a class without a name is refused, as such a test cannot run alone. In RSpec it is what `rspec TEST_PATH:LINE` runs: the example on that line, or every example of a `describe` or `context` on that line, found with a dry run that runs no example and no hook. A test on the line that never started in any run, for example because an earlier test always takes the process down or a filter of the file leaves it out, makes deflake say so and exit with 1; the JSON report lists it with the verdict `not_run`.
+
+Files ending in `_spec.rb` go to RSpec, unless they mention Minitest (a `minitest_helper` counts, and so do expectations such as `_(x).must_equal`) and neither RSpec nor `rails_helper`: those are `Minitest::Spec` files.
+
+A Minitest test is known by its class and method, `UserTest#test_name`. When two classes have the same name, as two `describe` blocks with the same description do, the second one's tests are `Name (2)#test_...`. A failing test in a class without a name, made with `Class.new` and never given one, cannot be run again by name: its failure shows up under *errors outside of tests*. A test a file brings in from another one with `require_relative` belongs to that other file, not to the line of the `require_relative`.
+
+### Options
+
+| Option | Default | |
+| --- | --- | --- |
+| `-n`, `--runs N` | 10 | How many times to run the file |
+| `-j`, `--jobs N` | 1 | How many test processes to run at once |
+| `-t`, `--timeout SECONDS` | none | Stop a test process that runs longer, in whole seconds; the test it was running counts as failed |
+| `-f`, `--format FORMAT` | `text` | `text` or `json` |
+
+Exit code: 0 when every test passed and nothing leaked (or every test skipped itself), 1 otherwise, 2 on usage errors (a missing or extra `TEST_PATH`, `--runs` or `--jobs` below 1), when the file has no tests or a filter of the project leaves them all out, or when the tests could not run (deflake then shows their output). When interrupted with Ctrl-C, deflake stops its test processes and ends as an interrupted program does, so a shell loop around it stops too.
+
+### Parallel runs and databases
+
+With `--jobs` several processes run at once. Each gets `TEST_ENV_NUMBER` set to `""`, `"2"`, `"3"`... like [parallel_tests](https://github.com/grosser/parallel_tests), so a Rails app can give each its own database:
+
+```yaml
+# config/database.yml
+test:
+  database: myapp_test<%= ENV["TEST_ENV_NUMBER"] %>
+```
+
+Every test process runs in a session of its own, without a terminal, as in CI: its input is empty and its output is not a TTY, so a test that skips itself unless `$stdin.tty?` always skips under deflake. In RSpec, deflake runs every example even when the project asks for `fail_fast`, and leaves out the profile; options in `SPEC_OPTS` that set the seed, the order, the formatters or `--fail-fast` are left out too, as deflake sets them itself.
+
+deflake sets `PARALLEL_WORKERS=1` and `MT_CPU=1` for every test process, so neither Rails' `parallelize` nor Minitest's `parallelize_me!` runs tests at once: the order a seed gives stays the same and the leaks of one test do not show up in another. To rerun a seed by hand, set them too, as the `reproduce:` command does; without `PARALLEL_WORKERS=1` Rails gives the same seed a different order.
+
+### Leak detectors
+
+| Detector | Reports |
+| --- | --- |
+| ENV | Variables set, changed or removed |
+| Clock | `Time.now` frozen or moved away from the real clock: `freeze_time`, `travel_to` or Timecop left on, Timecop's `mock_process_clock` included. Where the clock ends up counts, so a test that brings back a clock another test moved leaves nothing behind |
+| Globals | Global variables assigned or changed in place (not Ruby's own, such as `$CHILD_STATUS`). The whole content of what they hold is compared, as `Marshal` dumps it. An object `Marshal` cannot dump, such as a client holding a `Mutex` or a connection, counts by identity, so what it sets up inside on first use is not a leak; arrays, hashes, structs, sets and `StringIO` are still compared part by part, and queues by their size and whether they are closed. The order of a hash or a set does not matter. A value of any kind works, even a mock without Kernel's methods. `Marshal` dumps every global twice per test, so a global holding a lot of data makes every test slower |
+| Cache | New keys in `Rails.cache`, when it is a `MemoryStore` |
+| Current | `ActiveSupport::CurrentAttributes` left set; attributes with a proc default are not compared. A default changed in place, such as a list inside it, is reported too |
+| Rows | Rows created or deleted outside a rolled back transaction, in every database connected before the test; fixtures loaded during a test are not counted, nor fixtures a test without a transaction deletes again. The counts bypass ActiveRecord's query cache |
+| Jobs | Jobs left in the ActiveJob test queue. `ActiveJob::TestHelper` empties the queue before each test that includes it, so with every test including it these reports are noise: remove the detector as shown below |
+
+Detectors that do not apply, for example Rows in an app without ActiveRecord, stay quiet. A detector is any object with two methods: `snapshot` returns the current state, or `nil` when it does not apply, and `diff(before, after)` returns one message per leak. Add your own in `test_helper.rb` or `spec_helper.rb`:
+
+```ruby
+if ENV["DEFLAKE_OUTPUT"]
+  require "deflake/leaks"
+
+  class FeatureFlagLeaks
+    def snapshot = FeatureFlags.enabled.sort
+    def diff(before, after) = (after - before).map { |flag| "feature flag #{flag} left on" }
+  end
+
+  Deflake::Leaks.detectors << FeatureFlagLeaks.new
+end
+```
+
+`DEFLAKE_OUTPUT` is set only in the processes deflake starts, so your regular test runs are not affected. The Rows detector counts rows in every table before and after each test; if that is too slow for a big schema, remove it the same way: `Deflake::Leaks.detectors.reject! { |detector| detector.is_a?(Deflake::Leaks::Rows) }`. The same goes for `Deflake::Leaks::Jobs`.
+
+### JSON report
+
+`--format json` prints one object per test that ran or leaked (a test that skipped itself in every run and leaked nothing is left out, as in the text report) with `name`, `id`, `verdict` (`stable`, `failed`, `fails_on_its_own`, `order_dependent`, `random`, or `not_run` for a test on the line that never started; `failed` means it could not run alone), `runs` and `isolated_runs` counts (`isolated_runs` is `null` when the test never failed or could not run alone, as errors outside of tests cannot) (`total`, `failures`, `failure_rate` and `skipped`: `total` leaves out the runs where the test skipped itself), `errors` with their seeds, `leaks` (with `runs`, counting the runs where the test skipped itself too) and `culprits` (`ids`, `names`, `command`, `alone` when the test fails with no test before it, `hidden` when tests in a class without a name may be to blame, and `group_wide`). The format may change until 1.0.
+
+## Limitations
+
+- deflake looks at one file at a time. A test broken by a test in another file shows up as failing only when the two files run together, which deflake does not do yet.
+- Minitest files run as `ruby -Ilib -Itest FILE` (`lib` when there is one, and `-Ispec` too when there is a `spec` directory), RSpec files with RSpec's own defaults. A project that needs a different command is not supported yet.
+- A Minitest error outside of tests is seen only in a run where every test passed: a failing test also makes the process fail.
+- Leak detection sees what changed in the test process. It cannot see changes made by other processes, such as a browser in a system test.
+
+## Development
+
+```bash
+bin/setup          # install dependencies
+bundle exec rake   # run tests and RuboCop
+```
+
+To test against a specific Minitest major version:
+
+```bash
+MINITEST_VERSION="~> 6.0" bundle update minitest
+MINITEST_VERSION="~> 6.0" bundle exec rake test
+```
+
+## License
+
+MIT, see [LICENSE.txt](LICENSE.txt).

data/exe/deflake

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "deflake"
+
+code = Deflake::CLI.new(ARGV).run
+# After Ctrl-C, end the way an interrupted program does, so a shell loop around deflake stops too
+if code == 130
+  # SYSTEM_DEFAULT: Ruby's own handler would raise Interrupt and print its backtrace
+  trap("INT", "SYSTEM_DEFAULT")
+  Process.kill("INT", Process.pid)
+  # In case the signal arrives a moment later
+  sleep 1
+end
+exit code

data/lib/deflake/adapters/minitest.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require "shellwords"
+require "tmpdir"
+
+module Deflake
+  module Adapters
+    # Runs a Minitest file, optionally only some of its tests, by ids such as "Class#test_name"
+    module Minitest
+      module_function
+
+      # Which tests to run goes through env. lib and test are on the load path as with rake test,
+      # and spec too when there is one, for Minitest::Spec files that require spec_helper
+      def command(path, seed:, **)
+        [RbConfig.ruby, "-I", LIB, *load_path.flat_map { |dir| ["-I", dir] }, "-r", "deflake/minitest", path,
+         "--seed", seed.to_s]
+      end
+
+      def load_path
+        %w[lib test spec].select { |dir| dir == "test" || File.directory?(dir) }
+      end
+
+      # The ids go through the environment, not --name: a regexp on the command line follows
+      # the locale, and under LANG=C it never matches a name that is not ASCII
+      def env(only)
+        only ? { ResultFile::ONLY_KEY => ResultFile.pack_ids(only) } : {}
+      end
+
+      # Matches the tests defined closest above this line. Minitest itself says which tests there
+      # are, and where in this file each one is declared, so Rails' test "...", Minitest::Spec's
+      # it "..." and tests a macro defines work too. Returns their ids
+      def focus(path, line)
+        above = definitions(path).select { |test| test[:line] <= line }
+        nearest = above.map { |test| test[:line] }.max
+        raise Error, "no test found at #{path}:#{line}" unless nearest
+
+        named = above.filter_map { |test| test[:id] if test[:line] == nearest }
+        return named if named.any?
+
+        raise Error, "the test at #{path}:#{line} is in a class without a name, which deflake cannot run alone; " \
+                     "run the whole file instead"
+      end
+
+      # Tests declared in this file: { id:, line: }, with a nil id for a test in a class without a name
+      def definitions(path)
+        here = File.expand_path(path).b
+        env = { ResultFile::LIST_KEY => ResultFile.clean(File.expand_path(path)) }
+        rows = Execution.retrying { Execution.call(self, path, seed: 0, env: env) }
+        rows.select { |row| row["file"] && File.expand_path(row["file"]).b == here }
+            .map { |row| { id: row["listed"], line: row["line"] } }
+      end
+
+      # The order goes through a file: an environment variable cannot hold a big file's test ids
+      def run_in_order(path, ids, seed:)
+        Dir.mktmpdir do |dir|
+          order = File.join(dir, "order")
+          File.write(order, ResultFile.pack_ids(ids))
+          Execution.call(self, path, seed: seed, env: { ResultFile::ORDER_KEY => order })
+        end
+      end
+
+      def bisect(path, victim, seed:)
+        bisect = Bisect.new(path, seed: seed, adapter: self)
+        ids = bisect.call(victim)
+        Culprits.new(ids: ids, command: reproduce(path, seed), alone: bisect.alone?, hidden: bisect.hidden?,
+                     group_wide: bisect.group_wide)
+      end
+
+      # The class: its hooks, such as minitest-hooks' before_all, run with any of its tests
+      def group(id)
+        id.split("#", 2).first
+      end
+
+      # With the variables Execution sets: without them Rails' parallelize gives a seed another order
+      def reproduce(path, seed)
+        variables = Execution::SERIAL.map { |name, value| "#{name}=#{value}" }.join(" ")
+        "#{variables} ruby #{load_path.map { |dir| "-I#{dir}" }.join(" ")} #{Shellwords.escape(path)} --seed #{seed}"
+      end
+    end
+  end
+end

data/lib/deflake/adapters/rspec.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+require "shellwords"
+require "tmpdir"
+
+module Deflake
+  module Adapters
+    # Runs an RSpec file, optionally only some examples (by their ids, "./spec/foo_spec.rb[1:2]")
+    module RSpec
+      # Like the rspec executable: invoke turns off rspec/autorun, which would run everything twice
+      RUN = "RSpec::Core::Runner.invoke"
+      # Options in SPEC_OPTS that would take over what deflake sets: the seed and order of a run,
+      # where the output goes and when the run stops. They win over the command line
+      OWN = %w[--seed --order --rand --random --format -f --out -o --fail-fast --no-fail-fast --bisect --profile -p
+               --no-profile --dry-run].freeze
+      # Those of them that take a separate value
+      VALUED = %w[--seed --order --format -f --out -o].freeze
+
+      module_function
+
+      # only: an example id, a "path:line", or a list of example ids. The profile would fill the
+      # end of the output, where errors outside of examples are looked for
+      def command(path, seed:, only: nil)
+        # deflake/rspec adds deflake's own formatter
+        [*rspec(only || path, seed), "--format", "progress", "--no-profile"]
+      end
+
+      def env(_only)
+        options = ENV.fetch("SPEC_OPTS", nil)
+        options ? { "SPEC_OPTS" => Shellwords.join(project_options(Shellwords.split(options))) } : {}
+      end
+
+      # SPEC_OPTS without the options deflake sets itself
+      def project_options(words)
+        values = 0
+        words.reject do |word|
+          next (values -= 1) >= 0 if values.positive?
+
+          # "--seed 5" takes the next word too, "--seed=5" does not
+          values = VALUED.include?(word) ? 1 : 0
+          OWN.include?(word.split("=", 2).first)
+        end
+      end
+
+      # Lets RSpec itself say which examples are at this line, as rspec path:line does: on the line
+      # of a group, all its examples. A dry run lists them without running any example or hook,
+      # so nothing a run does, such as --fail-fast or a hook that exits, can hide some of them.
+      # Returns their ids
+      def focus(path, line)
+        ids = Execution.retrying { examples_at(path, line) }
+        raise Error, "no test found at #{path}:#{line}" if ids.empty?
+
+        ids
+      end
+
+      def examples_at(path, line)
+        Execution.call(self, path, seed: 0, only: ["--dry-run", "#{path}:#{line}"]).map { |row| row["id"] }.uniq
+      rescue Execution::NoResults => e
+        # A project with fail_if_no_examples fails a run that found none
+        raise unless e.success? || e.output.match?(/^0 examples, 0 failures/)
+
+        []
+      end
+
+      # RSpec orders examples by a hash of the seed and each id, so a subset of examples runs in the
+      # same relative order as in the whole file. The ids go through a file: a command line cannot
+      # hold the ids of a file with thousands of examples
+      def run_in_order(path, ids, seed:)
+        Dir.mktmpdir do |dir|
+          order = File.join(dir, "order")
+          File.write(order, ResultFile.pack_ids(ids))
+          Execution.call(self, path, seed: seed, env: { ResultFile::ORDER_KEY => order })
+        end
+      end
+
+      # The same bisect as for Minitest, which checks only this victim. rspec --bisect
+      # keeps every example that fails with the seed, and hangs when one exits the process
+      def bisect(path, victim, seed:)
+        bisect = Bisect.new(path, seed: seed, adapter: self)
+        ids = bisect.call(victim)
+        targets = if ids.any? then compact(ids + [victim])
+                  elsif bisect.alone? then [victim]
+                  else [path]
+                  end
+        Culprits.new(ids: ids, command: "rspec #{Shellwords.join(targets)} --seed #{seed}",
+                     alone: bisect.alone?, hidden: bisect.hidden?, group_wide: bisect.group_wide)
+      end
+
+      # The group an example is in: "./a_spec.rb[1:2:3]" => "./a_spec.rb[1:2]"
+      def group(id)
+        file, scope = id.match(/\A(.+)\[([^\]]+):\d+\]\z/)&.captures
+        "#{file}[#{scope}]" if file
+      end
+
+      # ["./a_spec.rb[1:1]", "./a_spec.rb[1:3]"] => ["./a_spec.rb[1:1,1:3]"], as rspec --bisect prints them
+      def compact(ids)
+        ids.group_by { |id| id[/\A(.+)\[[^\]]+\]\z/, 1] }.flat_map do |file, group|
+          file ? "#{file}[#{group.map { |id| id[/\[([^\]]+)\]\z/, 1] }.join(",")}]" : group
+        end
+      end
+
+      def rspec(target, seed)
+        [RbConfig.ruby, "-I", LIB, "-r", "deflake/rspec", "-e", RUN, "--", *Array(target), "--seed", seed.to_s]
+      end
+    end
+  end
+end

data/lib/deflake/adapters.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require "rbconfig"
+require_relative "adapters/minitest"
+require_relative "adapters/rspec"
+
+module Deflake
+  # Each adapter builds the command that runs a test file with deflake's recorder loaded
+  module Adapters
+    # Minitest itself, a helper named after it such as minitest_helper, or its expectations: _(x).must_equal
+    MINITEST = /minitest|mini_test|\b_\(.*\)\.(?:must|wont)_/i
+
+    module_function
+
+    # Minitest::Spec files are often named *_spec.rb too, so a spec file is sent to
+    # Minitest when it mentions Minitest and neither RSpec nor rails_helper. A spec_helper
+    # says nothing: Minitest::Spec projects use one as well
+    def for(path)
+      return Minitest unless path.to_s.end_with?("_spec.rb")
+
+      # Read as UTF-8 whatever the locale, as Ruby reads the test file itself
+      source = File.read(path, mode: "r:UTF-8").scrub
+      source.match?(MINITEST) && !source.match?(/\bRSpec\b|rails_helper/) ? Minitest : RSpec
+    rescue SystemCallError
+      RSpec
+    end
+  end
+end

data/lib/deflake/bisect.rb

@@ -0,0 +1,114 @@
+# frozen_string_literal: true
+
+module Deflake
+  # Finds the smallest set of tests that make a test fail when they run before it.
+  # Works with any adapter that can run a list of tests in a given order (run_in_order), and
+  # tell the group of a test (group) if it can.
+  # A failure counts only when it repeats: a test that fails at random would otherwise
+  # get culprits that have nothing to do with it
+  class Bisect
+    # How many runs in a row a failure must happen in to count
+    CONFIRM = 2
+    # How many other tests of a group must stand in for a culprit
+    STAND_INS = 2
+
+    def initialize(path, seed:, adapter:)
+      @path = path
+      @seed = seed
+      @adapter = adapter
+      @alone = false
+      @group_wide = []
+      @hidden = false
+    end
+
+    # Whether the failure did not happen again, and tests that cannot run by name, such as those
+    # in a class without a name, ran in the whole file: one of them may be the culprit
+    def hidden? = @hidden
+
+    # Whether the victim failed with this seed with no test before it
+    def alone? = @alone
+
+    # Culprits that other tests of their group stand in for, so their group's hook is to blame
+    attr_reader :group_wide
+
+    # Returns culprit ids, or an empty array if the failure does not reproduce or happens alone
+    def call(victim)
+      @order, unnamed = Execution.retrying { Execution.order(@adapter, @path, seed: @seed) }
+      before = @order.take_while { |id| id != victim }
+      return hidden(unnamed) unless fails?(before, victim, CONFIRM)
+      return alone if fails?([], victim, CONFIRM)
+
+      confirmed(minimize(before) { |tests| fails?(tests, victim) }, victim)
+    rescue Execution::NoResults
+      # The process broke before its first test every time, so the failure did not reproduce
+      []
+    end
+
+    private
+
+    def alone
+      @alone = true
+      []
+    end
+
+    def hidden(unnamed)
+      @hidden = unnamed
+      []
+    end
+
+    # Smaller steps ran once each, so the end result must break the victim again
+    def confirmed(culprits, victim)
+      return [] unless culprits.any? && fails?(culprits, victim, CONFIRM)
+
+      @group_wide = group_wide_of(culprits, victim)
+      culprits
+    end
+
+    # Only this victim counts: other tests may fail too, a skipped victim did not fail, and neither
+    # did one stopped by the timeout, which a shorter list of tests would not reach. A run that
+    # broke before its first test, as a flaky boot does now and then, is tried again
+    def fails?(tests, victim, times = 1)
+      times.times.all? do
+        rows = Execution.retrying { @adapter.run_in_order(@path, tests + [victim], seed: @seed) }
+        rows.any? { |row| row["id"] == victim && row["passed"] == false && !row["timed_out"] }
+      rescue Execution::NoResults
+        false
+      end
+    end
+
+    # Drops ever smaller chunks of candidates while the failure still happens
+    def minimize(candidates)
+      size = (candidates.size / 2.0).ceil
+      while size.positive?
+        candidates.each_slice(size).to_a.each do |chunk|
+          rest = candidates - chunk
+          candidates = rest if yield(rest)
+        end
+        size /= 2
+      end
+      candidates
+    end
+
+    # A culprit that other tests of its group replace with the same failure is not to blame
+    # itself: the group's hooks are, such as before(:context) or minitest-hooks' before_all.
+    # Two of them must, as one may break the victim on its own
+    def group_wide_of(culprits, victim)
+      culprits.select do |culprit|
+        stand_ins = stand_ins(culprit, culprits + [victim])
+        stand_ins.any? && stand_ins.all? do |stand_in|
+          fails?(in_order(culprits - [culprit] + [stand_in]), victim, CONFIRM)
+        end
+      end
+    end
+
+    # Other tests of the culprit's group, none of these. An adapter that knows no groups has none
+    def stand_ins(culprit, taken)
+      group = @adapter.respond_to?(:group) && @adapter.group(culprit) or return []
+      @order.select { |id| !taken.include?(id) && @adapter.group(id) == group }.first(STAND_INS)
+    end
+
+    def in_order(ids)
+      ids.sort_by { |id| @order.index(id) }
+    end
+  end
+end

data/lib/deflake/child.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Deflake
+  # Starts a test process and waits for it. The output goes to a file, not a pipe: a process the
+  # tests started in the background would keep a pipe open, and deflake would wait for it. The
+  # test process gets a session of its own, without a terminal: in deflake's process group a test
+  # that touches the terminal settings would be stopped, and a timeout or an interrupt stops
+  # whatever it started too. After an interrupt the killed process is reaped before deflake exits,
+  # not left to init
+  module Child
+    # The status of a process killed because it ran longer than the timeout
+    TimedOut = Struct.new(:status, :seconds) do
+      def success? = false
+      def signaled? = false
+      def exitstatus = nil
+    end
+
+    module_function
+
+    # Returns the process status, or TimedOut. timeout: seconds, nil for no limit
+    def run(env, command, log, timeout)
+      pid = start(env, command, log)
+      waiter = Process.detach(pid)
+      return waiter.value if waiter.join(timeout)
+
+      kill(pid)
+      TimedOut.new(waiter.value, timeout)
+    ensure
+      if waiter&.alive?
+        kill(pid)
+        waiter.join(5)
+      end
+    end
+
+    def start(env, command, log)
+      Process.fork do
+        Process.setsid
+        exec(env, *command, in: File::NULL, %i[out err] => log)
+      rescue SystemCallError => e
+        File.write(log, "deflake: #{e.message}\n")
+        exit!(127)
+      end
+    end
+
+    # Before the new process starts its session, it has no group of its own yet
+    def kill(pid)
+      Process.kill("KILL", -pid)
+    rescue SystemCallError
+      begin
+        Process.kill("KILL", pid)
+      rescue SystemCallError
+        nil
+      end
+    end
+
+    def ending(status)
+      return "timed out after #{status.seconds} seconds" if status.is_a?(TimedOut)
+      return "was killed by SIG#{Signal.signame(status.termsig)}" if status.signaled?
+
+      "exited with status #{status.exitstatus}"
+    end
+  end
+end