minitest-impact 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 729696ea279a7dac126254382c8761f9e26ae5972e0047c1961b5c0b57e6037c
+  data.tar.gz: d9abac4812e8a11862e87bc4c69c6b6b96f368c3b8b269b2f643c9547479557b
+SHA512:
+  metadata.gz: 5b5cb7ba14512aead7ede47003cd271ccbbe749f9c0880666b0cb372d2a6f69f5a28aed4ca96e6c998b7ee30e4b12df0e6a1e7be8b32896f9d541fe150fb23a6
+  data.tar.gz: 0d83d45f8d06b7762b375e7428ea63b85f20189dce5bcf2d9d4bb08039995c165eda2855d323710d5b09809fc953c72c724734041ed222d02975939c6348ff3f

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Bruno Costanzo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,205 @@
+# minitest-impact
+
+Pick the Minitest test files a change is most likely to break, or that were written to verify it,
+so a coding agent runs a handful of tests inside its loop instead of the whole suite. The full
+suite stays the final gate: this gem decides what to run *while working*, never what is allowed
+to ship.
+
+```console
+$ minitest-impact select --since main
+Confidence: medium
+  exact        app/models/invoice.rb
+  convention   config/locales/es.yml  (keys: invoices.show.title)
+4 test files, most likely first:
+  1.00  test/models/invoice_test.rb  - runs Invoice#total; is the test for app/models/invoice.rb
+  0.95  test/controllers/invoices_controller_test.rb  - uses the key invoices.show.title
+  ...
+```
+
+It works in two layers:
+
+1. **A coverage map, exact and free.** One recording run of your suite notes which project lines
+   every test file executed (Ruby's `Coverage`, with `eval: true` so ERB views count too). Given a
+   diff, a changed line is traced to the method around it, the method is found by name in the
+   recorded version of the file (so moved lines still match), and the tests that ran that method
+   are selected. A change outside any method (a constant, a `validates`, a `has_many`) selects
+   every test that loaded the file, weighted down when the file is loaded by nearly everything.
+2. **Rules for what the map cannot see**, all in `lib/minitest/impact/rules.rb`: new files (the
+   conventional test path, and tests that mention the new constant), locale keys (tests that use
+   the key, views that render it), routes (their controllers), migrations and `schema.rb` (the
+   models of the changed tables), fixtures, Stimulus controllers (the views that use them), files a
+   test reads by path, and files that need the whole suite (`Gemfile.lock`, `test_helper.rb`, boot
+   configuration).
+3. **Jev, optionally**, when the map and the rules are not enough: some file could not be traced,
+   or the selection is too large to run in a loop. See [Jev](#jev) below.
+
+The output is an ordered list of test files, each with its reasons, and an overall confidence:
+
+- **high**: every changed file was traced exactly: a changed test, a changed method the map saw
+  run, or a file no test reads.
+- **medium**: some file was traced by the whole file or by convention.
+- **low**: some file could not be traced, or the change needs the whole suite. Run the suite.
+
+## Install
+
+```ruby
+# Gemfile
+group :development, :test do
+  gem "minitest-impact"
+end
+```
+
+Ruby 3.3 or newer (it uses the Prism parser from the standard library). No runtime dependencies.
+
+## Record the map
+
+```console
+$ minitest-impact record -- bin/rails test
+$ minitest-impact record -- sh -c 'bin/rails test; bin/rails test:system'
+```
+
+`record` runs the command with `RUBYOPT` loading a small recorder before your app boots, so
+nothing changes in your test helper. Each test process (Rails' forked parallel workers included)
+appends one JSON line per test to its own file; when the command ends they are merged into
+`tmp/minitest-impact/map.json` (`--map PATH` to change it), stamped with the commit it describes.
+
+- **Turn SimpleCov off while recording** (`SimpleCov.start unless ENV["MINITEST_IMPACT_RECORD"]`,
+  or your app's own switch). Ruby allows one coverage setup per process.
+- Recording is slower than a normal run (about 2 to 3 times on a Rails app), because every test
+  reads and clears the coverage counters. Record on a quiet machine, or in CI, and refresh the map
+  when it drifts: a map a few hundred commits old still works, because methods are matched by name.
+- Record from a clean working tree; the map is stamped with `HEAD`.
+
+## Select and run
+
+```console
+$ minitest-impact select                      # the uncommitted change
+$ minitest-impact select --since main         # everything since main
+$ minitest-impact select --since main --format json
+$ bin/rails test $(minitest-impact select --since main --format paths)
+$ minitest-impact run --since main            # select, then bin/rails test the selection
+$ bin/rails test:impact SINCE=main            # the same, as a Rake task (added by a Railtie)
+```
+
+`--format paths` exits with status 10, and prints nothing, when the change needs the whole suite.
+`run` and `test:impact` run the whole suite themselves in that case. `--max N` keeps the N most
+likely files.
+
+### For coding agents
+
+Put this in the agent's instructions (`CLAUDE.md`, `AGENTS.md`):
+
+> While you work, run `bin/rails test:impact SINCE=main` instead of the whole suite. When it says
+> "Confidence: low", or when you are done, the full gate runs; you do not run it yourself.
+
+and run the full gate as code after the agent says it is done, feeding back only failures.
+
+## Jev
+
+[Jev](https://docs.typesafe.ai) is TypeSafe's System One model: a fast, cheap classifier that
+answers typed questions (yes/no, choice, score) about a state, with calibrated probabilities. It
+is used here the way TypeSafe's own guidance says to use it:
+
+- **Rules stay in code.** Jev never decides what a test file is, which files need the whole suite,
+  or anything else a path can tell.
+- **One narrow question per judgment, all in one request.** The state is the change (paths, a
+  trimmed diff, your `--intent`) and up to 48 candidate test files with their test names. The
+  questions: one yes/no per candidate ("do these tests call, render or assert on something the
+  change modifies?"), one choice of the candidate most directly written for the change, **with a
+  "none" option**, and one yes/no for "does every test depend on this?".
+- **Fast search first, then re-rank**, as in TypeSafe's re-ranking cookbook: the candidates are
+  the map's selection plus test files whose paths and test names share words with the change.
+- **Exact map hits are never dropped.** Jev can add tests, drop weak non-exact ones and reorder,
+  but a test the map saw run the changed method stays.
+- **Thresholds live in one file** (`lib/minitest/impact/jev/questions.rb`) and **the model version
+  is pinned** (`jev-1.13.0`), because a threshold tuned on one version does not carry over.
+
+Set `TYPESAFE_API_KEY` to turn it on (`TYPESAFE_BASE_URL` for another endpoint,
+`MINITEST_IMPACT_JEV_MODEL` to move the pin). Without a key, or with `--no-jev`, everything runs
+offline on the map and the rules. If Jev fails, the map's selection is used and the error is
+reported; the agent's loop never breaks on it.
+
+Cost: Jev bills input tokens only, $0.042 per million (docs.typesafe.ai/models). A request with
+48 candidates and a 12,000-character diff is under 10,000 tokens: about $0.0004.
+
+### Tuning
+
+The thresholds ship **untuned** (0.5 to keep a candidate, 0.5 confidence for the "most direct"
+choice, 0.8 for "whole suite"). Tune them on your own history before trusting them:
+
+```console
+$ minitest-impact eval --cases cases.json --jev --format json
+```
+
+and move the constants in `questions.rb` to the values that give the recall you need at the
+smallest selection.
+
+## Measure it on your history
+
+```console
+$ minitest-impact eval --co-changed 200          # commits that changed code and its tests together
+$ ruby eval/ci_failures.rb --repo . --out cases.json   # failed CI runs, via the gh CLI
+$ minitest-impact eval --cases cases.json
+```
+
+Two kinds of labelled cases:
+
+- **Tests a change broke** (`eval/ci_failures.rb`): for each failed GitHub Actions run on the
+  default branch, the change since the last green run, and the test files the failed jobs
+  reported. A test already failing in the run before is not counted again.
+- **Tests written for a change** (`--co-changed N`): commits that changed code and existing test
+  files together. The selector sees only the code; the tests the commit edited are the answer.
+
+Reported per case and on average: whether any expected test was selected (`caught`), whether all
+were (`all_caught`), recall, precision, and the share of the suite selected.
+
+### First numbers: Piou Piou, map only (2026-09-28)
+
+The map was recorded at one commit: 396 test files (2,673 unit and 88 system tests), 580 KB of
+JSON (72 KB gzipped). Recording made the unit suite about 2 to 3 times slower. The evaluation made
+no Jev calls.
+
+| Cases | Caught (any expected test selected) | All caught | Recall | Precision | Share of suite selected | Share of suite time |
+|---|---:|---:|---:|---:|---:|---:|
+| 150 commits that changed code and tests together, method-level | 94.0% | 90.7% | 0.93 | 0.17 | 12.1% | 24.0% |
+| The same 150, file-level | 94.0% | 90.7% | 0.93 | 0.15 | 12.3% | 24.4% |
+| 17 failed CI runs on main | 70.6% | 64.7% | 0.68 | 0.07 | 38.5% | 42.2% |
+| The same, without 4 runs where only a flaky system test failed | 12 of 13 | 11 of 13 | 0.88 | 0.10 | 50.2% | 55.1% |
+
+How to read them:
+
+- Six of the 13 real CI breaks changed `Gemfile.lock`, `ci.yml`-tested config or `test_helper.rb`,
+  so the rule selected the whole suite. That is correct but costly. On the other seven, the
+  selection was 7.5% of the suite.
+- The one real break missed: a `config/piou.yml` change that failed
+  `test/services/sandbox_container_test.rb`, which reads the setting through the app and never
+  names the file.
+- Method-level tracing barely beats file-level on this history. Most changes land in small,
+  focused files, where the two agree.
+
+## Prior art
+
+Nothing did most of this for Minitest, offline, when this gem was written (September 2026):
+
+| Project | What it is | What we took |
+|---|---|---|
+| [Crystalball](https://github.com/toptal/crystalball) (and GitLab's fork) | Coverage-map test selection for RSpec | The shape: record per-test coverage, predict from the diff; views, locales and schema as separate strategies |
+| [affected_tests](https://rubygems.org/gems/affected_tests), [test_impact](https://rubygems.org/gems/test_impact) | 2026 map-based selectors, RSpec only | `Coverage.result(clear: true)` per test; exit code for "run everything"; a staleness warning |
+| [fast_cov](https://github.com/Gusto/fast_cov) | A C-extension file tracker that leaves `Coverage` to SimpleCov | A candidate recorder backend if recording beside SimpleCov matters more than line numbers |
+| [Datadog Test Impact Analysis](https://docs.datadoghq.com/tests/test_impact_analysis/) | Minitest support, but the decision needs Datadog's service | Nothing adopted |
+| [Launchable / CloudBees Smart Tests](https://docs.cloudbees.com/docs/cloudbees-smart-tests/latest/features/predictive-test-selection) | Predictive selection as a service | Nothing adopted |
+| Google TAP (Memon et al., 2017), Facebook's predictive test selection (Machalica et al., 2019), Ekstazi (Gligoric et al., 2015) | Research | File-level dynamic selection is safe and cheap; most failures are close to the change |
+| [`jev`](https://rubygems.org/gems/jev) 0.2.0 | A Ruby client for Jev | Not used yet: it always sends `jev-latest` to one fixed URL, and thresholds need a pinned version |
+
+## Limits
+
+- Code that runs only at boot (initializers, `config/*.rb`) is not in the map; a change there is
+  traced by rules or reported as low confidence.
+- Views are traced by file, not by line: compiled templates' line numbers are not the ERB's.
+- Line coverage cannot see what a test *would* run after the change (a new branch, a new
+  callback). That is what the conventional test and Jev are for, and why the full suite stays the
+  final gate.
+
+## License
+
+MIT.

data/eval/ci_failures.rb

@@ -0,0 +1,72 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Builds evaluation cases from a repository's failed GitHub Actions runs on its default branch:
+# for each failed run, the change is everything since the last green run before it, and the
+# expected tests are the test files the failed jobs reported. Logs are cached, so a second run is
+# offline.
+#
+#   ruby eval/ci_failures.rb --repo ~/app --workflow ci.yml --out cases.json [--cache DIR] [--limit 200]
+#
+# Needs the gh CLI, authenticated for the repository. Costs nothing but GitHub API calls.
+
+require "json"
+require "open3"
+require "optparse"
+require "fileutils"
+
+options = { workflow: "ci.yml", limit: 300, cache: "tmp/minitest-impact/ci-logs", branch: "main" }
+OptionParser.new do |parser|
+  parser.on("--repo DIR") { |v| options[:repo] = File.expand_path(v) }
+  parser.on("--workflow FILE") { |v| options[:workflow] = v }
+  parser.on("--branch NAME") { |v| options[:branch] = v }
+  parser.on("--out FILE") { |v| options[:out] = v }
+  parser.on("--cache DIR") { |v| options[:cache] = v }
+  parser.on("--limit N", Integer) { |v| options[:limit] = v }
+end.parse!
+abort "usage: ci_failures.rb --repo DIR --out FILE" unless options[:repo] && options[:out]
+
+def sh(*cmd, dir:)
+  out, err, status = Open3.capture3(*cmd, chdir: dir)
+  raise "#{cmd.join(" ")}: #{err}" unless status.success?
+
+  out
+end
+
+FileUtils.mkdir_p(options[:cache])
+runs = JSON.parse(sh("gh", "run", "list", "--workflow", options[:workflow], "--branch", options[:branch], "--event", "push",
+                     "--limit", options[:limit].to_s, "--json", "databaseId,headSha,conclusion,createdAt", dir: options[:repo]))
+runs.sort_by! { |run| run["createdAt"] }
+
+# Rails prints a rerun line per failure ("bin/rails test test/models/user_test.rb:12"), and
+# Minitest a location in brackets ("[test/models/user_test.rb:12]").
+FAILING = %r{(?:bin/rails test |\[)(test/[\w/.-]+_test\.rb):\d+}
+
+last_green = nil
+still_failing = []
+cases = []
+runs.each do |run|
+  if run["conclusion"] == "success"
+    last_green = run["headSha"]
+    still_failing = []
+    next
+  end
+  next unless run["conclusion"] == "failure" && last_green
+
+  cache = File.join(options[:cache], "#{run["databaseId"]}.log")
+  unless File.exist?(cache)
+    log, _err, status = Open3.capture3("gh", "run", "view", run["databaseId"].to_s, "--log-failed", chdir: options[:repo])
+    File.write(cache, status.success? ? log : "")
+  end
+  reported = File.read(cache).scan(FAILING).flatten.uniq.sort
+  # A test that already failed in the run before was broken by an earlier push, not this one.
+  failing = reported - still_failing
+  still_failing |= reported
+  next if failing.empty?
+
+  cases << { "id" => "run-#{run["databaseId"]}", "base" => last_green, "head" => run["headSha"], "expected" => failing,
+             "source" => "ci_failure" }
+end
+
+File.write(options[:out], JSON.pretty_generate(cases))
+puts "#{cases.size} cases from #{runs.count { |r| r["conclusion"] == "failure" }} failed runs into #{options[:out]}"

data/exe/minitest-impact

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "../lib/minitest/impact"
+require_relative "../lib/minitest/impact/cli"
+
+exit Minitest::Impact::CLI.start(ARGV.dup)

data/lib/minitest/impact/cli.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+require "json"
+require "optparse"
+require "tmpdir"
+
+module Minitest
+  module Impact
+    # minitest-impact record | select | run | eval
+    class CLI
+      WHOLE_SUITE_EXIT = 10
+
+      def self.start(argv, out: $stdout, err: $stderr, env: ENV) = new(out: out, err: err, env: env).call(argv)
+
+      def initialize(out:, err:, env:)
+        @out = out
+        @err = err
+        @env = env
+      end
+
+      def call(argv)
+        command = argv.shift
+        case command
+        when "record" then record(argv)
+        when "select" then select(argv)
+        when "run" then run(argv)
+        when "eval" then evaluate(argv)
+        else
+          @out.puts(USAGE)
+          command.nil? || %w[-h --help help].include?(command) ? 0 : 1
+        end
+      rescue Error, OptionParser::ParseError => error
+        @err.puts("minitest-impact: #{error.message}")
+        1
+      end
+
+      USAGE = <<~TEXT
+        Usage:
+          minitest-impact record [--map PATH] -- COMMAND...   Run your suite and record the coverage map
+          minitest-impact select [--since REV] [options]      Print the tests a change should run
+          minitest-impact run    [--since REV] [options]      Select, then run them with bin/rails test
+          minitest-impact eval   (--cases FILE | --co-changed N) [--jev] [--granularity method|file]
+                                                               Measure selection on history
+
+        Select options:
+          --since REV       compare against REV (default HEAD: the uncommitted change)
+          --head REV        compare REV instead of the working tree
+          --intent TEXT     what the change is for, in words (used by Jev only)
+          --map PATH        coverage map (default #{Map::DEFAULT_PATH})
+          --format FORMAT   text (default), json or paths
+          --max N           keep the N most likely test files
+          --no-jev          map and rules only, even with TYPESAFE_API_KEY set
+
+        Exit status 10 from select/run with --format paths means: run the whole suite.
+      TEXT
+
+      private
+
+      def record(argv)
+        map_path = Map::DEFAULT_PATH
+        if argv.first == "--map"
+          argv.shift
+          map_path = argv.shift
+        end
+        argv.shift if argv.first == "--"
+        raise Error, "record needs a command, e.g. minitest-impact record -- bin/rails test" if argv.empty?
+
+        repo = Repo.new
+        Dir.mktmpdir("minitest-impact") do |dir|
+          lib = File.expand_path("../..", __dir__)
+          bootstrap = File.join(__dir__, "record_bootstrap.rb")
+          env = {
+            "MINITEST_IMPACT_RECORD" => dir,
+            "MINITEST_IMPACT_ROOT" => repo.root,
+            "RUBYOPT" => ["-I#{lib}", "-r#{bootstrap}", @env["RUBYOPT"]].compact.join(" ")
+          }
+          ok = system(env, *argv)
+          map = Map.merge(File.join(dir, Recorder::PARTS), commit: repo.head)
+          raise Error, "no test recorded anything; is the command a Minitest run?" if map.tests.empty?
+
+          map.save(map_path)
+          dirty = !repo.git("status", "--porcelain", "--untracked-files=no").empty?
+          @err.puts("minitest-impact: the working tree has changes; the map describes them, not #{repo.head[0, 10]}") if dirty
+          @out.puts("Recorded #{map.tests.size} test files at #{repo.head[0, 10]} into #{map_path}")
+          ok ? 0 : 1
+        end
+      end
+
+      def select_options(argv)
+        options = { since: "HEAD", format: "text", map: Map::DEFAULT_PATH, jev: true }
+        OptionParser.new do |parser|
+          parser.on("--since REV") { |v| options[:since] = v }
+          parser.on("--head REV") { |v| options[:head] = v }
+          parser.on("--intent TEXT") { |v| options[:intent] = v }
+          parser.on("--map PATH") { |v| options[:map] = v }
+          parser.on("--format FORMAT", %w[text json paths]) { |v| options[:format] = v }
+          parser.on("--max N", Integer) { |v| options[:max] = v }
+          parser.on("--[no-]jev") { |v| options[:jev] = v }
+        end.parse!(argv)
+        options
+      end
+
+      def selection(options)
+        raise Error, "no coverage map at #{options[:map]}; run: minitest-impact record -- bin/rails test" unless File.exist?(options[:map])
+
+        repo = Repo.new
+        ranker = options[:jev] ? jev_ranker : nil
+        selection = Selector.new(repo: repo, map: Map.load(options[:map]), base: options[:since], head: options[:head],
+                                 intent: options[:intent], ranker: ranker).call
+        selection.picks = selection.picks.first(options[:max]) if options[:max]
+        selection
+      end
+
+      def jev_ranker
+        client = Jev::Client.from_env(@env)
+        client && Jev::Ranker.new(client: client)
+      end
+
+      def select(argv)
+        options = select_options(argv)
+        result = selection(options)
+        case options[:format]
+        when "json" then @out.puts(JSON.pretty_generate(result.to_h))
+        when "paths"
+          return WHOLE_SUITE_EXIT if result.whole_suite
+
+          result.tests.each { |test| @out.puts(test) }
+        else print_text(result)
+        end
+        0
+      end
+
+      def run(argv)
+        result = selection(select_options(argv))
+        print_text(result)
+        return WHOLE_SUITE_EXIT if result.whole_suite
+        return 0 if result.tests.empty?
+
+        runner = File.exist?("bin/rails") ? ["bin/rails", "test"] : ["ruby", "-Itest", "-e", "ARGV.each { |f| require File.expand_path(f) }"]
+        system(*runner, *result.tests) ? 0 : 1
+      end
+
+      def print_text(result)
+        @out.puts("Confidence: #{result.confidence}#{result.whole_suite ? " (run the whole suite)" : ""}")
+        result.warnings.each { |warning| @out.puts("Warning: #{warning}") }
+        @out.puts("Jev: #{result.jev.map { |k, v| "#{k}=#{v}" }.join(" ")}") if result.jev
+        result.resolutions.each do |resolution|
+          @out.puts("  #{resolution.how.to_s.ljust(12)} #{resolution.path}#{resolution.note ? "  (#{resolution.note})" : ""}")
+        end
+        @out.puts(result.picks.empty? ? "No tests selected." : "#{result.picks.size} test files, most likely first:")
+        result.picks.each do |pick|
+          @out.puts(format("  %.2f  %s  - %s", pick.score, pick.test, pick.reasons.first(2).join("; ")))
+        end
+      end
+
+      def evaluate(argv)
+        options = { map: Map::DEFAULT_PATH, jev: false, format: "text", granularity: :method }
+        OptionParser.new do |parser|
+          parser.on("--cases FILE") { |v| options[:cases] = v }
+          parser.on("--co-changed N", Integer) { |v| options[:co_changed] = v }
+          parser.on("--map PATH") { |v| options[:map] = v }
+          parser.on("--[no-]jev") { |v| options[:jev] = v }
+          parser.on("--format FORMAT", %w[text json]) { |v| options[:format] = v }
+          parser.on("--granularity LEVEL", %w[method file]) { |v| options[:granularity] = v.to_sym }
+        end.parse!(argv)
+        repo = Repo.new
+        map = Map.load(options[:map])
+        cases = if options[:cases]
+                  JSON.parse(File.read(options[:cases]))
+                elsif options[:co_changed]
+                  Evaluation.co_changed_cases(repo, map, limit: options[:co_changed])
+                else
+                  raise Error, "eval needs --cases FILE or --co-changed N"
+                end
+        results = Evaluation.new(repo: repo, map: map, ranker: options[:jev] ? jev_ranker : nil, granularity: options[:granularity]).run(cases)
+        summary = Evaluation.summary(results)
+        if options[:format] == "json"
+          @out.puts(JSON.pretty_generate("summary" => summary, "cases" => results.map { |r| r.to_h.merge(recall: r.recall, precision: r.precision) }))
+        else
+          results.each do |r|
+            @out.puts(format("%-12s caught=%-5s recall=%.2f precision=%.2f selected=%d/%d %s", r.id, r.caught?, r.recall, r.precision,
+                             r.selected.size, r.suite_size, r.confidence))
+          end
+          @out.puts(JSON.pretty_generate(summary))
+        end
+        0
+      end
+    end
+  end
+end

data/lib/minitest/impact/diff.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module Minitest
+  module Impact
+    # One file in a change. Line numbers are on the old side of the diff, because the coverage
+    # map was recorded against old code; an insertion is placed on the line it follows.
+    ChangedFile = Struct.new(:path, :old_path, :status, :old_lines, :new_lines, :added_text, :removed_text, keyword_init: true) do
+      def added? = status == :added
+      def deleted? = status == :deleted
+      def text = [removed_text, added_text].join("\n")
+    end
+
+    module Diff
+      HUNK = /\A@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@/
+
+      # Parses `git diff --unified=0` output.
+      def self.parse(text)
+        files = []
+        current = nil
+        text.each_line do |line|
+          line = line.chomp
+          case line
+          when /\Adiff --git a\/(.+) b\/(.+)\z/
+            current = ChangedFile.new(path: Regexp.last_match(2), old_path: Regexp.last_match(1), status: :modified,
+                                      old_lines: [], new_lines: [], added_text: +"", removed_text: +"")
+            files << current
+          when /\Anew file mode/ then current.status = :added
+          when /\Adeleted file mode/ then current.status = :deleted
+          when /\Arename from (.+)\z/ then current.old_path = Regexp.last_match(1)
+          when /\Arename to (.+)\z/ then current.status = :renamed if current.status == :modified
+          when HUNK
+            start = Regexp.last_match(1).to_i
+            count = (Regexp.last_match(2) || "1").to_i
+            current.old_lines.concat(count.zero? ? [[start, 1].max] : (start...(start + count)).to_a)
+            new_start = Regexp.last_match(3).to_i
+            new_count = (Regexp.last_match(4) || "1").to_i
+            current.new_lines.concat((new_start...(new_start + new_count)).to_a)
+          when /\A\+\+\+ |\A--- / then next
+          when /\A\+(.*)\z/ then current&.added_text&.<<(Regexp.last_match(1) + "\n")
+          when /\A-(.*)\z/ then current&.removed_text&.<<(Regexp.last_match(1) + "\n")
+          end
+        end
+        files.each { |file| file.old_path = nil if file.status == :added }
+      end
+    end
+  end
+end

data/lib/minitest/impact/evaluation.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Minitest
+  module Impact
+    # Replays a selection on past changes whose right answer is known and measures it.
+    #
+    # A case is {"id", "base", "head", "expected" => [test files]}: the tests a change broke (from
+    # CI failures) or the tests written for it (from commits that changed code and its tests
+    # together, with the test edits hidden from the selector).
+    class Evaluation
+      Result = Struct.new(:id, :expected, :selected, :hit, :suite_size, :seconds_share, :whole_suite, :confidence, keyword_init: true) do
+        def caught? = hit.any?
+        def all_caught? = (expected - hit).empty?
+        def recall = hit.size.to_f / expected.size
+        def precision = selected.empty? ? 0.0 : hit.size.to_f / selected.size
+      end
+
+      # Cases from commits that changed code and existing tests together. The tests they edited
+      # are the ones written to verify the change.
+      def self.co_changed_cases(repo, map, limit: 200)
+        shas = repo.git("log", "--no-merges", "--format=%H", "-n", (limit * 10).to_s).split("\n")
+        shas.each_with_object([]) do |sha, cases|
+          break cases if cases.size >= limit
+
+          files = Diff.parse(repo.git("diff", "--no-color", "-M", "--unified=0", "#{sha}^", sha, allow_failure: true).to_s)
+          tests = files.select { |f| Rules.test_file?(f.path) && f.status == :modified && map.tests.key?(f.path) }.map(&:path)
+          code = files.reject { |f| f.path.start_with?(Rules::TEST_ROOT) || Rules.quiet?(f.path) }
+          next if tests.empty? || code.empty?
+
+          cases << { "id" => sha[0, 10], "base" => "#{sha}^", "head" => sha, "expected" => tests, "hide_tests" => true }
+        end
+      end
+
+      def initialize(repo:, map:, ranker: nil, granularity: :method)
+        @repo = repo
+        @map = map
+        @ranker = ranker
+        @granularity = granularity
+      end
+
+      def run(cases)
+        cases.filter_map do |kase|
+          expected = kase.fetch("expected") & @map.test_files
+          next if expected.empty? || !@repo.commit?(kase.fetch("head"))
+
+          exclude = kase["hide_tests"] ? ->(path) { path.start_with?(Rules::TEST_ROOT) } : nil
+          selection = Selector.new(repo: @repo, map: @map, base: kase.fetch("base"), head: kase.fetch("head"),
+                                   intent: kase["intent"], ranker: @ranker, exclude: exclude, granularity: @granularity).call
+          selected = selection.whole_suite ? @map.test_files : selection.tests
+          Result.new(id: kase.fetch("id"), expected: expected, selected: selected, hit: expected & selected,
+                     suite_size: @map.test_files.size, seconds_share: seconds_share(selected), whole_suite: selection.whole_suite,
+                     confidence: selection.confidence)
+        end
+      end
+
+      def seconds_share(selected)
+        total = @map.tests.values.sum(&:seconds)
+        total.zero? ? 0.0 : selected.sum { |test| @map.seconds(test).to_f } / total
+      end
+
+      def self.summary(results)
+        return {} if results.empty?
+
+        n = results.size.to_f
+        {
+          "cases" => results.size,
+          "caught" => (results.count(&:caught?) / n).round(3),
+          "all_caught" => (results.count(&:all_caught?) / n).round(3),
+          "recall" => (results.sum(&:recall) / n).round(3),
+          "precision" => (results.sum(&:precision) / n).round(3),
+          "selected_share" => (results.sum { |r| r.selected.size.to_f / r.suite_size } / n).round(3),
+          "seconds_share" => (results.sum(&:seconds_share) / n).round(3),
+          "whole_suite" => results.count(&:whole_suite)
+        }
+      end
+    end
+  end
+end