minitest-impact 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 729696ea279a7dac126254382c8761f9e26ae5972e0047c1961b5c0b57e6037c
-  data.tar.gz: d9abac4812e8a11862e87bc4c69c6b6b96f368c3b8b269b2f643c9547479557b
+  metadata.gz: db9b1457ea4098827caa398da16219f2ca2fa5034da0e2ccb9dfb2940c74c4f2
+  data.tar.gz: 50993034bce9425627b4f20784bb7154bff6a3dcd2ab55233ec72b3dda0c59db
 SHA512:
-  metadata.gz: 5b5cb7ba14512aead7ede47003cd271ccbbe749f9c0880666b0cb372d2a6f69f5a28aed4ca96e6c998b7ee30e4b12df0e6a1e7be8b32896f9d541fe150fb23a6
-  data.tar.gz: 0d83d45f8d06b7762b375e7428ea63b85f20189dce5bcf2d9d4bb08039995c165eda2855d323710d5b09809fc953c72c724734041ed222d02975939c6348ff3f
+  metadata.gz: 86d954e6cf0db47b51b0cacf6e672001097f039a2cc526a09b06424a3ff32a45b473780bdde073b1cfe4ceb90b3f610769b8b2a052d63c8669df752c4391c913
+  data.tar.gz: 6604fd3f172a91df47f5b76b4d22d9f582dee833c42cc7e2ace3c16eaba9b7d5a8a22a8e52c1ae7307571e3cc6fcb90a549e01349b66f8ecf91645f739acab20

data/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## 0.2.0 (2026-09-30)
+
+- Record an app from outside its bundle: the recorder loads `json` only when it first writes, so an
+  app that pins another `json` still boots, and `SimpleCov.start` does nothing while recording.
+- `run` switches SimpleCov off while it runs the selection.
+- Record projects that live under a `tmp` directory (Linux's temporary directories, some CI
+  workspaces); before, every file was skipped.
+- Trace files the application reads by path; Jev only ranks, it never drops an exact map hit.
+- `run` exits 10 without running anything when the change needs the whole suite.
+- `eval/baselines.rb` scores simple strategies on the same cases, and the README reports them for
+  Piou Piou and Fizzy.
+
+## 0.1.0 (2026-09-29)
+
+- First release.

data/README.md

@@ -16,7 +16,7 @@ Confidence: medium
   ...
 ```
 
-It works in two layers:
+It works in three layers; the third is optional:
 
 1. **A coverage map, exact and free.** One recording run of your suite notes which project lines
    every test file executed (Ruby's `Coverage`, with `eval: true` so ERB views count too). Given a
@@ -28,8 +28,9 @@ It works in two layers:
    conventional test path, and tests that mention the new constant), locale keys (tests that use
    the key, views that render it), routes (their controllers), migrations and `schema.rb` (the
    models of the changed tables), fixtures, Stimulus controllers (the views that use them), files a
-   test reads by path, and files that need the whole suite (`Gemfile.lock`, `test_helper.rb`, boot
-   configuration).
+   test reads by path, files the application reads by path, folder or file name (prompts,
+   templates: the tests that ran the reading code), config keys the application reads by name, and
+   files that need the whole suite (`Gemfile.lock`, `test_helper.rb`, boot configuration).
 3. **Jev, optionally**, when the map and the rules are not enough: some file could not be traced,
    or the selection is too large to run in a loop. See [Jev](#jev) below.
 
@@ -63,10 +64,15 @@ nothing changes in your test helper. Each test process (Rails' forked parallel w
 appends one JSON line per test to its own file; when the command ends they are merged into
 `tmp/minitest-impact/map.json` (`--map PATH` to change it), stamped with the commit it describes.
 
-- **Turn SimpleCov off while recording** (`SimpleCov.start unless ENV["MINITEST_IMPACT_RECORD"]`,
-  or your app's own switch). Ruby allows one coverage setup per process.
-- Recording is slower than a normal run (about 2 to 3 times on a Rails app), because every test
-  reads and clears the coverage counters. Record on a quiet machine, or in CI, and refresh the map
+- **SimpleCov is off while recording.** Ruby allows one coverage setup per process, so the
+  recorder makes `SimpleCov.start` do nothing for the run, with no change to your test helper.
+  Another tool that calls `Coverage.start` itself has to be turned off by you
+  (`ENV["MINITEST_IMPACT_RECORD"]` is set while recording).
+- **It does not need to be in your Gemfile.** The recorder loads no gem before your app's bundle
+  does, so the CLI can run from a checkout of this repository, outside the app's bundle:
+  `ruby -I path/to/minitest-impact/lib path/to/minitest-impact/exe/minitest-impact record -- bin/rails test`.
+- Recording is 2 to 3 times slower than a normal run on a Rails app, because every test reads
+  and clears the coverage counters. Record on a quiet machine, or in CI, and refresh the map
   when it drifts: a map a few hundred commits old still works, because methods are matched by name.
 - Record from a clean working tree; the map is stamped with `HEAD`.
 
@@ -81,38 +87,44 @@ $ minitest-impact run --since main            # select, then bin/rails test the
 $ bin/rails test:impact SINCE=main            # the same, as a Rake task (added by a Railtie)
 ```
 
-`--format paths` exits with status 10, and prints nothing, when the change needs the whole suite.
-`run` and `test:impact` run the whole suite themselves in that case. `--max N` keeps the N most
-likely files.
+When the change needs the whole suite, `select --format paths` prints nothing and `run` runs
+nothing, and both exit with status 10; `test:impact` runs the whole suite in that case. `run` and
+`test:impact` switch SimpleCov off for the selected tests, since a minimum-coverage check on a
+handful of tests always fails; with your own runner on `--format paths`, turn coverage off
+yourself. `--max N` keeps the N most likely files.
 
 ### For coding agents
 
 Put this in the agent's instructions (`CLAUDE.md`, `AGENTS.md`):
 
-> While you work, run `bin/rails test:impact SINCE=main` instead of the whole suite. When it says
-> "Confidence: low", or when you are done, the full gate runs; you do not run it yourself.
+> While you work, run `bin/rails test:impact SINCE=main` instead of the whole suite. Do not run
+> the whole suite yourself: it runs after you finish.
 
-and run the full gate as code after the agent says it is done, feeding back only failures.
+Then make that true in your harness: run the full suite as code once the agent says it is done,
+and feed back only the failures. On "Confidence: low", `test:impact` already runs the whole suite.
 
 ## Jev
 
-[Jev](https://docs.typesafe.ai) is TypeSafe's System One model: a fast, cheap classifier that
-answers typed questions (yes/no, choice, score) about a state, with calibrated probabilities. It
-is used here the way TypeSafe's own guidance says to use it:
+[Jev](https://docs.typesafe.ai) is TypeSafe's fast, cheap classifier: it answers typed questions
+(yes/no, choice, score) about a state, with calibrated probabilities.
 
-- **Rules stay in code.** Jev never decides what a test file is, which files need the whole suite,
+Measured on one app so far (see [the numbers](#with-jev-2026-09-30)): it puts the test written for
+the change first far more often, and it never found a test the map and the rules had missed. It is
+used as a ranker, not a filter.
+
+- Rules stay in code. Jev never decides what a test file is, which files need the whole suite,
   or anything else a path can tell.
-- **One narrow question per judgment, all in one request.** The state is the change (paths, a
+- One request per selection, one narrow question per judgment. The state is the change (paths, a
   trimmed diff, your `--intent`) and up to 48 candidate test files with their test names. The
   questions: one yes/no per candidate ("do these tests call, render or assert on something the
-  change modifies?"), one choice of the candidate most directly written for the change, **with a
-  "none" option**, and one yes/no for "does every test depend on this?".
-- **Fast search first, then re-rank**, as in TypeSafe's re-ranking cookbook: the candidates are
-  the map's selection plus test files whose paths and test names share words with the change.
-- **Exact map hits are never dropped.** Jev can add tests, drop weak non-exact ones and reorder,
-  but a test the map saw run the changed method stays.
-- **Thresholds live in one file** (`lib/minitest/impact/jev/questions.rb`) and **the model version
-  is pinned** (`jev-1.13.0`), because a threshold tuned on one version does not carry over.
+  change modifies?"), one choice of the candidate most directly written for the change, with a
+  "none" option, and one yes/no for "does every test depend on this?".
+- The candidates come from a fast search, and Jev only re-ranks them: the map's selection plus
+  test files whose paths and test names share words with the change.
+- Jev never drops a test. It can add tests and reorder them. Letting it drop weak picks lost tests
+  the change needed (7 in 158 cases) and bought little.
+- The thresholds live in one file (`lib/minitest/impact/jev/questions.rb`) and the model version
+  is pinned (`jev-1.13.0`), because a threshold tuned on one version does not carry over.
 
 Set `TYPESAFE_API_KEY` to turn it on (`TYPESAFE_BASE_URL` for another endpoint,
 `MINITEST_IMPACT_JEV_MODEL` to move the pin). Without a key, or with `--no-jev`, everything runs
@@ -153,35 +165,121 @@ Two kinds of labelled cases:
 Reported per case and on average: whether any expected test was selected (`caught`), whether all
 were (`all_caught`), recall, precision, and the share of the suite selected.
 
-### First numbers: Piou Piou, map only (2026-09-28)
+### First numbers: one Rails app, map only (2026-09-28)
 
-The map was recorded at one commit: 396 test files (2,673 unit and 88 system tests), 580 KB of
-JSON (72 KB gzipped). Recording made the unit suite about 2 to 3 times slower. The evaluation made
-no Jev calls.
+Measured on Piou Piou, a Rails 8.1 app, with the map recorded at one commit: 396 test files (2,673
+unit and 88 system tests), 580 KB of JSON (72 KB gzipped). The evaluation made no Jev calls.
 
 | Cases | Caught (any expected test selected) | All caught | Recall | Precision | Share of suite selected | Share of suite time |
 |---|---:|---:|---:|---:|---:|---:|
 | 150 commits that changed code and tests together, method-level | 94.0% | 90.7% | 0.93 | 0.17 | 12.1% | 24.0% |
 | The same 150, file-level | 94.0% | 90.7% | 0.93 | 0.15 | 12.3% | 24.4% |
 | 17 failed CI runs on main | 70.6% | 64.7% | 0.68 | 0.07 | 38.5% | 42.2% |
-| The same, without 4 runs where only a flaky system test failed | 12 of 13 | 11 of 13 | 0.88 | 0.10 | 50.2% | 55.1% |
+| The same, without 4 runs where only a flaky system test failed | 92.3% (12 of 13) | 84.6% (11 of 13) | 0.88 | 0.10 | 50.2% | 55.1% |
 
 How to read them:
 
-- Six of the 13 real CI breaks changed `Gemfile.lock`, `ci.yml`-tested config or `test_helper.rb`,
-  so the rule selected the whole suite. That is correct but costly. On the other seven, the
-  selection was 7.5% of the suite.
+- Six of the 13 real CI breaks changed `Gemfile.lock`, `test_helper.rb` or boot configuration,
+  so the rules selected the whole suite. That is correct but costly, and it is why the share
+  selected rises to 50% once the flaky runs are left out. On the other seven, the selection was
+  7.5% of the suite.
 - The one real break missed: a `config/piou.yml` change that failed
   `test/services/sandbox_container_test.rb`, which reads the setting through the app and never
   names the file.
 - Method-level tracing barely beats file-level on this history. Most changes land in small,
   focused files, where the two agree.
 
+### With Jev (2026-09-30)
+
+The same app two days later, with a map recorded at one commit (371 test files), the rules for
+files and config keys the application reads, and Jev as a ranker. "First" and "in the top 5" count
+the cases where an expected test was ranked there; `--format json` lists `selected` in rank order.
+
+| 150 commits that changed code and tests together | Caught | All caught | Recall | First | In the top 5 | Share of suite selected | Share of suite time |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Map and the older rules | 95.3% | 91.3% | 0.94 | 39% | 79% | 11.3% | 19.5% |
+| Map and the older rules, Jev allowed to drop picks | 94.0% | 88.0% | 0.92 | 55% | 82% | 9.4% | 16.5% |
+| Map and the current rules | 97.3% | 95.3% | 0.97 | 39% | 81% | 13.0% | 21.8% |
+| Map, the current rules and Jev as a ranker | 97.3% | 95.3% | 0.97 | 61% | 85% | 13.0% | 21.9% |
+
+How to read them:
+
+- The rules for what the application reads recovered 6 of the 15 tests the older rules missed,
+  all of them prompts and templates read through a service. They also select 18% more files.
+- Jev's gain is the order. An agent running `--max 5` gets the test written for its change in 85%
+  of cases, against 81% without it.
+- Of the 9 tests still missed, 3 belong to a commit that added a config key and its tests, with no
+  application code reading it yet: nothing but those tests could have pointed at them. Two are
+  architecture tests that read the whole source tree. The rest follow a seeds change, an importmap
+  change, a one-word config key (`technical`, too common to search for), and one change spread
+  over an agent, two jobs and a model.
+- Jev answered in about 4.7 seconds per selection, one request each.
+- Only 8 failed CI runs fit the newer map, too few to report.
+
+### A second app: Fizzy (2026-09-30)
+
+[Fizzy](https://github.com/basecamp/fizzy), Basecamp's open-source Rails app, with the map
+recorded at `a703bf1de` (2026-09-29): 261 test files (1,703 unit tests; the system tests were not
+recorded), 686 KB of JSON (62 KB gzipped), on SQLite in a Docker container. Recording took 44
+seconds against 39 for a plain run. The evaluation made no Jev calls.
+
+| Cases | Caught | All caught | Recall | Precision | Share of suite selected | Share of suite time |
+|---|---:|---:|---:|---:|---:|---:|
+| 200 commits that changed code and tests together (December 2025 to September 2026), method-level | 95.5% | 93.5% | 0.95 | 0.30 | 13.1% | 20.7% |
+
+How to read them:
+
+- 6 cases changed a file that needs the whole suite. 43 ended with "Confidence: low"; an agent
+  that runs the suite on those, as `test:impact` does, gets 95.5% all caught at 35% of suite time.
+- 13 cases missed a test. 5 of them selected nothing: a new Action Text patch in `lib/rails_ext`,
+  a service worker view, a SQLite search adapter that no longer exists at the map's commit, a
+  partial changed with the SaaS lockfile, and a commit whose only code change was
+  `test/test_helper.rb`, which the evaluation hides from the selector along with the tests. A
+  `config/routes.rb` change selected its controllers but not `test/routes_test.rb`.
+- No failed CI runs could be used. GitHub keeps Actions logs for 90 days; the 15 failed runs on
+  `main` whose logs were still there failed installing packages or gems, or on a flaky system
+  test in the SaaS bundle. None reported a failing unit test.
+
+### Against simpler strategies
+
+`eval/baselines.rb` replays the same cases with two strategies a coding agent can follow with
+no map: **conventional**, the changed tests plus the test named after each changed file
+(`app/models/invoice.rb` to `test/models/invoice_test.rb`), and **mentions**, the tests that name
+the constant a changed file defines. Both keep the gem's whole-suite rule, which needs no map.
+
+```console
+$ ruby -Ilib eval/baselines.rb --repo ../app --map map.json --results eval.json [--cases cases.json]
+```
+
+| Cases | Strategy | Caught | All caught | Recall | Precision | Share of suite selected | Share of suite time |
+|---|---|---:|---:|---:|---:|---:|---:|
+| Piou Piou, 150 commits | minitest-impact | 94.0% | 90.7% | 0.93 | 0.17 | 12.1% | 24.0% |
+| | conventional | 78.0% | 50.7% | 0.66 | 0.53 | 2.8% | 4.0% |
+| | mentions | 78.0% | 62.0% | 0.72 | 0.18 | 6.4% | 12.4% |
+| | both | 81.3% | 66.0% | 0.76 | 0.21 | 6.4% | 12.4% |
+| Piou Piou, 17 failed CI runs | minitest-impact | 70.6% | 64.7% | 0.68 | 0.07 | 38.5% | 42.2% |
+| | conventional, mentions or both | 47.1% | 47.1% | 0.47 | 0.02 | 36.3% | 37.3% |
+| Fizzy, 200 commits | minitest-impact | 95.5% | 93.5% | 0.95 | 0.30 | 13.1% | 20.7% |
+| | conventional | 68.5% | 50.0% | 0.60 | 0.54 | 3.6% | 4.4% |
+| | mentions | 68.5% | 54.0% | 0.62 | 0.40 | 5.1% | 6.3% |
+| | both | 75.5% | 61.0% | 0.69 | 0.46 | 5.2% | 6.6% |
+
+How to read them:
+
+- On commits, the map found every test a change needed in 25 (Piou Piou) and 32 (Fizzy) more
+  cases in 100 than the best simple strategy, at twice its test time on Piou Piou and three times
+  on Fizzy.
+- The simple strategies are more precise. When a change touches one model and its test, they
+  pick that test; the map also picks the controllers and jobs that ran the changed method.
+- Of the 13 real CI breaks on Piou Piou (the 4 other runs failed on a flaky system test), the map
+  caught 12 and every simple strategy 8, at about 40% of suite time for all: six of them needed
+  the whole suite.
+
 ## Prior art
 
 Nothing did most of this for Minitest, offline, when this gem was written (September 2026):
 
-| Project | What it is | What we took |
+| Project | What it is | What this gem took |
 |---|---|---|
 | [Crystalball](https://github.com/toptal/crystalball) (and GitLab's fork) | Coverage-map test selection for RSpec | The shape: record per-test coverage, predict from the diff; views, locales and schema as separate strategies |
 | [affected_tests](https://rubygems.org/gems/affected_tests), [test_impact](https://rubygems.org/gems/test_impact) | 2026 map-based selectors, RSpec only | `Coverage.result(clear: true)` per test; exit code for "run everything"; a staleness warning |

data/eval/baselines.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+# Compares minitest-impact's selection with two strategies a coding agent could follow with no
+# map at all, on the same labelled cases, so the gem has to earn its place:
+#
+# - conventional: the changed test files, plus the test named after each changed file
+#   (app/models/invoice.rb => test/models/invoice_test.rb).
+# - mentions: tests whose source names the constant a changed file defines.
+#
+# Both apply the gem's own whole-suite rule (Gemfile.lock, test_helper.rb...), which needs no map.
+#
+#   ruby -Ilib eval/baselines.rb --repo ../app --map map.json --results eval-co.json [--cases ci-cases.json]
+#
+# --results is the JSON `minitest-impact eval --format json` wrote for the same cases; its per-case
+# ids say which commits to replay. --cases gives base/head for cases that are not single commits.
+
+require "json"
+require "optparse"
+require "minitest/impact"
+
+module Minitest
+  module Impact
+    module Baselines
+      module_function
+
+      def selections(repo, map, base:, head:, hide_tests:)
+        files = Diff.parse(repo.git("diff", "--no-color", "-M", "--unified=0", base, head, allow_failure: true).to_s)
+        files = files.reject { |file| file.path.start_with?(Rules::TEST_ROOT) } if hide_tests
+        known = map.test_files
+        return { whole: true } if files.any? { |file| Rules.whole_suite?(file.path) }
+
+        changed_tests = files.map(&:path).select { |path| Rules.test_file?(path) }
+        conventional = changed_tests + files.flat_map { |file| Rules.conventional_tests(file.path) }
+        mentions = files.filter_map { |file| Rules.constant_for(file.path) }.uniq
+                        .flat_map { |constant| repo.grep(constant, paths: ["test"], rev: head) }
+                        .select { |path| Rules.test_file?(path) }
+        { whole: false, conventional: (conventional & known).uniq, mentions: ((changed_tests + mentions) & known).uniq }
+      end
+
+      def score(expected, selected, map)
+        hit = expected & selected
+        total = map.tests.values.sum(&:seconds)
+        { caught: hit.any?, all_caught: (expected - hit).empty?, recall: hit.size.to_f / expected.size,
+          precision: selected.empty? ? 0.0 : hit.size.to_f / selected.size,
+          selected_share: selected.size.to_f / map.test_files.size,
+          seconds_share: total.zero? ? 0.0 : selected.sum { |test| map.seconds(test).to_f } / total }
+      end
+
+      def summarize(rows)
+        n = rows.size.to_f
+        %i[caught all_caught].to_h { |key| [key, (rows.count { |row| row[key] } / n).round(3)] }
+          .merge(%i[recall precision selected_share seconds_share].to_h { |key| [key, (rows.sum { |row| row[key] } / n).round(3)] })
+      end
+    end
+  end
+end
+
+options = {}
+OptionParser.new do |opts|
+  opts.on("--repo PATH") { options[:repo] = it }
+  opts.on("--map PATH") { options[:map] = it }
+  opts.on("--results PATH") { options[:results] = it }
+  opts.on("--cases PATH") { options[:cases] = it }
+end.parse!
+
+repo = Minitest::Impact::Repo.new(options.fetch(:repo))
+map = Minitest::Impact::Map.load(options.fetch(:map))
+results = JSON.parse(File.read(options.fetch(:results))).fetch("cases")
+cases = options[:cases] ? JSON.parse(File.read(options[:cases])).to_h { [it["id"], it] } : {}
+
+rows = Hash.new { |hash, key| hash[key] = [] }
+results.each do |result|
+  kase = cases[result["id"]]
+  head = kase ? kase["head"] : result["id"]
+  base = kase ? kase["base"] : "#{head}^"
+  expected = result["expected"]
+  picks = Minitest::Impact::Baselines.selections(repo, map, base: base, head: head, hide_tests: kase.nil?)
+  all = map.test_files
+  conventional = picks[:whole] ? all : picks[:conventional]
+  mentions = picks[:whole] ? all : picks[:mentions]
+  rows[:gem] << Minitest::Impact::Baselines.score(expected, result["selected"], map)
+  rows[:conventional] << Minitest::Impact::Baselines.score(expected, conventional, map)
+  rows[:mentions] << Minitest::Impact::Baselines.score(expected, mentions, map)
+  rows[:conventional_or_mentions] << Minitest::Impact::Baselines.score(expected, (conventional | mentions), map)
+end
+
+puts JSON.pretty_generate(rows.transform_values { Minitest::Impact::Baselines.summarize(it) }.merge(cases: results.size))

data/lib/minitest/impact/cli.rb

@@ -67,12 +67,10 @@ module Minitest
 
         repo = Repo.new
         Dir.mktmpdir("minitest-impact") do |dir|
-          lib = File.expand_path("../..", __dir__)
-          bootstrap = File.join(__dir__, "record_bootstrap.rb")
           env = {
             "MINITEST_IMPACT_RECORD" => dir,
             "MINITEST_IMPACT_ROOT" => repo.root,
-            "RUBYOPT" => ["-I#{lib}", "-r#{bootstrap}", @env["RUBYOPT"]].compact.join(" ")
+            "RUBYOPT" => rubyopt("record_bootstrap.rb")
           }
           ok = system(env, *argv)
           map = Map.merge(File.join(dir, Recorder::PARTS), commit: repo.head)
@@ -137,7 +135,12 @@ module Minitest
         return 0 if result.tests.empty?
 
         runner = File.exist?("bin/rails") ? ["bin/rails", "test"] : ["ruby", "-Itest", "-e", "ARGV.each { |f| require File.expand_path(f) }"]
-        system(*runner, *result.tests) ? 0 : 1
+        system({ "RUBYOPT" => rubyopt("partial_bootstrap.rb") }, *runner, *result.tests) ? 0 : 1
+      end
+
+      # Loads +bootstrap+ into every Ruby process the command starts, before the application.
+      def rubyopt(bootstrap)
+        ["-I#{File.expand_path("../..", __dir__)}", "-r#{File.join(__dir__, bootstrap)}", @env["RUBYOPT"]].compact.join(" ")
       end
 
       def print_text(result)

data/lib/minitest/impact/jev/ranker.rb

@@ -8,7 +8,9 @@ module Minitest
       # over test paths and test names builds a shortlist; Jev reads the change once and answers one
       # question per candidate, in one request (docs.typesafe.ai/cookbooks/rerank_typesafe).
       #
-      # Exact map hits are never dropped: the map saw those tests run the changed code.
+      # Jev only adds and reorders; it never drops a pick. Measured on Piou Piou's history, dropping
+      # lost tests the change needed and bought little, while the new order put the test written
+      # for the change first far more often.
       class Ranker
         EXACT = 0.9
 
@@ -96,8 +98,6 @@ module Minitest
               next if noul < Questions::KEEP
 
               by_test[test] = Pick.new(test: test, score: noul * 0.8, reasons: ["Jev: exercises the change (#{noul.round(2)})"], seconds: map.seconds(test))
-            elsif pick.score < EXACT && noul < Questions::KEEP
-              by_test.delete(test)
             else
               pick.score = pick.score >= EXACT ? pick.score : (pick.score + noul) / 2
               pick.reasons += ["Jev: #{noul.round(2)}"]

data/lib/minitest/impact/partial_bootstrap.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+# Required through RUBYOPT by `minitest-impact run`, before the application loads. Coverage measured
+# on a few selected tests says nothing, and an app's minimum-coverage check would turn a green run
+# red; so SimpleCov does nothing for the run, as while recording.
+module_name = Module.instance_method(:name)
+trace = TracePoint.new(:class) do |event|
+  next unless module_name.bind_call(event.self) == "SimpleCov"
+
+  require_relative "simplecov_off"
+  event.self.singleton_class.prepend(Minitest::Impact::SimpleCovOff)
+  trace.disable
+end
+trace.enable

data/lib/minitest/impact/record_bootstrap.rb

@@ -2,18 +2,31 @@
 
 # Required through RUBYOPT by `minitest-impact record`, before the application or Bundler load, so
 # coverage sees every project file from its first line. Minitest 6 no longer loads plugins on its
-# own, so a TracePoint waits for Minitest::Test to be defined and wraps every test from there.
+# own, so a TracePoint waits for Minitest::Test to be defined and wraps every test from there. The
+# same TracePoint switches SimpleCov off as soon as it is defined: Ruby allows one coverage setup
+# per process, and an app that starts SimpleCov unconditionally would otherwise fail to boot.
 require_relative "recorder"
 
 if (dir = ENV["MINITEST_IMPACT_RECORD"])
   Minitest::Impact::Recorder.start(dir: dir, root: ENV.fetch("MINITEST_IMPACT_ROOT", Dir.pwd))
 
+  hooked = []
+  module_name = Module.instance_method(:name)
   trace = TracePoint.new(:class) do |event|
-    next unless event.self.name == "Minitest::Test"
+    name = module_name.bind_call(event.self)
+    next if hooked.include?(name)
 
-    trace.disable
-    require_relative "recording_hooks"
-    event.self.prepend(Minitest::Impact::RecordingHooks)
+    case name
+    when "Minitest::Test"
+      require_relative "recording_hooks"
+      event.self.prepend(Minitest::Impact::RecordingHooks)
+    when "SimpleCov"
+      require_relative "simplecov_off"
+      event.self.singleton_class.prepend(Minitest::Impact::SimpleCovOff)
+    else next
+    end
+    hooked << name
+    trace.disable if hooked.size == 2
   end
   trace.enable
 end

data/lib/minitest/impact/recorder.rb

@@ -1,12 +1,14 @@
 # frozen_string_literal: true
 
 require "coverage"
-require "json"
 
 module Minitest
   module Impact
     # Records which project lines each test executes. Loaded through RUBYOPT before the
-    # application boots (see `minitest-impact record`), so it depends on the standard library only.
+    # application boots (see `minitest-impact record`), so it depends on the standard library only,
+    # and loads no gem there: json, a default gem, would be activated at its newest installed version
+    # and the app's bundle would then refuse to boot on any other. It is required at the first write,
+    # after the bundle chose its version.
     #
     # Each test appends one JSON line to a file named after its process, so forked parallel
     # workers never share a file handle; `Map.merge` folds the parts into one map afterwards.
@@ -94,6 +96,7 @@ module Minitest
         end
 
         def write(record)
+          require "json"
           File.open(File.join(dir, PARTS, "#{Process.pid}.ndjson"), "a") do |file|
             file.puts(JSON.generate(record))
           end

data/lib/minitest/impact/rules.rb

@@ -25,6 +25,15 @@ module Minitest
       QUIET = [%r{\A(?:docs|tmp|log)/}, /\.md\z/, %r{\A\.github/}, /\ALICENSE/, /\.txt\z/, %r{\A\.claude/}, /\A\.gitignore\z/].freeze
 
       ROUTES = "config/routes.rb"
+      CONFIG_YAML = %r{\Aconfig/.+\.ya?ml\z}
+      CONFIG_KEY = /\A\s*([A-Za-z_][\w-]*):/
+
+      # Where application code lives, for files it reads by path and config keys it reads by name.
+      CODE_ROOTS = %w[app lib].freeze
+
+      # A folder or bare file name found in more code files than this is too common to say who reads
+      # the changed file.
+      MAX_READERS = 3
       LOCALE = %r{\Aconfig/locales/.+\.ya?ml\z}
       MIGRATION = %r{\Adb/(migrate/.+\.rb|schema\.rb|.+_schema\.rb)\z}
       VIEW = %r{\Aapp/views/(.+?)/(_?)([^/]+?)\.[^/]+\z}
@@ -53,6 +62,10 @@ module Minitest
       def quiet?(path) = QUIET.any? { |pattern| pattern.match?(path) }
       def ruby?(path) = path.end_with?(".rb")
 
+      # A key named like `sandbox_ready_poll_seconds` is found in code by name; `model` or
+      # `technical` would match unrelated code everywhere.
+      def distinctive_key?(key) = key.include?("_") && key.size >= 8
+
       # app/models/billing/invoice.rb => test/models/billing/invoice_test.rb
       # lib/tasks/x.rb => test/lib/tasks/x_test.rb, test/tasks/x_test.rb
       def conventional_tests(path)

data/lib/minitest/impact/selector.rb

@@ -215,6 +215,8 @@ module Minitest
           resolved(change, found || reads.any? ? :convention : :unresolved)
         elsif Rules.ruby?(change.path) && Rules.constant_for(change.path)
           resolve_by_name(change, reads.any?)
+        elsif resolve_read_by_code(change)
+          resolved(change, :convention, "the code reads it")
         elsif reads.any?
           resolved(change, :exact, "tests read it by path")
         elsif Rules.quiet?(change.path)
@@ -282,6 +284,46 @@ module Minitest
         resolved(change, found ? :convention : :unresolved, constant)
       end
 
+      # A file the map cannot see (a prompt, a template, a config file) that application code reads:
+      # by its path, by the folder it sits in, by its bare file name, or, for config, by the keys
+      # the change touched. The tests that ran that code are the ones that can notice.
+      def resolve_read_by_code(change)
+        path = change.path
+        found = false
+        readers = code_mentioning(path, change)
+        reason = "reads #{path}"
+        [[File.dirname(path), "reads files under #{File.dirname(path)}"], [File.basename(path), reason]].each do |needle, why|
+          break unless readers.empty?
+
+          candidates = code_mentioning(needle, change)
+          readers, reason = candidates, why if candidates.size <= Rules::MAX_READERS
+        end
+        readers.each { |file| found = true if map_tests(file, :via_file, "#{reason} through #{file}") }
+        found |= resolve_config_keys(change) if Rules::CONFIG_YAML.match?(path)
+        found
+      end
+
+      def resolve_config_keys(change)
+        keys = config_keys(@repo.read(change.path, @head), change.new_lines) |
+               config_keys(@repo.read(change.old_path || change.path, @base), change.old_lines)
+        found = false
+        keys.select { |key| Rules.distinctive_key?(key) }.each do |key|
+          code_mentioning(key, change).each do |file|
+            found = true if map_tests(file, :via_file, "reads #{key} from #{change.path} through #{file}")
+          end
+        end
+        found
+      end
+
+      def config_keys(text, lines)
+        rows = text.to_s.lines
+        lines.filter_map { |line| rows[line - 1]&.[](Rules::CONFIG_KEY, 1) }
+      end
+
+      def code_mentioning(needle, change)
+        @repo.grep(needle, paths: Rules::CODE_ROOTS, rev: @head) - [change.path]
+      end
+
       def resolve_view_like(view, reason)
         return map_tests(view, :view, reason || "renders #{view}") if @map.known_file?(view)
 

data/lib/minitest/impact/simplecov_off.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module Minitest
+  module Impact
+    # Prepended to SimpleCov while recording and during `run`. `SimpleCov.start` does nothing: no
+    # second coverage setup beside the recorder's, no report, and no minimum-coverage exit status
+    # measured on counters the recorder clears or on a handful of selected tests.
+    module SimpleCovOff
+      def start(*) = nil
+    end
+  end
+end

data/lib/minitest/impact/version.rb

@@ -2,6 +2,6 @@
 
 module Minitest
   module Impact
-    VERSION = "0.1.0"
+    VERSION = "0.2.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: minitest-impact
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Bruno Costanzo
@@ -20,8 +20,10 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- CHANGELOG.md
 - LICENSE.txt
 - README.md
+- eval/baselines.rb
 - eval/ci_failures.rb
 - exe/minitest-impact
 - lib/minitest/impact.rb
@@ -34,6 +36,7 @@ files:
 - lib/minitest/impact/locale_keys.rb
 - lib/minitest/impact/map.rb
 - lib/minitest/impact/methods.rb
+- lib/minitest/impact/partial_bootstrap.rb
 - lib/minitest/impact/railtie.rb
 - lib/minitest/impact/rake_task.rb
 - lib/minitest/impact/record_bootstrap.rb
@@ -42,13 +45,14 @@ files:
 - lib/minitest/impact/repo.rb
 - lib/minitest/impact/rules.rb
 - lib/minitest/impact/selector.rb
+- lib/minitest/impact/simplecov_off.rb
 - lib/minitest/impact/version.rb
 homepage: https://github.com/bruno-costanzo/minitest-impact
 licenses:
 - MIT
 metadata:
   source_code_uri: https://github.com/bruno-costanzo/minitest-impact
-  changelog_uri: https://github.com/bruno-costanzo/minitest-impact/releases
+  changelog_uri: https://github.com/bruno-costanzo/minitest-impact/blob/main/CHANGELOG.md
   rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths: