mutation_tester 1.4.1 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/Gemfile.lock +1 -1
- data/docs/ci.md +24 -1
- data/docs/execution-runners.md +12 -0
- data/exe/mutation_test +34 -8
- data/lib/mutation_tester/batch_runner.rb +41 -3
- data/lib/mutation_tester/configuration.rb +10 -2
- data/lib/mutation_tester/fork_runner/worker.rb +11 -0
- data/lib/mutation_tester/fork_runner.rb +12 -6
- data/lib/mutation_tester/mutation_runner.rb +38 -6
- data/lib/mutation_tester/version.rb +1 -1
- data/readme.md +144 -27
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 95884456f3cb22f463001c227181cbb9250fe09c3e94eec4039cc0fac54c6e18
|
|
4
|
+
data.tar.gz: 984f636d787af8f79d4d117684f57a69bdd48347651ecbb6421468d2730522e8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 890a83603e65df88fcad21e62a27faa57c8540c403d8ca92ae218b9471d04471272f01f409f7c32cff8bf567feda042167c75b7be83962315c06976626ec142b
|
|
7
|
+
data.tar.gz: 943bb88ba38f80d37c78ecde8f91567000e952a4b68209cfac9d94aa336129b8ce2bdec2eaac4759fa328fb334daa79fd34a8a7a587d069804fcf1d4c907c943
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.5.0] - 2026-08-08
|
|
4
|
+
|
|
5
|
+
- Added `--after-fork FILE` (and `config.after_fork_file` / the `MUTATION_TESTER_AFTER_FORK` environment variable): a Ruby file loaded inside each preloaded in-memory clone right after it forks and receives its per-worker environment, so the app can re-establish per-worker state such as its ActiveRecord connection. With `--worker-env` set, this keeps the `in_memory` runner available in parallel runs instead of falling back to `fork`: each clone now receives its own per-worker value of the variable through the existing per-clone environment plumbing and runs the after-fork file once, giving in-memory execution and per-worker database isolation at the same time. Failure modes stay loud: a clone whose after-fork file raises reports the error on stderr and is dropped, its worker deciding its share of mutants file-based, and a missing after-fork file makes the whole run fall back to file-based execution with a warning. A serial in-memory run (`-p 1`) has only one worker and now stays in memory with `--worker-env` even without the hook.
|
|
6
|
+
- A serial file-based run (`-p 1`) with `--worker-env` now decides each mutant in a shadow workspace instead of writing mutants into the real source file with a `.mutation_backup` alongside, so a process killed mid-run can no longer leave a mutated source file in the checkout. This combination occurs naturally when a caller uses one code path for both a parallel run and a serial re-run. Without `--worker-env` serial file-based runs keep the in-place strategy (the next run still restores a leftover backup automatically).
|
|
7
|
+
|
|
8
|
+
## [1.4.2] - 2026-08-05
|
|
9
|
+
|
|
10
|
+
- Added `--spec-map 'PATTERN=>REPLACEMENT'`, a repeatable rule that builds the whole spec path from the whole source path with a regular expression, for the layouts the `{name}` template cannot express. `{name}` is the source path minus a leading `lib/`, so a template can only wrap it with a prefix and a suffix; every mapping that substitutes *inside* the path, after a variable-length prefix, was out of reach: `packs/identity/app/models/party.rb -> packs/identity/test/models/party_test.rb` (Packwerk / packs-rails), the same shape with Rails engines, and the plain Rails `app/ -> test/` rule. Because `--since` requires `--glob` and `--glob` maps through `--spec-glob`, such a layout previously lost the entire batch tier at once and had to be reimplemented in a wrapper script. The first matching rule wins and a source matching no rule falls back to `--spec-glob`, so one run can cover `app/` through a rule and `lib/` through the template. The flag works with `--glob`, a positional `FILE` list and `--staged`; a malformed rule is a usage error (exit code `2`) reported before any mutation runs.
|
|
11
|
+
- Fixed `--glob` finishing with status `0` when every matched file was skipped. The two batch paths used different predicates over the same result, and the `--glob` one gated on "something matched and every processed file passed", which is vacuously true over an empty processed list. A run that matched four files and skipped all four (a wrong `--spec-glob`, a test directory moved by a refactor) printed `All processed files met the mutation score threshold`, set `"passed": true` in the JSON envelope and passed the run, which is the worst direction for a quality gate: it vouches for a score it never measured. Both paths now share one predicate: a batch that matched files but measured none fails with status `1` and prints `No files were mutation-tested`, while a `--since` run whose matched files were all unchanged still passes, so a pull request that touches nothing keeps its documented green run.
|
|
12
|
+
- Added `--minimum-score N` to set the threshold a file must reach for a single run. `config.minimum_score` (default 80) had no CLI flag and no environment variable, so the CLI's own gate was fixed at 80 and the shipped `examples/hooks/pre-push` worked around it with `--json` plus `jq`. An existing codebase can now start below 80 and raise the number over time without a wrapper.
|
|
13
|
+
|
|
3
14
|
## [1.4.1] - 2026-08-03
|
|
4
15
|
|
|
5
16
|
- Fixed parallel runs reporting every mutant as survived (`Killed: 0`, score `0.0%`) on projects whose test file reaches its source through `$LOAD_PATH` (a Rails app run with `-Itest` in `RUBYOPT`, for example). A preloaded worker is started once in the real project root, so Ruby absolutizes every `-I`/`RUBYLIB` entry against that directory; the forked child then changed into the shadow workspace but kept resolving `require "test_helper"` back to the original tree, which loaded the unmutated source. Each job now rewrites the `$LOAD_PATH` entries of the mirrored project root into the workspace, so a preloaded worker resolves project code exactly like the `spawn` runner that starts inside the workspace. Only Minitest suites hit this in practice: RSpec re-adds `lib` and its default path at run time, after the child has changed directory. Serial runs and the `spawn` runner were never affected, which is why the same file scored 96.55% serially and 0.0% with `-p 8`.
|
data/Gemfile.lock
CHANGED
data/docs/ci.md
CHANGED
|
@@ -29,7 +29,8 @@ The same thing, condensed to a copy-pasteable minimal workflow. It matches the
|
|
|
29
29
|
maintained template above; reach for the template when you want the parallel and
|
|
30
30
|
multi-file variants. The quality gate needs no extra configuration: the CLI exits
|
|
31
31
|
non-zero when the mutation score is below the threshold, which fails the step (and
|
|
32
|
-
the job).
|
|
32
|
+
the job). The threshold defaults to 80 and `--minimum-score N` sets it for a
|
|
33
|
+
single run, so an existing codebase can start lower and ratchet the number up.
|
|
33
34
|
|
|
34
35
|
```yaml
|
|
35
36
|
name: Mutation Testing
|
|
@@ -108,6 +109,28 @@ jobs:
|
|
|
108
109
|
A PR that touches nothing under `lib/` exits `0` with a "Nothing to mutate"
|
|
109
110
|
message, so the gate never blocks unrelated changes.
|
|
110
111
|
|
|
112
|
+
### What the batch exit code guarantees
|
|
113
|
+
|
|
114
|
+
A batch run exits `0` only when at least one file was actually mutation-tested
|
|
115
|
+
and every processed file met the threshold, or when the `--since` filter left
|
|
116
|
+
nothing to mutate. If the glob matched files but *every* one of them was skipped
|
|
117
|
+
(most often `no matching spec file`), the run exits `1`: nothing was measured, so
|
|
118
|
+
the gate has nothing to vouch for. That keeps a typo in `--spec-glob` /
|
|
119
|
+
`--spec-map`, or a refactor that moves the test directory, from silently turning
|
|
120
|
+
a green job into a job that measures nothing.
|
|
121
|
+
|
|
122
|
+
For an application whose tests do not sit under `spec/`, map the paths with
|
|
123
|
+
`--spec-map` (see
|
|
124
|
+
[Mapping sources to specs](../readme.md#mapping-sources-to-specs)):
|
|
125
|
+
|
|
126
|
+
```yaml
|
|
127
|
+
- name: Run mutation tests on changed files only
|
|
128
|
+
run: |
|
|
129
|
+
bundle exec mutation_test --glob '{app,packs/*/app}/**/*.rb' \
|
|
130
|
+
--spec-map '\A((?:packs/[^/]+/)?)app/(.+)\.rb\z=>\1test/\2_test.rb' \
|
|
131
|
+
--since "origin/${{ github.base_ref }}" --fail-fast
|
|
132
|
+
```
|
|
133
|
+
|
|
111
134
|
## Machine mode as a CI gate and artifact
|
|
112
135
|
|
|
113
136
|
`--json` makes the CLI double as the gate (its exit code follows the threshold)
|
data/docs/execution-runners.md
CHANGED
|
@@ -134,6 +134,18 @@ no fallback is possible.
|
|
|
134
134
|
in a fresh fork of its own clone. If a pooled worker dies mid-run, that
|
|
135
135
|
worker finishes its share of mutants through the file-based path with a
|
|
136
136
|
warning; the other workers stay in memory.
|
|
137
|
+
- With `--worker-env` the preloaded clones would all share the primary
|
|
138
|
+
worker's already-established database connection, so a parallel run
|
|
139
|
+
(`-p N`, N > 1) skips the in-memory runner with a warning unless
|
|
140
|
+
`--after-fork FILE` is also given. With `--after-fork`, each clone
|
|
141
|
+
receives its per-worker value of the `--worker-env` variable and loads
|
|
142
|
+
`FILE` right after forking, and that file re-establishes the per-worker
|
|
143
|
+
state (typically the ActiveRecord connection), which keeps the whole run
|
|
144
|
+
in memory with per-worker database isolation. A serial run (`-p 1`) has
|
|
145
|
+
only one worker and stays in memory without any hook. A clone whose
|
|
146
|
+
after-fork file raises reports the error on stderr and is dropped, and
|
|
147
|
+
its worker falls back to the file-based path; a missing after-fork file
|
|
148
|
+
makes the whole run fall back with a warning.
|
|
137
149
|
- Before any mutant runs, the runner re-applies the **unmutated** source in a
|
|
138
150
|
probe child and runs the suite. If that probe fails (for example the file has
|
|
139
151
|
top-level side effects that break on a second execution, or the class is
|
data/exe/mutation_test
CHANGED
|
@@ -60,6 +60,14 @@ OptionParser.new do |opts|
|
|
|
60
60
|
options[:spec_glob] = template
|
|
61
61
|
end
|
|
62
62
|
|
|
63
|
+
opts.on('--spec-map RULE', "Regex spec-mapping rule 'PATTERN#{MutationTester::BatchRunner::SPEC_MAP_SEPARATOR}REPLACEMENT' applied to the whole source path to build the whole spec path (e.g. '\\A((?:packs/[^/]+/)?)app/(.+)\\.rb\\z#{MutationTester::BatchRunner::SPEC_MAP_SEPARATOR}\\1test/\\2_test.rb'). Repeatable; the first matching rule wins and a source matching no rule falls back to --spec-glob. Requires a FILE list, --staged, or --glob.") do |rule|
|
|
64
|
+
(options[:spec_map] ||= []) << rule
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
opts.on('--minimum-score N', Float, "Mutation score percentage a file must reach to pass (default: #{MutationTester::Configuration::DEFAULT_MINIMUM_SCORE})") do |score|
|
|
68
|
+
options[:minimum_score] = score
|
|
69
|
+
end
|
|
70
|
+
|
|
63
71
|
opts.on('--since REV', 'Incremental batch mode: mutate only the files matched by --glob that changed since git revision REV (new files count as changed). Requires --glob.') do |rev|
|
|
64
72
|
options[:since] = rev
|
|
65
73
|
end
|
|
@@ -80,10 +88,14 @@ OptionParser.new do |opts|
|
|
|
80
88
|
options[:strict_equality] = true
|
|
81
89
|
end
|
|
82
90
|
|
|
83
|
-
opts.on('--worker-env NAME', 'Set environment variable NAME to a distinct per-worker value before each parallel worker boots (parallel_tests TEST_ENV_NUMBER convention: worker 0 -> "", worker N -> N+1), so a parallel_tests-style database.yml selects a per-worker database. Provision the databases yourself (e.g. rake parallel:prepare).
|
|
91
|
+
opts.on('--worker-env NAME', 'Set environment variable NAME to a distinct per-worker value before each parallel worker boots (parallel_tests TEST_ENV_NUMBER convention: worker 0 -> "", worker N -> N+1), so a parallel_tests-style database.yml selects a per-worker database. Provision the databases yourself (e.g. rake parallel:prepare). Without --after-fork a parallel in_memory run falls back to fork (preloaded clones share one database connection); pair it with --after-fork to keep the in_memory runner. A serial run (-p 1) decides mutants in a shadow workspace instead of mutating the checkout in place.') do |name|
|
|
84
92
|
options[:worker_env] = name
|
|
85
93
|
end
|
|
86
94
|
|
|
95
|
+
opts.on('--after-fork FILE', 'Ruby file loaded inside each preloaded in-memory clone right after it forks and receives its per-worker environment (see --worker-env), so the app can re-establish per-worker state such as its database connection (e.g. a file calling ActiveRecord::Base.establish_connection). With --worker-env set, this keeps the in_memory runner available in parallel runs.') do |file|
|
|
96
|
+
options[:after_fork] = file
|
|
97
|
+
end
|
|
98
|
+
|
|
87
99
|
opts.on('-h', '--help', 'Show this help message') do
|
|
88
100
|
puts opts
|
|
89
101
|
exit
|
|
@@ -150,6 +162,13 @@ end
|
|
|
150
162
|
glob_mode = !options[:glob].nil?
|
|
151
163
|
staged_mode = options.fetch(:staged, false)
|
|
152
164
|
|
|
165
|
+
spec_map = begin
|
|
166
|
+
(options[:spec_map] || []).map { |rule| MutationTester::BatchRunner.parse_spec_map(rule) }
|
|
167
|
+
rescue MutationTester::Error => e
|
|
168
|
+
usage_error.call(e.message,
|
|
169
|
+
"Quote the rule so the shell keeps it intact, e.g. --spec-map '\\Aapp/(.+)\\.rb\\z=>test/\\1_test.rb'.")
|
|
170
|
+
end
|
|
171
|
+
|
|
153
172
|
apply_configuration = lambda do
|
|
154
173
|
MutationTester.configure do |config|
|
|
155
174
|
config.parallel_processes = options[:parallel] if options[:parallel]
|
|
@@ -158,10 +177,12 @@ apply_configuration = lambda do
|
|
|
158
177
|
config.show_progress = !options[:no_progress]
|
|
159
178
|
config.reporters = selected_reporters if selected_reporters
|
|
160
179
|
config.output_dir = options[:output_dir] if options[:output_dir]
|
|
180
|
+
config.minimum_score = options[:minimum_score] if options[:minimum_score]
|
|
161
181
|
config.test_selection = false if options[:no_test_selection]
|
|
162
182
|
config.timeout_factor = options[:timeout_factor] if options[:timeout_factor]
|
|
163
183
|
config.timeout_policy = options[:timeout_policy] if options[:timeout_policy]
|
|
164
184
|
config.worker_env_var = options[:worker_env] if options.key?(:worker_env)
|
|
185
|
+
config.after_fork_file = options[:after_fork] if options.key?(:after_fork)
|
|
165
186
|
config.fail_fast = options.fetch(:fail_fast, false)
|
|
166
187
|
config.mutation_types[:strict_equality] = true if options[:strict_equality]
|
|
167
188
|
|
|
@@ -224,18 +245,17 @@ if glob_mode
|
|
|
224
245
|
batch = MutationTester::BatchRunner.new(
|
|
225
246
|
glob: options[:glob],
|
|
226
247
|
spec_template: options[:spec_glob],
|
|
248
|
+
spec_map: spec_map,
|
|
227
249
|
config: MutationTester.configuration,
|
|
228
250
|
since: options[:since],
|
|
229
251
|
changed_files: changed_files
|
|
230
252
|
)
|
|
231
253
|
|
|
232
|
-
run_batch_machine.call(batch, ->(result) { result.
|
|
254
|
+
run_batch_machine.call(batch, ->(result) { result.gate_passed? }) if json_mode
|
|
233
255
|
|
|
234
256
|
result = guard_interrupt.call { batch.run }
|
|
235
257
|
|
|
236
|
-
exit
|
|
237
|
-
|
|
238
|
-
exit(result.success? ? 0 : 1)
|
|
258
|
+
exit(result.gate_passed? ? 0 : 1)
|
|
239
259
|
end
|
|
240
260
|
|
|
241
261
|
if !staged_mode && ARGV.empty?
|
|
@@ -255,6 +275,11 @@ if options[:spec_glob] && legacy_pair
|
|
|
255
275
|
'Use --spec-glob with a positional FILE list, --staged, or --glob.')
|
|
256
276
|
end
|
|
257
277
|
|
|
278
|
+
if options[:spec_map] && legacy_pair
|
|
279
|
+
usage_error.call('--spec-map does not apply to an explicit SOURCE_FILE TEST_FILE pair.',
|
|
280
|
+
'Use --spec-map with a positional FILE list, --staged, or --glob.')
|
|
281
|
+
end
|
|
282
|
+
|
|
258
283
|
if legacy_pair
|
|
259
284
|
source_file, test_file = ARGV
|
|
260
285
|
else
|
|
@@ -278,7 +303,7 @@ else
|
|
|
278
303
|
single_file_json = json_mode && !staged_mode && sources.length == 1
|
|
279
304
|
|
|
280
305
|
if single_file_json
|
|
281
|
-
outcome = MutationTester::BatchRunner.new(files: sources, spec_template: options[:spec_glob]).classify(sources.first)
|
|
306
|
+
outcome = MutationTester::BatchRunner.new(files: sources, spec_template: options[:spec_glob], spec_map: spec_map).classify(sources.first)
|
|
282
307
|
if outcome.is_a?(MutationTester::BatchRunner::SkippedEntry)
|
|
283
308
|
reason = MutationTester::BatchRunner::SKIP_REASONS.fetch(outcome.reason)
|
|
284
309
|
detail = outcome.expected_spec ? " (expected #{outcome.expected_spec})" : ''
|
|
@@ -293,14 +318,15 @@ else
|
|
|
293
318
|
batch = MutationTester::BatchRunner.new(
|
|
294
319
|
files: sources,
|
|
295
320
|
spec_template: options[:spec_glob],
|
|
321
|
+
spec_map: spec_map,
|
|
296
322
|
config: MutationTester.configuration
|
|
297
323
|
)
|
|
298
324
|
|
|
299
|
-
run_batch_machine.call(batch, ->(result) { result.
|
|
325
|
+
run_batch_machine.call(batch, ->(result) { result.gate_passed? }) if json_mode
|
|
300
326
|
|
|
301
327
|
result = guard_interrupt.call { batch.run }
|
|
302
328
|
|
|
303
|
-
exit(result.
|
|
329
|
+
exit(result.gate_passed? ? 0 : 1)
|
|
304
330
|
end
|
|
305
331
|
end
|
|
306
332
|
|
|
@@ -9,6 +9,8 @@ module MutationTester
|
|
|
9
9
|
|
|
10
10
|
DEFAULT_SPEC_TEMPLATE = 'spec/{name}_spec.rb'.freeze
|
|
11
11
|
|
|
12
|
+
SPEC_MAP_SEPARATOR = '=>'.freeze
|
|
13
|
+
|
|
12
14
|
TEST_FILE_SUFFIXES = ['_spec.rb', '.spec.rb', '_test.rb'].freeze
|
|
13
15
|
|
|
14
16
|
SKIP_REASONS = {
|
|
@@ -57,6 +59,18 @@ module MutationTester
|
|
|
57
59
|
!(processed.empty? && skipped.empty? && (unchanged || []).empty?)
|
|
58
60
|
end
|
|
59
61
|
|
|
62
|
+
def nothing_measured?
|
|
63
|
+
processed.empty? && !skipped.empty?
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def gate_passed?
|
|
67
|
+
return false unless matched_any?
|
|
68
|
+
return false if nothing_measured?
|
|
69
|
+
return true if processed.empty?
|
|
70
|
+
|
|
71
|
+
success?
|
|
72
|
+
end
|
|
73
|
+
|
|
60
74
|
def interrupted?
|
|
61
75
|
!!interrupted
|
|
62
76
|
end
|
|
@@ -100,6 +114,24 @@ module MutationTester
|
|
|
100
114
|
.map { |path| Pathname.new(File.expand_path(path, toplevel)).relative_path_from(base).to_s }
|
|
101
115
|
end
|
|
102
116
|
|
|
117
|
+
def self.parse_spec_map(rule)
|
|
118
|
+
pattern, replacement = rule.to_s.split(SPEC_MAP_SEPARATOR, 2)
|
|
119
|
+
if replacement.nil?
|
|
120
|
+
raise MutationTester::Error,
|
|
121
|
+
"--spec-map #{rule.to_s.inspect} has no #{SPEC_MAP_SEPARATOR} separator; " \
|
|
122
|
+
"write it as 'PATTERN#{SPEC_MAP_SEPARATOR}REPLACEMENT'"
|
|
123
|
+
end
|
|
124
|
+
if pattern.empty?
|
|
125
|
+
raise MutationTester::Error, "--spec-map #{rule.to_s.inspect} has an empty pattern"
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
begin
|
|
129
|
+
[Regexp.new(pattern), replacement]
|
|
130
|
+
rescue RegexpError => e
|
|
131
|
+
raise MutationTester::Error, "--spec-map pattern #{pattern.inspect} is not a valid regular expression: #{e.message}"
|
|
132
|
+
end
|
|
133
|
+
end
|
|
134
|
+
|
|
103
135
|
def self.test_file?(path)
|
|
104
136
|
basename = File.basename(path)
|
|
105
137
|
return true if TEST_FILE_SUFFIXES.any? { |suffix| basename.end_with?(suffix) }
|
|
@@ -118,12 +150,13 @@ module MutationTester
|
|
|
118
150
|
end
|
|
119
151
|
private_class_method :git_capture
|
|
120
152
|
|
|
121
|
-
def initialize(glob: nil, files: nil, spec_template: nil, config: MutationTester.configuration, since: nil, changed_files: nil)
|
|
153
|
+
def initialize(glob: nil, files: nil, spec_template: nil, spec_map: nil, config: MutationTester.configuration, since: nil, changed_files: nil)
|
|
122
154
|
raise ArgumentError, 'provide exactly one of glob: or files:' unless glob.nil? ^ files.nil?
|
|
123
155
|
|
|
124
156
|
@glob = glob
|
|
125
157
|
@files = files
|
|
126
158
|
@spec_template = spec_template.nil? || spec_template.empty? ? DEFAULT_SPEC_TEMPLATE : spec_template
|
|
159
|
+
@spec_map = spec_map || []
|
|
127
160
|
@config = config
|
|
128
161
|
@since = since
|
|
129
162
|
@changed_files = changed_files
|
|
@@ -203,6 +236,10 @@ module MutationTester
|
|
|
203
236
|
end
|
|
204
237
|
|
|
205
238
|
def spec_path_for(source_file)
|
|
239
|
+
normalized = source_file.sub(%r{\A\./}, '')
|
|
240
|
+
pattern, replacement = @spec_map.find { |rule_pattern, _| rule_pattern.match?(normalized) }
|
|
241
|
+
return normalized.sub(pattern, replacement) if pattern
|
|
242
|
+
|
|
206
243
|
@spec_template.gsub(NAME_PLACEHOLDER, source_name(source_file))
|
|
207
244
|
end
|
|
208
245
|
|
|
@@ -254,8 +291,9 @@ module MutationTester
|
|
|
254
291
|
puts Rainbow('=' * 80).bright
|
|
255
292
|
failed = result.processed.reject(&:passed?)
|
|
256
293
|
degraded = failed.select(&:degraded?)
|
|
257
|
-
if
|
|
258
|
-
|
|
294
|
+
if result.nothing_measured?
|
|
295
|
+
scope = @files ? 'listed' : 'matched'
|
|
296
|
+
puts Rainbow("❌ No files were mutation-tested: every #{scope} file was skipped").red
|
|
259
297
|
elsif result.success?
|
|
260
298
|
puts Rainbow('✓ All processed files met the mutation score threshold').green
|
|
261
299
|
elsif degraded.size == failed.size
|
|
@@ -5,6 +5,7 @@ module MutationTester
|
|
|
5
5
|
RUNNER_MODES = %i[auto fork spawn in_memory].freeze
|
|
6
6
|
TIMEOUT_POLICIES = %i[killed separate].freeze
|
|
7
7
|
AUTO_PARALLEL_CAP = 8
|
|
8
|
+
DEFAULT_MINIMUM_SCORE = 80.0
|
|
8
9
|
DEFAULT_TIMEOUT = 30
|
|
9
10
|
DEFAULT_TIMEOUT_FACTOR = 5
|
|
10
11
|
CALIBRATED_TIMEOUT_FLOOR = 5
|
|
@@ -18,7 +19,8 @@ module MutationTester
|
|
|
18
19
|
number <= 0 ? '' : (number + 1).to_s
|
|
19
20
|
end
|
|
20
21
|
|
|
21
|
-
attr_reader :parallel_processes, :runner, :worker_env_var, :timeout, :timeout_factor,
|
|
22
|
+
attr_reader :parallel_processes, :runner, :worker_env_var, :after_fork_file, :timeout, :timeout_factor,
|
|
23
|
+
:timeout_policy
|
|
22
24
|
|
|
23
25
|
attr_accessor :baseline_duration,
|
|
24
26
|
:baseline_timeout,
|
|
@@ -37,6 +39,7 @@ module MutationTester
|
|
|
37
39
|
self.parallel_processes = ENV['MUTATION_TESTER_PARALLEL_PROCESSES'] || self.class.auto_parallel_processes
|
|
38
40
|
self.runner = ENV['MUTATION_TESTER_RUNNER'] || :auto
|
|
39
41
|
self.worker_env_var = ENV['MUTATION_TESTER_WORKER_ENV']
|
|
42
|
+
self.after_fork_file = ENV['MUTATION_TESTER_AFTER_FORK']
|
|
40
43
|
@timeout = DEFAULT_TIMEOUT
|
|
41
44
|
@timeout_factor = DEFAULT_TIMEOUT_FACTOR
|
|
42
45
|
@timeout_policy = :killed
|
|
@@ -56,7 +59,7 @@ module MutationTester
|
|
|
56
59
|
}
|
|
57
60
|
@reporters = %i[console html json]
|
|
58
61
|
@output_dir = 'tmp/mutation_reports'
|
|
59
|
-
@minimum_score =
|
|
62
|
+
@minimum_score = DEFAULT_MINIMUM_SCORE
|
|
60
63
|
@fail_on_threshold = true
|
|
61
64
|
@verbose = false
|
|
62
65
|
@show_file_path = true
|
|
@@ -135,6 +138,11 @@ module MutationTester
|
|
|
135
138
|
@worker_env_var = normalized.empty? ? nil : normalized
|
|
136
139
|
end
|
|
137
140
|
|
|
141
|
+
def after_fork_file=(value)
|
|
142
|
+
normalized = value.to_s.strip
|
|
143
|
+
@after_fork_file = normalized.empty? ? nil : File.expand_path(normalized)
|
|
144
|
+
end
|
|
145
|
+
|
|
138
146
|
def worker_env_assignment(index)
|
|
139
147
|
return nil unless @worker_env_var
|
|
140
148
|
|
|
@@ -238,6 +238,17 @@ spawn_clone = lambda do |request, out|
|
|
|
238
238
|
out.close
|
|
239
239
|
STDIN.reopen(File::NULL)
|
|
240
240
|
request['env']&.each { |name, value| ENV[name] = value }
|
|
241
|
+
if request['after_fork']
|
|
242
|
+
begin
|
|
243
|
+
load(request['after_fork'])
|
|
244
|
+
rescue ScriptError, StandardError => e
|
|
245
|
+
clone_out.puts(JSON.generate(
|
|
246
|
+
'event' => 'clone_error',
|
|
247
|
+
'message' => "the after-fork file #{request['after_fork']} raised #{e.class}: #{e.message}; this clone is unavailable"
|
|
248
|
+
))
|
|
249
|
+
raise
|
|
250
|
+
end
|
|
251
|
+
end
|
|
241
252
|
serve.call(input, clone_out)
|
|
242
253
|
end
|
|
243
254
|
Process.detach(child)
|
|
@@ -39,10 +39,10 @@ module MutationTester
|
|
|
39
39
|
refill_pool(pool_key(use_bundle_exec, framework), count, primary, env_for: env_for)
|
|
40
40
|
end
|
|
41
41
|
|
|
42
|
-
def prepare_in_memory_pool(count, primary)
|
|
42
|
+
def prepare_in_memory_pool(count, primary, env_for: nil, after_fork: nil)
|
|
43
43
|
return [] unless available? && primary&.ready?
|
|
44
44
|
|
|
45
|
-
refill_pool(IN_MEMORY_POOL_KEY, count, primary)
|
|
45
|
+
refill_pool(IN_MEMORY_POOL_KEY, count, primary, env_for: env_for, after_fork: after_fork)
|
|
46
46
|
end
|
|
47
47
|
|
|
48
48
|
def in_memory_pool_prepared?
|
|
@@ -116,13 +116,13 @@ module MutationTester
|
|
|
116
116
|
Parallel.worker_number if defined?(Parallel) && Parallel.respond_to?(:worker_number)
|
|
117
117
|
end
|
|
118
118
|
|
|
119
|
-
def refill_pool(key, count, primary, env_for: nil)
|
|
119
|
+
def refill_pool(key, count, primary, env_for: nil, after_fork: nil)
|
|
120
120
|
entry = (pool[key] ||= { owner: Process.pid, runners: [] })
|
|
121
121
|
entry[:runners] = entry[:runners].each_with_index.map do |runner, index|
|
|
122
|
-
runner&.ready? ? runner : primary.fork_clone(env: env_for&.call(index))
|
|
122
|
+
runner&.ready? ? runner : primary.fork_clone(env: env_for&.call(index), after_fork: after_fork)
|
|
123
123
|
end
|
|
124
124
|
entry[:runners].size.upto(count - 1) do |index|
|
|
125
|
-
entry[:runners] << primary.fork_clone(env: env_for&.call(index))
|
|
125
|
+
entry[:runners] << primary.fork_clone(env: env_for&.call(index), after_fork: after_fork)
|
|
126
126
|
end
|
|
127
127
|
entry[:runners]
|
|
128
128
|
end
|
|
@@ -198,7 +198,7 @@ module MutationTester
|
|
|
198
198
|
fail_worker
|
|
199
199
|
end
|
|
200
200
|
|
|
201
|
-
def fork_clone(env: nil)
|
|
201
|
+
def fork_clone(env: nil, after_fork: nil)
|
|
202
202
|
return nil unless ready?
|
|
203
203
|
return nil unless File.respond_to?(:mkfifo)
|
|
204
204
|
|
|
@@ -210,6 +210,7 @@ module MutationTester
|
|
|
210
210
|
|
|
211
211
|
clone_request = { 'job' => job_path, 'events' => events_path }
|
|
212
212
|
clone_request['env'] = env if env
|
|
213
|
+
clone_request['after_fork'] = after_fork if after_fork
|
|
213
214
|
@job_writer.puts(JSON.generate('clone' => clone_request))
|
|
214
215
|
event = read_event(monotonic_time + CLONE_TIMEOUT)
|
|
215
216
|
return nil unless event.is_a?(Hash) && event['event'] == 'cloned'
|
|
@@ -264,6 +265,11 @@ module MutationTester
|
|
|
264
265
|
next
|
|
265
266
|
end
|
|
266
267
|
|
|
268
|
+
if event['event'] == 'clone_error'
|
|
269
|
+
warn("[MutationTester] #{event['message']}")
|
|
270
|
+
return false
|
|
271
|
+
end
|
|
272
|
+
|
|
267
273
|
return event['event'] == 'ready'
|
|
268
274
|
end
|
|
269
275
|
end
|
|
@@ -56,16 +56,17 @@ module MutationTester
|
|
|
56
56
|
elsif in_memory_first?
|
|
57
57
|
run_in_memory_series(mutations, &progress_callback)
|
|
58
58
|
elsif @config.parallel_processes == 1
|
|
59
|
-
|
|
59
|
+
run_file_based_series(mutations, &progress_callback)
|
|
60
60
|
else
|
|
61
61
|
run_in_shadow_parallel(mutations, &progress_callback)
|
|
62
62
|
end
|
|
63
63
|
end
|
|
64
64
|
|
|
65
65
|
def in_memory_first?
|
|
66
|
-
return false
|
|
66
|
+
return false unless %i[in_memory auto].include?(@config.runner)
|
|
67
|
+
return true unless @config.worker_env_var
|
|
67
68
|
|
|
68
|
-
|
|
69
|
+
@config.parallel_processes == 1 || !@config.after_fork_file.nil?
|
|
69
70
|
end
|
|
70
71
|
|
|
71
72
|
def run_in_memory_series(mutations, &progress_callback)
|
|
@@ -100,7 +101,12 @@ module MutationTester
|
|
|
100
101
|
blocker = prepare_in_memory_execution
|
|
101
102
|
return fall_back_to_parallel_file_based(blocker, mutations, &progress_callback) if blocker
|
|
102
103
|
|
|
103
|
-
pool = ForkRunner.prepare_in_memory_pool(
|
|
104
|
+
pool = ForkRunner.prepare_in_memory_pool(
|
|
105
|
+
[@config.parallel_processes, mutations.size].min,
|
|
106
|
+
@in_memory_runner,
|
|
107
|
+
env_for: worker_env_for,
|
|
108
|
+
after_fork: @config.after_fork_file
|
|
109
|
+
)
|
|
104
110
|
if pool.compact.empty?
|
|
105
111
|
return fall_back_to_parallel_file_based('the preloaded worker pool could not be cloned', mutations, &progress_callback)
|
|
106
112
|
end
|
|
@@ -161,6 +167,28 @@ module MutationTester
|
|
|
161
167
|
cleanup_shadow_workspaces
|
|
162
168
|
end
|
|
163
169
|
|
|
170
|
+
def run_file_based_series(mutations, &progress_callback)
|
|
171
|
+
project_root = @config.worker_env_var ? discoverable_project_root : nil
|
|
172
|
+
return run_in_place_series(mutations, &progress_callback) unless project_root
|
|
173
|
+
|
|
174
|
+
run_in_shadow_series(mutations, project_root, &progress_callback)
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
def run_in_shadow_series(mutations, project_root, &progress_callback)
|
|
178
|
+
warn "[MutationTester] --worker-env #{@config.worker_env_var} is set, so the serial run decides each mutant in a shadow workspace and never mutates the checkout in place."
|
|
179
|
+
shadow_run_root
|
|
180
|
+
results = []
|
|
181
|
+
mutations.each_with_index do |mutation, index|
|
|
182
|
+
result = run_single_mutation(mutation, :shadow, project_root)
|
|
183
|
+
progress_callback.call(mutation, index + 1) if progress_callback
|
|
184
|
+
results << result
|
|
185
|
+
break if stop_early?(result)
|
|
186
|
+
end
|
|
187
|
+
results
|
|
188
|
+
ensure
|
|
189
|
+
cleanup_shadow_workspaces
|
|
190
|
+
end
|
|
191
|
+
|
|
164
192
|
def run_in_place_series(mutations, &progress_callback)
|
|
165
193
|
write_in_place_backup
|
|
166
194
|
results = []
|
|
@@ -401,6 +429,9 @@ module MutationTester
|
|
|
401
429
|
|
|
402
430
|
def prepare_in_memory_execution
|
|
403
431
|
return 'Process.fork is not supported on this platform' unless ForkRunner.available?
|
|
432
|
+
if @config.after_fork_file && !File.exist?(@config.after_fork_file)
|
|
433
|
+
return "the after-fork file #{@config.after_fork_file} does not exist"
|
|
434
|
+
end
|
|
404
435
|
if InMemoryLoader.load_time_defined_guard?(@original_content)
|
|
405
436
|
return 'the source file uses defined? at load time, so redefinition would silently skip the guarded code'
|
|
406
437
|
end
|
|
@@ -451,7 +482,7 @@ module MutationTester
|
|
|
451
482
|
offset_callback = progress_callback && lambda do |mutation, index|
|
|
452
483
|
progress_callback.call(mutation, completed + index)
|
|
453
484
|
end
|
|
454
|
-
|
|
485
|
+
run_file_based_series(mutations, &offset_callback)
|
|
455
486
|
end
|
|
456
487
|
|
|
457
488
|
def fall_back_to_parallel_file_based(reason, mutations, &progress_callback)
|
|
@@ -470,8 +501,9 @@ module MutationTester
|
|
|
470
501
|
def announce_worker_env_in_memory_opt_out
|
|
471
502
|
return unless @config.worker_env_var
|
|
472
503
|
return unless %i[in_memory auto].include?(@config.runner)
|
|
504
|
+
return if @config.parallel_processes == 1 || @config.after_fork_file
|
|
473
505
|
|
|
474
|
-
warn "[MutationTester] --worker-env #{@config.worker_env_var} is set, so the in-memory runner is skipped (its clones share one preloaded database connection); using the fork runner for per-worker database isolation."
|
|
506
|
+
warn "[MutationTester] --worker-env #{@config.worker_env_var} is set without --after-fork, so the in-memory runner is skipped (its clones share one preloaded database connection); using the fork runner for per-worker database isolation. Pass --after-fork FILE to keep the in-memory runner and re-establish per-worker connections inside each clone."
|
|
475
507
|
end
|
|
476
508
|
|
|
477
509
|
def run_mutation_load_time(mutation, result, project_root)
|
data/readme.md
CHANGED
|
@@ -204,11 +204,14 @@ mutation_test [OPTIONS] --glob 'lib/**/*.rb'
|
|
|
204
204
|
| `--staged` | Mutation-test the files staged in git (`git diff --cached --name-only`; files staged as deleted are ignored), mapping each to its spec like a positional `FILE` list. Cannot be combined with positional arguments or `--glob`. See [File lists and --staged](#file-lists-and---staged-test-what-you-changed). |
|
|
205
205
|
| `--glob PATTERN` | Batch mode: mutation-test every source file matching `PATTERN`, mapping each to its spec by convention (see [Batch mode](#batch-mode-run-many-files-in-one-command)). |
|
|
206
206
|
| `--spec-glob TEMPLATE` | Spec-mapping template with a `{name}` placeholder (default: `spec/{name}_spec.rb`). Requires a positional `FILE` list, `--staged`, or `--glob`. |
|
|
207
|
+
| `--spec-map RULE` | Spec-mapping rule `'PATTERN=>REPLACEMENT'`: a regular expression applied to the whole source path to build the whole spec path, for layouts a `{name}` template cannot express (Rails `app/` -> `test/`, engines, Packwerk packs). Repeatable, first matching rule wins, a source matching no rule falls back to `--spec-glob`. Requires a positional `FILE` list, `--staged`, or `--glob`. See [Mapping sources to specs](#mapping-sources-to-specs). |
|
|
208
|
+
| `--minimum-score N` | Mutation score percentage a file must reach to pass (default: 80). Drives the `PASS`/`FAIL` verdict and the exit code. |
|
|
207
209
|
| `--since REV` | Incremental batch mode: mutate only the files matched by `--glob` that changed since git revision `REV` (new files count as changed). Requires `--glob`. See [Incremental mode](#incremental-mode-mutate-only-what-changed). |
|
|
208
210
|
| `--fail-fast` | Stop the run at the first surviving mutant and finish with a failing status. Works in single-file mode and with `--glob`. |
|
|
209
211
|
| `--timeout-factor N` | Per-mutant timeout budget as `N` times the measured baseline test run, never below 5 s (default: 5, must be > 0). Ignored when `config.timeout` is set explicitly, which keeps a fixed budget. See [Configuration](#configuration). |
|
|
210
212
|
| `--timeout-policy MODE` | Scoring policy for timed-out mutants: `killed` (default) counts a timeout as a kill; `separate` keeps timeouts out of the score entirely (`killed / (killed + survived)`) and reports them only as their own category in the console, JSON and HTML reports. |
|
|
211
|
-
| `--worker-env NAME` | Set environment variable `NAME` to a distinct per-worker value before each parallel worker boots (`parallel_tests` `TEST_ENV_NUMBER` convention: worker 0 -> `""`, worker N -> `N+1`), so a `parallel_tests`-style `database.yml` selects a per-worker database. You provision the databases (e.g. `rake parallel:prepare`).
|
|
213
|
+
| `--worker-env NAME` | Set environment variable `NAME` to a distinct per-worker value before each parallel worker boots (`parallel_tests` `TEST_ENV_NUMBER` convention: worker 0 -> `""`, worker N -> `N+1`), so a `parallel_tests`-style `database.yml` selects a per-worker database. You provision the databases (e.g. `rake parallel:prepare`). A parallel `in_memory` run falls back to `fork` unless `--after-fork` is also given; a serial run (`-p 1`) decides mutants in a shadow workspace instead of mutating the checkout in place. See [Making parallelism work with Rails](#making-parallelism-work-with-rails). |
|
|
214
|
+
| `--after-fork FILE` | Ruby file loaded inside each preloaded in-memory clone right after it forks and receives its per-worker environment (see `--worker-env`), so the app can re-establish per-worker state such as its database connection. With `--worker-env` set, this keeps the `in_memory` runner available in parallel runs. A clone whose after-fork file raises is dropped with a stderr warning and its worker falls back to file-based execution. See [Making parallelism work with Rails](#making-parallelism-work-with-rails). |
|
|
212
215
|
| `--strict-equality` | Enable the opt-in strict-equality probes (`==` → `eql?` and `==` → `equal?`). Default off; expect noise on code that does not distinguish numeric types or object identity. See [Strict Equality Mutations](docs/mutation-types.md#strict-equality-mutations-opt-in). |
|
|
213
216
|
| `-h, --help` | Show help message. |
|
|
214
217
|
| `-v, --version` | Show version. |
|
|
@@ -314,8 +317,8 @@ reported as `SKIPPED` with an explicit reason and never count as a success:
|
|
|
314
317
|
- **not a Ruby source file** - e.g. a staged `.md` or config file.
|
|
315
318
|
- **a test file, not a mutable source** - a test file passed directly
|
|
316
319
|
(`*_spec.rb`, `*.spec.rb`, `*_test.rb`, `test_*.rb`, or minitest content).
|
|
317
|
-
- **no matching spec file** - the convention (or `--spec-glob`
|
|
318
|
-
spec that does not exist; the expected path is printed.
|
|
320
|
+
- **no matching spec file** - the convention (or `--spec-glob` / `--spec-map`)
|
|
321
|
+
points at a spec that does not exist; the expected path is printed.
|
|
319
322
|
|
|
320
323
|
Exit codes: `0` when at least one file was processed and every processed file
|
|
321
324
|
met the threshold; `1` when any processed file was below threshold or when
|
|
@@ -336,16 +339,8 @@ run, so you no longer need to script a loop around `mutation_test` or depend on
|
|
|
336
339
|
the Rails-only `rake mutation:test_models` task.
|
|
337
340
|
|
|
338
341
|
Each matched source file is mapped to its spec by convention: `lib/X.rb` becomes
|
|
339
|
-
`spec/X_spec.rb`.
|
|
340
|
-
|
|
341
|
-
stripped, subdirectories preserved (`lib/foo/bar.rb` -> `spec/foo/bar_spec.rb`).
|
|
342
|
-
|
|
343
|
-
Override the convention with `--spec-glob TEMPLATE`, a template containing the
|
|
344
|
-
`{name}` placeholder. For a Minitest project laid out under `test/`:
|
|
345
|
-
|
|
346
|
-
```bash
|
|
347
|
-
bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
348
|
-
```
|
|
342
|
+
`spec/X_spec.rb`. See [Mapping sources to specs](#mapping-sources-to-specs) for
|
|
343
|
+
the two ways to override that convention.
|
|
349
344
|
|
|
350
345
|
Behaviour:
|
|
351
346
|
|
|
@@ -359,18 +354,27 @@ Behaviour:
|
|
|
359
354
|
file with its score and `PASS`/`FAIL` against the threshold, followed by a
|
|
360
355
|
clearly separated `SKIPPED` list.
|
|
361
356
|
- **A source file with no matching spec is `SKIPPED`**, reported explicitly and
|
|
362
|
-
never counted as a success. A skipped file
|
|
357
|
+
never counted as a success. A single skipped file next to processed ones does
|
|
358
|
+
not by itself fail the run, but a run that skipped *every* matched file
|
|
359
|
+
measured nothing and fails (see the exit codes below).
|
|
363
360
|
|
|
364
361
|
Exit codes:
|
|
365
362
|
|
|
366
|
-
- `0` -
|
|
367
|
-
`--since` run where nothing changed
|
|
363
|
+
- `0` - at least one file was mutation-tested and every processed file met the
|
|
364
|
+
mutation score threshold. A `--since` run where nothing changed also exits `0`
|
|
365
|
+
(see below).
|
|
368
366
|
- `1` - at least one processed file was below threshold, the glob matched no
|
|
369
|
-
source files at all,
|
|
370
|
-
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
367
|
+
source files at all, every matched file was skipped so nothing was actually
|
|
368
|
+
mutation-tested, or `--fail-fast` stopped the run at a surviving mutant.
|
|
369
|
+
- `2` - a usage error: `--spec-glob` or `--spec-map` with an explicit
|
|
370
|
+
`SOURCE_FILE TEST_FILE` pair, a malformed `--spec-map` rule, `--since` given
|
|
371
|
+
without `--glob`, `--staged` combined with positional arguments or `--glob`,
|
|
372
|
+
or `--since`/`--staged` used outside a git repository (for `--since` also an
|
|
373
|
+
unknown revision).
|
|
374
|
+
|
|
375
|
+
The "every matched file was skipped" case is deliberate: a typo in
|
|
376
|
+
`--spec-glob`/`--spec-map`, or a refactor that moves the test directory, would
|
|
377
|
+
otherwise leave a green CI step that measured nothing.
|
|
374
378
|
|
|
375
379
|
```bash
|
|
376
380
|
# Minitest project, JSON report per file, custom output directory
|
|
@@ -378,6 +382,66 @@ bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
|
378
382
|
--reporters json --output-dir build/mutation
|
|
379
383
|
```
|
|
380
384
|
|
|
385
|
+
### Mapping sources to specs
|
|
386
|
+
|
|
387
|
+
Every mode that takes more than an explicit `SOURCE_FILE TEST_FILE` pair (a
|
|
388
|
+
positional `FILE` list, `--staged`, `--glob`) derives the test path from the
|
|
389
|
+
source path. Two mechanisms do that, checked in this order:
|
|
390
|
+
|
|
391
|
+
1. `--spec-map 'PATTERN=>REPLACEMENT'` - regular-expression rules.
|
|
392
|
+
2. `--spec-glob TEMPLATE` - a `{name}` template (default `spec/{name}_spec.rb`).
|
|
393
|
+
|
|
394
|
+
**`--spec-glob TEMPLATE`** substitutes `{name}`, which is the source path with a
|
|
395
|
+
leading `lib/` segment removed and the `.rb` extension stripped, subdirectories
|
|
396
|
+
preserved (`lib/foo/bar.rb` -> `spec/foo/bar_spec.rb`). Because `{name}` is one
|
|
397
|
+
value, a template can only add a prefix and a suffix around the source path. For
|
|
398
|
+
a Minitest project laid out under `test/` that is enough:
|
|
399
|
+
|
|
400
|
+
```bash
|
|
401
|
+
bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
**`--spec-map 'PATTERN=>REPLACEMENT'`** covers the layouts a template cannot
|
|
405
|
+
express: those that substitute *inside* the path, after a variable-length
|
|
406
|
+
prefix. `PATTERN` is a Ruby regular expression matched against the whole source
|
|
407
|
+
path (a leading `./` removed); the first `=>` separates it from `REPLACEMENT`,
|
|
408
|
+
which may use `\1`, `\2`, ... backreferences and produces the whole spec path.
|
|
409
|
+
Only the first match in the path is replaced.
|
|
410
|
+
|
|
411
|
+
- The flag is repeatable and the first matching rule wins.
|
|
412
|
+
- A source that matches no rule falls back to `--spec-glob` (or the default
|
|
413
|
+
convention), so one command can cover `app/` and `lib/` at once.
|
|
414
|
+
- Quote the rule in single quotes so the shell leaves the backslashes alone.
|
|
415
|
+
|
|
416
|
+
Rails and Rails-shaped layouts, where the rule is "replace the `app/` segment
|
|
417
|
+
with `test/`, keep whatever prefix comes before it":
|
|
418
|
+
|
|
419
|
+
```bash
|
|
420
|
+
# Plain Rails, Minitest: app/models/current.rb -> test/models/current_test.rb
|
|
421
|
+
bundle exec mutation_test --glob 'app/**/*.rb' \
|
|
422
|
+
--spec-map '\Aapp/(.+)\.rb\z=>test/\1_test.rb'
|
|
423
|
+
|
|
424
|
+
# Plain Rails, RSpec: app/models/user.rb -> spec/models/user_spec.rb
|
|
425
|
+
bundle exec mutation_test --glob 'app/**/*.rb' \
|
|
426
|
+
--spec-map '\Aapp/(.+)\.rb\z=>spec/\1_spec.rb'
|
|
427
|
+
|
|
428
|
+
# Packwerk / packs-rails and engines, with the app root as an optional prefix:
|
|
429
|
+
# app/models/current.rb -> test/models/current_test.rb
|
|
430
|
+
# packs/identity/app/models/party.rb -> packs/identity/test/models/party_test.rb
|
|
431
|
+
# engines/billing/app/jobs/send_job.rb -> engines/billing/test/jobs/send_job_test.rb
|
|
432
|
+
bundle exec mutation_test --glob '{app,packs/*/app,engines/*/app}/**/*.rb' \
|
|
433
|
+
--spec-map '\A((?:(?:packs|engines)/[^/]+/)?)app/(.+)\.rb\z=>\1test/\2_test.rb'
|
|
434
|
+
|
|
435
|
+
# app/ through the rule, lib/ through the template, in one run
|
|
436
|
+
bundle exec mutation_test --glob '{app,lib}/**/*.rb' \
|
|
437
|
+
--spec-map '\Aapp/(.+)\.rb\z=>test/\1_test.rb' \
|
|
438
|
+
--spec-glob 'test/{name}_test.rb'
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
When a rule produces a path that does not exist, the file is reported as
|
|
442
|
+
`SKIPPED (no matching spec file)` with the expected path printed, and a run in
|
|
443
|
+
which *every* file was skipped that way fails with exit code `1`.
|
|
444
|
+
|
|
381
445
|
### Incremental mode: mutate only what changed
|
|
382
446
|
|
|
383
447
|
`--since REV` narrows a `--glob` batch to the files that changed since a git
|
|
@@ -447,6 +511,19 @@ MutationTester.configure do |config|
|
|
|
447
511
|
# setting a specific mode forces it. See the Execution model section below.
|
|
448
512
|
config.runner = :auto
|
|
449
513
|
|
|
514
|
+
# Per-worker database isolation (parallel_tests TEST_ENV_NUMBER convention).
|
|
515
|
+
# Set to an environment variable name to give each parallel worker a distinct
|
|
516
|
+
# value before it boots its test environment. Equivalent to the --worker-env
|
|
517
|
+
# CLI flag. See Making parallelism work with Rails below.
|
|
518
|
+
# config.worker_env_var = "TEST_ENV_NUMBER"
|
|
519
|
+
|
|
520
|
+
# Ruby file loaded inside each preloaded in-memory clone right after it forks
|
|
521
|
+
# and receives its per-worker environment, so the app can re-establish
|
|
522
|
+
# per-worker state such as its database connection. Keeps the in_memory
|
|
523
|
+
# runner available in parallel worker-env runs. Equivalent to the
|
|
524
|
+
# --after-fork CLI flag.
|
|
525
|
+
# config.after_fork_file = "db/mutation_after_fork.rb"
|
|
526
|
+
|
|
450
527
|
# Two-phase test selection (RSpec only): fast-kill on a matching example
|
|
451
528
|
# subset, always confirmed by the full file before a mutant is reported as
|
|
452
529
|
# survived. Set to false to always run the full file. See Execution model.
|
|
@@ -463,7 +540,7 @@ MutationTester.configure do |config|
|
|
|
463
540
|
# Set it explicitly to write elsewhere (this example uses "mutation_reports"):
|
|
464
541
|
config.output_dir = "mutation_reports"
|
|
465
542
|
|
|
466
|
-
# Quality thresholds
|
|
543
|
+
# Quality thresholds. The CLI flag --minimum-score overrides this per run.
|
|
467
544
|
config.minimum_score = 80.0
|
|
468
545
|
config.fail_on_threshold = true
|
|
469
546
|
|
|
@@ -581,16 +658,48 @@ bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb \
|
|
|
581
658
|
-p 4 --worker-env TEST_ENV_NUMBER
|
|
582
659
|
|
|
583
660
|
# Batch over a whole directory the same way
|
|
584
|
-
bundle exec mutation_test --glob 'app/models/**/*.rb'
|
|
661
|
+
bundle exec mutation_test --glob 'app/models/**/*.rb' \
|
|
662
|
+
--spec-map '\Aapp/(.+)\.rb\z=>spec/\1_spec.rb' \
|
|
585
663
|
-p 4 --worker-env TEST_ENV_NUMBER
|
|
586
664
|
```
|
|
587
665
|
|
|
588
666
|
`MUTATION_TESTER_WORKER_ENV=TEST_ENV_NUMBER` is equivalent to passing the flag.
|
|
589
667
|
|
|
590
|
-
**Runner support.** `--worker-env` works with the `fork` and `spawn` runners,
|
|
591
|
-
environment freshly and picks up the variable. The `in_memory` runner clones a single preloaded worker that
|
|
592
|
-
connected to one database, so
|
|
593
|
-
skips `in_memory` and uses `fork`, announcing the
|
|
668
|
+
**Runner support.** `--worker-env` works with the `fork` and `spawn` runners out of the box, because each mutant boots
|
|
669
|
+
its test environment freshly and picks up the variable. The `in_memory` runner clones a single preloaded worker that
|
|
670
|
+
has already connected to one database, so setting the variable alone cannot re-point an existing connection; in a
|
|
671
|
+
parallel run without `--after-fork` the runner selection therefore skips `in_memory` and uses `fork`, announcing the
|
|
672
|
+
reason on stderr. A serial run (`-p 1`) has only one worker and stays in memory.
|
|
673
|
+
|
|
674
|
+
**Keeping the in-memory runner with `--after-fork`.** Pass `--after-fork FILE` (or
|
|
675
|
+
`MUTATION_TESTER_AFTER_FORK=FILE`) to keep the `in_memory` runner in parallel worker-env runs. Each preloaded clone
|
|
676
|
+
then receives its per-worker value of the `--worker-env` variable and loads `FILE` right after forking, and that file
|
|
677
|
+
is where your app re-establishes its per-worker state. For a Rails app with a `TEST_ENV_NUMBER`-keyed `database.yml`
|
|
678
|
+
that usually means reconnecting ActiveRecord:
|
|
679
|
+
|
|
680
|
+
```ruby
|
|
681
|
+
# db/mutation_after_fork.rb
|
|
682
|
+
ActiveRecord::Base.establish_connection(
|
|
683
|
+
ActiveRecord::Base.configurations
|
|
684
|
+
.configs_for(env_name: 'test', name: 'primary')
|
|
685
|
+
.configuration_hash
|
|
686
|
+
.merge(database: "myapp_test#{ENV['TEST_ENV_NUMBER']}")
|
|
687
|
+
)
|
|
688
|
+
```
|
|
689
|
+
|
|
690
|
+
```bash
|
|
691
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb \
|
|
692
|
+
-p 4 --worker-env TEST_ENV_NUMBER --after-fork db/mutation_after_fork.rb
|
|
693
|
+
```
|
|
694
|
+
|
|
695
|
+
The file runs once per clone, inside the clone only (never in the primary preloaded worker or in your shell process).
|
|
696
|
+
If it raises, the clone reports the error on stderr and is dropped, and its worker decides its share of mutants through
|
|
697
|
+
the file-based path; if the file does not exist, the whole run falls back to file-based execution with a warning.
|
|
698
|
+
|
|
699
|
+
**Serial runs never mutate your checkout.** With `--worker-env` set, a `-p 1` run on the file-based runners decides
|
|
700
|
+
each mutant in a shadow workspace instead of writing mutants into the real source file, so a killed process cannot
|
|
701
|
+
leave a mutated file behind. (Without `--worker-env`, a serial file-based run still uses in-place mutation with a
|
|
702
|
+
`.mutation_backup` file that the next run restores automatically.)
|
|
594
703
|
|
|
595
704
|
**Still simplest without a parallel database setup:** if you have not provisioned per-worker databases, keep Rails
|
|
596
705
|
model runs on serial `-p 1`. `--worker-env` is only useful once the databases exist.
|
|
@@ -853,6 +962,14 @@ can drop the `jq` comparison and let the exit code be the gate:
|
|
|
853
962
|
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb || exit 1
|
|
854
963
|
```
|
|
855
964
|
|
|
965
|
+
`--minimum-score N` sets that threshold for a single run, which is how you start
|
|
966
|
+
below 80 in an existing codebase and ratchet the number up over time:
|
|
967
|
+
|
|
968
|
+
```sh
|
|
969
|
+
bundle exec mutation_test --glob 'app/**/*.rb' \
|
|
970
|
+
--spec-map '\Aapp/(.+)\.rb\z=>spec/\1_spec.rb' --minimum-score 60 || exit 1
|
|
971
|
+
```
|
|
972
|
+
|
|
856
973
|
lefthook or overcommit users: call the shipped hook from your `pre-push` step
|
|
857
974
|
instead of writing to `.git/hooks/`.
|
|
858
975
|
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutation_tester
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kamil Dzierbicki
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-08-
|
|
11
|
+
date: 2026-08-08 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: parallel
|