mutation_tester 1.4.1 → 1.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/Gemfile.lock +1 -1
- data/docs/ci.md +24 -1
- data/exe/mutation_test +28 -7
- data/lib/mutation_tester/batch_runner.rb +41 -3
- data/lib/mutation_tester/configuration.rb +2 -1
- data/lib/mutation_tester/version.rb +1 -1
- data/readme.md +94 -22
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c743765bd0adb455bf1162627a35cb5b3232d45cfb83dca91ab139248a3acac1
|
|
4
|
+
data.tar.gz: 71eb3f98c61271896c6c4cc2e3b9b69c62fee77d50b3d94c0237b0dac9eefd53
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3fac687f4fd5496daced6e16cf30e3dc39a0816fc5be7fe9f37caae4a05d3e088bdf1f80ee5b23f72610741d767cd410de81c6fcec09b4456d2936549f32266c
|
|
7
|
+
data.tar.gz: 13842db8fe228decfe6d3e165e5f80ea3c18af549e09051a3b12342f1ff3251ea5f81f8d375d1f0bdc5e76b6320f101ce791900016b8054630a3e0c2e14f3227
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.4.2] - 2026-08-05
|
|
4
|
+
|
|
5
|
+
- Added `--spec-map 'PATTERN=>REPLACEMENT'`, a repeatable rule that builds the whole spec path from the whole source path with a regular expression, for the layouts the `{name}` template cannot express. `{name}` is the source path minus a leading `lib/`, so a template can only wrap it with a prefix and a suffix; every mapping that substitutes *inside* the path, after a variable-length prefix, was out of reach: `packs/identity/app/models/party.rb -> packs/identity/test/models/party_test.rb` (Packwerk / packs-rails), the same shape with Rails engines, and the plain Rails `app/ -> test/` rule. Because `--since` requires `--glob` and `--glob` maps through `--spec-glob`, such a layout previously lost the entire batch tier at once and had to be reimplemented in a wrapper script. The first matching rule wins and a source matching no rule falls back to `--spec-glob`, so one run can cover `app/` through a rule and `lib/` through the template. The flag works with `--glob`, a positional `FILE` list and `--staged`; a malformed rule is a usage error (exit code `2`) reported before any mutation runs.
|
|
6
|
+
- Fixed `--glob` finishing with status `0` when every matched file was skipped. The two batch paths used different predicates over the same result, and the `--glob` one gated on "something matched and every processed file passed", which is vacuously true over an empty processed list. A run that matched four files and skipped all four (a wrong `--spec-glob`, a test directory moved by a refactor) printed `All processed files met the mutation score threshold`, set `"passed": true` in the JSON envelope and passed the run, which is the worst direction for a quality gate: it vouches for a score it never measured. Both paths now share one predicate: a batch that matched files but measured none fails with status `1` and prints `No files were mutation-tested`, while a `--since` run whose matched files were all unchanged still passes, so a pull request that touches nothing keeps its documented green run.
|
|
7
|
+
- Added `--minimum-score N` to set the threshold a file must reach for a single run. `config.minimum_score` (default 80) had no CLI flag and no environment variable, so the CLI's own gate was fixed at 80 and the shipped `examples/hooks/pre-push` worked around it with `--json` plus `jq`. An existing codebase can now start below 80 and raise the number over time without a wrapper.
|
|
8
|
+
|
|
3
9
|
## [1.4.1] - 2026-08-03
|
|
4
10
|
|
|
5
11
|
- Fixed parallel runs reporting every mutant as survived (`Killed: 0`, score `0.0%`) on projects whose test file reaches its source through `$LOAD_PATH` (a Rails app run with `-Itest` in `RUBYOPT`, for example). A preloaded worker is started once in the real project root, so Ruby absolutizes every `-I`/`RUBYLIB` entry against that directory; the forked child then changed into the shadow workspace but kept resolving `require "test_helper"` back to the original tree, which loaded the unmutated source. Each job now rewrites the `$LOAD_PATH` entries of the mirrored project root into the workspace, so a preloaded worker resolves project code exactly like the `spawn` runner that starts inside the workspace. Only Minitest suites hit this in practice: RSpec re-adds `lib` and its default path at run time, after the child has changed directory. Serial runs and the `spawn` runner were never affected, which is why the same file scored 96.55% serially and 0.0% with `-p 8`.
|
data/Gemfile.lock
CHANGED
data/docs/ci.md
CHANGED
|
@@ -29,7 +29,8 @@ The same thing, condensed to a copy-pasteable minimal workflow. It matches the
|
|
|
29
29
|
maintained template above; reach for the template when you want the parallel and
|
|
30
30
|
multi-file variants. The quality gate needs no extra configuration: the CLI exits
|
|
31
31
|
non-zero when the mutation score is below the threshold, which fails the step (and
|
|
32
|
-
the job).
|
|
32
|
+
the job). The threshold defaults to 80 and `--minimum-score N` sets it for a
|
|
33
|
+
single run, so an existing codebase can start lower and ratchet the number up.
|
|
33
34
|
|
|
34
35
|
```yaml
|
|
35
36
|
name: Mutation Testing
|
|
@@ -108,6 +109,28 @@ jobs:
|
|
|
108
109
|
A PR that touches nothing under `lib/` exits `0` with a "Nothing to mutate"
|
|
109
110
|
message, so the gate never blocks unrelated changes.
|
|
110
111
|
|
|
112
|
+
### What the batch exit code guarantees
|
|
113
|
+
|
|
114
|
+
A batch run exits `0` only when at least one file was actually mutation-tested
|
|
115
|
+
and every processed file met the threshold, or when the `--since` filter left
|
|
116
|
+
nothing to mutate. If the glob matched files but *every* one of them was skipped
|
|
117
|
+
(most often `no matching spec file`), the run exits `1`: nothing was measured, so
|
|
118
|
+
the gate has nothing to vouch for. That keeps a typo in `--spec-glob` /
|
|
119
|
+
`--spec-map`, or a refactor that moves the test directory, from silently turning
|
|
120
|
+
a green job into a job that measures nothing.
|
|
121
|
+
|
|
122
|
+
For an application whose tests do not sit under `spec/`, map the paths with
|
|
123
|
+
`--spec-map` (see
|
|
124
|
+
[Mapping sources to specs](../readme.md#mapping-sources-to-specs)):
|
|
125
|
+
|
|
126
|
+
```yaml
|
|
127
|
+
- name: Run mutation tests on changed files only
|
|
128
|
+
run: |
|
|
129
|
+
bundle exec mutation_test --glob '{app,packs/*/app}/**/*.rb' \
|
|
130
|
+
--spec-map '\A((?:packs/[^/]+/)?)app/(.+)\.rb\z=>\1test/\2_test.rb' \
|
|
131
|
+
--since "origin/${{ github.base_ref }}" --fail-fast
|
|
132
|
+
```
|
|
133
|
+
|
|
111
134
|
## Machine mode as a CI gate and artifact
|
|
112
135
|
|
|
113
136
|
`--json` makes the CLI double as the gate (its exit code follows the threshold)
|
data/exe/mutation_test
CHANGED
|
@@ -60,6 +60,14 @@ OptionParser.new do |opts|
|
|
|
60
60
|
options[:spec_glob] = template
|
|
61
61
|
end
|
|
62
62
|
|
|
63
|
+
opts.on('--spec-map RULE', "Regex spec-mapping rule 'PATTERN#{MutationTester::BatchRunner::SPEC_MAP_SEPARATOR}REPLACEMENT' applied to the whole source path to build the whole spec path (e.g. '\\A((?:packs/[^/]+/)?)app/(.+)\\.rb\\z#{MutationTester::BatchRunner::SPEC_MAP_SEPARATOR}\\1test/\\2_test.rb'). Repeatable; the first matching rule wins and a source matching no rule falls back to --spec-glob. Requires a FILE list, --staged, or --glob.") do |rule|
|
|
64
|
+
(options[:spec_map] ||= []) << rule
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
opts.on('--minimum-score N', Float, "Mutation score percentage a file must reach to pass (default: #{MutationTester::Configuration::DEFAULT_MINIMUM_SCORE})") do |score|
|
|
68
|
+
options[:minimum_score] = score
|
|
69
|
+
end
|
|
70
|
+
|
|
63
71
|
opts.on('--since REV', 'Incremental batch mode: mutate only the files matched by --glob that changed since git revision REV (new files count as changed). Requires --glob.') do |rev|
|
|
64
72
|
options[:since] = rev
|
|
65
73
|
end
|
|
@@ -150,6 +158,13 @@ end
|
|
|
150
158
|
glob_mode = !options[:glob].nil?
|
|
151
159
|
staged_mode = options.fetch(:staged, false)
|
|
152
160
|
|
|
161
|
+
spec_map = begin
|
|
162
|
+
(options[:spec_map] || []).map { |rule| MutationTester::BatchRunner.parse_spec_map(rule) }
|
|
163
|
+
rescue MutationTester::Error => e
|
|
164
|
+
usage_error.call(e.message,
|
|
165
|
+
"Quote the rule so the shell keeps it intact, e.g. --spec-map '\\Aapp/(.+)\\.rb\\z=>test/\\1_test.rb'.")
|
|
166
|
+
end
|
|
167
|
+
|
|
153
168
|
apply_configuration = lambda do
|
|
154
169
|
MutationTester.configure do |config|
|
|
155
170
|
config.parallel_processes = options[:parallel] if options[:parallel]
|
|
@@ -158,6 +173,7 @@ apply_configuration = lambda do
|
|
|
158
173
|
config.show_progress = !options[:no_progress]
|
|
159
174
|
config.reporters = selected_reporters if selected_reporters
|
|
160
175
|
config.output_dir = options[:output_dir] if options[:output_dir]
|
|
176
|
+
config.minimum_score = options[:minimum_score] if options[:minimum_score]
|
|
161
177
|
config.test_selection = false if options[:no_test_selection]
|
|
162
178
|
config.timeout_factor = options[:timeout_factor] if options[:timeout_factor]
|
|
163
179
|
config.timeout_policy = options[:timeout_policy] if options[:timeout_policy]
|
|
@@ -224,18 +240,17 @@ if glob_mode
|
|
|
224
240
|
batch = MutationTester::BatchRunner.new(
|
|
225
241
|
glob: options[:glob],
|
|
226
242
|
spec_template: options[:spec_glob],
|
|
243
|
+
spec_map: spec_map,
|
|
227
244
|
config: MutationTester.configuration,
|
|
228
245
|
since: options[:since],
|
|
229
246
|
changed_files: changed_files
|
|
230
247
|
)
|
|
231
248
|
|
|
232
|
-
run_batch_machine.call(batch, ->(result) { result.
|
|
249
|
+
run_batch_machine.call(batch, ->(result) { result.gate_passed? }) if json_mode
|
|
233
250
|
|
|
234
251
|
result = guard_interrupt.call { batch.run }
|
|
235
252
|
|
|
236
|
-
exit
|
|
237
|
-
|
|
238
|
-
exit(result.success? ? 0 : 1)
|
|
253
|
+
exit(result.gate_passed? ? 0 : 1)
|
|
239
254
|
end
|
|
240
255
|
|
|
241
256
|
if !staged_mode && ARGV.empty?
|
|
@@ -255,6 +270,11 @@ if options[:spec_glob] && legacy_pair
|
|
|
255
270
|
'Use --spec-glob with a positional FILE list, --staged, or --glob.')
|
|
256
271
|
end
|
|
257
272
|
|
|
273
|
+
if options[:spec_map] && legacy_pair
|
|
274
|
+
usage_error.call('--spec-map does not apply to an explicit SOURCE_FILE TEST_FILE pair.',
|
|
275
|
+
'Use --spec-map with a positional FILE list, --staged, or --glob.')
|
|
276
|
+
end
|
|
277
|
+
|
|
258
278
|
if legacy_pair
|
|
259
279
|
source_file, test_file = ARGV
|
|
260
280
|
else
|
|
@@ -278,7 +298,7 @@ else
|
|
|
278
298
|
single_file_json = json_mode && !staged_mode && sources.length == 1
|
|
279
299
|
|
|
280
300
|
if single_file_json
|
|
281
|
-
outcome = MutationTester::BatchRunner.new(files: sources, spec_template: options[:spec_glob]).classify(sources.first)
|
|
301
|
+
outcome = MutationTester::BatchRunner.new(files: sources, spec_template: options[:spec_glob], spec_map: spec_map).classify(sources.first)
|
|
282
302
|
if outcome.is_a?(MutationTester::BatchRunner::SkippedEntry)
|
|
283
303
|
reason = MutationTester::BatchRunner::SKIP_REASONS.fetch(outcome.reason)
|
|
284
304
|
detail = outcome.expected_spec ? " (expected #{outcome.expected_spec})" : ''
|
|
@@ -293,14 +313,15 @@ else
|
|
|
293
313
|
batch = MutationTester::BatchRunner.new(
|
|
294
314
|
files: sources,
|
|
295
315
|
spec_template: options[:spec_glob],
|
|
316
|
+
spec_map: spec_map,
|
|
296
317
|
config: MutationTester.configuration
|
|
297
318
|
)
|
|
298
319
|
|
|
299
|
-
run_batch_machine.call(batch, ->(result) { result.
|
|
320
|
+
run_batch_machine.call(batch, ->(result) { result.gate_passed? }) if json_mode
|
|
300
321
|
|
|
301
322
|
result = guard_interrupt.call { batch.run }
|
|
302
323
|
|
|
303
|
-
exit(result.
|
|
324
|
+
exit(result.gate_passed? ? 0 : 1)
|
|
304
325
|
end
|
|
305
326
|
end
|
|
306
327
|
|
|
@@ -9,6 +9,8 @@ module MutationTester
|
|
|
9
9
|
|
|
10
10
|
DEFAULT_SPEC_TEMPLATE = 'spec/{name}_spec.rb'.freeze
|
|
11
11
|
|
|
12
|
+
SPEC_MAP_SEPARATOR = '=>'.freeze
|
|
13
|
+
|
|
12
14
|
TEST_FILE_SUFFIXES = ['_spec.rb', '.spec.rb', '_test.rb'].freeze
|
|
13
15
|
|
|
14
16
|
SKIP_REASONS = {
|
|
@@ -57,6 +59,18 @@ module MutationTester
|
|
|
57
59
|
!(processed.empty? && skipped.empty? && (unchanged || []).empty?)
|
|
58
60
|
end
|
|
59
61
|
|
|
62
|
+
def nothing_measured?
|
|
63
|
+
processed.empty? && !skipped.empty?
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def gate_passed?
|
|
67
|
+
return false unless matched_any?
|
|
68
|
+
return false if nothing_measured?
|
|
69
|
+
return true if processed.empty?
|
|
70
|
+
|
|
71
|
+
success?
|
|
72
|
+
end
|
|
73
|
+
|
|
60
74
|
def interrupted?
|
|
61
75
|
!!interrupted
|
|
62
76
|
end
|
|
@@ -100,6 +114,24 @@ module MutationTester
|
|
|
100
114
|
.map { |path| Pathname.new(File.expand_path(path, toplevel)).relative_path_from(base).to_s }
|
|
101
115
|
end
|
|
102
116
|
|
|
117
|
+
def self.parse_spec_map(rule)
|
|
118
|
+
pattern, replacement = rule.to_s.split(SPEC_MAP_SEPARATOR, 2)
|
|
119
|
+
if replacement.nil?
|
|
120
|
+
raise MutationTester::Error,
|
|
121
|
+
"--spec-map #{rule.to_s.inspect} has no #{SPEC_MAP_SEPARATOR} separator; " \
|
|
122
|
+
"write it as 'PATTERN#{SPEC_MAP_SEPARATOR}REPLACEMENT'"
|
|
123
|
+
end
|
|
124
|
+
if pattern.empty?
|
|
125
|
+
raise MutationTester::Error, "--spec-map #{rule.to_s.inspect} has an empty pattern"
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
begin
|
|
129
|
+
[Regexp.new(pattern), replacement]
|
|
130
|
+
rescue RegexpError => e
|
|
131
|
+
raise MutationTester::Error, "--spec-map pattern #{pattern.inspect} is not a valid regular expression: #{e.message}"
|
|
132
|
+
end
|
|
133
|
+
end
|
|
134
|
+
|
|
103
135
|
def self.test_file?(path)
|
|
104
136
|
basename = File.basename(path)
|
|
105
137
|
return true if TEST_FILE_SUFFIXES.any? { |suffix| basename.end_with?(suffix) }
|
|
@@ -118,12 +150,13 @@ module MutationTester
|
|
|
118
150
|
end
|
|
119
151
|
private_class_method :git_capture
|
|
120
152
|
|
|
121
|
-
def initialize(glob: nil, files: nil, spec_template: nil, config: MutationTester.configuration, since: nil, changed_files: nil)
|
|
153
|
+
def initialize(glob: nil, files: nil, spec_template: nil, spec_map: nil, config: MutationTester.configuration, since: nil, changed_files: nil)
|
|
122
154
|
raise ArgumentError, 'provide exactly one of glob: or files:' unless glob.nil? ^ files.nil?
|
|
123
155
|
|
|
124
156
|
@glob = glob
|
|
125
157
|
@files = files
|
|
126
158
|
@spec_template = spec_template.nil? || spec_template.empty? ? DEFAULT_SPEC_TEMPLATE : spec_template
|
|
159
|
+
@spec_map = spec_map || []
|
|
127
160
|
@config = config
|
|
128
161
|
@since = since
|
|
129
162
|
@changed_files = changed_files
|
|
@@ -203,6 +236,10 @@ module MutationTester
|
|
|
203
236
|
end
|
|
204
237
|
|
|
205
238
|
def spec_path_for(source_file)
|
|
239
|
+
normalized = source_file.sub(%r{\A\./}, '')
|
|
240
|
+
pattern, replacement = @spec_map.find { |rule_pattern, _| rule_pattern.match?(normalized) }
|
|
241
|
+
return normalized.sub(pattern, replacement) if pattern
|
|
242
|
+
|
|
206
243
|
@spec_template.gsub(NAME_PLACEHOLDER, source_name(source_file))
|
|
207
244
|
end
|
|
208
245
|
|
|
@@ -254,8 +291,9 @@ module MutationTester
|
|
|
254
291
|
puts Rainbow('=' * 80).bright
|
|
255
292
|
failed = result.processed.reject(&:passed?)
|
|
256
293
|
degraded = failed.select(&:degraded?)
|
|
257
|
-
if
|
|
258
|
-
|
|
294
|
+
if result.nothing_measured?
|
|
295
|
+
scope = @files ? 'listed' : 'matched'
|
|
296
|
+
puts Rainbow("❌ No files were mutation-tested: every #{scope} file was skipped").red
|
|
259
297
|
elsif result.success?
|
|
260
298
|
puts Rainbow('✓ All processed files met the mutation score threshold').green
|
|
261
299
|
elsif degraded.size == failed.size
|
|
@@ -5,6 +5,7 @@ module MutationTester
|
|
|
5
5
|
RUNNER_MODES = %i[auto fork spawn in_memory].freeze
|
|
6
6
|
TIMEOUT_POLICIES = %i[killed separate].freeze
|
|
7
7
|
AUTO_PARALLEL_CAP = 8
|
|
8
|
+
DEFAULT_MINIMUM_SCORE = 80.0
|
|
8
9
|
DEFAULT_TIMEOUT = 30
|
|
9
10
|
DEFAULT_TIMEOUT_FACTOR = 5
|
|
10
11
|
CALIBRATED_TIMEOUT_FLOOR = 5
|
|
@@ -56,7 +57,7 @@ module MutationTester
|
|
|
56
57
|
}
|
|
57
58
|
@reporters = %i[console html json]
|
|
58
59
|
@output_dir = 'tmp/mutation_reports'
|
|
59
|
-
@minimum_score =
|
|
60
|
+
@minimum_score = DEFAULT_MINIMUM_SCORE
|
|
60
61
|
@fail_on_threshold = true
|
|
61
62
|
@verbose = false
|
|
62
63
|
@show_file_path = true
|
data/readme.md
CHANGED
|
@@ -204,6 +204,8 @@ mutation_test [OPTIONS] --glob 'lib/**/*.rb'
|
|
|
204
204
|
| `--staged` | Mutation-test the files staged in git (`git diff --cached --name-only`; files staged as deleted are ignored), mapping each to its spec like a positional `FILE` list. Cannot be combined with positional arguments or `--glob`. See [File lists and --staged](#file-lists-and---staged-test-what-you-changed). |
|
|
205
205
|
| `--glob PATTERN` | Batch mode: mutation-test every source file matching `PATTERN`, mapping each to its spec by convention (see [Batch mode](#batch-mode-run-many-files-in-one-command)). |
|
|
206
206
|
| `--spec-glob TEMPLATE` | Spec-mapping template with a `{name}` placeholder (default: `spec/{name}_spec.rb`). Requires a positional `FILE` list, `--staged`, or `--glob`. |
|
|
207
|
+
| `--spec-map RULE` | Spec-mapping rule `'PATTERN=>REPLACEMENT'`: a regular expression applied to the whole source path to build the whole spec path, for layouts a `{name}` template cannot express (Rails `app/` -> `test/`, engines, Packwerk packs). Repeatable, first matching rule wins, a source matching no rule falls back to `--spec-glob`. Requires a positional `FILE` list, `--staged`, or `--glob`. See [Mapping sources to specs](#mapping-sources-to-specs). |
|
|
208
|
+
| `--minimum-score N` | Mutation score percentage a file must reach to pass (default: 80). Drives the `PASS`/`FAIL` verdict and the exit code. |
|
|
207
209
|
| `--since REV` | Incremental batch mode: mutate only the files matched by `--glob` that changed since git revision `REV` (new files count as changed). Requires `--glob`. See [Incremental mode](#incremental-mode-mutate-only-what-changed). |
|
|
208
210
|
| `--fail-fast` | Stop the run at the first surviving mutant and finish with a failing status. Works in single-file mode and with `--glob`. |
|
|
209
211
|
| `--timeout-factor N` | Per-mutant timeout budget as `N` times the measured baseline test run, never below 5 s (default: 5, must be > 0). Ignored when `config.timeout` is set explicitly, which keeps a fixed budget. See [Configuration](#configuration). |
|
|
@@ -314,8 +316,8 @@ reported as `SKIPPED` with an explicit reason and never count as a success:
|
|
|
314
316
|
- **not a Ruby source file** - e.g. a staged `.md` or config file.
|
|
315
317
|
- **a test file, not a mutable source** - a test file passed directly
|
|
316
318
|
(`*_spec.rb`, `*.spec.rb`, `*_test.rb`, `test_*.rb`, or minitest content).
|
|
317
|
-
- **no matching spec file** - the convention (or `--spec-glob`
|
|
318
|
-
spec that does not exist; the expected path is printed.
|
|
319
|
+
- **no matching spec file** - the convention (or `--spec-glob` / `--spec-map`)
|
|
320
|
+
points at a spec that does not exist; the expected path is printed.
|
|
319
321
|
|
|
320
322
|
Exit codes: `0` when at least one file was processed and every processed file
|
|
321
323
|
met the threshold; `1` when any processed file was below threshold or when
|
|
@@ -336,16 +338,8 @@ run, so you no longer need to script a loop around `mutation_test` or depend on
|
|
|
336
338
|
the Rails-only `rake mutation:test_models` task.
|
|
337
339
|
|
|
338
340
|
Each matched source file is mapped to its spec by convention: `lib/X.rb` becomes
|
|
339
|
-
`spec/X_spec.rb`.
|
|
340
|
-
|
|
341
|
-
stripped, subdirectories preserved (`lib/foo/bar.rb` -> `spec/foo/bar_spec.rb`).
|
|
342
|
-
|
|
343
|
-
Override the convention with `--spec-glob TEMPLATE`, a template containing the
|
|
344
|
-
`{name}` placeholder. For a Minitest project laid out under `test/`:
|
|
345
|
-
|
|
346
|
-
```bash
|
|
347
|
-
bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
348
|
-
```
|
|
341
|
+
`spec/X_spec.rb`. See [Mapping sources to specs](#mapping-sources-to-specs) for
|
|
342
|
+
the two ways to override that convention.
|
|
349
343
|
|
|
350
344
|
Behaviour:
|
|
351
345
|
|
|
@@ -359,18 +353,27 @@ Behaviour:
|
|
|
359
353
|
file with its score and `PASS`/`FAIL` against the threshold, followed by a
|
|
360
354
|
clearly separated `SKIPPED` list.
|
|
361
355
|
- **A source file with no matching spec is `SKIPPED`**, reported explicitly and
|
|
362
|
-
never counted as a success. A skipped file
|
|
356
|
+
never counted as a success. A single skipped file next to processed ones does
|
|
357
|
+
not by itself fail the run, but a run that skipped *every* matched file
|
|
358
|
+
measured nothing and fails (see the exit codes below).
|
|
363
359
|
|
|
364
360
|
Exit codes:
|
|
365
361
|
|
|
366
|
-
- `0` -
|
|
367
|
-
`--since` run where nothing changed
|
|
362
|
+
- `0` - at least one file was mutation-tested and every processed file met the
|
|
363
|
+
mutation score threshold. A `--since` run where nothing changed also exits `0`
|
|
364
|
+
(see below).
|
|
368
365
|
- `1` - at least one processed file was below threshold, the glob matched no
|
|
369
|
-
source files at all,
|
|
370
|
-
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
366
|
+
source files at all, every matched file was skipped so nothing was actually
|
|
367
|
+
mutation-tested, or `--fail-fast` stopped the run at a surviving mutant.
|
|
368
|
+
- `2` - a usage error: `--spec-glob` or `--spec-map` with an explicit
|
|
369
|
+
`SOURCE_FILE TEST_FILE` pair, a malformed `--spec-map` rule, `--since` given
|
|
370
|
+
without `--glob`, `--staged` combined with positional arguments or `--glob`,
|
|
371
|
+
or `--since`/`--staged` used outside a git repository (for `--since` also an
|
|
372
|
+
unknown revision).
|
|
373
|
+
|
|
374
|
+
The "every matched file was skipped" case is deliberate: a typo in
|
|
375
|
+
`--spec-glob`/`--spec-map`, or a refactor that moves the test directory, would
|
|
376
|
+
otherwise leave a green CI step that measured nothing.
|
|
374
377
|
|
|
375
378
|
```bash
|
|
376
379
|
# Minitest project, JSON report per file, custom output directory
|
|
@@ -378,6 +381,66 @@ bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
|
378
381
|
--reporters json --output-dir build/mutation
|
|
379
382
|
```
|
|
380
383
|
|
|
384
|
+
### Mapping sources to specs
|
|
385
|
+
|
|
386
|
+
Every mode that takes more than an explicit `SOURCE_FILE TEST_FILE` pair (a
|
|
387
|
+
positional `FILE` list, `--staged`, `--glob`) derives the test path from the
|
|
388
|
+
source path. Two mechanisms do that, checked in this order:
|
|
389
|
+
|
|
390
|
+
1. `--spec-map 'PATTERN=>REPLACEMENT'` - regular-expression rules.
|
|
391
|
+
2. `--spec-glob TEMPLATE` - a `{name}` template (default `spec/{name}_spec.rb`).
|
|
392
|
+
|
|
393
|
+
**`--spec-glob TEMPLATE`** substitutes `{name}`, which is the source path with a
|
|
394
|
+
leading `lib/` segment removed and the `.rb` extension stripped, subdirectories
|
|
395
|
+
preserved (`lib/foo/bar.rb` -> `spec/foo/bar_spec.rb`). Because `{name}` is one
|
|
396
|
+
value, a template can only add a prefix and a suffix around the source path. For
|
|
397
|
+
a Minitest project laid out under `test/` that is enough:
|
|
398
|
+
|
|
399
|
+
```bash
|
|
400
|
+
bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
**`--spec-map 'PATTERN=>REPLACEMENT'`** covers the layouts a template cannot
|
|
404
|
+
express: those that substitute *inside* the path, after a variable-length
|
|
405
|
+
prefix. `PATTERN` is a Ruby regular expression matched against the whole source
|
|
406
|
+
path (a leading `./` removed); the first `=>` separates it from `REPLACEMENT`,
|
|
407
|
+
which may use `\1`, `\2`, ... backreferences and produces the whole spec path.
|
|
408
|
+
Only the first match in the path is replaced.
|
|
409
|
+
|
|
410
|
+
- The flag is repeatable and the first matching rule wins.
|
|
411
|
+
- A source that matches no rule falls back to `--spec-glob` (or the default
|
|
412
|
+
convention), so one command can cover `app/` and `lib/` at once.
|
|
413
|
+
- Quote the rule in single quotes so the shell leaves the backslashes alone.
|
|
414
|
+
|
|
415
|
+
Rails and Rails-shaped layouts, where the rule is "replace the `app/` segment
|
|
416
|
+
with `test/`, keep whatever prefix comes before it":
|
|
417
|
+
|
|
418
|
+
```bash
|
|
419
|
+
# Plain Rails, Minitest: app/models/current.rb -> test/models/current_test.rb
|
|
420
|
+
bundle exec mutation_test --glob 'app/**/*.rb' \
|
|
421
|
+
--spec-map '\Aapp/(.+)\.rb\z=>test/\1_test.rb'
|
|
422
|
+
|
|
423
|
+
# Plain Rails, RSpec: app/models/user.rb -> spec/models/user_spec.rb
|
|
424
|
+
bundle exec mutation_test --glob 'app/**/*.rb' \
|
|
425
|
+
--spec-map '\Aapp/(.+)\.rb\z=>spec/\1_spec.rb'
|
|
426
|
+
|
|
427
|
+
# Packwerk / packs-rails and engines, with the app root as an optional prefix:
|
|
428
|
+
# app/models/current.rb -> test/models/current_test.rb
|
|
429
|
+
# packs/identity/app/models/party.rb -> packs/identity/test/models/party_test.rb
|
|
430
|
+
# engines/billing/app/jobs/send_job.rb -> engines/billing/test/jobs/send_job_test.rb
|
|
431
|
+
bundle exec mutation_test --glob '{app,packs/*/app,engines/*/app}/**/*.rb' \
|
|
432
|
+
--spec-map '\A((?:(?:packs|engines)/[^/]+/)?)app/(.+)\.rb\z=>\1test/\2_test.rb'
|
|
433
|
+
|
|
434
|
+
# app/ through the rule, lib/ through the template, in one run
|
|
435
|
+
bundle exec mutation_test --glob '{app,lib}/**/*.rb' \
|
|
436
|
+
--spec-map '\Aapp/(.+)\.rb\z=>test/\1_test.rb' \
|
|
437
|
+
--spec-glob 'test/{name}_test.rb'
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
When a rule produces a path that does not exist, the file is reported as
|
|
441
|
+
`SKIPPED (no matching spec file)` with the expected path printed, and a run in
|
|
442
|
+
which *every* file was skipped that way fails with exit code `1`.
|
|
443
|
+
|
|
381
444
|
### Incremental mode: mutate only what changed
|
|
382
445
|
|
|
383
446
|
`--since REV` narrows a `--glob` batch to the files that changed since a git
|
|
@@ -463,7 +526,7 @@ MutationTester.configure do |config|
|
|
|
463
526
|
# Set it explicitly to write elsewhere (this example uses "mutation_reports"):
|
|
464
527
|
config.output_dir = "mutation_reports"
|
|
465
528
|
|
|
466
|
-
# Quality thresholds
|
|
529
|
+
# Quality thresholds. The CLI flag --minimum-score overrides this per run.
|
|
467
530
|
config.minimum_score = 80.0
|
|
468
531
|
config.fail_on_threshold = true
|
|
469
532
|
|
|
@@ -581,7 +644,8 @@ bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb \
|
|
|
581
644
|
-p 4 --worker-env TEST_ENV_NUMBER
|
|
582
645
|
|
|
583
646
|
# Batch over a whole directory the same way
|
|
584
|
-
bundle exec mutation_test --glob 'app/models/**/*.rb'
|
|
647
|
+
bundle exec mutation_test --glob 'app/models/**/*.rb' \
|
|
648
|
+
--spec-map '\Aapp/(.+)\.rb\z=>spec/\1_spec.rb' \
|
|
585
649
|
-p 4 --worker-env TEST_ENV_NUMBER
|
|
586
650
|
```
|
|
587
651
|
|
|
@@ -853,6 +917,14 @@ can drop the `jq` comparison and let the exit code be the gate:
|
|
|
853
917
|
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb || exit 1
|
|
854
918
|
```
|
|
855
919
|
|
|
920
|
+
`--minimum-score N` sets that threshold for a single run, which is how you start
|
|
921
|
+
below 80 in an existing codebase and ratchet the number up over time:
|
|
922
|
+
|
|
923
|
+
```sh
|
|
924
|
+
bundle exec mutation_test --glob 'app/**/*.rb' \
|
|
925
|
+
--spec-map '\Aapp/(.+)\.rb\z=>spec/\1_spec.rb' --minimum-score 60 || exit 1
|
|
926
|
+
```
|
|
927
|
+
|
|
856
928
|
lefthook or overcommit users: call the shipped hook from your `pre-push` step
|
|
857
929
|
instead of writing to `.git/hooks/`.
|
|
858
930
|
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutation_tester
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.4.
|
|
4
|
+
version: 1.4.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kamil Dzierbicki
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-08-
|
|
11
|
+
date: 2026-08-05 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: parallel
|