mutation_tester 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +5 -0
- data/Gemfile.lock +1 -1
- data/docs/json-schema.md +4 -4
- data/exe/mutation_test +12 -1
- data/lib/mutation_tester/batch_runner.rb +14 -2
- data/lib/mutation_tester/configuration.rb +50 -3
- data/lib/mutation_tester/core.rb +32 -2
- data/lib/mutation_tester/mutation_runner.rb +4 -4
- data/lib/mutation_tester/reporters/base_reporter.rb +9 -5
- data/lib/mutation_tester/reporters/batch_json_reporter.rb +1 -1
- data/lib/mutation_tester/version.rb +1 -1
- data/lib/tasks/mutation_tester.rake +1 -1
- data/readme.md +39 -3
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 03a2b2364ebede84e6b56cfe4fba7d0450475262d59aa754abe0ce34ee1a83ba
|
|
4
|
+
data.tar.gz: 7439181f228ffbf2d1cf2e603ac21a91a46c764cd66f0ebfcc40dab30a2f2188
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6be68e2928ca7ecc1a382e01c66cb5ab7d0afa35e5cf5b6008a62d0599fe399af17a8960a3f4e62025d40e640850c4fde423cff5aa2de69d34db2f6139661342
|
|
7
|
+
data.tar.gz: fb12cf8fd83bc37529e7e0fcdc8bb72bc902df0c846b05ad8fb856d8db8b382cf18fb2058b42f5efa2b161f573517b368d7aee92563c4115c7b3669a3d2e5ba9
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.3.0] - 2026-07-15
|
|
4
|
+
|
|
5
|
+
- Calibrated the per-mutant timeout against the measured baseline run: unless `config.timeout` is set explicitly, each mutant now gets `max(5s, timeout_factor * baseline duration)` (factor configurable via `config.timeout_factor` / `--timeout-factor N`, default 5) instead of a fixed 30 s, so a loaded machine no longer inflates the mutation score by killing healthy-but-slow runs as timeouts. An explicit `config.timeout` (including `nil` for no deadline) keeps today's fixed-budget behavior and disables calibration. Added the opt-in `config.timeout_policy = :separate` / `--timeout-policy separate`, which scores `killed / (killed + survived)` with timeouts excluded from the score and reported only as their own category; the default `:killed` policy and its output are unchanged.
|
|
6
|
+
- Runs that finish without a single scored mutant (every mutant errored or was stillborn) are now reported as an infrastructure/runner failure with the error/stillborn counts instead of a misleading "score 0.0% is below threshold" verdict. `Core` exposes `infrastructure_failure?` to distinguish such degraded runs (including the shadow-workspace abort) from a genuine threshold failure, and `mutation_test` maps them to the new exit code `3` in single-file mode (`0` pass, `1` threshold failure, `2` usage error, `130` interrupt). The batch summary names degraded files instead of blaming the threshold.
|
|
7
|
+
|
|
3
8
|
## [1.2.0] - 2026-07-14
|
|
4
9
|
|
|
5
10
|
- Fixed the in-memory runner reporting false survivors for mutations that only take effect at class-load time (constants consumed by macros, `validates`/`has_many`/`before_save`/`scope`/`attribute`, and anything inside an `included do` block). The mutator now classifies each mutation by AST context, and the in-memory run routes load-time mutations to the file-based path while keeping method-body mutations in memory, so the default (`auto`) score matches a full `fork` run on Rails concerns and models. Measured on a real Rails concern the default score went from a misleading 1.63% to the correct 35.77%.
|
data/Gemfile.lock
CHANGED
data/docs/json-schema.md
CHANGED
|
@@ -52,13 +52,13 @@ history of changes.
|
|
|
52
52
|
| `metadata.source_file` | string | Absolute path of the mutated source file. |
|
|
53
53
|
| `metadata.spec_file` | string | Absolute path of the test file that was run. |
|
|
54
54
|
| `summary.total` | integer | Total number of mutations produced (all statuses). |
|
|
55
|
-
| `summary.killed` | integer | Effective kills (a `timeout` counts as a kill). Kept for backward compatibility.
|
|
55
|
+
| `summary.killed` | integer | Effective kills (a `timeout` counts as a kill; under the opt-in `timeout_policy: :separate` only real kills are counted). Kept for backward compatibility. |
|
|
56
56
|
| `summary.survived` | integer | Number of surviving mutants. |
|
|
57
|
-
| `summary.mutation_score` | number | Percentage `(killed + timeout) / (killed + timeout + survived) * 100`, rounded to 2 decimals. `stillborn` and `error` are excluded from the denominator.
|
|
57
|
+
| `summary.mutation_score` | number | Percentage `(killed + timeout) / (killed + timeout + survived) * 100`, rounded to 2 decimals; under the opt-in `timeout_policy: :separate` it is `killed / (killed + survived) * 100` with timeouts excluded. `stillborn` and `error` are excluded from the denominator. |
|
|
58
58
|
| `summary.quality_rating` | string | Human label derived from the score (Excellent/Good/Fair/Poor/Critical). |
|
|
59
59
|
| `summary.categories.killed` | integer | Mutants whose covering tests failed. |
|
|
60
60
|
| `summary.categories.survived` | integer | Mutants whose covering tests still passed (a test gap candidate). |
|
|
61
|
-
| `summary.categories.timeout` | integer | Mutants that exceeded the per-mutant deadline (counted as kills in the score).
|
|
61
|
+
| `summary.categories.timeout` | integer | Mutants that exceeded the per-mutant deadline (counted as kills in the score by default; excluded from the score under `timeout_policy: :separate`). |
|
|
62
62
|
| `summary.categories.stillborn` | integer | Mutants whose code no longer parses. Never run; excluded from the score. |
|
|
63
63
|
| `summary.categories.error` | integer | Mutants that hit a runner-side error. Excluded from the score. |
|
|
64
64
|
| `mutations[].id` | integer | Stable id of the mutation within this run. |
|
|
@@ -144,7 +144,7 @@ the exit code keeps its meaning (`0` passed, `1` failed).
|
|
|
144
144
|
| `summary.files` | integer | Files that entered the run: processed plus skipped. Files a `--since` filter left out as unchanged are not counted; they are reported on stderr. |
|
|
145
145
|
| `summary.processed` | integer | Files that were actually mutation-tested. |
|
|
146
146
|
| `summary.skipped` | array | One entry per skipped file: `file` (the path as given) and `reason`, one of `file not found`, `not a Ruby source file`, `a test file, not a mutable source`, `no matching spec file`. |
|
|
147
|
-
| `summary.score` | number | Aggregate mutation score over every mutant of every processed file, with the per-file formula: `(killed + timeout) / (killed + timeout + survived) * 100
|
|
147
|
+
| `summary.score` | number | Aggregate mutation score over every mutant of every processed file, with the per-file formula: `(killed + timeout) / (killed + timeout + survived) * 100` (or `killed / (killed + survived) * 100` under `timeout_policy: :separate`), rounded to 2 decimals; `stillborn` and `error` are excluded from the denominator. |
|
|
148
148
|
| `summary.passed` | boolean | `true` exactly when the process exits `0`: the run matched/processed files and every processed file met the threshold. |
|
|
149
149
|
| `summary.interrupted` | boolean | `true` when `--fail-fast` stopped the batch at a surviving mutant, so remaining files were not run and the envelope is partial. |
|
|
150
150
|
| `survivors[].file` | string | Absolute path of the mutated source file, the same convention as `files[].metadata.source_file`, so a survivor joins its full per-file report by an exact string match on this value. |
|
data/exe/mutation_test
CHANGED
|
@@ -68,6 +68,14 @@ OptionParser.new do |opts|
|
|
|
68
68
|
options[:fail_fast] = true
|
|
69
69
|
end
|
|
70
70
|
|
|
71
|
+
opts.on('--timeout-factor N', Float, "Per-mutant timeout budget as N times the measured baseline test run, never below #{MutationTester::Configuration::CALIBRATED_TIMEOUT_FLOOR}s (default: #{MutationTester::Configuration::DEFAULT_TIMEOUT_FACTOR}; ignored when config.timeout is set explicitly, which keeps a fixed budget)") do |n|
|
|
72
|
+
options[:timeout_factor] = n
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
opts.on('--timeout-policy MODE', 'Scoring policy for timed-out mutants: killed (default) counts a timeout as a kill; separate keeps timeouts out of the score entirely (killed / (killed + survived)) and reports them only as their own category') do |mode|
|
|
76
|
+
options[:timeout_policy] = mode
|
|
77
|
+
end
|
|
78
|
+
|
|
71
79
|
opts.on('--strict-equality', 'Enable strict equality probes (== to eql? and equal?); default off, expect noise on code that does not distinguish numeric types or object identity') do
|
|
72
80
|
options[:strict_equality] = true
|
|
73
81
|
end
|
|
@@ -151,6 +159,8 @@ apply_configuration = lambda do
|
|
|
151
159
|
config.reporters = selected_reporters if selected_reporters
|
|
152
160
|
config.output_dir = options[:output_dir] if options[:output_dir]
|
|
153
161
|
config.test_selection = false if options[:no_test_selection]
|
|
162
|
+
config.timeout_factor = options[:timeout_factor] if options[:timeout_factor]
|
|
163
|
+
config.timeout_policy = options[:timeout_policy] if options[:timeout_policy]
|
|
154
164
|
config.worker_env_var = options[:worker_env] if options.key?(:worker_env)
|
|
155
165
|
config.fail_fast = options.fetch(:fail_fast, false)
|
|
156
166
|
config.mutation_types[:strict_equality] = true if options[:strict_equality]
|
|
@@ -342,4 +352,5 @@ else
|
|
|
342
352
|
success = guard_interrupt.call { core.run }
|
|
343
353
|
end
|
|
344
354
|
|
|
345
|
-
exit
|
|
355
|
+
exit 0 if success
|
|
356
|
+
exit(core.infrastructure_failure? ? 3 : 1)
|
|
@@ -20,10 +20,14 @@ module MutationTester
|
|
|
20
20
|
|
|
21
21
|
SKIP_ORDER = %i[missing not_ruby test_file no_spec].freeze
|
|
22
22
|
|
|
23
|
-
ProcessedEntry = Struct.new(:source_file, :spec_file, :score, :passed, :output_dir, :results, :interrupted, keyword_init: true) do
|
|
23
|
+
ProcessedEntry = Struct.new(:source_file, :spec_file, :score, :passed, :output_dir, :results, :interrupted, :degraded, keyword_init: true) do
|
|
24
24
|
def passed?
|
|
25
25
|
passed
|
|
26
26
|
end
|
|
27
|
+
|
|
28
|
+
def degraded?
|
|
29
|
+
!!degraded
|
|
30
|
+
end
|
|
27
31
|
end
|
|
28
32
|
|
|
29
33
|
SkippedEntry = Struct.new(:source_file, :expected_spec, :reason, keyword_init: true)
|
|
@@ -192,7 +196,8 @@ module MutationTester
|
|
|
192
196
|
passed: passed,
|
|
193
197
|
output_dir: file_config.output_dir,
|
|
194
198
|
results: core.results,
|
|
195
|
-
interrupted: core.interrupted
|
|
199
|
+
interrupted: core.interrupted?,
|
|
200
|
+
degraded: core.infrastructure_failure?
|
|
196
201
|
)
|
|
197
202
|
[entry, core.interrupted?]
|
|
198
203
|
end
|
|
@@ -247,12 +252,19 @@ module MutationTester
|
|
|
247
252
|
end
|
|
248
253
|
|
|
249
254
|
puts Rainbow('=' * 80).bright
|
|
255
|
+
failed = result.processed.reject(&:passed?)
|
|
256
|
+
degraded = failed.select(&:degraded?)
|
|
250
257
|
if @files && result.processed.empty?
|
|
251
258
|
puts Rainbow('❌ No files were mutation-tested: every listed file was skipped').red
|
|
252
259
|
elsif result.success?
|
|
253
260
|
puts Rainbow('✓ All processed files met the mutation score threshold').green
|
|
261
|
+
elsif degraded.size == failed.size
|
|
262
|
+
puts Rainbow('❌ The failing files produced no scored mutants; this indicates an infrastructure or runner problem, not a test-quality gap').red
|
|
254
263
|
else
|
|
255
264
|
puts Rainbow('❌ One or more files did not meet the mutation score threshold').red
|
|
265
|
+
unless degraded.empty?
|
|
266
|
+
puts Rainbow("⚠️ #{degraded.size} of the failing files produced no scored mutants (infrastructure or runner problem); see the per-file output above").yellow
|
|
267
|
+
end
|
|
256
268
|
end
|
|
257
269
|
|
|
258
270
|
print_survivors(result.survivors)
|
|
@@ -3,7 +3,11 @@ require 'etc'
|
|
|
3
3
|
module MutationTester
|
|
4
4
|
class Configuration
|
|
5
5
|
RUNNER_MODES = %i[auto fork spawn in_memory].freeze
|
|
6
|
+
TIMEOUT_POLICIES = %i[killed separate].freeze
|
|
6
7
|
AUTO_PARALLEL_CAP = 8
|
|
8
|
+
DEFAULT_TIMEOUT = 30
|
|
9
|
+
DEFAULT_TIMEOUT_FACTOR = 5
|
|
10
|
+
CALIBRATED_TIMEOUT_FLOOR = 5
|
|
7
11
|
|
|
8
12
|
def self.auto_parallel_processes
|
|
9
13
|
[[Etc.nprocessors, AUTO_PARALLEL_CAP].min, 1].max
|
|
@@ -14,9 +18,9 @@ module MutationTester
|
|
|
14
18
|
number <= 0 ? '' : (number + 1).to_s
|
|
15
19
|
end
|
|
16
20
|
|
|
17
|
-
attr_reader :parallel_processes, :runner, :worker_env_var
|
|
21
|
+
attr_reader :parallel_processes, :runner, :worker_env_var, :timeout, :timeout_factor, :timeout_policy
|
|
18
22
|
|
|
19
|
-
attr_accessor :
|
|
23
|
+
attr_accessor :baseline_duration,
|
|
20
24
|
:baseline_timeout,
|
|
21
25
|
:mutation_types,
|
|
22
26
|
:reporters,
|
|
@@ -33,7 +37,9 @@ module MutationTester
|
|
|
33
37
|
self.parallel_processes = ENV['MUTATION_TESTER_PARALLEL_PROCESSES'] || self.class.auto_parallel_processes
|
|
34
38
|
self.runner = ENV['MUTATION_TESTER_RUNNER'] || :auto
|
|
35
39
|
self.worker_env_var = ENV['MUTATION_TESTER_WORKER_ENV']
|
|
36
|
-
@timeout =
|
|
40
|
+
@timeout = DEFAULT_TIMEOUT
|
|
41
|
+
@timeout_factor = DEFAULT_TIMEOUT_FACTOR
|
|
42
|
+
@timeout_policy = :killed
|
|
37
43
|
@baseline_timeout = 300
|
|
38
44
|
@mutation_types = {
|
|
39
45
|
arithmetic: true,
|
|
@@ -73,6 +79,39 @@ module MutationTester
|
|
|
73
79
|
@reporters = source.reporters.dup
|
|
74
80
|
end
|
|
75
81
|
|
|
82
|
+
def timeout=(value)
|
|
83
|
+
@timeout_explicit = true
|
|
84
|
+
@timeout = value
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def timeout_explicitly_set?
|
|
88
|
+
!!@timeout_explicit
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def timeout_factor=(value)
|
|
92
|
+
factor = coerce_numeric(value)
|
|
93
|
+
unless factor.positive?
|
|
94
|
+
warn "[MutationTester] timeout_factor must be a number greater than 0; got #{value.inspect}, falling back to #{DEFAULT_TIMEOUT_FACTOR}."
|
|
95
|
+
factor = DEFAULT_TIMEOUT_FACTOR
|
|
96
|
+
end
|
|
97
|
+
@timeout_factor = factor
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def timeout_policy=(value)
|
|
101
|
+
policy = value.to_s.strip.downcase.to_sym
|
|
102
|
+
unless TIMEOUT_POLICIES.include?(policy)
|
|
103
|
+
warn "[MutationTester] timeout_policy must be one of #{TIMEOUT_POLICIES.join(", ")}; got #{value.inspect}, falling back to killed."
|
|
104
|
+
policy = :killed
|
|
105
|
+
end
|
|
106
|
+
@timeout_policy = policy
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def effective_timeout
|
|
110
|
+
return @timeout if timeout_explicitly_set? || @baseline_duration.nil?
|
|
111
|
+
|
|
112
|
+
[CALIBRATED_TIMEOUT_FLOOR, @timeout_factor * @baseline_duration].max
|
|
113
|
+
end
|
|
114
|
+
|
|
76
115
|
def parallel_processes=(value)
|
|
77
116
|
count = value.to_i
|
|
78
117
|
if count < 1
|
|
@@ -101,5 +140,13 @@ module MutationTester
|
|
|
101
140
|
|
|
102
141
|
{ @worker_env_var => self.class.worker_env_value(index) }
|
|
103
142
|
end
|
|
143
|
+
|
|
144
|
+
private
|
|
145
|
+
|
|
146
|
+
def coerce_numeric(value)
|
|
147
|
+
Float(value)
|
|
148
|
+
rescue ArgumentError, TypeError
|
|
149
|
+
0
|
|
150
|
+
end
|
|
104
151
|
end
|
|
105
152
|
end
|
data/lib/mutation_tester/core.rb
CHANGED
|
@@ -11,6 +11,7 @@ module MutationTester
|
|
|
11
11
|
@mutations = []
|
|
12
12
|
@results = []
|
|
13
13
|
@parse_failed = false
|
|
14
|
+
@shadow_aborted = false
|
|
14
15
|
MutationRunner.recover_in_place_backup(@source_file)
|
|
15
16
|
@original_content = File.read(@source_file)
|
|
16
17
|
end
|
|
@@ -27,11 +28,15 @@ module MutationTester
|
|
|
27
28
|
return true
|
|
28
29
|
end
|
|
29
30
|
|
|
30
|
-
|
|
31
|
+
unless shadow_environment_reliable?
|
|
32
|
+
@shadow_aborted = true
|
|
33
|
+
return false
|
|
34
|
+
end
|
|
31
35
|
|
|
32
36
|
run_mutations
|
|
33
37
|
generate_reports
|
|
34
38
|
return report_interruption if interrupted?
|
|
39
|
+
return report_infrastructure_failure if infrastructure_failure?
|
|
35
40
|
|
|
36
41
|
check_threshold
|
|
37
42
|
ensure
|
|
@@ -46,7 +51,11 @@ module MutationTester
|
|
|
46
51
|
end
|
|
47
52
|
|
|
48
53
|
def mutation_score
|
|
49
|
-
Reporters::BaseReporter.score(@results)
|
|
54
|
+
Reporters::BaseReporter.score(@results, policy: @config.timeout_policy)
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def infrastructure_failure?
|
|
58
|
+
@shadow_aborted || (!@results.empty? && scored_count.zero?)
|
|
50
59
|
end
|
|
51
60
|
|
|
52
61
|
def threshold_met?
|
|
@@ -69,7 +78,9 @@ module MutationTester
|
|
|
69
78
|
def run_original_tests
|
|
70
79
|
puts Rainbow("\n🧪 Running original tests...").yellow
|
|
71
80
|
command = test_command
|
|
81
|
+
baseline_started = monotonic_time
|
|
72
82
|
result = command.run(timeout: @config.baseline_timeout, capture: true)
|
|
83
|
+
baseline_elapsed = monotonic_time - baseline_started
|
|
73
84
|
|
|
74
85
|
if result.timed_out?
|
|
75
86
|
puts Rainbow("❌ Original tests exceeded the baseline deadline of #{@config.baseline_timeout}s and were terminated.").red
|
|
@@ -85,9 +96,14 @@ module MutationTester
|
|
|
85
96
|
return false
|
|
86
97
|
end
|
|
87
98
|
puts Rainbow('✓ Original tests passed').green
|
|
99
|
+
@config.baseline_duration = baseline_elapsed
|
|
88
100
|
true
|
|
89
101
|
end
|
|
90
102
|
|
|
103
|
+
def monotonic_time
|
|
104
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
105
|
+
end
|
|
106
|
+
|
|
91
107
|
def replay_baseline_output(output)
|
|
92
108
|
return if output.nil? || output.strip.empty?
|
|
93
109
|
|
|
@@ -178,6 +194,20 @@ module MutationTester
|
|
|
178
194
|
false
|
|
179
195
|
end
|
|
180
196
|
|
|
197
|
+
def scored_count
|
|
198
|
+
@results.count { |r| %i[killed timeout survived].include?(Reporters::BaseReporter.status_for(r)) }
|
|
199
|
+
end
|
|
200
|
+
|
|
201
|
+
def status_count(status)
|
|
202
|
+
@results.count { |r| Reporters::BaseReporter.status_for(r) == status }
|
|
203
|
+
end
|
|
204
|
+
|
|
205
|
+
def report_infrastructure_failure
|
|
206
|
+
puts Rainbow("\n❌ Run failed: none of the #{@results.size} mutants could be scored (#{status_count(:error)} error, #{status_count(:stillborn)} stillborn).").red
|
|
207
|
+
puts Rainbow(' This indicates an infrastructure or runner problem (test environment, workspace, or test command), not a test-quality gap.').red
|
|
208
|
+
false
|
|
209
|
+
end
|
|
210
|
+
|
|
181
211
|
def check_threshold
|
|
182
212
|
score = mutation_score
|
|
183
213
|
|
|
@@ -285,7 +285,7 @@ module MutationTester
|
|
|
285
285
|
end
|
|
286
286
|
|
|
287
287
|
def run_specs_in_shadow(spec_file, working_dir, example_filters: [])
|
|
288
|
-
test_command(spec_file, example_filters: example_filters).run(timeout: @config.
|
|
288
|
+
test_command(spec_file, example_filters: example_filters).run(timeout: @config.effective_timeout, chdir: working_dir)
|
|
289
289
|
end
|
|
290
290
|
|
|
291
291
|
def discoverable_project_root
|
|
@@ -325,7 +325,7 @@ module MutationTester
|
|
|
325
325
|
end
|
|
326
326
|
|
|
327
327
|
def run_specs_in_place(spec_file, example_filters: [])
|
|
328
|
-
test_command(spec_file, example_filters: example_filters).run(timeout: @config.
|
|
328
|
+
test_command(spec_file, example_filters: example_filters).run(timeout: @config.effective_timeout)
|
|
329
329
|
end
|
|
330
330
|
|
|
331
331
|
def run_mutation_in_memory(mutation, result, project_root = nil)
|
|
@@ -335,7 +335,7 @@ module MutationTester
|
|
|
335
335
|
outcome = runner.execute_in_memory(
|
|
336
336
|
source: mutation[:code],
|
|
337
337
|
path: @source_file,
|
|
338
|
-
timeout: @config.
|
|
338
|
+
timeout: @config.effective_timeout,
|
|
339
339
|
chdir: Dir.pwd
|
|
340
340
|
)
|
|
341
341
|
|
|
@@ -419,7 +419,7 @@ module MutationTester
|
|
|
419
419
|
outcome = runner.execute_in_memory(
|
|
420
420
|
source: @original_content,
|
|
421
421
|
path: @source_file,
|
|
422
|
-
timeout: @config.
|
|
422
|
+
timeout: @config.effective_timeout,
|
|
423
423
|
chdir: Dir.pwd
|
|
424
424
|
)
|
|
425
425
|
return nil if outcome.status == 'pass'
|
|
@@ -3,12 +3,14 @@ module MutationTester
|
|
|
3
3
|
class BaseReporter
|
|
4
4
|
attr_reader :results, :source_file, :spec_file, :config
|
|
5
5
|
|
|
6
|
-
def self.score(results)
|
|
7
|
-
|
|
8
|
-
|
|
6
|
+
def self.score(results, policy: :killed)
|
|
7
|
+
killing_statuses = policy == :separate ? %i[killed] : %i[killed timeout]
|
|
8
|
+
scoring_statuses = killing_statuses + %i[survived]
|
|
9
|
+
kills = results.count { |r| killing_statuses.include?(status_for(r)) }
|
|
10
|
+
scored = results.count { |r| scoring_statuses.include?(status_for(r)) }
|
|
9
11
|
return 0.0 if scored.zero?
|
|
10
12
|
|
|
11
|
-
(
|
|
13
|
+
(kills.to_f / scored * 100).round(2)
|
|
12
14
|
end
|
|
13
15
|
|
|
14
16
|
def self.status_for(result)
|
|
@@ -62,6 +64,8 @@ module MutationTester
|
|
|
62
64
|
end
|
|
63
65
|
|
|
64
66
|
def effective_killed_count
|
|
67
|
+
return killed_count if @config.timeout_policy == :separate
|
|
68
|
+
|
|
65
69
|
killed_count + timeout_count
|
|
66
70
|
end
|
|
67
71
|
|
|
@@ -70,7 +74,7 @@ module MutationTester
|
|
|
70
74
|
end
|
|
71
75
|
|
|
72
76
|
def mutation_score
|
|
73
|
-
self.class.score(@results)
|
|
77
|
+
self.class.score(@results, policy: @config.timeout_policy)
|
|
74
78
|
end
|
|
75
79
|
|
|
76
80
|
def quality_rating
|
|
@@ -53,7 +53,7 @@ module MutationTester
|
|
|
53
53
|
end
|
|
54
54
|
|
|
55
55
|
def aggregate_score
|
|
56
|
-
BaseReporter.score(@result.processed.flat_map { |entry| entry.results || [] })
|
|
56
|
+
BaseReporter.score(@result.processed.flat_map { |entry| entry.results || [] }, policy: @config.timeout_policy)
|
|
57
57
|
end
|
|
58
58
|
|
|
59
59
|
def file_reports
|
|
@@ -45,7 +45,7 @@ namespace :mutation do
|
|
|
45
45
|
end
|
|
46
46
|
|
|
47
47
|
unless all_passed
|
|
48
|
-
puts Rainbow("\n❌ One or more files
|
|
48
|
+
puts Rainbow("\n❌ One or more files failed mutation testing (see the per-file output above)").red
|
|
49
49
|
abort
|
|
50
50
|
end
|
|
51
51
|
end
|
data/readme.md
CHANGED
|
@@ -206,6 +206,8 @@ mutation_test [OPTIONS] --glob 'lib/**/*.rb'
|
|
|
206
206
|
| `--spec-glob TEMPLATE` | Spec-mapping template with a `{name}` placeholder (default: `spec/{name}_spec.rb`). Requires a positional `FILE` list, `--staged`, or `--glob`. |
|
|
207
207
|
| `--since REV` | Incremental batch mode: mutate only the files matched by `--glob` that changed since git revision `REV` (new files count as changed). Requires `--glob`. See [Incremental mode](#incremental-mode-mutate-only-what-changed). |
|
|
208
208
|
| `--fail-fast` | Stop the run at the first surviving mutant and finish with a failing status. Works in single-file mode and with `--glob`. |
|
|
209
|
+
| `--timeout-factor N` | Per-mutant timeout budget as `N` times the measured baseline test run, never below 5 s (default: 5, must be > 0). Ignored when `config.timeout` is set explicitly, which keeps a fixed budget. See [Configuration](#configuration). |
|
|
210
|
+
| `--timeout-policy MODE` | Scoring policy for timed-out mutants: `killed` (default) counts a timeout as a kill; `separate` keeps timeouts out of the score entirely (`killed / (killed + survived)`) and reports them only as their own category in the console, JSON and HTML reports. |
|
|
209
211
|
| `--worker-env NAME` | Set environment variable `NAME` to a distinct per-worker value before each parallel worker boots (`parallel_tests` `TEST_ENV_NUMBER` convention: worker 0 -> `""`, worker N -> `N+1`), so a `parallel_tests`-style `database.yml` selects a per-worker database. You provision the databases (e.g. `rake parallel:prepare`). Not supported by the `in_memory` runner (it falls back to `fork`). See [Making parallelism work with Rails](#making-parallelism-work-with-rails). |
|
|
210
212
|
| `--strict-equality` | Enable the opt-in strict-equality probes (`==` → `eql?` and `==` → `equal?`). Default off; expect noise on code that does not distinguish numeric types or object identity. See [Strict Equality Mutations](docs/mutation-types.md#strict-equality-mutations-opt-in). |
|
|
211
213
|
| `-h, --help` | Show help message. |
|
|
@@ -223,6 +225,22 @@ bundle exec mutation_test --reporters json,html --output-dir build/mutation \
|
|
|
223
225
|
app/models/user.rb spec/models/user_spec.rb
|
|
224
226
|
```
|
|
225
227
|
|
|
228
|
+
### Exit codes (single-file mode)
|
|
229
|
+
|
|
230
|
+
- `0` - the run passed (score met the threshold, or `fail_on_threshold` is disabled).
|
|
231
|
+
- `1` - the mutation score is below the threshold, or the input is unusable (missing
|
|
232
|
+
file, unknown reporter, source with a syntax error).
|
|
233
|
+
- `2` - a usage error (conflicting flags; see the batch sections below).
|
|
234
|
+
- `3` - the run aborted or degraded before reaching a verdict: the shadow workspace
|
|
235
|
+
was unreliable, or every mutant ended as `error`/`stillborn` so nothing was scored.
|
|
236
|
+
This signals an infrastructure or runner problem, not a test-quality gap, so CI
|
|
237
|
+
hooks can distinguish it from a genuine threshold failure.
|
|
238
|
+
- `130` - interrupted with Ctrl+C.
|
|
239
|
+
|
|
240
|
+
Batch modes (`FILE...` lists, `--staged`, `--glob`) keep the exit codes documented in
|
|
241
|
+
their sections below (`0`/`1`/`2`); a degraded file is named explicitly in the batch
|
|
242
|
+
summary instead of being blamed on the threshold.
|
|
243
|
+
|
|
226
244
|
### Running with rake
|
|
227
245
|
|
|
228
246
|
You can also run mutation tests through rake tasks.
|
|
@@ -403,8 +421,24 @@ MutationTester.configure do |config|
|
|
|
403
421
|
# config.parallel_processes = 1
|
|
404
422
|
config.parallel_processes = 4
|
|
405
423
|
|
|
406
|
-
#
|
|
407
|
-
|
|
424
|
+
# Per-mutant timeout (seconds). Calibrated automatically by default:
|
|
425
|
+
# max(5, timeout_factor * measured baseline duration), so a loaded machine
|
|
426
|
+
# cannot inflate the score by turning healthy-but-slow runs into timeout
|
|
427
|
+
# kills; on code paths without a measured baseline the fixed default of 30
|
|
428
|
+
# applies. Setting config.timeout explicitly (a number, or nil for no
|
|
429
|
+
# deadline at all) disables calibration and keeps that fixed budget:
|
|
430
|
+
# config.timeout = 30
|
|
431
|
+
|
|
432
|
+
# Multiplier for the baseline-calibrated per-mutant timeout (> 0; an
|
|
433
|
+
# invalid value falls back to the default with a warning):
|
|
434
|
+
config.timeout_factor = 5
|
|
435
|
+
|
|
436
|
+
# Scoring policy for timed-out mutants. :killed (default) counts a timeout
|
|
437
|
+
# as a kill: (killed + timeout) / (killed + timeout + survived). :separate
|
|
438
|
+
# keeps timeouts out of the score entirely, killed / (killed + survived),
|
|
439
|
+
# and reports them only as their own category, so timeouts under load can
|
|
440
|
+
# never raise the score:
|
|
441
|
+
config.timeout_policy = :killed
|
|
408
442
|
|
|
409
443
|
# Mutant execution runner: :auto (default), :fork, :spawn or :in_memory.
|
|
410
444
|
# :auto picks the fastest safe path and announces any fallback on stderr;
|
|
@@ -725,7 +759,9 @@ to get a single, clean JSON document on **stdout** and nothing else: the banner,
|
|
|
725
759
|
progress spinner, colours and the "report saved" notice all go to **stderr**, so
|
|
726
760
|
the stream is safe to pipe straight into `jq` or a parser. The process still
|
|
727
761
|
exits `0` when the mutation score meets the configured threshold and `1` when it
|
|
728
|
-
does not
|
|
762
|
+
does not (a degraded single-file run exits `3`, see
|
|
763
|
+
[Exit codes](#exit-codes-single-file-mode)), so the exit code remains a
|
|
764
|
+
pass/fail signal.
|
|
729
765
|
|
|
730
766
|
```bash
|
|
731
767
|
bundle exec mutation_test --json examples/calculator.rb examples/calculator_spec.rb | jq .
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutation_tester
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kamil Dzierbicki
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-07-
|
|
11
|
+
date: 2026-07-15 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: parallel
|