mutineer 0.6.2 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +30 -0
- data/README.md +30 -2
- data/lib/mutineer/baseline.rb +70 -0
- data/lib/mutineer/cli.rb +80 -9
- data/lib/mutineer/config.rb +9 -2
- data/lib/mutineer/coverage_map.rb +49 -7
- data/lib/mutineer/mutant_id.rb +47 -0
- data/lib/mutineer/pairing.rb +67 -0
- data/lib/mutineer/reporter.rb +117 -8
- data/lib/mutineer/result.rb +55 -27
- data/lib/mutineer/runner.rb +78 -9
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +2 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ae2c0b5167b5070a3afc7834ffa6d4d012267b070dd6b73f3d95affa3752c4c0
|
|
4
|
+
data.tar.gz: 22a328c5b5446d8c937288763d5ab050869b49b58c0bee715c3ec9dbb54005fd
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 858c634ce049e4a55ee2c94151a11df0b620edb19630d53719bf94cf6efa9212a4bd6d55c67607a1447ff42fed7804293ae3ec8c28b703b6478f2324aa5593b9
|
|
7
|
+
data.tar.gz: 2c47217873142ba8dbdffe1adbeb3772532925a951a2d52f058ca4b5e26732877019299de6a98c5c68ad89a5d9c6f8513b06ac424806fe7258e4ef1a7e305a27
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,35 @@ All notable changes to this project are documented here. The format is based on
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/), and this project adheres to
|
|
5
5
|
[Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [0.7.0] - 2026-06-30
|
|
8
|
+
|
|
9
|
+
Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- **Equivalent-mutant suppression** (#10) — inline `# mutineer:disable-line [ops]`
|
|
13
|
+
and a `.mutineer.yml` `ignore:` list keyed on a stable, offset-free mutant id;
|
|
14
|
+
suppressed mutants are excluded from the score (100% reachable). The stable id
|
|
15
|
+
(and readable token) is emitted per survivor in JSON.
|
|
16
|
+
- **Source→test auto-pairing + multi-source runs** (#11) — pass a directory or
|
|
17
|
+
several sources with `--test` omitted; tests are inferred by convention
|
|
18
|
+
(`app/`,`lib/` → `test/…_test.rb` / `spec/…_spec.rb`) and run under one boot,
|
|
19
|
+
with per-source results (human + JSON `per_source`).
|
|
20
|
+
- **`--baseline <file.json>` CI gating** (#13) — diff against a prior run by stable
|
|
21
|
+
id; exit 1 on new survivors or a score drop (with `--baseline-epsilon`), naming
|
|
22
|
+
what regressed. Combines with `--threshold` via max exit code.
|
|
23
|
+
- **`--verbose`/`--debug`** (#8) — surface the real error when a fork capture fails.
|
|
24
|
+
- **`:uncapturable` status** (#9) — distinct from `no_coverage`; reported separately
|
|
25
|
+
("tests failed to run" vs "genuinely uncovered"). Both excluded from the score.
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
- **Fork capture no longer drops fixture transactions** (#8) — `reconnect` skips
|
|
29
|
+
`clear_all_connections!` when a fixture transaction is open, and stops swallowing
|
|
30
|
+
the child error; write-heavy Rails tests are mutation-testable again.
|
|
31
|
+
|
|
32
|
+
### Changed
|
|
33
|
+
- JSON `schema_version` → `1.1` (additive: survivor `id`/`token`, `ignored`,
|
|
34
|
+
`uncapturable`, `per_source`).
|
|
35
|
+
|
|
7
36
|
## [0.6.2] - 2026-06-29
|
|
8
37
|
|
|
9
38
|
### Fixed
|
|
@@ -100,6 +129,7 @@ All notable changes to this project are documented here. The format is based on
|
|
|
100
129
|
- `.mutineer.yml` configuration (CLI > config > default precedence).
|
|
101
130
|
- Byte-correct source handling for multibyte (UTF-8) sources.
|
|
102
131
|
|
|
132
|
+
[0.7.0]: https://github.com/davidteren/mutineer/releases/tag/v0.7.0
|
|
103
133
|
[0.6.2]: https://github.com/davidteren/mutineer/releases/tag/v0.6.2
|
|
104
134
|
[0.6.1]: https://github.com/davidteren/mutineer/releases/tag/v0.6.1
|
|
105
135
|
[0.6.0]: https://github.com/davidteren/mutineer/releases/tag/v0.6.0
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Mutineer
|
|
2
2
|
|
|
3
|
-
[](https://socket.dev/rubygems/package/mutineer/overview/0.7.0?platform=ruby)
|
|
4
4
|
|
|
5
5
|
A clean-room mutation-testing tool for Ruby. Mutineer mutates your source one
|
|
6
6
|
change at a time, runs your test suite (Minitest or RSpec) against each mutant, and reports the
|
|
@@ -49,7 +49,10 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
49
49
|
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
50
50
|
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
51
51
|
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`) — ideal for PR CI |
|
|
52
|
-
| `--
|
|
52
|
+
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (see [CI](#ci-gating)) |
|
|
53
|
+
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
54
|
+
| `--jobs N` | Parallel worker count (default: processor count; `1` under `--rails`) |
|
|
55
|
+
| `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
|
|
53
56
|
| `--strategy NAME` | Mutation application: `reload` whole-file (default) or `redefine` surgical (`7a`/`7b` accepted as deprecated aliases) |
|
|
54
57
|
| `--format human\|json` | Report format (default: human) |
|
|
55
58
|
| `--output FILE` | Write the report to FILE instead of stdout |
|
|
@@ -96,6 +99,31 @@ Add Mutineer to your Gemfile's test group:
|
|
|
96
99
|
gem "mutineer", group: :test, require: false
|
|
97
100
|
```
|
|
98
101
|
|
|
102
|
+
## Suppressing equivalent mutants
|
|
103
|
+
|
|
104
|
+
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
105
|
+
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
106
|
+
|
|
107
|
+
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`).
|
|
108
|
+
- **Config:** a `.mutineer.yml` `ignore:` list of stable mutant ids. Each survivor's
|
|
109
|
+
`id` is printed in the JSON report, so copy it straight into `ignore:`.
|
|
110
|
+
|
|
111
|
+
Suppressed mutants are excluded from the score (so 100% becomes reachable).
|
|
112
|
+
|
|
113
|
+
## CI gating
|
|
114
|
+
|
|
115
|
+
Store a JSON run as a baseline, then fail the build only when a PR makes things
|
|
116
|
+
worse:
|
|
117
|
+
|
|
118
|
+
```sh
|
|
119
|
+
mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
`--baseline` reports which survivors are new (by stable id) and any score drop. It
|
|
123
|
+
combines with `--threshold` (the worse of the two sets the exit code). Pass a
|
|
124
|
+
directory (or several sources) to audit a whole layer in one boot — tests are
|
|
125
|
+
auto-paired by convention and the report breaks down per source.
|
|
126
|
+
|
|
99
127
|
## Configuration
|
|
100
128
|
|
|
101
129
|
Mutineer reads an optional `.mutineer.yml` from the project root (nearest one,
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require_relative "config" # for Mutineer::ConfigError
|
|
5
|
+
|
|
6
|
+
module Mutineer
|
|
7
|
+
# #13: CI baseline/delta gating. A baseline is literally a prior
|
|
8
|
+
# `mutineer run --format json` document (KTD-1) — no bespoke format to version.
|
|
9
|
+
# We diff the current run against it by the #10 stable survivor id (KTD-2): a
|
|
10
|
+
# NEW survivor (id present now, absent in the baseline) OR a score drop is a
|
|
11
|
+
# regression the CLI turns into exit 1. Pure data — stdlib `json` only, no fork,
|
|
12
|
+
# no Rails — so it's testable in isolation from a canned JSON + a hand-built
|
|
13
|
+
# AggregateResult.
|
|
14
|
+
class Baseline
|
|
15
|
+
# The verdict of diffing a current run against the baseline.
|
|
16
|
+
# new_survivors — current Result objects whose stable id is absent from the
|
|
17
|
+
# baseline (the regressions to name).
|
|
18
|
+
# fixed_survivors — baseline survivor hashes absent from the current run
|
|
19
|
+
# (informational, never gates).
|
|
20
|
+
# score_drop — current score < baseline score - epsilon. nil on either
|
|
21
|
+
# side skips the check (see #diff).
|
|
22
|
+
# regressed — any new survivors OR a score drop.
|
|
23
|
+
Delta = Data.define(:new_survivors, :fixed_survivors,
|
|
24
|
+
:score_before, :score_after, :score_drop, :regressed)
|
|
25
|
+
|
|
26
|
+
# Load a prior --format json run. Raises ConfigError (NOT exit — R8: a data
|
|
27
|
+
# class must never kill the host) on a missing/unreadable file, unparseable
|
|
28
|
+
# JSON, or a doc that isn't a baseline shape, so the CLI maps it to exit 2
|
|
29
|
+
# (usage) like every other bad-path flag.
|
|
30
|
+
def self.load(path)
|
|
31
|
+
doc = JSON.parse(File.read(path))
|
|
32
|
+
unless doc.is_a?(Hash) && doc["schema_version"] && doc["survivors"].is_a?(Array)
|
|
33
|
+
raise ConfigError, "not a Mutineer JSON report: #{path}"
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
new(doc)
|
|
37
|
+
rescue JSON::ParserError => e
|
|
38
|
+
raise ConfigError, "invalid baseline JSON in #{path}: #{e.message}"
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
attr_reader :score
|
|
42
|
+
|
|
43
|
+
def initialize(doc)
|
|
44
|
+
@survivors = doc["survivors"] || []
|
|
45
|
+
@score = doc.dig("summary", "score")
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# Diff a current AggregateResult against this baseline by stable survivor id.
|
|
49
|
+
# `epsilon` tolerates float jitter on the score (default 0.0 = any drop gates).
|
|
50
|
+
def diff(aggregate, epsilon: 0.0)
|
|
51
|
+
current = aggregate.surviving_mutants
|
|
52
|
+
current_ids = current.map(&:id)
|
|
53
|
+
baseline_ids = @survivors.map { |h| h["id"] }
|
|
54
|
+
|
|
55
|
+
new_survivors = current.reject { |r| baseline_ids.include?(r.id) }
|
|
56
|
+
fixed = @survivors.reject { |h| current_ids.include?(h["id"]) }
|
|
57
|
+
|
|
58
|
+
current_score = aggregate.mutation_score
|
|
59
|
+
# nil-score discipline (mirrors Reporter#exit_code): a score absent on either
|
|
60
|
+
# side can't be compared — skip the drop check, keep the new-survivor check.
|
|
61
|
+
score_drop = !@score.nil? && !current_score.nil? &&
|
|
62
|
+
current_score < @score - epsilon
|
|
63
|
+
|
|
64
|
+
Delta.new(new_survivors: new_survivors, fixed_survivors: fixed,
|
|
65
|
+
score_before: @score, score_after: current_score,
|
|
66
|
+
score_drop: score_drop,
|
|
67
|
+
regressed: !new_survivors.empty? || score_drop)
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
data/lib/mutineer/cli.rb
CHANGED
|
@@ -7,9 +7,11 @@ require_relative "version"
|
|
|
7
7
|
require_relative "config"
|
|
8
8
|
require_relative "parser"
|
|
9
9
|
require_relative "project"
|
|
10
|
+
require_relative "pairing"
|
|
10
11
|
require_relative "changed_lines"
|
|
11
12
|
require_relative "runner"
|
|
12
13
|
require_relative "reporter"
|
|
14
|
+
require_relative "baseline"
|
|
13
15
|
require_relative "mutator_registry"
|
|
14
16
|
|
|
15
17
|
module Mutineer
|
|
@@ -33,6 +35,9 @@ module Mutineer
|
|
|
33
35
|
--test FILE Test file covering the sources (repeatable)
|
|
34
36
|
--operators LIST Comma-separated operator names (default: Tier 1 set)
|
|
35
37
|
--threshold FLOAT Fail (exit 1) when score < FLOAT (default: 0 = off)
|
|
38
|
+
--baseline FILE Fail (exit 1) on NEW survivors / score drop vs a prior
|
|
39
|
+
--format json run (CI delta gate)
|
|
40
|
+
--baseline-epsilon FLOAT Score-drop tolerance for --baseline (default: 0)
|
|
36
41
|
--only NAME Restrict to one fully-qualified subject
|
|
37
42
|
--since REF Only mutate lines changed since git REF (e.g. origin/main)
|
|
38
43
|
--jobs N Parallel worker count (default: processor count)
|
|
@@ -44,6 +49,7 @@ module Mutineer
|
|
|
44
49
|
--format human|json Report format (default: human)
|
|
45
50
|
--output FILE Write the report to FILE instead of stdout
|
|
46
51
|
--dry-run List mutations without executing
|
|
52
|
+
--verbose Surface the real error when a fork capture fails (alias: --debug)
|
|
47
53
|
|
|
48
54
|
Options:
|
|
49
55
|
--list-operators List available operators (default vs optional) and exit
|
|
@@ -84,8 +90,14 @@ module Mutineer
|
|
|
84
90
|
o.on("--framework NAME") { |v| opts[:framework] = v; explicit << :framework }
|
|
85
91
|
o.on("--boot FILE") { |v| opts[:boot] = v; explicit << :boot }
|
|
86
92
|
o.on("--rails") { opts[:rails] = true }
|
|
93
|
+
o.on("--verbose") { opts[:verbose] = true }
|
|
94
|
+
o.on("--debug") { opts[:verbose] = true } # alias of --verbose
|
|
87
95
|
o.on("--format FORMAT") { |v| opts[:format] = v }
|
|
88
96
|
o.on("--output FILE") { |v| opts[:output] = v }
|
|
97
|
+
# #13: --baseline is also a .mutineer.yml key, so mark it explicit when
|
|
98
|
+
# typed (CLI wins over the file). --baseline-epsilon is CLI-only.
|
|
99
|
+
o.on("--baseline FILE") { |v| opts[:baseline] = v; explicit << :baseline }
|
|
100
|
+
o.on("--baseline-epsilon FLOAT") { |v| opts[:baseline_epsilon] = v.to_f }
|
|
89
101
|
end
|
|
90
102
|
|
|
91
103
|
begin
|
|
@@ -118,8 +130,10 @@ module Mutineer
|
|
|
118
130
|
|
|
119
131
|
case argv.first
|
|
120
132
|
when "run"
|
|
121
|
-
|
|
122
|
-
|
|
133
|
+
# #11: a directory source expands to its **/*.rb files; literal files pass
|
|
134
|
+
# through. Test inference (when --test is omitted) happens in validate!.
|
|
135
|
+
config.sources = Pairing.expand_sources(argv[1..], project_root: config.project_root)
|
|
136
|
+
run(config, explicit)
|
|
123
137
|
else
|
|
124
138
|
warn "mutineer: unknown command '#{argv.first}'"
|
|
125
139
|
exit 2
|
|
@@ -134,12 +148,12 @@ module Mutineer
|
|
|
134
148
|
end
|
|
135
149
|
end
|
|
136
150
|
|
|
137
|
-
def self.run(config)
|
|
151
|
+
def self.run(config, explicit = Set.new)
|
|
138
152
|
if config.sources.empty?
|
|
139
153
|
warn "mutineer: run requires at least one source file"
|
|
140
154
|
exit 2
|
|
141
155
|
end
|
|
142
|
-
validate!(config)
|
|
156
|
+
validate!(config, explicit)
|
|
143
157
|
|
|
144
158
|
config.dry_run ? dry_run(config) : execute(config)
|
|
145
159
|
rescue ArgumentError => e
|
|
@@ -163,7 +177,7 @@ module Mutineer
|
|
|
163
177
|
|
|
164
178
|
# Flag validation: every flag/usage failure exits 2 (C7), consistent with the
|
|
165
179
|
# taxonomy above — CI can tell "mistyped flag" from "tests too weak."
|
|
166
|
-
def self.validate!(config)
|
|
180
|
+
def self.validate!(config, explicit = Set.new)
|
|
167
181
|
unless (0.0..100.0).cover?(config.threshold)
|
|
168
182
|
warn "mutineer: --threshold must be between 0 and 100"
|
|
169
183
|
exit 2
|
|
@@ -194,6 +208,17 @@ module Mutineer
|
|
|
194
208
|
exit 2
|
|
195
209
|
end
|
|
196
210
|
|
|
211
|
+
validate_since!(config) if config.since
|
|
212
|
+
preflight_output!(config.output) if config.output
|
|
213
|
+
preflight_baseline!(config.baseline) if config.baseline
|
|
214
|
+
|
|
215
|
+
# #11: when --test is omitted, infer each source's test by convention so the
|
|
216
|
+
# boot-once/fork-per-test core (which pairs empirically by coverage) gets a
|
|
217
|
+
# populated config.tests. Runs after every flag/usage check above so a
|
|
218
|
+
# mistyped flag still reports the flag; skipped under --dry-run (no tests
|
|
219
|
+
# needed). validate_paths! then sees the inferred (real) tests.
|
|
220
|
+
autopair!(config, explicit) unless config.dry_run
|
|
221
|
+
|
|
197
222
|
# Boot mode does no coverage selection — every mutant runs the given tests —
|
|
198
223
|
# so at least one --test file is mandatory (there is nothing to select from).
|
|
199
224
|
if config.boot && config.tests.empty?
|
|
@@ -201,8 +226,6 @@ module Mutineer
|
|
|
201
226
|
exit 2
|
|
202
227
|
end
|
|
203
228
|
|
|
204
|
-
validate_since!(config) if config.since
|
|
205
|
-
preflight_output!(config.output) if config.output
|
|
206
229
|
validate_paths!(config)
|
|
207
230
|
end
|
|
208
231
|
|
|
@@ -238,6 +261,45 @@ module Mutineer
|
|
|
238
261
|
exit 2
|
|
239
262
|
end
|
|
240
263
|
|
|
264
|
+
# #11: auto-pair sources to tests by path convention when no --test was given
|
|
265
|
+
# (explicit --test wins — R5). Each source with an inferred test on disk joins
|
|
266
|
+
# the run; a source with none is dropped with a one-line stderr warning (R3)
|
|
267
|
+
# and the run continues with the rest. If every source is dropped: in boot mode
|
|
268
|
+
# the dedicated --boot/--rails-requires-test check reports it; otherwise exit 2
|
|
269
|
+
# with a usage message. The framework is re-detected from the inferred set
|
|
270
|
+
# unless it was set explicitly (a spec-only project loads/reports as rspec).
|
|
271
|
+
def self.autopair!(config, explicit)
|
|
272
|
+
return unless config.tests.empty?
|
|
273
|
+
|
|
274
|
+
paired = config.sources.filter_map do |s|
|
|
275
|
+
t = Pairing.infer_test(s, project_root: config.project_root, prefer: config.framework)
|
|
276
|
+
[s, t] if t
|
|
277
|
+
end
|
|
278
|
+
(config.sources - paired.map(&:first)).each do |s|
|
|
279
|
+
warn "[mutineer] no test found by convention for #{s}; skipping"
|
|
280
|
+
end
|
|
281
|
+
config.sources = paired.map(&:first)
|
|
282
|
+
config.tests = paired.map(&:last).uniq
|
|
283
|
+
config.framework = Config.detect_framework(config.tests) unless explicit.include?(:framework)
|
|
284
|
+
|
|
285
|
+
return unless config.sources.empty?
|
|
286
|
+
return if config.boot # let the --boot/--rails-requires-test check report it
|
|
287
|
+
|
|
288
|
+
warn "mutineer: no test files found by convention; pass --test or add tests"
|
|
289
|
+
exit 2
|
|
290
|
+
end
|
|
291
|
+
|
|
292
|
+
# #13: a missing/unreadable/unparseable baseline is a usage error (exit 2),
|
|
293
|
+
# mirroring --output/--since preflight, so CI sees "bad invocation," not a
|
|
294
|
+
# backtrace mid-run. Validating up front = attempting the load (it raises
|
|
295
|
+
# ConfigError/SystemCallError; the actual diff reloads in execute).
|
|
296
|
+
def self.preflight_baseline!(path)
|
|
297
|
+
Baseline.load(path)
|
|
298
|
+
rescue Mutineer::ConfigError, SystemCallError => e
|
|
299
|
+
warn "mutineer: #{e.message}"
|
|
300
|
+
exit 2
|
|
301
|
+
end
|
|
302
|
+
|
|
241
303
|
def self.preflight_output!(path)
|
|
242
304
|
dir = File.dirname(File.expand_path(path))
|
|
243
305
|
return if File.directory?(dir) && File.writable?(dir)
|
|
@@ -255,8 +317,14 @@ module Mutineer
|
|
|
255
317
|
|
|
256
318
|
aggregate, source_map = Runner.execute(config)
|
|
257
319
|
reporter = Reporter.new(aggregate, source_map)
|
|
320
|
+
|
|
321
|
+
# #13: diff the current run against the baseline (preflighted above) by the
|
|
322
|
+
# stable survivor id. The delta is rendered inline (human section / additive
|
|
323
|
+
# json block) and gates exit independently of --threshold.
|
|
324
|
+
delta = (Baseline.load(config.baseline).diff(aggregate, epsilon: config.baseline_epsilon) if config.baseline)
|
|
325
|
+
|
|
258
326
|
reporter.report(out: $stdout, err: $stderr, threshold: config.threshold,
|
|
259
|
-
format: config.format, output: config.output)
|
|
327
|
+
format: config.format, output: config.output, baseline: delta)
|
|
260
328
|
|
|
261
329
|
# #14: nudge toward the opt-in tier-2 operators (human report only — never
|
|
262
330
|
# pollute JSON output).
|
|
@@ -264,7 +332,10 @@ module Mutineer
|
|
|
264
332
|
puts hint
|
|
265
333
|
end
|
|
266
334
|
|
|
267
|
-
|
|
335
|
+
# #13/KTD-4: --baseline and --threshold are independent gates OR'd together.
|
|
336
|
+
# `max` of two 0/1 codes is the OR; usage (2) is handled earlier and wins.
|
|
337
|
+
baseline_exit = delta&.regressed ? 1 : 0
|
|
338
|
+
exit [reporter.exit_code(threshold: config.threshold), baseline_exit].max
|
|
268
339
|
end
|
|
269
340
|
|
|
270
341
|
# The tier-2 operators not in the active set, as a one-line hint (or nil when
|
data/lib/mutineer/config.rb
CHANGED
|
@@ -24,12 +24,13 @@ module Mutineer
|
|
|
24
24
|
:sources, :tests, :operators, :threshold, :only, :dry_run,
|
|
25
25
|
:cache_dir, :project_root, :load_paths,
|
|
26
26
|
:jobs, :format, :output, :strategy, :require_paths,
|
|
27
|
-
:boot, :rails, :since, :framework,
|
|
27
|
+
:boot, :rails, :since, :framework, :verbose, :ignore,
|
|
28
|
+
:baseline, :baseline_epsilon,
|
|
28
29
|
keyword_init: true
|
|
29
30
|
) do
|
|
30
31
|
CONFIG_FILE = ".mutineer.yml"
|
|
31
32
|
# Keys accepted in .mutineer.yml (R7). `require` maps to the :require_paths field.
|
|
32
|
-
KNOWN_KEYS = %w[operators jobs threshold only require boot rails since framework].freeze
|
|
33
|
+
KNOWN_KEYS = %w[operators jobs threshold only require boot rails since framework verbose ignore baseline].freeze
|
|
33
34
|
|
|
34
35
|
def initialize(**kwargs)
|
|
35
36
|
super
|
|
@@ -45,6 +46,9 @@ module Mutineer
|
|
|
45
46
|
self.strategy ||= "reload"
|
|
46
47
|
self.require_paths ||= []
|
|
47
48
|
self.rails = false if rails.nil?
|
|
49
|
+
self.verbose = false if verbose.nil?
|
|
50
|
+
self.ignore ||= []
|
|
51
|
+
self.baseline_epsilon ||= 0.0
|
|
48
52
|
end
|
|
49
53
|
|
|
50
54
|
# Walk from `start` toward `home`, returning the first .mutineer.yml path found
|
|
@@ -143,6 +147,9 @@ module Mutineer
|
|
|
143
147
|
when "boot" then value.to_s
|
|
144
148
|
when "framework" then value.to_s
|
|
145
149
|
when "rails" then value == true || value.to_s == "true"
|
|
150
|
+
when "verbose" then value == true || value.to_s == "true"
|
|
151
|
+
when "ignore" then Array(value).map(&:to_s)
|
|
152
|
+
when "baseline" then value.to_s
|
|
146
153
|
else value
|
|
147
154
|
end
|
|
148
155
|
end
|
|
@@ -6,6 +6,7 @@ require "digest"
|
|
|
6
6
|
require "fileutils"
|
|
7
7
|
require "rbconfig"
|
|
8
8
|
require "coverage"
|
|
9
|
+
require "set"
|
|
9
10
|
require_relative "minitest_integration"
|
|
10
11
|
require_relative "test_runners"
|
|
11
12
|
|
|
@@ -25,7 +26,7 @@ module Mutineer
|
|
|
25
26
|
def initialize(source_paths:, test_paths:, cache_dir: ".mutineer",
|
|
26
27
|
load_paths: ["lib"], project_root: Dir.pwd,
|
|
27
28
|
capture_timeout: DEFAULT_CAPTURE_TIMEOUT, boot_path: nil,
|
|
28
|
-
framework: "minitest")
|
|
29
|
+
framework: "minitest", verbose: false)
|
|
29
30
|
@source_paths = Array(source_paths)
|
|
30
31
|
@test_paths = Array(test_paths)
|
|
31
32
|
@cache_dir = cache_dir
|
|
@@ -34,6 +35,7 @@ module Mutineer
|
|
|
34
35
|
@capture_timeout = capture_timeout
|
|
35
36
|
@boot_path = boot_path
|
|
36
37
|
@framework = framework || "minitest"
|
|
38
|
+
@verbose = verbose
|
|
37
39
|
@map = {}
|
|
38
40
|
@failed_test_files = []
|
|
39
41
|
@phase_a_ran = false
|
|
@@ -64,8 +66,40 @@ module Mutineer
|
|
|
64
66
|
@map["#{relativize(file)}:#{line}"] || []
|
|
65
67
|
end
|
|
66
68
|
|
|
69
|
+
# #9: is this source file's empty coverage the result of an *errored* capture
|
|
70
|
+
# rather than a genuine coverage gap? True iff (KTD-2) some capture failed this
|
|
71
|
+
# run AND this file got zero coverage from any successful capture AND a failed
|
|
72
|
+
# test file maps to it by the standard _test/_spec naming convention. Derived
|
|
73
|
+
# purely from already-persisted state (@map keys + @failed_test_files); no rerun,
|
|
74
|
+
# no new cached field, no digest change.
|
|
75
|
+
#
|
|
76
|
+
# ponytail: file-level, convention-based attribution. A line covered only by a
|
|
77
|
+
# failed test in an otherwise-covered file stays no_coverage (condition 2), and
|
|
78
|
+
# a source with no naming-convention test match is never tainted. Upgrade path:
|
|
79
|
+
# persist per-file coverage per successful run and diff against the failed set,
|
|
80
|
+
# or record test->source targets explicitly. Not needed for the #8/#9 cases.
|
|
81
|
+
def uncapturable_source?(file)
|
|
82
|
+
return false if @failed_test_files.empty?
|
|
83
|
+
|
|
84
|
+
rel = relativize(absolute(file))
|
|
85
|
+
return false if covered_source_files.include?(rel)
|
|
86
|
+
|
|
87
|
+
failed_test_targets.include?(File.basename(rel, ".rb"))
|
|
88
|
+
end
|
|
89
|
+
|
|
67
90
|
private
|
|
68
91
|
|
|
92
|
+
# Source rel-paths that received coverage from any successful capture.
|
|
93
|
+
def covered_source_files
|
|
94
|
+
@map.keys.map { |k| k.rpartition(":").first }.to_set
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# Basenames of failed test files with a trailing _test/_spec (and .rb) stripped,
|
|
98
|
+
# i.e. the source basenames they would have covered by convention.
|
|
99
|
+
def failed_test_targets
|
|
100
|
+
@failed_test_files.map { |t| File.basename(t, ".rb").sub(/_(test|spec)\z/, "") }.to_set
|
|
101
|
+
end
|
|
102
|
+
|
|
69
103
|
# Shared cache dance for both build paths: hit the digest-keyed cache, else
|
|
70
104
|
# yield to populate @map and persist it.
|
|
71
105
|
def cached_or
|
|
@@ -108,10 +142,16 @@ module Mutineer
|
|
|
108
142
|
abs_sources = abs_source_paths
|
|
109
143
|
|
|
110
144
|
@test_paths.each do |test_path|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
145
|
+
# Tri-state payload (KTD-1): Hash = coverage, String = error diagnostic
|
|
146
|
+
# from the child, nil = pipe gone / empty.
|
|
147
|
+
# ponytail/#9: this String diagnostic is what #9 turns into an :uncapturable status.
|
|
148
|
+
case (coverage = fork_capture(absolute(test_path), abs_sources, rails))
|
|
149
|
+
when Hash then record(coverage, test_path)
|
|
150
|
+
when String
|
|
151
|
+
fail_test(test_path, @verbose ? "fork capture failed: #{coverage}" :
|
|
152
|
+
"fork capture produced no result (re-run with --verbose for the error)")
|
|
153
|
+
else fail_test(test_path, "fork capture produced no result")
|
|
154
|
+
end
|
|
115
155
|
end
|
|
116
156
|
end
|
|
117
157
|
|
|
@@ -132,8 +172,10 @@ module Mutineer
|
|
|
132
172
|
Coverage.result(stop: false)
|
|
133
173
|
.select { |f, _| abs_sources.include?(f) }
|
|
134
174
|
.transform_values { |v| v.is_a?(Hash) ? v[:lines] : v }
|
|
135
|
-
rescue Exception # rubocop:disable Lint/RescueException
|
|
136
|
-
|
|
175
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
176
|
+
# KTD-1: stringify (an arbitrary Exception may not marshal); the parent
|
|
177
|
+
# surfaces this under --verbose. A String marshals safely over the pipe.
|
|
178
|
+
"#{e.class}: #{e.message}#{e.backtrace&.first ? " @ #{e.backtrace.first}" : ''}"
|
|
137
179
|
end
|
|
138
180
|
begin
|
|
139
181
|
wr.write(Marshal.dump(payload))
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "digest"
|
|
4
|
+
|
|
5
|
+
module Mutineer
|
|
6
|
+
# Content-based stable id for a mutant — NOT byte offsets. Pure function, reused
|
|
7
|
+
# by the Runner (matching the ignore list), the Reporter (emitting a copy-
|
|
8
|
+
# pasteable id per survivor), and #13 baseline gating (diffing id-sets run to
|
|
9
|
+
# run). `digest` is stdlib, so zero new deps.
|
|
10
|
+
#
|
|
11
|
+
# Offset-free by design: keyed on the subject's qualified_name (a method, not a
|
|
12
|
+
# byte position) + operator + the normalized mutated token + an occurrence
|
|
13
|
+
# ordinal among same-(operator, token) twins WITHIN the subject. So it survives
|
|
14
|
+
# any edit outside the subject method — where raw start/end offsets shift on
|
|
15
|
+
# every edit earlier in the file and would silently stop matching.
|
|
16
|
+
module MutantId
|
|
17
|
+
module_function
|
|
18
|
+
|
|
19
|
+
# NUL-joined so token delimiters (`||=`, spaces, `::`, `#`) can never collide
|
|
20
|
+
# with the separator; SHA256[0,12] gives a fixed-length, copy-pasteable key.
|
|
21
|
+
def for(subject, mutation, source, occurrence = 0)
|
|
22
|
+
Digest::SHA256.hexdigest(
|
|
23
|
+
[subject.qualified_name, mutation.operator,
|
|
24
|
+
normalized_token(mutation, source), occurrence].join("\x00")
|
|
25
|
+
)[0, 12]
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Ids for a subject's full mutation list, in input order, assigning each its
|
|
29
|
+
# 0-based occurrence among twins sharing the same (operator, token). This is
|
|
30
|
+
# what disambiguates `a + b + c`'s two `+` mutants without an offset.
|
|
31
|
+
def for_subject(subject, source, mutations)
|
|
32
|
+
seen = Hash.new(0)
|
|
33
|
+
mutations.map do |m|
|
|
34
|
+
key = [m.operator, normalized_token(m, source)]
|
|
35
|
+
occ = seen[key]
|
|
36
|
+
seen[key] += 1
|
|
37
|
+
self.for(subject, m, source, occ)
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# The exact code being mutated, whitespace-collapsed — same normalization the
|
|
42
|
+
# Reporter's diff_for uses for its token label.
|
|
43
|
+
def normalized_token(mutation, source)
|
|
44
|
+
source.byteslice(mutation.start_offset...mutation.end_offset).gsub(/\s+/, " ").strip
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
# Source -> test pairing by path convention (#11). Pure stdlib path logic:
|
|
5
|
+
# no Rails, no class loading, no process. Two jobs:
|
|
6
|
+
# * expand_sources — a directory argument becomes its sorted **/*.rb files.
|
|
7
|
+
# * infer_test — a source's test file by convention (app/ and lib/ sources
|
|
8
|
+
# map to test/.../_test.rb or spec/.../_spec.rb), preserving
|
|
9
|
+
# namespaced subdirectories. First EXISTING candidate wins.
|
|
10
|
+
#
|
|
11
|
+
# Independently unit-testable: every method is pure in/out over the filesystem,
|
|
12
|
+
# so the pairing contract is exercised with plain fixtures, no Rails, no fork.
|
|
13
|
+
module Pairing
|
|
14
|
+
module_function
|
|
15
|
+
|
|
16
|
+
# Expand each positional source: a directory -> its sorted **/*.rb files
|
|
17
|
+
# (relative to project_root); a file (or glob, or anything non-directory) ->
|
|
18
|
+
# itself. Flattened, deduped, order-stable.
|
|
19
|
+
def expand_sources(args, project_root:)
|
|
20
|
+
root = File.expand_path(project_root)
|
|
21
|
+
Array(args).flat_map do |arg|
|
|
22
|
+
abs = File.expand_path(arg, root)
|
|
23
|
+
if File.directory?(abs)
|
|
24
|
+
Dir.glob(File.join(abs, "**", "*.rb")).sort.map { |f| f.delete_prefix("#{root}/") }
|
|
25
|
+
else
|
|
26
|
+
[arg]
|
|
27
|
+
end
|
|
28
|
+
end.uniq
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# The first EXISTING candidate test path for a source (relative to
|
|
32
|
+
# project_root), or nil. `prefer` is the resolved framework ("minitest" |
|
|
33
|
+
# "rspec"): its candidates are tried first, the other framework's as fallback,
|
|
34
|
+
# so a minitest default still finds a spec and vice-versa.
|
|
35
|
+
def infer_test(source_rel, project_root:, prefer: "minitest")
|
|
36
|
+
base, lib = logical_path(source_rel)
|
|
37
|
+
candidates(base, lib, prefer).find do |rel|
|
|
38
|
+
File.exist?(File.expand_path(rel, project_root))
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Strip the source root to a logical path (no ".rb") and flag lib/ sources.
|
|
43
|
+
# app/foo/bar.rb and lib/foo/bar.rb both -> "foo/bar"; anything else -> the
|
|
44
|
+
# path minus ".rb" (still attempted). Namespaced subdirs are preserved
|
|
45
|
+
# verbatim — structural, never constant resolution.
|
|
46
|
+
def logical_path(source_rel)
|
|
47
|
+
no_ext = source_rel.sub(/\.rb\z/, "")
|
|
48
|
+
if no_ext.start_with?("app/")
|
|
49
|
+
[no_ext.sub(%r{\Aapp/}, ""), false]
|
|
50
|
+
elsif no_ext.start_with?("lib/")
|
|
51
|
+
[no_ext.sub(%r{\Alib/}, ""), true]
|
|
52
|
+
else
|
|
53
|
+
[no_ext, false]
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Ordered candidate test paths. lib/ sources also get test/lib/... and
|
|
58
|
+
# spec/lib/... (Rails apps put lib tests under either layout).
|
|
59
|
+
def candidates(base, lib, prefer)
|
|
60
|
+
minitest = ["test/#{base}_test.rb"]
|
|
61
|
+
minitest << "test/lib/#{base}_test.rb" if lib
|
|
62
|
+
rspec = ["spec/#{base}_spec.rb"]
|
|
63
|
+
rspec << "spec/lib/#{base}_spec.rb" if lib
|
|
64
|
+
prefer == "rspec" ? rspec + minitest : minitest + rspec
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
end
|
data/lib/mutineer/reporter.rb
CHANGED
|
@@ -21,13 +21,14 @@ module Mutineer
|
|
|
21
21
|
# Single entry point (R20/R21). Branches on `format` ("human" | "json") and
|
|
22
22
|
# routes the rendered report to `output` (a file, with a stderr confirmation)
|
|
23
23
|
# or to `out`. Diagnostics always go to `err`.
|
|
24
|
-
def report(out: $stdout, err: $stderr, threshold: 0.0, format: "human", output: nil)
|
|
24
|
+
def report(out: $stdout, err: $stderr, threshold: 0.0, format: "human", output: nil, baseline: nil)
|
|
25
25
|
rendered =
|
|
26
26
|
if format == "json"
|
|
27
|
-
json_report
|
|
27
|
+
json_report(baseline)
|
|
28
28
|
else
|
|
29
29
|
sio = StringIO.new
|
|
30
30
|
human_report(sio, err, threshold)
|
|
31
|
+
baseline_section(sio, baseline) if baseline
|
|
31
32
|
sio.string
|
|
32
33
|
end
|
|
33
34
|
|
|
@@ -53,6 +54,7 @@ module Mutineer
|
|
|
53
54
|
summary(out)
|
|
54
55
|
out.puts
|
|
55
56
|
score_line(out, err)
|
|
57
|
+
per_source(out)
|
|
56
58
|
|
|
57
59
|
survivors(out)
|
|
58
60
|
verdict(out, threshold) if threshold && threshold.positive?
|
|
@@ -73,7 +75,7 @@ module Mutineer
|
|
|
73
75
|
# Canonical machine-readable schema (KTD7). survivors/no_coverage are sorted
|
|
74
76
|
# by (file, line, operator) so output is byte-stable regardless of --jobs
|
|
75
77
|
# worker finish order (R22).
|
|
76
|
-
def json_report
|
|
78
|
+
def json_report(baseline = nil)
|
|
77
79
|
killed = @agg.killed_count
|
|
78
80
|
survived = @agg.survived_count
|
|
79
81
|
# C8: null (not 0.0) on an empty denominator, matching the nil-vs-0.0
|
|
@@ -82,27 +84,70 @@ module Mutineer
|
|
|
82
84
|
score = @agg.mutation_score
|
|
83
85
|
|
|
84
86
|
doc = {
|
|
85
|
-
schema_version: "1.
|
|
87
|
+
schema_version: "1.1",
|
|
86
88
|
summary: {
|
|
87
89
|
total: @agg.total, killed: killed, survived: survived,
|
|
88
90
|
no_coverage: @agg.no_coverage_count,
|
|
91
|
+
uncapturable: @agg.uncapturable_count,
|
|
89
92
|
skipped_invalid: @agg.skipped_invalid_count,
|
|
90
93
|
errored: @agg.errored_count, timeout: @agg.timeout_count,
|
|
94
|
+
ignored: @agg.ignored_count,
|
|
91
95
|
score: score
|
|
92
96
|
},
|
|
93
97
|
survivors: @agg.surviving_mutants.map { |r| survivor_json(r) }
|
|
94
98
|
.sort_by { |h| [h[:file], h[:line], h[:operator]] },
|
|
95
99
|
no_coverage: @agg.results.select(&:no_coverage?).map { |r| no_coverage_json(r) }
|
|
96
|
-
.sort_by { |h| [h[:file], h[:line]] }
|
|
100
|
+
.sort_by { |h| [h[:file], h[:line]] },
|
|
101
|
+
# #9: same shape as no_coverage; additive key.
|
|
102
|
+
uncapturable: @agg.results.select(&:uncapturable?).map { |r| no_coverage_json(r) }
|
|
103
|
+
.sort_by { |h| [h[:file], h[:line]] },
|
|
104
|
+
# #10: equivalent mutants the user suppressed — emitted with their stable
|
|
105
|
+
# id so the user can audit what is silenced (and copy ids for survivors
|
|
106
|
+
# they want to add). Excluded from the score; never in `survivors`.
|
|
107
|
+
ignored: @agg.results.select(&:ignored?).map { |r| ignored_json(r) }
|
|
108
|
+
.sort_by { |h| [h[:file], h[:line], h[:operator]] },
|
|
109
|
+
# #11: per-source breakdown (additive; #13 consumes it). Sorted by file so
|
|
110
|
+
# output is byte-stable. Reuses AggregateResult via by_source.
|
|
111
|
+
per_source: @agg.by_source.map { |file, agg| per_source_json(file, agg) }
|
|
112
|
+
.sort_by { |h| h[:file] }
|
|
97
113
|
}
|
|
114
|
+
# #13: additive baseline-delta block, present only with --baseline. Existing
|
|
115
|
+
# consumers ignore the extra key; schema_version stays 1.1.
|
|
116
|
+
doc[:baseline] = baseline_json(baseline) if baseline
|
|
98
117
|
"#{JSON.generate(doc)}\n"
|
|
99
118
|
end
|
|
100
119
|
|
|
120
|
+
# #13: the same delta facts the human report prints, for dashboards. new_survivors
|
|
121
|
+
# reuse the ignored_json shape (subject/file/line/operator/token/id) and sort
|
|
122
|
+
# byte-stably so output doesn't depend on --jobs finish order.
|
|
123
|
+
def baseline_json(delta)
|
|
124
|
+
{
|
|
125
|
+
regressed: delta.regressed,
|
|
126
|
+
score_before: delta.score_before,
|
|
127
|
+
score_after: delta.score_after,
|
|
128
|
+
score_dropped: delta.score_drop,
|
|
129
|
+
new_survivors: delta.new_survivors.map { |r| ignored_json(r) }
|
|
130
|
+
.sort_by { |h| [h[:file], h[:line], h[:operator]] },
|
|
131
|
+
fixed_survivors: delta.fixed_survivors.map do |h|
|
|
132
|
+
{ subject: h["subject"], file: h["file"], line: h["line"],
|
|
133
|
+
operator: h["operator"], id: h["id"] }
|
|
134
|
+
end.sort_by { |h| [h[:file].to_s, h[:line].to_i, h[:operator].to_s] }
|
|
135
|
+
}
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
def per_source_json(file, agg)
|
|
139
|
+
{
|
|
140
|
+
file: file, total: agg.total,
|
|
141
|
+
killed: agg.killed_count, survived: agg.survived_count,
|
|
142
|
+
no_coverage: agg.no_coverage_count, score: agg.mutation_score
|
|
143
|
+
}
|
|
144
|
+
end
|
|
145
|
+
|
|
101
146
|
def survivor_json(result)
|
|
102
147
|
m = result.mutation
|
|
103
148
|
file = result.subject.file
|
|
104
149
|
source = @source_map[file] || File.read(file)
|
|
105
|
-
start_line, original_block, mutated_block, = diff_for(m, source)
|
|
150
|
+
start_line, original_block, mutated_block, token = diff_for(m, source)
|
|
106
151
|
minus = original_block.each_line.map { |l| "-#{l.chomp}" }.join("\n")
|
|
107
152
|
plus = mutated_block.each_line.map { |l| "+#{l.chomp}" }.join("\n")
|
|
108
153
|
{
|
|
@@ -110,10 +155,30 @@ module Mutineer
|
|
|
110
155
|
file: file,
|
|
111
156
|
line: start_line,
|
|
112
157
|
operator: m.operator.to_s,
|
|
158
|
+
# #10: the stable, copy-pasteable id (next to the human-readable token) so a
|
|
159
|
+
# user can paste it straight into .mutineer.yml `ignore:`.
|
|
160
|
+
id: result.id,
|
|
161
|
+
token: token,
|
|
113
162
|
diff: "--- a/#{file}\n+++ b/#{file}\n@@ -#{start_line} +#{start_line} @@\n#{minus}\n#{plus}\n"
|
|
114
163
|
}
|
|
115
164
|
end
|
|
116
165
|
|
|
166
|
+
# An entry under the JSON `ignored:` key — what the user already suppressed.
|
|
167
|
+
def ignored_json(result)
|
|
168
|
+
m = result.mutation
|
|
169
|
+
file = result.subject.file
|
|
170
|
+
source = @source_map[file] || File.read(file)
|
|
171
|
+
start_line, _orig, _mut, token = diff_for(m, source)
|
|
172
|
+
{
|
|
173
|
+
subject: result.subject.qualified_name,
|
|
174
|
+
file: file,
|
|
175
|
+
line: start_line,
|
|
176
|
+
operator: m.operator.to_s,
|
|
177
|
+
token: token,
|
|
178
|
+
id: result.id
|
|
179
|
+
}
|
|
180
|
+
end
|
|
181
|
+
|
|
117
182
|
# Builds a line-aligned diff for a mutation whose byte range may span several
|
|
118
183
|
# lines (e.g. statement-removal of a multi-line statement). Returns the
|
|
119
184
|
# mutation's 1-based start line, the full original line-block it touches, the
|
|
@@ -151,12 +216,18 @@ module Mutineer
|
|
|
151
216
|
out.puts format("Survived: %-6d No coverage: %d", @agg.survived_count, @agg.no_coverage_count)
|
|
152
217
|
out.puts format("Skipped: %-6d Errored: %d", @agg.skipped_invalid_count,
|
|
153
218
|
@agg.errored_count + @agg.timeout_count)
|
|
219
|
+
# #9: a broken harness, not a coverage gap — report it distinctly from No coverage.
|
|
220
|
+
out.puts format("Uncapturable: %-6d (tests failed to run)", @agg.uncapturable_count)
|
|
221
|
+
# #10: equivalent mutants the user suppressed; excluded from the denominator.
|
|
222
|
+
out.puts format("Ignored: %-6d (equivalent, suppressed)", @agg.ignored_count)
|
|
154
223
|
end
|
|
155
224
|
|
|
156
225
|
def score_line(out, err)
|
|
157
226
|
score = @agg.mutation_score
|
|
158
|
-
excluded = "#{@agg.no_coverage_count} no-coverage, #{@agg.
|
|
159
|
-
"#{@agg.
|
|
227
|
+
excluded = "#{@agg.no_coverage_count} no-coverage, #{@agg.uncapturable_count} uncapturable, " \
|
|
228
|
+
"#{@agg.skipped_invalid_count} skipped, " \
|
|
229
|
+
"#{@agg.errored_count + @agg.timeout_count} errored, " \
|
|
230
|
+
"#{@agg.ignored_count} ignored excluded"
|
|
160
231
|
if score.nil?
|
|
161
232
|
out.puts "Mutation score: N/A (no covered mutants)"
|
|
162
233
|
err.puts "[mutineer] no covered mutations; mutation score is N/A and the threshold check is skipped."
|
|
@@ -165,6 +236,44 @@ module Mutineer
|
|
|
165
236
|
end
|
|
166
237
|
end
|
|
167
238
|
|
|
239
|
+
# #11: one line per source after the global summary, so a multi-source run
|
|
240
|
+
# shows which file is weak. Omitted for a single-source run — the global
|
|
241
|
+
# summary already says everything (ponytail: no redundant one-line block).
|
|
242
|
+
def per_source(out)
|
|
243
|
+
sources = @agg.by_source
|
|
244
|
+
return if sources.size <= 1
|
|
245
|
+
|
|
246
|
+
out.puts
|
|
247
|
+
out.puts "Per-source"
|
|
248
|
+
out.puts "----------"
|
|
249
|
+
sources.sort.each do |file, agg|
|
|
250
|
+
score = agg.mutation_score
|
|
251
|
+
out.puts format("%s %s (%d killed / %d survived / %d no-cov)",
|
|
252
|
+
file, score.nil? ? "N/A" : "#{score}%",
|
|
253
|
+
agg.killed_count, agg.survived_count, agg.no_coverage_count)
|
|
254
|
+
end
|
|
255
|
+
end
|
|
256
|
+
|
|
257
|
+
# #13: the --baseline delta, appended after the normal report. Names every NEW
|
|
258
|
+
# survivor (subject (file:line) operator) and the score delta when it dropped,
|
|
259
|
+
# then a one-line REGRESSION/OK verdict so CI logs show which gate fired.
|
|
260
|
+
def baseline_section(out, delta)
|
|
261
|
+
out.puts
|
|
262
|
+
out.puts "Baseline comparison"
|
|
263
|
+
out.puts "-------------------"
|
|
264
|
+
out.puts "killed #{@agg.killed_count}, #{delta.new_survivors.size} new survivors vs baseline"
|
|
265
|
+
delta.new_survivors
|
|
266
|
+
.sort_by { |r| [r.subject.file, r.mutation.start_offset] }
|
|
267
|
+
.each do |r|
|
|
268
|
+
file = r.subject.file
|
|
269
|
+
source = @source_map[file] || File.read(file)
|
|
270
|
+
line, = diff_for(r.mutation, source)
|
|
271
|
+
out.puts " + #{r.subject.qualified_name} (#{file}:#{line}) #{r.mutation.operator}"
|
|
272
|
+
end
|
|
273
|
+
out.puts "score dropped #{delta.score_before}% -> #{delta.score_after}%" if delta.score_drop
|
|
274
|
+
out.puts(delta.regressed ? "REGRESSION vs baseline" : "OK: no regression vs baseline")
|
|
275
|
+
end
|
|
276
|
+
|
|
168
277
|
def survivors(out)
|
|
169
278
|
mutants = @agg.surviving_mutants
|
|
170
279
|
return if mutants.empty?
|
data/lib/mutineer/result.rb
CHANGED
|
@@ -1,42 +1,57 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module Mutineer
|
|
4
|
-
# Immutable outcome of running one mutant.
|
|
5
|
-
# killed
|
|
6
|
-
# survived
|
|
7
|
-
# error
|
|
8
|
-
# timeout
|
|
9
|
-
# skipped
|
|
10
|
-
# no_coverage
|
|
4
|
+
# Immutable outcome of running one mutant. Seven distinct states:
|
|
5
|
+
# killed — a test failed/errored, so the mutation was caught.
|
|
6
|
+
# survived — every test passed, so the mutation went undetected.
|
|
7
|
+
# error — the child crashed (unhandled exception): exit status 2.
|
|
8
|
+
# timeout — the parent SIGKILLed a child that overran its wall clock.
|
|
9
|
+
# skipped — the mutated source failed to re-parse (invalid); no fork.
|
|
10
|
+
# no_coverage — no test exercises the mutated line; not run, not scored.
|
|
11
|
+
# uncapturable — the line's would-be covering test errored during capture (#9),
|
|
12
|
+
# so coverage was lost. Excluded from the denominator exactly
|
|
13
|
+
# like no_coverage, but reported separately: it signals a broken
|
|
14
|
+
# harness (a test that failed to run), not a genuine coverage gap.
|
|
15
|
+
# ignored — a known-equivalent mutant the user suppressed (#10), via an
|
|
16
|
+
# inline `# mutineer:disable-line` comment or a `.mutineer.yml`
|
|
17
|
+
# `ignore:` id. A pre-fork classification (never run); excluded
|
|
18
|
+
# from the denominator so a strong file can reach 100%.
|
|
11
19
|
#
|
|
12
20
|
# `error` and `skipped` are deliberately distinct: skipped is a pre-fork
|
|
13
21
|
# validity failure (counted separately by the reporter), error is a runtime
|
|
14
|
-
# crash. Never conflate them via `details` string parsing. `no_coverage`
|
|
15
|
-
# pre-fork selection
|
|
22
|
+
# crash. Never conflate them via `details` string parsing. `no_coverage` and
|
|
23
|
+
# `uncapturable` are pre-fork selection results (M3/#9): both excluded from the
|
|
24
|
+
# score denominator.
|
|
16
25
|
#
|
|
17
|
-
# `subject` and `
|
|
18
|
-
# (which only know the outcome); the orchestrator attaches them afterwards
|
|
19
|
-
# `result.with(subject:, mutation:)` so the Reporter can render survivor
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
def self.
|
|
23
|
-
def self.
|
|
24
|
-
def self.
|
|
25
|
-
def self.
|
|
26
|
-
def self.
|
|
26
|
+
# `subject`, `mutation`, and `id` are nil when the Result is built by Isolation/
|
|
27
|
+
# Runner (which only know the outcome); the orchestrator attaches them afterwards
|
|
28
|
+
# via `result.with(subject:, mutation:, id:)` so the Reporter can render survivor
|
|
29
|
+
# diffs and emit the stable id. `id` is the content-based MutantId (#10).
|
|
30
|
+
Result = Data.define(:status, :details, :subject, :mutation, :id) do
|
|
31
|
+
def self.killed = new(status: :killed, details: nil, subject: nil, mutation: nil, id: nil)
|
|
32
|
+
def self.survived = new(status: :survived, details: nil, subject: nil, mutation: nil, id: nil)
|
|
33
|
+
def self.error(details = nil) = new(status: :error, details: details, subject: nil, mutation: nil, id: nil)
|
|
34
|
+
def self.timeout = new(status: :timeout, details: nil, subject: nil, mutation: nil, id: nil)
|
|
35
|
+
def self.skipped(details = nil) = new(status: :skipped, details: details, subject: nil, mutation: nil, id: nil)
|
|
36
|
+
def self.no_coverage = new(status: :no_coverage, details: nil, subject: nil, mutation: nil, id: nil)
|
|
37
|
+
def self.uncapturable = new(status: :uncapturable, details: nil, subject: nil, mutation: nil, id: nil)
|
|
38
|
+
def self.ignored = new(status: :ignored, details: nil, subject: nil, mutation: nil, id: nil)
|
|
27
39
|
|
|
28
|
-
def killed?
|
|
29
|
-
def survived?
|
|
30
|
-
def error?
|
|
31
|
-
def timeout?
|
|
32
|
-
def skipped?
|
|
33
|
-
def no_coverage?
|
|
40
|
+
def killed? = status == :killed
|
|
41
|
+
def survived? = status == :survived
|
|
42
|
+
def error? = status == :error
|
|
43
|
+
def timeout? = status == :timeout
|
|
44
|
+
def skipped? = status == :skipped
|
|
45
|
+
def no_coverage? = status == :no_coverage
|
|
46
|
+
def uncapturable? = status == :uncapturable
|
|
47
|
+
def ignored? = status == :ignored
|
|
34
48
|
end
|
|
35
49
|
|
|
36
50
|
# Aggregates a flat list of Results into counts, the mutation score, and the
|
|
37
51
|
# surviving-mutant list. The score denominator is killed + survived ONLY
|
|
38
|
-
# (KTD-4): no-coverage, skipped (invalid), errored,
|
|
39
|
-
#
|
|
52
|
+
# (KTD-4): no-coverage, uncapturable, skipped (invalid), errored, timeout, and
|
|
53
|
+
# ignored (#10 equivalent-mutant suppression) are each excluded and surfaced
|
|
54
|
+
# separately — so suppressing every survivor reaches 100%. An empty denominator yields a nil score
|
|
40
55
|
# (rendered "N/A"), never 0.0 — distinguishing "no testable mutants" from
|
|
41
56
|
# "0% killed".
|
|
42
57
|
class AggregateResult
|
|
@@ -50,9 +65,11 @@ module Mutineer
|
|
|
50
65
|
def killed_count = count(:killed)
|
|
51
66
|
def survived_count = count(:survived)
|
|
52
67
|
def no_coverage_count = count(:no_coverage)
|
|
68
|
+
def uncapturable_count = count(:uncapturable)
|
|
53
69
|
def skipped_invalid_count = count(:skipped)
|
|
54
70
|
def errored_count = count(:error)
|
|
55
71
|
def timeout_count = count(:timeout)
|
|
72
|
+
def ignored_count = count(:ignored)
|
|
56
73
|
|
|
57
74
|
# Every generated, classified mutation. NOT the score denominator.
|
|
58
75
|
def total = @results.size
|
|
@@ -70,6 +87,17 @@ module Mutineer
|
|
|
70
87
|
|
|
71
88
|
def surviving_mutants = @results.select(&:survived?)
|
|
72
89
|
|
|
90
|
+
# #11: split into { source_file => AggregateResult } so the Reporter (per-source
|
|
91
|
+
# breakdown) and #13 (per-source roll-up / baseline diff) share one shape with
|
|
92
|
+
# all the score/count methods. After Runner.execute every result carries a
|
|
93
|
+
# subject, so the grouping is total; bare results (no subject, only in unit
|
|
94
|
+
# tests) are skipped so file keys stay sortable strings.
|
|
95
|
+
def by_source
|
|
96
|
+
@results.select { |r| r.subject }
|
|
97
|
+
.group_by { |r| r.subject.file }
|
|
98
|
+
.transform_values { |rs| AggregateResult.new(rs) }
|
|
99
|
+
end
|
|
100
|
+
|
|
73
101
|
private
|
|
74
102
|
|
|
75
103
|
def count(status) = (@by_status[status] || []).size
|
data/lib/mutineer/runner.rb
CHANGED
|
@@ -10,6 +10,8 @@ require_relative "coverage_map"
|
|
|
10
10
|
require_relative "changed_lines"
|
|
11
11
|
require_relative "mutator_registry"
|
|
12
12
|
require_relative "worker_pool"
|
|
13
|
+
require_relative "mutant_id"
|
|
14
|
+
require "set"
|
|
13
15
|
|
|
14
16
|
module Mutineer
|
|
15
17
|
# Orchestrates one mutation end-to-end: apply it textually, validate the
|
|
@@ -70,7 +72,8 @@ module Mutineer
|
|
|
70
72
|
source_paths: config.sources, test_paths: config.tests,
|
|
71
73
|
cache_dir: config.cache_dir, project_root: config.project_root,
|
|
72
74
|
load_paths: config.load_paths, framework: config.framework,
|
|
73
|
-
boot_path: File.expand_path(config.boot, config.project_root)
|
|
75
|
+
boot_path: File.expand_path(config.boot, config.project_root),
|
|
76
|
+
verbose: config.verbose
|
|
74
77
|
).build_via_fork(rails: config.rails)
|
|
75
78
|
else
|
|
76
79
|
coverage_map = CoverageMap.new(
|
|
@@ -81,13 +84,28 @@ module Mutineer
|
|
|
81
84
|
end
|
|
82
85
|
|
|
83
86
|
# Collect every (subject, mutation) up front so the pool can fan them out.
|
|
87
|
+
# #10: a mutant the user marked known-equivalent (inline disable-line comment
|
|
88
|
+
# or .mutineer.yml ignore id) is classified :ignored here and NEVER forked —
|
|
89
|
+
# it is removed from the killed+survived denominator so a strong file reaches
|
|
90
|
+
# 100%. The stable id is computed per subject (occurrence needs the full list)
|
|
91
|
+
# and carried on every job so the parent can reattach it after the run.
|
|
84
92
|
source_map = {}
|
|
93
|
+
disabled_map = {}
|
|
94
|
+
ignore_set = config.ignore.to_set
|
|
85
95
|
jobs = []
|
|
96
|
+
ignored_results = []
|
|
86
97
|
Project.discover(config.sources, only: config.only).each do |subject|
|
|
87
98
|
source = (source_map[subject.file] ||= File.read(subject.file))
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
99
|
+
disabled = (disabled_map[subject.file] ||= suppress_map(source))
|
|
100
|
+
mutations = operator_classes.flat_map { |klass| klass.new.mutations_for(subject, source) }
|
|
101
|
+
ids = MutantId.for_subject(subject, source, mutations)
|
|
102
|
+
mutations.each_with_index do |mutation, i|
|
|
103
|
+
id = ids[i]
|
|
104
|
+
line = source.byteslice(0, mutation.start_offset).count("\n") + 1
|
|
105
|
+
if suppressed?(mutation.operator, line, id, disabled, ignore_set)
|
|
106
|
+
ignored_results << Result.ignored.with(subject: subject, mutation: mutation, id: id)
|
|
107
|
+
else
|
|
108
|
+
jobs << [subject, mutation, id]
|
|
91
109
|
end
|
|
92
110
|
end
|
|
93
111
|
end
|
|
@@ -111,13 +129,42 @@ module Mutineer
|
|
|
111
129
|
subject: subject, strategy: strategy, rails: config.rails, framework: framework)
|
|
112
130
|
end
|
|
113
131
|
# The bare Results carry only status (Subjects hold live AST nodes that
|
|
114
|
-
# do not marshal); reattach subject+mutation in the parent, in order.
|
|
115
|
-
bare.each_with_index.map { |r, i| r.with(subject: jobs[i][0], mutation: jobs[i][1]) }
|
|
132
|
+
# do not marshal); reattach subject+mutation+id in the parent, in order.
|
|
133
|
+
bare.each_with_index.map { |r, i| r.with(subject: jobs[i][0], mutation: jobs[i][1], id: jobs[i][2]) }
|
|
116
134
|
ensure
|
|
117
135
|
sweep_orphans(source_dirs)
|
|
118
136
|
end
|
|
119
137
|
|
|
120
|
-
[AggregateResult.new(results), source_map]
|
|
138
|
+
[AggregateResult.new(results + ignored_results), source_map]
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
# Scan a source once into { line_number => :all | Set[operator_syms] } from
|
|
142
|
+
# inline `# mutineer:disable-line [ops]` markers (RuboCop semantics: the marker
|
|
143
|
+
# sits on the same physical line as the code it silences). A bare marker
|
|
144
|
+
# disables every operator on that line; `disable-line a, b` only the listed
|
|
145
|
+
# operators. Block-form disable/enable ranges are intentionally not supported.
|
|
146
|
+
def self.suppress_map(source)
|
|
147
|
+
map = {}
|
|
148
|
+
source.each_line.with_index(1) do |text, line|
|
|
149
|
+
next unless (m = text.match(/#\s*mutineer:disable-line(?:\s+([\w,\s]+))?/))
|
|
150
|
+
|
|
151
|
+
ops = m[1]
|
|
152
|
+
map[line] = ops ? ops.split(",").map { |o| o.strip.to_sym }.reject(&:empty?).to_set : :all
|
|
153
|
+
end
|
|
154
|
+
map
|
|
155
|
+
end
|
|
156
|
+
|
|
157
|
+
# True when this mutant is suppressed: its line bears a disable-line marker
|
|
158
|
+
# (bare, or scoped to its operator), OR its stable id is in the config ignore
|
|
159
|
+
# list. Checked at job-build time so a suppressed mutant is never forked.
|
|
160
|
+
def self.suppressed?(operator, line, id, disabled, ignore_set)
|
|
161
|
+
return true if ignore_set.include?(id)
|
|
162
|
+
|
|
163
|
+
case (entry = disabled[line])
|
|
164
|
+
when :all then true
|
|
165
|
+
when Set then entry.include?(operator)
|
|
166
|
+
else false
|
|
167
|
+
end
|
|
121
168
|
end
|
|
122
169
|
|
|
123
170
|
# --since: keep only jobs whose mutation lands on a line changed since the git
|
|
@@ -189,7 +236,11 @@ module Mutineer
|
|
|
189
236
|
# covering test files run in the child.
|
|
190
237
|
line = source.byteslice(0, mutation.start_offset).count("\n") + 1
|
|
191
238
|
chosen = coverage_map.tests_for(source_file, line)
|
|
192
|
-
|
|
239
|
+
# #9: distinguish a genuine coverage gap from a line whose would-be test
|
|
240
|
+
# errored during capture (coverage lost) — the latter is :uncapturable.
|
|
241
|
+
if chosen.empty?
|
|
242
|
+
return coverage_map.uncapturable_source?(source_file) ? Result.uncapturable : Result.no_coverage
|
|
243
|
+
end
|
|
193
244
|
|
|
194
245
|
abs_tests = chosen.map { |t| File.expand_path(t, coverage_map.project_root) }
|
|
195
246
|
|
|
@@ -209,10 +260,28 @@ module Mutineer
|
|
|
209
260
|
def self.reconnect_active_record
|
|
210
261
|
return unless defined?(ActiveRecord::Base)
|
|
211
262
|
|
|
212
|
-
ActiveRecord::Base
|
|
263
|
+
base = ActiveRecord::Base
|
|
264
|
+
# #8: clearing connections here drops an open transactional-fixture
|
|
265
|
+
# transaction, so the test loses its fixture rows and fails. Skip the clear
|
|
266
|
+
# when a transaction is open; otherwise clear (v0.2 per-fork write-safety).
|
|
267
|
+
return if fixture_transaction_open?(base)
|
|
268
|
+
|
|
269
|
+
base.connection_handler.clear_all_connections!
|
|
213
270
|
rescue StandardError
|
|
214
271
|
nil
|
|
215
272
|
end
|
|
216
273
|
private_class_method :reconnect_active_record
|
|
274
|
+
|
|
275
|
+
# Pure, injectable predicate: true when a transactional-fixture transaction is
|
|
276
|
+
# already open on the connection. Keys off open_transactions (KTD-2) so it is
|
|
277
|
+
# correct whenever the transaction exists, regardless of when it opened. Any
|
|
278
|
+
# probe error degrades safe to false -> caller clears (existing behaviour).
|
|
279
|
+
def self.fixture_transaction_open?(base)
|
|
280
|
+
pool = base.connection_pool
|
|
281
|
+
pool.active_connection? && base.connection.open_transactions.positive?
|
|
282
|
+
rescue StandardError
|
|
283
|
+
false
|
|
284
|
+
end
|
|
285
|
+
private_class_method :fixture_transaction_open?
|
|
217
286
|
end
|
|
218
287
|
end
|
data/lib/mutineer/version.rb
CHANGED
data/lib/mutineer.rb
CHANGED
|
@@ -6,6 +6,7 @@ require_relative "mutineer/parser"
|
|
|
6
6
|
require_relative "mutineer/subject"
|
|
7
7
|
require_relative "mutineer/mutation"
|
|
8
8
|
require_relative "mutineer/project"
|
|
9
|
+
require_relative "mutineer/pairing"
|
|
9
10
|
require_relative "mutineer/result"
|
|
10
11
|
require_relative "mutineer/coverage_map"
|
|
11
12
|
require_relative "mutineer/changed_lines"
|
|
@@ -25,6 +26,7 @@ require_relative "mutineer/mutator_registry"
|
|
|
25
26
|
require_relative "mutineer/worker_pool"
|
|
26
27
|
require_relative "mutineer/runner"
|
|
27
28
|
require_relative "mutineer/reporter"
|
|
29
|
+
require_relative "mutineer/baseline"
|
|
28
30
|
require_relative "mutineer/cli"
|
|
29
31
|
|
|
30
32
|
module Mutineer
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutineer
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- David Teren
|
|
@@ -52,12 +52,14 @@ files:
|
|
|
52
52
|
- README.md
|
|
53
53
|
- bin/mutineer
|
|
54
54
|
- lib/mutineer.rb
|
|
55
|
+
- lib/mutineer/baseline.rb
|
|
55
56
|
- lib/mutineer/changed_lines.rb
|
|
56
57
|
- lib/mutineer/cli.rb
|
|
57
58
|
- lib/mutineer/config.rb
|
|
58
59
|
- lib/mutineer/coverage_map.rb
|
|
59
60
|
- lib/mutineer/isolation.rb
|
|
60
61
|
- lib/mutineer/minitest_integration.rb
|
|
62
|
+
- lib/mutineer/mutant_id.rb
|
|
61
63
|
- lib/mutineer/mutation.rb
|
|
62
64
|
- lib/mutineer/mutator_registry.rb
|
|
63
65
|
- lib/mutineer/mutators/arithmetic.rb
|
|
@@ -69,6 +71,7 @@ files:
|
|
|
69
71
|
- lib/mutineer/mutators/literal_mutation.rb
|
|
70
72
|
- lib/mutineer/mutators/return_nil.rb
|
|
71
73
|
- lib/mutineer/mutators/statement_removal.rb
|
|
74
|
+
- lib/mutineer/pairing.rb
|
|
72
75
|
- lib/mutineer/parser.rb
|
|
73
76
|
- lib/mutineer/project.rb
|
|
74
77
|
- lib/mutineer/reporter.rb
|