mutineer 0.6.2 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +59 -0
- data/README.md +63 -2
- data/lib/mutineer/baseline.rb +86 -0
- data/lib/mutineer/changed_lines.rb +29 -12
- data/lib/mutineer/cli.rb +137 -9
- data/lib/mutineer/config.rb +30 -2
- data/lib/mutineer/coverage_map.rb +135 -11
- data/lib/mutineer/isolation.rb +89 -40
- data/lib/mutineer/minitest_integration.rb +13 -9
- data/lib/mutineer/mutant_id.rb +66 -0
- data/lib/mutineer/mutation.rb +12 -6
- data/lib/mutineer/mutator_registry.rb +21 -3
- data/lib/mutineer/mutators/arithmetic.rb +8 -2
- data/lib/mutineer/mutators/base.rb +11 -3
- data/lib/mutineer/mutators/boolean_connector.rb +15 -5
- data/lib/mutineer/mutators/boolean_literal.rb +19 -6
- data/lib/mutineer/mutators/comparison.rb +7 -8
- data/lib/mutineer/mutators/condition_negation.rb +14 -5
- data/lib/mutineer/mutators/literal_mutation.rb +15 -4
- data/lib/mutineer/mutators/return_nil.rb +22 -7
- data/lib/mutineer/mutators/statement_removal.rb +6 -9
- data/lib/mutineer/pairing.rb +86 -0
- data/lib/mutineer/parser.rb +17 -7
- data/lib/mutineer/project.rb +73 -2
- data/lib/mutineer/reporter.rb +172 -8
- data/lib/mutineer/result.rb +128 -32
- data/lib/mutineer/runner.rb +101 -9
- data/lib/mutineer/subject.rb +10 -6
- data/lib/mutineer/test_runners/minitest.rb +5 -4
- data/lib/mutineer/test_runners/rspec.rb +10 -17
- data/lib/mutineer/test_runners.rb +9 -2
- data/lib/mutineer/version.rb +2 -1
- data/lib/mutineer/worker_pool.rb +34 -21
- data/lib/mutineer.rb +3 -0
- metadata +18 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bfadc70b171d8778c8afa95021c2989ad33ab2880200a22886fec274e358efcc
|
|
4
|
+
data.tar.gz: f6f6c91f6e5fffc0a2d05d93b0981df165d6d5b2df1e1434949c7dea1c0f70a6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 01de0a56c54a7b85fd98d92831681a258a4420b3e5b3d34ce932811259a776764c888aae13a8975632c66851798f9b4a53f343864518fd8fe608e7dddff20e07
|
|
7
|
+
data.tar.gz: 63c802e363a3f87a8ecd43a5bf726d2ce23af2aadf890b87b7ad6616076e9f2461876c0af5faa70c83caf93027fd9e8754bf6866c5ab25e7d2a51bd51f9f3d4c
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,62 @@ All notable changes to this project are documented here. The format is based on
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/), and this project adheres to
|
|
5
5
|
[Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [0.8.0] - 2026-06-30
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
- **Singleton methods are now mutated** (#20) — `class << self` and
|
|
11
|
+
`module_function` methods were discovered but applied to the instance scope, so
|
|
12
|
+
the mutant never ran on the singleton the caller dispatches to; every such
|
|
13
|
+
mutant falsely survived and the file read a false 0%. `module_function` methods
|
|
14
|
+
are now discovered as singletons, and the redefine strategy re-opens
|
|
15
|
+
`class << self` so the mutation lands on the called method. (Scores for
|
|
16
|
+
singleton-heavy files will rise to their true values.)
|
|
17
|
+
- **Fork-capture failures are diagnosable** (#19, part 1) — capture/worker pipes
|
|
18
|
+
are `binmode` (a binary Marshal payload could otherwise be lost to an encoding
|
|
19
|
+
error and swallowed), and a child that dies without writing now reports how it
|
|
20
|
+
died (exit status / signal) instead of a silent "produced no result". `--verbose`
|
|
21
|
+
always surfaces a real reason now. (The residual write-heavy capture failure
|
|
22
|
+
on some real Rails suites remains under investigation — now diagnosable.)
|
|
23
|
+
|
|
24
|
+
## [0.7.1] - 2026-06-30
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
- **GitHub Action** (`action.yml`, composite) wrapping the CLI for CI — gate a PR
|
|
28
|
+
on new survivors / score drop with `sources`, `since`, `baseline`, `threshold`,
|
|
29
|
+
etc. Inputs are passed via `env` (no `${{ }}` interpolation into the run script)
|
|
30
|
+
for command-injection safety.
|
|
31
|
+
- Docs site (GitHub Pages) with Open Graph / Twitter Card share image; YARD doc
|
|
32
|
+
comments across the library.
|
|
33
|
+
|
|
34
|
+
## [0.7.0] - 2026-06-30
|
|
35
|
+
|
|
36
|
+
Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
- **Equivalent-mutant suppression** (#10) — inline `# mutineer:disable-line [ops]`
|
|
40
|
+
and a `.mutineer.yml` `ignore:` list keyed on a stable, offset-free mutant id;
|
|
41
|
+
suppressed mutants are excluded from the score (100% reachable). The stable id
|
|
42
|
+
(and readable token) is emitted per survivor in JSON.
|
|
43
|
+
- **Source→test auto-pairing + multi-source runs** (#11) — pass a directory or
|
|
44
|
+
several sources with `--test` omitted; tests are inferred by convention
|
|
45
|
+
(`app/`,`lib/` → `test/…_test.rb` / `spec/…_spec.rb`) and run under one boot,
|
|
46
|
+
with per-source results (human + JSON `per_source`).
|
|
47
|
+
- **`--baseline <file.json>` CI gating** (#13) — diff against a prior run by stable
|
|
48
|
+
id; exit 1 on new survivors or a score drop (with `--baseline-epsilon`), naming
|
|
49
|
+
what regressed. Combines with `--threshold` via max exit code.
|
|
50
|
+
- **`--verbose`/`--debug`** (#8) — surface the real error when a fork capture fails.
|
|
51
|
+
- **`:uncapturable` status** (#9) — distinct from `no_coverage`; reported separately
|
|
52
|
+
("tests failed to run" vs "genuinely uncovered"). Both excluded from the score.
|
|
53
|
+
|
|
54
|
+
### Fixed
|
|
55
|
+
- **Fork capture no longer drops fixture transactions** (#8) — `reconnect` skips
|
|
56
|
+
`clear_all_connections!` when a fixture transaction is open, and stops swallowing
|
|
57
|
+
the child error; write-heavy Rails tests are mutation-testable again.
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
- JSON `schema_version` → `1.1` (additive: survivor `id`/`token`, `ignored`,
|
|
61
|
+
`uncapturable`, `per_source`).
|
|
62
|
+
|
|
7
63
|
## [0.6.2] - 2026-06-29
|
|
8
64
|
|
|
9
65
|
### Fixed
|
|
@@ -100,6 +156,9 @@ All notable changes to this project are documented here. The format is based on
|
|
|
100
156
|
- `.mutineer.yml` configuration (CLI > config > default precedence).
|
|
101
157
|
- Byte-correct source handling for multibyte (UTF-8) sources.
|
|
102
158
|
|
|
159
|
+
[0.8.0]: https://github.com/davidteren/mutineer/releases/tag/v0.8.0
|
|
160
|
+
[0.7.1]: https://github.com/davidteren/mutineer/releases/tag/v0.7.1
|
|
161
|
+
[0.7.0]: https://github.com/davidteren/mutineer/releases/tag/v0.7.0
|
|
103
162
|
[0.6.2]: https://github.com/davidteren/mutineer/releases/tag/v0.6.2
|
|
104
163
|
[0.6.1]: https://github.com/davidteren/mutineer/releases/tag/v0.6.1
|
|
105
164
|
[0.6.0]: https://github.com/davidteren/mutineer/releases/tag/v0.6.0
|
data/README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Mutineer
|
|
2
2
|
|
|
3
|
-
[](https://rubygems.org/gems/mutineer)
|
|
4
|
+
[](https://github.com/marketplace/actions/mutineer-ruby)
|
|
5
|
+
[](https://socket.dev/rubygems/package/mutineer/overview/0.7.1?platform=ruby)
|
|
4
6
|
|
|
5
7
|
A clean-room mutation-testing tool for Ruby. Mutineer mutates your source one
|
|
6
8
|
change at a time, runs your test suite (Minitest or RSpec) against each mutant, and reports the
|
|
@@ -49,7 +51,10 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
49
51
|
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
50
52
|
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
51
53
|
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`) — ideal for PR CI |
|
|
52
|
-
| `--
|
|
54
|
+
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (see [CI](#ci-gating)) |
|
|
55
|
+
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
56
|
+
| `--jobs N` | Parallel worker count (default: processor count; `1` under `--rails`) |
|
|
57
|
+
| `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
|
|
53
58
|
| `--strategy NAME` | Mutation application: `reload` whole-file (default) or `redefine` surgical (`7a`/`7b` accepted as deprecated aliases) |
|
|
54
59
|
| `--format human\|json` | Report format (default: human) |
|
|
55
60
|
| `--output FILE` | Write the report to FILE instead of stdout |
|
|
@@ -96,6 +101,62 @@ Add Mutineer to your Gemfile's test group:
|
|
|
96
101
|
gem "mutineer", group: :test, require: false
|
|
97
102
|
```
|
|
98
103
|
|
|
104
|
+
## Suppressing equivalent mutants
|
|
105
|
+
|
|
106
|
+
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
107
|
+
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
108
|
+
|
|
109
|
+
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`).
|
|
110
|
+
- **Config:** a `.mutineer.yml` `ignore:` list of stable mutant ids. Each survivor's
|
|
111
|
+
`id` is printed in the JSON report, so copy it straight into `ignore:`.
|
|
112
|
+
|
|
113
|
+
Suppressed mutants are excluded from the score (so 100% becomes reachable).
|
|
114
|
+
|
|
115
|
+
## CI gating
|
|
116
|
+
|
|
117
|
+
Store a JSON run as a baseline, then fail the build only when a PR makes things
|
|
118
|
+
worse:
|
|
119
|
+
|
|
120
|
+
```sh
|
|
121
|
+
mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
`--baseline` reports which survivors are new (by stable id) and any score drop. It
|
|
125
|
+
combines with `--threshold` (the worse of the two sets the exit code). Pass a
|
|
126
|
+
directory (or several sources) to audit a whole layer in one boot — tests are
|
|
127
|
+
auto-paired by convention and the report breaks down per source.
|
|
128
|
+
|
|
129
|
+
### GitHub Action
|
|
130
|
+
|
|
131
|
+
This repo ships a composite action (`action.yml`) that wraps the CLI for CI:
|
|
132
|
+
|
|
133
|
+
```yaml
|
|
134
|
+
- uses: actions/checkout@v4
|
|
135
|
+
with: { fetch-depth: 0 } # --since needs full history
|
|
136
|
+
- uses: ruby/setup-ruby@v1
|
|
137
|
+
with: { ruby-version: "3.4", bundler-cache: true }
|
|
138
|
+
- uses: davidteren/mutineer@v0
|
|
139
|
+
with:
|
|
140
|
+
sources: app/
|
|
141
|
+
since: origin/${{ github.base_ref }}
|
|
142
|
+
baseline: .mutineer/baseline.json
|
|
143
|
+
threshold: "90"
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
## For AI agents & pipelines
|
|
147
|
+
|
|
148
|
+
Mutineer is built for programmatic use — versioned JSON, stable mutant ids,
|
|
149
|
+
structured exit codes, and diff-scoped runs. See:
|
|
150
|
+
|
|
151
|
+
- **AI agents & CI recipes** — the agent inner-loop and CI-gate recipes (and how
|
|
152
|
+
to avoid infinite loops on equivalent mutants):
|
|
153
|
+
[rendered](https://davidteren.github.io/mutineer/agentic-coding.html) ·
|
|
154
|
+
[source](docs/agentic-coding.md)
|
|
155
|
+
- **JSON schema reference** — the `--format json` schema and its versioning
|
|
156
|
+
contract:
|
|
157
|
+
[rendered](https://davidteren.github.io/mutineer/json-schema.html) ·
|
|
158
|
+
[source](docs/json-schema.md)
|
|
159
|
+
|
|
99
160
|
## Configuration
|
|
100
161
|
|
|
101
162
|
Mutineer reads an optional `.mutineer.yml` from the project root (nearest one,
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require_relative "config" # for Mutineer::ConfigError
|
|
5
|
+
|
|
6
|
+
module Mutineer
|
|
7
|
+
# #13: CI baseline/delta gating. A baseline is literally a prior
|
|
8
|
+
# `mutineer run --format json` document (KTD-1) — no bespoke format to
|
|
9
|
+
# version.
|
|
10
|
+
# We diff the current run against it by the #10 stable survivor id (KTD-2):
|
|
11
|
+
# a NEW survivor (id present now, absent in the baseline) OR a score drop is
|
|
12
|
+
# a regression the CLI turns into exit 1. Pure data — stdlib `json` only, no
|
|
13
|
+
# fork, no Rails — so it's testable in isolation from a canned JSON + a hand-
|
|
14
|
+
# built AggregateResult.
|
|
15
|
+
class Baseline
|
|
16
|
+
# The verdict of diffing a current run against the baseline.
|
|
17
|
+
# new_survivors — current Result objects whose stable id is absent from
|
|
18
|
+
# the baseline (the regressions to name).
|
|
19
|
+
# fixed_survivors — baseline survivor hashes absent from the current run
|
|
20
|
+
# (informational, never gates).
|
|
21
|
+
# score_drop — current score < baseline score - epsilon. nil on
|
|
22
|
+
# either side skips the check (see #diff).
|
|
23
|
+
# regressed — any new survivors OR a score drop.
|
|
24
|
+
Delta = Data.define(:new_survivors, :fixed_survivors,
|
|
25
|
+
:score_before, :score_after, :score_drop, :regressed)
|
|
26
|
+
|
|
27
|
+
# Load a prior --format json run. Raises ConfigError (NOT exit — R8: a data
|
|
28
|
+
# class must never kill the host) on a missing/unreadable file, unparseable
|
|
29
|
+
# JSON, or a doc that isn't a baseline shape, so the CLI maps it to exit 2
|
|
30
|
+
# (usage) like every other bad-path flag.
|
|
31
|
+
#
|
|
32
|
+
# @param path [String] baseline JSON file path.
|
|
33
|
+
# @return [Mutineer::Baseline] baseline object.
|
|
34
|
+
# @raise [Mutineer::ConfigError] when the file is missing or invalid.
|
|
35
|
+
def self.load(path)
|
|
36
|
+
doc = JSON.parse(File.read(path))
|
|
37
|
+
unless doc.is_a?(Hash) && doc["schema_version"] && doc["survivors"].is_a?(Array)
|
|
38
|
+
raise ConfigError, "not a Mutineer JSON report: #{path}"
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
new(doc)
|
|
42
|
+
rescue JSON::ParserError => e
|
|
43
|
+
raise ConfigError, "invalid baseline JSON in #{path}: #{e.message}"
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
attr_reader :score
|
|
47
|
+
|
|
48
|
+
# Builds a baseline from a JSON document.
|
|
49
|
+
#
|
|
50
|
+
# The baseline retains the survivor document and score from the JSON report.
|
|
51
|
+
#
|
|
52
|
+
# @param doc [Hash] parsed JSON document.
|
|
53
|
+
def initialize(doc)
|
|
54
|
+
@survivors = doc["survivors"] || []
|
|
55
|
+
@score = doc.dig("summary", "score")
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Diff a current AggregateResult against this baseline by stable survivor id.
|
|
59
|
+
# `epsilon` tolerates float jitter on the score (default 0.0 = any drop
|
|
60
|
+
# gates).
|
|
61
|
+
#
|
|
62
|
+
# @param aggregate [Mutineer::AggregateResult] current results.
|
|
63
|
+
# @param epsilon [Float] score-drop tolerance.
|
|
64
|
+
# @return [Mutineer::Baseline::Delta] delta summary.
|
|
65
|
+
def diff(aggregate, epsilon: 0.0)
|
|
66
|
+
current = aggregate.surviving_mutants
|
|
67
|
+
current_ids = current.map(&:id)
|
|
68
|
+
baseline_ids = @survivors.map { |h| h["id"] }
|
|
69
|
+
|
|
70
|
+
new_survivors = current.reject { |r| baseline_ids.include?(r.id) }
|
|
71
|
+
fixed = @survivors.reject { |h| current_ids.include?(h["id"]) }
|
|
72
|
+
|
|
73
|
+
current_score = aggregate.mutation_score
|
|
74
|
+
# nil-score discipline (mirrors Reporter#exit_code): a score absent on
|
|
75
|
+
# either side can't be compared — skip the drop check, keep the new-
|
|
76
|
+
# survivor check.
|
|
77
|
+
score_drop = !@score.nil? && !current_score.nil? &&
|
|
78
|
+
current_score < @score - epsilon
|
|
79
|
+
|
|
80
|
+
Delta.new(new_survivors: new_survivors, fixed_survivors: fixed,
|
|
81
|
+
score_before: @score, score_after: current_score,
|
|
82
|
+
score_drop: score_drop,
|
|
83
|
+
regressed: !new_survivors.empty? || score_drop)
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
end
|
|
@@ -4,23 +4,30 @@ require "set"
|
|
|
4
4
|
require "open3"
|
|
5
5
|
|
|
6
6
|
module Mutineer
|
|
7
|
-
# Maps each source file to the set of NEW-side line numbers
|
|
8
|
-
#
|
|
9
|
-
# to only the diff (issue #2): on a PR you care whether the changed code is
|
|
10
|
-
# tested, so mutating just those lines is fast and actionable.
|
|
7
|
+
# Maps each source file to the set of NEW-side line numbers changed since a
|
|
8
|
+
# git ref.
|
|
11
9
|
#
|
|
12
|
-
#
|
|
10
|
+
# By parsing `git diff --unified=0`, this restricts mutations to only the
|
|
11
|
+
# diff (issue #2): on a PR you care whether the changed code is tested, so
|
|
12
|
+
# mutating just those lines is fast and actionable.
|
|
13
|
+
#
|
|
14
|
+
# git is an external tool, not a gem dependency — shelling out is fine. The
|
|
13
15
|
# `parse` carries the logic; `git_diff` is injectable so it stays testable
|
|
14
16
|
# without invoking git.
|
|
15
17
|
module ChangedLines
|
|
18
|
+
# Matches unified-diff hunks and captures the new-file start/count.
|
|
16
19
|
HUNK = /^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/
|
|
17
20
|
|
|
18
21
|
module_function
|
|
19
22
|
|
|
20
|
-
#
|
|
21
|
-
#
|
|
22
|
-
#
|
|
23
|
-
# deletion and
|
|
23
|
+
# Parses unified diff text into the set of NEW-side line numbers.
|
|
24
|
+
#
|
|
25
|
+
# With `--unified=0` each hunk's `+c,d` block is exactly the changed lines:
|
|
26
|
+
# `c..c+d-1`. `d` absent means 1 line; `d == 0` is a pure deletion and
|
|
27
|
+
# contributes nothing.
|
|
28
|
+
#
|
|
29
|
+
# @param diff_text [String] raw `git diff --unified=0` output.
|
|
30
|
+
# @return [Set<Integer>] changed line numbers on the new side.
|
|
24
31
|
def parse(diff_text)
|
|
25
32
|
lines = Set.new
|
|
26
33
|
diff_text.each_line do |row|
|
|
@@ -34,8 +41,13 @@ module Mutineer
|
|
|
34
41
|
lines
|
|
35
42
|
end
|
|
36
43
|
|
|
37
|
-
#
|
|
38
|
-
#
|
|
44
|
+
# Builds a per-file map of changed new-side lines.
|
|
45
|
+
#
|
|
46
|
+
# @param ref [String] git ref to diff against.
|
|
47
|
+
# @param files [Array<String>] source files to inspect.
|
|
48
|
+
# @param project_root [String] repository root for `git -C`.
|
|
49
|
+
# @param runner [#call] injectable diff producer.
|
|
50
|
+
# @return [Hash<String, Set<Integer>>] absolute file path to changed lines.
|
|
39
51
|
def for(ref:, files:, project_root:, runner: method(:git_diff))
|
|
40
52
|
files.each_with_object({}) do |file, acc|
|
|
41
53
|
abs = File.expand_path(file, project_root)
|
|
@@ -43,7 +55,12 @@ module Mutineer
|
|
|
43
55
|
end
|
|
44
56
|
end
|
|
45
57
|
|
|
46
|
-
# stdout of `git -C <root> diff --unified=0 <ref> -- <file
|
|
58
|
+
# Returns the stdout of `git -C <root> diff --unified=0 <ref> -- <file>`.
|
|
59
|
+
#
|
|
60
|
+
# @param ref [String] git ref to diff against.
|
|
61
|
+
# @param abs_file [String] absolute path of the file being diffed.
|
|
62
|
+
# @param project_root [String] repository root for `git -C`.
|
|
63
|
+
# @return [String] diff text, or `""` on failure.
|
|
47
64
|
def git_diff(ref, abs_file, project_root)
|
|
48
65
|
out, _err, status = Open3.capture3(
|
|
49
66
|
"git", "-C", project_root, "diff", "--unified=0", ref, "--", abs_file
|
data/lib/mutineer/cli.rb
CHANGED
|
@@ -7,9 +7,11 @@ require_relative "version"
|
|
|
7
7
|
require_relative "config"
|
|
8
8
|
require_relative "parser"
|
|
9
9
|
require_relative "project"
|
|
10
|
+
require_relative "pairing"
|
|
10
11
|
require_relative "changed_lines"
|
|
11
12
|
require_relative "runner"
|
|
12
13
|
require_relative "reporter"
|
|
14
|
+
require_relative "baseline"
|
|
13
15
|
require_relative "mutator_registry"
|
|
14
16
|
|
|
15
17
|
module Mutineer
|
|
@@ -22,6 +24,7 @@ module Mutineer
|
|
|
22
24
|
# 2 usage / flag error (unknown subcommand, invalid flag, unknown operator,
|
|
23
25
|
# out-of-range threshold)
|
|
24
26
|
class CLI
|
|
27
|
+
# Command-line usage banner.
|
|
25
28
|
BANNER = <<~USAGE
|
|
26
29
|
Usage: mutineer [options] <command> [args]
|
|
27
30
|
|
|
@@ -33,6 +36,9 @@ module Mutineer
|
|
|
33
36
|
--test FILE Test file covering the sources (repeatable)
|
|
34
37
|
--operators LIST Comma-separated operator names (default: Tier 1 set)
|
|
35
38
|
--threshold FLOAT Fail (exit 1) when score < FLOAT (default: 0 = off)
|
|
39
|
+
--baseline FILE Fail (exit 1) on NEW survivors / score drop vs a prior
|
|
40
|
+
--format json run (CI delta gate)
|
|
41
|
+
--baseline-epsilon FLOAT Score-drop tolerance for --baseline (default: 0)
|
|
36
42
|
--only NAME Restrict to one fully-qualified subject
|
|
37
43
|
--since REF Only mutate lines changed since git REF (e.g. origin/main)
|
|
38
44
|
--jobs N Parallel worker count (default: processor count)
|
|
@@ -44,6 +50,7 @@ module Mutineer
|
|
|
44
50
|
--format human|json Report format (default: human)
|
|
45
51
|
--output FILE Write the report to FILE instead of stdout
|
|
46
52
|
--dry-run List mutations without executing
|
|
53
|
+
--verbose Surface the real error when a fork capture fails (alias: --debug)
|
|
47
54
|
|
|
48
55
|
Options:
|
|
49
56
|
--list-operators List available operators (default vs optional) and exit
|
|
@@ -57,6 +64,10 @@ module Mutineer
|
|
|
57
64
|
# Deprecated internal strategy names, mapped to their canonical equivalents.
|
|
58
65
|
STRATEGY_ALIASES = { "7a" => "reload", "7b" => "redefine" }.freeze
|
|
59
66
|
|
|
67
|
+
# Parses arguments, executes the command, and exits.
|
|
68
|
+
#
|
|
69
|
+
# @param argv [Array<String>] raw command-line arguments.
|
|
70
|
+
# @return [void]
|
|
60
71
|
def self.start(argv)
|
|
61
72
|
opts = {} # symbol => value, the CLI-provided Config fields
|
|
62
73
|
explicit = Set.new # precedence keys the user typed (KTD3)
|
|
@@ -84,8 +95,14 @@ module Mutineer
|
|
|
84
95
|
o.on("--framework NAME") { |v| opts[:framework] = v; explicit << :framework }
|
|
85
96
|
o.on("--boot FILE") { |v| opts[:boot] = v; explicit << :boot }
|
|
86
97
|
o.on("--rails") { opts[:rails] = true }
|
|
98
|
+
o.on("--verbose") { opts[:verbose] = true }
|
|
99
|
+
o.on("--debug") { opts[:verbose] = true } # alias of --verbose
|
|
87
100
|
o.on("--format FORMAT") { |v| opts[:format] = v }
|
|
88
101
|
o.on("--output FILE") { |v| opts[:output] = v }
|
|
102
|
+
# #13: --baseline is also a .mutineer.yml key, so mark it explicit when
|
|
103
|
+
# typed (CLI wins over the file). --baseline-epsilon is CLI-only.
|
|
104
|
+
o.on("--baseline FILE") { |v| opts[:baseline] = v; explicit << :baseline }
|
|
105
|
+
o.on("--baseline-epsilon FLOAT") { |v| opts[:baseline_epsilon] = v.to_f }
|
|
89
106
|
end
|
|
90
107
|
|
|
91
108
|
begin
|
|
@@ -118,14 +135,19 @@ module Mutineer
|
|
|
118
135
|
|
|
119
136
|
case argv.first
|
|
120
137
|
when "run"
|
|
121
|
-
|
|
122
|
-
|
|
138
|
+
# #11: a directory source expands to its **/*.rb files; literal files pass
|
|
139
|
+
# through. Test inference (when --test is omitted) happens in validate!.
|
|
140
|
+
config.sources = Pairing.expand_sources(argv[1..], project_root: config.project_root)
|
|
141
|
+
run(config, explicit)
|
|
123
142
|
else
|
|
124
143
|
warn "mutineer: unknown command '#{argv.first}'"
|
|
125
144
|
exit 2
|
|
126
145
|
end
|
|
127
146
|
end
|
|
128
147
|
|
|
148
|
+
# Lists available operators.
|
|
149
|
+
#
|
|
150
|
+
# @return [void]
|
|
129
151
|
def self.list_operators
|
|
130
152
|
MutatorRegistry::ALL.each_key do |name|
|
|
131
153
|
state = MutatorRegistry.default?(name) ? "default" : "disabled"
|
|
@@ -134,12 +156,17 @@ module Mutineer
|
|
|
134
156
|
end
|
|
135
157
|
end
|
|
136
158
|
|
|
137
|
-
|
|
159
|
+
# Runs the requested command after validation.
|
|
160
|
+
#
|
|
161
|
+
# @param config [Mutineer::Config] run configuration.
|
|
162
|
+
# @param explicit [Set<Symbol>] explicit CLI fields.
|
|
163
|
+
# @return [void]
|
|
164
|
+
def self.run(config, explicit = Set.new)
|
|
138
165
|
if config.sources.empty?
|
|
139
166
|
warn "mutineer: run requires at least one source file"
|
|
140
167
|
exit 2
|
|
141
168
|
end
|
|
142
|
-
validate!(config)
|
|
169
|
+
validate!(config, explicit)
|
|
143
170
|
|
|
144
171
|
config.dry_run ? dry_run(config) : execute(config)
|
|
145
172
|
rescue ArgumentError => e
|
|
@@ -163,7 +190,13 @@ module Mutineer
|
|
|
163
190
|
|
|
164
191
|
# Flag validation: every flag/usage failure exits 2 (C7), consistent with the
|
|
165
192
|
# taxonomy above — CI can tell "mistyped flag" from "tests too weak."
|
|
166
|
-
|
|
193
|
+
# Validates the run configuration.
|
|
194
|
+
#
|
|
195
|
+
# @api private
|
|
196
|
+
# @param config [Mutineer::Config] run configuration.
|
|
197
|
+
# @param explicit [Set<Symbol>] explicit CLI fields.
|
|
198
|
+
# @return [void]
|
|
199
|
+
def self.validate!(config, explicit = Set.new)
|
|
167
200
|
unless (0.0..100.0).cover?(config.threshold)
|
|
168
201
|
warn "mutineer: --threshold must be between 0 and 100"
|
|
169
202
|
exit 2
|
|
@@ -194,6 +227,17 @@ module Mutineer
|
|
|
194
227
|
exit 2
|
|
195
228
|
end
|
|
196
229
|
|
|
230
|
+
validate_since!(config) if config.since
|
|
231
|
+
preflight_output!(config.output) if config.output
|
|
232
|
+
preflight_baseline!(config.baseline) if config.baseline
|
|
233
|
+
|
|
234
|
+
# #11: when --test is omitted, infer each source's test by convention so the
|
|
235
|
+
# boot-once/fork-per-test core (which pairs empirically by coverage) gets a
|
|
236
|
+
# populated config.tests. Runs after every flag/usage check above so a
|
|
237
|
+
# mistyped flag still reports the flag; skipped under --dry-run (no tests
|
|
238
|
+
# needed). validate_paths! then sees the inferred (real) tests.
|
|
239
|
+
autopair!(config, explicit) unless config.dry_run
|
|
240
|
+
|
|
197
241
|
# Boot mode does no coverage selection — every mutant runs the given tests —
|
|
198
242
|
# so at least one --test file is mandatory (there is nothing to select from).
|
|
199
243
|
if config.boot && config.tests.empty?
|
|
@@ -201,13 +245,16 @@ module Mutineer
|
|
|
201
245
|
exit 2
|
|
202
246
|
end
|
|
203
247
|
|
|
204
|
-
validate_since!(config) if config.since
|
|
205
|
-
preflight_output!(config.output) if config.output
|
|
206
248
|
validate_paths!(config)
|
|
207
249
|
end
|
|
208
250
|
|
|
209
251
|
# --since needs a real git repo and a resolvable ref; either failure is a
|
|
210
252
|
# usage error (exit 2) so CI sees "bad invocation," not "tests too weak."
|
|
253
|
+
# Validates the --since ref.
|
|
254
|
+
#
|
|
255
|
+
# @api private
|
|
256
|
+
# @param config [Mutineer::Config] run configuration.
|
|
257
|
+
# @return [void]
|
|
211
258
|
def self.validate_since!(config)
|
|
212
259
|
_out, _err, status = Open3.capture3(
|
|
213
260
|
"git", "-C", config.project_root, "rev-parse", "--verify", "--quiet",
|
|
@@ -229,6 +276,11 @@ module Mutineer
|
|
|
229
276
|
# R5: validate path existence up front so a typo is a clean usage error (exit
|
|
230
277
|
# 2), not an Errno::ENOENT backtrace from deep in the run. Flag checks run
|
|
231
278
|
# first so a bad flag still reports the flag, not the missing file.
|
|
279
|
+
# Validates source and test paths.
|
|
280
|
+
#
|
|
281
|
+
# @api private
|
|
282
|
+
# @param config [Mutineer::Config] run configuration.
|
|
283
|
+
# @return [void]
|
|
232
284
|
def self.validate_paths!(config)
|
|
233
285
|
missing = (config.sources + config.tests)
|
|
234
286
|
.reject { |p| File.exist?(File.expand_path(p, config.project_root)) }
|
|
@@ -238,6 +290,61 @@ module Mutineer
|
|
|
238
290
|
exit 2
|
|
239
291
|
end
|
|
240
292
|
|
|
293
|
+
# #11: auto-pair sources to tests by path convention when no --test was given
|
|
294
|
+
# (explicit --test wins — R5). Each source with an inferred test on disk joins
|
|
295
|
+
# the run; a source with none is dropped with a one-line stderr warning (R3)
|
|
296
|
+
# and the run continues with the rest. If every source is dropped: in boot mode
|
|
297
|
+
# the dedicated --boot/--rails-requires-test check reports it; otherwise exit 2
|
|
298
|
+
# with a usage message. The framework is re-detected from the inferred set
|
|
299
|
+
# unless it was set explicitly (a spec-only project loads/reports as rspec).
|
|
300
|
+
# Auto-pairs sources and tests when --test is absent.
|
|
301
|
+
#
|
|
302
|
+
# @api private
|
|
303
|
+
# @param config [Mutineer::Config] run configuration.
|
|
304
|
+
# @param explicit [Set<Symbol>] explicit CLI fields.
|
|
305
|
+
# @return [void]
|
|
306
|
+
def self.autopair!(config, explicit)
|
|
307
|
+
return unless config.tests.empty?
|
|
308
|
+
|
|
309
|
+
paired = config.sources.filter_map do |s|
|
|
310
|
+
t = Pairing.infer_test(s, project_root: config.project_root, prefer: config.framework)
|
|
311
|
+
[s, t] if t
|
|
312
|
+
end
|
|
313
|
+
(config.sources - paired.map(&:first)).each do |s|
|
|
314
|
+
warn "[mutineer] no test found by convention for #{s}; skipping"
|
|
315
|
+
end
|
|
316
|
+
config.sources = paired.map(&:first)
|
|
317
|
+
config.tests = paired.map(&:last).uniq
|
|
318
|
+
config.framework = Config.detect_framework(config.tests) unless explicit.include?(:framework)
|
|
319
|
+
|
|
320
|
+
return unless config.sources.empty?
|
|
321
|
+
return if config.boot # let the --boot/--rails-requires-test check report it
|
|
322
|
+
|
|
323
|
+
warn "mutineer: no test files found by convention; pass --test or add tests"
|
|
324
|
+
exit 2
|
|
325
|
+
end
|
|
326
|
+
|
|
327
|
+
# #13: a missing/unreadable/unparseable baseline is a usage error (exit 2),
|
|
328
|
+
# mirroring --output/--since preflight, so CI sees "bad invocation," not a
|
|
329
|
+
# backtrace mid-run. Validating up front = attempting the load (it raises
|
|
330
|
+
# ConfigError/SystemCallError; the actual diff reloads in execute).
|
|
331
|
+
# Preflights a baseline file.
|
|
332
|
+
#
|
|
333
|
+
# @api private
|
|
334
|
+
# @param path [String] baseline file path.
|
|
335
|
+
# @return [void]
|
|
336
|
+
def self.preflight_baseline!(path)
|
|
337
|
+
Baseline.load(path)
|
|
338
|
+
rescue Mutineer::ConfigError, SystemCallError => e
|
|
339
|
+
warn "mutineer: #{e.message}"
|
|
340
|
+
exit 2
|
|
341
|
+
end
|
|
342
|
+
|
|
343
|
+
# Preflights an output path.
|
|
344
|
+
#
|
|
345
|
+
# @api private
|
|
346
|
+
# @param path [String] output file path.
|
|
347
|
+
# @return [void]
|
|
241
348
|
def self.preflight_output!(path)
|
|
242
349
|
dir = File.dirname(File.expand_path(path))
|
|
243
350
|
return if File.directory?(dir) && File.writable?(dir)
|
|
@@ -247,6 +354,10 @@ module Mutineer
|
|
|
247
354
|
exit 2
|
|
248
355
|
end
|
|
249
356
|
|
|
357
|
+
# Executes the run command.
|
|
358
|
+
#
|
|
359
|
+
# @param config [Mutineer::Config] run configuration.
|
|
360
|
+
# @return [void]
|
|
250
361
|
def self.execute(config)
|
|
251
362
|
if config.tests.empty?
|
|
252
363
|
warn "mutineer: run requires at least one --test file (or use --dry-run)"
|
|
@@ -255,8 +366,14 @@ module Mutineer
|
|
|
255
366
|
|
|
256
367
|
aggregate, source_map = Runner.execute(config)
|
|
257
368
|
reporter = Reporter.new(aggregate, source_map)
|
|
369
|
+
|
|
370
|
+
# #13: diff the current run against the baseline (preflighted above) by the
|
|
371
|
+
# stable survivor id. The delta is rendered inline (human section / additive
|
|
372
|
+
# json block) and gates exit independently of --threshold.
|
|
373
|
+
delta = (Baseline.load(config.baseline).diff(aggregate, epsilon: config.baseline_epsilon) if config.baseline)
|
|
374
|
+
|
|
258
375
|
reporter.report(out: $stdout, err: $stderr, threshold: config.threshold,
|
|
259
|
-
format: config.format, output: config.output)
|
|
376
|
+
format: config.format, output: config.output, baseline: delta)
|
|
260
377
|
|
|
261
378
|
# #14: nudge toward the opt-in tier-2 operators (human report only — never
|
|
262
379
|
# pollute JSON output).
|
|
@@ -264,11 +381,18 @@ module Mutineer
|
|
|
264
381
|
puts hint
|
|
265
382
|
end
|
|
266
383
|
|
|
267
|
-
|
|
384
|
+
# #13/KTD-4: --baseline and --threshold are independent gates OR'd together.
|
|
385
|
+
# `max` of two 0/1 codes is the OR; usage (2) is handled earlier and wins.
|
|
386
|
+
baseline_exit = delta&.regressed ? 1 : 0
|
|
387
|
+
exit [reporter.exit_code(threshold: config.threshold), baseline_exit].max
|
|
268
388
|
end
|
|
269
389
|
|
|
270
390
|
# The tier-2 operators not in the active set, as a one-line hint (or nil when
|
|
271
391
|
# they're all already enabled). `active` nil means the default (Tier-1) set.
|
|
392
|
+
# Builds a Tier-2 operator hint.
|
|
393
|
+
#
|
|
394
|
+
# @param active [Array<String>, nil] active operator names.
|
|
395
|
+
# @return [String, nil] hint text or nil.
|
|
272
396
|
def self.tier2_hint(active)
|
|
273
397
|
active ||= MutatorRegistry::DEFAULT_NAMES
|
|
274
398
|
unused = MutatorRegistry::TIER2_NAMES - active
|
|
@@ -278,6 +402,10 @@ module Mutineer
|
|
|
278
402
|
"enable with --operators <list>."
|
|
279
403
|
end
|
|
280
404
|
|
|
405
|
+
# Runs dry-run mode.
|
|
406
|
+
#
|
|
407
|
+
# @param config [Mutineer::Config] run configuration.
|
|
408
|
+
# @return [void]
|
|
281
409
|
def self.dry_run(config)
|
|
282
410
|
operator_classes = MutatorRegistry.resolve(config.operators || MutatorRegistry::DEFAULT_NAMES)
|
|
283
411
|
sources = {}
|