saboteur 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +113 -1
- data/README.md +248 -63
- data/lib/generators/saboteur/install/install_generator.rb +21 -0
- data/lib/saboteur/agent_skill.rb +40 -0
- data/lib/saboteur/cli.rb +150 -22
- data/lib/saboteur/config.rb +27 -15
- data/lib/saboteur/config_file.rb +97 -0
- data/lib/saboteur/custom_map.rb +170 -0
- data/lib/saboteur/doctor.rb +17 -1
- data/lib/saboteur/engine/baseline_guard.rb +20 -3
- data/lib/saboteur/engine/boot.rb +9 -0
- data/lib/saboteur/engine/coverage_store.rb +31 -4
- data/lib/saboteur/engine/data_inputs.rb +43 -0
- data/lib/saboteur/engine/diagnostics.rb +28 -0
- data/lib/saboteur/engine/diff_scope.rb +8 -0
- data/lib/saboteur/engine/run_fingerprint.rb +34 -6
- data/lib/saboteur/engine/scheduler.rb +7 -16
- data/lib/saboteur/engine/suite_files.rb +8 -1
- data/lib/saboteur/engine.rb +74 -13
- data/lib/saboteur/execution/reevaluator.rb +7 -4
- data/lib/saboteur/execution/supervisor.rb +23 -11
- data/lib/saboteur/frameworks/rails.rb +329 -13
- data/lib/saboteur/help.rb +33 -17
- data/lib/saboteur/ignore_list.rb +71 -17
- data/lib/saboteur/install.rb +132 -0
- data/lib/saboteur/integration/rspec.rb +47 -3
- data/lib/saboteur/mutation_source/argument_removal.rb +86 -0
- data/lib/saboteur/mutation_source/builtin.rb +8 -2
- data/lib/saboteur/mutation_source/pair_removal.rb +101 -0
- data/lib/saboteur/replay.rb +1 -1
- data/lib/saboteur/reporter/agent.rb +87 -0
- data/lib/saboteur/reporter/agent_clipping.rb +29 -0
- data/lib/saboteur/reporter/agent_commands.rb +61 -0
- data/lib/saboteur/reporter/agent_incomplete.rb +37 -0
- data/lib/saboteur/reporter/agent_summary.rb +131 -0
- data/lib/saboteur/reporter/findings.rb +8 -0
- data/lib/saboteur/reporter/html.rb +49 -1
- data/lib/saboteur/reporter/stryker.rb +11 -4
- data/lib/saboteur/reporter/terminal.rb +65 -2
- data/lib/saboteur/result.rb +46 -5
- data/lib/saboteur/score.rb +89 -0
- data/lib/saboteur/version.rb +1 -1
- data/lib/saboteur/working_tree.rb +48 -0
- data/lib/saboteur.rb +4 -0
- metadata +17 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a5c52cb7485073b569359bf940160095370103747d55a6065b7f067726a1a53c
|
|
4
|
+
data.tar.gz: 40fca0c1c0f8391de86d7f17960d908add226682d93be2610775fc888ab36f60
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 061e0b5a97c95048048767d3724ea883ea144e7530b9a52631d52c896f4154e1a63f7b65ef7b5817053a42ab31eb97c168aa3079c87e9d9a3e8be1ae949d7bdf
|
|
7
|
+
data.tar.gz: e29bcfafdaf2fc7ecffbc4f8746349d79c908dfc2de1530998b525c691135c2bfb4ade67575374178742b1fddb29ab96a7d9ff67656d29dbd7c096ba0b50d0dc
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,118 @@ All notable changes are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.4.0] - 2026-10-12
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
- **Breaking:** configuration moved from YAML to Ruby. `.saboteur.yml` becomes
|
|
13
|
+
`config/saboteur.rb` (`Saboteur.configure do |c| ... end`, same key names),
|
|
14
|
+
and `.saboteur-ignore.yml` becomes `config/saboteur_ignore.rb`
|
|
15
|
+
(`Saboteur.ignore do |i| i.mutation "<digest>", subject:, reason: end`,
|
|
16
|
+
reason still mandatory). A legacy YAML file stops the run with a pointer to
|
|
17
|
+
the new file instead of being read. Unknown settings fail with the line
|
|
18
|
+
number and a did-you-mean suggestion. `rails generate saboteur:install`
|
|
19
|
+
writes `config/saboteur.rb` (never overwriting without `--force`);
|
|
20
|
+
`config/saboteur_ignore.rb` and `config/saboteur_custom_map.rb` are optional
|
|
21
|
+
and created by hand when needed.
|
|
22
|
+
- A run in which more than half of the executed mutants errored now exits 3
|
|
23
|
+
(the tool failed). Before, only a run where every executed mutant errored
|
|
24
|
+
did. Errored mutants are outside the score, so such a run tested almost
|
|
25
|
+
nothing. This applies with and without `--threshold`. The JSON report records the reason under
|
|
26
|
+
`saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`, `errorMajority`,
|
|
27
|
+
`incompleteUnderThreshold`; in `--agent` mode `ok`, `survivors`,
|
|
28
|
+
`errorMajority` or `incomplete`).
|
|
29
|
+
- A whole-suite coverage build that records no lines for any included file
|
|
30
|
+
now exits 3 and the empty map is not cached, instead of testing nothing.
|
|
31
|
+
With `--test`, an empty map stays a legitimate "no coverage" outcome.
|
|
32
|
+
- Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
|
|
33
|
+
`db/schema.rb` and `config/*.yml` are part of the coverage and verdict
|
|
34
|
+
cache fingerprint, so a changed fixture or schema invalidates cached
|
|
35
|
+
results. The new `fingerprint_extra` setting adds other globs whose
|
|
36
|
+
contents should invalidate caches, for example `data/*.csv`.
|
|
37
|
+
- The baseline and coverage phases show progress and write real logs, so a
|
|
38
|
+
slow or failing phase can be diagnosed from the output.
|
|
39
|
+
- When the baseline child is killed by a signal, the message names it
|
|
40
|
+
(`SIGKILL` adds a possible-OOM hint) instead of reporting a bare failure.
|
|
41
|
+
|
|
42
|
+
### Removed
|
|
43
|
+
- The coarse-survivor gate: a surviving whole-body mutant no longer skips the
|
|
44
|
+
method's fine mutants; every eligible fine mutant runs.
|
|
45
|
+
|
|
46
|
+
### Fixed
|
|
47
|
+
- Mutations inside private `module_function` helpers are preserved when the
|
|
48
|
+
method is re-evaluated; they previously reverted and reported as false
|
|
49
|
+
survivors.
|
|
50
|
+
- Rails apps that test against in-memory SQLite (`:memory:`) no longer fail
|
|
51
|
+
in forked workers with "Could not find table": saboteur never disconnects,
|
|
52
|
+
renames or copies an in-memory pool. Each in-memory database is
|
|
53
|
+
snapshotted to a temporary file after boot, and every forked child
|
|
54
|
+
restores the snapshot into its own fresh database.
|
|
55
|
+
- With a Rails app under Bootsnap, Coverage no longer sees an empty map:
|
|
56
|
+
saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its own process when
|
|
57
|
+
the variable is unset, because bytecode from Bootsnap's compile cache
|
|
58
|
+
carries no coverage counters.
|
|
59
|
+
- A relative SQLite `database:` path resolves against `Rails.root` (as Rails
|
|
60
|
+
7.1+ does) for the per-worker copy and its cleanup. The copy uses SQLite's
|
|
61
|
+
backup API after a WAL checkpoint, refuses to run when a reader keeps the
|
|
62
|
+
WAL busy, and a missing source file is an error rather than a skipped copy.
|
|
63
|
+
- Per-worker database copies (`jobs` above 1) are dropped when the run ends,
|
|
64
|
+
best effort: a failed drop is a warning and never changes the outcome. A
|
|
65
|
+
copy left by a killed run is reclaimed by the next run. A failed copy
|
|
66
|
+
names the sessions blocking it.
|
|
67
|
+
- For `rails: true`, the app's `spec/` directory is on the load path before
|
|
68
|
+
`require` is processed, as with rspec, so a boot file such as a dummy app's
|
|
69
|
+
`spec/rails_helper.rb` can require its siblings.
|
|
70
|
+
- When Rails did not load, the error points engine users at the dummy app's
|
|
71
|
+
environment file or helper (`DummyApp.setup` in `spec/rails_helper.rb`).
|
|
72
|
+
|
|
73
|
+
### Added
|
|
74
|
+
- `rails generate saboteur:install` and a Rails quickstart in the README. The
|
|
75
|
+
generator writes `config/saboteur.rb` and never overwrites without `--force`.
|
|
76
|
+
- The warning about threads alive when a worker forks names those threads,
|
|
77
|
+
falling back to their backtraces for unnamed ones.
|
|
78
|
+
- `--test` no longer requires source operands when `--since` is given: the
|
|
79
|
+
changed files are the sources, so `saboteur run --since origin/main --test
|
|
80
|
+
spec/billing_spec.rb` scopes a pull-request run to chosen specs. A `--since`
|
|
81
|
+
run without `--test` now warns before it starts that the two baseline passes
|
|
82
|
+
run the loaded suite (a cold coverage cache adds one more pass), with the
|
|
83
|
+
spec-file count and how to scope them, instead of going quiet for hours on a
|
|
84
|
+
large app.
|
|
85
|
+
- `config/saboteur_custom_map.rb` declares explicit source-file to spec-file
|
|
86
|
+
mappings (`SABOTEUR_CUSTOM_MAP`): exact paths, globs (the more specific glob
|
|
87
|
+
wins; equal specificity is an error) and Regexps with capture substitution.
|
|
88
|
+
Invalid maps fail the run with the file and key; stale and duplicate keys
|
|
89
|
+
warn on runs, and `saboteur doctor` fails on stale keys. The map is parsed
|
|
90
|
+
and validated only for now - it does not change which specs run yet.
|
|
91
|
+
|
|
92
|
+
- An integration target can declare a known error profile
|
|
93
|
+
(`allow_error_fraction` per strategy in integration/targets.yml): the
|
|
94
|
+
majority-error exit is accepted only when the run's own fresh report
|
|
95
|
+
carries the CLI's `errorMajority` marker, the errored fraction stays
|
|
96
|
+
within the bound, and no mutant hit a resource limit. The JSON report
|
|
97
|
+
records the run's exit cause under `saboteur.exitCause`.
|
|
98
|
+
- A survivor lists the first five tests that covered its line and still passed
|
|
99
|
+
with the mutant applied, in the terminal and the HTML page.
|
|
100
|
+
- In diff mode (`--since`), each survivor names the changed line it relates to
|
|
101
|
+
(`changed line: path:N` in the terminal, `changedLines` in each JSON finding),
|
|
102
|
+
because a multi-line statement is reported at its first line.
|
|
103
|
+
- The score is split into assertion kills and crash kills, in the terminal,
|
|
104
|
+
the JSON report (`saboteur.scores`) and the HTML page. A second score leaves
|
|
105
|
+
out whole-body mutants (`body_nil`, `body_true`, `body_false`, `body_raise`). The headline score and
|
|
106
|
+
`--threshold` are unchanged.
|
|
107
|
+
- `saboteur run --agent` (`--format agent`): one bounded JSON summary (under 8 KiB) on stdout
|
|
108
|
+
with status, verdict, provenance, the first three findings and the commands for full
|
|
109
|
+
detail. Exit 0 means a complete, clean run only: 1 survivors, 2 usage or config error,
|
|
110
|
+
3 incomplete or failed. `--threshold` cannot be combined with it. The next commands are shell-escaped and keep the
|
|
111
|
+
full selection; the working-tree state is read before the run; target output stays on stderr. `saboteur agent-skill`
|
|
112
|
+
prints an AGENTS.md block for coding agents (also in `docs/agent-skill.md`). Classic
|
|
113
|
+
exit codes are unchanged.
|
|
114
|
+
- Fine `pair_removal` mutants for hash literals and keyword lists, including
|
|
115
|
+
sole associations, with token-based separators and the normal method cap.
|
|
116
|
+
- Fine `argument_removal` mutants: remove one ordinary argument from a
|
|
117
|
+
parenthesized call while preserving keywords and blocks. Token-based comma
|
|
118
|
+
edits respect comments and strings. Existing candidate validation, order and
|
|
119
|
+
per-method caps apply. Unparenthesized calls and heredoc arguments are skipped.
|
|
120
|
+
|
|
9
121
|
## [0.3.0] - 2026-10-11
|
|
10
122
|
|
|
11
123
|
### Added
|
|
@@ -64,7 +176,7 @@ First release that does mutation testing. MIT licensed.
|
|
|
64
176
|
### Added
|
|
65
177
|
- `saboteur run` with `--since`, `--subject`, `--timeout-factor`, `--jobs`,
|
|
66
178
|
`--reeval`, `--json`, `--no-cache`, `--incremental`, `--config`.
|
|
67
|
-
- Subjects found with Prism; mutations are byte-range edits: whole-
|
|
179
|
+
- Subjects found with Prism; mutations are byte-range edits: whole-body
|
|
68
180
|
`nil`/`true`/`false`/`raise` first, then comparison, boolean, arithmetic,
|
|
69
181
|
numeric literal, statement removal, condition, negation, safe navigation
|
|
70
182
|
and `||=`.
|
data/README.md
CHANGED
|
@@ -31,14 +31,22 @@ fail?" for many small breakages and lists the ones nobody noticed. Read the
|
|
|
31
31
|
|
|
32
32
|
### The unchanged suite is checked first
|
|
33
33
|
|
|
34
|
-
Every run with selected subjects runs the
|
|
34
|
+
Every run with selected subjects runs the loaded unchanged suite twice (the specs chosen with `--test`, if given) in fresh
|
|
35
35
|
supervised children before coverage lookup or incremental verdict reuse. This
|
|
36
36
|
costs two suite runs even on a cache hit, plus a coverage-building suite run on
|
|
37
37
|
a cache miss. Both baseline passes must be green, complete and in the same
|
|
38
|
-
RSpec order at the run seed. A failure exits 3, stops mutation and names the log and fix.
|
|
38
|
+
RSpec order at the run seed. A failure exits 3, stops mutation and names the log and fix. Long phases
|
|
39
|
+
print their elapsed time about every 30 seconds, and each phase's log holds
|
|
40
|
+
the suite's progress and full failure detail. Each suite pass
|
|
39
41
|
uses `coverage_timeout` and configured memory/process limits (the suite memory
|
|
40
42
|
budget is four times `max_memory_mb`, as for coverage). Older benchmark numbers
|
|
41
43
|
need a rerun before they describe this extra fixed cost.
|
|
44
|
+
A whole-suite coverage build that records no lines for any included file
|
|
45
|
+
means the setup hid the app from Coverage (Bootsnap's compile cache is the
|
|
46
|
+
common cause), so the run fails with exit 3 instead of testing nothing, and
|
|
47
|
+
the empty map is not cached. With `--test` an empty map instead stays a
|
|
48
|
+
legitimate "no coverage" outcome, because the chosen specs may not touch the
|
|
49
|
+
selected sources.
|
|
42
50
|
Two passes detect observed instability, not every possible flaky test. See
|
|
43
51
|
[baseline stability](docs/baseline-stability.md) for limits and hook behaviour.
|
|
44
52
|
|
|
@@ -57,9 +65,8 @@ bundle exec saboteur run --subject 'YourClass#your_method'
|
|
|
57
65
|
```
|
|
58
66
|
|
|
59
67
|
Outside a Bundler project, `gem install saboteur` installs the same executable.
|
|
60
|
-
Rails
|
|
61
|
-
`rails:
|
|
62
|
-
gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
|
|
68
|
+
Rails app? Follow the [Rails quickstart](#rails-apps) below: the defaults are for
|
|
69
|
+
plain gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
|
|
63
70
|
|
|
64
71
|
Start with one method. You get counts per status, a score, timing, and a
|
|
65
72
|
unified diff for every survivor, so you can read the exact change your tests
|
|
@@ -74,22 +81,24 @@ line coverage touched the changed statement. Tests that killed earlier mutants
|
|
|
74
81
|
of the same method go first, then the fastest, and the first failure stops the
|
|
75
82
|
run. A change no test reaches is reported as "no coverage", not as a survivor.
|
|
76
83
|
|
|
77
|
-
**It
|
|
78
|
-
mutants
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
84
|
+
**It runs coarse mutants first, then fine mutants.** Each method first gets
|
|
85
|
+
whole-body mutants: `nil` (`true`/`false` for predicates) unless the body is empty or
|
|
86
|
+
only logs, and `raise` unless a catch-all rescue would absorb it. Fine-grained mutants run
|
|
87
|
+
regardless of the coarse verdict, including a survivor or a reused verdict.
|
|
88
|
+
Deleting a body unnoticed does not prove its smaller changes are harmless.
|
|
89
|
+
Subjects with no tests are skipped before any mutants are generated, and each
|
|
90
|
+
method is capped at `max_mutants_per_method` mutants (100 by default). Running
|
|
91
|
+
fine mutants after coarse survivors can increase run time and change scores.
|
|
83
92
|
|
|
84
93
|
**It prepares memory before it forks.** Before workers fork, saboteur warms and
|
|
85
94
|
compacts the Ruby heap ([`ForkReadiness`](lib/saboteur/execution/fork_readiness.rb)),
|
|
86
95
|
which is meant to let workers share more pages with their parent and copy less.
|
|
87
|
-
|
|
96
|
+
Observed on the pinned hashie workload (paired runs, below): about 17-18% lower peak PSS at 2 and 6 workers. Not a general guarantee.
|
|
88
97
|
|
|
89
98
|
**It is aimed at pull requests.** `--since origin/main` mutates only the methods
|
|
90
99
|
your branch changed: fine-grained mutants on the changed lines, plus the
|
|
91
100
|
whole-body mutants of those methods. On one real change in saboteur's own history, that took a file
|
|
92
|
-
from 139 mutants to 7, and median wall time fell 22.9
|
|
101
|
+
from 139 mutants to 7, and median wall time fell 22.9% (measured at d2b7abb, before the coarse-survivor gate was removed; a rerun is pending). The time saving is
|
|
93
102
|
smaller than the mutant count because building the coverage map is a fixed
|
|
94
103
|
cost, and this is one file, not a Rails app, so it is not a general speed
|
|
95
104
|
claim. The full method, raw timings and
|
|
@@ -97,8 +106,8 @@ caveats are in [bench/since](bench/since/README.md).
|
|
|
97
106
|
|
|
98
107
|
**It checks its own kills.** After a method's mutants run, saboteur re-runs the
|
|
99
108
|
killing tests against the *unchanged* method. If they fail there too, the kill
|
|
100
|
-
is not credited and becomes an error. In
|
|
101
|
-
|
|
109
|
+
is not credited and becomes an error. In one comparison on lobsters, 0 of 20
|
|
110
|
+
randomly sampled kills were fake. Every mutant runs in a supervised child with deadline,
|
|
102
111
|
memory and process limits, and saboteur tracks and kills the processes it
|
|
103
112
|
starts.
|
|
104
113
|
|
|
@@ -124,17 +133,20 @@ Positional operands name the sources to mutate: Ruby files, or directories
|
|
|
124
133
|
(all their `.rb` files). Each must exist; a typo or a non-Ruby file exits 2
|
|
125
134
|
before anything runs. They replace `include` and `exclude`. `--test`
|
|
126
135
|
(repeatable; spec files or directories, with an optional `:LINE` or `[1:2]`
|
|
127
|
-
selector; needs at least one source
|
|
136
|
+
selector; needs at least one source, unless `--since` is given - the changed
|
|
137
|
+
files are the sources) replaces only the files RSpec runs. Other
|
|
128
138
|
`rspec_args` and `.rspec` options such as `--require` stay. The coverage map
|
|
129
139
|
is built from only those specs, each mutant runs only the ones that cover it,
|
|
130
140
|
and the selection is part of the coverage and `--incremental` cache keys.
|
|
131
141
|
|
|
132
|
-
`--threshold PERCENT` (0 to 100, or `threshold
|
|
142
|
+
`--threshold PERCENT` (0 to 100, or `c.threshold` in `config/saboteur.rb`) is for CI.
|
|
133
143
|
The run exits 1 when the score is below PERCENT and 0 when it is at or above,
|
|
134
144
|
even with survivors. It fails closed: a run that tested no mutant has no score
|
|
135
145
|
and exits 1, and a run where any mutant errored or hit a resource limit exits 3,
|
|
136
146
|
because the score covers only part of the run. Without a threshold, any
|
|
137
|
-
unignored survivor exits 1.
|
|
147
|
+
unignored survivor exits 1. Either way, a run where more than half of the
|
|
148
|
+
executed mutants errored exits 3: errored mutants are outside the score,
|
|
149
|
+
so such a run tested almost nothing.
|
|
138
150
|
|
|
139
151
|
Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
|
|
140
152
|
`--config PATH`, `--integration NAME`, `--reeval class_eval|nesting`,
|
|
@@ -142,7 +154,11 @@ Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
|
|
|
142
154
|
|
|
143
155
|
For pull requests, start with `--since` against your target branch. This is a
|
|
144
156
|
workflow recommendation, not a CLI default: plain `run` still checks the whole
|
|
145
|
-
configured project.
|
|
157
|
+
configured project. To scope the specs as well, pass them with `--test` and no
|
|
158
|
+
source operands: `bundle exec saboteur run --since origin/main --test spec/billing_spec.rb`.
|
|
159
|
+
With `--since` and no `--test`, the run warns before it starts that the two baseline passes
|
|
160
|
+
run the loaded suite (a cold coverage cache adds one more pass), and how to
|
|
161
|
+
scope them. Keep whole-project runs on main: diff mode does not check every method affected by a
|
|
146
162
|
dependency change. [The pull-request guide](docs/pull-requests.md) covers
|
|
147
163
|
checkout history, a GitHub Actions example, configuration and limits.
|
|
148
164
|
|
|
@@ -152,7 +168,7 @@ mutation reports or coverage/verdict caches. Target boot files and tests still
|
|
|
152
168
|
run, so their normal side effects remain. See the [doctor guide](docs/guides/doctor.md).
|
|
153
169
|
|
|
154
170
|
`--id` runs exactly one current mutant with its coverage-selected tests, bypassing
|
|
155
|
-
per-method caps,
|
|
171
|
+
per-method caps, ignore suppression and incremental verdict reuse.
|
|
156
172
|
It still honors `--subject` and `--since` changed-line selection. See the
|
|
157
173
|
[replay guide](docs/guides/replay.md) for digest format, checks and limits.
|
|
158
174
|
|
|
@@ -167,16 +183,42 @@ unchanged. It cannot see a change in a method this one calls, so run without
|
|
|
167
183
|
it now and then (and in CI on the main branch).
|
|
168
184
|
|
|
169
185
|
Output: counts per status, the score, timing, and a unified diff for every
|
|
170
|
-
survivor.
|
|
186
|
+
survivor. Under each diff the terminal lists the tests that covered that line
|
|
187
|
+
and still passed with the mutant applied (the first five, likeliest killers
|
|
188
|
+
first; the rest are under `coveredBy` in the JSON report). The missing
|
|
189
|
+
check may belong in one of them. Line coverage can overstate which tests
|
|
190
|
+
exercise a line, so treat the list as a lead. A survivor reused with
|
|
191
|
+
`--incremental` was not run again, and the list says its tests passed in an
|
|
192
|
+
earlier run. The HTML page shows the same list in the
|
|
193
|
+
survivor's notes. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
|
|
171
194
|
report (one file, no scripts, no network) to `.saboteur/report.html`. Change
|
|
172
|
-
the paths with `--json` / `--html` or `json_report` / `html_report` in
|
|
173
|
-
|
|
195
|
+
the paths with `--json` / `--html` or `c.json_report` / `c.html_report` in
|
|
196
|
+
`config/saboteur.rb`; set either to `nil` to turn that report off. The two paths
|
|
174
197
|
must differ. The HTML page shows the covered mutation score
|
|
175
198
|
`(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage
|
|
176
199
|
out and keeps survivors accepted on the ignore list in the denominator. A tool
|
|
177
200
|
that reads the JSON report may show a different score, because the schema counts
|
|
178
201
|
NoCoverage as undetected.
|
|
179
202
|
|
|
203
|
+
The score is split by how the mutant was killed. A crash kill (a `NoMethodError`,
|
|
204
|
+
a `TypeError`, a parse error) shows the code ran and broke, not that a test
|
|
205
|
+
checked a value. The terminal prints
|
|
206
|
+
`score 100.0 % (assertion 50.0 %, crash 50.0 %)`: each share is a part of
|
|
207
|
+
tested mutants, so the parts add up to the score. Timeouts and kills with no
|
|
208
|
+
recorded cause (a verdict reused from an older run, a test that called `exit`,
|
|
209
|
+
a failure outside any example) are named when there are some. Whole-body
|
|
210
|
+
mutants (`body_nil`, `body_true`, `body_false`, `body_raise`) are killed by
|
|
211
|
+
almost any test that runs the method, so a second line scores the run without
|
|
212
|
+
them, and shows them alone. The headline score and `--threshold` still count every
|
|
213
|
+
mutant. The JSON report carries both splits under `saboteur.scores`
|
|
214
|
+
(`all` and `withoutWholeBodyMutants`), and the HTML page prints them under
|
|
215
|
+
the score. The JSON also records why the run exited the way it did under
|
|
216
|
+
`saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`,
|
|
217
|
+
`errorMajority`, `incompleteUnderThreshold`). In `--agent` mode the cause
|
|
218
|
+
follows the agent verdict, so the report and the process exit classify the
|
|
219
|
+
run the same way: `ok` for clean, `survivors` for survivors, and otherwise
|
|
220
|
+
`errorMajority` when most mutants errored or `incomplete`.
|
|
221
|
+
|
|
180
222
|
For code-review surfaces, the JSON report also carries a `saboteur.findings`
|
|
181
223
|
section: a bounded, deterministically ordered list of survivors, coarse
|
|
182
224
|
mutants first (a surviving coarse mutant hints that no test pins the method's
|
|
@@ -188,6 +230,15 @@ Survivors accepted on the ignore list are listed separately with their
|
|
|
188
230
|
reason, and `survivorsTruncated` and `acceptedTruncated` count what the bound
|
|
189
231
|
left out; the full data stays in the schema's `files` section.
|
|
190
232
|
|
|
233
|
+
In diff mode (`--since`), a mutant is reported at the line where its statement
|
|
234
|
+
starts. A statement that spans several lines can start a line or two above the
|
|
235
|
+
line that changed. Under each survivor the terminal therefore prints
|
|
236
|
+
`changed line: path:N`, the changed line or lines (the first three) the mutant
|
|
237
|
+
touches, and each finding carries them as `changedLines`. Use `changedLines`,
|
|
238
|
+
not `line`, to put a note on a changed line of the pull-request diff. A
|
|
239
|
+
whole-body mutant of a method whose only change is its signature touches no
|
|
240
|
+
changed line and prints none.
|
|
241
|
+
|
|
191
242
|
Exit codes: 0 no survivors (or all on the ignore list), or the score meets
|
|
192
243
|
`--threshold`; 1 survivors, or the score is below `--threshold` or missing;
|
|
193
244
|
2 bad configuration or a missing source or test; 3 the tool failed (with
|
|
@@ -203,38 +254,126 @@ other errors and reused cached kills are not included. See
|
|
|
203
254
|
|
|
204
255
|
## ⚙️ Configuration
|
|
205
256
|
|
|
206
|
-
|
|
257
|
+
`config/saboteur.rb` in the project root; CLI flags override it. Keep the shared
|
|
207
258
|
config usable for whole-project runs and pass `--since` in the pull-request
|
|
208
|
-
job. A PR-only config may set `since
|
|
209
|
-
`since
|
|
210
|
-
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
259
|
+
job. A PR-only config may set `c.since = "origin/main"`; omit `since` (or use
|
|
260
|
+
`c.since = nil`) in the config for audits on main. Use your actual target branch.
|
|
261
|
+
|
|
262
|
+
```ruby
|
|
263
|
+
Saboteur.configure do |c|
|
|
264
|
+
c.include = ["lib/**/*.rb"]
|
|
265
|
+
c.exclude = ["lib/my_gem/version.rb"]
|
|
266
|
+
c.require = [] # boot files, e.g. ./config/environment
|
|
267
|
+
c.reeval = "nesting" # or "class_eval"; see docs/design.md 6.1
|
|
268
|
+
c.timeout_factor = 3 # per-mutant deadline = factor x its tests' recorded time
|
|
269
|
+
c.timeout_min = 5
|
|
270
|
+
c.max_memory_mb = 2048 # memory growth cap per mutant process tree
|
|
271
|
+
c.max_processes = 64
|
|
272
|
+
c.max_mutants_per_method = 100 # positive integer; raise it for a broader run
|
|
273
|
+
c.ignore_file = "config/saboteur_ignore.rb"
|
|
274
|
+
c.seed = 1 # Ruby's random seed in every test run
|
|
275
|
+
c.fingerprint_extra = [] # extra globs whose contents invalidate caches, e.g. ["data/*.csv"]
|
|
276
|
+
end
|
|
223
277
|
```
|
|
224
278
|
|
|
225
279
|
### Rails apps
|
|
226
280
|
|
|
227
|
-
The defaults target a plain gem, so a Rails
|
|
281
|
+
Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a Rails
|
|
282
|
+
app needs a few settings; the generator writes them for you.
|
|
283
|
+
|
|
284
|
+
**1. Add the gem**
|
|
285
|
+
|
|
286
|
+
```sh
|
|
287
|
+
bundle add saboteur --group test
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
**2. Generate the setup**
|
|
291
|
+
|
|
292
|
+
```sh
|
|
293
|
+
bundle exec rails generate saboteur:install
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets `RAILS_ENV=test`
|
|
297
|
+
for you) and a `.gitignore` entry for `.saboteur/`. It never overwrites an
|
|
298
|
+
existing file unless you pass `--force`, and never edits your Gemfile.
|
|
228
299
|
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
300
|
+
Prefer to do it by hand, or on a version without the generator? Put this in
|
|
301
|
+
`config/saboteur.rb`:
|
|
302
|
+
|
|
303
|
+
```ruby
|
|
304
|
+
Saboteur.configure do |c|
|
|
305
|
+
c.include = ["app/**/*.rb"]
|
|
306
|
+
c.require = ["./config/environment"] # boots the app
|
|
307
|
+
c.rails = true # reconnects ActiveRecord after fork, per-worker databases
|
|
308
|
+
c.jobs = 4 # optional; the default is 1
|
|
309
|
+
end
|
|
234
310
|
```
|
|
235
311
|
|
|
236
|
-
|
|
237
|
-
|
|
312
|
+
**3. Check the setup**
|
|
313
|
+
|
|
314
|
+
```sh
|
|
315
|
+
bin/saboteur doctor
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
Doctor boots your app, checks the config and the test database host, and runs
|
|
319
|
+
your suite twice before any mutation. Fix whatever it reports first. Turn
|
|
320
|
+
SimpleCov off for the run if you use it.
|
|
321
|
+
|
|
322
|
+
**4. Test what your branch changed**
|
|
323
|
+
|
|
324
|
+
```sh
|
|
325
|
+
bin/saboteur run --since main
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
It mutates only the methods changed since `main` and reports each survivor with
|
|
329
|
+
its diff: a change your tests did not notice. Write a test that fails on that
|
|
330
|
+
diff, then run again. Use `--since origin/main` in CI. Without the generator, use
|
|
331
|
+
`bundle exec saboteur` and `RAILS_ENV=test` instead of `bin/saboteur`.
|
|
332
|
+
|
|
333
|
+
saboteur puts the app's `spec/` directory on the load path before
|
|
334
|
+
processing `require`, matching rspec, so a boot file like a dummy app's
|
|
335
|
+
`spec/rails_helper.rb` can require its siblings.
|
|
336
|
+
|
|
337
|
+
With `jobs` above 1 on SQLite, each extra worker gets a copy of the test
|
|
338
|
+
database made through SQLite's backup API, so the copy is a consistent
|
|
339
|
+
snapshot. A relative `database:` path resolves against `Rails.root`, the
|
|
340
|
+
same as Rails 7.1+ does, for the copy and for its cleanup. The copy
|
|
341
|
+
checkpoints the source first (`PRAGMA wal_checkpoint(TRUNCATE)`) and
|
|
342
|
+
refuses to run when a reader keeps the WAL busy, because copying on
|
|
343
|
+
would silently miss committed rows. A source file that does not exist is
|
|
344
|
+
an error, not a skipped copy. An in-memory SQLite database (`:memory:`) is
|
|
345
|
+
never copied, renamed or disconnected. After the app boots, each in-memory
|
|
346
|
+
database is snapshotted to a temporary file through SQLite's backup API,
|
|
347
|
+
and every forked child restores the snapshot into its own fresh database -
|
|
348
|
+
a child cannot keep the parent's connection (ActiveRecord discards
|
|
349
|
+
inherited pools and sqlite3 closes inherited writable handles), so without
|
|
350
|
+
the snapshot it would see an empty database.
|
|
351
|
+
|
|
352
|
+
For a Rails engine, point `require` at the dummy app's environment -
|
|
353
|
+
`test/dummy/config/environment.rb`, usually loaded for you by
|
|
354
|
+
`spec/rails_helper.rb`. Only a dynamically created dummy app has no
|
|
355
|
+
environment file; there, require the helper that calls setup instead,
|
|
356
|
+
such as `DummyApp.setup` in `spec/rails_helper.rb`.
|
|
357
|
+
|
|
358
|
+
Bootsnap needs no extra setting: bytecode from its compile cache carries no
|
|
359
|
+
coverage counters, so saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its
|
|
360
|
+
own process when the variable is not already set (Bootsnap treats any set
|
|
361
|
+
value as off) and runs the eager load with Coverage active. App code is seen
|
|
362
|
+
by Coverage with the app unchanged.
|
|
363
|
+
|
|
364
|
+
With `jobs: 4` (or more), each extra worker gets a copy of every test
|
|
365
|
+
database, named `<database>_saboteur2` and up, made with
|
|
366
|
+
`CREATE DATABASE ... TEMPLATE` on Postgres or a file copy on SQLite.
|
|
367
|
+
saboteur drops the copies when the run ends, on a best-effort basis: a
|
|
368
|
+
drop that fails is reported as a warning and never changes the run's
|
|
369
|
+
outcome. A copy left behind - by a failed drop or a killed run - is
|
|
370
|
+
reclaimed by the next run, whose prepare starts from
|
|
371
|
+
`DROP DATABASE IF EXISTS`. MySQL is not supported for `jobs` above 1.
|
|
372
|
+
|
|
373
|
+
Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
|
|
374
|
+
`db/schema.rb` and `config/*.yml` (including `database.yml`) are part of the
|
|
375
|
+
cache fingerprint automatically. Use `fingerprint_extra` for any other file your
|
|
376
|
+
tests read.
|
|
238
377
|
|
|
239
378
|
### Skipped subjects and the mutant cap
|
|
240
379
|
|
|
@@ -261,11 +400,13 @@ mutants; the limit does not bound how many proposals a custom source creates.
|
|
|
261
400
|
|
|
262
401
|
Every survivor is printed with a digest. Accept it with a reason:
|
|
263
402
|
|
|
264
|
-
```
|
|
265
|
-
# .
|
|
266
|
-
|
|
267
|
-
mutation
|
|
268
|
-
|
|
403
|
+
```ruby
|
|
404
|
+
# config/saboteur_ignore.rb
|
|
405
|
+
Saboteur.ignore do |i|
|
|
406
|
+
i.mutation "1a2b3c4d5e6f7a8b",
|
|
407
|
+
subject: "Billing::Invoice#total",
|
|
408
|
+
reason: "equivalent: amounts are always whole cents"
|
|
409
|
+
end
|
|
269
410
|
```
|
|
270
411
|
|
|
271
412
|
An entry without a reason is rejected.
|
|
@@ -320,6 +461,46 @@ No speed, type-safety or real-fault detection improvement is claimed. See
|
|
|
320
461
|
[the literal operator rationale](docs/literal-operators.md) for fault evidence
|
|
321
462
|
and limits, including string encoding/frozen-value behavior.
|
|
322
463
|
|
|
464
|
+
## Argument removal
|
|
465
|
+
|
|
466
|
+
Builtin removes one ordinary positional argument from a parenthesized call.
|
|
467
|
+
For `Model.includes(:a, :b, c: :d)`, the two proposals remove `:a` or `:b`,
|
|
468
|
+
without deleting the statement or changing `c: :d`. This lets a preload check
|
|
469
|
+
exercise missing associations instead of only a missing return value.
|
|
470
|
+
The operator and source name is `argument_removal`.
|
|
471
|
+
|
|
472
|
+
Only parenthesized calls are supported in this first version. Keyword hashes,
|
|
473
|
+
splats, argument forwarding, bracket indexing/assignment syntax and arguments containing
|
|
474
|
+
heredocs are unchanged. Explicit hash arguments count as positional arguments.
|
|
475
|
+
Blocks and keyword pairs remain intact; separators come from Prism tokens,
|
|
476
|
+
so commas inside strings or comments are not mistaken for separators.
|
|
477
|
+
Arid statements, parameter defaults and nested definitions are skipped.
|
|
478
|
+
|
|
479
|
+
These fine proposals follow existing builtin proposals, with the same validation,
|
|
480
|
+
deduplication and `max_mutants_per_method` cap. Earlier digests and relative order
|
|
481
|
+
stay unchanged for unchanged subjects. Extra proposals can change scores or be
|
|
482
|
+
omitted by the cap. Removing an argument can still raise an arity error; no
|
|
483
|
+
non-crashing guarantee is made. The regression uses a small preload-like receiver
|
|
484
|
+
and checks the missing associations, not an ActiveRecord SQL benchmark. See
|
|
485
|
+
[the validation notes](docs/argument-removal.md) for the pinned snapshot migration.
|
|
486
|
+
|
|
487
|
+
## Hash and keyword pair removal
|
|
488
|
+
|
|
489
|
+
Builtin removes one key/value association from a hash literal or keyword list.
|
|
490
|
+
`Model.includes(:a, :b, c: :d)` becomes `Model.includes(:a, :b )`, so a nested
|
|
491
|
+
preload guard can be exercised without deleting the whole statement. A sole
|
|
492
|
+
hash pair becomes `{}`. Operator and source name: `pair_removal`.
|
|
493
|
+
|
|
494
|
+
Both parenthesized and command calls are supported. Splats stay intact; pairs
|
|
495
|
+
containing heredocs, parameter defaults, arid statements and nested definitions
|
|
496
|
+
are skipped. Token-based separator edits preserve comments and blocks. Nested
|
|
497
|
+
hash values and callable bodies are visited. Removing a required keyword can
|
|
498
|
+
still crash; the regression uses a preload-like receiver, not an ActiveRecord
|
|
499
|
+
SQL benchmark. These fine proposals follow earlier builtin proposals and use
|
|
500
|
+
the usual validation, deduplication, order and `max_mutants_per_method` cap.
|
|
501
|
+
More proposals can change scores and cap coverage. See
|
|
502
|
+
[the validation notes](docs/pair-removal.md) for snapshot changes.
|
|
503
|
+
|
|
323
504
|
## 🔌 Extending
|
|
324
505
|
|
|
325
506
|
Public API, stable across minor versions:
|
|
@@ -334,9 +515,10 @@ Public API, stable across minor versions:
|
|
|
334
515
|
|
|
335
516
|
## 🌍 Where it has run
|
|
336
517
|
|
|
337
|
-
An early release (0.
|
|
518
|
+
An early release (0.4.0). Tested on the RSpec suites of plain gems
|
|
338
519
|
(dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
|
|
339
|
-
human-essentials on Postgres (4,080 mutants, 6 workers
|
|
520
|
+
human-essentials on Postgres (4,080 mutants, 6 workers; the 2026-10-10 run, before
|
|
521
|
+
the baseline check and the coarse-gate removal) and lobsters on SQLite,
|
|
340
522
|
both on macOS. Not yet tried on MySQL or older Rails versions. RSpec only, by
|
|
341
523
|
design. Results: [docs/validation](docs/validation).
|
|
342
524
|
|
|
@@ -344,17 +526,20 @@ design. Results: [docs/validation](docs/validation).
|
|
|
344
526
|
|
|
345
527
|
A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
|
|
346
528
|
1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
|
|
347
|
-
ran once on sampled files, so treat the numbers as indicative.
|
|
529
|
+
ran once on sampled files, so treat the numbers as indicative. The table was
|
|
530
|
+
measured at 2b6ec02, before the literal, filter and argument operators and
|
|
531
|
+
before the coarse-survivor gate was removed, so mutant counts and times will
|
|
532
|
+
change; a rerun is pending. mutineer is ahead
|
|
348
533
|
on number of operators and CI extras, and it supports Minitest.
|
|
349
534
|
|
|
350
535
|
| Metric | saboteur | mutineer |
|
|
351
536
|
|---|---|---|
|
|
352
|
-
| Fake kills (lobsters) | 0 of 20
|
|
537
|
+
| Fake kills (lobsters) | 0 of 20 sampled at random | 8 of 8 hand-checked (identical mutants, not a random sample) |
|
|
353
538
|
| Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
|
|
354
539
|
| Runs on the Postgres app | yes | no |
|
|
355
540
|
| Time per mutant, hashie | 0.008 s | 0.092 s |
|
|
356
541
|
| Peak memory, hashie (summed process RSS) | 595 MB | 471 MB |
|
|
357
|
-
| Choice of operators, CI features | fixed set, fewer |
|
|
542
|
+
| Choice of operators, CI features | fixed set, fewer | 22 operators, baseline, HTML, GitHub Action |
|
|
358
543
|
|
|
359
544
|
Full results, method and limitations:
|
|
360
545
|
[benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
|
|
@@ -380,12 +565,12 @@ variants, as before.
|
|
|
380
565
|
|
|
381
566
|
Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not
|
|
382
567
|
touch less memory, it keeps more of it shared between workers after fork, which
|
|
383
|
-
is what PSS measures. On
|
|
384
|
-
|
|
568
|
+
is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers on a
|
|
569
|
+
2 GB sandbox. Swap was not measured.
|
|
385
570
|
|
|
386
|
-
Wall time on uncontended runs
|
|
387
|
-
|
|
388
|
-
|
|
571
|
+
Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs
|
|
572
|
+
32s at jobs=6. These are noisy per-cell minima from historical runs and do not
|
|
573
|
+
show a direction either way. The shared sandbox had multi-minute CPU-starvation
|
|
389
574
|
windows during this session (walls up to 594s); affected runs are flagged via
|
|
390
575
|
sampler cadence and excluded from wall analysis. Absolute times are specific to
|
|
391
576
|
this 2-core sandbox. Environment, method and the contention log are on the
|
|
@@ -397,7 +582,7 @@ this 2-core sandbox. Environment, method and the contention log are on the
|
|
|
397
582
|
team workflow (GitHub Action, baseline, HTML report), then 1.0. saboteur stays
|
|
398
583
|
RSpec-only. Design
|
|
399
584
|
details are in [docs/design.md](docs/design.md), prior art in
|
|
400
|
-
[docs/prior-art.md](docs/prior-art.md), and
|
|
585
|
+
[docs/prior-art.md](docs/prior-art.md), and project status in
|
|
401
586
|
[docs/status.md](docs/status.md).
|
|
402
587
|
|
|
403
588
|
## 🤝 Contributing
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# SPDX-License-Identifier: MIT
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
require "rails/generators"
|
|
5
|
+
require "saboteur/install"
|
|
6
|
+
|
|
7
|
+
module Saboteur
|
|
8
|
+
module Generators
|
|
9
|
+
# rails generate saboteur:install [--force]
|
|
10
|
+
class InstallGenerator < Rails::Generators::Base
|
|
11
|
+
desc "Writes config/saboteur.rb, bin/saboteur and a .gitignore entry for a Rails + RSpec app."
|
|
12
|
+
|
|
13
|
+
def install_saboteur
|
|
14
|
+
installer = Saboteur::Install.new(root: destination_root, force: options[:force])
|
|
15
|
+
installer.call.each { |result| say_status(result.status, result.path) }
|
|
16
|
+
hint = installer.gemfile_hint
|
|
17
|
+
say("\n#{hint}") if hint
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|