saboteur 0.6.0 β 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +1024 -0
- data/README.md +199 -952
- data/exe/saboteur +34 -7
- data/lib/saboteur/baseline/workflow.rb +2 -1
- data/lib/saboteur/baseline_health.rb +40 -6
- data/lib/saboteur/cli/exit_status.rb +80 -4
- data/lib/saboteur/cli/help_options.rb +17 -0
- data/lib/saboteur/cli/history_command.rb +135 -0
- data/lib/saboteur/cli/presets.rb +40 -0
- data/lib/saboteur/cli/run_budget_options.rb +37 -0
- data/lib/saboteur/cli/run_reporting.rb +74 -5
- data/lib/saboteur/cli/selection_options.rb +31 -0
- data/lib/saboteur/cli/shard_options.rb +17 -0
- data/lib/saboteur/cli/source_set.rb +53 -0
- data/lib/saboteur/cli/survivors_command.rb +157 -0
- data/lib/saboteur/cli.rb +136 -98
- data/lib/saboteur/config.rb +163 -15
- data/lib/saboteur/config_file.rb +8 -0
- data/lib/saboteur/constant_reads.rb +182 -0
- data/lib/saboteur/constant_subject.rb +70 -0
- data/lib/saboteur/constant_subject_finder.rb +281 -0
- data/lib/saboteur/coverage_reads.rb +29 -0
- data/lib/saboteur/crash_report.rb +154 -0
- data/lib/saboteur/custom_map.rb +3 -1
- data/lib/saboteur/dirty_tree_warning.rb +128 -0
- data/lib/saboteur/doctor.rb +49 -1
- data/lib/saboteur/engine/baseline_guard.rb +136 -14
- data/lib/saboteur/engine/boot.rb +16 -1
- data/lib/saboteur/engine/canary.rb +41 -10
- data/lib/saboteur/engine/candidate_selection.rb +28 -1
- data/lib/saboteur/engine/candidates.rb +43 -3
- data/lib/saboteur/engine/coverage_guard.rb +29 -0
- data/lib/saboteur/engine/coverage_store.rb +79 -15
- data/lib/saboteur/engine/defined_order_prefix.rb +43 -0
- data/lib/saboteur/engine/diagnostics.rb +41 -2
- data/lib/saboteur/engine/diff_scope.rb +24 -5
- data/lib/saboteur/engine/distribution.rb +134 -8
- data/lib/saboteur/engine/early_stop_recording.rb +47 -0
- data/lib/saboteur/engine/gated_run.rb +35 -0
- data/lib/saboteur/engine/load_coverage.rb +45 -0
- data/lib/saboteur/engine/out_of_scope_guard.rb +21 -0
- data/lib/saboteur/engine/pipelines.rb +107 -0
- data/lib/saboteur/engine/result_cache.rb +67 -9
- data/lib/saboteur/engine/run_budget.rb +90 -0
- data/lib/saboteur/engine/run_fingerprint.rb +4 -0
- data/lib/saboteur/engine/sandbox_policy.rb +56 -11
- data/lib/saboteur/engine/scheduler.rb +222 -20
- data/lib/saboteur/engine/selection_announcement.rb +54 -0
- data/lib/saboteur/engine/shard.rb +69 -0
- data/lib/saboteur/engine/subject_selection.rb +373 -19
- data/lib/saboteur/engine/suite_files.rb +3 -1
- data/lib/saboteur/engine/test_guess.rb +438 -25
- data/lib/saboteur/engine/thread_warnings.rb +53 -0
- data/lib/saboteur/engine/unchanged_syntax.rb +17 -2
- data/lib/saboteur/engine/unscoped_suite_warning.rb +41 -0
- data/lib/saboteur/engine/verdict_check.rb +54 -18
- data/lib/saboteur/engine.rb +202 -158
- data/lib/saboteur/execution/external_runner.rb +471 -0
- data/lib/saboteur/execution/marker_sweep.rb +79 -0
- data/lib/saboteur/execution/peak_memory.rb +28 -0
- data/lib/saboteur/execution/process_table.rb +12 -0
- data/lib/saboteur/execution/process_tree.rb +65 -5
- data/lib/saboteur/execution/protocol.rb +2 -1
- data/lib/saboteur/execution/reevaluator.rb +68 -12
- data/lib/saboteur/execution/runner.rb +225 -24
- data/lib/saboteur/execution/sandbox.rb +21 -3
- data/lib/saboteur/execution/seatbelt.rb +226 -0
- data/lib/saboteur/execution/supervisor.rb +45 -13
- data/lib/saboteur/execution/worker_pool.rb +110 -8
- data/lib/saboteur/frameworks/sqlite_memory_database.rb +49 -0
- data/lib/saboteur/gates.rb +43 -2
- data/lib/saboteur/help.rb +108 -13
- data/lib/saboteur/ignore_list.rb +40 -7
- data/lib/saboteur/install.rb +4 -3
- data/lib/saboteur/integration/base.rb +7 -2
- data/lib/saboteur/integration/rspec.rb +175 -9
- data/lib/saboteur/mutation_source/argument_removal.rb +1 -1
- data/lib/saboteur/mutation_source/builtin.rb +44 -14
- data/lib/saboteur/mutation_source/chain_removal.rb +1 -1
- data/lib/saboteur/mutation_source/condition_negation.rb +1 -1
- data/lib/saboteur/mutation_source/constant_literals.rb +66 -0
- data/lib/saboteur/mutation_source/element_removal.rb +73 -0
- data/lib/saboteur/mutation_source/endpoint_replacement.rb +1 -1
- data/lib/saboteur/mutation_source/entry_edits.rb +44 -0
- data/lib/saboteur/mutation_source/filter_replacement.rb +1 -1
- data/lib/saboteur/mutation_source/hazard_guard.rb +129 -3
- data/lib/saboteur/mutation_source/interpolation_removal.rb +41 -0
- data/lib/saboteur/mutation_source/literal_replacement.rb +1 -1
- data/lib/saboteur/mutation_source/method_swap.rb +193 -0
- data/lib/saboteur/mutation_source/mutation.rb +5 -3
- data/lib/saboteur/mutation_source/operand_replacement.rb +54 -0
- data/lib/saboteur/mutation_source/operator_assignment.rb +1 -1
- data/lib/saboteur/mutation_source/operator_examples.rb +6 -1
- data/lib/saboteur/mutation_source/operator_registry.rb +7 -2
- data/lib/saboteur/mutation_source/operator_visit.rb +39 -0
- data/lib/saboteur/mutation_source/pair_removal.rb +13 -31
- data/lib/saboteur/mutation_source/range_replacement.rb +1 -1
- data/lib/saboteur/mutation_source/regex_replacement.rb +93 -0
- data/lib/saboteur/preview.rb +122 -13
- data/lib/saboteur/refinement_activations.rb +60 -0
- data/lib/saboteur/reporter/agent.rb +9 -1
- data/lib/saboteur/reporter/agent_commands.rb +47 -15
- data/lib/saboteur/reporter/agent_incomplete.rb +38 -2
- data/lib/saboteur/reporter/agent_summary.rb +4 -3
- data/lib/saboteur/reporter/findings.rb +5 -2
- data/lib/saboteur/reporter/history_policy.rb +11 -0
- data/lib/saboteur/reporter/history_reader.rb +47 -0
- data/lib/saboteur/reporter/html.rb +26 -4
- data/lib/saboteur/reporter/mutation_text.rb +6 -0
- data/lib/saboteur/reporter/next_steps.rb +14 -4
- data/lib/saboteur/reporter/score_history.rb +5 -1
- data/lib/saboteur/reporter/stryker.rb +40 -7
- data/lib/saboteur/reporter/survivor_classification.rb +101 -0
- data/lib/saboteur/reporter/survivor_hints.rb +3 -0
- data/lib/saboteur/reporter/survivor_list.rb +15 -9
- data/lib/saboteur/reporter/terminal.rb +74 -17
- data/lib/saboteur/result.rb +38 -7
- data/lib/saboteur/selection/coverage_map.rb +107 -11
- data/lib/saboteur/selection/dependency_shield.rb +80 -0
- data/lib/saboteur/selection/external_test_map.rb +27 -0
- data/lib/saboteur/selection/git_diff.rb +105 -13
- data/lib/saboteur/selection/layout_only_lines.rb +80 -0
- data/lib/saboteur/selection/recorder.rb +22 -4
- data/lib/saboteur/selection/source_scope.rb +22 -0
- data/lib/saboteur/selection/untracked_changes.rb +126 -0
- data/lib/saboteur/subject.rb +53 -7
- data/lib/saboteur/subject_finder.rb +173 -29
- data/lib/saboteur/text.rb +42 -0
- data/lib/saboteur/version.rb +1 -1
- data/lib/saboteur/working_tree.rb +38 -15
- data/lib/saboteur.rb +29 -1
- metadata +50 -2
data/README.md
CHANGED
|
@@ -10,996 +10,243 @@
|
|
|
10
10
|
[](https://rubygems.org/gems/saboteur)
|
|
11
11
|
[](LICENSE)
|
|
12
12
|
|
|
13
|
-
|
|
14
|
-
|
|
13
|
+
Line coverage tells you which code ran. It does not tell you if a test
|
|
14
|
+
would notice when that code breaks. Mutation testing does.
|
|
15
15
|
|
|
16
|
-
saboteur
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
change that no test could notice, a flaky test, or a limit of the tool, so
|
|
21
|
-
each one needs a human look.
|
|
16
|
+
saboteur makes one small change to your code at a time and runs only the
|
|
17
|
+
tests that touch it, so it is fast. If the tests still pass, you see the
|
|
18
|
+
exact change they missed. Every kill is re-checked against the unchanged
|
|
19
|
+
code, so a broken suite never counts as a kill.
|
|
22
20
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
```
|
|
21
|
+
**Docs, benchmarks and guides live on the site:
|
|
22
|
+
[vamsipavanmahesh.github.io/saboteur](https://vamsipavanmahesh.github.io/saboteur/)**
|
|
26
23
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
[design notes](docs/design.md) when you want the details. For adopting it on a team, see the
|
|
30
|
-
[best practices guide](docs/best-practices.md).
|
|
31
|
-
|
|
32
|
-
Declared external inputs are checked against cache and report paths using their
|
|
33
|
-
physical paths, including aliased parent directories. An overlap makes the run
|
|
34
|
-
fingerprint incomplete, so it cannot authorize cache reuse.
|
|
35
|
-
|
|
36
|
-
Internal run reports retain typed candidate-drop and cap evidence for gate
|
|
37
|
-
validation. These records do not change selection, scores, exit codes or the
|
|
38
|
-
public JSON format. Empty generation is unverified, not proof of equivalence.
|
|
39
|
-
|
|
40
|
-
The internal engine can opt in to fresh baseline evidence capture. It records
|
|
41
|
-
inputs before boot, planned candidate identities before scheduling and reconciled
|
|
42
|
-
terminal results after execution. Caps, uncovered work, invalid or empty
|
|
43
|
-
production, reused outcomes and changed inputs make that evidence incomplete.
|
|
44
|
-
The public opt-in `baseline propose`, `baseline accept` and `baseline check`
|
|
45
|
-
commands use this evidence to review and compare survivors. They never skip
|
|
46
|
-
accepted mutants or change their scores. See [Opt-in survivor baseline](#opt-in-survivor-baseline).
|
|
47
|
-
|
|
48
|
-
### The unchanged suite is checked first
|
|
49
|
-
|
|
50
|
-
Every run with selected subjects runs the loaded unchanged suite twice (the specs chosen with `--test`, if given) in fresh
|
|
51
|
-
supervised children before coverage lookup or incremental verdict reuse. This
|
|
52
|
-
costs two suite runs even on a cache hit, plus a coverage-building suite run on
|
|
53
|
-
a cache miss. Both baseline passes must be green, complete and in the same
|
|
54
|
-
RSpec order at the run seed. A failure exits 3, stops mutation, prints the failing example ids and the first ten lines of the
|
|
55
|
-
failure inline, and names the log and fix. Long phases
|
|
56
|
-
print their elapsed time about every 30 seconds, with the examples finished so far, the percentage and a rough estimate of the time left (it assumes equally long examples) once the suite child has logged its first count, and each phase's log holds
|
|
57
|
-
the suite's progress and full failure detail. Each suite pass
|
|
58
|
-
uses `coverage_timeout` and configured memory/process limits (the suite memory
|
|
59
|
-
budget is four times `max_memory_mb`, as for coverage). A suite child that has
|
|
60
|
-
already reported its result gets up to 5 seconds to finish exiting, under the
|
|
61
|
-
same deadline and memory/process limits as before; if it is still running then,
|
|
62
|
-
saboteur stops it and keeps the result, so a slow exit is not reported as a
|
|
63
|
-
failure. A child that ends by itself in that window keeps its own exit status. A child killed by `SIGKILL` is called a possible
|
|
64
|
-
out-of-memory kill only when the process tree was seen growing toward the
|
|
65
|
-
memory limit. Older benchmark numbers
|
|
66
|
-
need a rerun before they describe this extra fixed cost.
|
|
67
|
-
A whole-suite coverage build that records no lines for any included file
|
|
68
|
-
means the setup hid the app from Coverage (Bootsnap's compile cache is the
|
|
69
|
-
common cause), so the run fails with exit 3 instead of testing nothing, and
|
|
70
|
-
the empty map is not cached. With `--test` an empty map instead stays a
|
|
71
|
-
legitimate "no coverage" outcome, because the chosen specs may not touch the
|
|
72
|
-
selected sources.
|
|
73
|
-
Two passes detect observed instability, not every possible flaky test. See
|
|
74
|
-
[baseline stability](docs/baseline-stability.md) for limits and hook behaviour.
|
|
75
|
-
|
|
76
|
-
### Hazardous calls are not mutated, and a canary watches the tree
|
|
77
|
-
|
|
78
|
-
Mutants run in your checkout, so a mutant inside a destructive call could do
|
|
79
|
-
real damage: a string mutation inside `FileUtils.rm_rf(File.join(Dir.pwd,
|
|
80
|
-
"build"))` once turned the target into the working directory itself. The
|
|
81
|
-
hazard guard is a syntactic filter. It recognizes destructive calls written
|
|
82
|
-
as direct calls: `FileUtils.rm*`/`remove_entry*` (also as bare names under
|
|
83
|
-
`include FileUtils`), `File.delete`/`unlink`, `Dir.rmdir`/`delete`/`unlink`,
|
|
84
|
-
`rmtree`, `system`/`exec`/`spawn` (bare or on `Kernel`/`Process`), backticks
|
|
85
|
-
and `%x`, `IO.popen`, any `Open3` method, and `open` with a leading `|`. It
|
|
86
|
-
suppresses any mutant that would leave such a call in place while altering
|
|
87
|
-
code inside it or anywhere else in the same method. A mutant that removes
|
|
88
|
-
every hazardous call in the method still runs, provided its replacement
|
|
89
|
-
introduces no new hazard. The match is by name on the written call: calls
|
|
90
|
-
reached through aliases, variables, `send` or other indirection are not
|
|
91
|
-
recognized, so the guard cannot promise to find every destructive call. The
|
|
92
|
-
summary reports how many mutants were suppressed. Extend or replace the list
|
|
93
|
-
with `c.hazard` (same tag shapes: `"FileUtils.rm_rf"`, or a bare name like
|
|
94
|
-
`"system"`); `--allow-hazardous` runs the suppressed mutants anyway.
|
|
95
|
-
|
|
96
|
-
Behind the guard, a canary snapshots the tracked files (or every file,
|
|
97
|
-
outside a repository) before mutating and re-checks after each method's
|
|
98
|
-
mutants. If anything vanished, the run aborts with exit 3 and names the most
|
|
99
|
-
recent mutant's digest. That check happens after the damage and restores
|
|
100
|
-
nothing; it limits how far a bad mutant gets. Recover from your own backup,
|
|
101
|
-
a fresh checkout or a reclone - a deleted `.git` or uncommitted work is gone
|
|
102
|
-
unless you have another copy. The canary is the backstop for the destructive
|
|
103
|
-
forms the syntactic guard cannot see.
|
|
104
|
-
|
|
105
|
-
On Linux 6.2 and later (Landlock ABI 3) each mutant child also runs under
|
|
106
|
-
a Landlock sandbox: the child cannot create, write, rename, unlink or
|
|
107
|
-
truncate files in the project tree, except `tmp/`, the cache dir and the
|
|
108
|
-
system temp dir, plus anything in `c.sandbox_writable`. Every other path
|
|
109
|
-
is denied for writing, `/dev/shm` and `/dev/tty` included; `/dev/null`
|
|
110
|
-
stays writable. Inherited
|
|
111
|
-
descriptors already open for writing outside the granted dirs are closed
|
|
112
|
-
before the sandbox engages - stdio redirected to a regular file and files
|
|
113
|
-
already deleted from the tree included (stdio on a terminal, pipe or
|
|
114
|
-
`/dev/null` stays), and a descriptor whose target or flags cannot be
|
|
115
|
-
inspected is closed too rather than trusted. A `c.sandbox_writable` grant
|
|
116
|
-
that contains the project root or another granted dir warns, since it
|
|
117
|
-
defeats or duplicates the protection; reads and process execution are
|
|
118
|
-
unaffected.
|
|
119
|
-
This DOES change what your tests can write: a suite that writes elsewhere
|
|
120
|
-
in the project (`db/`, `log/`, ...) fails with EACCES in mutant children -
|
|
121
|
-
grant that directory with `c.sandbox_writable` or opt out with
|
|
122
|
-
`c.sandbox = false`. Landlock covers the filesystem only - network,
|
|
123
|
-
sockets and IPC are never restricted - and a checkout inside the system
|
|
124
|
-
temp dir cannot be protected, because everything under it stays writable
|
|
125
|
-
so that real test suites keep working - the run warns when the effective
|
|
126
|
-
write set (every grant realpath'd, the cache dir and the system temp dir
|
|
127
|
-
included) contains the project root, however it happened: a checkout
|
|
128
|
-
inside the system temp dir, `cache_dir = "."`, a symlinked `tmp`, or an
|
|
129
|
-
overlapping `c.sandbox_writable` grant. `c.sandbox = "strict"` refuses to
|
|
130
|
-
run at all in that situation, since the sandbox would protect nothing.
|
|
131
|
-
The sandbox applies to every mutant
|
|
132
|
-
child and fails closed: a child that cannot restrict itself reports an
|
|
133
|
-
error rather than run, and a kernel probe that errors (as opposed to
|
|
134
|
-
cleanly reporting Landlock absent) refuses the run.
|
|
135
|
-
|
|
136
|
-
On kernels from 5.13 to 6.1 (Landlock ABI 1-2) `truncate()` cannot be
|
|
137
|
-
restricted, so the tree is not fully read-only and the claim above does
|
|
138
|
-
not hold; the run notes once that children run with the guard and canary
|
|
139
|
-
only, exactly as on kernels without Landlock - and as on non-glibc Linux
|
|
140
|
-
whose C library cannot be located (musl is tried by soname first).
|
|
141
|
-
`c.sandbox = "strict"` turns that degraded note into a refusal;
|
|
142
|
-
`c.sandbox = false` opts out.
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
## π Try it in a minute
|
|
146
|
-
|
|
147
|
-
Add saboteur to the test group of your project's Gemfile (Ruby 3.3+, RSpec):
|
|
24
|
+
<a id="60-second-quick-start"></a>
|
|
25
|
+
## 30-second quick start
|
|
148
26
|
|
|
149
|
-
|
|
150
|
-
gem "saboteur", group: :test
|
|
151
|
-
```
|
|
27
|
+
You need Ruby 3.3 or newer, RSpec, and Linux or macOS with `fork`.
|
|
152
28
|
|
|
153
29
|
```sh
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
```
|
|
30
|
+
# Install the gem:
|
|
31
|
+
gem install saboteur
|
|
157
32
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
plain gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
|
|
161
|
-
|
|
162
|
-
Start with one method. You get counts per status, a score, timing, and a
|
|
163
|
-
unified diff for every survivor, so you can read the exact change your tests
|
|
164
|
-
missed. The JSON report lands in `.saboteur/report.json`.
|
|
165
|
-
|
|
166
|
-
π Requirements: Ruby 3.3 or newer on Linux or macOS (it needs `fork`), and RSpec.
|
|
167
|
-
Minitest is not supported; see [Minitest](docs/minitest.md) for why and what it would take.
|
|
168
|
-
|
|
169
|
-
## π§ Why it is built the way it is
|
|
170
|
-
|
|
171
|
-
**It only runs tests that can matter.** Each mutant runs only the tests whose
|
|
172
|
-
line coverage touched the changed statement. Tests that killed earlier mutants
|
|
173
|
-
of the same method go first, then the fastest, and the first failure stops the
|
|
174
|
-
run. A change no test reaches is reported as "no coverage", not as a survivor. Branch coverage separates arms on the same line: an edit inside an arm no test took reports "no coverage (branch not taken)". Ruby does not expose `&&`/`||` right operands as branches, so those keep line-based selection. `case/in` pattern edits use the enclosing case and clause execution lines; edits in an untested clause body still stay uncovered. Whole-body replacements use every test of the method, including methods that start with a heredoc. Endless defs (`def x = ...`) are matched to tests through per-test method call coverage, since line coverage never counts their bodies. Methods defined inside blocks (`Class.new do`, `Data.define do`, `included do`) are not mutated yet; the run lists them as skipped, "defined inside a block".
|
|
175
|
-
|
|
176
|
-
**It runs coarse mutants first, then fine mutants.** Each method first gets
|
|
177
|
-
whole-body mutants: `nil` (`true`/`false` for predicates) unless the body is empty or
|
|
178
|
-
only logs, and `raise` unless a catch-all rescue would absorb it. Fine-grained mutants run
|
|
179
|
-
regardless of the coarse verdict, including a survivor or a reused verdict.
|
|
180
|
-
Deleting a body unnoticed does not prove its smaller changes are harmless.
|
|
181
|
-
Subjects with no tests are skipped before any mutants are generated, and each
|
|
182
|
-
method is capped at `max_mutants_per_method` mutants (100 by default). Running
|
|
183
|
-
fine mutants after coarse survivors can increase run time and change scores.
|
|
184
|
-
|
|
185
|
-
**It prepares memory before it forks.** Before workers fork, saboteur warms and
|
|
186
|
-
compacts the Ruby heap ([`ForkReadiness`](lib/saboteur/execution/fork_readiness.rb)),
|
|
187
|
-
which is meant to let workers share more pages with their parent and copy less.
|
|
188
|
-
Observed on the pinned hashie workload (paired runs, below): about 17-18% lower peak PSS at 2 and 6 workers. Not a general guarantee.
|
|
189
|
-
|
|
190
|
-
**It is aimed at pull requests.** `--since origin/main` mutates only the methods
|
|
191
|
-
your branch changed: fine-grained mutants on the changed lines, plus the
|
|
192
|
-
whole-body mutants of those methods. On one real change in saboteur's own history, that took a file
|
|
193
|
-
from 139 mutants to 7, and median wall time fell 22.9% (measured at d2b7abb, before the coarse-survivor gate was removed; a rerun is pending). The time saving is
|
|
194
|
-
smaller than the mutant count because building the coverage map is a fixed
|
|
195
|
-
cost, and this is one file, not a Rails app, so it is not a general speed
|
|
196
|
-
claim. The full method, raw timings and
|
|
197
|
-
caveats are in [bench/since](bench/since/README.md).
|
|
198
|
-
|
|
199
|
-
**It checks its own kills.** After a method's mutants run, saboteur re-runs the
|
|
200
|
-
killing tests against the *unchanged* method. If they fail there too, the kill
|
|
201
|
-
is not credited and becomes an error. In one comparison on lobsters, 0 of 20
|
|
202
|
-
randomly sampled kills were fake. Every mutant runs in a supervised child with deadline,
|
|
203
|
-
memory and process limits, and saboteur tracks and kills the processes it
|
|
204
|
-
starts.
|
|
205
|
-
|
|
206
|
-
**It is open.** The report is JSON in the Stryker mutation-testing-report
|
|
207
|
-
schema, and you can add your own mutants through a small public interface.
|
|
208
|
-
|
|
209
|
-
## π οΈ Usage
|
|
33
|
+
# Or add it to the Gemfile:
|
|
34
|
+
bundle add saboteur --group test
|
|
210
35
|
|
|
211
|
-
|
|
212
|
-
bundle exec saboteur run
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
bundle exec saboteur
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
bundle exec saboteur
|
|
36
|
+
# Either way:
|
|
37
|
+
bundle exec saboteur run
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The first run builds a coverage map of which tests touch which code. The
|
|
41
|
+
map is cached and reused while your code and tests stay unchanged, so
|
|
42
|
+
later runs start fast.
|
|
43
|
+
|
|
44
|
+
On a branch, `bundle exec saboteur pr` mutates only the methods you
|
|
45
|
+
changed relative to `main`, with one baseline pass and a ten-minute
|
|
46
|
+
budget. If the budget stops the run, it exits 4 and prints `partial: X
|
|
47
|
+
of Y mutants not run` (or says the count is unknown when planning did
|
|
48
|
+
not finish). Pass `--max-duration-ok` to accept a partial verdict; the
|
|
49
|
+
warning stays, and other failure gates still apply. Pass `--since` for a
|
|
50
|
+
different base, for example `bundle exec saboteur pr --since master`.
|
|
51
|
+
|
|
52
|
+
Setup, configuration and CI:
|
|
53
|
+
[the quickstart](https://vamsipavanmahesh.github.io/saboteur/quickstart.html).
|
|
54
|
+
|
|
55
|
+
## Why RSpec-only
|
|
56
|
+
|
|
57
|
+
A mutation verdict is only worth anything if the test selection behind it
|
|
58
|
+
is exact. RSpec's example structure lets saboteur map each method to the
|
|
59
|
+
examples that cover it, rerun just those against each change, and re-check
|
|
60
|
+
every kill against the code before the change. One framework, done
|
|
61
|
+
properly - that is the bet. Minitest is not supported today;
|
|
62
|
+
[why, and what it would take](docs/minitest.md).
|
|
63
|
+
|
|
64
|
+
## What a run tells you
|
|
65
|
+
|
|
66
|
+
```text
|
|
67
|
+
saboteur 0.7.0: 1 subjects, 10 mutants, wall 0.70 s, seed 1
|
|
68
|
+
killed 8 survived 2 timeout 0 resource limit 0 error 0 score 80.0 %
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Each survivor comes with its diff, the tests that ran, and a triage label
|
|
72
|
+
(`likely_equivalent`, `arid`, `missing_test`) to sort what to read first.
|
|
73
|
+
The labels are heuristics for reading order, never a diagnosis, and they
|
|
74
|
+
never change the score.
|
|
75
|
+
|
|
76
|
+
A run on a tree with uncommitted changes opens with a warning that splits
|
|
77
|
+
the count into changes inside the mutation scope, outside it, and
|
|
78
|
+
saboteur's own setup files. Mutants run your code, so run on a clean,
|
|
79
|
+
committed tree or a CI checkout.
|
|
80
|
+
|
|
81
|
+
Code that only runs while the app loads cannot be judged by replacing the
|
|
82
|
+
method later, so it reports as `ran at load`, outside the score. The same
|
|
83
|
+
holds for def-time macros - a `deprecate`-style helper, a `before_action`
|
|
84
|
+
block, any DSL whose only caller is a class body: the closure captured at
|
|
85
|
+
load keeps the pristine definition, so the boot's def-call counters mark
|
|
86
|
+
the macro itself `ran at load` instead of letting its mutants survive.
|
|
87
|
+
|
|
88
|
+
## What you get
|
|
89
|
+
|
|
90
|
+
- **Fast pull-request runs.** `--since main` scopes the run to the methods
|
|
91
|
+
you changed, refuses (exit 2) when the resolved range has no changes -
|
|
92
|
+
`saboteur pr $GITHUB_SHA` on a checkout at the merge commit diffs the
|
|
93
|
+
merge against itself - and fails loudly (exit 3) when the changed code
|
|
94
|
+
cannot be verified - for example when every changed method is skipped
|
|
95
|
+
because a refinement is active where it sits. A changed method skipped
|
|
96
|
+
for no coverage names the specs the run loaded; when its coverage
|
|
97
|
+
lives in a sibling spec file, pass `--test spec/models` (the directory
|
|
98
|
+
holding it) to load it. The GitHub Action runs on every pull request
|
|
99
|
+
and caches the coverage map between runs. `--shard K/N` splits one
|
|
100
|
+
planned inventory deterministically across CI jobs; each shard reports
|
|
101
|
+
partial evidence.
|
|
102
|
+
- **Honest verdicts.** Every kill is verified against the unchanged
|
|
103
|
+
method; a kill the suite cannot confirm stays out of the score.
|
|
104
|
+
Suspect a flaky test? `--flaky-check survivors` confirms each new
|
|
105
|
+
survivor with one reordered rerun (`decisive` also confirms fresh
|
|
106
|
+
kills); a verdict the rerun disagrees with becomes `unverified`,
|
|
107
|
+
named with its tests, and stays out of the score. A spec
|
|
108
|
+
file whose examples depend on earlier ones is retried in defined order
|
|
109
|
+
before a kill is rejected, so its mutants are still scored.
|
|
110
|
+
- **Reports for people and tools.** Terminal, HTML, a Stryker-compatible
|
|
111
|
+
JSON report, and a stable agent JSON API with a deprecation policy.
|
|
112
|
+
- **A plan before the run.** `saboteur run --dry-run --format json` prints
|
|
113
|
+
the versioned [mutant inventory](docs/dry-run-json.md) - every planned
|
|
114
|
+
mutant with its stable digest and edits - without booting the suite.
|
|
115
|
+
- **A review workflow.** An ignore list with reasons and triage states,
|
|
116
|
+
an opt-in survivor baseline, score thresholds and score history.
|
|
117
|
+
`saboteur survivors .saboteur/report.json` browses survivors in the
|
|
118
|
+
terminal straight from the JSON report - list, filter
|
|
119
|
+
(`--file`/`--operator`/`--triage`/`--ranked`), and `--show DIGEST`
|
|
120
|
+
for the diff, the exact `run --id` replay command and an ignore-file
|
|
121
|
+
entry template. It never boots your project or edits the ignore file.
|
|
122
|
+
- **A local README badge.** `saboteur badge` writes Shields endpoint JSON
|
|
123
|
+
from the score history and `saboteur trend` prints the score delta -
|
|
124
|
+
no hosted service:
|
|
125
|
+
|
|
126
|
+
```md
|
|
127
|
+

|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Incomplete runs are refused as badge evidence unless `--mark-partial`
|
|
131
|
+
labels them, and the trend never compares runs with different
|
|
132
|
+
selection, inventory or operator policy.
|
|
133
|
+
- **Safety controls.** A hazard guard, canary values, and time, memory
|
|
134
|
+
and process-count limits on every mutant in the default in-process
|
|
135
|
+
mode; a Landlock sandbox on Linux
|
|
136
|
+
6.2 and newer.
|
|
137
|
+
- **Extension points.** Add your own mutation operators through the
|
|
138
|
+
public interface.
|
|
139
|
+
|
|
140
|
+
Per-path score gates hold one subtree to a stricter floor: `c.score_gate
|
|
141
|
+
"app/payments/**/*", min_score: 95` in config/saboteur.rb (repeatable). Each
|
|
142
|
+
gate scores only the mutants under its pattern and fails the run under its
|
|
143
|
+
floor; a gate with nothing executed under it passes vacuously and says so.
|
|
144
|
+
See
|
|
145
|
+
[choosing and raising a threshold](docs/best-practices.md#choosing-and-raising-a-threshold)
|
|
146
|
+
for how to pick a number.
|
|
147
|
+
|
|
148
|
+
## Honest limits
|
|
149
|
+
|
|
150
|
+
- saboteur is RSpec-only and needs `fork`: no Windows, no Minitest.
|
|
151
|
+
- A survivor is review evidence, not proof of a bug in your tests. It can
|
|
152
|
+
be a missing test, an equivalent change, a flaky test, or a limit of
|
|
153
|
+
the tool. Read it before you write a test for it.
|
|
154
|
+
- `--since` and `--incremental` are iteration aids. Run the whole project
|
|
155
|
+
on main and before a release.
|
|
156
|
+
- The hazard guard is syntactic, sandboxing varies by platform (Landlock
|
|
157
|
+
on Linux 6.2+; on macOS mutants run unsandboxed unless `c.sandbox = "seatbelt"`
|
|
158
|
+
opts into a probe-gated Seatbelt profile; see
|
|
159
|
+
[the macOS Seatbelt sandbox](docs/guides/macos-seatbelt-sandbox.md)), and network access is
|
|
160
|
+
never restricted. Stub real APIs and use a container when that matters.
|
|
161
|
+
|
|
162
|
+
### Custom spec mapping
|
|
163
|
+
|
|
164
|
+
When the naming guess does not fit the project's layout, teach the mapping
|
|
165
|
+
once in `config/saboteur_custom_map.rb` instead of passing `--test` on every
|
|
166
|
+
run:
|
|
167
|
+
|
|
168
|
+
```ruby
|
|
169
|
+
SABOTEUR_CUSTOM_MAP = {
|
|
170
|
+
"lib/aasm/core/state.rb" => ["spec/unit/state_spec.rb"],
|
|
171
|
+
"app/jobs/**/*.rb" => ["spec/jobs/**/*_spec.rb"],
|
|
172
|
+
%r{\Aapp/jobs/(.*)\.rb\z} => ['spec/jobs/\1_spec.rb'],
|
|
173
|
+
"app/generated/**/*.rb" => [], # intentionally untested
|
|
174
|
+
}.freeze
|
|
226
175
|
```
|
|
227
176
|
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
files are the sources) replaces only the files RSpec runs. Other
|
|
234
|
-
`rspec_args` and `.rspec` options such as `--require` stay. The coverage map
|
|
235
|
-
is built from only those specs, each mutant runs only the ones that cover it,
|
|
236
|
-
and the selection is part of the coverage and `--incremental` cache keys.
|
|
237
|
-
|
|
238
|
-
`--threshold PERCENT` (0 to 100, or `c.threshold` in `config/saboteur.rb`) is for CI.
|
|
239
|
-
The run exits 1 when the score is below PERCENT and 0 when it is at or above,
|
|
240
|
-
even with survivors. It fails closed: a run that tested no mutant has no score
|
|
241
|
-
and exits 1, and a run where any mutant errored or hit a resource limit exits 3,
|
|
242
|
-
because the score covers only part of the run. Without a threshold, any
|
|
243
|
-
unignored survivor exits 1. Either way, a run where more than half of the
|
|
244
|
-
executed mutants errored exits 3: errored mutants are outside the score,
|
|
245
|
-
so such a run tested almost nothing.
|
|
246
|
-
|
|
247
|
-
`--max-timeouts N` and `--max-errors N` (or `c.max_timeouts` / `c.max_errors`)
|
|
248
|
-
are two more CI gates. Each takes a count or a percentage of the executed
|
|
249
|
-
mutants (`--max-errors 5%`). The run exits 1 when the count (or share) is
|
|
250
|
-
above the limit, with `exitCause` `maxTimeouts` or `maxErrors` in the JSON report.
|
|
251
|
-
`--threshold` already compares the covered score, `(killed + timeout) /
|
|
252
|
-
(killed + survived + timeout)`, so no separate score flag exists. With
|
|
253
|
-
`--max-errors`, errored mutants are judged by that limit instead of making a
|
|
254
|
-
`--threshold` run exit 3. Survivors and a score below the threshold keep
|
|
255
|
-
their own causes when they fail too. The gates cannot be combined with `--agent`.
|
|
256
|
-
The GitHub Action has no input for them yet.
|
|
257
|
-
|
|
258
|
-
Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
|
|
259
|
-
`--config PATH`, `--integration NAME`, `--reeval class_eval|nesting`,
|
|
260
|
-
`--json PATH`, `--no-cache` (rebuild the coverage map), `--allow-focus`.
|
|
261
|
-
|
|
262
|
-
A stray `fit`, `fdescribe` or `:focus` example, with `filter_run_when_matching :focus`, makes RSpec skip
|
|
263
|
-
every other example. The baseline stays green and the score would cover only the focused ones, so
|
|
264
|
-
saboteur exits 2 and names the focused examples. Remove the focus, or pass `--allow-focus` (config
|
|
265
|
-
`c.allow_focus = true`) to run the narrowed suite. Filters you name in `rspec_args` or `.rspec`, such
|
|
266
|
-
as `--tag slow`, and example selectors such as `--test spec/a_spec.rb:12`, are on purpose and are not refused. The
|
|
267
|
-
check compares what RSpec runs with what it would run without the stray focus, so examples RSpec leaves out anyway
|
|
268
|
-
(`:if` / `:unless`, `--tag`, exclusions, selectors in another file) never count as hidden.
|
|
269
|
-
|
|
270
|
-
For pull requests, start with `--since` against your target branch. This is a
|
|
271
|
-
workflow recommendation, not a CLI default: plain `run` still checks the whole
|
|
272
|
-
configured project. To scope the specs as well, pass them with `--test` and no
|
|
273
|
-
source operands: `bundle exec saboteur run --since origin/main --test spec/billing_spec.rb`.
|
|
274
|
-
With `--since` and no `--test`, the run warns before it starts that the two baseline passes
|
|
275
|
-
run the loaded suite (a cold coverage cache adds one more pass), and how to
|
|
276
|
-
scope them.
|
|
277
|
-
|
|
278
|
-
`--since origin/main --guess-tests` does that scoping for you: it picks the spec
|
|
279
|
-
named after each changed source file (`lib/a/b.rb` runs `spec/a/b_spec.rb`; `app/`
|
|
280
|
-
files and `spec/lib/` work too) and runs only those for the baseline passes, the
|
|
281
|
-
coverage pass and every mutant, as `--test` would. It is a naming guess, not
|
|
282
|
-
coverage: a spec that reaches the changed file only through another file does not
|
|
283
|
-
run: a mutant only that spec would kill can survive, and a line that none of the
|
|
284
|
-
guessed specs executes is reported as NoCoverage. The agent JSON records the guessed specs as its
|
|
285
|
-
`tests`, so its follow-up commands keep the same `--test` scope (with `--no-guess-tests`, so a `guess_tests`
|
|
286
|
-
in the config file cannot conflict) instead of widening to the whole suite. If any changed file has no such
|
|
287
|
-
spec, or nothing changed, the whole suite runs as before and the run says so.
|
|
288
|
-
`--guess-tests` needs `--since` and cannot be combined with `--test`. Keep whole-project runs on main: diff mode does not check every method affected by a
|
|
289
|
-
dependency change. [The pull-request guide](docs/pull-requests.md) covers
|
|
290
|
-
checkout history, a GitHub Actions example, configuration and limits.
|
|
291
|
-
|
|
292
|
-
Doctor warns when an ignore subject matches no method discovered in the included
|
|
293
|
-
source files. Check the exact `Class#method` or `Class.method` form and the
|
|
294
|
-
include/exclude settings; warnings never edit or delete ignore entries.
|
|
295
|
-
A malformed ignore file fails doctor with a config error before target boot.
|
|
296
|
-
|
|
297
|
-
`saboteur doctor` prints Ruby, coverage, integration, Rails/SimpleCov and boot-file
|
|
298
|
-
checks. Add `--suite` to also run the unchanged suite twice with the chosen seed
|
|
299
|
-
(`--test PATH`, repeatable, limits those runs to some specs and implies `--suite`),
|
|
300
|
-
since that takes hours on a large suite. It writes no
|
|
301
|
-
mutation reports or coverage/verdict caches. Target boot files and tests still
|
|
302
|
-
run, so their normal side effects remain. See the [doctor guide](docs/guides/doctor.md).
|
|
303
|
-
|
|
304
|
-
`--id` runs exactly one current mutant with its coverage-selected tests, bypassing
|
|
305
|
-
per-method caps, ignore suppression and incremental verdict reuse.
|
|
306
|
-
It still honors `--subject` and `--since` changed-line selection, and the
|
|
307
|
-
hazard guard: a digest that names a suppressed mutant is refused unless
|
|
308
|
-
`--allow-hazardous` is also given. See the
|
|
309
|
-
[replay guide](docs/guides/replay.md) for digest format, checks and limits.
|
|
310
|
-
|
|
311
|
-
Every test run starts from the same random seed (1 by default), so tests that
|
|
312
|
-
use `rand` see the same values in each mutant's run and in the check that
|
|
313
|
-
confirms a kill. The report prints the seed; pass it to `--seed` to repeat a
|
|
314
|
-
run.
|
|
315
|
-
|
|
316
|
-
`--incremental` reuses an earlier verdict only while the method, the whole
|
|
317
|
-
file it lives in, the mutant, its covering tests and their files, the lockfile
|
|
318
|
-
and the config are all unchanged. So a constant or helper changed in the same
|
|
319
|
-
file is seen. It cannot see a change in another file that the method uses (a
|
|
320
|
-
method it calls, a constant defined elsewhere), so run without it now and then
|
|
321
|
-
(and in CI on the main branch).
|
|
322
|
-
|
|
323
|
-
Output: survivors first, then a short summary last, so the summary stays on
|
|
324
|
-
screen. Survivors are grouped by file, then method, then line. Each line is
|
|
325
|
-
printed once as `path:line in Subject`, with the source line and the tests that
|
|
326
|
-
covered it, then one compact line per mutation: `operator original -> replacement digest`,
|
|
327
|
-
and a hint on what kind of test would kill it. The summary has the counts per
|
|
328
|
-
status, the score, a table of scores per file (worst first) and a one-line
|
|
329
|
-
legend. `--verbose` adds the score split, mean time, whole-body score and
|
|
330
|
-
kill causes. Colour is on when stdout is a terminal and `NO_COLOR` is not set;
|
|
331
|
-
`--color` and `--no-color` override that. While mutants run, stderr shows
|
|
332
|
-
`done/total (percent) eta`: a live counter on a terminal, and dots with a
|
|
333
|
-
counter every 50 mutants otherwise. Skipped and reused mutants are not
|
|
334
|
-
counted, so the count can end below the total.
|
|
335
|
-
The tests that covered the survivor's line and
|
|
336
|
-
still passed with the mutant applied are listed (the first five, likeliest killers
|
|
337
|
-
first; the rest are under `coveredBy` in the JSON report). The missing
|
|
338
|
-
check may belong in one of them. Line coverage can overstate which tests
|
|
339
|
-
exercise a line, so treat the list as a lead. A survivor reused with
|
|
340
|
-
`--incremental` was not run again, and the list says its tests passed in an
|
|
341
|
-
earlier run. `--fail-fast` (serial runs only) stops after the first subject
|
|
342
|
-
that has a survivor the ignore list does not accept. The report then covers
|
|
343
|
-
only the subjects that ran: the terminal says which subjects did not run, the
|
|
344
|
-
JSON report has an `earlyStop` field naming them (`exitCause` is `failFast`),
|
|
345
|
-
`--agent` reports status `partial` with the reason `fail_fast`, and the run
|
|
346
|
-
always exits 1, so no `--threshold` can pass it. It cannot be used with
|
|
347
|
-
`baseline` evidence. Survivors that share a line are printed together only
|
|
348
|
-
when their covering tests, reuse status and changed lines are identical;
|
|
349
|
-
otherwise each distinct set of evidence is printed above its own mutations.
|
|
350
|
-
Hints are suggestions, not diagnoses: a survivor can also be an equivalent
|
|
351
|
-
mutant. The HTML page shows the same list in the
|
|
352
|
-
survivor's notes. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
|
|
353
|
-
report (one file, no scripts, no network) to `.saboteur/report.html`. Change
|
|
354
|
-
the paths with `--json` / `--html` or `c.json_report` / `c.html_report` in
|
|
355
|
-
`config/saboteur.rb`; set either to `nil` to turn that report off. The two paths
|
|
356
|
-
must differ. The HTML page shows the covered mutation score
|
|
357
|
-
`(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage,
|
|
358
|
-
RanAtLoad and Unreachable out and keeps survivors accepted on the ignore list
|
|
359
|
-
in the denominator. A tool
|
|
360
|
-
that reads the JSON report may show a different score, because the schema counts
|
|
361
|
-
NoCoverage as undetected.
|
|
362
|
-
|
|
363
|
-
`--matrix` runs every covering test against every mutant instead of stopping at
|
|
364
|
-
the first failing one, so a killed mutant lists all of its killers. After the
|
|
365
|
-
survivors the report prints two lists: blind tests (they covered mutants and
|
|
366
|
-
killed none) and redundant tests (they killed some, and every one of those
|
|
367
|
-
mutants had another killer). The run is slower, by up to the number of covering
|
|
368
|
-
tests per mutant. `--matrix` cannot be combined with `--incremental`, because a
|
|
369
|
-
reused verdict lists only the first killer. A redundant test is a lead, not a
|
|
370
|
-
verdict: it may guard behavior no selected mutant touches. A selection with
|
|
371
|
-
`--since` or `--subject` describes only the selected methods.
|
|
372
|
-
|
|
373
|
-
[Scores](docs/scores.md) defines each number on this page: the covered score, the
|
|
374
|
-
score a Stryker dashboard shows, the assertion and crash split, and what makes a run
|
|
375
|
-
incomplete.
|
|
376
|
-
|
|
377
|
-
The score is split by how the mutant was killed. A crash kill (a `NoMethodError`,
|
|
378
|
-
a `TypeError`, a parse error) shows the code ran and broke, not that a test
|
|
379
|
-
checked a value. The terminal prints
|
|
380
|
-
`score 100.0 % (assertion 50.0 %, crash 50.0 %)`: each share is a part of
|
|
381
|
-
tested mutants, so the parts add up to the score. Timeouts and kills with no
|
|
382
|
-
recorded cause (a verdict reused from an older run, a test that called `exit`,
|
|
383
|
-
a failure outside any example) are named when there are some. Whole-body
|
|
384
|
-
mutants (`body_nil`, `body_true`, `body_false`, `body_raise`) are killed by
|
|
385
|
-
almost any test that runs the method, so a second line scores the run without
|
|
386
|
-
them, and shows them alone. The headline score and `--threshold` still count every
|
|
387
|
-
mutant. The JSON report carries both splits under `saboteur.scores`
|
|
388
|
-
(`all` and `withoutWholeBodyMutants`), and the HTML page prints them under
|
|
389
|
-
the score. The JSON also records why the run exited the way it did under
|
|
390
|
-
`saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`,
|
|
391
|
-
`errorMajority`, `incompleteUnderThreshold`). In `--agent` mode the cause
|
|
392
|
-
follows the agent verdict, so the report and the process exit classify the
|
|
393
|
-
run the same way: `ok` for clean, `survivors` for survivors, and otherwise
|
|
394
|
-
`errorMajority` when most mutants errored or `incomplete`.
|
|
395
|
-
|
|
396
|
-
For code-review surfaces, the JSON report also carries a `saboteur.findings`
|
|
397
|
-
section: a bounded, deterministically ordered list of survivors, coarse
|
|
398
|
-
mutants first (a surviving coarse mutant hints that no test pins the method's
|
|
399
|
-
behavior - it can also be a re-evaluation limit or an equivalent mutant),
|
|
400
|
-
then file, line and digest. Each finding holds the file, line, digest,
|
|
401
|
-
subject, operator, original and replacement a note on a pull-request diff
|
|
402
|
-
needs, plus one entry per edit for mutants that edit more than one spot.
|
|
403
|
-
Survivors accepted on the ignore list are listed separately with their
|
|
404
|
-
reason, and `survivorsTruncated` and `acceptedTruncated` count what the bound
|
|
405
|
-
left out; the full data stays in the schema's `files` section.
|
|
406
|
-
|
|
407
|
-
In diff mode (`--since`), a mutant is reported at the line where its statement
|
|
408
|
-
starts. A statement that spans several lines can start a line or two above the
|
|
409
|
-
line that changed. Under each survivor the terminal therefore prints
|
|
410
|
-
`changed line: path:N`, the changed line or lines (the first three) the mutant
|
|
411
|
-
touches, and each finding carries them as `changedLines`. Use `changedLines`,
|
|
412
|
-
not `line`, to put a note on a changed line of the pull-request diff. A
|
|
413
|
-
whole-body mutant of a method whose only change is its signature touches no
|
|
414
|
-
changed line and prints none.
|
|
415
|
-
|
|
416
|
-
New to survivors? The [reading survivors guide](docs/guides/reading-survivors.md) walks one from the diff to the test that kills it.
|
|
417
|
-
|
|
418
|
-
Exit codes: 0 no survivors (or all on the ignore list), or the score meets
|
|
419
|
-
`--threshold`; 1 survivors, or the score is below `--threshold` or missing;
|
|
420
|
-
2 bad configuration or a missing source or test; 3 the tool failed (with
|
|
421
|
-
`--threshold`, also any errored or resource-limited mutant).
|
|
177
|
+
A matching entry wins over the naming convention: the file stops being
|
|
178
|
+
unmatched and its mapped specs run as the inferred ones. Keys are exact
|
|
179
|
+
paths, globs or Regexps; an empty array marks the file intentionally
|
|
180
|
+
untested, so it needs no inferred spec. The file loads before the app
|
|
181
|
+
boots, so keep it plain Ruby. `saboteur doctor` validates it.
|
|
422
182
|
|
|
423
183
|
### Rejected kills
|
|
424
184
|
|
|
425
|
-
The terminal
|
|
426
|
-
`
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
`config/saboteur.rb` in the project root; CLI flags override it. Keep the shared
|
|
434
|
-
config usable for whole-project runs and pass `--since` in the pull-request
|
|
435
|
-
job. A PR-only config may set `c.since = "origin/main"`; omit `since` (or use
|
|
436
|
-
`c.since = nil`) in the config for audits on main. Use your actual target branch.
|
|
437
|
-
|
|
438
|
-
```ruby
|
|
439
|
-
Saboteur.configure do |c|
|
|
440
|
-
c.include = ["lib/**/*.rb"]
|
|
441
|
-
c.exclude = ["lib/my_gem/version.rb"]
|
|
442
|
-
c.require = [] # boot files, e.g. ./config/environment
|
|
443
|
-
c.whole_file = [] # subject ids or globs to re-evaluate as a whole file (see below)
|
|
444
|
-
c.reeval = "nesting" # or "class_eval"; see docs/design.md 6.1
|
|
445
|
-
c.timeout_factor = 3 # per-mutant deadline = factor x its tests' recorded time
|
|
446
|
-
c.timeout_min = 5
|
|
447
|
-
c.max_memory_mb = 2048 # memory growth cap per mutant process tree
|
|
448
|
-
c.max_processes = 64
|
|
449
|
-
c.max_mutants_per_method = 100 # positive integer; raise it for a broader run
|
|
450
|
-
c.hazard = Saboteur::MutationSource::HazardGuard::DEFAULT_HAZARD # calls whose arguments are never mutated
|
|
451
|
-
c.sandbox = true # read-only project tree for mutant children (Linux 6.2+); "strict" refuses to run degraded
|
|
452
|
-
c.sandbox_writable = [] # extra dirs mutant children may write, e.g. ["db", "log"]
|
|
453
|
-
c.ignore_file = "config/saboteur_ignore.rb"
|
|
454
|
-
c.seed = 1 # Ruby's random seed in every test run
|
|
455
|
-
c.allow_focus = false # true runs a suite narrowed by fit/fdescribe/:focus instead of exit 2
|
|
456
|
-
c.fingerprint_extra = [] # extra globs whose contents invalidate caches, e.g. ["data/*.csv"]
|
|
457
|
-
end
|
|
458
|
-
```
|
|
185
|
+
The terminal lists them in a `Rejected kills` section and annotates the error
|
|
186
|
+
count (`error 3 (2 rejected kills)`); JSON flags each rejected mutant with
|
|
187
|
+
`rejectedKill: true` and keeps the `rejectedKills` count. These are fresh
|
|
188
|
+
per-mutant kill claims whose tests also fail with the method unchanged. They
|
|
189
|
+
stay errors outside the score. Timeout checks, other errors and reused cached
|
|
190
|
+
kills are not included. See [rejected kills](docs/rejected-kills.md) for
|
|
191
|
+
details.
|
|
459
192
|
|
|
460
193
|
### Rails apps
|
|
461
194
|
|
|
462
|
-
Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a
|
|
463
|
-
app needs a few settings; the generator writes them for you
|
|
464
|
-
|
|
465
|
-
**1. Add the gem**
|
|
195
|
+
Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a
|
|
196
|
+
Rails app needs a few settings; the generator writes them for you:
|
|
466
197
|
|
|
467
198
|
```sh
|
|
468
199
|
bundle add saboteur --group test
|
|
469
|
-
```
|
|
470
|
-
|
|
471
|
-
**2. Generate the setup**
|
|
472
|
-
|
|
473
|
-
```sh
|
|
474
200
|
bundle exec rails generate saboteur:install
|
|
475
201
|
```
|
|
476
202
|
|
|
477
|
-
This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets
|
|
478
|
-
for you) and a `.gitignore` entry for `.saboteur/`. It
|
|
479
|
-
existing file unless you pass `--force
|
|
203
|
+
This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets
|
|
204
|
+
`RAILS_ENV=test` for you) and a `.gitignore` entry for `.saboteur/`. It
|
|
205
|
+
never overwrites an existing file unless you pass `--force`.
|
|
480
206
|
|
|
481
|
-
|
|
482
|
-
|
|
207
|
+
Until a config names your sources, a run whose include selects no Ruby
|
|
208
|
+
file refuses with exit 2 and points at the install generator above -
|
|
209
|
+
a first CI run on an unconfigured tree fails loud instead of passing
|
|
210
|
+
with 0 subjects.
|
|
211
|
+
|
|
212
|
+
Prefer to do it by hand, or on a version without the generator? Put this
|
|
213
|
+
in `config/saboteur.rb`:
|
|
483
214
|
|
|
484
215
|
```ruby
|
|
485
216
|
Saboteur.configure do |c|
|
|
486
|
-
c.include = ["app/**/*.rb"]
|
|
217
|
+
c.include = ["app/**/*.rb"] # spec/test dirs vendored under lib/ are excluded by default
|
|
487
218
|
c.require = ["./config/environment"] # boots the app
|
|
488
219
|
c.rails = true # reconnects ActiveRecord after fork, per-worker databases
|
|
489
220
|
c.jobs = 4 # optional; the default is 1
|
|
490
221
|
end
|
|
491
222
|
```
|
|
492
223
|
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
```sh
|
|
496
|
-
bin/saboteur doctor
|
|
497
|
-
```
|
|
498
|
-
|
|
499
|
-
Doctor boots your app, checks the config and the test database host, and runs
|
|
500
|
-
your suite twice before any mutation. Fix whatever it reports first. Turn
|
|
501
|
-
SimpleCov off for the run if you use it.
|
|
502
|
-
|
|
503
|
-
The database check refuses any test database whose host is not local
|
|
504
|
-
(`localhost`, `127.0.0.1`, `::1`, `0.0.0.0` or a socket), so a remote
|
|
505
|
-
database is never mutated by accident. `0.0.0.0` counts as local: compose
|
|
506
|
-
setups write it into database URLs, and connecting to it reaches this
|
|
507
|
-
machine. For another disposable test database host, assert it yourself:
|
|
508
|
-
`c.allow_db_hosts = ["db.internal"]` in `config/saboteur.rb`.
|
|
509
|
-
|
|
510
|
-
**4. Test what your branch changed**
|
|
511
|
-
|
|
512
|
-
```sh
|
|
513
|
-
bin/saboteur run --since main
|
|
514
|
-
```
|
|
515
|
-
|
|
516
|
-
It mutates only the methods changed since `main` and reports each survivor with
|
|
517
|
-
its diff: a change your tests did not notice. Write a test that fails on that
|
|
518
|
-
diff, then run again. Use `--since origin/main` in CI. Without the generator, use
|
|
519
|
-
`bundle exec saboteur` and `RAILS_ENV=test` instead of `bin/saboteur`.
|
|
520
|
-
|
|
521
|
-
saboteur puts the app's `spec/` directory on the load path before
|
|
522
|
-
processing `require`, matching rspec, so a boot file like a dummy app's
|
|
523
|
-
`spec/rails_helper.rb` can require its siblings.
|
|
524
|
-
|
|
525
|
-
With `jobs` above 1 on SQLite, each extra worker gets a copy of the test
|
|
526
|
-
database made through SQLite's backup API, so the copy is a consistent
|
|
527
|
-
snapshot. A relative `database:` path resolves against `Rails.root`, the
|
|
528
|
-
same as Rails 7.1+ does, for the copy and for its cleanup. The copy
|
|
529
|
-
checkpoints the source first (`PRAGMA wal_checkpoint(TRUNCATE)`) and
|
|
530
|
-
refuses to run when a reader keeps the WAL busy, because copying on
|
|
531
|
-
would silently miss committed rows. A source file that does not exist is
|
|
532
|
-
an error, not a skipped copy. An in-memory SQLite database (`:memory:`) is
|
|
533
|
-
never copied, renamed or disconnected. After the app boots, each in-memory
|
|
534
|
-
database is snapshotted to a temporary file through SQLite's backup API,
|
|
535
|
-
and every forked child restores the snapshot into its own fresh database -
|
|
536
|
-
a child cannot keep the parent's connection (ActiveRecord discards
|
|
537
|
-
inherited pools and sqlite3 closes inherited writable handles), so without
|
|
538
|
-
the snapshot it would see an empty database.
|
|
539
|
-
|
|
540
|
-
For a Rails engine, point `require` at the dummy app's environment -
|
|
541
|
-
`test/dummy/config/environment.rb`, usually loaded for you by
|
|
542
|
-
`spec/rails_helper.rb`. Only a dynamically created dummy app has no
|
|
543
|
-
environment file; there, require the helper that calls setup instead,
|
|
544
|
-
such as `DummyApp.setup` in `spec/rails_helper.rb`.
|
|
545
|
-
|
|
546
|
-
Bootsnap needs no extra setting: bytecode from its compile cache carries no
|
|
547
|
-
coverage counters, so saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its
|
|
548
|
-
own process when the variable is not already set (Bootsnap treats any set
|
|
549
|
-
value as off) and runs the eager load with Coverage active. App code is seen
|
|
550
|
-
by Coverage with the app unchanged.
|
|
551
|
-
|
|
552
|
-
With `jobs: 4` (or more), each extra worker gets a copy of every test
|
|
553
|
-
database, named `<database>_saboteur2` and up, made with
|
|
554
|
-
`CREATE DATABASE ... TEMPLATE` on Postgres or a file copy on SQLite.
|
|
555
|
-
saboteur drops the copies when the run ends, on a best-effort basis: a
|
|
556
|
-
drop that fails is reported as a warning and never changes the run's
|
|
557
|
-
outcome. A copy left behind - by a failed drop or a killed run - is
|
|
558
|
-
reclaimed by the next run, whose prepare starts from
|
|
559
|
-
`DROP DATABASE IF EXISTS`. MySQL is not supported for `jobs` above 1.
|
|
560
|
-
|
|
561
|
-
Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
|
|
562
|
-
`db/schema.rb` and `config/*.yml` (including `database.yml`) are part of the
|
|
563
|
-
cache fingerprint automatically. Use `fingerprint_extra` for any other file your
|
|
564
|
-
tests read.
|
|
565
|
-
|
|
566
|
-
### Load-time code, captured methods and Sorbet signatures
|
|
567
|
-
|
|
568
|
-
Code that runs while the target boots, before tests run, cannot be checked by
|
|
569
|
-
replacing a method later. saboteur reports these changes as `ran_at_load` rather
|
|
570
|
-
than survivors. Calls from test setup, including RSpec `before(:all)`, are still
|
|
571
|
-
checked. Load-time detection follows the `ran_at_load` idea in
|
|
572
|
-
[mutineer #187](https://github.com/davidteren/mutineer/pull/187).
|
|
573
|
-
|
|
574
|
-
A saved `Method` or `UnboundMethod` can keep calling the old definition after
|
|
575
|
-
saboteur replaces it. When tests pass without calling the replacement and a
|
|
576
|
-
saved method still points to the old definition, the result is `unreachable`.
|
|
577
|
-
A targeted execution check must also confirm the replacement was not called.
|
|
578
|
-
Each mutant is checked separately; one unreachable result does not skip later mutants.
|
|
579
|
-
The reason names the old definition's source location. A lambda that looks up
|
|
580
|
-
the method when it runs is still checked normally.
|
|
581
|
-
|
|
582
|
-
Coverage caches now store both per-test method hits and load-time line hits.
|
|
583
|
-
Older caches are rebuilt; load-time accounting preserves endless-method coverage.
|
|
584
|
-
|
|
585
|
-
One-line and endless method definitions need boot method-call counters to prove
|
|
586
|
-
the body ran at load; merely defining them is not execution evidence. If methods
|
|
587
|
-
share a physical line and any sibling is uncalled, the line is ambiguous and is
|
|
588
|
-
conservatively reported as no coverage, even for its called sibling.
|
|
589
|
-
Agent mode reports ran-at-load and unreachable work as incomplete.
|
|
590
|
-
|
|
591
|
-
A whole-suite map with only load-time hits is valid and reports load-only subjects
|
|
592
|
-
with their ran-at-load reason, rather than failing as an empty coverage build.
|
|
593
|
-
It has no mutation score when no mutant can be tested.
|
|
594
|
-
|
|
595
|
-
Both statuses are listed separately in terminal output and excluded from the
|
|
596
|
-
mutation score. Stryker JSON reports them as `Ignored`, with the explanation in
|
|
597
|
-
`statusReason`; they are not entries on your survivor ignore list.
|
|
598
|
-
|
|
599
|
-
Methods wrapped by `sorbet-runtime` signatures are checked rather than skipped
|
|
600
|
-
as shadowed definitions. saboteur re-evaluates the adjacent `sig` with the
|
|
601
|
-
changed method so the replacement keeps its signature wrapper. Missing adjacent signatures and unknown wrappers are skipped when saboteur cannot
|
|
602
|
-
preserve the loaded method semantics.
|
|
603
|
-
|
|
604
|
-
### Whole-file re-evaluation (opt-in)
|
|
605
|
-
|
|
606
|
-
Surgical re-evaluation refuses a method whose owner it cannot resolve or whose
|
|
607
|
-
loaded definition is not the one in the file. The skip reason ends with a hint.
|
|
608
|
-
List such subjects in `c.whole_file` (ids or globs such as `"Point#distance"` or
|
|
609
|
-
`"Geometry::*"`) and the child evaluates the whole mutated file instead. This
|
|
610
|
-
applies to methods saboteur already finds as subjects; it does not make new
|
|
611
|
-
kinds of definitions selectable (methods inside `Data.define` or `Struct.new`
|
|
612
|
-
blocks are still not found). The file's top-level code then runs again in every
|
|
613
|
-
child, after the run's seed is set: load-time side effects repeat and constants
|
|
614
|
-
it assigns are rebuilt, with redefinition warnings silenced. If a later
|
|
615
|
-
definition in the same file replaces the mutated method, the mutant could never
|
|
616
|
-
run, so a survivor there is reported as unreachable instead. It is off by
|
|
617
|
-
default, and the cache key includes the list.
|
|
618
|
-
Signature text alone does not remove mutants. `body_nil` and last-statement
|
|
619
|
-
removal remain available even when `sig` declares a non-nil return type.
|
|
620
|
-
`checked(:never)`, unchecked `checked(:tests)` signatures, configured default
|
|
621
|
-
levels, custom validation handlers and wrapper shims can all allow nil to
|
|
622
|
-
reach real assertions. saboteur does not verify enforcement, evaluate return
|
|
623
|
-
type expressions or infer types from constant names. Aliases, enclosing
|
|
624
|
-
namespace constants and transformed generic types therefore cannot change
|
|
625
|
-
mutation generation. RBS signatures are not read yet. Type-only crash kills
|
|
626
|
-
remain visible in the crash share, not silently pruned.
|
|
627
|
-
|
|
628
|
-
The Sorbet E2E fixture is a wrapper/registry shim without runtime type checking.
|
|
629
|
-
It tests re-evaluation and assertion detection, not real Sorbet enforcement or
|
|
630
|
-
proof of type-check noise reduction.
|
|
631
|
-
|
|
632
|
-
### Skipped subjects and the mutant cap
|
|
633
|
-
|
|
634
|
-
Subjects with no test coverage are skipped before mutation sources run. They
|
|
635
|
-
remain in the subject count, are listed as skipped subjects in terminal output,
|
|
636
|
-
and appear under `skippedSubjects` in JSON. They are not invented mutants and do
|
|
637
|
-
not enter saboteur's mutation score. Stryker dashboards may ignore the extra
|
|
638
|
-
`skippedSubjects` field and calculate scores from mutant entries alone, so
|
|
639
|
-
removing previously generated `NoCoverage` mutants can raise a dashboard's score
|
|
640
|
-
even when test detection has not improved. Changes inside a covered method that
|
|
641
|
-
no test reaches still produce mutant-level "no coverage" results.
|
|
642
|
-
|
|
643
|
-
Each covered method keeps at most `max_mutants_per_method` valid, distinct
|
|
644
|
-
mutants (100 by default). Coarse mutants come first, then fine mutants in
|
|
645
|
-
configured mutation-source order and each source's proposal order. The cap keeps
|
|
646
|
-
a prefix of that order, so earlier sources can use the entire limit within a
|
|
647
|
-
method. Later methods have their own limit. In diff mode, changed-line selection
|
|
648
|
-
happens before the cap. A warning names each capped method and the number
|
|
649
|
-
omitted; JSON keeps these warnings under `warnings`. The score describes this
|
|
650
|
-
selected subset, not every possible mutation. Raise the limit to examine more
|
|
651
|
-
mutants; the limit does not bound how many proposals a custom source creates.
|
|
652
|
-
|
|
653
|
-
### Ignoring a survivor
|
|
654
|
-
|
|
655
|
-
Every survivor is printed with a digest. Accept it with a reason:
|
|
656
|
-
|
|
657
|
-
```ruby
|
|
658
|
-
# config/saboteur_ignore.rb
|
|
659
|
-
Saboteur.ignore do |i|
|
|
660
|
-
i.mutation "1a2b3c4d5e6f7a8b",
|
|
661
|
-
subject: "Billing::Invoice#total",
|
|
662
|
-
reason: "equivalent: amounts are always whole cents"
|
|
663
|
-
end
|
|
664
|
-
```
|
|
665
|
-
|
|
666
|
-
An entry without a reason is rejected.
|
|
224
|
+
An engine gem whose specs live under `spec/dummy` needs an exclude for them; see
|
|
225
|
+
[Rails engines and dummy apps](docs/guides/rails-engines-and-dummy-apps.md).
|
|
667
226
|
|
|
668
|
-
|
|
227
|
+
## SimpleCov
|
|
669
228
|
|
|
670
|
-
saboteur needs Ruby's `Coverage` for itself
|
|
671
|
-
|
|
229
|
+
saboteur needs Ruby's `Coverage` for itself, so leave SimpleCov off for
|
|
230
|
+
saboteur runs - for example, start it only when `ENV["COVERAGE"]` is set.
|
|
231
|
+
The [recipes](docs/recipes/README.md) cover this and VCR, WebMock and
|
|
232
|
+
Sidekiq.
|
|
672
233
|
|
|
673
|
-
|
|
234
|
+
<a id="dated-validation-runs"></a>
|
|
235
|
+
<a id="where-it-has-run"></a>
|
|
236
|
+
## Benchmarks and validation
|
|
674
237
|
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
678
|
-
|
|
238
|
+
Speed and memory numbers live with the harness and the raw data that
|
|
239
|
+
reproduce them, framed as dated measurements rather than standing claims:
|
|
240
|
+
the [benchmarks page](https://vamsipavanmahesh.github.io/saboteur/benchmark/)
|
|
241
|
+
on the site and [docs/benchmark](docs/benchmark/) in this repo.
|
|
679
242
|
|
|
680
|
-
|
|
681
|
-
commands.reject! { |name, _| name == replaced_name }
|
|
682
|
-
# mutant: commands.select! { |name, _| name == replaced_name }
|
|
683
|
-
```
|
|
243
|
+
## Project
|
|
684
244
|
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
Ruby dispatch is dynamic: a matching name does not prove the receiver is an
|
|
690
|
-
Enumerable, and a custom receiver can make the replacement fail. No type or
|
|
691
|
-
real-fault detection guarantee is made.
|
|
692
|
-
|
|
693
|
-
These are fine mutants, appended after the existing built-in operators. For unchanged subjects, old
|
|
694
|
-
mutant digests and relative order stay unchanged, but runs can produce more
|
|
695
|
-
mutants, change their score, and hit the per-method cap. The operator and source
|
|
696
|
-
name in reports is `filter_replacement`.
|
|
697
|
-
|
|
698
|
-
See [the operator rationale](docs/filter-operators.md) for the Ruby fault used
|
|
699
|
-
to choose this family and the limits of that evidence.
|
|
700
|
-
|
|
701
|
-
## Literal mutations
|
|
702
|
-
|
|
703
|
-
Builtin now swaps `true` / `false` literals and replaces nonempty static strings
|
|
704
|
-
with `""`. These test boolean choices and the difference between meaningful
|
|
705
|
-
text and empty input. Operator names are `literal_boolean` and `literal_string`;
|
|
706
|
-
the source name is `literal_replacement`.
|
|
707
|
-
|
|
708
|
-
Only method bodies are visited. Default arguments, arid statements, nested
|
|
709
|
-
definitions, empty strings, heredocs, interpolation, word/symbol lists, symbols,
|
|
710
|
-
regexes and command strings are unchanged. Quoted and static percent strings
|
|
711
|
-
are supported. The new proposals follow the earlier built-in operators; the
|
|
712
|
-
normal duplicate check and per-method cap still apply. Old digests and order
|
|
713
|
-
remain stable for unchanged subjects, but more mutants can change a run's score.
|
|
714
|
-
No speed, type-safety or real-fault detection improvement is claimed. See
|
|
715
|
-
[the literal operator rationale](docs/literal-operators.md) for fault evidence
|
|
716
|
-
and limits, including string encoding/frozen-value behavior.
|
|
717
|
-
|
|
718
|
-
## Chain removal
|
|
719
|
-
|
|
720
|
-
Builtin drops one link from the middle of a dotted method chain: `user.name.strip`
|
|
721
|
-
becomes `user.strip`, and `a.b.c.d` gives `a.c.d` and `a.b.d`. The first receiver
|
|
722
|
-
and the last call stay, so a test has to notice that a step in between went
|
|
723
|
-
missing. The link goes whole, with its arguments and block. The operator and
|
|
724
|
-
source name is `chain_removal`.
|
|
725
|
-
|
|
726
|
-
Operators, index calls and bare-name receivers are not chains and are unchanged.
|
|
727
|
-
The target of a write (`user.account.profile.name = 1`, `+=`, `||=`, `a.b[0] = 1`)
|
|
728
|
-
is left whole: no link of its receiver chain is dropped, however long. Only
|
|
729
|
-
the value on the right can be mutated. `freeze`, `dup`, `clone`, `itself`
|
|
730
|
-
and `tap` links are never removed. Tests can observe frozenness, copies and a
|
|
731
|
-
`tap` block's effects, but those are other contracts than a missing step in a
|
|
732
|
-
query chain, and dropping such a link mostly yields noise, so this operator
|
|
733
|
-
leaves them out. Arid statements and nested definitions are skipped.
|
|
734
|
-
|
|
735
|
-
Chain removal is on by default, like `argument_removal` and `pair_removal`.
|
|
736
|
-
Its proposals follow the other fine proposals, with the same validation,
|
|
737
|
-
deduplication and `max_mutants_per_method` cap.
|
|
738
|
-
|
|
739
|
-
## Argument removal
|
|
740
|
-
|
|
741
|
-
Builtin removes one ordinary positional argument from a parenthesized call.
|
|
742
|
-
For `Model.includes(:a, :b, c: :d)`, the two proposals remove `:a` or `:b`,
|
|
743
|
-
without deleting the statement or changing `c: :d`. This lets a preload check
|
|
744
|
-
exercise missing associations instead of only a missing return value.
|
|
745
|
-
The operator and source name is `argument_removal`.
|
|
746
|
-
|
|
747
|
-
Only parenthesized calls are supported in this first version. Keyword hashes,
|
|
748
|
-
splats, argument forwarding, bracket indexing/assignment syntax and arguments containing
|
|
749
|
-
heredocs are unchanged. Explicit hash arguments count as positional arguments.
|
|
750
|
-
Blocks and keyword pairs remain intact; separators come from Prism tokens,
|
|
751
|
-
so commas inside strings or comments are not mistaken for separators.
|
|
752
|
-
Arid statements, parameter defaults and nested definitions are skipped.
|
|
753
|
-
|
|
754
|
-
These fine proposals follow existing builtin proposals, with the same validation,
|
|
755
|
-
deduplication and `max_mutants_per_method` cap. Earlier digests and relative order
|
|
756
|
-
stay unchanged for unchanged subjects. Extra proposals can change scores or be
|
|
757
|
-
omitted by the cap. Removing an argument can still raise an arity error; no
|
|
758
|
-
non-crashing guarantee is made. The regression uses a small preload-like receiver
|
|
759
|
-
and checks the missing associations, not an ActiveRecord SQL benchmark. See
|
|
760
|
-
[the validation notes](docs/argument-removal.md) for the pinned snapshot migration.
|
|
761
|
-
|
|
762
|
-
## Hash and keyword pair removal
|
|
763
|
-
|
|
764
|
-
Builtin removes one key/value association from a hash literal or keyword list.
|
|
765
|
-
`Model.includes(:a, :b, c: :d)` becomes `Model.includes(:a, :b )`, so a nested
|
|
766
|
-
preload guard can be exercised without deleting the whole statement. A sole
|
|
767
|
-
hash pair becomes `{}`. Operator and source name: `pair_removal`.
|
|
768
|
-
|
|
769
|
-
Both parenthesized and command calls are supported. Splats stay intact; pairs
|
|
770
|
-
containing heredocs, parameter defaults, arid statements and nested definitions
|
|
771
|
-
are skipped. Token-based separator edits preserve comments and blocks. Nested
|
|
772
|
-
hash values and callable bodies are visited. Removing a required keyword can
|
|
773
|
-
still crash; the regression uses a preload-like receiver, not an ActiveRecord
|
|
774
|
-
SQL benchmark. These fine proposals follow earlier builtin proposals and use
|
|
775
|
-
the usual validation, deduplication, order and `max_mutants_per_method` cap.
|
|
776
|
-
More proposals can change scores and cap coverage. See
|
|
777
|
-
[the validation notes](docs/pair-removal.md) for snapshot changes.
|
|
778
|
-
|
|
779
|
-
## π Extending
|
|
780
|
-
|
|
781
|
-
Public API, stable across minor versions:
|
|
782
|
-
|
|
783
|
-
- `Saboteur::MutationSource::Base#mutations_for(subject)` returning
|
|
784
|
-
`Saboteur::MutationSource::Mutation` objects made of `Edit`s (file, byte
|
|
785
|
-
range, replacement). Register with `sources: [builtin, path.rb:ClassName]`.
|
|
786
|
-
- `Saboteur::Integration::Base` for other test frameworks (RSpec is the only one
|
|
787
|
-
that ships).
|
|
788
|
-
- `Saboteur.hook(:before_boot | :after_boot | :before_fork | :after_fork |
|
|
789
|
-
:around_mutant)` in files listed under `hooks:`.
|
|
790
|
-
|
|
791
|
-
## π Where it has run
|
|
792
|
-
|
|
793
|
-
An early release (0.4.0). Tested on the RSpec suites of plain gems
|
|
794
|
-
(dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
|
|
795
|
-
human-essentials on Postgres (4,080 mutants, 6 workers; the 2026-10-10 run, before
|
|
796
|
-
the baseline check and the coarse-gate removal) and lobsters on SQLite,
|
|
797
|
-
both on macOS. Not yet tried on MySQL or older Rails versions. RSpec only, by
|
|
798
|
-
design. Results: [docs/validation](docs/validation).
|
|
799
|
-
|
|
800
|
-
## βοΈ How it compares
|
|
801
|
-
|
|
802
|
-
A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
|
|
803
|
-
1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
|
|
804
|
-
ran once on sampled files, so treat the numbers as indicative. The table was
|
|
805
|
-
measured at 2b6ec02, before the literal, filter and argument operators and
|
|
806
|
-
before the coarse-survivor gate was removed, so mutant counts and times will
|
|
807
|
-
change; a rerun is pending. mutineer is ahead
|
|
808
|
-
on number of operators and CI extras, and it supports Minitest.
|
|
809
|
-
|
|
810
|
-
| Metric | saboteur | mutineer |
|
|
811
|
-
|---|---|---|
|
|
812
|
-
| Fake kills (lobsters) | 0 of 20 sampled at random | 8 of 8 hand-checked (identical mutants, not a random sample) |
|
|
813
|
-
| Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
|
|
814
|
-
| Runs on the Postgres app | yes | no |
|
|
815
|
-
| Time per mutant, hashie | 0.008 s | 0.092 s |
|
|
816
|
-
| Peak memory, hashie (summed process RSS) | 595 MB | 471 MB |
|
|
817
|
-
| Choice of operators, CI features | fixed set, fewer | 22 operators, baseline, HTML, GitHub Action |
|
|
818
|
-
|
|
819
|
-
Full results, method and limitations:
|
|
820
|
-
[benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
|
|
821
|
-
Scripts and raw data: [bench/vs-mutineer](bench/vs-mutineer). The memory row sums
|
|
822
|
-
per-process RSS, which counts shared pages more than once, and it was measured
|
|
823
|
-
before the fork-memory work. Read it as history, not a ranking of physical
|
|
824
|
-
memory.
|
|
825
|
-
|
|
826
|
-
### Independent review, saboteur 0.4.0 vs mutineer 1.6.0
|
|
827
|
-
|
|
828
|
-
A private consultant review ("saboteur 0.4.0 - Deep Review", 12 October 2026,
|
|
829
|
-
not published) ran saboteur 0.4.0 at 38b1834 and mutineer 1.6.0 at a766c96 on
|
|
830
|
-
the same machine and projects: Linux x86_64, 1 vCPU, 4 GB, Ruby 3.4.5, Prism
|
|
831
|
-
1.2.0, RSpec 3.13, no Rails or sqlite3. mutineer ran with its default 5
|
|
832
|
-
operators and with all 20. The project and timing rows are one run each. The
|
|
833
|
-
rows describe saboteur at 38b1834; several of its findings have been fixed on
|
|
834
|
-
main since. The rerun that replaces this table is still pending.
|
|
835
|
-
|
|
836
|
-
| Measure | saboteur 0.4.0 | mutineer 1.6.0 |
|
|
837
|
-
|---|---|---|
|
|
838
|
-
| Snippets project (11 planted gaps) | 177 mutants, 14 s, 10/11 gaps found, 9 noise survivors | Default: 56 mutants, 18 s, about 5/11. All operators: 191 mutants, 51 s, 10/11, 8 noise and duplicates |
|
|
839
|
-
| Dependency project (A tested only via B) | 239 mutants, 6 s | 254 mutants, 26 s (needed hand-ordered source files) |
|
|
840
|
-
| dry-inflector (real gem) | 551 mutants, 24 survivors, 95.6 %, 133 s | 480 mutants, 47 survivors (31 distinct), 90.1 %, 134 s |
|
|
841
|
-
| Test selection | Whole suite, per example, automatic | Only files passed with --test, per file |
|
|
842
|
-
| Kill verification | Yes (re-runs killers on unchanged code) | No |
|
|
843
|
-
| sorbet sig methods | Skipped at 0.4.0 | Mutated, 24/24 killed |
|
|
844
|
-
| Data.define / Struct.new block methods | Absent at 0.4.0 | Mutated |
|
|
845
|
-
| Load-time code | False survivors at 0.4.0 | ran_at_load status |
|
|
846
|
-
| Endless defs | Tested ones reported as no coverage at 0.4.0 | Tested ones work; untested ones show as survived |
|
|
847
|
-
| Order-dependent specs | Smell exposed, bogus kills rejected | Hidden (whole file runs) |
|
|
848
|
-
| Destructive mutant | Project wiped except .saboteur | Project wiped |
|
|
849
|
-
| Baseline stability | Race at 0.4.0: 4 of 30 green runs failed (13 %), 5 of 25 under CPU contention (20 %) | No failures observed (15/15) |
|
|
850
|
-
|
|
851
|
-
Not in the review and still pending: a rerun on current main, memory and
|
|
852
|
-
per-mutant time on the benchmark apps, and any mutineer release after 1.6.0.
|
|
853
|
-
|
|
854
|
-
### Memory after the fork-memory work (hashie @ 9c81e83)
|
|
855
|
-
|
|
856
|
-
Paired A/B runs against the pre-PR commit (32c0f21), 5 pairs per job level with
|
|
857
|
-
AB/BA alternation and cold caches, on a 2 vCPU / 2 GB shared sandbox. Peak
|
|
858
|
-
memory is per-process-tree PSS from /proc/smaps_rollup sampled at 0.3s. Verdict
|
|
859
|
-
parity: **0 verdict differences in 23,700 mutant outcomes** (1,580 mutants x 15
|
|
860
|
-
pairs); kill reasons (assertion vs crash class) vary run to run on both
|
|
861
|
-
variants, as before.
|
|
862
|
-
|
|
863
|
-
| jobs | peak PSS baseline (median, range) | peak PSS PR (median, range) | shared-memory delta |
|
|
864
|
-
|------|-----------------------------------|-----------------------------|---------------------|
|
|
865
|
-
| 1 | 101 MB (101-101) | 100 MB (100-102) | ~0 (no workers to share with) |
|
|
866
|
-
| 2 | 133 MB (116-134) | 109 MB (106-121) | -18% |
|
|
867
|
-
| 6 | 235 MB (192-269) | 196 MB (175-201) | -17% |
|
|
868
|
-
|
|
869
|
-
Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not
|
|
870
|
-
touch less memory, it keeps more of it shared between workers after fork, which
|
|
871
|
-
is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers on a
|
|
872
|
-
2 GB sandbox. Swap was not measured.
|
|
873
|
-
|
|
874
|
-
Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs
|
|
875
|
-
32s at jobs=6. These are noisy per-cell minima from historical runs and do not
|
|
876
|
-
show a direction either way. The shared sandbox had multi-minute CPU-starvation
|
|
877
|
-
windows during this session (walls up to 594s); affected runs are flagged via
|
|
878
|
-
sampler cadence and excluded from wall analysis. Absolute times are specific to
|
|
879
|
-
this 2-core sandbox. Environment, method and the contention log are on the
|
|
880
|
-
[benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
|
|
881
|
-
|
|
882
|
-
## πΊοΈ Where it is going
|
|
883
|
-
|
|
884
|
-
[ROADMAP.md](ROADMAP.md) lays out the plan: CI-fast runs, trusted results, a
|
|
885
|
-
team workflow (GitHub Action, baseline, HTML report), then 1.0. saboteur stays
|
|
886
|
-
RSpec-only. Design
|
|
887
|
-
details are in [docs/design.md](docs/design.md), prior art in
|
|
888
|
-
[docs/prior-art.md](docs/prior-art.md), and project status in
|
|
889
|
-
[docs/status.md](docs/status.md).
|
|
890
|
-
|
|
891
|
-
## π€ Contributing
|
|
892
|
-
|
|
893
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md). Commits need a DCO `Signed-off-by` line.
|
|
894
|
-
|
|
895
|
-
Before you ask for review, run the local preflight check on your exact head:
|
|
896
|
-
|
|
897
|
-
```sh
|
|
898
|
-
ruby script/preflight --base <full main sha> --head <full head sha> --base-ref main --mode full
|
|
899
|
-
```
|
|
900
|
-
|
|
901
|
-
It runs scope, sign-off, SPDX, full RuboCop and the specs, and writes a report under `tmp/preflight/`. A pass is not an approval; fast mode is partial. See [docs/preflight.md](docs/preflight.md).
|
|
902
|
-
|
|
903
|
-
## Acknowledgements
|
|
904
|
-
|
|
905
|
-
Thanks to [Glomo](https://glomo.one) for providing the AI tokens used to build saboteur.
|
|
245
|
+
- [Changelog](CHANGELOG.md)
|
|
246
|
+
- [All documentation](docs/README.md) - configuration, operators, CI, recipes
|
|
247
|
+
- [Contributing](CONTRIBUTING.md) - every commit needs a DCO `Signed-off-by` line
|
|
248
|
+
- [Design notes](docs/design.md)
|
|
906
249
|
|
|
907
250
|
## License
|
|
908
251
|
|
|
909
|
-
MIT
|
|
910
|
-
|
|
911
|
-
Internal baseline capture blocks a subject with empty generation, including a subject for which the builtin source yields no candidates. Context comparison is an extra consistency check; file fingerprint verification normally also catches source drift. Neither check proves external state stability.
|
|
912
|
-
|
|
913
|
-
### Opt-in survivor baseline
|
|
914
|
-
|
|
915
|
-
`baseline check` runs every selected mutant fresh, then compares accepted survivors. It does not skip accepted mutants, change verdicts or change scores. Use whole configured projects or `--subject` filters; diff, replay, incremental reuse and score thresholds are refused. Caps, uncovered work, empty generation, errors, resource limits, omitted malformed files, changed inputs and missing results are incomplete, even with zero survivors.
|
|
916
|
-
|
|
917
|
-
```sh
|
|
918
|
-
saboteur baseline propose --baseline baseline.json --proposal proposal.json
|
|
919
|
-
# Edit only document.accepted[*].reason in proposal.json. Give each survivor a review reason.
|
|
920
|
-
saboteur baseline accept --baseline baseline.json --proposal proposal.json
|
|
921
|
-
saboteur baseline check --baseline baseline.json
|
|
922
|
-
```
|
|
923
|
-
|
|
924
|
-
Proposal writes independent owner-private execution evidence under the configured cache directory. Keep that directory until acceptance; deleting it prevents acceptance. Acceptance verifies the private record and current input fingerprint before atomically publishing the manifest. For replacement, give both propose and accept `--previous SHA256`, the hash you reviewed of the current baseline bytes. Output files must not overlap project inputs or the private evidence directory. A baseline normally lives in a separate JSON file outside fixture/config input trees.
|
|
925
|
-
|
|
926
|
-
Check prints one JSON object: accepted, new, changed, resolved_kills, detected_timeouts, unmatched and errors are separate categories. Exit 0 means a compatible complete clean check; 1 needs survivor review; 2 means incompatible or invalid configuration; 3 means incomplete execution or tool failure. Propose and accept exit 0 only after their file operation succeeds. Context changes require fresh review. Timeouts remain a distinct detected category, not assertion kills.
|
|
927
|
-
|
|
928
|
-
Score-drop gate: `saboteur baseline check --baseline baseline.json --epsilon POINTS` also fails (exit 1,
|
|
929
|
-
state `review_required`) when the fresh score is more than POINTS percentage points below the score recorded in
|
|
930
|
-
the baseline. Both scores use saboteur's own formula, (killed + timeout) / (killed + timeout + survived), over
|
|
931
|
-
every candidate. The baseline already stores each candidate's status, so no new file format is needed. This
|
|
932
|
-
catches what the survivor comparison cannot: a score that falls because killed mutants disappeared (deleted or
|
|
933
|
-
rewritten code) while every accepted survivor is still matched. A drop of exactly POINTS passes; a rise never
|
|
934
|
-
fails. The check prints a `score` object (`baseline`, `current`, `drop`, `epsilon`, `exceeded`) only when
|
|
935
|
-
`--epsilon` is given. Without it, behavior is unchanged. POINTS is 0 to 100. It lives on `baseline check`,
|
|
936
|
-
not on `run`: the delta gate needs the baseline's fresh whole-scope evidence, which `run` does not produce.
|
|
937
|
-
It is not the same as `--threshold`, which gates an absolute score.
|
|
938
|
-
|
|
939
|
-
In GitHub Actions, the action's `baseline` and `baseline-epsilon` inputs run this check; see
|
|
940
|
-
[docs/github-action.md](docs/github-action.md).
|
|
941
|
-
|
|
942
|
-
Local evidence protects against edits to the public proposal except review reasons. It is not remote attestation, a signature, or protection against the account owner changing their private files. File fingerprints cannot prove external database, network or environment stability. `config/saboteur_ignore.rb` remains a separate legacy ignore mechanism; this gate compares every survivor regardless of ignore status.
|
|
943
|
-
|
|
944
|
-
A baseline that changes during `check` is rejected as configuration/evidence error (exit 2), not an incomplete execution result. Acceptance leaves a private `baseline.json.lock` beside the destination: keep it for cooperating-process publication locks. It contains no evidence or results. Symlinked checkout parents are normalized; symlink artifacts themselves and private-store/lock symlinks are still refused.
|
|
945
|
-
|
|
946
|
-
The builtin `endpoint_replacement` operator swaps explicit, no-argument `first`/`last` calls. It targets wrong-end/order-selection faults in collection handling. Argument-bearing slices, bare calls and block-bearing APIs are left unchanged. Receivers are not statically typed: a custom object may implement either name differently, so a crash remains a crash kill, not proof of a collection assertion.
|
|
947
|
-
|
|
948
|
-
JSON findings are ordered by a deterministic [review ranking](docs/survivor-ranking.md): coarse first, then covering-test count, edited-line impact and changed-line proximity, with file/line/digest ties. The terminal groups survivors by file, method and line first; the ranking only orders the survivors on one line, and every survivor is listed, in review mode too. Only consecutive survivors with matching evidence share a test/evidence block; repeated evidence after another group is printed again to keep that order. Ranking does not change scores or execution.
|
|
949
|
-
|
|
950
|
-
Cache and baseline identities include the policy. Re-propose and review old survivor baselines with `baseline propose`; never auto-accept new survivors. Mutation digests for unchanged edits stay stable.
|
|
951
|
-
|
|
952
|
-
### Preview and choose operators
|
|
953
|
-
|
|
954
|
-
`bundle exec saboteur run --dry-run` discovers source methods and generates candidates without booting the app, running tests, forking, or writing reports/caches. It honors `--since`, `--subject` and source arguments. It prints per-method counts and cap omissions. After a matching normal run, a cached estimate shows baseline covering-test work (cached original-test durations times planned mutants; it is not a bound on mutant runtime and excludes boot and fork costs). If any file cannot be parsed, the total is marked INCOMPLETE and no estimate is given.
|
|
955
|
-
|
|
956
|
-
Trust boundary: `--dry-run` never boots the app or runs its tests, but custom `sources` entries (`path.rb:ClassName`) are still `require`d to generate candidates, so their code runs. Only preview with sources you trust.
|
|
957
|
-
|
|
958
|
-
`bundle exec saboteur run --list-operators` lists every builtin name and description. All builtin operators run by default, including [`range`](docs/operators/range.md), [`operator_assignment`](docs/operators/operator_assignment.md), and [`condition_negation`](docs/operators/condition_negation.md). `--operators arithmetic,comparison` or `c.operators = ["arithmetic", "comparison"]` still selects a subset; `all` selects every name. There is no opt-in tier. `--exclude-operators range,condition_negation` or `c.exclude_operators = ["range", "condition_negation"]` removes named operators from that selection; exclusion wins if a name is in both lists. An empty selection means all, while an empty exclusion means none. CLI lists replace their corresponding config lists. These controls apply to builtin operators, not custom source plugins. Selection and exclusions invalidate cached verdict identity and are carried into agent rerun/replay commands.
|
|
959
|
-
|
|
960
|
-
|
|
961
|
-
All operators run unless you select a smaller set; exclusions always remove operators from that set.
|
|
962
|
-
Repeated `--operators` or `--exclude-operators` flags use the last list, not a union.
|
|
963
|
-
To re-enable an operator excluded in config: `bundle exec saboteur run --operators range --exclude-operators ''`.
|
|
964
|
-
To keep the 0.5.0 default inventory: `bundle exec saboteur run --exclude-operators range,operator_assignment,condition_negation`.
|
|
965
|
-
The old `core` and `extended` names now give a migration error, not a changed selection in disguise.
|
|
966
|
-
An empty effective selection is a usage error before boot. `--list-operators` loads current config and shows on/off plus examples; preview and the run header show the effective set.
|
|
967
|
-
|
|
968
|
-
```ruby
|
|
969
|
-
Saboteur.configure do |c|
|
|
970
|
-
c.operators = [] # all by default; or ["arithmetic", "range"] for a subset
|
|
971
|
-
c.exclude_operators = ["condition_negation"]
|
|
972
|
-
end
|
|
973
|
-
```
|
|
974
|
-
|
|
975
|
-
Inspired by mutineer's --dry-run, --list-operators and --operators tiers (github.com/davidteren/mutineer, MIT, by David Teren).
|
|
976
|
-
|
|
977
|
-
### Agent survivor hints and score history
|
|
978
|
-
|
|
979
|
-
`--agent` findings include operator-specific `hint` suggestions and a shared
|
|
980
|
-
`hintCaveat`. Hints do not diagnose a missing test or assertion; equivalent
|
|
981
|
-
mutants and limited observation are still possible. Unknown operators have no
|
|
982
|
-
hint unless marked coarse. The normal JSON byte budget still applies.
|
|
983
|
-
|
|
984
|
-
Use `--score-history PATH` (or `c.score_history = PATH` in your config) to append
|
|
985
|
-
one score-history JSONL record per finished report. History is off by default;
|
|
986
|
-
changes are printed to stderr, not agent stdout. See [score definitions and
|
|
987
|
-
history limits](docs/scores.md). The file can be uploaded by your own artifact
|
|
988
|
-
step; the bundled Action's upload behavior is unchanged.
|
|
989
|
-
|
|
990
|
-
Score history now refuses non-regular files, symlink paths and report/history
|
|
991
|
-
inode aliases before running the engine or writing reports. Imported records
|
|
992
|
-
have a 64 KiB per-line bound and strict validation of supported version 1
|
|
993
|
-
and version 2 records, including counts, scores and duplicate keys. History is an output, excluded from baseline
|
|
994
|
-
fingerprints. See [history safety and comparison limits](docs/scores.md).
|
|
995
|
-
|
|
996
|
-
Operator policy version 2, selected names, excluded names and effective names are in JSON/agent/baseline evidence and fresh score-history v2 rows. Old history v1 rows remain readable but are not compared across policy versions.
|
|
997
|
-
### Inline suppression comments
|
|
998
|
-
|
|
999
|
-
`total = a + b # saboteur:disable-line arithmetic reason="rounding checked elsewhere"` suppresses edits starting on that line, restricted to the named operators. Omit names to suppress all operators. Multiple names are space-separated. Only actual Ruby comments count; directive text in strings does not. A missing or blank reason or unknown operator is a usage error naming file:line. Terminal, JSON and HTML list every suppressed candidate with its reason and digest, outside the score. Moving the commented code keeps the suppression. Multi-line edits use their starting line.
|
|
1000
|
-
|
|
1001
|
-
Inspired by mutineer's inline suppression comments (github.com/davidteren/mutineer, MIT, by David Teren).
|
|
1002
|
-
|
|
1003
|
-
Inline suppressions bound completeness: classic scores still exclude suppressed candidates, but baseline evidence is incomplete when any inline candidate is suppressed. Agent output reports `inline_suppressed` with its count and points to the full audit. Reasons are audited, not proof that suppressed work was checked.
|
|
1004
|
-
|
|
1005
|
-
Source-only `--dry-run` lists the suppressed count and each audit row in selected methods, before diff restriction and the per-method cap. They are outside the planned total. Preview still does not boot the app or write reports.
|
|
252
|
+
[MIT](LICENSE).
|