saboteur 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +280 -1
- data/README.md +635 -77
- data/exe/saboteur +6 -2
- data/lib/generators/saboteur/install/install_generator.rb +21 -0
- data/lib/saboteur/agent_skill.rb +46 -0
- data/lib/saboteur/baseline/acceptance.rb +218 -0
- data/lib/saboteur/baseline/command.rb +80 -0
- data/lib/saboteur/baseline/comparison.rb +105 -0
- data/lib/saboteur/baseline/completeness.rb +53 -0
- data/lib/saboteur/baseline/evidence.rb +114 -0
- data/lib/saboteur/baseline/identity.rb +78 -0
- data/lib/saboteur/baseline/manifest.rb +164 -0
- data/lib/saboteur/baseline/run_evidence.rb +117 -0
- data/lib/saboteur/baseline/workflow.rb +121 -0
- data/lib/saboteur/baseline_health.rb +1 -1
- data/lib/saboteur/cli/exit_status.rb +93 -0
- data/lib/saboteur/cli/run_reporting.rb +75 -0
- data/lib/saboteur/cli.rb +171 -63
- data/lib/saboteur/config.rb +101 -19
- data/lib/saboteur/config_file.rb +97 -0
- data/lib/saboteur/custom_map.rb +170 -0
- data/lib/saboteur/doctor.rb +57 -9
- data/lib/saboteur/engine/baseline_guard.rb +44 -5
- data/lib/saboteur/engine/boot.rb +9 -0
- data/lib/saboteur/engine/bytecode_equivalence.rb +95 -0
- data/lib/saboteur/engine/canary.rb +74 -0
- data/lib/saboteur/engine/candidate_selection.rb +18 -5
- data/lib/saboteur/engine/candidates.rb +59 -7
- data/lib/saboteur/engine/coverage_store.rb +53 -6
- data/lib/saboteur/engine/data_inputs.rb +43 -0
- data/lib/saboteur/engine/diagnostics.rb +176 -2
- data/lib/saboteur/engine/diff_scope.rb +18 -0
- data/lib/saboteur/engine/distribution.rb +54 -13
- data/lib/saboteur/engine/result_cache.rb +10 -5
- data/lib/saboteur/engine/run_fingerprint.rb +40 -10
- data/lib/saboteur/engine/sandbox_policy.rb +108 -0
- data/lib/saboteur/engine/scheduler.rb +99 -36
- data/lib/saboteur/engine/subject_selection.rb +24 -1
- data/lib/saboteur/engine/suite_files.rb +8 -1
- data/lib/saboteur/engine/test_guess.rb +52 -0
- data/lib/saboteur/engine/unchanged_syntax.rb +87 -0
- data/lib/saboteur/engine/verdict_check.rb +7 -6
- data/lib/saboteur/engine.rb +257 -22
- data/lib/saboteur/execution/protocol.rb +2 -1
- data/lib/saboteur/execution/reevaluator.rb +79 -14
- data/lib/saboteur/execution/runner.rb +120 -17
- data/lib/saboteur/execution/sandbox.rb +271 -0
- data/lib/saboteur/execution/stale_captures.rb +43 -0
- data/lib/saboteur/execution/supervisor.rb +113 -9
- data/lib/saboteur/execution/worker_pool.rb +20 -4
- data/lib/saboteur/execution/wrapper.rb +45 -0
- data/lib/saboteur/frameworks/rails.rb +329 -13
- data/lib/saboteur/gates.rb +83 -0
- data/lib/saboteur/help.rb +63 -23
- data/lib/saboteur/ignore_list.rb +76 -17
- data/lib/saboteur/inline_suppression.rb +65 -0
- data/lib/saboteur/install.rb +132 -0
- data/lib/saboteur/integration/base.rb +1 -0
- data/lib/saboteur/integration/focus_guard.rb +83 -0
- data/lib/saboteur/integration/rspec.rb +68 -4
- data/lib/saboteur/mutation_source/argument_removal.rb +86 -0
- data/lib/saboteur/mutation_source/builtin.rb +30 -5
- data/lib/saboteur/mutation_source/chain_removal.rb +80 -0
- data/lib/saboteur/mutation_source/condition_negation.rb +50 -0
- data/lib/saboteur/mutation_source/endpoint_replacement.rb +61 -0
- data/lib/saboteur/mutation_source/hazard_guard.rb +136 -0
- data/lib/saboteur/mutation_source/operator_assignment.rb +52 -0
- data/lib/saboteur/mutation_source/operator_registry.rb +57 -0
- data/lib/saboteur/mutation_source/pair_removal.rb +101 -0
- data/lib/saboteur/mutation_source/range_replacement.rb +43 -0
- data/lib/saboteur/preview.rb +106 -0
- data/lib/saboteur/replay.rb +1 -1
- data/lib/saboteur/reporter/agent.rb +87 -0
- data/lib/saboteur/reporter/agent_clipping.rb +29 -0
- data/lib/saboteur/reporter/agent_commands.rb +67 -0
- data/lib/saboteur/reporter/agent_incomplete.rb +60 -0
- data/lib/saboteur/reporter/agent_summary.rb +137 -0
- data/lib/saboteur/reporter/findings.rb +12 -19
- data/lib/saboteur/reporter/history_paths.rb +38 -0
- data/lib/saboteur/reporter/history_record.rb +132 -0
- data/lib/saboteur/reporter/html.rb +61 -2
- data/lib/saboteur/reporter/mutation_text.rb +36 -0
- data/lib/saboteur/reporter/next_steps.rb +30 -0
- data/lib/saboteur/reporter/palette.rb +37 -0
- data/lib/saboteur/reporter/score_history.rb +123 -0
- data/lib/saboteur/reporter/score_table.rb +57 -0
- data/lib/saboteur/reporter/stryker.rb +25 -5
- data/lib/saboteur/reporter/survivor_hints.rb +68 -0
- data/lib/saboteur/reporter/survivor_list.rb +126 -0
- data/lib/saboteur/reporter/survivor_ranking.rb +34 -0
- data/lib/saboteur/reporter/terminal.rb +157 -42
- data/lib/saboteur/reporter/test_matrix.rb +57 -0
- data/lib/saboteur/result.rb +86 -10
- data/lib/saboteur/score.rb +89 -0
- data/lib/saboteur/selection/coverage_map.rb +119 -13
- data/lib/saboteur/selection/git_diff.rb +28 -1
- data/lib/saboteur/selection/recorder.rb +55 -13
- data/lib/saboteur/subject.rb +40 -4
- data/lib/saboteur/subject_finder.rb +62 -12
- data/lib/saboteur/version.rb +1 -1
- data/lib/saboteur/working_tree.rb +56 -0
- data/lib/saboteur.rb +10 -0
- metadata +58 -2
data/README.md
CHANGED
|
@@ -29,19 +29,118 @@ fail?" for many small breakages and lists the ones nobody noticed. Read the
|
|
|
29
29
|
[design notes](docs/design.md) when you want the details. For adopting it on a team, see the
|
|
30
30
|
[best practices guide](docs/best-practices.md).
|
|
31
31
|
|
|
32
|
+
Declared external inputs are checked against cache and report paths using their
|
|
33
|
+
physical paths, including aliased parent directories. An overlap makes the run
|
|
34
|
+
fingerprint incomplete, so it cannot authorize cache reuse.
|
|
35
|
+
|
|
36
|
+
Internal run reports retain typed candidate-drop and cap evidence for gate
|
|
37
|
+
validation. These records do not change selection, scores, exit codes or the
|
|
38
|
+
public JSON format. Empty generation is unverified, not proof of equivalence.
|
|
39
|
+
|
|
40
|
+
The internal engine can opt in to fresh baseline evidence capture. It records
|
|
41
|
+
inputs before boot, planned candidate identities before scheduling and reconciled
|
|
42
|
+
terminal results after execution. Caps, uncovered work, invalid or empty
|
|
43
|
+
production, reused outcomes and changed inputs make that evidence incomplete.
|
|
44
|
+
The public opt-in `baseline propose`, `baseline accept` and `baseline check`
|
|
45
|
+
commands use this evidence to review and compare survivors. They never skip
|
|
46
|
+
accepted mutants or change their scores. See [Opt-in survivor baseline](#opt-in-survivor-baseline).
|
|
47
|
+
|
|
32
48
|
### The unchanged suite is checked first
|
|
33
49
|
|
|
34
|
-
Every run with selected subjects runs the
|
|
50
|
+
Every run with selected subjects runs the loaded unchanged suite twice (the specs chosen with `--test`, if given) in fresh
|
|
35
51
|
supervised children before coverage lookup or incremental verdict reuse. This
|
|
36
52
|
costs two suite runs even on a cache hit, plus a coverage-building suite run on
|
|
37
53
|
a cache miss. Both baseline passes must be green, complete and in the same
|
|
38
|
-
RSpec order at the run seed. A failure exits 3, stops mutation
|
|
54
|
+
RSpec order at the run seed. A failure exits 3, stops mutation, prints the failing example ids and the first ten lines of the
|
|
55
|
+
failure inline, and names the log and fix. Long phases
|
|
56
|
+
print their elapsed time about every 30 seconds, with the examples finished so far, the percentage and a rough estimate of the time left (it assumes equally long examples) once the suite child has logged its first count, and each phase's log holds
|
|
57
|
+
the suite's progress and full failure detail. Each suite pass
|
|
39
58
|
uses `coverage_timeout` and configured memory/process limits (the suite memory
|
|
40
|
-
budget is four times `max_memory_mb`, as for coverage).
|
|
59
|
+
budget is four times `max_memory_mb`, as for coverage). A suite child that has
|
|
60
|
+
already reported its result gets up to 5 seconds to finish exiting, under the
|
|
61
|
+
same deadline and memory/process limits as before; if it is still running then,
|
|
62
|
+
saboteur stops it and keeps the result, so a slow exit is not reported as a
|
|
63
|
+
failure. A child that ends by itself in that window keeps its own exit status. A child killed by `SIGKILL` is called a possible
|
|
64
|
+
out-of-memory kill only when the process tree was seen growing toward the
|
|
65
|
+
memory limit. Older benchmark numbers
|
|
41
66
|
need a rerun before they describe this extra fixed cost.
|
|
67
|
+
A whole-suite coverage build that records no lines for any included file
|
|
68
|
+
means the setup hid the app from Coverage (Bootsnap's compile cache is the
|
|
69
|
+
common cause), so the run fails with exit 3 instead of testing nothing, and
|
|
70
|
+
the empty map is not cached. With `--test` an empty map instead stays a
|
|
71
|
+
legitimate "no coverage" outcome, because the chosen specs may not touch the
|
|
72
|
+
selected sources.
|
|
42
73
|
Two passes detect observed instability, not every possible flaky test. See
|
|
43
74
|
[baseline stability](docs/baseline-stability.md) for limits and hook behaviour.
|
|
44
75
|
|
|
76
|
+
### Hazardous calls are not mutated, and a canary watches the tree
|
|
77
|
+
|
|
78
|
+
Mutants run in your checkout, so a mutant inside a destructive call could do
|
|
79
|
+
real damage: a string mutation inside `FileUtils.rm_rf(File.join(Dir.pwd,
|
|
80
|
+
"build"))` once turned the target into the working directory itself. The
|
|
81
|
+
hazard guard is a syntactic filter. It recognizes destructive calls written
|
|
82
|
+
as direct calls: `FileUtils.rm*`/`remove_entry*` (also as bare names under
|
|
83
|
+
`include FileUtils`), `File.delete`/`unlink`, `Dir.rmdir`/`delete`/`unlink`,
|
|
84
|
+
`rmtree`, `system`/`exec`/`spawn` (bare or on `Kernel`/`Process`), backticks
|
|
85
|
+
and `%x`, `IO.popen`, any `Open3` method, and `open` with a leading `|`. It
|
|
86
|
+
suppresses any mutant that would leave such a call in place while altering
|
|
87
|
+
code inside it or anywhere else in the same method. A mutant that removes
|
|
88
|
+
every hazardous call in the method still runs, provided its replacement
|
|
89
|
+
introduces no new hazard. The match is by name on the written call: calls
|
|
90
|
+
reached through aliases, variables, `send` or other indirection are not
|
|
91
|
+
recognized, so the guard cannot promise to find every destructive call. The
|
|
92
|
+
summary reports how many mutants were suppressed. Extend or replace the list
|
|
93
|
+
with `c.hazard` (same tag shapes: `"FileUtils.rm_rf"`, or a bare name like
|
|
94
|
+
`"system"`); `--allow-hazardous` runs the suppressed mutants anyway.
|
|
95
|
+
|
|
96
|
+
Behind the guard, a canary snapshots the tracked files (or every file,
|
|
97
|
+
outside a repository) before mutating and re-checks after each method's
|
|
98
|
+
mutants. If anything vanished, the run aborts with exit 3 and names the most
|
|
99
|
+
recent mutant's digest. That check happens after the damage and restores
|
|
100
|
+
nothing; it limits how far a bad mutant gets. Recover from your own backup,
|
|
101
|
+
a fresh checkout or a reclone - a deleted `.git` or uncommitted work is gone
|
|
102
|
+
unless you have another copy. The canary is the backstop for the destructive
|
|
103
|
+
forms the syntactic guard cannot see.
|
|
104
|
+
|
|
105
|
+
On Linux 6.2 and later (Landlock ABI 3) each mutant child also runs under
|
|
106
|
+
a Landlock sandbox: the child cannot create, write, rename, unlink or
|
|
107
|
+
truncate files in the project tree, except `tmp/`, the cache dir and the
|
|
108
|
+
system temp dir, plus anything in `c.sandbox_writable`. Every other path
|
|
109
|
+
is denied for writing, `/dev/shm` and `/dev/tty` included; `/dev/null`
|
|
110
|
+
stays writable. Inherited
|
|
111
|
+
descriptors already open for writing outside the granted dirs are closed
|
|
112
|
+
before the sandbox engages - stdio redirected to a regular file and files
|
|
113
|
+
already deleted from the tree included (stdio on a terminal, pipe or
|
|
114
|
+
`/dev/null` stays), and a descriptor whose target or flags cannot be
|
|
115
|
+
inspected is closed too rather than trusted. A `c.sandbox_writable` grant
|
|
116
|
+
that contains the project root or another granted dir warns, since it
|
|
117
|
+
defeats or duplicates the protection; reads and process execution are
|
|
118
|
+
unaffected.
|
|
119
|
+
This DOES change what your tests can write: a suite that writes elsewhere
|
|
120
|
+
in the project (`db/`, `log/`, ...) fails with EACCES in mutant children -
|
|
121
|
+
grant that directory with `c.sandbox_writable` or opt out with
|
|
122
|
+
`c.sandbox = false`. Landlock covers the filesystem only - network,
|
|
123
|
+
sockets and IPC are never restricted - and a checkout inside the system
|
|
124
|
+
temp dir cannot be protected, because everything under it stays writable
|
|
125
|
+
so that real test suites keep working - the run warns when the effective
|
|
126
|
+
write set (every grant realpath'd, the cache dir and the system temp dir
|
|
127
|
+
included) contains the project root, however it happened: a checkout
|
|
128
|
+
inside the system temp dir, `cache_dir = "."`, a symlinked `tmp`, or an
|
|
129
|
+
overlapping `c.sandbox_writable` grant. `c.sandbox = "strict"` refuses to
|
|
130
|
+
run at all in that situation, since the sandbox would protect nothing.
|
|
131
|
+
The sandbox applies to every mutant
|
|
132
|
+
child and fails closed: a child that cannot restrict itself reports an
|
|
133
|
+
error rather than run, and a kernel probe that errors (as opposed to
|
|
134
|
+
cleanly reporting Landlock absent) refuses the run.
|
|
135
|
+
|
|
136
|
+
On kernels from 5.13 to 6.1 (Landlock ABI 1-2) `truncate()` cannot be
|
|
137
|
+
restricted, so the tree is not fully read-only and the claim above does
|
|
138
|
+
not hold; the run notes once that children run with the guard and canary
|
|
139
|
+
only, exactly as on kernels without Landlock - and as on non-glibc Linux
|
|
140
|
+
whose C library cannot be located (musl is tried by soname first).
|
|
141
|
+
`c.sandbox = "strict"` turns that degraded note into a refusal;
|
|
142
|
+
`c.sandbox = false` opts out.
|
|
143
|
+
|
|
45
144
|
|
|
46
145
|
## 🚀 Try it in a minute
|
|
47
146
|
|
|
@@ -57,39 +156,41 @@ bundle exec saboteur run --subject 'YourClass#your_method'
|
|
|
57
156
|
```
|
|
58
157
|
|
|
59
158
|
Outside a Bundler project, `gem install saboteur` installs the same executable.
|
|
60
|
-
Rails
|
|
61
|
-
`rails:
|
|
62
|
-
gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
|
|
159
|
+
Rails app? Follow the [Rails quickstart](#rails-apps) below: the defaults are for
|
|
160
|
+
plain gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
|
|
63
161
|
|
|
64
162
|
Start with one method. You get counts per status, a score, timing, and a
|
|
65
163
|
unified diff for every survivor, so you can read the exact change your tests
|
|
66
164
|
missed. The JSON report lands in `.saboteur/report.json`.
|
|
67
165
|
|
|
68
166
|
📋 Requirements: Ruby 3.3 or newer on Linux or macOS (it needs `fork`), and RSpec.
|
|
167
|
+
Minitest is not supported; see [Minitest](docs/minitest.md) for why and what it would take.
|
|
69
168
|
|
|
70
169
|
## 🧠 Why it is built the way it is
|
|
71
170
|
|
|
72
171
|
**It only runs tests that can matter.** Each mutant runs only the tests whose
|
|
73
172
|
line coverage touched the changed statement. Tests that killed earlier mutants
|
|
74
173
|
of the same method go first, then the fastest, and the first failure stops the
|
|
75
|
-
run. A change no test reaches is reported as "no coverage", not as a survivor.
|
|
174
|
+
run. A change no test reaches is reported as "no coverage", not as a survivor. Branch coverage separates arms on the same line: an edit inside an arm no test took reports "no coverage (branch not taken)". Ruby does not expose `&&`/`||` right operands as branches, so those keep line-based selection. `case/in` pattern edits use the enclosing case and clause execution lines; edits in an untested clause body still stay uncovered. Whole-body replacements use every test of the method, including methods that start with a heredoc. Endless defs (`def x = ...`) are matched to tests through per-test method call coverage, since line coverage never counts their bodies. Methods defined inside blocks (`Class.new do`, `Data.define do`, `included do`) are not mutated yet; the run lists them as skipped, "defined inside a block".
|
|
76
175
|
|
|
77
|
-
**It
|
|
78
|
-
mutants
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
176
|
+
**It runs coarse mutants first, then fine mutants.** Each method first gets
|
|
177
|
+
whole-body mutants: `nil` (`true`/`false` for predicates) unless the body is empty or
|
|
178
|
+
only logs, and `raise` unless a catch-all rescue would absorb it. Fine-grained mutants run
|
|
179
|
+
regardless of the coarse verdict, including a survivor or a reused verdict.
|
|
180
|
+
Deleting a body unnoticed does not prove its smaller changes are harmless.
|
|
181
|
+
Subjects with no tests are skipped before any mutants are generated, and each
|
|
182
|
+
method is capped at `max_mutants_per_method` mutants (100 by default). Running
|
|
183
|
+
fine mutants after coarse survivors can increase run time and change scores.
|
|
83
184
|
|
|
84
185
|
**It prepares memory before it forks.** Before workers fork, saboteur warms and
|
|
85
186
|
compacts the Ruby heap ([`ForkReadiness`](lib/saboteur/execution/fork_readiness.rb)),
|
|
86
187
|
which is meant to let workers share more pages with their parent and copy less.
|
|
87
|
-
|
|
188
|
+
Observed on the pinned hashie workload (paired runs, below): about 17-18% lower peak PSS at 2 and 6 workers. Not a general guarantee.
|
|
88
189
|
|
|
89
190
|
**It is aimed at pull requests.** `--since origin/main` mutates only the methods
|
|
90
191
|
your branch changed: fine-grained mutants on the changed lines, plus the
|
|
91
192
|
whole-body mutants of those methods. On one real change in saboteur's own history, that took a file
|
|
92
|
-
from 139 mutants to 7, and median wall time fell 22.9
|
|
193
|
+
from 139 mutants to 7, and median wall time fell 22.9% (measured at d2b7abb, before the coarse-survivor gate was removed; a rerun is pending). The time saving is
|
|
93
194
|
smaller than the mutant count because building the coverage map is a fixed
|
|
94
195
|
cost, and this is one file, not a Rails app, so it is not a general speed
|
|
95
196
|
claim. The full method, raw timings and
|
|
@@ -97,8 +198,8 @@ caveats are in [bench/since](bench/since/README.md).
|
|
|
97
198
|
|
|
98
199
|
**It checks its own kills.** After a method's mutants run, saboteur re-runs the
|
|
99
200
|
killing tests against the *unchanged* method. If they fail there too, the kill
|
|
100
|
-
is not credited and becomes an error. In
|
|
101
|
-
|
|
201
|
+
is not credited and becomes an error. In one comparison on lobsters, 0 of 20
|
|
202
|
+
randomly sampled kills were fake. Every mutant runs in a supervised child with deadline,
|
|
102
203
|
memory and process limits, and saboteur tracks and kills the processes it
|
|
103
204
|
starts.
|
|
104
205
|
|
|
@@ -109,7 +210,8 @@ schema, and you can add your own mutants through a small public interface.
|
|
|
109
210
|
|
|
110
211
|
```sh
|
|
111
212
|
bundle exec saboteur run # whole project
|
|
112
|
-
bundle exec saboteur doctor
|
|
213
|
+
bundle exec saboteur doctor # environment checks only
|
|
214
|
+
bundle exec saboteur doctor --suite --seed 42 # plus two unchanged suite checks
|
|
113
215
|
bundle exec saboteur run --since origin/main # methods changed on this branch
|
|
114
216
|
bundle exec saboteur run --subject 'Billing::Invoice#total'
|
|
115
217
|
bundle exec saboteur run --subject Billing # a namespace
|
|
@@ -118,42 +220,92 @@ bundle exec saboteur run --seed 1234 # repeat a run that used seed
|
|
|
118
220
|
bundle exec saboteur run lib/a.rb lib/b.rb --test spec/a_spec.rb --test spec/b_spec.rb
|
|
119
221
|
bundle exec saboteur run --threshold 90 # exit 1 when the score is below 90 %
|
|
120
222
|
bundle exec saboteur run --id DIGEST --seed 1234 # replay one full current mutant digest
|
|
223
|
+
bundle exec saboteur run --fail-fast # stop after the first subject with a survivor
|
|
224
|
+
bundle exec saboteur run --matrix # every covering test per mutant: blind and redundant tests
|
|
225
|
+
bundle exec saboteur run --verbose --no-color # add score split, timings and kill causes; no colour
|
|
121
226
|
```
|
|
122
227
|
|
|
123
228
|
Positional operands name the sources to mutate: Ruby files, or directories
|
|
124
229
|
(all their `.rb` files). Each must exist; a typo or a non-Ruby file exits 2
|
|
125
230
|
before anything runs. They replace `include` and `exclude`. `--test`
|
|
126
231
|
(repeatable; spec files or directories, with an optional `:LINE` or `[1:2]`
|
|
127
|
-
selector; needs at least one source
|
|
232
|
+
selector; needs at least one source, unless `--since` is given - the changed
|
|
233
|
+
files are the sources) replaces only the files RSpec runs. Other
|
|
128
234
|
`rspec_args` and `.rspec` options such as `--require` stay. The coverage map
|
|
129
235
|
is built from only those specs, each mutant runs only the ones that cover it,
|
|
130
236
|
and the selection is part of the coverage and `--incremental` cache keys.
|
|
131
237
|
|
|
132
|
-
`--threshold PERCENT` (0 to 100, or `threshold
|
|
238
|
+
`--threshold PERCENT` (0 to 100, or `c.threshold` in `config/saboteur.rb`) is for CI.
|
|
133
239
|
The run exits 1 when the score is below PERCENT and 0 when it is at or above,
|
|
134
240
|
even with survivors. It fails closed: a run that tested no mutant has no score
|
|
135
241
|
and exits 1, and a run where any mutant errored or hit a resource limit exits 3,
|
|
136
242
|
because the score covers only part of the run. Without a threshold, any
|
|
137
|
-
unignored survivor exits 1.
|
|
243
|
+
unignored survivor exits 1. Either way, a run where more than half of the
|
|
244
|
+
executed mutants errored exits 3: errored mutants are outside the score,
|
|
245
|
+
so such a run tested almost nothing.
|
|
246
|
+
|
|
247
|
+
`--max-timeouts N` and `--max-errors N` (or `c.max_timeouts` / `c.max_errors`)
|
|
248
|
+
are two more CI gates. Each takes a count or a percentage of the executed
|
|
249
|
+
mutants (`--max-errors 5%`). The run exits 1 when the count (or share) is
|
|
250
|
+
above the limit, with `exitCause` `maxTimeouts` or `maxErrors` in the JSON report.
|
|
251
|
+
`--threshold` already compares the covered score, `(killed + timeout) /
|
|
252
|
+
(killed + survived + timeout)`, so no separate score flag exists. With
|
|
253
|
+
`--max-errors`, errored mutants are judged by that limit instead of making a
|
|
254
|
+
`--threshold` run exit 3. Survivors and a score below the threshold keep
|
|
255
|
+
their own causes when they fail too. The gates cannot be combined with `--agent`.
|
|
256
|
+
The GitHub Action has no input for them yet.
|
|
138
257
|
|
|
139
258
|
Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
|
|
140
259
|
`--config PATH`, `--integration NAME`, `--reeval class_eval|nesting`,
|
|
141
|
-
`--json PATH`, `--no-cache` (rebuild the coverage map)
|
|
260
|
+
`--json PATH`, `--no-cache` (rebuild the coverage map), `--allow-focus`.
|
|
261
|
+
|
|
262
|
+
A stray `fit`, `fdescribe` or `:focus` example, with `filter_run_when_matching :focus`, makes RSpec skip
|
|
263
|
+
every other example. The baseline stays green and the score would cover only the focused ones, so
|
|
264
|
+
saboteur exits 2 and names the focused examples. Remove the focus, or pass `--allow-focus` (config
|
|
265
|
+
`c.allow_focus = true`) to run the narrowed suite. Filters you name in `rspec_args` or `.rspec`, such
|
|
266
|
+
as `--tag slow`, and example selectors such as `--test spec/a_spec.rb:12`, are on purpose and are not refused. The
|
|
267
|
+
check compares what RSpec runs with what it would run without the stray focus, so examples RSpec leaves out anyway
|
|
268
|
+
(`:if` / `:unless`, `--tag`, exclusions, selectors in another file) never count as hidden.
|
|
142
269
|
|
|
143
270
|
For pull requests, start with `--since` against your target branch. This is a
|
|
144
271
|
workflow recommendation, not a CLI default: plain `run` still checks the whole
|
|
145
|
-
configured project.
|
|
272
|
+
configured project. To scope the specs as well, pass them with `--test` and no
|
|
273
|
+
source operands: `bundle exec saboteur run --since origin/main --test spec/billing_spec.rb`.
|
|
274
|
+
With `--since` and no `--test`, the run warns before it starts that the two baseline passes
|
|
275
|
+
run the loaded suite (a cold coverage cache adds one more pass), and how to
|
|
276
|
+
scope them.
|
|
277
|
+
|
|
278
|
+
`--since origin/main --guess-tests` does that scoping for you: it picks the spec
|
|
279
|
+
named after each changed source file (`lib/a/b.rb` runs `spec/a/b_spec.rb`; `app/`
|
|
280
|
+
files and `spec/lib/` work too) and runs only those for the baseline passes, the
|
|
281
|
+
coverage pass and every mutant, as `--test` would. It is a naming guess, not
|
|
282
|
+
coverage: a spec that reaches the changed file only through another file does not
|
|
283
|
+
run: a mutant only that spec would kill can survive, and a line that none of the
|
|
284
|
+
guessed specs executes is reported as NoCoverage. The agent JSON records the guessed specs as its
|
|
285
|
+
`tests`, so its follow-up commands keep the same `--test` scope (with `--no-guess-tests`, so a `guess_tests`
|
|
286
|
+
in the config file cannot conflict) instead of widening to the whole suite. If any changed file has no such
|
|
287
|
+
spec, or nothing changed, the whole suite runs as before and the run says so.
|
|
288
|
+
`--guess-tests` needs `--since` and cannot be combined with `--test`. Keep whole-project runs on main: diff mode does not check every method affected by a
|
|
146
289
|
dependency change. [The pull-request guide](docs/pull-requests.md) covers
|
|
147
290
|
checkout history, a GitHub Actions example, configuration and limits.
|
|
148
291
|
|
|
292
|
+
Doctor warns when an ignore subject matches no method discovered in the included
|
|
293
|
+
source files. Check the exact `Class#method` or `Class.method` form and the
|
|
294
|
+
include/exclude settings; warnings never edit or delete ignore entries.
|
|
295
|
+
A malformed ignore file fails doctor with a config error before target boot.
|
|
296
|
+
|
|
149
297
|
`saboteur doctor` prints Ruby, coverage, integration, Rails/SimpleCov and boot-file
|
|
150
|
-
checks
|
|
298
|
+
checks. Add `--suite` to also run the unchanged suite twice with the chosen seed
|
|
299
|
+
(`--test PATH`, repeatable, limits those runs to some specs and implies `--suite`),
|
|
300
|
+
since that takes hours on a large suite. It writes no
|
|
151
301
|
mutation reports or coverage/verdict caches. Target boot files and tests still
|
|
152
302
|
run, so their normal side effects remain. See the [doctor guide](docs/guides/doctor.md).
|
|
153
303
|
|
|
154
304
|
`--id` runs exactly one current mutant with its coverage-selected tests, bypassing
|
|
155
|
-
per-method caps,
|
|
156
|
-
It still honors `--subject` and `--since` changed-line selection
|
|
305
|
+
per-method caps, ignore suppression and incremental verdict reuse.
|
|
306
|
+
It still honors `--subject` and `--since` changed-line selection, and the
|
|
307
|
+
hazard guard: a digest that names a suppressed mutant is refused unless
|
|
308
|
+
`--allow-hazardous` is also given. See the
|
|
157
309
|
[replay guide](docs/guides/replay.md) for digest format, checks and limits.
|
|
158
310
|
|
|
159
311
|
Every test run starts from the same random seed (1 by default), so tests that
|
|
@@ -161,22 +313,86 @@ use `rand` see the same values in each mutant's run and in the check that
|
|
|
161
313
|
confirms a kill. The report prints the seed; pass it to `--seed` to repeat a
|
|
162
314
|
run.
|
|
163
315
|
|
|
164
|
-
`--incremental` reuses an earlier verdict only while the method, the
|
|
165
|
-
its covering tests and their files, the lockfile
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
316
|
+
`--incremental` reuses an earlier verdict only while the method, the whole
|
|
317
|
+
file it lives in, the mutant, its covering tests and their files, the lockfile
|
|
318
|
+
and the config are all unchanged. So a constant or helper changed in the same
|
|
319
|
+
file is seen. It cannot see a change in another file that the method uses (a
|
|
320
|
+
method it calls, a constant defined elsewhere), so run without it now and then
|
|
321
|
+
(and in CI on the main branch).
|
|
322
|
+
|
|
323
|
+
Output: survivors first, then a short summary last, so the summary stays on
|
|
324
|
+
screen. Survivors are grouped by file, then method, then line. Each line is
|
|
325
|
+
printed once as `path:line in Subject`, with the source line and the tests that
|
|
326
|
+
covered it, then one compact line per mutation: `operator original -> replacement digest`,
|
|
327
|
+
and a hint on what kind of test would kill it. The summary has the counts per
|
|
328
|
+
status, the score, a table of scores per file (worst first) and a one-line
|
|
329
|
+
legend. `--verbose` adds the score split, mean time, whole-body score and
|
|
330
|
+
kill causes. Colour is on when stdout is a terminal and `NO_COLOR` is not set;
|
|
331
|
+
`--color` and `--no-color` override that. While mutants run, stderr shows
|
|
332
|
+
`done/total (percent) eta`: a live counter on a terminal, and dots with a
|
|
333
|
+
counter every 50 mutants otherwise. Skipped and reused mutants are not
|
|
334
|
+
counted, so the count can end below the total.
|
|
335
|
+
The tests that covered the survivor's line and
|
|
336
|
+
still passed with the mutant applied are listed (the first five, likeliest killers
|
|
337
|
+
first; the rest are under `coveredBy` in the JSON report). The missing
|
|
338
|
+
check may belong in one of them. Line coverage can overstate which tests
|
|
339
|
+
exercise a line, so treat the list as a lead. A survivor reused with
|
|
340
|
+
`--incremental` was not run again, and the list says its tests passed in an
|
|
341
|
+
earlier run. `--fail-fast` (serial runs only) stops after the first subject
|
|
342
|
+
that has a survivor the ignore list does not accept. The report then covers
|
|
343
|
+
only the subjects that ran: the terminal says which subjects did not run, the
|
|
344
|
+
JSON report has an `earlyStop` field naming them (`exitCause` is `failFast`),
|
|
345
|
+
`--agent` reports status `partial` with the reason `fail_fast`, and the run
|
|
346
|
+
always exits 1, so no `--threshold` can pass it. It cannot be used with
|
|
347
|
+
`baseline` evidence. Survivors that share a line are printed together only
|
|
348
|
+
when their covering tests, reuse status and changed lines are identical;
|
|
349
|
+
otherwise each distinct set of evidence is printed above its own mutations.
|
|
350
|
+
Hints are suggestions, not diagnoses: a survivor can also be an equivalent
|
|
351
|
+
mutant. The HTML page shows the same list in the
|
|
352
|
+
survivor's notes. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
|
|
171
353
|
report (one file, no scripts, no network) to `.saboteur/report.html`. Change
|
|
172
|
-
the paths with `--json` / `--html` or `json_report` / `html_report` in
|
|
173
|
-
|
|
354
|
+
the paths with `--json` / `--html` or `c.json_report` / `c.html_report` in
|
|
355
|
+
`config/saboteur.rb`; set either to `nil` to turn that report off. The two paths
|
|
174
356
|
must differ. The HTML page shows the covered mutation score
|
|
175
|
-
`(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage
|
|
176
|
-
out and keeps survivors accepted on the ignore list
|
|
357
|
+
`(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage,
|
|
358
|
+
RanAtLoad and Unreachable out and keeps survivors accepted on the ignore list
|
|
359
|
+
in the denominator. A tool
|
|
177
360
|
that reads the JSON report may show a different score, because the schema counts
|
|
178
361
|
NoCoverage as undetected.
|
|
179
362
|
|
|
363
|
+
`--matrix` runs every covering test against every mutant instead of stopping at
|
|
364
|
+
the first failing one, so a killed mutant lists all of its killers. After the
|
|
365
|
+
survivors the report prints two lists: blind tests (they covered mutants and
|
|
366
|
+
killed none) and redundant tests (they killed some, and every one of those
|
|
367
|
+
mutants had another killer). The run is slower, by up to the number of covering
|
|
368
|
+
tests per mutant. `--matrix` cannot be combined with `--incremental`, because a
|
|
369
|
+
reused verdict lists only the first killer. A redundant test is a lead, not a
|
|
370
|
+
verdict: it may guard behavior no selected mutant touches. A selection with
|
|
371
|
+
`--since` or `--subject` describes only the selected methods.
|
|
372
|
+
|
|
373
|
+
[Scores](docs/scores.md) defines each number on this page: the covered score, the
|
|
374
|
+
score a Stryker dashboard shows, the assertion and crash split, and what makes a run
|
|
375
|
+
incomplete.
|
|
376
|
+
|
|
377
|
+
The score is split by how the mutant was killed. A crash kill (a `NoMethodError`,
|
|
378
|
+
a `TypeError`, a parse error) shows the code ran and broke, not that a test
|
|
379
|
+
checked a value. The terminal prints
|
|
380
|
+
`score 100.0 % (assertion 50.0 %, crash 50.0 %)`: each share is a part of
|
|
381
|
+
tested mutants, so the parts add up to the score. Timeouts and kills with no
|
|
382
|
+
recorded cause (a verdict reused from an older run, a test that called `exit`,
|
|
383
|
+
a failure outside any example) are named when there are some. Whole-body
|
|
384
|
+
mutants (`body_nil`, `body_true`, `body_false`, `body_raise`) are killed by
|
|
385
|
+
almost any test that runs the method, so a second line scores the run without
|
|
386
|
+
them, and shows them alone. The headline score and `--threshold` still count every
|
|
387
|
+
mutant. The JSON report carries both splits under `saboteur.scores`
|
|
388
|
+
(`all` and `withoutWholeBodyMutants`), and the HTML page prints them under
|
|
389
|
+
the score. The JSON also records why the run exited the way it did under
|
|
390
|
+
`saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`,
|
|
391
|
+
`errorMajority`, `incompleteUnderThreshold`). In `--agent` mode the cause
|
|
392
|
+
follows the agent verdict, so the report and the process exit classify the
|
|
393
|
+
run the same way: `ok` for clean, `survivors` for survivors, and otherwise
|
|
394
|
+
`errorMajority` when most mutants errored or `incomplete`.
|
|
395
|
+
|
|
180
396
|
For code-review surfaces, the JSON report also carries a `saboteur.findings`
|
|
181
397
|
section: a bounded, deterministically ordered list of survivors, coarse
|
|
182
398
|
mutants first (a surviving coarse mutant hints that no test pins the method's
|
|
@@ -188,6 +404,17 @@ Survivors accepted on the ignore list are listed separately with their
|
|
|
188
404
|
reason, and `survivorsTruncated` and `acceptedTruncated` count what the bound
|
|
189
405
|
left out; the full data stays in the schema's `files` section.
|
|
190
406
|
|
|
407
|
+
In diff mode (`--since`), a mutant is reported at the line where its statement
|
|
408
|
+
starts. A statement that spans several lines can start a line or two above the
|
|
409
|
+
line that changed. Under each survivor the terminal therefore prints
|
|
410
|
+
`changed line: path:N`, the changed line or lines (the first three) the mutant
|
|
411
|
+
touches, and each finding carries them as `changedLines`. Use `changedLines`,
|
|
412
|
+
not `line`, to put a note on a changed line of the pull-request diff. A
|
|
413
|
+
whole-body mutant of a method whose only change is its signature touches no
|
|
414
|
+
changed line and prints none.
|
|
415
|
+
|
|
416
|
+
New to survivors? The [reading survivors guide](docs/guides/reading-survivors.md) walks one from the diff to the test that kills it.
|
|
417
|
+
|
|
191
418
|
Exit codes: 0 no survivors (or all on the ignore list), or the score meets
|
|
192
419
|
`--threshold`; 1 survivors, or the score is below `--threshold` or missing;
|
|
193
420
|
2 bad configuration or a missing source or test; 3 the tool failed (with
|
|
@@ -203,38 +430,197 @@ other errors and reused cached kills are not included. See
|
|
|
203
430
|
|
|
204
431
|
## ⚙️ Configuration
|
|
205
432
|
|
|
206
|
-
|
|
433
|
+
`config/saboteur.rb` in the project root; CLI flags override it. Keep the shared
|
|
207
434
|
config usable for whole-project runs and pass `--since` in the pull-request
|
|
208
|
-
job. A PR-only config may set `since
|
|
209
|
-
`since
|
|
210
|
-
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
435
|
+
job. A PR-only config may set `c.since = "origin/main"`; omit `since` (or use
|
|
436
|
+
`c.since = nil`) in the config for audits on main. Use your actual target branch.
|
|
437
|
+
|
|
438
|
+
```ruby
|
|
439
|
+
Saboteur.configure do |c|
|
|
440
|
+
c.include = ["lib/**/*.rb"]
|
|
441
|
+
c.exclude = ["lib/my_gem/version.rb"]
|
|
442
|
+
c.require = [] # boot files, e.g. ./config/environment
|
|
443
|
+
c.whole_file = [] # subject ids or globs to re-evaluate as a whole file (see below)
|
|
444
|
+
c.reeval = "nesting" # or "class_eval"; see docs/design.md 6.1
|
|
445
|
+
c.timeout_factor = 3 # per-mutant deadline = factor x its tests' recorded time
|
|
446
|
+
c.timeout_min = 5
|
|
447
|
+
c.max_memory_mb = 2048 # memory growth cap per mutant process tree
|
|
448
|
+
c.max_processes = 64
|
|
449
|
+
c.max_mutants_per_method = 100 # positive integer; raise it for a broader run
|
|
450
|
+
c.hazard = Saboteur::MutationSource::HazardGuard::DEFAULT_HAZARD # calls whose arguments are never mutated
|
|
451
|
+
c.sandbox = true # read-only project tree for mutant children (Linux 6.2+); "strict" refuses to run degraded
|
|
452
|
+
c.sandbox_writable = [] # extra dirs mutant children may write, e.g. ["db", "log"]
|
|
453
|
+
c.ignore_file = "config/saboteur_ignore.rb"
|
|
454
|
+
c.seed = 1 # Ruby's random seed in every test run
|
|
455
|
+
c.allow_focus = false # true runs a suite narrowed by fit/fdescribe/:focus instead of exit 2
|
|
456
|
+
c.fingerprint_extra = [] # extra globs whose contents invalidate caches, e.g. ["data/*.csv"]
|
|
457
|
+
end
|
|
223
458
|
```
|
|
224
459
|
|
|
225
460
|
### Rails apps
|
|
226
461
|
|
|
227
|
-
The defaults target a plain gem, so a Rails
|
|
462
|
+
Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a Rails
|
|
463
|
+
app needs a few settings; the generator writes them for you.
|
|
464
|
+
|
|
465
|
+
**1. Add the gem**
|
|
466
|
+
|
|
467
|
+
```sh
|
|
468
|
+
bundle add saboteur --group test
|
|
469
|
+
```
|
|
470
|
+
|
|
471
|
+
**2. Generate the setup**
|
|
472
|
+
|
|
473
|
+
```sh
|
|
474
|
+
bundle exec rails generate saboteur:install
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets `RAILS_ENV=test`
|
|
478
|
+
for you) and a `.gitignore` entry for `.saboteur/`. It never overwrites an
|
|
479
|
+
existing file unless you pass `--force`, and never edits your Gemfile.
|
|
480
|
+
|
|
481
|
+
Prefer to do it by hand, or on a version without the generator? Put this in
|
|
482
|
+
`config/saboteur.rb`:
|
|
483
|
+
|
|
484
|
+
```ruby
|
|
485
|
+
Saboteur.configure do |c|
|
|
486
|
+
c.include = ["app/**/*.rb"]
|
|
487
|
+
c.require = ["./config/environment"] # boots the app
|
|
488
|
+
c.rails = true # reconnects ActiveRecord after fork, per-worker databases
|
|
489
|
+
c.jobs = 4 # optional; the default is 1
|
|
490
|
+
end
|
|
491
|
+
```
|
|
492
|
+
|
|
493
|
+
**3. Check the setup**
|
|
228
494
|
|
|
229
|
-
```
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
495
|
+
```sh
|
|
496
|
+
bin/saboteur doctor
|
|
497
|
+
```
|
|
498
|
+
|
|
499
|
+
Doctor boots your app, checks the config and the test database host, and runs
|
|
500
|
+
your suite twice before any mutation. Fix whatever it reports first. Turn
|
|
501
|
+
SimpleCov off for the run if you use it.
|
|
502
|
+
|
|
503
|
+
**4. Test what your branch changed**
|
|
504
|
+
|
|
505
|
+
```sh
|
|
506
|
+
bin/saboteur run --since main
|
|
234
507
|
```
|
|
235
508
|
|
|
236
|
-
|
|
237
|
-
|
|
509
|
+
It mutates only the methods changed since `main` and reports each survivor with
|
|
510
|
+
its diff: a change your tests did not notice. Write a test that fails on that
|
|
511
|
+
diff, then run again. Use `--since origin/main` in CI. Without the generator, use
|
|
512
|
+
`bundle exec saboteur` and `RAILS_ENV=test` instead of `bin/saboteur`.
|
|
513
|
+
|
|
514
|
+
saboteur puts the app's `spec/` directory on the load path before
|
|
515
|
+
processing `require`, matching rspec, so a boot file like a dummy app's
|
|
516
|
+
`spec/rails_helper.rb` can require its siblings.
|
|
517
|
+
|
|
518
|
+
With `jobs` above 1 on SQLite, each extra worker gets a copy of the test
|
|
519
|
+
database made through SQLite's backup API, so the copy is a consistent
|
|
520
|
+
snapshot. A relative `database:` path resolves against `Rails.root`, the
|
|
521
|
+
same as Rails 7.1+ does, for the copy and for its cleanup. The copy
|
|
522
|
+
checkpoints the source first (`PRAGMA wal_checkpoint(TRUNCATE)`) and
|
|
523
|
+
refuses to run when a reader keeps the WAL busy, because copying on
|
|
524
|
+
would silently miss committed rows. A source file that does not exist is
|
|
525
|
+
an error, not a skipped copy. An in-memory SQLite database (`:memory:`) is
|
|
526
|
+
never copied, renamed or disconnected. After the app boots, each in-memory
|
|
527
|
+
database is snapshotted to a temporary file through SQLite's backup API,
|
|
528
|
+
and every forked child restores the snapshot into its own fresh database -
|
|
529
|
+
a child cannot keep the parent's connection (ActiveRecord discards
|
|
530
|
+
inherited pools and sqlite3 closes inherited writable handles), so without
|
|
531
|
+
the snapshot it would see an empty database.
|
|
532
|
+
|
|
533
|
+
For a Rails engine, point `require` at the dummy app's environment -
|
|
534
|
+
`test/dummy/config/environment.rb`, usually loaded for you by
|
|
535
|
+
`spec/rails_helper.rb`. Only a dynamically created dummy app has no
|
|
536
|
+
environment file; there, require the helper that calls setup instead,
|
|
537
|
+
such as `DummyApp.setup` in `spec/rails_helper.rb`.
|
|
538
|
+
|
|
539
|
+
Bootsnap needs no extra setting: bytecode from its compile cache carries no
|
|
540
|
+
coverage counters, so saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its
|
|
541
|
+
own process when the variable is not already set (Bootsnap treats any set
|
|
542
|
+
value as off) and runs the eager load with Coverage active. App code is seen
|
|
543
|
+
by Coverage with the app unchanged.
|
|
544
|
+
|
|
545
|
+
With `jobs: 4` (or more), each extra worker gets a copy of every test
|
|
546
|
+
database, named `<database>_saboteur2` and up, made with
|
|
547
|
+
`CREATE DATABASE ... TEMPLATE` on Postgres or a file copy on SQLite.
|
|
548
|
+
saboteur drops the copies when the run ends, on a best-effort basis: a
|
|
549
|
+
drop that fails is reported as a warning and never changes the run's
|
|
550
|
+
outcome. A copy left behind - by a failed drop or a killed run - is
|
|
551
|
+
reclaimed by the next run, whose prepare starts from
|
|
552
|
+
`DROP DATABASE IF EXISTS`. MySQL is not supported for `jobs` above 1.
|
|
553
|
+
|
|
554
|
+
Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
|
|
555
|
+
`db/schema.rb` and `config/*.yml` (including `database.yml`) are part of the
|
|
556
|
+
cache fingerprint automatically. Use `fingerprint_extra` for any other file your
|
|
557
|
+
tests read.
|
|
558
|
+
|
|
559
|
+
### Load-time code, captured methods and Sorbet signatures
|
|
560
|
+
|
|
561
|
+
Code that runs while the target boots, before tests run, cannot be checked by
|
|
562
|
+
replacing a method later. saboteur reports these changes as `ran_at_load` rather
|
|
563
|
+
than survivors. Calls from test setup, including RSpec `before(:all)`, are still
|
|
564
|
+
checked. Load-time detection follows the `ran_at_load` idea in
|
|
565
|
+
[mutineer #187](https://github.com/davidteren/mutineer/pull/187).
|
|
566
|
+
|
|
567
|
+
A saved `Method` or `UnboundMethod` can keep calling the old definition after
|
|
568
|
+
saboteur replaces it. When tests pass without calling the replacement and a
|
|
569
|
+
saved method still points to the old definition, the result is `unreachable`.
|
|
570
|
+
A targeted execution check must also confirm the replacement was not called.
|
|
571
|
+
Each mutant is checked separately; one unreachable result does not skip later mutants.
|
|
572
|
+
The reason names the old definition's source location. A lambda that looks up
|
|
573
|
+
the method when it runs is still checked normally.
|
|
574
|
+
|
|
575
|
+
Coverage caches now store both per-test method hits and load-time line hits.
|
|
576
|
+
Older caches are rebuilt; load-time accounting preserves endless-method coverage.
|
|
577
|
+
|
|
578
|
+
One-line and endless method definitions need boot method-call counters to prove
|
|
579
|
+
the body ran at load; merely defining them is not execution evidence. If methods
|
|
580
|
+
share a physical line and any sibling is uncalled, the line is ambiguous and is
|
|
581
|
+
conservatively reported as no coverage, even for its called sibling.
|
|
582
|
+
Agent mode reports ran-at-load and unreachable work as incomplete.
|
|
583
|
+
|
|
584
|
+
A whole-suite map with only load-time hits is valid and reports load-only subjects
|
|
585
|
+
with their ran-at-load reason, rather than failing as an empty coverage build.
|
|
586
|
+
It has no mutation score when no mutant can be tested.
|
|
587
|
+
|
|
588
|
+
Both statuses are listed separately in terminal output and excluded from the
|
|
589
|
+
mutation score. Stryker JSON reports them as `Ignored`, with the explanation in
|
|
590
|
+
`statusReason`; they are not entries on your survivor ignore list.
|
|
591
|
+
|
|
592
|
+
Methods wrapped by `sorbet-runtime` signatures are checked rather than skipped
|
|
593
|
+
as shadowed definitions. saboteur re-evaluates the adjacent `sig` with the
|
|
594
|
+
changed method so the replacement keeps its signature wrapper. Missing adjacent signatures and unknown wrappers are skipped when saboteur cannot
|
|
595
|
+
preserve the loaded method semantics.
|
|
596
|
+
|
|
597
|
+
### Whole-file re-evaluation (opt-in)
|
|
598
|
+
|
|
599
|
+
Surgical re-evaluation refuses a method whose owner it cannot resolve or whose
|
|
600
|
+
loaded definition is not the one in the file. The skip reason ends with a hint.
|
|
601
|
+
List such subjects in `c.whole_file` (ids or globs such as `"Point#distance"` or
|
|
602
|
+
`"Geometry::*"`) and the child evaluates the whole mutated file instead. This
|
|
603
|
+
applies to methods saboteur already finds as subjects; it does not make new
|
|
604
|
+
kinds of definitions selectable (methods inside `Data.define` or `Struct.new`
|
|
605
|
+
blocks are still not found). The file's top-level code then runs again in every
|
|
606
|
+
child, after the run's seed is set: load-time side effects repeat and constants
|
|
607
|
+
it assigns are rebuilt, with redefinition warnings silenced. If a later
|
|
608
|
+
definition in the same file replaces the mutated method, the mutant could never
|
|
609
|
+
run, so a survivor there is reported as unreachable instead. It is off by
|
|
610
|
+
default, and the cache key includes the list.
|
|
611
|
+
Signature text alone does not remove mutants. `body_nil` and last-statement
|
|
612
|
+
removal remain available even when `sig` declares a non-nil return type.
|
|
613
|
+
`checked(:never)`, unchecked `checked(:tests)` signatures, configured default
|
|
614
|
+
levels, custom validation handlers and wrapper shims can all allow nil to
|
|
615
|
+
reach real assertions. saboteur does not verify enforcement, evaluate return
|
|
616
|
+
type expressions or infer types from constant names. Aliases, enclosing
|
|
617
|
+
namespace constants and transformed generic types therefore cannot change
|
|
618
|
+
mutation generation. RBS signatures are not read yet. Type-only crash kills
|
|
619
|
+
remain visible in the crash share, not silently pruned.
|
|
620
|
+
|
|
621
|
+
The Sorbet E2E fixture is a wrapper/registry shim without runtime type checking.
|
|
622
|
+
It tests re-evaluation and assertion detection, not real Sorbet enforcement or
|
|
623
|
+
proof of type-check noise reduction.
|
|
238
624
|
|
|
239
625
|
### Skipped subjects and the mutant cap
|
|
240
626
|
|
|
@@ -261,11 +647,13 @@ mutants; the limit does not bound how many proposals a custom source creates.
|
|
|
261
647
|
|
|
262
648
|
Every survivor is printed with a digest. Accept it with a reason:
|
|
263
649
|
|
|
264
|
-
```
|
|
265
|
-
# .
|
|
266
|
-
|
|
267
|
-
mutation
|
|
268
|
-
|
|
650
|
+
```ruby
|
|
651
|
+
# config/saboteur_ignore.rb
|
|
652
|
+
Saboteur.ignore do |i|
|
|
653
|
+
i.mutation "1a2b3c4d5e6f7a8b",
|
|
654
|
+
subject: "Billing::Invoice#total",
|
|
655
|
+
reason: "equivalent: amounts are always whole cents"
|
|
656
|
+
end
|
|
269
657
|
```
|
|
270
658
|
|
|
271
659
|
An entry without a reason is rejected.
|
|
@@ -320,6 +708,67 @@ No speed, type-safety or real-fault detection improvement is claimed. See
|
|
|
320
708
|
[the literal operator rationale](docs/literal-operators.md) for fault evidence
|
|
321
709
|
and limits, including string encoding/frozen-value behavior.
|
|
322
710
|
|
|
711
|
+
## Chain removal
|
|
712
|
+
|
|
713
|
+
Builtin drops one link from the middle of a dotted method chain: `user.name.strip`
|
|
714
|
+
becomes `user.strip`, and `a.b.c.d` gives `a.c.d` and `a.b.d`. The first receiver
|
|
715
|
+
and the last call stay, so a test has to notice that a step in between went
|
|
716
|
+
missing. The link goes whole, with its arguments and block. The operator and
|
|
717
|
+
source name is `chain_removal`.
|
|
718
|
+
|
|
719
|
+
Operators, index calls and bare-name receivers are not chains and are unchanged.
|
|
720
|
+
The target of a write (`user.account.profile.name = 1`, `+=`, `||=`, `a.b[0] = 1`)
|
|
721
|
+
is left whole: no link of its receiver chain is dropped, however long. Only
|
|
722
|
+
the value on the right can be mutated. `freeze`, `dup`, `clone`, `itself`
|
|
723
|
+
and `tap` links are never removed. Tests can observe frozenness, copies and a
|
|
724
|
+
`tap` block's effects, but those are other contracts than a missing step in a
|
|
725
|
+
query chain, and dropping such a link mostly yields noise, so this operator
|
|
726
|
+
leaves them out. Arid statements and nested definitions are skipped.
|
|
727
|
+
|
|
728
|
+
Chain removal is on by default, like `argument_removal` and `pair_removal`.
|
|
729
|
+
Its proposals follow the other fine proposals, with the same validation,
|
|
730
|
+
deduplication and `max_mutants_per_method` cap.
|
|
731
|
+
|
|
732
|
+
## Argument removal
|
|
733
|
+
|
|
734
|
+
Builtin removes one ordinary positional argument from a parenthesized call.
|
|
735
|
+
For `Model.includes(:a, :b, c: :d)`, the two proposals remove `:a` or `:b`,
|
|
736
|
+
without deleting the statement or changing `c: :d`. This lets a preload check
|
|
737
|
+
exercise missing associations instead of only a missing return value.
|
|
738
|
+
The operator and source name is `argument_removal`.
|
|
739
|
+
|
|
740
|
+
Only parenthesized calls are supported in this first version. Keyword hashes,
|
|
741
|
+
splats, argument forwarding, bracket indexing/assignment syntax and arguments containing
|
|
742
|
+
heredocs are unchanged. Explicit hash arguments count as positional arguments.
|
|
743
|
+
Blocks and keyword pairs remain intact; separators come from Prism tokens,
|
|
744
|
+
so commas inside strings or comments are not mistaken for separators.
|
|
745
|
+
Arid statements, parameter defaults and nested definitions are skipped.
|
|
746
|
+
|
|
747
|
+
These fine proposals follow existing builtin proposals, with the same validation,
|
|
748
|
+
deduplication and `max_mutants_per_method` cap. Earlier digests and relative order
|
|
749
|
+
stay unchanged for unchanged subjects. Extra proposals can change scores or be
|
|
750
|
+
omitted by the cap. Removing an argument can still raise an arity error; no
|
|
751
|
+
non-crashing guarantee is made. The regression uses a small preload-like receiver
|
|
752
|
+
and checks the missing associations, not an ActiveRecord SQL benchmark. See
|
|
753
|
+
[the validation notes](docs/argument-removal.md) for the pinned snapshot migration.
|
|
754
|
+
|
|
755
|
+
## Hash and keyword pair removal
|
|
756
|
+
|
|
757
|
+
Builtin removes one key/value association from a hash literal or keyword list.
|
|
758
|
+
`Model.includes(:a, :b, c: :d)` becomes `Model.includes(:a, :b )`, so a nested
|
|
759
|
+
preload guard can be exercised without deleting the whole statement. A sole
|
|
760
|
+
hash pair becomes `{}`. Operator and source name: `pair_removal`.
|
|
761
|
+
|
|
762
|
+
Both parenthesized and command calls are supported. Splats stay intact; pairs
|
|
763
|
+
containing heredocs, parameter defaults, arid statements and nested definitions
|
|
764
|
+
are skipped. Token-based separator edits preserve comments and blocks. Nested
|
|
765
|
+
hash values and callable bodies are visited. Removing a required keyword can
|
|
766
|
+
still crash; the regression uses a preload-like receiver, not an ActiveRecord
|
|
767
|
+
SQL benchmark. These fine proposals follow earlier builtin proposals and use
|
|
768
|
+
the usual validation, deduplication, order and `max_mutants_per_method` cap.
|
|
769
|
+
More proposals can change scores and cap coverage. See
|
|
770
|
+
[the validation notes](docs/pair-removal.md) for snapshot changes.
|
|
771
|
+
|
|
323
772
|
## 🔌 Extending
|
|
324
773
|
|
|
325
774
|
Public API, stable across minor versions:
|
|
@@ -334,9 +783,10 @@ Public API, stable across minor versions:
|
|
|
334
783
|
|
|
335
784
|
## 🌍 Where it has run
|
|
336
785
|
|
|
337
|
-
An early release (0.
|
|
786
|
+
An early release (0.4.0). Tested on the RSpec suites of plain gems
|
|
338
787
|
(dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
|
|
339
|
-
human-essentials on Postgres (4,080 mutants, 6 workers
|
|
788
|
+
human-essentials on Postgres (4,080 mutants, 6 workers; the 2026-10-10 run, before
|
|
789
|
+
the baseline check and the coarse-gate removal) and lobsters on SQLite,
|
|
340
790
|
both on macOS. Not yet tried on MySQL or older Rails versions. RSpec only, by
|
|
341
791
|
design. Results: [docs/validation](docs/validation).
|
|
342
792
|
|
|
@@ -344,17 +794,20 @@ design. Results: [docs/validation](docs/validation).
|
|
|
344
794
|
|
|
345
795
|
A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
|
|
346
796
|
1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
|
|
347
|
-
ran once on sampled files, so treat the numbers as indicative.
|
|
797
|
+
ran once on sampled files, so treat the numbers as indicative. The table was
|
|
798
|
+
measured at 2b6ec02, before the literal, filter and argument operators and
|
|
799
|
+
before the coarse-survivor gate was removed, so mutant counts and times will
|
|
800
|
+
change; a rerun is pending. mutineer is ahead
|
|
348
801
|
on number of operators and CI extras, and it supports Minitest.
|
|
349
802
|
|
|
350
803
|
| Metric | saboteur | mutineer |
|
|
351
804
|
|---|---|---|
|
|
352
|
-
| Fake kills (lobsters) | 0 of 20
|
|
805
|
+
| Fake kills (lobsters) | 0 of 20 sampled at random | 8 of 8 hand-checked (identical mutants, not a random sample) |
|
|
353
806
|
| Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
|
|
354
807
|
| Runs on the Postgres app | yes | no |
|
|
355
808
|
| Time per mutant, hashie | 0.008 s | 0.092 s |
|
|
356
809
|
| Peak memory, hashie (summed process RSS) | 595 MB | 471 MB |
|
|
357
|
-
| Choice of operators, CI features | fixed set, fewer |
|
|
810
|
+
| Choice of operators, CI features | fixed set, fewer | 22 operators, baseline, HTML, GitHub Action |
|
|
358
811
|
|
|
359
812
|
Full results, method and limitations:
|
|
360
813
|
[benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
|
|
@@ -363,6 +816,34 @@ per-process RSS, which counts shared pages more than once, and it was measured
|
|
|
363
816
|
before the fork-memory work. Read it as history, not a ranking of physical
|
|
364
817
|
memory.
|
|
365
818
|
|
|
819
|
+
### Independent review, saboteur 0.4.0 vs mutineer 1.6.0
|
|
820
|
+
|
|
821
|
+
A private consultant review ("saboteur 0.4.0 - Deep Review", 12 October 2026,
|
|
822
|
+
not published) ran saboteur 0.4.0 at 38b1834 and mutineer 1.6.0 at a766c96 on
|
|
823
|
+
the same machine and projects: Linux x86_64, 1 vCPU, 4 GB, Ruby 3.4.5, Prism
|
|
824
|
+
1.2.0, RSpec 3.13, no Rails or sqlite3. mutineer ran with its default 5
|
|
825
|
+
operators and with all 20. The project and timing rows are one run each. The
|
|
826
|
+
rows describe saboteur at 38b1834; several of its findings have been fixed on
|
|
827
|
+
main since. The rerun that replaces this table is still pending.
|
|
828
|
+
|
|
829
|
+
| Measure | saboteur 0.4.0 | mutineer 1.6.0 |
|
|
830
|
+
|---|---|---|
|
|
831
|
+
| Snippets project (11 planted gaps) | 177 mutants, 14 s, 10/11 gaps found, 9 noise survivors | Default: 56 mutants, 18 s, about 5/11. All operators: 191 mutants, 51 s, 10/11, 8 noise and duplicates |
|
|
832
|
+
| Dependency project (A tested only via B) | 239 mutants, 6 s | 254 mutants, 26 s (needed hand-ordered source files) |
|
|
833
|
+
| dry-inflector (real gem) | 551 mutants, 24 survivors, 95.6 %, 133 s | 480 mutants, 47 survivors (31 distinct), 90.1 %, 134 s |
|
|
834
|
+
| Test selection | Whole suite, per example, automatic | Only files passed with --test, per file |
|
|
835
|
+
| Kill verification | Yes (re-runs killers on unchanged code) | No |
|
|
836
|
+
| sorbet sig methods | Skipped at 0.4.0 | Mutated, 24/24 killed |
|
|
837
|
+
| Data.define / Struct.new block methods | Absent at 0.4.0 | Mutated |
|
|
838
|
+
| Load-time code | False survivors at 0.4.0 | ran_at_load status |
|
|
839
|
+
| Endless defs | Tested ones reported as no coverage at 0.4.0 | Tested ones work; untested ones show as survived |
|
|
840
|
+
| Order-dependent specs | Smell exposed, bogus kills rejected | Hidden (whole file runs) |
|
|
841
|
+
| Destructive mutant | Project wiped except .saboteur | Project wiped |
|
|
842
|
+
| Baseline stability | Race at 0.4.0: 4 of 30 green runs failed (13 %), 5 of 25 under CPU contention (20 %) | No failures observed (15/15) |
|
|
843
|
+
|
|
844
|
+
Not in the review and still pending: a rerun on current main, memory and
|
|
845
|
+
per-mutant time on the benchmark apps, and any mutineer release after 1.6.0.
|
|
846
|
+
|
|
366
847
|
### Memory after the fork-memory work (hashie @ 9c81e83)
|
|
367
848
|
|
|
368
849
|
Paired A/B runs against the pre-PR commit (32c0f21), 5 pairs per job level with
|
|
@@ -380,12 +861,12 @@ variants, as before.
|
|
|
380
861
|
|
|
381
862
|
Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not
|
|
382
863
|
touch less memory, it keeps more of it shared between workers after fork, which
|
|
383
|
-
is what PSS measures. On
|
|
384
|
-
|
|
864
|
+
is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers on a
|
|
865
|
+
2 GB sandbox. Swap was not measured.
|
|
385
866
|
|
|
386
|
-
Wall time on uncontended runs
|
|
387
|
-
|
|
388
|
-
|
|
867
|
+
Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs
|
|
868
|
+
32s at jobs=6. These are noisy per-cell minima from historical runs and do not
|
|
869
|
+
show a direction either way. The shared sandbox had multi-minute CPU-starvation
|
|
389
870
|
windows during this session (walls up to 594s); affected runs are flagged via
|
|
390
871
|
sampler cadence and excluded from wall analysis. Absolute times are specific to
|
|
391
872
|
this 2-core sandbox. Environment, method and the contention log are on the
|
|
@@ -397,7 +878,7 @@ this 2-core sandbox. Environment, method and the contention log are on the
|
|
|
397
878
|
team workflow (GitHub Action, baseline, HTML report), then 1.0. saboteur stays
|
|
398
879
|
RSpec-only. Design
|
|
399
880
|
details are in [docs/design.md](docs/design.md), prior art in
|
|
400
|
-
[docs/prior-art.md](docs/prior-art.md), and
|
|
881
|
+
[docs/prior-art.md](docs/prior-art.md), and project status in
|
|
401
882
|
[docs/status.md](docs/status.md).
|
|
402
883
|
|
|
403
884
|
## 🤝 Contributing
|
|
@@ -411,3 +892,80 @@ Thanks to [Glomo](https://glomo.one) for providing the AI tokens used to build s
|
|
|
411
892
|
## License
|
|
412
893
|
|
|
413
894
|
MIT. Copyright (c) 2026 Vamsi Pavan Mahesh Gunturu. See [LICENSE](LICENSE).
|
|
895
|
+
|
|
896
|
+
Internal baseline capture blocks a subject with empty generation, including a subject for which the builtin source yields no candidates. Context comparison is an extra consistency check; file fingerprint verification normally also catches source drift. Neither check proves external state stability.
|
|
897
|
+
|
|
898
|
+
### Opt-in survivor baseline
|
|
899
|
+
|
|
900
|
+
`baseline check` runs every selected mutant fresh, then compares accepted survivors. It does not skip accepted mutants, change verdicts or change scores. Use whole configured projects or `--subject` filters; diff, replay, incremental reuse and score thresholds are refused. Caps, uncovered work, empty generation, errors, resource limits, omitted malformed files, changed inputs and missing results are incomplete, even with zero survivors.
|
|
901
|
+
|
|
902
|
+
```sh
|
|
903
|
+
saboteur baseline propose --baseline baseline.json --proposal proposal.json
|
|
904
|
+
# Edit only document.accepted[*].reason in proposal.json. Give each survivor a review reason.
|
|
905
|
+
saboteur baseline accept --baseline baseline.json --proposal proposal.json
|
|
906
|
+
saboteur baseline check --baseline baseline.json
|
|
907
|
+
```
|
|
908
|
+
|
|
909
|
+
Proposal writes independent owner-private execution evidence under the configured cache directory. Keep that directory until acceptance; deleting it prevents acceptance. Acceptance verifies the private record and current input fingerprint before atomically publishing the manifest. For replacement, give both propose and accept `--previous SHA256`, the hash you reviewed of the current baseline bytes. Output files must not overlap project inputs or the private evidence directory. A baseline normally lives in a separate JSON file outside fixture/config input trees.
|
|
910
|
+
|
|
911
|
+
Check prints one JSON object: accepted, new, changed, resolved_kills, detected_timeouts, unmatched and errors are separate categories. Exit 0 means a compatible complete clean check; 1 needs survivor review; 2 means incompatible or invalid configuration; 3 means incomplete execution or tool failure. Propose and accept exit 0 only after their file operation succeeds. Context changes require fresh review. Timeouts remain a distinct detected category, not assertion kills.
|
|
912
|
+
|
|
913
|
+
Score-drop gate: `saboteur baseline check --baseline baseline.json --epsilon POINTS` also fails (exit 1,
|
|
914
|
+
state `review_required`) when the fresh score is more than POINTS percentage points below the score recorded in
|
|
915
|
+
the baseline. Both scores use saboteur's own formula, (killed + timeout) / (killed + timeout + survived), over
|
|
916
|
+
every candidate. The baseline already stores each candidate's status, so no new file format is needed. This
|
|
917
|
+
catches what the survivor comparison cannot: a score that falls because killed mutants disappeared (deleted or
|
|
918
|
+
rewritten code) while every accepted survivor is still matched. A drop of exactly POINTS passes; a rise never
|
|
919
|
+
fails. The check prints a `score` object (`baseline`, `current`, `drop`, `epsilon`, `exceeded`) only when
|
|
920
|
+
`--epsilon` is given. Without it, behavior is unchanged. POINTS is 0 to 100. It lives on `baseline check`,
|
|
921
|
+
not on `run`: the delta gate needs the baseline's fresh whole-scope evidence, which `run` does not produce.
|
|
922
|
+
It is not the same as `--threshold`, which gates an absolute score.
|
|
923
|
+
|
|
924
|
+
In GitHub Actions, the action's `baseline` and `baseline-epsilon` inputs run this check; see
|
|
925
|
+
[docs/github-action.md](docs/github-action.md).
|
|
926
|
+
|
|
927
|
+
Local evidence protects against edits to the public proposal except review reasons. It is not remote attestation, a signature, or protection against the account owner changing their private files. File fingerprints cannot prove external database, network or environment stability. `config/saboteur_ignore.rb` remains a separate legacy ignore mechanism; this gate compares every survivor regardless of ignore status.
|
|
928
|
+
|
|
929
|
+
A baseline that changes during `check` is rejected as configuration/evidence error (exit 2), not an incomplete execution result. Acceptance leaves a private `baseline.json.lock` beside the destination: keep it for cooperating-process publication locks. It contains no evidence or results. Symlinked checkout parents are normalized; symlink artifacts themselves and private-store/lock symlinks are still refused.
|
|
930
|
+
|
|
931
|
+
The builtin `endpoint_replacement` operator swaps explicit, no-argument `first`/`last` calls. It targets wrong-end/order-selection faults in collection handling. Argument-bearing slices, bare calls and block-bearing APIs are left unchanged. Receivers are not statically typed: a custom object may implement either name differently, so a crash remains a crash kill, not proof of a collection assertion.
|
|
932
|
+
|
|
933
|
+
JSON findings are ordered by a deterministic [review ranking](docs/survivor-ranking.md): coarse first, then covering-test count, edited-line impact and changed-line proximity, with file/line/digest ties. The terminal groups survivors by file, method and line first; the ranking only orders the survivors on one line, and every survivor is listed, in review mode too. Only consecutive survivors with matching evidence share a test/evidence block; repeated evidence after another group is printed again to keep that order. Ranking does not change scores or execution.
|
|
934
|
+
|
|
935
|
+
### Preview and choose operators
|
|
936
|
+
|
|
937
|
+
`bundle exec saboteur run --dry-run` discovers source methods and generates candidates without booting the app, running tests, forking, or writing reports/caches. It honors `--since`, `--subject` and source arguments. It prints per-method counts and cap omissions. After a matching normal run, a cached estimate shows baseline covering-test work (cached original-test durations times planned mutants; it is not a bound on mutant runtime and excludes boot and fork costs). If any file cannot be parsed, the total is marked INCOMPLETE and no estimate is given.
|
|
938
|
+
|
|
939
|
+
Trust boundary: `--dry-run` never boots the app or runs its tests, but custom `sources` entries (`path.rb:ClassName`) are still `require`d to generate candidates, so their code runs. Only preview with sources you trust.
|
|
940
|
+
|
|
941
|
+
`bundle exec saboteur run --list-operators` lists every builtin name and description. `--operators arithmetic,comparison` or `c.operators = ["arithmetic", "comparison"]` selects names. Tiers are `core`, `extended`, and `all`; core contains the current default operators. Extended adds opt-in [`range`](docs/operators/range.md), [`operator_assignment`](docs/operators/operator_assignment.md), and [`condition_negation`](docs/operators/condition_negation.md). Empty/default selection keeps only core operators. Selection changes invalidate incremental verdict identity.
|
|
942
|
+
|
|
943
|
+
Inspired by mutineer's --dry-run, --list-operators and --operators tiers (github.com/davidteren/mutineer, MIT, by David Teren).
|
|
944
|
+
|
|
945
|
+
### Agent survivor hints and score history
|
|
946
|
+
|
|
947
|
+
`--agent` findings include operator-specific `hint` suggestions and a shared
|
|
948
|
+
`hintCaveat`. Hints do not diagnose a missing test or assertion; equivalent
|
|
949
|
+
mutants and limited observation are still possible. Unknown operators have no
|
|
950
|
+
hint unless marked coarse. The normal JSON byte budget still applies.
|
|
951
|
+
|
|
952
|
+
Use `--score-history PATH` (or `c.score_history = PATH` in your config) to append
|
|
953
|
+
one score-history JSONL record per finished report. History is off by default;
|
|
954
|
+
changes are printed to stderr, not agent stdout. See [score definitions and
|
|
955
|
+
history limits](docs/scores.md). The file can be uploaded by your own artifact
|
|
956
|
+
step; the bundled Action's upload behavior is unchanged.
|
|
957
|
+
|
|
958
|
+
Score history now refuses non-regular files, symlink paths and report/history
|
|
959
|
+
inode aliases before running the engine or writing reports. Imported records
|
|
960
|
+
have a 64 KiB per-line bound and strict version-one validation, including
|
|
961
|
+
counts, scores and duplicate keys. History is an output, excluded from baseline
|
|
962
|
+
fingerprints. See [history safety and comparison limits](docs/scores.md).
|
|
963
|
+
### Inline suppression comments
|
|
964
|
+
|
|
965
|
+
`total = a + b # saboteur:disable-line arithmetic reason="rounding checked elsewhere"` suppresses edits starting on that line, restricted to the named operators. Omit names to suppress all operators. Multiple names are space-separated. Only actual Ruby comments count; directive text in strings does not. A missing or blank reason or unknown operator is a usage error naming file:line. Terminal, JSON and HTML list every suppressed candidate with its reason and digest, outside the score. Moving the commented code keeps the suppression. Multi-line edits use their starting line.
|
|
966
|
+
|
|
967
|
+
Inspired by mutineer's inline suppression comments (github.com/davidteren/mutineer, MIT, by David Teren).
|
|
968
|
+
|
|
969
|
+
Inline suppressions bound completeness: classic scores still exclude suppressed candidates, but baseline evidence is incomplete when any inline candidate is suppressed. Agent output reports `inline_suppressed` with its count and points to the full audit. Reasons are audited, not proof that suppressed work was checked.
|
|
970
|
+
|
|
971
|
+
Source-only `--dry-run` lists the suppressed count and each audit row in selected methods, before diff restriction and the per-method cap. They are outside the planned total. Preview still does not boot the app or write reports.
|