saboteur 0.7.0 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (114) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +772 -0
  3. data/README.md +221 -1072
  4. data/lib/saboteur/attribute_finder.rb +113 -0
  5. data/lib/saboteur/cli/exit_status.rb +32 -1
  6. data/lib/saboteur/cli/help_options.rb +2 -1
  7. data/lib/saboteur/cli/history_command.rb +135 -0
  8. data/lib/saboteur/cli/presets.rb +47 -3
  9. data/lib/saboteur/cli/run_budget_options.rb +3 -1
  10. data/lib/saboteur/cli/selection_options.rb +31 -0
  11. data/lib/saboteur/cli/shard_options.rb +17 -0
  12. data/lib/saboteur/cli/source_set.rb +35 -0
  13. data/lib/saboteur/cli/survivors_command.rb +157 -0
  14. data/lib/saboteur/cli.rb +74 -55
  15. data/lib/saboteur/config.rb +102 -6
  16. data/lib/saboteur/config_file.rb +19 -7
  17. data/lib/saboteur/constant_reads.rb +182 -0
  18. data/lib/saboteur/constant_subject.rb +70 -0
  19. data/lib/saboteur/constant_subject_finder.rb +281 -0
  20. data/lib/saboteur/coverage_reads.rb +29 -0
  21. data/lib/saboteur/custom_map.rb +12 -1
  22. data/lib/saboteur/dirty_tree_warning.rb +130 -0
  23. data/lib/saboteur/doctor.rb +13 -0
  24. data/lib/saboteur/engine/baseline_guard.rb +135 -21
  25. data/lib/saboteur/engine/boot.rb +11 -3
  26. data/lib/saboteur/engine/bytecode_equivalence.rb +17 -1
  27. data/lib/saboteur/engine/candidate_selection.rb +20 -0
  28. data/lib/saboteur/engine/candidates.rb +43 -3
  29. data/lib/saboteur/engine/coverage_store.rb +58 -14
  30. data/lib/saboteur/engine/defined_order_prefix.rb +43 -0
  31. data/lib/saboteur/engine/diagnostics.rb +19 -1
  32. data/lib/saboteur/engine/distribution.rb +10 -0
  33. data/lib/saboteur/engine/early_stop_recording.rb +3 -2
  34. data/lib/saboteur/engine/gated_run.rb +37 -0
  35. data/lib/saboteur/engine/load_coverage.rb +63 -0
  36. data/lib/saboteur/engine/out_of_scope_guard.rb +21 -0
  37. data/lib/saboteur/engine/pipelines.rb +115 -0
  38. data/lib/saboteur/engine/result_cache.rb +65 -8
  39. data/lib/saboteur/engine/run_budget.rb +32 -2
  40. data/lib/saboteur/engine/sandbox_policy.rb +59 -3
  41. data/lib/saboteur/engine/scheduler.rb +131 -17
  42. data/lib/saboteur/engine/selection_announcement.rb +59 -0
  43. data/lib/saboteur/engine/shard.rb +69 -0
  44. data/lib/saboteur/engine/subject_selection.rb +380 -21
  45. data/lib/saboteur/engine/test_guess.rb +407 -27
  46. data/lib/saboteur/engine/unchanged_syntax.rb +17 -2
  47. data/lib/saboteur/engine.rb +103 -129
  48. data/lib/saboteur/execution/external_runner.rb +489 -0
  49. data/lib/saboteur/execution/fiddle_loader.rb +34 -0
  50. data/lib/saboteur/execution/marker_sweep.rb +80 -0
  51. data/lib/saboteur/execution/peak_memory.rb +28 -0
  52. data/lib/saboteur/execution/process_table.rb +19 -3
  53. data/lib/saboteur/execution/process_tree.rb +65 -5
  54. data/lib/saboteur/execution/reevaluator.rb +75 -12
  55. data/lib/saboteur/execution/runner.rb +46 -7
  56. data/lib/saboteur/execution/sandbox.rb +25 -6
  57. data/lib/saboteur/execution/seatbelt.rb +233 -0
  58. data/lib/saboteur/execution/supervisor.rb +29 -11
  59. data/lib/saboteur/execution/test_command_shape.rb +149 -0
  60. data/lib/saboteur/execution/worker_pool.rb +23 -1
  61. data/lib/saboteur/frameworks/sqlite_memory_database.rb +79 -0
  62. data/lib/saboteur/gates.rb +43 -2
  63. data/lib/saboteur/help.rb +60 -9
  64. data/lib/saboteur/ignore_list.rb +29 -6
  65. data/lib/saboteur/integration/rspec.rb +223 -5
  66. data/lib/saboteur/mutation_source/argument_removal.rb +1 -1
  67. data/lib/saboteur/mutation_source/argument_swap.rb +54 -0
  68. data/lib/saboteur/mutation_source/builtin.rb +28 -5
  69. data/lib/saboteur/mutation_source/chain_removal.rb +1 -1
  70. data/lib/saboteur/mutation_source/condition_negation.rb +1 -1
  71. data/lib/saboteur/mutation_source/constant_literals.rb +66 -0
  72. data/lib/saboteur/mutation_source/element_removal.rb +73 -0
  73. data/lib/saboteur/mutation_source/endpoint_replacement.rb +1 -1
  74. data/lib/saboteur/mutation_source/entry_edits.rb +44 -0
  75. data/lib/saboteur/mutation_source/filter_replacement.rb +1 -1
  76. data/lib/saboteur/mutation_source/hazard_guard.rb +189 -0
  77. data/lib/saboteur/mutation_source/literal_replacement.rb +1 -1
  78. data/lib/saboteur/mutation_source/method_swap.rb +148 -0
  79. data/lib/saboteur/mutation_source/operator_assignment.rb +1 -1
  80. data/lib/saboteur/mutation_source/operator_examples.rb +4 -1
  81. data/lib/saboteur/mutation_source/operator_registry.rb +5 -2
  82. data/lib/saboteur/mutation_source/operator_visit.rb +1 -1
  83. data/lib/saboteur/mutation_source/pair_removal.rb +13 -31
  84. data/lib/saboteur/mutation_source/range_replacement.rb +1 -1
  85. data/lib/saboteur/mutation_source/regex_replacement.rb +93 -0
  86. data/lib/saboteur/preview.rb +125 -12
  87. data/lib/saboteur/refinement_activations.rb +60 -0
  88. data/lib/saboteur/reporter/agent.rb +9 -1
  89. data/lib/saboteur/reporter/agent_commands.rb +10 -3
  90. data/lib/saboteur/reporter/agent_incomplete.rb +31 -5
  91. data/lib/saboteur/reporter/agent_summary.rb +4 -3
  92. data/lib/saboteur/reporter/findings.rb +5 -2
  93. data/lib/saboteur/reporter/history_policy.rb +11 -0
  94. data/lib/saboteur/reporter/history_reader.rb +47 -0
  95. data/lib/saboteur/reporter/html.rb +26 -4
  96. data/lib/saboteur/reporter/mutation_text.rb +6 -0
  97. data/lib/saboteur/reporter/stryker.rb +27 -3
  98. data/lib/saboteur/reporter/survivor_classification.rb +101 -0
  99. data/lib/saboteur/reporter/survivor_list.rb +15 -9
  100. data/lib/saboteur/reporter/terminal.rb +56 -8
  101. data/lib/saboteur/result.rb +19 -3
  102. data/lib/saboteur/selection/coverage_map.rb +120 -10
  103. data/lib/saboteur/selection/dependency_shield.rb +80 -0
  104. data/lib/saboteur/selection/external_test_map.rb +27 -0
  105. data/lib/saboteur/selection/git_diff.rb +93 -11
  106. data/lib/saboteur/selection/layout_only_lines.rb +80 -0
  107. data/lib/saboteur/selection/recorder.rb +22 -4
  108. data/lib/saboteur/selection/source_scope.rb +22 -0
  109. data/lib/saboteur/subject.rb +42 -7
  110. data/lib/saboteur/subject_finder.rb +173 -29
  111. data/lib/saboteur/version.rb +1 -1
  112. data/lib/saboteur/working_tree.rb +38 -15
  113. data/lib/saboteur.rb +14 -0
  114. metadata +39 -16
data/README.md CHANGED
@@ -10,1122 +10,271 @@
10
10
  [![Gem Version](https://img.shields.io/gem/v/saboteur)](https://rubygems.org/gems/saboteur)
11
11
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
12
12
 
13
- Your suite is green and coverage says 95%. That tells you the tests *ran* your
14
- code. It does not tell you they would *notice* if the code were wrong.
13
+ Line coverage tells you which code ran. It does not tell you if a test
14
+ would notice when that code breaks. Mutation testing does.
15
15
 
16
- saboteur finds out. It makes small deliberate changes to your methods (flips a
17
- `<` to `<=`, swaps `+` for `-`, replaces a body with `nil`), runs the tests that
18
- cover each change, and reports every change your suite let through. Each of
19
- those survivors can point at a missing test. It can also be an equivalent
20
- change that no test could notice, a flaky test, or a limit of the tool, so
21
- each one needs a human look.
16
+ saboteur makes one small change to your code at a time and runs only the
17
+ tests that touch it, so it is fast. If the tests still pass, you see the
18
+ exact change they missed. Every kill is re-checked against the unchanged
19
+ code, so a broken suite never counts as a kill.
22
20
 
23
- ```
24
- $ bundle exec saboteur run --since origin/main
25
- ```
26
-
27
- Never heard of mutation testing? It asks "if I broke this, would a test
28
- fail?" for many small breakages and lists the ones nobody noticed. Read the
29
- [design notes](docs/design.md) when you want the details. For adopting it on a team, see the
30
- [best practices guide](docs/best-practices.md).
31
-
32
- Declared external inputs are checked against cache and report paths using their
33
- physical paths, including aliased parent directories. An overlap makes the run
34
- fingerprint incomplete, so it cannot authorize cache reuse.
35
-
36
- Internal run reports retain typed candidate-drop and cap evidence for gate
37
- validation. These records do not change selection, scores, exit codes or the
38
- public JSON format. Empty generation is unverified, not proof of equivalence.
39
-
40
- The internal engine can opt in to fresh baseline evidence capture. It records
41
- inputs before boot, planned candidate identities before scheduling and reconciled
42
- terminal results after execution. Caps, uncovered work, invalid or empty
43
- production, reused outcomes and changed inputs make that evidence incomplete.
44
- The public opt-in `baseline propose`, `baseline accept` and `baseline check`
45
- commands use this evidence to review and compare survivors. They never skip
46
- accepted mutants or change their scores. See [Opt-in survivor baseline](#opt-in-survivor-baseline).
47
-
48
- ### The unchanged suite is checked first
49
-
50
- Every run with selected subjects runs the loaded unchanged suite twice (the specs chosen with `--test` if given,
51
- otherwise the inferred specs) in fresh
52
- supervised children before coverage lookup or incremental verdict reuse. This
53
- costs two suite runs even on a cache hit, plus a coverage-building suite run on
54
- a cache miss. Before the first pass, a run over the whole suite announces the example
55
- count and the real number of full passes (a cached map only counts as
56
- reused when it loads clean); once the first baseline pass is timed, the
57
- run estimates the time left for the remaining baseline and coverage
58
- passes from it - mutation time is not part of that estimate. Past 1,000
59
- examples the announcement also names `--test` as the way to scope the run. Both baseline passes must be green, complete and in the same
60
- RSpec order at the run seed. Set `c.baseline_passes = 1` (or `--baseline-passes 1`) for a faster PR loop: the run says so up front and gives up the order-stability comparison between passes (#55). A failure exits 3, stops mutation, prints the failing example ids and the first ten lines of the
61
- failure inline, and names the log and fix. Long phases
62
- print their elapsed time about every 30 seconds, with the examples finished so far, the percentage and a rough estimate of the time left (it assumes equally long examples) once the suite child has logged its first count, and each phase's log holds
63
- the suite's progress and full failure detail. Each suite pass
64
- uses `coverage_timeout` and configured memory/process limits (the suite memory
65
- budget is four times `max_memory_mb`, as for coverage). A suite child that has
66
- already reported its result gets up to 5 seconds to finish exiting, under the
67
- same deadline and memory/process limits as before; if it is still running then,
68
- saboteur stops it and keeps the result, so a slow exit is not reported as a
69
- failure. A child that ends by itself in that window keeps its own exit status. A child killed by `SIGKILL` is called a possible
70
- out-of-memory kill only when the process tree was seen growing toward the
71
- memory limit. Older benchmark numbers
72
- need a rerun before they describe this extra fixed cost.
73
- A whole-suite coverage build that records no lines for any included file
74
- means the setup hid the app from Coverage (Bootsnap's compile cache is the
75
- common cause), so the run fails with exit 3 instead of testing nothing, and
76
- the empty map is not cached. With `--test` an empty map instead stays a
77
- legitimate "no coverage" outcome, because the chosen specs may not touch the
78
- selected sources.
79
- Two passes detect observed instability, not every possible flaky test. See
80
- [baseline stability](docs/baseline-stability.md) for limits and hook behaviour.
81
-
82
- ### Hazardous calls are not mutated, and a canary watches the tree
83
-
84
- Mutants run in your checkout, so a mutant inside a destructive call could do
85
- real damage: a string mutation inside `FileUtils.rm_rf(File.join(Dir.pwd,
86
- "build"))` once turned the target into the working directory itself. The
87
- hazard guard is a syntactic filter. It recognizes destructive calls written
88
- as direct calls: `FileUtils.rm*`/`remove_entry*` (also as bare names under
89
- `include FileUtils`), `File.delete`/`unlink`, `Dir.rmdir`/`delete`/`unlink`,
90
- `rmtree`, `system`/`exec`/`spawn` (bare or on `Kernel`/`Process`), backticks
91
- and `%x`, `IO.popen`, any `Open3` method, `open` with a leading `|`, and
92
- `send`/`public_send`/`__send__` with a literal symbol or string selector
93
- naming one of these (dynamic selectors stay out of scope). It
94
- suppresses any mutant that would leave such a call in place while altering
95
- code inside it or anywhere else in the same method. A mutant that removes
96
- every hazardous call in the method still runs, provided its replacement
97
- introduces no new hazard. The match is by name on the written call: calls
98
- reached through aliases, variables or other indirection are not
99
- recognized, so the guard cannot promise to find every destructive call. The
100
- summary reports how many mutants were suppressed. Extend or replace the list
101
- with `c.hazard` (same tag shapes: `"FileUtils.rm_rf"`, or a bare name like
102
- `"system"`); `--allow-hazardous` runs the suppressed mutants anyway.
103
-
104
- Behind the guard, a canary snapshots the tracked files (or every file,
105
- outside a repository) before mutating and probes the snapshot after every
106
- mutant - a cheap stat pass, no tree walk. If anything vanished, the run
107
- aborts with exit 3 and names the digest of the mutant that just ran, not
108
- the method's last mutant. A fuller check still runs after each method. That check happens after the damage and restores
109
- nothing; it limits how far a bad mutant gets. Recover from your own backup,
110
- a fresh checkout or a reclone - a deleted `.git` or uncommitted work is gone
111
- unless you have another copy. The canary is the backstop for the destructive
112
- forms the syntactic guard cannot see.
113
-
114
- On Linux 6.2 and later (Landlock ABI 3) each mutant child also runs under
115
- a Landlock sandbox: the child cannot create, write, rename, unlink or
116
- truncate files in the project tree, except `tmp/` and the
117
- system temp dir, plus anything in `c.sandbox_writable`. The project
118
- `tmp/` grant applies only when `tmp/` already exists - the run never
119
- creates it for the grant - so create it before the run if your suite
120
- writes there, or grant the directory with `c.sandbox_writable`. The cache dir
121
- is denied too - a mutant cannot delete the baseline, the reports or
122
- other children's logs; its own log is opened before the fork and
123
- arrives as an inherited descriptor. Grants are additive, though: a
124
- cache dir placed inside `tmp/`, the system temp dir or a
125
- `c.sandbox_writable` entry stays writable, and the tool cannot deny
126
- what another grant allows - keep the cache outside every grant for
127
- the isolation to bite. Every other path
128
- is denied for writing, `/dev/shm` and `/dev/tty` included; `/dev/null`
129
- stays writable. Inherited
130
- descriptors already open for writing outside the granted dirs are closed
131
- before the sandbox engages - stdio redirected to a regular file and files
132
- already deleted from the tree included (stdio on a terminal, pipe or
133
- `/dev/null` stays), and a descriptor whose target or flags cannot be
134
- inspected is closed too rather than trusted. A `c.sandbox_writable` grant
135
- that contains the project root or another granted dir warns, since it
136
- defeats or duplicates the protection; reads and process execution are
137
- unaffected.
138
- This DOES change what your tests can write: a suite that writes elsewhere
139
- in the project (`db/`, `log/`, ...) fails with EACCES in mutant children -
140
- grant that directory with `c.sandbox_writable` or opt out with
141
- `c.sandbox = false`. Landlock covers the filesystem only - network,
142
- sockets and IPC are never restricted - and a checkout inside the system
143
- temp dir cannot be protected, because everything under it stays writable
144
- so that real test suites keep working - the run warns when the effective
145
- write set (every grant realpath'd, the system temp dir
146
- included) contains the project root, however it happened: a checkout
147
- inside the system temp dir, a symlinked `tmp`, or an
148
- overlapping `c.sandbox_writable` grant. `c.sandbox = "strict"` refuses to
149
- run at all in that situation, since the sandbox would protect nothing.
150
- The sandbox applies to every mutant
151
- child and fails closed: a child that cannot restrict itself reports an
152
- error rather than run, and a kernel probe that errors (as opposed to
153
- cleanly reporting Landlock absent) refuses the run.
21
+ **Docs, benchmarks and guides live on the site:
22
+ [vamsipavanmahesh.github.io/saboteur](https://vamsipavanmahesh.github.io/saboteur/)**
154
23
 
155
- On kernels from 5.13 to 6.1 (Landlock ABI 1-2) `truncate()` cannot be
156
- restricted, so the tree is not fully read-only and the claim above does
157
- not hold; the run notes once that children run with the guard and canary
158
- only, exactly as on kernels without Landlock - and as on non-glibc Linux
159
- whose C library cannot be located (musl is tried by soname first).
160
- `c.sandbox = "strict"` turns that degraded note into a refusal;
161
- `c.sandbox = false` opts out.
24
+ <a id="30-second-quick-start"></a>
25
+ ## 30-second quick start
162
26
 
27
+ You need Ruby 3.3 or newer, RSpec, and Linux or macOS with `fork`.
163
28
 
164
- ## 🚀 Try it in a minute
29
+ Prerequisite: the sandbox layers use fiddle, which ships with Ruby
30
+ (a default gem on Ruby 3.x). On Ruby 4.0+, where fiddle is a bundled
31
+ gem, add `gem "fiddle"` to your Gemfile. If saboteur refuses a run
32
+ because fiddle cannot load: reinstall Ruby, run `gem install fiddle`,
33
+ or install libffi headers (libffi-dev on Debian/Ubuntu, libffi-devel
34
+ on Fedora/RHEL) and reinstall fiddle. libffi headers are required only
35
+ when Ruby's shipped fiddle is missing or must be installed standalone.
165
36
 
166
- Add saboteur to the test group of your project's Gemfile (Ruby 3.3+, RSpec):
37
+ ```sh
38
+ # Add it to your project's Gemfile:
39
+ bundle add saboteur --group test
167
40
 
168
- ```ruby
169
- gem "saboteur", group: :test
41
+ # Then:
42
+ bundle exec saboteur run
170
43
  ```
171
44
 
172
- ```sh
173
- bundle install
174
- bundle exec saboteur run --subject 'YourClass#your_method'
45
+ Outside a Bundler project, `gem install saboteur` gives the same
46
+ executable - run it there as plain `saboteur run`, without
47
+ `bundle exec`.
48
+
49
+ The first run builds a coverage map of which tests touch which code. The
50
+ map is cached and reused while your code and tests stay unchanged, so
51
+ later runs start fast.
52
+
53
+ If the first run stops with exit 2 and `no spec found for ...`, saboteur
54
+ could not map a source file to its spec: it matches by path first, then
55
+ by parent directory and basename when the layout strips directories,
56
+ and it refuses to guess wrong. Point it at the specs yourself - set
57
+ `test_selection = :all` in `config/saboteur.rb` to run the whole suite
58
+ against each mutant, pass `--test spec/...` to name the specs, or
59
+ declare a custom map
60
+ ([monorepos and the custom map](docs/guides/monorepo-and-custom-map.md)).
61
+
62
+ On a branch, `bundle exec saboteur pr` mutates only the methods you
63
+ changed relative to `main`, with one baseline pass and a ten-minute
64
+ budget. If the budget stops the run, it exits 4 and prints `partial: X
65
+ of Y mutants not run` (or says the count is unknown when planning did
66
+ not finish). Pass `--max-duration-ok` to accept a partial verdict; the
67
+ warning stays, and other failure gates still apply. Pass `--since` for a
68
+ different base, for example `bundle exec saboteur pr --since master`.
69
+
70
+ Setup, configuration and CI:
71
+ [the quickstart](https://vamsipavanmahesh.github.io/saboteur/quickstart.html).
72
+
73
+ ## Why RSpec-only
74
+
75
+ A mutation verdict is only worth anything if the test selection behind it
76
+ is exact. RSpec's example structure lets saboteur map each method to the
77
+ examples that cover it, rerun just those against each change, and re-check
78
+ every kill against the code before the change. One framework, done
79
+ properly - that is the bet. Minitest is not supported today;
80
+ [why, and what it would take](docs/minitest.md).
81
+
82
+ ## What a run tells you
83
+
84
+ ```text
85
+ saboteur 0.7.0: 1 subjects, 10 mutants, wall 0.70 s, seed 1
86
+ killed 8 survived 2 timeout 0 resource limit 0 error 0 score 80.0 %
175
87
  ```
176
88
 
177
- Outside a Bundler project, `gem install saboteur` installs the same executable.
178
- Rails app? Follow the [Rails quickstart](#rails-apps) below: the defaults are for
179
- plain gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
180
-
181
- Start with one method. You get counts per status, a score, timing, and a
182
- unified diff for every survivor, so you can read the exact change your tests
183
- missed. The JSON report lands in `.saboteur/report.json`.
184
-
185
- 📋 Requirements: Ruby 3.3 or newer on Linux or macOS (it needs `fork`), and RSpec.
186
- Minitest is not supported; see [Minitest](docs/minitest.md) for why and what it would take.
187
-
188
- ## 🧠 Why it is built the way it is
189
-
190
- **It only runs tests that can matter.** Each mutant runs only the tests whose
191
- line coverage touched the changed statement. Tests that killed earlier mutants
192
- of the same method go first, then the fastest, and the first failure stops the
193
- run. A change no test reaches is reported as "no coverage", not as a survivor. Branch coverage separates arms on the same line: an edit inside an arm no test took reports "no coverage (branch not taken)". Ruby does not expose `&&`/`||` right operands as branches, so those keep line-based selection. `case/in` pattern edits use the enclosing case and clause execution lines; edits in an untested clause body still stay uncovered. Whole-body replacements use every test of the method, including methods that start with a heredoc. Endless defs (`def x = ...`) are matched to tests through per-test method call coverage, since line coverage never counts their bodies. Methods defined inside blocks (`Class.new do`, `Data.define do`, `included do`) are not mutated yet; the run lists them as skipped, "defined inside a block".
194
-
195
- **It runs coarse mutants first, then fine mutants.** Each method first gets
196
- whole-body mutants: `nil` (`true`/`false` for predicates) unless the body is empty or
197
- only logs, and `raise` unless a catch-all rescue would absorb it. Fine-grained mutants run
198
- regardless of the coarse verdict, including a survivor or a reused verdict.
199
- Deleting a body unnoticed does not prove its smaller changes are harmless.
200
- Subjects with no tests are skipped before any mutants are generated, and each
201
- method is capped at `max_mutants_per_method` mutants (100 by default). Running
202
- fine mutants after coarse survivors can increase run time and change scores.
203
-
204
- **It prepares memory before it forks.** Before workers fork, saboteur warms and
205
- compacts the Ruby heap ([`ForkReadiness`](lib/saboteur/execution/fork_readiness.rb)),
206
- which is meant to let workers share more pages with their parent and copy less.
207
- Observed on the pinned hashie workload (paired runs, below): about 17-18% lower peak PSS at 2 and 6 workers. Not a general guarantee.
208
-
209
- **It is aimed at pull requests.** `--since origin/main` mutates only the methods
210
- your branch changed: fine-grained mutants on the changed lines, plus the
211
- whole-body mutants of those methods. On one real change in saboteur's own history, that took a file
212
- from 139 mutants to 7, and median wall time fell 22.9% (measured at d2b7abb, before the coarse-survivor gate was removed; a rerun is pending). The time saving is
213
- smaller than the mutant count because building the coverage map is a fixed
214
- cost, and this is one file, not a Rails app, so it is not a general speed
215
- claim. The full method, raw timings and
216
- caveats are in [bench/since](bench/since/README.md).
217
-
218
- **It checks its own kills.** After a method's mutants run, saboteur re-runs the
219
- killing tests against the *unchanged* method. If they fail there too, the kill
220
- is not credited and becomes an error. In one comparison on lobsters, 0 of 20
221
- randomly sampled kills were fake. Every mutant runs in a supervised child with deadline,
222
- memory and process limits, and saboteur tracks and kills the processes it
223
- starts.
89
+ Each survivor comes with its diff, the tests that ran, and a triage label
90
+ (`likely_equivalent`, `arid`, `missing_test`) to sort what to read first.
91
+ The labels are heuristics for reading order, never a diagnosis, and they
92
+ never change the score.
93
+
94
+ A run on a tree with uncommitted changes opens with a warning that splits
95
+ the count into changes inside the mutation scope, outside it, and
96
+ saboteur's own setup files. Mutants run your code, so run on a clean,
97
+ committed tree or a CI checkout.
98
+
99
+ Code that only runs while the app loads cannot be judged by replacing the
100
+ method later, so it reports as `ran at load`, outside the score. The same
101
+ holds for def-time macros - a `deprecate`-style helper, a `before_action`
102
+ block, any DSL whose only caller is a class body: the closure captured at
103
+ load keeps the pristine definition, so the boot's def-call counters mark
104
+ the macro itself `ran at load` instead of letting its mutants survive.
105
+
106
+ ## What you get
107
+
108
+ - **Fast pull-request runs.** `--since main` scopes the run to the methods
109
+ you changed, refuses (exit 2) when the resolved range has no changes -
110
+ `saboteur pr $GITHUB_SHA` on a checkout at the merge commit diffs the
111
+ merge against itself - and fails loudly (exit 3) when the changed code
112
+ cannot be verified - for example when every changed method is skipped
113
+ because a refinement is active where it sits. A changed method skipped
114
+ for no coverage names the specs the run loaded; when its coverage
115
+ lives in a sibling spec file, pass `--test spec/models` (the directory
116
+ holding it) to load it. The GitHub Action runs on every pull request
117
+ and caches the coverage map between runs. `--shard K/N` splits one
118
+ planned inventory deterministically across CI jobs; each shard reports
119
+ partial evidence.
120
+ - **Honest verdicts.** Every kill is verified against the unchanged
121
+ method; a kill the suite cannot confirm stays out of the score.
122
+ Suspect a flaky test? `--flaky-check survivors` confirms each new
123
+ survivor with one reordered rerun (`decisive` also confirms fresh
124
+ kills); a verdict the rerun disagrees with becomes `unverified`,
125
+ named with its tests, and stays out of the score. A spec
126
+ file whose examples depend on earlier ones is retried in defined order
127
+ before a kill is rejected, so its mutants are still scored.
128
+ - **Reports for people and tools.** Terminal, HTML, a Stryker-compatible
129
+ JSON report, and a stable agent JSON API with a deprecation policy.
130
+ - **A plan before the run.** `saboteur run --dry-run --format json` prints
131
+ the versioned [mutant inventory](docs/dry-run-json.md) - every planned
132
+ mutant with its stable digest and edits - without booting the suite.
133
+ - **A review workflow.** An ignore list with reasons and triage states,
134
+ an opt-in survivor baseline, score thresholds and score history.
135
+ `saboteur survivors .saboteur/report.json` browses survivors in the
136
+ terminal straight from the JSON report - list, filter
137
+ (`--file`/`--operator`/`--triage`/`--ranked`), and `--show DIGEST`
138
+ for the diff, the exact `run --id` replay command and an ignore-file
139
+ entry template. It never boots your project or edits the ignore file.
140
+ - **A local README badge.** `saboteur badge` writes Shields endpoint JSON
141
+ from the score history and `saboteur trend` prints the score delta -
142
+ no hosted service:
143
+
144
+ ```md
145
+ ![mutation score](https://img.shields.io/endpoint?url=<raw-url-of-the-badge-json>)
146
+ ```
147
+
148
+ Incomplete runs are refused as badge evidence unless `--mark-partial`
149
+ labels them, and the trend never compares runs with different
150
+ selection, inventory or operator policy.
151
+ - **Safety controls.** A hazard guard, canary values, and time, memory
152
+ and process-count limits on every mutant in the default in-process
153
+ mode; a Landlock sandbox on Linux
154
+ 6.2 and newer.
155
+ - **Extension points.** Add your own mutation operators through the
156
+ public interface.
157
+
158
+ Per-path score gates hold one subtree to a stricter floor: `c.score_gate
159
+ "app/payments/**/*", min_score: 95` in config/saboteur.rb (repeatable). Each
160
+ gate scores only the mutants under its pattern and fails the run under its
161
+ floor; a gate with nothing executed under it passes vacuously and says so.
162
+ See
163
+ [choosing and raising a threshold](docs/best-practices.md#choosing-and-raising-a-threshold)
164
+ for how to pick a number.
165
+
166
+ ## Honest limits
167
+
168
+ - saboteur is RSpec-only and needs `fork`: no Windows, no Minitest.
169
+ - A survivor is review evidence, not proof of a bug in your tests. It can
170
+ be a missing test, an equivalent change, a flaky test, or a limit of
171
+ the tool. Read it before you write a test for it.
172
+ - `--since` and `--incremental` are iteration aids. Run the whole project
173
+ on main and before a release.
174
+ - The hazard guard is syntactic, sandboxing varies by platform (Landlock
175
+ on Linux 6.2+; on macOS mutants run unsandboxed unless `c.sandbox = "seatbelt"`
176
+ opts into a probe-gated Seatbelt profile; see
177
+ [the macOS Seatbelt sandbox](docs/guides/macos-seatbelt-sandbox.md)), and network access is
178
+ never restricted. Stub real APIs and use a container when that matters.
179
+
180
+ ### Custom spec mapping
181
+
182
+ When the naming guess does not fit the project's layout, teach the mapping
183
+ once in `config/saboteur_custom_map.rb` instead of passing `--test` on every
184
+ run:
224
185
 
225
- **It is open.** The report is JSON in the Stryker mutation-testing-report
226
- schema, and you can add your own mutants through a small public interface.
227
-
228
- ## 🛠️ Usage
229
-
230
- ```sh
231
- bundle exec saboteur run # whole project
232
- bundle exec saboteur doctor # environment checks only
233
- bundle exec saboteur doctor --suite --seed 42 # plus two unchanged suite checks
234
- bundle exec saboteur run --since origin/main # methods changed on this branch
235
- bundle exec saboteur run --subject 'Billing::Invoice#total'
236
- bundle exec saboteur run --subject Billing # a namespace
237
- bundle exec saboteur run --incremental # reuse verdicts whose inputs did not change
238
- bundle exec saboteur run --seed 1234 # repeat a run that used seed 1234
239
- bundle exec saboteur run lib/a.rb lib/b.rb --test spec/a_spec.rb --test spec/b_spec.rb
240
- bundle exec saboteur run --threshold 90 # exit 1 when the score is below 90 %
241
- bundle exec saboteur run --id DIGEST --seed 1234 # replay one full current mutant digest
242
- bundle exec saboteur run --fail-fast # stop after the first subject with a survivor
243
- bundle exec saboteur run --matrix # every covering test per mutant: blind and redundant tests
244
- bundle exec saboteur run --verbose --no-color # add score split, timings and kill causes; no colour
186
+ ```ruby
187
+ SABOTEUR_CUSTOM_MAP = {
188
+ "lib/aasm/core/state.rb" => ["spec/unit/state_spec.rb"],
189
+ "app/jobs/**/*.rb" => ["spec/jobs/**/*_spec.rb"],
190
+ %r{\Aapp/jobs/(.*)\.rb\z} => ['spec/jobs/\1_spec.rb'],
191
+ "app/generated/**/*.rb" => [], # intentionally untested
192
+ }.freeze
245
193
  ```
246
194
 
247
- Positional operands name the sources to mutate: Ruby files, or directories
248
- (all their `.rb` files). Each must exist; a typo or a non-Ruby file exits 2
249
- before anything runs. They replace `include` and `exclude`. `--test`
250
- (repeatable; spec files or directories, with an optional `:LINE` or `[1:2]`
251
- selector; needs at least one source, unless `--since` is given - the changed
252
- files are the sources) replaces only the files RSpec runs. Other
253
- `rspec_args` and `.rspec` options such as `--require` stay. The coverage map
254
- is built from only those specs, each mutant runs only the ones that cover it,
255
- and the selection is part of the coverage and `--incremental` cache keys.
256
-
257
- `--threshold PERCENT` (0 to 100, or `c.threshold` in `config/saboteur.rb`) is for CI.
258
- The run exits 1 when the score is below PERCENT and 0 when it is at or above,
259
- even with survivors. It fails closed: a run that tested no mutant has no score
260
- and exits 1, and a run where any mutant errored or hit a resource limit exits 3,
261
- because the score covers only part of the run. Without a threshold, any
262
- unignored survivor exits 1. Either way, a run where more than half of the
263
- executed mutants errored exits 3: errored mutants are outside the score,
264
- so such a run tested almost nothing.
265
-
266
- `--max-timeouts N` and `--max-errors N` (or `c.max_timeouts` / `c.max_errors`)
267
- are two more CI gates. Each takes a count or a percentage of the executed
268
- mutants (`--max-errors 5%`). The run exits 1 when the count (or share) is
269
- above the limit, with `exitCause` `maxTimeouts` or `maxErrors` in the JSON report.
270
- `--threshold` already compares the covered score, `(killed + timeout) /
271
- (killed + survived + timeout)`, so no separate score flag exists. With
272
- `--max-errors`, errored mutants are judged by that limit instead of making a
273
- `--threshold` run exit 3. Survivors and a score below the threshold keep
274
- their own causes when they fail too. The gates cannot be combined with `--agent`.
275
- The GitHub Action has no input for them yet.
276
-
277
- Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`, `--baseline-passes N`, `--max-duration DUR`,
278
- `--config PATH`, `--integration NAME`, `--reeval class_eval|nesting`,
279
- `--json PATH`, `--no-cache` (rebuild the coverage map), `--allow-focus`.
280
-
281
- Two presets bundle the common loops (#244): `saboteur pr [REF]` is the pull-request loop - `--since REF` (default `main`), one baseline check and a ten-minute `--max-duration` that opts out of the budget-specific exit 4 (normal and incomplete verdicts still fail the run); `saboteur audit` is the whole project: `--all`, all operators, both baseline checks and both reports (an explicit `--test` replaces the injected `--all`; configured selection limits such as `since`, subjects, exclusions and a configured budget still apply). The run prints the expansion up front, and an explicit flag after the preset wins.
282
-
283
- A stray `fit`, `fdescribe` or `:focus` example, with `filter_run_when_matching :focus`, makes RSpec skip
284
- every other example. The baseline stays green and the score would cover only the focused ones, so
285
- saboteur exits 2 and names the focused examples. Remove the focus, or pass `--allow-focus` (config
286
- `c.allow_focus = true`) to run the narrowed suite. Filters you name in `rspec_args` or `.rspec`, such
287
- as `--tag slow`, and example selectors such as `--test spec/a_spec.rb:12`, are on purpose and are not refused. The
288
- check compares what RSpec runs with what it would run without the stray focus, so examples RSpec leaves out anyway
289
- (`:if` / `:unless`, `--tag`, exclusions, selectors in another file) never count as hidden.
290
-
291
- For pull requests, start with `--since` against your target branch. This is a
292
- workflow recommendation, not a CLI default: plain `run` still checks the whole
293
- configured project. To scope the specs yourself, pass them with `--test` and no
294
- source operands: `bundle exec saboteur run --since origin/main --test spec/billing_spec.rb`.
295
- A changed Ruby file whose changed lines cover no method mutates nothing; the run
296
- warns and names the file instead of silently selecting zero subjects. An untracked
297
- file inside the configured `include` set counts as fully changed: a brand-new,
298
- never-committed source file is selected in full rather than silently skipped,
299
- because a diff cannot see it (#295). The union reads race-free
300
- (O_NOFOLLOW/O_NONBLOCK, validated on the opened descriptor) and streams its
301
- line count, so a swapped-in symlink or FIFO is skipped quietly and a huge
302
- untracked fixture outside `include` is never even opened. Comment-only
303
- or formatting-only changes within discovered methods are skipped without this
304
- warning; the run reports those skipped methods. A changed Ruby file that an
305
- include pattern covers but `c.exclude` removes mutates nothing either, and the
306
- run warns and names up to five such files - narrow or remove the `c.exclude`
307
- entry, or pass the file as a positional source operand (configured excludes are
308
- then ignored). `--subject` cannot help here: it filters subjects inside the
309
- files left after excludes.
310
-
311
- Without `--test`, the run infers the specs: the spec named after each mutated
312
- file (`lib/a/b.rb` runs `spec/a/b_spec.rb`; `app/` files and `spec/lib/` work
313
- too) runs for the baseline passes, the coverage pass and every mutant, as
314
- `--test` would. With `--since` only the changed files count; without it every
315
- included file does. It is a naming guess, not coverage: a spec that reaches the
316
- mutated file only through another file does not run, so a mutant only that
317
- spec would kill can survive, and a line none of the inferred specs executes is
318
- reported as NoCoverage. A file needs no inferred spec when it has no mutation
319
- subjects (with `--since`: none on its changed lines), such as a comment-only
320
- change within a method or a deletion leaving only blank/comment context; the
321
- run says so and skips those files. When no file has a mutation subject at all,
322
- the run reports that there is nothing to mutate. A `--subject` run infers only
323
- from the named subjects' files: an unrelated spec-less file never stops a
324
- one-method run, and a subject file with no conventional spec stops the run
325
- like any other unmatched file. With `--since` only named subjects on changed
326
- lines count: an unchanged subject file, one whose changes sit in other
327
- methods, or one with a comment-only change is skipped, not stopped.
328
-
329
- A file that needs a spec and has none stops the run before it boots (exit 2)
330
- and names both ways out: `--test` with the specs to run, or `--all` for the
331
- whole suite. An unmatched file no longer silently narrows the selected source
332
- scope or widens the test run to the whole suite. Whole-project audits should
333
- use `--all` explicitly. The agent JSON records the inferred specs as its
334
- `tests`, so its follow-up commands keep the same scope.
335
-
336
- `--all` (config `c.test_selection = :all`) runs the whole configured suite.
337
- With `--since` or `--all` and no inferred or named specs, the run warns before
338
- it starts that the two baseline passes run the loaded suite (a cold coverage
339
- cache adds one more pass). `--guess-tests` is a deprecated alias for the
340
- default inference and `--no-guess-tests` warns and is ignored; use `--all` for
341
- the whole suite. `--all` cannot be combined with `--test` or `--guess-tests`,
342
- and `--guess-tests` cannot be combined with `--test`. Keep whole-project runs
343
- on main: diff mode does not check every method affected by a dependency change.
344
- [The pull-request guide](docs/pull-requests.md) covers
345
- checkout history, a GitHub Actions example, configuration and limits.
346
-
347
- Doctor warns when an ignore subject matches no method discovered in the included
348
- source files. Check the exact `Class#method` or `Class.method` form and the
349
- include/exclude settings; warnings never edit or delete ignore entries.
350
- A malformed ignore file fails doctor with a config error before target boot.
351
-
352
- `saboteur doctor` prints Ruby, coverage, integration, Rails/SimpleCov and boot-file
353
- checks. Add `--suite` to also run the unchanged suite twice with the chosen seed
354
- (`--test PATH`, repeatable, limits those runs to some specs and implies `--suite`),
355
- since that takes hours on a large suite. It writes no
356
- mutation reports or coverage/verdict caches. Target boot files and tests still
357
- run, so their normal side effects remain. See the [doctor guide](docs/guides/doctor.md).
358
-
359
- `--id` runs exactly one current mutant with its coverage-selected tests, bypassing
360
- per-method caps, ignore suppression and incremental verdict reuse.
361
- It still honors `--subject` and `--since` changed-line selection, and the
362
- hazard guard: a digest that names a suppressed mutant is refused unless
363
- `--allow-hazardous` is also given. See the
364
- [replay guide](docs/guides/replay.md) for digest format, checks and limits.
365
-
366
- Every test run starts from the same random seed (1 by default), so tests that
367
- use `rand` see the same values in each mutant's run and in the check that
368
- confirms a kill. The report prints the seed; pass it to `--seed` to repeat a
369
- run.
370
-
371
- `--incremental` reuses an earlier verdict only while the method, the whole
372
- file it lives in, the mutant, its covering tests and their files, the lockfile
373
- and the config are all unchanged. So a constant or helper changed in the same
374
- file is seen. It cannot see a change in another file that the method uses (a
375
- method it calls, a constant defined elsewhere), so run without it now and then
376
- (and in CI on the main branch).
377
-
378
- Output: survivors first, then a short summary last, so the summary stays on
379
- screen. Survivors are grouped by file, then method, then line. Each line is
380
- printed once as `path:line in Subject`, with the source line and the tests that
381
- covered it, then one compact line per mutation: `operator original -> replacement digest`,
382
- and a hint on what kind of test would kill it. The summary has the counts per
383
- status, the score, a table of scores per file (worst first) and a one-line
384
- legend. When survivors remain, the Next steps footer prints the complete
385
- replay command - ref, subjects, specs, sources, seed and the run settings
386
- the agent protocol records, including the CLI-only switches that change
387
- candidate availability or boot (`--allow-hazardous`, `--allow-focus`) -
388
- shell-escaped and printed in full however long it gets, so running it
389
- as-is replays the survivor with the same selection. Framework and
390
- generation settings from the config file (arid/hazard patterns,
391
- whole-file, require, rspec_args, rails) ride along through the command's
392
- `--config` or the default config path. Only knobs that cannot change a
393
- by-digest replay (`--max-mutants-per-method`, `--fail-fast`,
394
- `--threshold`) are left out. The agent protocol instead omits a command
395
- longer than 1024 bytes, and the agent rebuilds it from the run's
396
- provenance. `--verbose` adds the score split, mean time, whole-body score and
397
- kill causes, and prints one line per finished subject with its result
398
- marks, so a survivor is attributable to a method before the run ends. Colour is on when stdout is a terminal and `NO_COLOR` is not set;
399
- `--color` and `--no-color` override that. While mutants run, stderr shows
400
- `done/total (percent) eta`: a live counter on a terminal, and dots with a
401
- counter every 50 mutants otherwise. Skipped and reused mutants are not
402
- counted, so the count can end below the total.
403
- The tests that covered the survivor's line and
404
- still passed with the mutant applied are listed (the first five, likeliest killers
405
- first; the rest are under `coveredBy` in the JSON report). The missing
406
- check may belong in one of them. Line coverage can overstate which tests
407
- exercise a line, so treat the list as a lead. A survivor reused with
408
- `--incremental` was not run again, and the list says its tests passed in an
409
- earlier run. `--fail-fast` (serial runs only) stops after the first subject
410
- that has a survivor the ignore list does not accept. The report then covers
411
- only the subjects that ran: the terminal says which subjects did not run, the
412
- JSON report has an `earlyStop` field naming them (`exitCause` is `failFast`),
413
- `--agent` reports status `partial` with the reason `fail_fast`, and the run
414
- always exits 1, so no `--threshold` can pass it. It cannot be used with
415
- `baseline` evidence. Survivors that share a line are printed together only
416
- when their covering tests, reuse status and changed lines are identical;
417
- otherwise each distinct set of evidence is printed above its own mutations.
418
- Hints are suggestions, not diagnoses: a survivor can also be an equivalent
419
- mutant. The HTML page shows the same list in the
420
- survivor's notes. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
421
- report (one file, no scripts, no network) to `.saboteur/report.html`. Change
422
- the paths with `--json` / `--html` or `c.json_report` / `c.html_report` in
423
- `config/saboteur.rb`; set either to `nil` to turn that report off. The two paths
424
- must differ. When an artifact cannot be written - a read-only tree inside a
425
- sandboxed run - the HTML report and the score history warn and the run still
426
- exits by its results. An explicitly requested JSON report (`--json` or
427
- `c.json_report`) is different: it is the completion evidence an outer harness
428
- trusts, so a failed write fails the run with exit 3 rather than leaving a
429
- stale or partial file presented as this run's; the default
430
- `.saboteur/report.json` stays best-effort. The HTML page shows the covered mutation score
431
- `(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage,
432
- RanAtLoad and Unreachable out and keeps survivors accepted on the ignore list
433
- in the denominator. A tool
434
- that reads the JSON report may show a different score, because the schema counts
435
- NoCoverage as undetected.
436
-
437
- `--matrix` runs every covering test against every mutant instead of stopping at
438
- the first failing one, so a killed mutant lists all of its killers. After the
439
- survivors the report prints two lists: blind tests (they covered mutants and
440
- killed none) and redundant tests (they killed some, and every one of those
441
- mutants had another killer). The run is slower, by up to the number of covering
442
- tests per mutant. `--matrix` cannot be combined with `--incremental`, because a
443
- reused verdict lists only the first killer. A redundant test is a lead, not a
444
- verdict: it may guard behavior no selected mutant touches. A selection with
445
- `--since` or `--subject` describes only the selected methods.
446
-
447
- [Scores](docs/scores.md) defines each number on this page: the covered score, the
448
- score a Stryker dashboard shows, the assertion and crash split, and what makes a run
449
- incomplete.
450
-
451
- The score is split by how the mutant was killed. A crash kill (a `NoMethodError`,
452
- a `TypeError`, a parse error) shows the code ran and broke, not that a test
453
- checked a value. The terminal prints
454
- `score 100.0 % (assertion 50.0 %, crash 50.0 %)`: each share is a part of
455
- tested mutants, so the parts add up to the score. Timeouts and kills with no
456
- recorded cause (a verdict reused from an older run, a test that called `exit`,
457
- a failure outside any example) are named when there are some. Whole-body
458
- mutants (`body_nil`, `body_true`, `body_false`, `body_raise`) are killed by
459
- almost any test that runs the method, so a second line scores the run without
460
- them, and shows them alone. The headline score and `--threshold` still count every
461
- mutant. The JSON report carries both splits under `saboteur.scores`
462
- (`all` and `withoutWholeBodyMutants`), and the HTML page prints them under
463
- the score. The JSON also records why the run exited the way it did under
464
- `saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`,
465
- `errorMajority`, `incompleteUnderThreshold`). In `--agent` mode the cause
466
- follows the agent verdict, so the report and the process exit classify the
467
- run the same way: `ok` for clean, `survivors` for survivors, and otherwise
468
- `errorMajority` when most mutants errored or `incomplete`.
469
-
470
- For code-review surfaces, the JSON report also carries a `saboteur.findings`
471
- section: a bounded, deterministically ordered list of survivors, coarse
472
- mutants first (a surviving coarse mutant hints that no test pins the method's
473
- behavior - it can also be a re-evaluation limit or an equivalent mutant),
474
- then file, line and digest. Each finding holds the file, line, digest,
475
- subject, operator, original and replacement a note on a pull-request diff
476
- needs, plus one entry per edit for mutants that edit more than one spot.
477
- Survivors accepted on the ignore list are listed separately with their
478
- reason, and `survivorsTruncated` and `acceptedTruncated` count what the bound
479
- left out; the full data stays in the schema's `files` section.
480
-
481
- In diff mode (`--since`), a mutant is reported at the line where its statement
482
- starts. A statement that spans several lines can start a line or two above the
483
- line that changed. Under each survivor the terminal therefore prints
484
- `changed line: path:N`, the changed line or lines (the first three) the mutant
485
- touches, and each finding carries them as `changedLines`. Use `changedLines`,
486
- not `line`, to put a note on a changed line of the pull-request diff. A
487
- whole-body mutant of a method whose only change is its signature touches no
488
- changed line and prints none.
489
-
490
- New to survivors? The [reading survivors guide](docs/guides/reading-survivors.md) walks one from the diff to the test that kills it.
491
-
492
- Exit codes: 0 no survivors (or all on the ignore list), or the score
493
- meets `--threshold`; 1 survivors, or the score is below `--threshold` or
494
- missing; 2 bad configuration or a missing source or test; 3 the tool
495
- failed (with `--threshold`, also any errored or resource-limited
496
- mutant). Unexpected `StandardError` or `LoadError` failures print
497
- `saboteur: internal error (...) - please report it` and exit 3, rather
498
- than dying with a bare stack trace and an accidental exit code. The full
499
- backtrace lands in `<cache_dir>/internal-error.log` (default
500
- `.saboteur/`), and `--verbose` also prints it to stderr. When the cache
501
- dir cannot hold the log, a private pid-keyed fallback in the system temp
502
- dir takes it (created with `O_EXCL | O_NOFOLLOW`, mode 0600, so it never
503
- follows or overwrites a pre-existing path). Logs are always written
504
- private, and a nested run reads them back only through a no-follow,
505
- non-blocking descriptor - a regular file, same owner, no group/other
506
- bits (or, where the platform lacks those flags, only the `O_EXCL` temp
507
- fallback is written and no excerpt is read). The message names the log only
508
- after a successful write: an unset `cache_dir`, or a log writable nowhere,
509
- leaves the one-liner standing alone. The executable
510
- loads the crash reporter before the CLI, so a failure while it is
511
- loading, being built, or parsing arguments gets the same report;
512
- failures earlier than that - coverage setup or the reporter's own load -
513
- get only the bare one-liner and still exit 3. Once the crash
514
- reporter has loaded, agent mode reports the same failure as agent
515
- JSON, using a standalone payload if the agent reporter has not
516
- loaded. Before the crash reporter loads, only the one-line
517
- diagnostic and exit 3 are available.
518
- `--max-duration 10m` budgets the whole run, baselines included: the budget is checked before every subject and between mutants inside one, so past the budget the run finishes only the mutant in flight (bounded by its timeout), then stops before the next subject. Admitted phases (baseline checks, the coverage map build) are capped at the time left, as are verdict checks and timeout calibrations - admission is per check and per sequence, so nothing forks with no time left. A phase that hits only its budget-shortened cap stops the run as budget-exhausted rather than failing as a suite error. A kill or timeout the budget leaves uncheckable is reported as `unverified` - outside the score, never cached, listed under the early stop - instead of becoming an error, reports which selected subjects never ran and which mutants were omitted, marks the JSON and agent output partial (`earlyStop.reason: "max_duration"`), and exits 4; `--max-duration-ok` (or `c.max_duration_ok = true`) leaves the exit to the plain verdict instead, which can still fail the run - 1 on survivors, 3 when the budget stopped the run before any mutant verdict was verified. In agent mode the exit code stays verdict-driven and the budget shows up in `incomplete[]` and `earlyStop` (#245).
195
+ A matching entry wins over the naming convention: the file stops being
196
+ unmatched and its mapped specs run as the inferred ones. Keys are exact
197
+ paths, globs or Regexps; an empty array marks the file intentionally
198
+ untested, so it needs no inferred spec. The file loads before the app
199
+ boots, so keep it plain Ruby. `saboteur doctor` validates it.
519
200
 
520
201
  ### Rejected kills
521
202
 
522
- The terminal prints `kills rejected by unchanged-method check: N`; JSON adds
523
- `rejectedKills`. This counts fresh per-mutant kill claims whose tests also fail
524
- with the method unchanged. They stay errors outside the score. Timeout checks,
525
- other errors and reused cached kills are not included. See
526
- [rejected kills](docs/rejected-kills.md) for details.
527
-
528
- ## ⚙️ Configuration
529
-
530
- `config/saboteur.rb` in the project root; CLI flags override it. Keep the shared
531
- config usable for whole-project runs and pass `--since` in the pull-request
532
- job. A PR-only config may set `c.since = "origin/main"`; omit `since` (or use
533
- `c.since = nil`) in the config for audits on main. Use your actual target branch.
534
-
535
- ```ruby
536
- Saboteur.configure do |c|
537
- c.include = ["lib/**/*.rb"]
538
- c.exclude = ["lib/my_gem/version.rb"]
539
- c.require = [] # boot files, e.g. ./config/environment
540
- c.whole_file = [] # subject ids or globs to re-evaluate as a whole file (see below)
541
- c.reeval = "nesting" # or "class_eval"; see docs/design.md 6.1
542
- c.timeout_factor = 3 # per-mutant deadline = factor x its tests' recorded time
543
- c.timeout_min = 5
544
- c.max_memory_mb = 2048 # memory growth cap per mutant process tree
545
- c.max_processes = 64
546
- c.max_mutants_per_method = 100 # positive integer; raise it for a broader run
547
- c.hazard = Saboteur::MutationSource::HazardGuard::DEFAULT_HAZARD # calls whose arguments are never mutated
548
- c.sandbox = true # read-only project tree for mutant children (Linux 6.2+); "strict" refuses to run degraded
549
- c.sandbox_writable = [] # extra dirs mutant children may write, e.g. ["db", "log"]
550
- c.ignore_file = "config/saboteur_ignore.rb"
551
- c.seed = 1 # Ruby's random seed in every test run
552
- c.baseline_passes = 2 # unchanged-suite checks before mutating; 1 skips the order-stability check (#55)
553
- c.max_duration = nil # wall-clock budget in seconds, e.g. 600; an exhausted budget exits 4 (#245)
554
- c.max_duration_ok = false # true: an exhausted budget does not fail the run
555
- c.allow_focus = false # true runs a suite narrowed by fit/fdescribe/:focus instead of exit 2
556
- c.fingerprint_extra = [] # extra globs whose contents invalidate caches, e.g. ["data/*.csv"]
557
- end
558
- ```
203
+ The terminal lists them in a `Rejected kills` section and annotates the error
204
+ count (`error 3 (2 rejected kills)`); JSON flags each rejected mutant with
205
+ `rejectedKill: true` and keeps the `rejectedKills` count. These are fresh
206
+ per-mutant kill claims whose tests also fail with the method unchanged. They
207
+ stay errors outside the score. Timeout checks, other errors and reused cached
208
+ kills are not included. See [rejected kills](docs/rejected-kills.md) for
209
+ details.
559
210
 
560
211
  ### Rails apps
561
212
 
562
- Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a Rails
563
- app needs a few settings; the generator writes them for you.
564
-
565
- **1. Add the gem**
213
+ Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a
214
+ Rails app needs a few settings; the generator writes them for you:
566
215
 
567
216
  ```sh
568
217
  bundle add saboteur --group test
569
- ```
570
-
571
- **2. Generate the setup**
572
-
573
- ```sh
574
218
  bundle exec rails generate saboteur:install
575
219
  ```
576
220
 
577
- This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets `RAILS_ENV=test`
578
- for you) and a `.gitignore` entry for `.saboteur/`. It never overwrites an
579
- existing file unless you pass `--force`, and never edits your Gemfile.
221
+ This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets
222
+ `RAILS_ENV=test` for you) and a `.gitignore` entry for `.saboteur/`. It
223
+ never overwrites an existing file unless you pass `--force`.
580
224
 
581
- Prefer to do it by hand, or on a version without the generator? Put this in
582
- `config/saboteur.rb`:
225
+ Until a config names your sources, a run whose include selects no Ruby
226
+ file refuses with exit 2 and points at the install generator above -
227
+ a first CI run on an unconfigured tree fails loud instead of passing
228
+ with 0 subjects.
229
+
230
+ Prefer to do it by hand, or on a version without the generator? Put this
231
+ in `config/saboteur.rb`:
583
232
 
584
233
  ```ruby
585
234
  Saboteur.configure do |c|
586
- c.include = ["app/**/*.rb"]
235
+ c.include = ["app/**/*.rb"] # spec/test dirs vendored under lib/ are excluded by default
587
236
  c.require = ["./config/environment"] # boots the app
588
237
  c.rails = true # reconnects ActiveRecord after fork, per-worker databases
589
238
  c.jobs = 4 # optional; the default is 1
590
239
  end
591
240
  ```
592
241
 
593
- **3. Check the setup**
594
-
595
- ```sh
596
- bin/saboteur doctor
597
- ```
598
-
599
- Doctor boots your app, checks the config and the test database host, and runs
600
- your suite twice before any mutation. Fix whatever it reports first. Turn
601
- SimpleCov off for the run if you use it.
602
-
603
- The database check refuses any test database whose host is not local
604
- (`localhost`, `127.0.0.1`, `::1`, `0.0.0.0` or a socket), so a remote
605
- database is never mutated by accident. `0.0.0.0` counts as local: compose
606
- setups write it into database URLs, and connecting to it reaches this
607
- machine. For another disposable test database host, assert it yourself:
608
- `c.allow_db_hosts = ["db.internal"]` in `config/saboteur.rb`.
609
-
610
- **4. Test what your branch changed**
611
-
612
- ```sh
613
- bin/saboteur run --since main
614
- ```
615
-
616
- It mutates only the methods changed since `main` and reports each survivor with
617
- its diff: a change your tests did not notice. Write a test that fails on that
618
- diff, then run again. Use `--since origin/main` in CI. Without the generator, use
619
- `bundle exec saboteur` and `RAILS_ENV=test` instead of `bin/saboteur`.
620
-
621
- saboteur puts the app's `spec/` directory on the load path before
622
- processing `require`, matching rspec, so a boot file like a dummy app's
623
- `spec/rails_helper.rb` can require its siblings.
624
-
625
- With `jobs` above 1 on SQLite, each extra worker gets a copy of the test
626
- database made through SQLite's backup API, so the copy is a consistent
627
- snapshot. A relative `database:` path resolves against `Rails.root`, the
628
- same as Rails 7.1+ does, for the copy and for its cleanup. The copy
629
- checkpoints the source first (`PRAGMA wal_checkpoint(TRUNCATE)`) and
630
- refuses to run when a reader keeps the WAL busy, because copying on
631
- would silently miss committed rows. A source file that does not exist is
632
- an error, not a skipped copy. An in-memory SQLite database (`:memory:`) is
633
- never copied, renamed or disconnected. After the app boots, each in-memory
634
- database is snapshotted to a temporary file through SQLite's backup API,
635
- and every forked child restores the snapshot into its own fresh database -
636
- a child cannot keep the parent's connection (ActiveRecord discards
637
- inherited pools and sqlite3 closes inherited writable handles), so without
638
- the snapshot it would see an empty database.
639
-
640
- For a Rails engine, point `require` at the dummy app's environment -
641
- `test/dummy/config/environment.rb`, usually loaded for you by
642
- `spec/rails_helper.rb`. Only a dynamically created dummy app has no
643
- environment file; there, require the helper that calls setup instead,
644
- such as `DummyApp.setup` in `spec/rails_helper.rb`.
645
-
646
- Bootsnap needs no extra setting: bytecode from its compile cache carries no
647
- coverage counters, so saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its
648
- own process when the variable is not already set (Bootsnap treats any set
649
- value as off) and runs the eager load with Coverage active. App code is seen
650
- by Coverage with the app unchanged.
651
-
652
- With `jobs: 4` (or more), each extra worker gets a copy of every test
653
- database, named `<database>_saboteur2` and up, made with
654
- `CREATE DATABASE ... TEMPLATE` on Postgres or a file copy on SQLite.
655
- saboteur drops the copies when the run ends, on a best-effort basis: a
656
- drop that fails is reported as a warning and never changes the run's
657
- outcome. A copy left behind - by a failed drop or a killed run - is
658
- reclaimed by the next run, whose prepare starts from
659
- `DROP DATABASE IF EXISTS`. MySQL is not supported for `jobs` above 1.
660
-
661
- Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
662
- `db/schema.rb` and `config/*.yml` (including `database.yml`) are part of the
663
- cache fingerprint automatically. Use `fingerprint_extra` for any other file your
664
- tests read.
665
-
666
- ### Load-time code, captured methods and Sorbet signatures
667
-
668
- Code that runs while the target boots, before tests run, cannot be checked by
669
- replacing a method later. saboteur reports these changes as `ran_at_load` rather
670
- than survivors. Calls from test setup, including RSpec `before(:all)`, are still
671
- checked. Load-time detection follows the `ran_at_load` idea in
672
- [mutineer #187](https://github.com/davidteren/mutineer/pull/187).
673
-
674
- A saved `Method` or `UnboundMethod` can keep calling the old definition after
675
- saboteur replaces it. When tests pass without calling the replacement and a
676
- saved method still points to the old definition, the result is `unreachable`.
677
- A targeted execution check must also confirm the replacement was not called.
678
- Each mutant is checked separately; one unreachable result does not skip later mutants.
679
- The reason names the old definition's source location. A lambda that looks up
680
- the method when it runs is still checked normally.
681
-
682
- Coverage caches now store both per-test method hits and load-time line hits.
683
- Older caches are rebuilt; load-time accounting preserves endless-method coverage.
684
-
685
- One-line and endless method definitions need boot method-call counters to prove
686
- the body ran at load; merely defining them is not execution evidence. If methods
687
- share a physical line and any sibling is uncalled, the line is ambiguous and is
688
- conservatively reported as no coverage, even for its called sibling.
689
- Agent mode reports ran-at-load and unreachable work as incomplete.
690
-
691
- A whole-suite map with only load-time hits is valid and reports load-only subjects
692
- with their ran-at-load reason, rather than failing as an empty coverage build.
693
- It has no mutation score when no mutant can be tested.
694
-
695
- Both statuses are listed separately in terminal output and excluded from the
696
- mutation score. Stryker JSON reports them as `Ignored`, with the explanation in
697
- `statusReason`; they are not entries on your survivor ignore list.
698
-
699
- Methods wrapped by `sorbet-runtime` signatures are checked rather than skipped
700
- as shadowed definitions. saboteur re-evaluates the adjacent `sig` with the
701
- changed method so the replacement keeps its signature wrapper. Missing adjacent signatures and unknown wrappers are skipped when saboteur cannot
702
- preserve the loaded method semantics.
703
-
704
- ### Whole-file re-evaluation (opt-in)
705
-
706
- Surgical re-evaluation refuses a method whose owner it cannot resolve or whose
707
- loaded definition is not the one in the file. The skip reason ends with a hint.
708
- List such subjects in `c.whole_file` (ids or globs such as `"Point#distance"` or
709
- `"Geometry::*"`) and the child evaluates the whole mutated file instead. This
710
- applies to methods saboteur already finds as subjects; it does not make new
711
- kinds of definitions selectable (methods inside `Data.define` or `Struct.new`
712
- blocks are still not found). The file's top-level code then runs again in every
713
- child, after the run's seed is set: load-time side effects repeat and constants
714
- it assigns are rebuilt, with redefinition warnings silenced. If a later
715
- definition in the same file replaces the mutated method, the mutant could never
716
- run, so a survivor there is reported as unreachable instead. It is off by
717
- default, and the cache key includes the list.
718
- Signature text alone does not remove mutants. `body_nil` and last-statement
719
- removal remain available even when `sig` declares a non-nil return type.
720
- `checked(:never)`, unchecked `checked(:tests)` signatures, configured default
721
- levels, custom validation handlers and wrapper shims can all allow nil to
722
- reach real assertions. saboteur does not verify enforcement, evaluate return
723
- type expressions or infer types from constant names. Aliases, enclosing
724
- namespace constants and transformed generic types therefore cannot change
725
- mutation generation. RBS signatures are not read yet. Type-only crash kills
726
- remain visible in the crash share, not silently pruned.
727
-
728
- The Sorbet E2E fixture is a wrapper/registry shim without runtime type checking.
729
- It tests re-evaluation and assertion detection, not real Sorbet enforcement or
730
- proof of type-check noise reduction.
731
-
732
- ### Skipped subjects and the mutant cap
733
-
734
- Subjects with no test coverage are skipped before mutation sources run. They
735
- remain in the subject count, are listed as skipped subjects in terminal output,
736
- and appear under `skippedSubjects` in JSON. They are not invented mutants and do
737
- not enter saboteur's mutation score. Stryker dashboards may ignore the extra
738
- `skippedSubjects` field and calculate scores from mutant entries alone, so
739
- removing previously generated `NoCoverage` mutants can raise a dashboard's score
740
- even when test detection has not improved. Changes inside a covered method that
741
- no test reaches still produce mutant-level "no coverage" results.
742
-
743
- Each covered method keeps at most `max_mutants_per_method` valid, distinct
744
- mutants (100 by default). Coarse mutants come first, then fine mutants in
745
- configured mutation-source order and each source's proposal order. The cap keeps
746
- a prefix of that order, so earlier sources can use the entire limit within a
747
- method. Later methods have their own limit. In diff mode, changed-line selection
748
- happens before the cap. A warning names each capped method and the number
749
- omitted; JSON keeps these warnings under `warnings`. The score describes this
750
- selected subset, not every possible mutation. Raise the limit to examine more
751
- mutants; the limit does not bound how many proposals a custom source creates.
752
-
753
- ### Ignoring a survivor
754
-
755
- Every survivor is printed with a digest. Accept it with a reason:
756
-
757
- ```ruby
758
- # config/saboteur_ignore.rb
759
- Saboteur.ignore do |i|
760
- i.mutation "1a2b3c4d5e6f7a8b",
761
- subject: "Billing::Invoice#total",
762
- reason: "equivalent: amounts are always whole cents"
763
- end
764
- ```
765
-
766
- An entry without a reason is rejected.
767
-
768
- ### SimpleCov
769
-
770
- saboteur needs Ruby's `Coverage` for itself. Disable SimpleCov for the run
771
- (most projects already guard it with an environment variable).
772
-
773
- ### Threads alive at fork time
774
-
775
- Threads do not survive `fork`, so a run names any that are alive when the
776
- workers fork. Two are well known:
777
-
778
- - `AR Pool Reaper` only maintains the parent's connection pool. It is
779
- expected harmless when children reconnect the database after fork -
780
- saboteur's rails mode does exactly that. Without verified
781
- reconnection, do not assume the inherited connection state is safe.
782
- - `DEBUGGER__::SESSION@server` is the debug gem's remote console. What
783
- it does in children depends on `RUBY_DEBUG_FORK_MODE`: with `parent`
784
- no debugger runs in children; with `child` or `both` (the gem's
785
- default) the debugger activates in children and can pause or
786
- interfere with supervised mutation runs. For unattended runs, do not
787
- start the remote debugger at all: drop `RUBY_DEBUG_OPEN` or require
788
- `debug` without `:open`.
789
-
790
- Any other thread gets a generic caveat: a thread that does not run in a
791
- child cannot do background work a mutant's test may rely on, so results
792
- can differ when one is left behind.
793
-
794
- ## Filter mutations
795
-
796
- The built-in source also swaps `select` / `reject` and `select!` / `reject!`.
797
- These catch a filter that keeps the items it should discard (or discards the
798
- items it should keep). For example, a command replacement should remove the
799
- old command, not keep only that command:
800
-
801
- ```ruby
802
- commands.reject! { |name, _| name == replaced_name }
803
- # mutant: commands.select! { |name, _| name == replaced_name }
804
- ```
805
-
806
- Only explicit-receiver calls with a block and no ordinary arguments are
807
- changed. A block passed as `&predicate` counts; safe navigation is preserved.
808
- `relation.select(:name)`, bare `select { ... }`, and `items.select` without a
809
- block are not changed. Arid statements and nested definitions remain skipped.
810
- Ruby dispatch is dynamic: a matching name does not prove the receiver is an
811
- Enumerable, and a custom receiver can make the replacement fail. No type or
812
- real-fault detection guarantee is made.
813
-
814
- These are fine mutants, appended after the existing built-in operators. For unchanged subjects, old
815
- mutant digests and relative order stay unchanged, but runs can produce more
816
- mutants, change their score, and hit the per-method cap. The operator and source
817
- name in reports is `filter_replacement`.
818
-
819
- See [the operator rationale](docs/filter-operators.md) for the Ruby fault used
820
- to choose this family and the limits of that evidence.
821
-
822
- ## Literal mutations
823
-
824
- Builtin now swaps `true` / `false` literals and replaces nonempty static strings
825
- with `""`. These test boolean choices and the difference between meaningful
826
- text and empty input. Operator names are `literal_boolean` and `literal_string`;
827
- the source name is `literal_replacement`.
828
-
829
- Only method bodies are visited. Default arguments, arid statements, nested
830
- definitions, empty strings, heredocs, interpolation, word/symbol lists, symbols,
831
- regexes and command strings are unchanged. Quoted and static percent strings
832
- are supported. The new proposals follow the earlier built-in operators; the
833
- normal duplicate check and per-method cap still apply. Old digests and order
834
- remain stable for unchanged subjects, but more mutants can change a run's score.
835
- No speed, type-safety or real-fault detection improvement is claimed. See
836
- [the literal operator rationale](docs/literal-operators.md) for fault evidence
837
- and limits, including string encoding/frozen-value behavior.
838
-
839
- ## Chain removal
840
-
841
- Builtin drops one link from the middle of a dotted method chain: `user.name.strip`
842
- becomes `user.strip`, and `a.b.c.d` gives `a.c.d` and `a.b.d`. The first receiver
843
- and the last call stay, so a test has to notice that a step in between went
844
- missing. The link goes whole, with its arguments and block. The operator and
845
- source name is `chain_removal`.
846
-
847
- Operators, index calls and bare-name receivers are not chains and are unchanged.
848
- The target of a write (`user.account.profile.name = 1`, `+=`, `||=`, `a.b[0] = 1`)
849
- is left whole: no link of its receiver chain is dropped, however long. Only
850
- the value on the right can be mutated. `freeze`, `dup`, `clone`, `itself`
851
- and `tap` links are never removed. Tests can observe frozenness, copies and a
852
- `tap` block's effects, but those are other contracts than a missing step in a
853
- query chain, and dropping such a link mostly yields noise, so this operator
854
- leaves them out. Arid statements and nested definitions are skipped.
855
-
856
- If you confirm the shorter chain is equivalent across the intended
857
- inputs (for example a duplicated column), simplify the code or ignore
858
- the mutant.
859
-
860
- Chain removal is on by default, like `argument_removal` and `pair_removal`.
861
- Its proposals follow the other fine proposals, with the same validation,
862
- deduplication and `max_mutants_per_method` cap.
863
-
864
- ## Argument removal
865
-
866
- Builtin removes one ordinary positional argument from a parenthesized call.
867
- For `Model.includes(:a, :b, c: :d)`, the two proposals remove `:a` or `:b`,
868
- without deleting the statement or changing `c: :d`. This lets a preload check
869
- exercise missing associations instead of only a missing return value.
870
- The operator and source name is `argument_removal`.
871
-
872
- Only parenthesized calls are supported in this first version. Keyword hashes,
873
- splats, argument forwarding, bracket indexing/assignment syntax and arguments containing
874
- heredocs are unchanged. Explicit hash arguments count as positional arguments.
875
- Blocks and keyword pairs remain intact; separators come from Prism tokens,
876
- so commas inside strings or comments are not mistaken for separators.
877
- Arid statements, parameter defaults and nested definitions are skipped.
878
-
879
- These fine proposals follow existing builtin proposals, with the same validation,
880
- deduplication and `max_mutants_per_method` cap. Earlier digests and relative order
881
- stay unchanged for unchanged subjects. Extra proposals can change scores or be
882
- omitted by the cap. Removing an argument can still raise an arity error; no
883
- non-crashing guarantee is made. The regression uses a small preload-like receiver
884
- and checks the missing associations, not an ActiveRecord SQL benchmark. See
885
- [the validation notes](docs/argument-removal.md) for the pinned snapshot migration.
886
-
887
- ## Hash and keyword pair removal
888
-
889
- Builtin removes one key/value association from a hash literal or keyword list.
890
- `Model.includes(:a, :b, c: :d)` becomes `Model.includes(:a, :b )`, so a nested
891
- preload guard can be exercised without deleting the whole statement. A sole
892
- hash pair becomes `{}`. Operator and source name: `pair_removal`.
893
-
894
- Both parenthesized and command calls are supported. Splats stay intact; pairs
895
- containing heredocs, parameter defaults, arid statements and nested definitions
896
- are skipped. Token-based separator edits preserve comments and blocks. Nested
897
- hash values and callable bodies are visited. Removing a required keyword can
898
- still crash; the regression uses a preload-like receiver, not an ActiveRecord
899
- SQL benchmark. These fine proposals follow earlier builtin proposals and use
900
- the usual validation, deduplication, order and `max_mutants_per_method` cap.
901
- More proposals can change scores and cap coverage. See
902
- [the validation notes](docs/pair-removal.md) for snapshot changes.
903
-
904
- ## 🔌 Extending
905
-
906
- Public API, stable across minor versions:
907
-
908
- - `Saboteur::MutationSource::Base#mutations_for(subject)` returning
909
- `Saboteur::MutationSource::Mutation` objects made of `Edit`s (file, byte
910
- range, replacement). Register with `sources: [builtin, path.rb:ClassName]`.
911
- - `Saboteur::Integration::Base` for other test frameworks (RSpec is the only one
912
- that ships).
913
- - `Saboteur.hook(:before_boot | :after_boot | :before_fork | :after_fork |
914
- :around_mutant)` in files listed under `hooks:`. An extensionless hooks
915
- path resolves its `.rb`, the way `require` does.
916
-
917
- ## 🌍 Where it has run
918
-
919
- Tested on the RSpec suites of plain gems
920
- (dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
921
- human-essentials on Postgres (4,080 mutants, 6 workers; the 2026-10-10 run, before
922
- the baseline check and the coarse-gate removal) and lobsters on SQLite,
923
- both on macOS. Not yet tried on MySQL or older Rails versions. RSpec only, by
924
- design. Results: [docs/validation](docs/validation).
925
-
926
- ## ⚖️ How it compares
927
-
928
- A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
929
- 1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
930
- ran once on sampled files, so treat the numbers as indicative. The table was
931
- measured at 2b6ec02, before the literal, filter and argument operators and
932
- before the coarse-survivor gate was removed, so mutant counts and times will
933
- change; a rerun is pending. mutineer is ahead
934
- on number of operators and CI extras, and it supports Minitest.
935
-
936
- | Metric | saboteur | mutineer |
937
- |---|---|---|
938
- | Fake kills (lobsters) | 0 of 20 sampled at random | 8 of 8 hand-checked (identical mutants, not a random sample) |
939
- | Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
940
- | Runs on the Postgres app | yes | no |
941
- | Time per mutant, hashie | 0.008 s | 0.092 s |
942
- | Peak memory, hashie (summed process RSS) | 595 MB | 471 MB |
943
- | Choice of operators, CI features | fixed set, fewer | 22 operators, baseline, HTML, GitHub Action |
944
-
945
- Full results, method and limitations:
946
- [benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
947
- Scripts and raw data: [bench/vs-mutineer](bench/vs-mutineer). The memory row sums
948
- per-process RSS, which counts shared pages more than once, and it was measured
949
- before the fork-memory work. Read it as history, not a ranking of physical
950
- memory.
951
-
952
- ### Independent review, saboteur 0.4.0 vs mutineer 1.6.0
953
-
954
- A private consultant review ("saboteur 0.4.0 - Deep Review", 12 October 2026,
955
- not published) ran saboteur 0.4.0 at 38b1834 and mutineer 1.6.0 at a766c96 on
956
- the same machine and projects: Linux x86_64, 1 vCPU, 4 GB, Ruby 3.4.5, Prism
957
- 1.2.0, RSpec 3.13, no Rails or sqlite3. mutineer ran with its default 5
958
- operators and with all 20. The project and timing rows are one run each. The
959
- rows describe saboteur at 38b1834; several of its findings have been fixed on
960
- main since. The rerun that replaces this table is still pending.
242
+ An engine gem whose specs live under `spec/dummy` needs an exclude for them; see
243
+ [Rails engines and dummy apps](docs/guides/rails-engines-and-dummy-apps.md).
961
244
 
962
- | Measure | saboteur 0.4.0 | mutineer 1.6.0 |
963
- |---|---|---|
964
- | Snippets project (11 planted gaps) | 177 mutants, 14 s, 10/11 gaps found, 9 noise survivors | Default: 56 mutants, 18 s, about 5/11. All operators: 191 mutants, 51 s, 10/11, 8 noise and duplicates |
965
- | Dependency project (A tested only via B) | 239 mutants, 6 s | 254 mutants, 26 s (needed hand-ordered source files) |
966
- | dry-inflector (real gem) | 551 mutants, 24 survivors, 95.6 %, 133 s | 480 mutants, 47 survivors (31 distinct), 90.1 %, 134 s |
967
- | Test selection | Whole suite, per example, automatic | Only files passed with --test, per file |
968
- | Kill verification | Yes (re-runs killers on unchanged code) | No |
969
- | sorbet sig methods | Skipped at 0.4.0 | Mutated, 24/24 killed |
970
- | Data.define / Struct.new block methods | Absent at 0.4.0 | Mutated |
971
- | Load-time code | False survivors at 0.4.0 | ran_at_load status |
972
- | Endless defs | Tested ones reported as no coverage at 0.4.0 | Tested ones work; untested ones show as survived |
973
- | Order-dependent specs | Smell exposed, bogus kills rejected | Hidden (whole file runs) |
974
- | Destructive mutant | Project wiped except .saboteur | Project wiped |
975
- | Baseline stability | Race at 0.4.0: 4 of 30 green runs failed (13 %), 5 of 25 under CPU contention (20 %) | No failures observed (15/15) |
245
+ ## SimpleCov
976
246
 
977
- Not in the review and still pending: a rerun on current main, memory and
978
- per-mutant time on the benchmark apps, and any mutineer release after 1.6.0.
247
+ saboteur needs Ruby's `Coverage` for itself, so leave SimpleCov off for
248
+ saboteur runs - for example, start it only when `ENV["COVERAGE"]` is set.
249
+ The [recipes](docs/recipes/README.md) cover this and VCR, WebMock and
250
+ Sidekiq.
979
251
 
980
- ### Memory after the fork-memory work (hashie @ 9c81e83)
252
+ <a id="dated-validation-runs"></a>
253
+ <a id="where-it-has-run"></a>
254
+ ## Benchmarks and validation
981
255
 
982
- Paired A/B runs against the pre-PR commit (32c0f21), 5 pairs per job level with
983
- AB/BA alternation and cold caches, on a 2 vCPU / 2 GB shared sandbox. Peak
984
- memory is per-process-tree PSS from /proc/smaps_rollup sampled at 0.3s. Verdict
985
- parity: **0 verdict differences in 23,700 mutant outcomes** (1,580 mutants x 15
986
- pairs); kill reasons (assertion vs crash class) vary run to run on both
987
- variants, as before.
256
+ Speed and memory numbers live with the harness and the raw data that
257
+ reproduce them, framed as dated measurements rather than standing claims:
258
+ the [benchmarks page](https://vamsipavanmahesh.github.io/saboteur/benchmark/)
259
+ on the site and [docs/benchmark](docs/benchmark/) in this repo.
988
260
 
989
- | jobs | peak PSS baseline (median, range) | peak PSS PR (median, range) | shared-memory delta |
990
- |------|-----------------------------------|-----------------------------|---------------------|
991
- | 1 | 101 MB (101-101) | 100 MB (100-102) | ~0 (no workers to share with) |
992
- | 2 | 133 MB (116-134) | 109 MB (106-121) | -18% |
993
- | 6 | 235 MB (192-269) | 196 MB (175-201) | -17% |
261
+ ## Project
994
262
 
995
- Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not
996
- touch less memory, it keeps more of it shared between workers after fork, which
997
- is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers on a
998
- 2 GB sandbox. Swap was not measured.
263
+ - [Changelog](CHANGELOG.md)
264
+ - [All documentation](docs/README.md) - configuration, operators, CI, recipes
265
+ - [Contributing](CONTRIBUTING.md) - every commit needs a DCO `Signed-off-by` line
266
+ - [Design notes](docs/design.md)
999
267
 
1000
- Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs
1001
- 32s at jobs=6. These are noisy per-cell minima from historical runs and do not
1002
- show a direction either way. The shared sandbox had multi-minute CPU-starvation
1003
- windows during this session (walls up to 594s); affected runs are flagged via
1004
- sampler cadence and excluded from wall analysis. Absolute times are specific to
1005
- this 2-core sandbox. Environment, method and the contention log are on the
1006
- [benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
268
+ ## Disclaimer
1007
269
 
1008
- ## 🗺️ Where it is going
1009
-
1010
- [ROADMAP.md](ROADMAP.md) lays out the plan: CI-fast runs, trusted results, a
1011
- team workflow (GitHub Action, baseline, HTML report), then 1.0. saboteur stays
1012
- RSpec-only. Design
1013
- details are in [docs/design.md](docs/design.md), prior art in
1014
- [docs/prior-art.md](docs/prior-art.md), and project status in
1015
- [docs/status.md](docs/status.md).
1016
-
1017
- ## 🤝 Contributing
1018
-
1019
- See [CONTRIBUTING.md](CONTRIBUTING.md). Commits need a DCO `Signed-off-by` line.
1020
-
1021
- Before you ask for review, run the local preflight check on your exact head:
1022
-
1023
- ```sh
1024
- ruby script/preflight --base <full main sha> --head <full head sha> --base-ref main --mode full
1025
- ```
1026
-
1027
- It runs scope, sign-off, SPDX, full RuboCop and the specs, and writes a report under `tmp/preflight/`. A pass is not an approval; fast mode is partial. See [docs/preflight.md](docs/preflight.md).
1028
-
1029
- ## Acknowledgements
1030
-
1031
- Thanks to [Glomo](https://glomo.one) for providing the AI tokens used to build saboteur.
270
+ saboteur is provided "as is", without warranty of any kind, express or
271
+ implied. In no event shall the authors or copyright holders be liable
272
+ for any claim, damages or other liability arising from its use -
273
+ including damage caused by mutating your code or by running unusual
274
+ DSLs and metaprogramming-heavy code. Mutation testing runs your test
275
+ suite against deliberately broken code; run it on a clean, committed
276
+ tree or a CI checkout. The full text is in [LICENSE](LICENSE).
1032
277
 
1033
278
  ## License
1034
279
 
1035
- MIT. Copyright (c) 2026 Vamsi Pavan Mahesh Gunturu. See [LICENSE](LICENSE).
1036
-
1037
- Internal baseline capture blocks a subject with empty generation, including a subject for which the builtin source yields no candidates. Context comparison is an extra consistency check; file fingerprint verification normally also catches source drift. Neither check proves external state stability.
1038
-
1039
- ### Opt-in survivor baseline
1040
-
1041
- `baseline check` runs every selected mutant fresh, then compares accepted survivors. It does not skip accepted mutants, change verdicts or change scores. Use whole configured projects or `--subject` filters; diff, replay, incremental reuse and score thresholds are refused. Caps, uncovered work, empty generation, errors, resource limits, omitted malformed files, changed inputs and missing results are incomplete, even with zero survivors.
1042
-
1043
- ```sh
1044
- saboteur baseline propose --baseline baseline.json --proposal proposal.json
1045
- # Edit only document.accepted[*].reason in proposal.json. Give each survivor a review reason.
1046
- saboteur baseline accept --baseline baseline.json --proposal proposal.json
1047
- saboteur baseline check --baseline baseline.json
1048
- ```
1049
-
1050
- Proposal writes independent owner-private execution evidence under the configured cache directory. Keep that directory until acceptance; deleting it prevents acceptance. Acceptance verifies the private record and current input fingerprint before atomically publishing the manifest. For replacement, give both propose and accept `--previous SHA256`, the hash you reviewed of the current baseline bytes. Output files must not overlap project inputs or the private evidence directory. A baseline normally lives in a separate JSON file outside fixture/config input trees.
1051
-
1052
- Check prints one JSON object: accepted, new, changed, resolved_kills, detected_timeouts, unmatched and errors are separate categories. Exit 0 means a compatible complete clean check; 1 needs survivor review; 2 means incompatible or invalid configuration; 3 means incomplete execution or tool failure. Propose and accept exit 0 only after their file operation succeeds. Context changes require fresh review. Timeouts remain a distinct detected category, not assertion kills.
1053
-
1054
- Score-drop gate: `saboteur baseline check --baseline baseline.json --epsilon POINTS` also fails (exit 1,
1055
- state `review_required`) when the fresh score is more than POINTS percentage points below the score recorded in
1056
- the baseline. Both scores use saboteur's own formula, (killed + timeout) / (killed + timeout + survived), over
1057
- every candidate. The baseline already stores each candidate's status, so no new file format is needed. This
1058
- catches what the survivor comparison cannot: a score that falls because killed mutants disappeared (deleted or
1059
- rewritten code) while every accepted survivor is still matched. A drop of exactly POINTS passes; a rise never
1060
- fails. The check prints a `score` object (`baseline`, `current`, `drop`, `epsilon`, `exceeded`) only when
1061
- `--epsilon` is given. Without it, behavior is unchanged. POINTS is 0 to 100. It lives on `baseline check`,
1062
- not on `run`: the delta gate needs the baseline's fresh whole-scope evidence, which `run` does not produce.
1063
- It is not the same as `--threshold`, which gates an absolute score.
1064
-
1065
- In GitHub Actions, the action's `baseline` and `baseline-epsilon` inputs run this check; see
1066
- [docs/github-action.md](docs/github-action.md).
1067
-
1068
- Local evidence protects against edits to the public proposal except review reasons. It is not remote attestation, a signature, or protection against the account owner changing their private files. File fingerprints cannot prove external database, network or environment stability. `config/saboteur_ignore.rb` remains a separate legacy ignore mechanism; this gate compares every survivor regardless of ignore status.
1069
-
1070
- A baseline that changes during `check` is rejected as configuration/evidence error (exit 2), not an incomplete execution result. Acceptance leaves a private `baseline.json.lock` beside the destination: keep it for cooperating-process publication locks. It contains no evidence or results. Symlinked checkout parents are normalized; symlink artifacts themselves and private-store/lock symlinks are still refused.
1071
-
1072
- The builtin `endpoint_replacement` operator swaps explicit, no-argument `first`/`last` calls. It targets wrong-end/order-selection faults in collection handling. Argument-bearing slices, bare calls and block-bearing APIs are left unchanged. Receivers are not statically typed: a custom object may implement either name differently, so a crash remains a crash kill, not proof of a collection assertion.
1073
-
1074
- JSON findings are ordered by a deterministic [review ranking](docs/survivor-ranking.md): coarse first, then covering-test count, edited-line impact and changed-line proximity, with file/line/digest ties. The terminal groups survivors by file, method and line first; the ranking only orders the survivors on one line, and every survivor is listed, in review mode too. Only consecutive survivors with matching evidence share a test/evidence block; repeated evidence after another group is printed again to keep that order. Ranking does not change scores or execution.
1075
-
1076
- Cache and baseline identities include the policy. Re-propose and review old survivor baselines with `baseline propose`; never auto-accept new survivors. Mutation digests for unchanged edits stay stable.
1077
-
1078
- ### Preview and choose operators
1079
-
1080
- `bundle exec saboteur run --dry-run` discovers source methods and generates candidates without booting the app, running tests, forking, or writing reports/caches. It honors `--since`, `--subject` and source arguments. It prints per-method counts and cap omissions. After a matching normal run, a cached estimate shows baseline covering-test work (cached original-test durations times planned mutants; it is not a bound on mutant runtime and excludes boot and fork costs). If any file cannot be parsed, the total is marked INCOMPLETE and no estimate is given.
1081
-
1082
- Trust boundary: `--dry-run` never boots the app or runs its tests, but custom `sources` entries (`path.rb:ClassName`) are still `require`d to generate candidates, so their code runs. Only preview with sources you trust.
1083
-
1084
- `bundle exec saboteur run --list-operators` lists every builtin name and description. All builtin operators run by default, including [`range`](docs/operators/range.md), [`operator_assignment`](docs/operators/operator_assignment.md), [`condition_negation`](docs/operators/condition_negation.md), [`method_swap`](docs/operators/method_swap.md), [`operand_replacement`](docs/operators/operand_replacement.md), and [`interpolation_removal`](docs/operators/interpolation_removal.md). `--operators arithmetic,comparison` or `c.operators = ["arithmetic", "comparison"]` still selects a subset; `all` selects every name. There is no opt-in tier. `--exclude-operators range,condition_negation` or `c.exclude_operators = ["range", "condition_negation"]` removes named operators from that selection; exclusion wins if a name is in both lists. An empty selection means all, while an empty exclusion means none. CLI lists replace their corresponding config lists. These controls apply to builtin operators, not custom source plugins. Selection and exclusions invalidate cached verdict identity and are carried into agent rerun/replay commands.
1085
-
1086
-
1087
- All operators run unless you select a smaller set; exclusions always remove operators from that set.
1088
- Repeated `--operators` or `--exclude-operators` flags use the last list, not a union.
1089
- To re-enable an operator excluded in config: `bundle exec saboteur run --operators range --exclude-operators ''`.
1090
- To keep the 0.5.0 default inventory: `bundle exec saboteur run --exclude-operators range,operator_assignment,condition_negation`.
1091
- The old `core` and `extended` names now give a migration error, not a changed selection in disguise.
1092
- An empty effective selection is a usage error before boot. `--list-operators` loads current config and shows on/off plus examples; preview and the run header show the effective set.
1093
-
1094
- ```ruby
1095
- Saboteur.configure do |c|
1096
- c.operators = [] # all by default; or ["arithmetic", "range"] for a subset
1097
- c.exclude_operators = ["condition_negation"]
1098
- end
1099
- ```
1100
-
1101
- Inspired by mutineer's --dry-run, --list-operators and --operators tiers (github.com/davidteren/mutineer, MIT, by David Teren).
1102
-
1103
- ### Agent survivor hints and score history
1104
-
1105
- `--agent` findings include operator-specific `hint` suggestions and a shared
1106
- `hintCaveat`. Hints do not diagnose a missing test or assertion; equivalent
1107
- mutants and limited observation are still possible. Unknown operators have no
1108
- hint unless marked coarse. The normal JSON byte budget still applies.
1109
-
1110
- Use `--score-history PATH` (or `c.score_history = PATH` in your config) to append
1111
- one score-history JSONL record per finished report. History is off by default;
1112
- changes are printed to stderr, not agent stdout. See [score definitions and
1113
- history limits](docs/scores.md). The file can be uploaded by your own artifact
1114
- step; the bundled Action's upload behavior is unchanged.
1115
-
1116
- Score history now refuses non-regular files, symlink paths and report/history
1117
- inode aliases before running the engine or writing reports. Imported records
1118
- have a 64 KiB per-line bound and strict validation of supported version 1
1119
- and version 2 records, including counts, scores and duplicate keys. History is an output, excluded from baseline
1120
- fingerprints. See [history safety and comparison limits](docs/scores.md).
1121
-
1122
- Operator policy version 2, selected names, excluded names and effective names are in JSON/agent/baseline evidence and fresh score-history v2 rows. Old history v1 rows remain readable but are not compared across policy versions.
1123
- ### Inline suppression comments
1124
-
1125
- `total = a + b # saboteur:disable-line arithmetic reason="rounding checked elsewhere"` suppresses edits starting on that line, restricted to the named operators. Omit names to suppress all operators. Multiple names are space-separated. Only actual Ruby comments count; directive text in strings does not. A missing or blank reason or unknown operator is a usage error naming file:line. Terminal, JSON and HTML list every suppressed candidate with its reason and digest, outside the score. Moving the commented code keeps the suppression. Multi-line edits use their starting line.
1126
-
1127
- Inspired by mutineer's inline suppression comments (github.com/davidteren/mutineer, MIT, by David Teren).
1128
-
1129
- Inline suppressions bound completeness: classic scores still exclude suppressed candidates, but baseline evidence is incomplete when any inline candidate is suppressed. Agent output reports `inline_suppressed` with its count and points to the full audit. Reasons are audited, not proof that suppressed work was checked.
1130
-
1131
- Source-only `--dry-run` lists the suppressed count and each audit row in selected methods, before diff restriction and the per-method cap. They are outside the planned total. Preview still does not boot the app or write reports.
280
+ [MIT](LICENSE).