saboteur 0.6.0 β†’ 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (133) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +1024 -0
  3. data/README.md +199 -952
  4. data/exe/saboteur +34 -7
  5. data/lib/saboteur/baseline/workflow.rb +2 -1
  6. data/lib/saboteur/baseline_health.rb +40 -6
  7. data/lib/saboteur/cli/exit_status.rb +80 -4
  8. data/lib/saboteur/cli/help_options.rb +17 -0
  9. data/lib/saboteur/cli/history_command.rb +135 -0
  10. data/lib/saboteur/cli/presets.rb +40 -0
  11. data/lib/saboteur/cli/run_budget_options.rb +37 -0
  12. data/lib/saboteur/cli/run_reporting.rb +74 -5
  13. data/lib/saboteur/cli/selection_options.rb +31 -0
  14. data/lib/saboteur/cli/shard_options.rb +17 -0
  15. data/lib/saboteur/cli/source_set.rb +53 -0
  16. data/lib/saboteur/cli/survivors_command.rb +157 -0
  17. data/lib/saboteur/cli.rb +136 -98
  18. data/lib/saboteur/config.rb +163 -15
  19. data/lib/saboteur/config_file.rb +8 -0
  20. data/lib/saboteur/constant_reads.rb +182 -0
  21. data/lib/saboteur/constant_subject.rb +70 -0
  22. data/lib/saboteur/constant_subject_finder.rb +281 -0
  23. data/lib/saboteur/coverage_reads.rb +29 -0
  24. data/lib/saboteur/crash_report.rb +154 -0
  25. data/lib/saboteur/custom_map.rb +3 -1
  26. data/lib/saboteur/dirty_tree_warning.rb +128 -0
  27. data/lib/saboteur/doctor.rb +49 -1
  28. data/lib/saboteur/engine/baseline_guard.rb +136 -14
  29. data/lib/saboteur/engine/boot.rb +16 -1
  30. data/lib/saboteur/engine/canary.rb +41 -10
  31. data/lib/saboteur/engine/candidate_selection.rb +28 -1
  32. data/lib/saboteur/engine/candidates.rb +43 -3
  33. data/lib/saboteur/engine/coverage_guard.rb +29 -0
  34. data/lib/saboteur/engine/coverage_store.rb +79 -15
  35. data/lib/saboteur/engine/defined_order_prefix.rb +43 -0
  36. data/lib/saboteur/engine/diagnostics.rb +41 -2
  37. data/lib/saboteur/engine/diff_scope.rb +24 -5
  38. data/lib/saboteur/engine/distribution.rb +134 -8
  39. data/lib/saboteur/engine/early_stop_recording.rb +47 -0
  40. data/lib/saboteur/engine/gated_run.rb +35 -0
  41. data/lib/saboteur/engine/load_coverage.rb +45 -0
  42. data/lib/saboteur/engine/out_of_scope_guard.rb +21 -0
  43. data/lib/saboteur/engine/pipelines.rb +107 -0
  44. data/lib/saboteur/engine/result_cache.rb +67 -9
  45. data/lib/saboteur/engine/run_budget.rb +90 -0
  46. data/lib/saboteur/engine/run_fingerprint.rb +4 -0
  47. data/lib/saboteur/engine/sandbox_policy.rb +56 -11
  48. data/lib/saboteur/engine/scheduler.rb +222 -20
  49. data/lib/saboteur/engine/selection_announcement.rb +54 -0
  50. data/lib/saboteur/engine/shard.rb +69 -0
  51. data/lib/saboteur/engine/subject_selection.rb +373 -19
  52. data/lib/saboteur/engine/suite_files.rb +3 -1
  53. data/lib/saboteur/engine/test_guess.rb +438 -25
  54. data/lib/saboteur/engine/thread_warnings.rb +53 -0
  55. data/lib/saboteur/engine/unchanged_syntax.rb +17 -2
  56. data/lib/saboteur/engine/unscoped_suite_warning.rb +41 -0
  57. data/lib/saboteur/engine/verdict_check.rb +54 -18
  58. data/lib/saboteur/engine.rb +202 -158
  59. data/lib/saboteur/execution/external_runner.rb +471 -0
  60. data/lib/saboteur/execution/marker_sweep.rb +79 -0
  61. data/lib/saboteur/execution/peak_memory.rb +28 -0
  62. data/lib/saboteur/execution/process_table.rb +12 -0
  63. data/lib/saboteur/execution/process_tree.rb +65 -5
  64. data/lib/saboteur/execution/protocol.rb +2 -1
  65. data/lib/saboteur/execution/reevaluator.rb +68 -12
  66. data/lib/saboteur/execution/runner.rb +225 -24
  67. data/lib/saboteur/execution/sandbox.rb +21 -3
  68. data/lib/saboteur/execution/seatbelt.rb +226 -0
  69. data/lib/saboteur/execution/supervisor.rb +45 -13
  70. data/lib/saboteur/execution/worker_pool.rb +110 -8
  71. data/lib/saboteur/frameworks/sqlite_memory_database.rb +49 -0
  72. data/lib/saboteur/gates.rb +43 -2
  73. data/lib/saboteur/help.rb +108 -13
  74. data/lib/saboteur/ignore_list.rb +40 -7
  75. data/lib/saboteur/install.rb +4 -3
  76. data/lib/saboteur/integration/base.rb +7 -2
  77. data/lib/saboteur/integration/rspec.rb +175 -9
  78. data/lib/saboteur/mutation_source/argument_removal.rb +1 -1
  79. data/lib/saboteur/mutation_source/builtin.rb +44 -14
  80. data/lib/saboteur/mutation_source/chain_removal.rb +1 -1
  81. data/lib/saboteur/mutation_source/condition_negation.rb +1 -1
  82. data/lib/saboteur/mutation_source/constant_literals.rb +66 -0
  83. data/lib/saboteur/mutation_source/element_removal.rb +73 -0
  84. data/lib/saboteur/mutation_source/endpoint_replacement.rb +1 -1
  85. data/lib/saboteur/mutation_source/entry_edits.rb +44 -0
  86. data/lib/saboteur/mutation_source/filter_replacement.rb +1 -1
  87. data/lib/saboteur/mutation_source/hazard_guard.rb +129 -3
  88. data/lib/saboteur/mutation_source/interpolation_removal.rb +41 -0
  89. data/lib/saboteur/mutation_source/literal_replacement.rb +1 -1
  90. data/lib/saboteur/mutation_source/method_swap.rb +193 -0
  91. data/lib/saboteur/mutation_source/mutation.rb +5 -3
  92. data/lib/saboteur/mutation_source/operand_replacement.rb +54 -0
  93. data/lib/saboteur/mutation_source/operator_assignment.rb +1 -1
  94. data/lib/saboteur/mutation_source/operator_examples.rb +6 -1
  95. data/lib/saboteur/mutation_source/operator_registry.rb +7 -2
  96. data/lib/saboteur/mutation_source/operator_visit.rb +39 -0
  97. data/lib/saboteur/mutation_source/pair_removal.rb +13 -31
  98. data/lib/saboteur/mutation_source/range_replacement.rb +1 -1
  99. data/lib/saboteur/mutation_source/regex_replacement.rb +93 -0
  100. data/lib/saboteur/preview.rb +122 -13
  101. data/lib/saboteur/refinement_activations.rb +60 -0
  102. data/lib/saboteur/reporter/agent.rb +9 -1
  103. data/lib/saboteur/reporter/agent_commands.rb +47 -15
  104. data/lib/saboteur/reporter/agent_incomplete.rb +38 -2
  105. data/lib/saboteur/reporter/agent_summary.rb +4 -3
  106. data/lib/saboteur/reporter/findings.rb +5 -2
  107. data/lib/saboteur/reporter/history_policy.rb +11 -0
  108. data/lib/saboteur/reporter/history_reader.rb +47 -0
  109. data/lib/saboteur/reporter/html.rb +26 -4
  110. data/lib/saboteur/reporter/mutation_text.rb +6 -0
  111. data/lib/saboteur/reporter/next_steps.rb +14 -4
  112. data/lib/saboteur/reporter/score_history.rb +5 -1
  113. data/lib/saboteur/reporter/stryker.rb +40 -7
  114. data/lib/saboteur/reporter/survivor_classification.rb +101 -0
  115. data/lib/saboteur/reporter/survivor_hints.rb +3 -0
  116. data/lib/saboteur/reporter/survivor_list.rb +15 -9
  117. data/lib/saboteur/reporter/terminal.rb +74 -17
  118. data/lib/saboteur/result.rb +38 -7
  119. data/lib/saboteur/selection/coverage_map.rb +107 -11
  120. data/lib/saboteur/selection/dependency_shield.rb +80 -0
  121. data/lib/saboteur/selection/external_test_map.rb +27 -0
  122. data/lib/saboteur/selection/git_diff.rb +105 -13
  123. data/lib/saboteur/selection/layout_only_lines.rb +80 -0
  124. data/lib/saboteur/selection/recorder.rb +22 -4
  125. data/lib/saboteur/selection/source_scope.rb +22 -0
  126. data/lib/saboteur/selection/untracked_changes.rb +126 -0
  127. data/lib/saboteur/subject.rb +53 -7
  128. data/lib/saboteur/subject_finder.rb +173 -29
  129. data/lib/saboteur/text.rb +42 -0
  130. data/lib/saboteur/version.rb +1 -1
  131. data/lib/saboteur/working_tree.rb +38 -15
  132. data/lib/saboteur.rb +29 -1
  133. metadata +50 -2
data/README.md CHANGED
@@ -10,996 +10,243 @@
10
10
  [![Gem Version](https://img.shields.io/gem/v/saboteur)](https://rubygems.org/gems/saboteur)
11
11
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
12
12
 
13
- Your suite is green and coverage says 95%. That tells you the tests *ran* your
14
- code. It does not tell you they would *notice* if the code were wrong.
13
+ Line coverage tells you which code ran. It does not tell you if a test
14
+ would notice when that code breaks. Mutation testing does.
15
15
 
16
- saboteur finds out. It makes small deliberate changes to your methods (flips a
17
- `<` to `<=`, swaps `+` for `-`, replaces a body with `nil`), runs the tests that
18
- cover each change, and reports every change your suite let through. Each of
19
- those survivors can point at a missing test. It can also be an equivalent
20
- change that no test could notice, a flaky test, or a limit of the tool, so
21
- each one needs a human look.
16
+ saboteur makes one small change to your code at a time and runs only the
17
+ tests that touch it, so it is fast. If the tests still pass, you see the
18
+ exact change they missed. Every kill is re-checked against the unchanged
19
+ code, so a broken suite never counts as a kill.
22
20
 
23
- ```
24
- $ bundle exec saboteur run --since origin/main
25
- ```
21
+ **Docs, benchmarks and guides live on the site:
22
+ [vamsipavanmahesh.github.io/saboteur](https://vamsipavanmahesh.github.io/saboteur/)**
26
23
 
27
- Never heard of mutation testing? It asks "if I broke this, would a test
28
- fail?" for many small breakages and lists the ones nobody noticed. Read the
29
- [design notes](docs/design.md) when you want the details. For adopting it on a team, see the
30
- [best practices guide](docs/best-practices.md).
31
-
32
- Declared external inputs are checked against cache and report paths using their
33
- physical paths, including aliased parent directories. An overlap makes the run
34
- fingerprint incomplete, so it cannot authorize cache reuse.
35
-
36
- Internal run reports retain typed candidate-drop and cap evidence for gate
37
- validation. These records do not change selection, scores, exit codes or the
38
- public JSON format. Empty generation is unverified, not proof of equivalence.
39
-
40
- The internal engine can opt in to fresh baseline evidence capture. It records
41
- inputs before boot, planned candidate identities before scheduling and reconciled
42
- terminal results after execution. Caps, uncovered work, invalid or empty
43
- production, reused outcomes and changed inputs make that evidence incomplete.
44
- The public opt-in `baseline propose`, `baseline accept` and `baseline check`
45
- commands use this evidence to review and compare survivors. They never skip
46
- accepted mutants or change their scores. See [Opt-in survivor baseline](#opt-in-survivor-baseline).
47
-
48
- ### The unchanged suite is checked first
49
-
50
- Every run with selected subjects runs the loaded unchanged suite twice (the specs chosen with `--test`, if given) in fresh
51
- supervised children before coverage lookup or incremental verdict reuse. This
52
- costs two suite runs even on a cache hit, plus a coverage-building suite run on
53
- a cache miss. Both baseline passes must be green, complete and in the same
54
- RSpec order at the run seed. A failure exits 3, stops mutation, prints the failing example ids and the first ten lines of the
55
- failure inline, and names the log and fix. Long phases
56
- print their elapsed time about every 30 seconds, with the examples finished so far, the percentage and a rough estimate of the time left (it assumes equally long examples) once the suite child has logged its first count, and each phase's log holds
57
- the suite's progress and full failure detail. Each suite pass
58
- uses `coverage_timeout` and configured memory/process limits (the suite memory
59
- budget is four times `max_memory_mb`, as for coverage). A suite child that has
60
- already reported its result gets up to 5 seconds to finish exiting, under the
61
- same deadline and memory/process limits as before; if it is still running then,
62
- saboteur stops it and keeps the result, so a slow exit is not reported as a
63
- failure. A child that ends by itself in that window keeps its own exit status. A child killed by `SIGKILL` is called a possible
64
- out-of-memory kill only when the process tree was seen growing toward the
65
- memory limit. Older benchmark numbers
66
- need a rerun before they describe this extra fixed cost.
67
- A whole-suite coverage build that records no lines for any included file
68
- means the setup hid the app from Coverage (Bootsnap's compile cache is the
69
- common cause), so the run fails with exit 3 instead of testing nothing, and
70
- the empty map is not cached. With `--test` an empty map instead stays a
71
- legitimate "no coverage" outcome, because the chosen specs may not touch the
72
- selected sources.
73
- Two passes detect observed instability, not every possible flaky test. See
74
- [baseline stability](docs/baseline-stability.md) for limits and hook behaviour.
75
-
76
- ### Hazardous calls are not mutated, and a canary watches the tree
77
-
78
- Mutants run in your checkout, so a mutant inside a destructive call could do
79
- real damage: a string mutation inside `FileUtils.rm_rf(File.join(Dir.pwd,
80
- "build"))` once turned the target into the working directory itself. The
81
- hazard guard is a syntactic filter. It recognizes destructive calls written
82
- as direct calls: `FileUtils.rm*`/`remove_entry*` (also as bare names under
83
- `include FileUtils`), `File.delete`/`unlink`, `Dir.rmdir`/`delete`/`unlink`,
84
- `rmtree`, `system`/`exec`/`spawn` (bare or on `Kernel`/`Process`), backticks
85
- and `%x`, `IO.popen`, any `Open3` method, and `open` with a leading `|`. It
86
- suppresses any mutant that would leave such a call in place while altering
87
- code inside it or anywhere else in the same method. A mutant that removes
88
- every hazardous call in the method still runs, provided its replacement
89
- introduces no new hazard. The match is by name on the written call: calls
90
- reached through aliases, variables, `send` or other indirection are not
91
- recognized, so the guard cannot promise to find every destructive call. The
92
- summary reports how many mutants were suppressed. Extend or replace the list
93
- with `c.hazard` (same tag shapes: `"FileUtils.rm_rf"`, or a bare name like
94
- `"system"`); `--allow-hazardous` runs the suppressed mutants anyway.
95
-
96
- Behind the guard, a canary snapshots the tracked files (or every file,
97
- outside a repository) before mutating and re-checks after each method's
98
- mutants. If anything vanished, the run aborts with exit 3 and names the most
99
- recent mutant's digest. That check happens after the damage and restores
100
- nothing; it limits how far a bad mutant gets. Recover from your own backup,
101
- a fresh checkout or a reclone - a deleted `.git` or uncommitted work is gone
102
- unless you have another copy. The canary is the backstop for the destructive
103
- forms the syntactic guard cannot see.
104
-
105
- On Linux 6.2 and later (Landlock ABI 3) each mutant child also runs under
106
- a Landlock sandbox: the child cannot create, write, rename, unlink or
107
- truncate files in the project tree, except `tmp/`, the cache dir and the
108
- system temp dir, plus anything in `c.sandbox_writable`. Every other path
109
- is denied for writing, `/dev/shm` and `/dev/tty` included; `/dev/null`
110
- stays writable. Inherited
111
- descriptors already open for writing outside the granted dirs are closed
112
- before the sandbox engages - stdio redirected to a regular file and files
113
- already deleted from the tree included (stdio on a terminal, pipe or
114
- `/dev/null` stays), and a descriptor whose target or flags cannot be
115
- inspected is closed too rather than trusted. A `c.sandbox_writable` grant
116
- that contains the project root or another granted dir warns, since it
117
- defeats or duplicates the protection; reads and process execution are
118
- unaffected.
119
- This DOES change what your tests can write: a suite that writes elsewhere
120
- in the project (`db/`, `log/`, ...) fails with EACCES in mutant children -
121
- grant that directory with `c.sandbox_writable` or opt out with
122
- `c.sandbox = false`. Landlock covers the filesystem only - network,
123
- sockets and IPC are never restricted - and a checkout inside the system
124
- temp dir cannot be protected, because everything under it stays writable
125
- so that real test suites keep working - the run warns when the effective
126
- write set (every grant realpath'd, the cache dir and the system temp dir
127
- included) contains the project root, however it happened: a checkout
128
- inside the system temp dir, `cache_dir = "."`, a symlinked `tmp`, or an
129
- overlapping `c.sandbox_writable` grant. `c.sandbox = "strict"` refuses to
130
- run at all in that situation, since the sandbox would protect nothing.
131
- The sandbox applies to every mutant
132
- child and fails closed: a child that cannot restrict itself reports an
133
- error rather than run, and a kernel probe that errors (as opposed to
134
- cleanly reporting Landlock absent) refuses the run.
135
-
136
- On kernels from 5.13 to 6.1 (Landlock ABI 1-2) `truncate()` cannot be
137
- restricted, so the tree is not fully read-only and the claim above does
138
- not hold; the run notes once that children run with the guard and canary
139
- only, exactly as on kernels without Landlock - and as on non-glibc Linux
140
- whose C library cannot be located (musl is tried by soname first).
141
- `c.sandbox = "strict"` turns that degraded note into a refusal;
142
- `c.sandbox = false` opts out.
143
-
144
-
145
- ## πŸš€ Try it in a minute
146
-
147
- Add saboteur to the test group of your project's Gemfile (Ruby 3.3+, RSpec):
24
+ <a id="60-second-quick-start"></a>
25
+ ## 30-second quick start
148
26
 
149
- ```ruby
150
- gem "saboteur", group: :test
151
- ```
27
+ You need Ruby 3.3 or newer, RSpec, and Linux or macOS with `fork`.
152
28
 
153
29
  ```sh
154
- bundle install
155
- bundle exec saboteur run --subject 'YourClass#your_method'
156
- ```
30
+ # Install the gem:
31
+ gem install saboteur
157
32
 
158
- Outside a Bundler project, `gem install saboteur` installs the same executable.
159
- Rails app? Follow the [Rails quickstart](#rails-apps) below: the defaults are for
160
- plain gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
161
-
162
- Start with one method. You get counts per status, a score, timing, and a
163
- unified diff for every survivor, so you can read the exact change your tests
164
- missed. The JSON report lands in `.saboteur/report.json`.
165
-
166
- πŸ“‹ Requirements: Ruby 3.3 or newer on Linux or macOS (it needs `fork`), and RSpec.
167
- Minitest is not supported; see [Minitest](docs/minitest.md) for why and what it would take.
168
-
169
- ## 🧠 Why it is built the way it is
170
-
171
- **It only runs tests that can matter.** Each mutant runs only the tests whose
172
- line coverage touched the changed statement. Tests that killed earlier mutants
173
- of the same method go first, then the fastest, and the first failure stops the
174
- run. A change no test reaches is reported as "no coverage", not as a survivor. Branch coverage separates arms on the same line: an edit inside an arm no test took reports "no coverage (branch not taken)". Ruby does not expose `&&`/`||` right operands as branches, so those keep line-based selection. `case/in` pattern edits use the enclosing case and clause execution lines; edits in an untested clause body still stay uncovered. Whole-body replacements use every test of the method, including methods that start with a heredoc. Endless defs (`def x = ...`) are matched to tests through per-test method call coverage, since line coverage never counts their bodies. Methods defined inside blocks (`Class.new do`, `Data.define do`, `included do`) are not mutated yet; the run lists them as skipped, "defined inside a block".
175
-
176
- **It runs coarse mutants first, then fine mutants.** Each method first gets
177
- whole-body mutants: `nil` (`true`/`false` for predicates) unless the body is empty or
178
- only logs, and `raise` unless a catch-all rescue would absorb it. Fine-grained mutants run
179
- regardless of the coarse verdict, including a survivor or a reused verdict.
180
- Deleting a body unnoticed does not prove its smaller changes are harmless.
181
- Subjects with no tests are skipped before any mutants are generated, and each
182
- method is capped at `max_mutants_per_method` mutants (100 by default). Running
183
- fine mutants after coarse survivors can increase run time and change scores.
184
-
185
- **It prepares memory before it forks.** Before workers fork, saboteur warms and
186
- compacts the Ruby heap ([`ForkReadiness`](lib/saboteur/execution/fork_readiness.rb)),
187
- which is meant to let workers share more pages with their parent and copy less.
188
- Observed on the pinned hashie workload (paired runs, below): about 17-18% lower peak PSS at 2 and 6 workers. Not a general guarantee.
189
-
190
- **It is aimed at pull requests.** `--since origin/main` mutates only the methods
191
- your branch changed: fine-grained mutants on the changed lines, plus the
192
- whole-body mutants of those methods. On one real change in saboteur's own history, that took a file
193
- from 139 mutants to 7, and median wall time fell 22.9% (measured at d2b7abb, before the coarse-survivor gate was removed; a rerun is pending). The time saving is
194
- smaller than the mutant count because building the coverage map is a fixed
195
- cost, and this is one file, not a Rails app, so it is not a general speed
196
- claim. The full method, raw timings and
197
- caveats are in [bench/since](bench/since/README.md).
198
-
199
- **It checks its own kills.** After a method's mutants run, saboteur re-runs the
200
- killing tests against the *unchanged* method. If they fail there too, the kill
201
- is not credited and becomes an error. In one comparison on lobsters, 0 of 20
202
- randomly sampled kills were fake. Every mutant runs in a supervised child with deadline,
203
- memory and process limits, and saboteur tracks and kills the processes it
204
- starts.
205
-
206
- **It is open.** The report is JSON in the Stryker mutation-testing-report
207
- schema, and you can add your own mutants through a small public interface.
208
-
209
- ## πŸ› οΈ Usage
33
+ # Or add it to the Gemfile:
34
+ bundle add saboteur --group test
210
35
 
211
- ```sh
212
- bundle exec saboteur run # whole project
213
- bundle exec saboteur doctor # environment checks only
214
- bundle exec saboteur doctor --suite --seed 42 # plus two unchanged suite checks
215
- bundle exec saboteur run --since origin/main # methods changed on this branch
216
- bundle exec saboteur run --subject 'Billing::Invoice#total'
217
- bundle exec saboteur run --subject Billing # a namespace
218
- bundle exec saboteur run --incremental # reuse verdicts whose inputs did not change
219
- bundle exec saboteur run --seed 1234 # repeat a run that used seed 1234
220
- bundle exec saboteur run lib/a.rb lib/b.rb --test spec/a_spec.rb --test spec/b_spec.rb
221
- bundle exec saboteur run --threshold 90 # exit 1 when the score is below 90 %
222
- bundle exec saboteur run --id DIGEST --seed 1234 # replay one full current mutant digest
223
- bundle exec saboteur run --fail-fast # stop after the first subject with a survivor
224
- bundle exec saboteur run --matrix # every covering test per mutant: blind and redundant tests
225
- bundle exec saboteur run --verbose --no-color # add score split, timings and kill causes; no colour
36
+ # Either way:
37
+ bundle exec saboteur run
38
+ ```
39
+
40
+ The first run builds a coverage map of which tests touch which code. The
41
+ map is cached and reused while your code and tests stay unchanged, so
42
+ later runs start fast.
43
+
44
+ On a branch, `bundle exec saboteur pr` mutates only the methods you
45
+ changed relative to `main`, with one baseline pass and a ten-minute
46
+ budget. If the budget stops the run, it exits 4 and prints `partial: X
47
+ of Y mutants not run` (or says the count is unknown when planning did
48
+ not finish). Pass `--max-duration-ok` to accept a partial verdict; the
49
+ warning stays, and other failure gates still apply. Pass `--since` for a
50
+ different base, for example `bundle exec saboteur pr --since master`.
51
+
52
+ Setup, configuration and CI:
53
+ [the quickstart](https://vamsipavanmahesh.github.io/saboteur/quickstart.html).
54
+
55
+ ## Why RSpec-only
56
+
57
+ A mutation verdict is only worth anything if the test selection behind it
58
+ is exact. RSpec's example structure lets saboteur map each method to the
59
+ examples that cover it, rerun just those against each change, and re-check
60
+ every kill against the code before the change. One framework, done
61
+ properly - that is the bet. Minitest is not supported today;
62
+ [why, and what it would take](docs/minitest.md).
63
+
64
+ ## What a run tells you
65
+
66
+ ```text
67
+ saboteur 0.7.0: 1 subjects, 10 mutants, wall 0.70 s, seed 1
68
+ killed 8 survived 2 timeout 0 resource limit 0 error 0 score 80.0 %
69
+ ```
70
+
71
+ Each survivor comes with its diff, the tests that ran, and a triage label
72
+ (`likely_equivalent`, `arid`, `missing_test`) to sort what to read first.
73
+ The labels are heuristics for reading order, never a diagnosis, and they
74
+ never change the score.
75
+
76
+ A run on a tree with uncommitted changes opens with a warning that splits
77
+ the count into changes inside the mutation scope, outside it, and
78
+ saboteur's own setup files. Mutants run your code, so run on a clean,
79
+ committed tree or a CI checkout.
80
+
81
+ Code that only runs while the app loads cannot be judged by replacing the
82
+ method later, so it reports as `ran at load`, outside the score. The same
83
+ holds for def-time macros - a `deprecate`-style helper, a `before_action`
84
+ block, any DSL whose only caller is a class body: the closure captured at
85
+ load keeps the pristine definition, so the boot's def-call counters mark
86
+ the macro itself `ran at load` instead of letting its mutants survive.
87
+
88
+ ## What you get
89
+
90
+ - **Fast pull-request runs.** `--since main` scopes the run to the methods
91
+ you changed, refuses (exit 2) when the resolved range has no changes -
92
+ `saboteur pr $GITHUB_SHA` on a checkout at the merge commit diffs the
93
+ merge against itself - and fails loudly (exit 3) when the changed code
94
+ cannot be verified - for example when every changed method is skipped
95
+ because a refinement is active where it sits. A changed method skipped
96
+ for no coverage names the specs the run loaded; when its coverage
97
+ lives in a sibling spec file, pass `--test spec/models` (the directory
98
+ holding it) to load it. The GitHub Action runs on every pull request
99
+ and caches the coverage map between runs. `--shard K/N` splits one
100
+ planned inventory deterministically across CI jobs; each shard reports
101
+ partial evidence.
102
+ - **Honest verdicts.** Every kill is verified against the unchanged
103
+ method; a kill the suite cannot confirm stays out of the score.
104
+ Suspect a flaky test? `--flaky-check survivors` confirms each new
105
+ survivor with one reordered rerun (`decisive` also confirms fresh
106
+ kills); a verdict the rerun disagrees with becomes `unverified`,
107
+ named with its tests, and stays out of the score. A spec
108
+ file whose examples depend on earlier ones is retried in defined order
109
+ before a kill is rejected, so its mutants are still scored.
110
+ - **Reports for people and tools.** Terminal, HTML, a Stryker-compatible
111
+ JSON report, and a stable agent JSON API with a deprecation policy.
112
+ - **A plan before the run.** `saboteur run --dry-run --format json` prints
113
+ the versioned [mutant inventory](docs/dry-run-json.md) - every planned
114
+ mutant with its stable digest and edits - without booting the suite.
115
+ - **A review workflow.** An ignore list with reasons and triage states,
116
+ an opt-in survivor baseline, score thresholds and score history.
117
+ `saboteur survivors .saboteur/report.json` browses survivors in the
118
+ terminal straight from the JSON report - list, filter
119
+ (`--file`/`--operator`/`--triage`/`--ranked`), and `--show DIGEST`
120
+ for the diff, the exact `run --id` replay command and an ignore-file
121
+ entry template. It never boots your project or edits the ignore file.
122
+ - **A local README badge.** `saboteur badge` writes Shields endpoint JSON
123
+ from the score history and `saboteur trend` prints the score delta -
124
+ no hosted service:
125
+
126
+ ```md
127
+ ![mutation score](https://img.shields.io/endpoint?url=<raw-url-of-the-badge-json>)
128
+ ```
129
+
130
+ Incomplete runs are refused as badge evidence unless `--mark-partial`
131
+ labels them, and the trend never compares runs with different
132
+ selection, inventory or operator policy.
133
+ - **Safety controls.** A hazard guard, canary values, and time, memory
134
+ and process-count limits on every mutant in the default in-process
135
+ mode; a Landlock sandbox on Linux
136
+ 6.2 and newer.
137
+ - **Extension points.** Add your own mutation operators through the
138
+ public interface.
139
+
140
+ Per-path score gates hold one subtree to a stricter floor: `c.score_gate
141
+ "app/payments/**/*", min_score: 95` in config/saboteur.rb (repeatable). Each
142
+ gate scores only the mutants under its pattern and fails the run under its
143
+ floor; a gate with nothing executed under it passes vacuously and says so.
144
+ See
145
+ [choosing and raising a threshold](docs/best-practices.md#choosing-and-raising-a-threshold)
146
+ for how to pick a number.
147
+
148
+ ## Honest limits
149
+
150
+ - saboteur is RSpec-only and needs `fork`: no Windows, no Minitest.
151
+ - A survivor is review evidence, not proof of a bug in your tests. It can
152
+ be a missing test, an equivalent change, a flaky test, or a limit of
153
+ the tool. Read it before you write a test for it.
154
+ - `--since` and `--incremental` are iteration aids. Run the whole project
155
+ on main and before a release.
156
+ - The hazard guard is syntactic, sandboxing varies by platform (Landlock
157
+ on Linux 6.2+; on macOS mutants run unsandboxed unless `c.sandbox = "seatbelt"`
158
+ opts into a probe-gated Seatbelt profile; see
159
+ [the macOS Seatbelt sandbox](docs/guides/macos-seatbelt-sandbox.md)), and network access is
160
+ never restricted. Stub real APIs and use a container when that matters.
161
+
162
+ ### Custom spec mapping
163
+
164
+ When the naming guess does not fit the project's layout, teach the mapping
165
+ once in `config/saboteur_custom_map.rb` instead of passing `--test` on every
166
+ run:
167
+
168
+ ```ruby
169
+ SABOTEUR_CUSTOM_MAP = {
170
+ "lib/aasm/core/state.rb" => ["spec/unit/state_spec.rb"],
171
+ "app/jobs/**/*.rb" => ["spec/jobs/**/*_spec.rb"],
172
+ %r{\Aapp/jobs/(.*)\.rb\z} => ['spec/jobs/\1_spec.rb'],
173
+ "app/generated/**/*.rb" => [], # intentionally untested
174
+ }.freeze
226
175
  ```
227
176
 
228
- Positional operands name the sources to mutate: Ruby files, or directories
229
- (all their `.rb` files). Each must exist; a typo or a non-Ruby file exits 2
230
- before anything runs. They replace `include` and `exclude`. `--test`
231
- (repeatable; spec files or directories, with an optional `:LINE` or `[1:2]`
232
- selector; needs at least one source, unless `--since` is given - the changed
233
- files are the sources) replaces only the files RSpec runs. Other
234
- `rspec_args` and `.rspec` options such as `--require` stay. The coverage map
235
- is built from only those specs, each mutant runs only the ones that cover it,
236
- and the selection is part of the coverage and `--incremental` cache keys.
237
-
238
- `--threshold PERCENT` (0 to 100, or `c.threshold` in `config/saboteur.rb`) is for CI.
239
- The run exits 1 when the score is below PERCENT and 0 when it is at or above,
240
- even with survivors. It fails closed: a run that tested no mutant has no score
241
- and exits 1, and a run where any mutant errored or hit a resource limit exits 3,
242
- because the score covers only part of the run. Without a threshold, any
243
- unignored survivor exits 1. Either way, a run where more than half of the
244
- executed mutants errored exits 3: errored mutants are outside the score,
245
- so such a run tested almost nothing.
246
-
247
- `--max-timeouts N` and `--max-errors N` (or `c.max_timeouts` / `c.max_errors`)
248
- are two more CI gates. Each takes a count or a percentage of the executed
249
- mutants (`--max-errors 5%`). The run exits 1 when the count (or share) is
250
- above the limit, with `exitCause` `maxTimeouts` or `maxErrors` in the JSON report.
251
- `--threshold` already compares the covered score, `(killed + timeout) /
252
- (killed + survived + timeout)`, so no separate score flag exists. With
253
- `--max-errors`, errored mutants are judged by that limit instead of making a
254
- `--threshold` run exit 3. Survivors and a score below the threshold keep
255
- their own causes when they fail too. The gates cannot be combined with `--agent`.
256
- The GitHub Action has no input for them yet.
257
-
258
- Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
259
- `--config PATH`, `--integration NAME`, `--reeval class_eval|nesting`,
260
- `--json PATH`, `--no-cache` (rebuild the coverage map), `--allow-focus`.
261
-
262
- A stray `fit`, `fdescribe` or `:focus` example, with `filter_run_when_matching :focus`, makes RSpec skip
263
- every other example. The baseline stays green and the score would cover only the focused ones, so
264
- saboteur exits 2 and names the focused examples. Remove the focus, or pass `--allow-focus` (config
265
- `c.allow_focus = true`) to run the narrowed suite. Filters you name in `rspec_args` or `.rspec`, such
266
- as `--tag slow`, and example selectors such as `--test spec/a_spec.rb:12`, are on purpose and are not refused. The
267
- check compares what RSpec runs with what it would run without the stray focus, so examples RSpec leaves out anyway
268
- (`:if` / `:unless`, `--tag`, exclusions, selectors in another file) never count as hidden.
269
-
270
- For pull requests, start with `--since` against your target branch. This is a
271
- workflow recommendation, not a CLI default: plain `run` still checks the whole
272
- configured project. To scope the specs as well, pass them with `--test` and no
273
- source operands: `bundle exec saboteur run --since origin/main --test spec/billing_spec.rb`.
274
- With `--since` and no `--test`, the run warns before it starts that the two baseline passes
275
- run the loaded suite (a cold coverage cache adds one more pass), and how to
276
- scope them.
277
-
278
- `--since origin/main --guess-tests` does that scoping for you: it picks the spec
279
- named after each changed source file (`lib/a/b.rb` runs `spec/a/b_spec.rb`; `app/`
280
- files and `spec/lib/` work too) and runs only those for the baseline passes, the
281
- coverage pass and every mutant, as `--test` would. It is a naming guess, not
282
- coverage: a spec that reaches the changed file only through another file does not
283
- run: a mutant only that spec would kill can survive, and a line that none of the
284
- guessed specs executes is reported as NoCoverage. The agent JSON records the guessed specs as its
285
- `tests`, so its follow-up commands keep the same `--test` scope (with `--no-guess-tests`, so a `guess_tests`
286
- in the config file cannot conflict) instead of widening to the whole suite. If any changed file has no such
287
- spec, or nothing changed, the whole suite runs as before and the run says so.
288
- `--guess-tests` needs `--since` and cannot be combined with `--test`. Keep whole-project runs on main: diff mode does not check every method affected by a
289
- dependency change. [The pull-request guide](docs/pull-requests.md) covers
290
- checkout history, a GitHub Actions example, configuration and limits.
291
-
292
- Doctor warns when an ignore subject matches no method discovered in the included
293
- source files. Check the exact `Class#method` or `Class.method` form and the
294
- include/exclude settings; warnings never edit or delete ignore entries.
295
- A malformed ignore file fails doctor with a config error before target boot.
296
-
297
- `saboteur doctor` prints Ruby, coverage, integration, Rails/SimpleCov and boot-file
298
- checks. Add `--suite` to also run the unchanged suite twice with the chosen seed
299
- (`--test PATH`, repeatable, limits those runs to some specs and implies `--suite`),
300
- since that takes hours on a large suite. It writes no
301
- mutation reports or coverage/verdict caches. Target boot files and tests still
302
- run, so their normal side effects remain. See the [doctor guide](docs/guides/doctor.md).
303
-
304
- `--id` runs exactly one current mutant with its coverage-selected tests, bypassing
305
- per-method caps, ignore suppression and incremental verdict reuse.
306
- It still honors `--subject` and `--since` changed-line selection, and the
307
- hazard guard: a digest that names a suppressed mutant is refused unless
308
- `--allow-hazardous` is also given. See the
309
- [replay guide](docs/guides/replay.md) for digest format, checks and limits.
310
-
311
- Every test run starts from the same random seed (1 by default), so tests that
312
- use `rand` see the same values in each mutant's run and in the check that
313
- confirms a kill. The report prints the seed; pass it to `--seed` to repeat a
314
- run.
315
-
316
- `--incremental` reuses an earlier verdict only while the method, the whole
317
- file it lives in, the mutant, its covering tests and their files, the lockfile
318
- and the config are all unchanged. So a constant or helper changed in the same
319
- file is seen. It cannot see a change in another file that the method uses (a
320
- method it calls, a constant defined elsewhere), so run without it now and then
321
- (and in CI on the main branch).
322
-
323
- Output: survivors first, then a short summary last, so the summary stays on
324
- screen. Survivors are grouped by file, then method, then line. Each line is
325
- printed once as `path:line in Subject`, with the source line and the tests that
326
- covered it, then one compact line per mutation: `operator original -> replacement digest`,
327
- and a hint on what kind of test would kill it. The summary has the counts per
328
- status, the score, a table of scores per file (worst first) and a one-line
329
- legend. `--verbose` adds the score split, mean time, whole-body score and
330
- kill causes. Colour is on when stdout is a terminal and `NO_COLOR` is not set;
331
- `--color` and `--no-color` override that. While mutants run, stderr shows
332
- `done/total (percent) eta`: a live counter on a terminal, and dots with a
333
- counter every 50 mutants otherwise. Skipped and reused mutants are not
334
- counted, so the count can end below the total.
335
- The tests that covered the survivor's line and
336
- still passed with the mutant applied are listed (the first five, likeliest killers
337
- first; the rest are under `coveredBy` in the JSON report). The missing
338
- check may belong in one of them. Line coverage can overstate which tests
339
- exercise a line, so treat the list as a lead. A survivor reused with
340
- `--incremental` was not run again, and the list says its tests passed in an
341
- earlier run. `--fail-fast` (serial runs only) stops after the first subject
342
- that has a survivor the ignore list does not accept. The report then covers
343
- only the subjects that ran: the terminal says which subjects did not run, the
344
- JSON report has an `earlyStop` field naming them (`exitCause` is `failFast`),
345
- `--agent` reports status `partial` with the reason `fail_fast`, and the run
346
- always exits 1, so no `--threshold` can pass it. It cannot be used with
347
- `baseline` evidence. Survivors that share a line are printed together only
348
- when their covering tests, reuse status and changed lines are identical;
349
- otherwise each distinct set of evidence is printed above its own mutations.
350
- Hints are suggestions, not diagnoses: a survivor can also be an equivalent
351
- mutant. The HTML page shows the same list in the
352
- survivor's notes. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
353
- report (one file, no scripts, no network) to `.saboteur/report.html`. Change
354
- the paths with `--json` / `--html` or `c.json_report` / `c.html_report` in
355
- `config/saboteur.rb`; set either to `nil` to turn that report off. The two paths
356
- must differ. The HTML page shows the covered mutation score
357
- `(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage,
358
- RanAtLoad and Unreachable out and keeps survivors accepted on the ignore list
359
- in the denominator. A tool
360
- that reads the JSON report may show a different score, because the schema counts
361
- NoCoverage as undetected.
362
-
363
- `--matrix` runs every covering test against every mutant instead of stopping at
364
- the first failing one, so a killed mutant lists all of its killers. After the
365
- survivors the report prints two lists: blind tests (they covered mutants and
366
- killed none) and redundant tests (they killed some, and every one of those
367
- mutants had another killer). The run is slower, by up to the number of covering
368
- tests per mutant. `--matrix` cannot be combined with `--incremental`, because a
369
- reused verdict lists only the first killer. A redundant test is a lead, not a
370
- verdict: it may guard behavior no selected mutant touches. A selection with
371
- `--since` or `--subject` describes only the selected methods.
372
-
373
- [Scores](docs/scores.md) defines each number on this page: the covered score, the
374
- score a Stryker dashboard shows, the assertion and crash split, and what makes a run
375
- incomplete.
376
-
377
- The score is split by how the mutant was killed. A crash kill (a `NoMethodError`,
378
- a `TypeError`, a parse error) shows the code ran and broke, not that a test
379
- checked a value. The terminal prints
380
- `score 100.0 % (assertion 50.0 %, crash 50.0 %)`: each share is a part of
381
- tested mutants, so the parts add up to the score. Timeouts and kills with no
382
- recorded cause (a verdict reused from an older run, a test that called `exit`,
383
- a failure outside any example) are named when there are some. Whole-body
384
- mutants (`body_nil`, `body_true`, `body_false`, `body_raise`) are killed by
385
- almost any test that runs the method, so a second line scores the run without
386
- them, and shows them alone. The headline score and `--threshold` still count every
387
- mutant. The JSON report carries both splits under `saboteur.scores`
388
- (`all` and `withoutWholeBodyMutants`), and the HTML page prints them under
389
- the score. The JSON also records why the run exited the way it did under
390
- `saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`,
391
- `errorMajority`, `incompleteUnderThreshold`). In `--agent` mode the cause
392
- follows the agent verdict, so the report and the process exit classify the
393
- run the same way: `ok` for clean, `survivors` for survivors, and otherwise
394
- `errorMajority` when most mutants errored or `incomplete`.
395
-
396
- For code-review surfaces, the JSON report also carries a `saboteur.findings`
397
- section: a bounded, deterministically ordered list of survivors, coarse
398
- mutants first (a surviving coarse mutant hints that no test pins the method's
399
- behavior - it can also be a re-evaluation limit or an equivalent mutant),
400
- then file, line and digest. Each finding holds the file, line, digest,
401
- subject, operator, original and replacement a note on a pull-request diff
402
- needs, plus one entry per edit for mutants that edit more than one spot.
403
- Survivors accepted on the ignore list are listed separately with their
404
- reason, and `survivorsTruncated` and `acceptedTruncated` count what the bound
405
- left out; the full data stays in the schema's `files` section.
406
-
407
- In diff mode (`--since`), a mutant is reported at the line where its statement
408
- starts. A statement that spans several lines can start a line or two above the
409
- line that changed. Under each survivor the terminal therefore prints
410
- `changed line: path:N`, the changed line or lines (the first three) the mutant
411
- touches, and each finding carries them as `changedLines`. Use `changedLines`,
412
- not `line`, to put a note on a changed line of the pull-request diff. A
413
- whole-body mutant of a method whose only change is its signature touches no
414
- changed line and prints none.
415
-
416
- New to survivors? The [reading survivors guide](docs/guides/reading-survivors.md) walks one from the diff to the test that kills it.
417
-
418
- Exit codes: 0 no survivors (or all on the ignore list), or the score meets
419
- `--threshold`; 1 survivors, or the score is below `--threshold` or missing;
420
- 2 bad configuration or a missing source or test; 3 the tool failed (with
421
- `--threshold`, also any errored or resource-limited mutant).
177
+ A matching entry wins over the naming convention: the file stops being
178
+ unmatched and its mapped specs run as the inferred ones. Keys are exact
179
+ paths, globs or Regexps; an empty array marks the file intentionally
180
+ untested, so it needs no inferred spec. The file loads before the app
181
+ boots, so keep it plain Ruby. `saboteur doctor` validates it.
422
182
 
423
183
  ### Rejected kills
424
184
 
425
- The terminal prints `kills rejected by unchanged-method check: N`; JSON adds
426
- `rejectedKills`. This counts fresh per-mutant kill claims whose tests also fail
427
- with the method unchanged. They stay errors outside the score. Timeout checks,
428
- other errors and reused cached kills are not included. See
429
- [rejected kills](docs/rejected-kills.md) for details.
430
-
431
- ## βš™οΈ Configuration
432
-
433
- `config/saboteur.rb` in the project root; CLI flags override it. Keep the shared
434
- config usable for whole-project runs and pass `--since` in the pull-request
435
- job. A PR-only config may set `c.since = "origin/main"`; omit `since` (or use
436
- `c.since = nil`) in the config for audits on main. Use your actual target branch.
437
-
438
- ```ruby
439
- Saboteur.configure do |c|
440
- c.include = ["lib/**/*.rb"]
441
- c.exclude = ["lib/my_gem/version.rb"]
442
- c.require = [] # boot files, e.g. ./config/environment
443
- c.whole_file = [] # subject ids or globs to re-evaluate as a whole file (see below)
444
- c.reeval = "nesting" # or "class_eval"; see docs/design.md 6.1
445
- c.timeout_factor = 3 # per-mutant deadline = factor x its tests' recorded time
446
- c.timeout_min = 5
447
- c.max_memory_mb = 2048 # memory growth cap per mutant process tree
448
- c.max_processes = 64
449
- c.max_mutants_per_method = 100 # positive integer; raise it for a broader run
450
- c.hazard = Saboteur::MutationSource::HazardGuard::DEFAULT_HAZARD # calls whose arguments are never mutated
451
- c.sandbox = true # read-only project tree for mutant children (Linux 6.2+); "strict" refuses to run degraded
452
- c.sandbox_writable = [] # extra dirs mutant children may write, e.g. ["db", "log"]
453
- c.ignore_file = "config/saboteur_ignore.rb"
454
- c.seed = 1 # Ruby's random seed in every test run
455
- c.allow_focus = false # true runs a suite narrowed by fit/fdescribe/:focus instead of exit 2
456
- c.fingerprint_extra = [] # extra globs whose contents invalidate caches, e.g. ["data/*.csv"]
457
- end
458
- ```
185
+ The terminal lists them in a `Rejected kills` section and annotates the error
186
+ count (`error 3 (2 rejected kills)`); JSON flags each rejected mutant with
187
+ `rejectedKill: true` and keeps the `rejectedKills` count. These are fresh
188
+ per-mutant kill claims whose tests also fail with the method unchanged. They
189
+ stay errors outside the score. Timeout checks, other errors and reused cached
190
+ kills are not included. See [rejected kills](docs/rejected-kills.md) for
191
+ details.
459
192
 
460
193
  ### Rails apps
461
194
 
462
- Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a Rails
463
- app needs a few settings; the generator writes them for you.
464
-
465
- **1. Add the gem**
195
+ Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a
196
+ Rails app needs a few settings; the generator writes them for you:
466
197
 
467
198
  ```sh
468
199
  bundle add saboteur --group test
469
- ```
470
-
471
- **2. Generate the setup**
472
-
473
- ```sh
474
200
  bundle exec rails generate saboteur:install
475
201
  ```
476
202
 
477
- This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets `RAILS_ENV=test`
478
- for you) and a `.gitignore` entry for `.saboteur/`. It never overwrites an
479
- existing file unless you pass `--force`, and never edits your Gemfile.
203
+ This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets
204
+ `RAILS_ENV=test` for you) and a `.gitignore` entry for `.saboteur/`. It
205
+ never overwrites an existing file unless you pass `--force`.
480
206
 
481
- Prefer to do it by hand, or on a version without the generator? Put this in
482
- `config/saboteur.rb`:
207
+ Until a config names your sources, a run whose include selects no Ruby
208
+ file refuses with exit 2 and points at the install generator above -
209
+ a first CI run on an unconfigured tree fails loud instead of passing
210
+ with 0 subjects.
211
+
212
+ Prefer to do it by hand, or on a version without the generator? Put this
213
+ in `config/saboteur.rb`:
483
214
 
484
215
  ```ruby
485
216
  Saboteur.configure do |c|
486
- c.include = ["app/**/*.rb"]
217
+ c.include = ["app/**/*.rb"] # spec/test dirs vendored under lib/ are excluded by default
487
218
  c.require = ["./config/environment"] # boots the app
488
219
  c.rails = true # reconnects ActiveRecord after fork, per-worker databases
489
220
  c.jobs = 4 # optional; the default is 1
490
221
  end
491
222
  ```
492
223
 
493
- **3. Check the setup**
494
-
495
- ```sh
496
- bin/saboteur doctor
497
- ```
498
-
499
- Doctor boots your app, checks the config and the test database host, and runs
500
- your suite twice before any mutation. Fix whatever it reports first. Turn
501
- SimpleCov off for the run if you use it.
502
-
503
- The database check refuses any test database whose host is not local
504
- (`localhost`, `127.0.0.1`, `::1`, `0.0.0.0` or a socket), so a remote
505
- database is never mutated by accident. `0.0.0.0` counts as local: compose
506
- setups write it into database URLs, and connecting to it reaches this
507
- machine. For another disposable test database host, assert it yourself:
508
- `c.allow_db_hosts = ["db.internal"]` in `config/saboteur.rb`.
509
-
510
- **4. Test what your branch changed**
511
-
512
- ```sh
513
- bin/saboteur run --since main
514
- ```
515
-
516
- It mutates only the methods changed since `main` and reports each survivor with
517
- its diff: a change your tests did not notice. Write a test that fails on that
518
- diff, then run again. Use `--since origin/main` in CI. Without the generator, use
519
- `bundle exec saboteur` and `RAILS_ENV=test` instead of `bin/saboteur`.
520
-
521
- saboteur puts the app's `spec/` directory on the load path before
522
- processing `require`, matching rspec, so a boot file like a dummy app's
523
- `spec/rails_helper.rb` can require its siblings.
524
-
525
- With `jobs` above 1 on SQLite, each extra worker gets a copy of the test
526
- database made through SQLite's backup API, so the copy is a consistent
527
- snapshot. A relative `database:` path resolves against `Rails.root`, the
528
- same as Rails 7.1+ does, for the copy and for its cleanup. The copy
529
- checkpoints the source first (`PRAGMA wal_checkpoint(TRUNCATE)`) and
530
- refuses to run when a reader keeps the WAL busy, because copying on
531
- would silently miss committed rows. A source file that does not exist is
532
- an error, not a skipped copy. An in-memory SQLite database (`:memory:`) is
533
- never copied, renamed or disconnected. After the app boots, each in-memory
534
- database is snapshotted to a temporary file through SQLite's backup API,
535
- and every forked child restores the snapshot into its own fresh database -
536
- a child cannot keep the parent's connection (ActiveRecord discards
537
- inherited pools and sqlite3 closes inherited writable handles), so without
538
- the snapshot it would see an empty database.
539
-
540
- For a Rails engine, point `require` at the dummy app's environment -
541
- `test/dummy/config/environment.rb`, usually loaded for you by
542
- `spec/rails_helper.rb`. Only a dynamically created dummy app has no
543
- environment file; there, require the helper that calls setup instead,
544
- such as `DummyApp.setup` in `spec/rails_helper.rb`.
545
-
546
- Bootsnap needs no extra setting: bytecode from its compile cache carries no
547
- coverage counters, so saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its
548
- own process when the variable is not already set (Bootsnap treats any set
549
- value as off) and runs the eager load with Coverage active. App code is seen
550
- by Coverage with the app unchanged.
551
-
552
- With `jobs: 4` (or more), each extra worker gets a copy of every test
553
- database, named `<database>_saboteur2` and up, made with
554
- `CREATE DATABASE ... TEMPLATE` on Postgres or a file copy on SQLite.
555
- saboteur drops the copies when the run ends, on a best-effort basis: a
556
- drop that fails is reported as a warning and never changes the run's
557
- outcome. A copy left behind - by a failed drop or a killed run - is
558
- reclaimed by the next run, whose prepare starts from
559
- `DROP DATABASE IF EXISTS`. MySQL is not supported for `jobs` above 1.
560
-
561
- Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
562
- `db/schema.rb` and `config/*.yml` (including `database.yml`) are part of the
563
- cache fingerprint automatically. Use `fingerprint_extra` for any other file your
564
- tests read.
565
-
566
- ### Load-time code, captured methods and Sorbet signatures
567
-
568
- Code that runs while the target boots, before tests run, cannot be checked by
569
- replacing a method later. saboteur reports these changes as `ran_at_load` rather
570
- than survivors. Calls from test setup, including RSpec `before(:all)`, are still
571
- checked. Load-time detection follows the `ran_at_load` idea in
572
- [mutineer #187](https://github.com/davidteren/mutineer/pull/187).
573
-
574
- A saved `Method` or `UnboundMethod` can keep calling the old definition after
575
- saboteur replaces it. When tests pass without calling the replacement and a
576
- saved method still points to the old definition, the result is `unreachable`.
577
- A targeted execution check must also confirm the replacement was not called.
578
- Each mutant is checked separately; one unreachable result does not skip later mutants.
579
- The reason names the old definition's source location. A lambda that looks up
580
- the method when it runs is still checked normally.
581
-
582
- Coverage caches now store both per-test method hits and load-time line hits.
583
- Older caches are rebuilt; load-time accounting preserves endless-method coverage.
584
-
585
- One-line and endless method definitions need boot method-call counters to prove
586
- the body ran at load; merely defining them is not execution evidence. If methods
587
- share a physical line and any sibling is uncalled, the line is ambiguous and is
588
- conservatively reported as no coverage, even for its called sibling.
589
- Agent mode reports ran-at-load and unreachable work as incomplete.
590
-
591
- A whole-suite map with only load-time hits is valid and reports load-only subjects
592
- with their ran-at-load reason, rather than failing as an empty coverage build.
593
- It has no mutation score when no mutant can be tested.
594
-
595
- Both statuses are listed separately in terminal output and excluded from the
596
- mutation score. Stryker JSON reports them as `Ignored`, with the explanation in
597
- `statusReason`; they are not entries on your survivor ignore list.
598
-
599
- Methods wrapped by `sorbet-runtime` signatures are checked rather than skipped
600
- as shadowed definitions. saboteur re-evaluates the adjacent `sig` with the
601
- changed method so the replacement keeps its signature wrapper. Missing adjacent signatures and unknown wrappers are skipped when saboteur cannot
602
- preserve the loaded method semantics.
603
-
604
- ### Whole-file re-evaluation (opt-in)
605
-
606
- Surgical re-evaluation refuses a method whose owner it cannot resolve or whose
607
- loaded definition is not the one in the file. The skip reason ends with a hint.
608
- List such subjects in `c.whole_file` (ids or globs such as `"Point#distance"` or
609
- `"Geometry::*"`) and the child evaluates the whole mutated file instead. This
610
- applies to methods saboteur already finds as subjects; it does not make new
611
- kinds of definitions selectable (methods inside `Data.define` or `Struct.new`
612
- blocks are still not found). The file's top-level code then runs again in every
613
- child, after the run's seed is set: load-time side effects repeat and constants
614
- it assigns are rebuilt, with redefinition warnings silenced. If a later
615
- definition in the same file replaces the mutated method, the mutant could never
616
- run, so a survivor there is reported as unreachable instead. It is off by
617
- default, and the cache key includes the list.
618
- Signature text alone does not remove mutants. `body_nil` and last-statement
619
- removal remain available even when `sig` declares a non-nil return type.
620
- `checked(:never)`, unchecked `checked(:tests)` signatures, configured default
621
- levels, custom validation handlers and wrapper shims can all allow nil to
622
- reach real assertions. saboteur does not verify enforcement, evaluate return
623
- type expressions or infer types from constant names. Aliases, enclosing
624
- namespace constants and transformed generic types therefore cannot change
625
- mutation generation. RBS signatures are not read yet. Type-only crash kills
626
- remain visible in the crash share, not silently pruned.
627
-
628
- The Sorbet E2E fixture is a wrapper/registry shim without runtime type checking.
629
- It tests re-evaluation and assertion detection, not real Sorbet enforcement or
630
- proof of type-check noise reduction.
631
-
632
- ### Skipped subjects and the mutant cap
633
-
634
- Subjects with no test coverage are skipped before mutation sources run. They
635
- remain in the subject count, are listed as skipped subjects in terminal output,
636
- and appear under `skippedSubjects` in JSON. They are not invented mutants and do
637
- not enter saboteur's mutation score. Stryker dashboards may ignore the extra
638
- `skippedSubjects` field and calculate scores from mutant entries alone, so
639
- removing previously generated `NoCoverage` mutants can raise a dashboard's score
640
- even when test detection has not improved. Changes inside a covered method that
641
- no test reaches still produce mutant-level "no coverage" results.
642
-
643
- Each covered method keeps at most `max_mutants_per_method` valid, distinct
644
- mutants (100 by default). Coarse mutants come first, then fine mutants in
645
- configured mutation-source order and each source's proposal order. The cap keeps
646
- a prefix of that order, so earlier sources can use the entire limit within a
647
- method. Later methods have their own limit. In diff mode, changed-line selection
648
- happens before the cap. A warning names each capped method and the number
649
- omitted; JSON keeps these warnings under `warnings`. The score describes this
650
- selected subset, not every possible mutation. Raise the limit to examine more
651
- mutants; the limit does not bound how many proposals a custom source creates.
652
-
653
- ### Ignoring a survivor
654
-
655
- Every survivor is printed with a digest. Accept it with a reason:
656
-
657
- ```ruby
658
- # config/saboteur_ignore.rb
659
- Saboteur.ignore do |i|
660
- i.mutation "1a2b3c4d5e6f7a8b",
661
- subject: "Billing::Invoice#total",
662
- reason: "equivalent: amounts are always whole cents"
663
- end
664
- ```
665
-
666
- An entry without a reason is rejected.
224
+ An engine gem whose specs live under `spec/dummy` needs an exclude for them; see
225
+ [Rails engines and dummy apps](docs/guides/rails-engines-and-dummy-apps.md).
667
226
 
668
- ### SimpleCov
227
+ ## SimpleCov
669
228
 
670
- saboteur needs Ruby's `Coverage` for itself. Disable SimpleCov for the run
671
- (most projects already guard it with an environment variable).
229
+ saboteur needs Ruby's `Coverage` for itself, so leave SimpleCov off for
230
+ saboteur runs - for example, start it only when `ENV["COVERAGE"]` is set.
231
+ The [recipes](docs/recipes/README.md) cover this and VCR, WebMock and
232
+ Sidekiq.
672
233
 
673
- ## Filter mutations
234
+ <a id="dated-validation-runs"></a>
235
+ <a id="where-it-has-run"></a>
236
+ ## Benchmarks and validation
674
237
 
675
- The built-in source also swaps `select` / `reject` and `select!` / `reject!`.
676
- These catch a filter that keeps the items it should discard (or discards the
677
- items it should keep). For example, a command replacement should remove the
678
- old command, not keep only that command:
238
+ Speed and memory numbers live with the harness and the raw data that
239
+ reproduce them, framed as dated measurements rather than standing claims:
240
+ the [benchmarks page](https://vamsipavanmahesh.github.io/saboteur/benchmark/)
241
+ on the site and [docs/benchmark](docs/benchmark/) in this repo.
679
242
 
680
- ```ruby
681
- commands.reject! { |name, _| name == replaced_name }
682
- # mutant: commands.select! { |name, _| name == replaced_name }
683
- ```
243
+ ## Project
684
244
 
685
- Only explicit-receiver calls with a block and no ordinary arguments are
686
- changed. A block passed as `&predicate` counts; safe navigation is preserved.
687
- `relation.select(:name)`, bare `select { ... }`, and `items.select` without a
688
- block are not changed. Arid statements and nested definitions remain skipped.
689
- Ruby dispatch is dynamic: a matching name does not prove the receiver is an
690
- Enumerable, and a custom receiver can make the replacement fail. No type or
691
- real-fault detection guarantee is made.
692
-
693
- These are fine mutants, appended after the existing built-in operators. For unchanged subjects, old
694
- mutant digests and relative order stay unchanged, but runs can produce more
695
- mutants, change their score, and hit the per-method cap. The operator and source
696
- name in reports is `filter_replacement`.
697
-
698
- See [the operator rationale](docs/filter-operators.md) for the Ruby fault used
699
- to choose this family and the limits of that evidence.
700
-
701
- ## Literal mutations
702
-
703
- Builtin now swaps `true` / `false` literals and replaces nonempty static strings
704
- with `""`. These test boolean choices and the difference between meaningful
705
- text and empty input. Operator names are `literal_boolean` and `literal_string`;
706
- the source name is `literal_replacement`.
707
-
708
- Only method bodies are visited. Default arguments, arid statements, nested
709
- definitions, empty strings, heredocs, interpolation, word/symbol lists, symbols,
710
- regexes and command strings are unchanged. Quoted and static percent strings
711
- are supported. The new proposals follow the earlier built-in operators; the
712
- normal duplicate check and per-method cap still apply. Old digests and order
713
- remain stable for unchanged subjects, but more mutants can change a run's score.
714
- No speed, type-safety or real-fault detection improvement is claimed. See
715
- [the literal operator rationale](docs/literal-operators.md) for fault evidence
716
- and limits, including string encoding/frozen-value behavior.
717
-
718
- ## Chain removal
719
-
720
- Builtin drops one link from the middle of a dotted method chain: `user.name.strip`
721
- becomes `user.strip`, and `a.b.c.d` gives `a.c.d` and `a.b.d`. The first receiver
722
- and the last call stay, so a test has to notice that a step in between went
723
- missing. The link goes whole, with its arguments and block. The operator and
724
- source name is `chain_removal`.
725
-
726
- Operators, index calls and bare-name receivers are not chains and are unchanged.
727
- The target of a write (`user.account.profile.name = 1`, `+=`, `||=`, `a.b[0] = 1`)
728
- is left whole: no link of its receiver chain is dropped, however long. Only
729
- the value on the right can be mutated. `freeze`, `dup`, `clone`, `itself`
730
- and `tap` links are never removed. Tests can observe frozenness, copies and a
731
- `tap` block's effects, but those are other contracts than a missing step in a
732
- query chain, and dropping such a link mostly yields noise, so this operator
733
- leaves them out. Arid statements and nested definitions are skipped.
734
-
735
- Chain removal is on by default, like `argument_removal` and `pair_removal`.
736
- Its proposals follow the other fine proposals, with the same validation,
737
- deduplication and `max_mutants_per_method` cap.
738
-
739
- ## Argument removal
740
-
741
- Builtin removes one ordinary positional argument from a parenthesized call.
742
- For `Model.includes(:a, :b, c: :d)`, the two proposals remove `:a` or `:b`,
743
- without deleting the statement or changing `c: :d`. This lets a preload check
744
- exercise missing associations instead of only a missing return value.
745
- The operator and source name is `argument_removal`.
746
-
747
- Only parenthesized calls are supported in this first version. Keyword hashes,
748
- splats, argument forwarding, bracket indexing/assignment syntax and arguments containing
749
- heredocs are unchanged. Explicit hash arguments count as positional arguments.
750
- Blocks and keyword pairs remain intact; separators come from Prism tokens,
751
- so commas inside strings or comments are not mistaken for separators.
752
- Arid statements, parameter defaults and nested definitions are skipped.
753
-
754
- These fine proposals follow existing builtin proposals, with the same validation,
755
- deduplication and `max_mutants_per_method` cap. Earlier digests and relative order
756
- stay unchanged for unchanged subjects. Extra proposals can change scores or be
757
- omitted by the cap. Removing an argument can still raise an arity error; no
758
- non-crashing guarantee is made. The regression uses a small preload-like receiver
759
- and checks the missing associations, not an ActiveRecord SQL benchmark. See
760
- [the validation notes](docs/argument-removal.md) for the pinned snapshot migration.
761
-
762
- ## Hash and keyword pair removal
763
-
764
- Builtin removes one key/value association from a hash literal or keyword list.
765
- `Model.includes(:a, :b, c: :d)` becomes `Model.includes(:a, :b )`, so a nested
766
- preload guard can be exercised without deleting the whole statement. A sole
767
- hash pair becomes `{}`. Operator and source name: `pair_removal`.
768
-
769
- Both parenthesized and command calls are supported. Splats stay intact; pairs
770
- containing heredocs, parameter defaults, arid statements and nested definitions
771
- are skipped. Token-based separator edits preserve comments and blocks. Nested
772
- hash values and callable bodies are visited. Removing a required keyword can
773
- still crash; the regression uses a preload-like receiver, not an ActiveRecord
774
- SQL benchmark. These fine proposals follow earlier builtin proposals and use
775
- the usual validation, deduplication, order and `max_mutants_per_method` cap.
776
- More proposals can change scores and cap coverage. See
777
- [the validation notes](docs/pair-removal.md) for snapshot changes.
778
-
779
- ## πŸ”Œ Extending
780
-
781
- Public API, stable across minor versions:
782
-
783
- - `Saboteur::MutationSource::Base#mutations_for(subject)` returning
784
- `Saboteur::MutationSource::Mutation` objects made of `Edit`s (file, byte
785
- range, replacement). Register with `sources: [builtin, path.rb:ClassName]`.
786
- - `Saboteur::Integration::Base` for other test frameworks (RSpec is the only one
787
- that ships).
788
- - `Saboteur.hook(:before_boot | :after_boot | :before_fork | :after_fork |
789
- :around_mutant)` in files listed under `hooks:`.
790
-
791
- ## 🌍 Where it has run
792
-
793
- An early release (0.4.0). Tested on the RSpec suites of plain gems
794
- (dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
795
- human-essentials on Postgres (4,080 mutants, 6 workers; the 2026-10-10 run, before
796
- the baseline check and the coarse-gate removal) and lobsters on SQLite,
797
- both on macOS. Not yet tried on MySQL or older Rails versions. RSpec only, by
798
- design. Results: [docs/validation](docs/validation).
799
-
800
- ## βš–οΈ How it compares
801
-
802
- A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
803
- 1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
804
- ran once on sampled files, so treat the numbers as indicative. The table was
805
- measured at 2b6ec02, before the literal, filter and argument operators and
806
- before the coarse-survivor gate was removed, so mutant counts and times will
807
- change; a rerun is pending. mutineer is ahead
808
- on number of operators and CI extras, and it supports Minitest.
809
-
810
- | Metric | saboteur | mutineer |
811
- |---|---|---|
812
- | Fake kills (lobsters) | 0 of 20 sampled at random | 8 of 8 hand-checked (identical mutants, not a random sample) |
813
- | Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
814
- | Runs on the Postgres app | yes | no |
815
- | Time per mutant, hashie | 0.008 s | 0.092 s |
816
- | Peak memory, hashie (summed process RSS) | 595 MB | 471 MB |
817
- | Choice of operators, CI features | fixed set, fewer | 22 operators, baseline, HTML, GitHub Action |
818
-
819
- Full results, method and limitations:
820
- [benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
821
- Scripts and raw data: [bench/vs-mutineer](bench/vs-mutineer). The memory row sums
822
- per-process RSS, which counts shared pages more than once, and it was measured
823
- before the fork-memory work. Read it as history, not a ranking of physical
824
- memory.
825
-
826
- ### Independent review, saboteur 0.4.0 vs mutineer 1.6.0
827
-
828
- A private consultant review ("saboteur 0.4.0 - Deep Review", 12 October 2026,
829
- not published) ran saboteur 0.4.0 at 38b1834 and mutineer 1.6.0 at a766c96 on
830
- the same machine and projects: Linux x86_64, 1 vCPU, 4 GB, Ruby 3.4.5, Prism
831
- 1.2.0, RSpec 3.13, no Rails or sqlite3. mutineer ran with its default 5
832
- operators and with all 20. The project and timing rows are one run each. The
833
- rows describe saboteur at 38b1834; several of its findings have been fixed on
834
- main since. The rerun that replaces this table is still pending.
835
-
836
- | Measure | saboteur 0.4.0 | mutineer 1.6.0 |
837
- |---|---|---|
838
- | Snippets project (11 planted gaps) | 177 mutants, 14 s, 10/11 gaps found, 9 noise survivors | Default: 56 mutants, 18 s, about 5/11. All operators: 191 mutants, 51 s, 10/11, 8 noise and duplicates |
839
- | Dependency project (A tested only via B) | 239 mutants, 6 s | 254 mutants, 26 s (needed hand-ordered source files) |
840
- | dry-inflector (real gem) | 551 mutants, 24 survivors, 95.6 %, 133 s | 480 mutants, 47 survivors (31 distinct), 90.1 %, 134 s |
841
- | Test selection | Whole suite, per example, automatic | Only files passed with --test, per file |
842
- | Kill verification | Yes (re-runs killers on unchanged code) | No |
843
- | sorbet sig methods | Skipped at 0.4.0 | Mutated, 24/24 killed |
844
- | Data.define / Struct.new block methods | Absent at 0.4.0 | Mutated |
845
- | Load-time code | False survivors at 0.4.0 | ran_at_load status |
846
- | Endless defs | Tested ones reported as no coverage at 0.4.0 | Tested ones work; untested ones show as survived |
847
- | Order-dependent specs | Smell exposed, bogus kills rejected | Hidden (whole file runs) |
848
- | Destructive mutant | Project wiped except .saboteur | Project wiped |
849
- | Baseline stability | Race at 0.4.0: 4 of 30 green runs failed (13 %), 5 of 25 under CPU contention (20 %) | No failures observed (15/15) |
850
-
851
- Not in the review and still pending: a rerun on current main, memory and
852
- per-mutant time on the benchmark apps, and any mutineer release after 1.6.0.
853
-
854
- ### Memory after the fork-memory work (hashie @ 9c81e83)
855
-
856
- Paired A/B runs against the pre-PR commit (32c0f21), 5 pairs per job level with
857
- AB/BA alternation and cold caches, on a 2 vCPU / 2 GB shared sandbox. Peak
858
- memory is per-process-tree PSS from /proc/smaps_rollup sampled at 0.3s. Verdict
859
- parity: **0 verdict differences in 23,700 mutant outcomes** (1,580 mutants x 15
860
- pairs); kill reasons (assertion vs crash class) vary run to run on both
861
- variants, as before.
862
-
863
- | jobs | peak PSS baseline (median, range) | peak PSS PR (median, range) | shared-memory delta |
864
- |------|-----------------------------------|-----------------------------|---------------------|
865
- | 1 | 101 MB (101-101) | 100 MB (100-102) | ~0 (no workers to share with) |
866
- | 2 | 133 MB (116-134) | 109 MB (106-121) | -18% |
867
- | 6 | 235 MB (192-269) | 196 MB (175-201) | -17% |
868
-
869
- Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not
870
- touch less memory, it keeps more of it shared between workers after fork, which
871
- is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers on a
872
- 2 GB sandbox. Swap was not measured.
873
-
874
- Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs
875
- 32s at jobs=6. These are noisy per-cell minima from historical runs and do not
876
- show a direction either way. The shared sandbox had multi-minute CPU-starvation
877
- windows during this session (walls up to 594s); affected runs are flagged via
878
- sampler cadence and excluded from wall analysis. Absolute times are specific to
879
- this 2-core sandbox. Environment, method and the contention log are on the
880
- [benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
881
-
882
- ## πŸ—ΊοΈ Where it is going
883
-
884
- [ROADMAP.md](ROADMAP.md) lays out the plan: CI-fast runs, trusted results, a
885
- team workflow (GitHub Action, baseline, HTML report), then 1.0. saboteur stays
886
- RSpec-only. Design
887
- details are in [docs/design.md](docs/design.md), prior art in
888
- [docs/prior-art.md](docs/prior-art.md), and project status in
889
- [docs/status.md](docs/status.md).
890
-
891
- ## 🀝 Contributing
892
-
893
- See [CONTRIBUTING.md](CONTRIBUTING.md). Commits need a DCO `Signed-off-by` line.
894
-
895
- Before you ask for review, run the local preflight check on your exact head:
896
-
897
- ```sh
898
- ruby script/preflight --base <full main sha> --head <full head sha> --base-ref main --mode full
899
- ```
900
-
901
- It runs scope, sign-off, SPDX, full RuboCop and the specs, and writes a report under `tmp/preflight/`. A pass is not an approval; fast mode is partial. See [docs/preflight.md](docs/preflight.md).
902
-
903
- ## Acknowledgements
904
-
905
- Thanks to [Glomo](https://glomo.one) for providing the AI tokens used to build saboteur.
245
+ - [Changelog](CHANGELOG.md)
246
+ - [All documentation](docs/README.md) - configuration, operators, CI, recipes
247
+ - [Contributing](CONTRIBUTING.md) - every commit needs a DCO `Signed-off-by` line
248
+ - [Design notes](docs/design.md)
906
249
 
907
250
  ## License
908
251
 
909
- MIT. Copyright (c) 2026 Vamsi Pavan Mahesh Gunturu. See [LICENSE](LICENSE).
910
-
911
- Internal baseline capture blocks a subject with empty generation, including a subject for which the builtin source yields no candidates. Context comparison is an extra consistency check; file fingerprint verification normally also catches source drift. Neither check proves external state stability.
912
-
913
- ### Opt-in survivor baseline
914
-
915
- `baseline check` runs every selected mutant fresh, then compares accepted survivors. It does not skip accepted mutants, change verdicts or change scores. Use whole configured projects or `--subject` filters; diff, replay, incremental reuse and score thresholds are refused. Caps, uncovered work, empty generation, errors, resource limits, omitted malformed files, changed inputs and missing results are incomplete, even with zero survivors.
916
-
917
- ```sh
918
- saboteur baseline propose --baseline baseline.json --proposal proposal.json
919
- # Edit only document.accepted[*].reason in proposal.json. Give each survivor a review reason.
920
- saboteur baseline accept --baseline baseline.json --proposal proposal.json
921
- saboteur baseline check --baseline baseline.json
922
- ```
923
-
924
- Proposal writes independent owner-private execution evidence under the configured cache directory. Keep that directory until acceptance; deleting it prevents acceptance. Acceptance verifies the private record and current input fingerprint before atomically publishing the manifest. For replacement, give both propose and accept `--previous SHA256`, the hash you reviewed of the current baseline bytes. Output files must not overlap project inputs or the private evidence directory. A baseline normally lives in a separate JSON file outside fixture/config input trees.
925
-
926
- Check prints one JSON object: accepted, new, changed, resolved_kills, detected_timeouts, unmatched and errors are separate categories. Exit 0 means a compatible complete clean check; 1 needs survivor review; 2 means incompatible or invalid configuration; 3 means incomplete execution or tool failure. Propose and accept exit 0 only after their file operation succeeds. Context changes require fresh review. Timeouts remain a distinct detected category, not assertion kills.
927
-
928
- Score-drop gate: `saboteur baseline check --baseline baseline.json --epsilon POINTS` also fails (exit 1,
929
- state `review_required`) when the fresh score is more than POINTS percentage points below the score recorded in
930
- the baseline. Both scores use saboteur's own formula, (killed + timeout) / (killed + timeout + survived), over
931
- every candidate. The baseline already stores each candidate's status, so no new file format is needed. This
932
- catches what the survivor comparison cannot: a score that falls because killed mutants disappeared (deleted or
933
- rewritten code) while every accepted survivor is still matched. A drop of exactly POINTS passes; a rise never
934
- fails. The check prints a `score` object (`baseline`, `current`, `drop`, `epsilon`, `exceeded`) only when
935
- `--epsilon` is given. Without it, behavior is unchanged. POINTS is 0 to 100. It lives on `baseline check`,
936
- not on `run`: the delta gate needs the baseline's fresh whole-scope evidence, which `run` does not produce.
937
- It is not the same as `--threshold`, which gates an absolute score.
938
-
939
- In GitHub Actions, the action's `baseline` and `baseline-epsilon` inputs run this check; see
940
- [docs/github-action.md](docs/github-action.md).
941
-
942
- Local evidence protects against edits to the public proposal except review reasons. It is not remote attestation, a signature, or protection against the account owner changing their private files. File fingerprints cannot prove external database, network or environment stability. `config/saboteur_ignore.rb` remains a separate legacy ignore mechanism; this gate compares every survivor regardless of ignore status.
943
-
944
- A baseline that changes during `check` is rejected as configuration/evidence error (exit 2), not an incomplete execution result. Acceptance leaves a private `baseline.json.lock` beside the destination: keep it for cooperating-process publication locks. It contains no evidence or results. Symlinked checkout parents are normalized; symlink artifacts themselves and private-store/lock symlinks are still refused.
945
-
946
- The builtin `endpoint_replacement` operator swaps explicit, no-argument `first`/`last` calls. It targets wrong-end/order-selection faults in collection handling. Argument-bearing slices, bare calls and block-bearing APIs are left unchanged. Receivers are not statically typed: a custom object may implement either name differently, so a crash remains a crash kill, not proof of a collection assertion.
947
-
948
- JSON findings are ordered by a deterministic [review ranking](docs/survivor-ranking.md): coarse first, then covering-test count, edited-line impact and changed-line proximity, with file/line/digest ties. The terminal groups survivors by file, method and line first; the ranking only orders the survivors on one line, and every survivor is listed, in review mode too. Only consecutive survivors with matching evidence share a test/evidence block; repeated evidence after another group is printed again to keep that order. Ranking does not change scores or execution.
949
-
950
- Cache and baseline identities include the policy. Re-propose and review old survivor baselines with `baseline propose`; never auto-accept new survivors. Mutation digests for unchanged edits stay stable.
951
-
952
- ### Preview and choose operators
953
-
954
- `bundle exec saboteur run --dry-run` discovers source methods and generates candidates without booting the app, running tests, forking, or writing reports/caches. It honors `--since`, `--subject` and source arguments. It prints per-method counts and cap omissions. After a matching normal run, a cached estimate shows baseline covering-test work (cached original-test durations times planned mutants; it is not a bound on mutant runtime and excludes boot and fork costs). If any file cannot be parsed, the total is marked INCOMPLETE and no estimate is given.
955
-
956
- Trust boundary: `--dry-run` never boots the app or runs its tests, but custom `sources` entries (`path.rb:ClassName`) are still `require`d to generate candidates, so their code runs. Only preview with sources you trust.
957
-
958
- `bundle exec saboteur run --list-operators` lists every builtin name and description. All builtin operators run by default, including [`range`](docs/operators/range.md), [`operator_assignment`](docs/operators/operator_assignment.md), and [`condition_negation`](docs/operators/condition_negation.md). `--operators arithmetic,comparison` or `c.operators = ["arithmetic", "comparison"]` still selects a subset; `all` selects every name. There is no opt-in tier. `--exclude-operators range,condition_negation` or `c.exclude_operators = ["range", "condition_negation"]` removes named operators from that selection; exclusion wins if a name is in both lists. An empty selection means all, while an empty exclusion means none. CLI lists replace their corresponding config lists. These controls apply to builtin operators, not custom source plugins. Selection and exclusions invalidate cached verdict identity and are carried into agent rerun/replay commands.
959
-
960
-
961
- All operators run unless you select a smaller set; exclusions always remove operators from that set.
962
- Repeated `--operators` or `--exclude-operators` flags use the last list, not a union.
963
- To re-enable an operator excluded in config: `bundle exec saboteur run --operators range --exclude-operators ''`.
964
- To keep the 0.5.0 default inventory: `bundle exec saboteur run --exclude-operators range,operator_assignment,condition_negation`.
965
- The old `core` and `extended` names now give a migration error, not a changed selection in disguise.
966
- An empty effective selection is a usage error before boot. `--list-operators` loads current config and shows on/off plus examples; preview and the run header show the effective set.
967
-
968
- ```ruby
969
- Saboteur.configure do |c|
970
- c.operators = [] # all by default; or ["arithmetic", "range"] for a subset
971
- c.exclude_operators = ["condition_negation"]
972
- end
973
- ```
974
-
975
- Inspired by mutineer's --dry-run, --list-operators and --operators tiers (github.com/davidteren/mutineer, MIT, by David Teren).
976
-
977
- ### Agent survivor hints and score history
978
-
979
- `--agent` findings include operator-specific `hint` suggestions and a shared
980
- `hintCaveat`. Hints do not diagnose a missing test or assertion; equivalent
981
- mutants and limited observation are still possible. Unknown operators have no
982
- hint unless marked coarse. The normal JSON byte budget still applies.
983
-
984
- Use `--score-history PATH` (or `c.score_history = PATH` in your config) to append
985
- one score-history JSONL record per finished report. History is off by default;
986
- changes are printed to stderr, not agent stdout. See [score definitions and
987
- history limits](docs/scores.md). The file can be uploaded by your own artifact
988
- step; the bundled Action's upload behavior is unchanged.
989
-
990
- Score history now refuses non-regular files, symlink paths and report/history
991
- inode aliases before running the engine or writing reports. Imported records
992
- have a 64 KiB per-line bound and strict validation of supported version 1
993
- and version 2 records, including counts, scores and duplicate keys. History is an output, excluded from baseline
994
- fingerprints. See [history safety and comparison limits](docs/scores.md).
995
-
996
- Operator policy version 2, selected names, excluded names and effective names are in JSON/agent/baseline evidence and fresh score-history v2 rows. Old history v1 rows remain readable but are not compared across policy versions.
997
- ### Inline suppression comments
998
-
999
- `total = a + b # saboteur:disable-line arithmetic reason="rounding checked elsewhere"` suppresses edits starting on that line, restricted to the named operators. Omit names to suppress all operators. Multiple names are space-separated. Only actual Ruby comments count; directive text in strings does not. A missing or blank reason or unknown operator is a usage error naming file:line. Terminal, JSON and HTML list every suppressed candidate with its reason and digest, outside the score. Moving the commented code keeps the suppression. Multi-line edits use their starting line.
1000
-
1001
- Inspired by mutineer's inline suppression comments (github.com/davidteren/mutineer, MIT, by David Teren).
1002
-
1003
- Inline suppressions bound completeness: classic scores still exclude suppressed candidates, but baseline evidence is incomplete when any inline candidate is suppressed. Agent output reports `inline_suppressed` with its count and points to the full audit. Reasons are audited, not proof that suppressed work was checked.
1004
-
1005
- Source-only `--dry-run` lists the suppressed count and each audit row in selected methods, before diff restriction and the per-method cap. They are outside the planned total. Preview still does not boot the app or write reports.
252
+ [MIT](LICENSE).