saboteur 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +113 -1
  3. data/README.md +248 -63
  4. data/lib/generators/saboteur/install/install_generator.rb +21 -0
  5. data/lib/saboteur/agent_skill.rb +40 -0
  6. data/lib/saboteur/cli.rb +150 -22
  7. data/lib/saboteur/config.rb +27 -15
  8. data/lib/saboteur/config_file.rb +97 -0
  9. data/lib/saboteur/custom_map.rb +170 -0
  10. data/lib/saboteur/doctor.rb +17 -1
  11. data/lib/saboteur/engine/baseline_guard.rb +20 -3
  12. data/lib/saboteur/engine/boot.rb +9 -0
  13. data/lib/saboteur/engine/coverage_store.rb +31 -4
  14. data/lib/saboteur/engine/data_inputs.rb +43 -0
  15. data/lib/saboteur/engine/diagnostics.rb +28 -0
  16. data/lib/saboteur/engine/diff_scope.rb +8 -0
  17. data/lib/saboteur/engine/run_fingerprint.rb +34 -6
  18. data/lib/saboteur/engine/scheduler.rb +7 -16
  19. data/lib/saboteur/engine/suite_files.rb +8 -1
  20. data/lib/saboteur/engine.rb +74 -13
  21. data/lib/saboteur/execution/reevaluator.rb +7 -4
  22. data/lib/saboteur/execution/supervisor.rb +23 -11
  23. data/lib/saboteur/frameworks/rails.rb +329 -13
  24. data/lib/saboteur/help.rb +33 -17
  25. data/lib/saboteur/ignore_list.rb +71 -17
  26. data/lib/saboteur/install.rb +132 -0
  27. data/lib/saboteur/integration/rspec.rb +47 -3
  28. data/lib/saboteur/mutation_source/argument_removal.rb +86 -0
  29. data/lib/saboteur/mutation_source/builtin.rb +8 -2
  30. data/lib/saboteur/mutation_source/pair_removal.rb +101 -0
  31. data/lib/saboteur/replay.rb +1 -1
  32. data/lib/saboteur/reporter/agent.rb +87 -0
  33. data/lib/saboteur/reporter/agent_clipping.rb +29 -0
  34. data/lib/saboteur/reporter/agent_commands.rb +61 -0
  35. data/lib/saboteur/reporter/agent_incomplete.rb +37 -0
  36. data/lib/saboteur/reporter/agent_summary.rb +131 -0
  37. data/lib/saboteur/reporter/findings.rb +8 -0
  38. data/lib/saboteur/reporter/html.rb +49 -1
  39. data/lib/saboteur/reporter/stryker.rb +11 -4
  40. data/lib/saboteur/reporter/terminal.rb +65 -2
  41. data/lib/saboteur/result.rb +46 -5
  42. data/lib/saboteur/score.rb +89 -0
  43. data/lib/saboteur/version.rb +1 -1
  44. data/lib/saboteur/working_tree.rb +48 -0
  45. data/lib/saboteur.rb +4 -0
  46. metadata +17 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 767101a27863dfddcfd14ed0f71737ba34764feebd1824625c2354119f5144cc
4
- data.tar.gz: 237fca7fc890cf682a4f4b10f2e2fbdfc23759d575ba5dfd804d60bc28febcb3
3
+ metadata.gz: a5c52cb7485073b569359bf940160095370103747d55a6065b7f067726a1a53c
4
+ data.tar.gz: 40fca0c1c0f8391de86d7f17960d908add226682d93be2610775fc888ab36f60
5
5
  SHA512:
6
- metadata.gz: 2b7e6f93cd8844cf886905e009157a7227389b1a11f47ead943f8075d52e45c8533925b60ee04c41e9678039090c0072b3ad19f64a7ea5135b8e311ada65e2bd
7
- data.tar.gz: 24faf1ebcfecaa018a4b415f86ceb40c73c2889fe49ff65718fba3e9709971e4c431756a6e4990436bc1fefbdb1fb8cdee72bf66953d5d76aa02d5b166435b70
6
+ metadata.gz: 061e0b5a97c95048048767d3724ea883ea144e7530b9a52631d52c896f4154e1a63f7b65ef7b5817053a42ab31eb97c168aa3079c87e9d9a3e8be1ae949d7bdf
7
+ data.tar.gz: e29bcfafdaf2fc7ecffbc4f8746349d79c908dfc2de1530998b525c691135c2bfb4ade67575374178742b1fddb29ab96a7d9ff67656d29dbd7c096ba0b50d0dc
data/CHANGELOG.md CHANGED
@@ -6,6 +6,118 @@ All notable changes are documented here. The format follows
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.4.0] - 2026-10-12
10
+
11
+ ### Changed
12
+ - **Breaking:** configuration moved from YAML to Ruby. `.saboteur.yml` becomes
13
+ `config/saboteur.rb` (`Saboteur.configure do |c| ... end`, same key names),
14
+ and `.saboteur-ignore.yml` becomes `config/saboteur_ignore.rb`
15
+ (`Saboteur.ignore do |i| i.mutation "<digest>", subject:, reason: end`,
16
+ reason still mandatory). A legacy YAML file stops the run with a pointer to
17
+ the new file instead of being read. Unknown settings fail with the line
18
+ number and a did-you-mean suggestion. `rails generate saboteur:install`
19
+ writes `config/saboteur.rb` (never overwriting without `--force`);
20
+ `config/saboteur_ignore.rb` and `config/saboteur_custom_map.rb` are optional
21
+ and created by hand when needed.
22
+ - A run in which more than half of the executed mutants errored now exits 3
23
+ (the tool failed). Before, only a run where every executed mutant errored
24
+ did. Errored mutants are outside the score, so such a run tested almost
25
+ nothing. This applies with and without `--threshold`. The JSON report records the reason under
26
+ `saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`, `errorMajority`,
27
+ `incompleteUnderThreshold`; in `--agent` mode `ok`, `survivors`,
28
+ `errorMajority` or `incomplete`).
29
+ - A whole-suite coverage build that records no lines for any included file
30
+ now exits 3 and the empty map is not cached, instead of testing nothing.
31
+ With `--test`, an empty map stays a legitimate "no coverage" outcome.
32
+ - Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
33
+ `db/schema.rb` and `config/*.yml` are part of the coverage and verdict
34
+ cache fingerprint, so a changed fixture or schema invalidates cached
35
+ results. The new `fingerprint_extra` setting adds other globs whose
36
+ contents should invalidate caches, for example `data/*.csv`.
37
+ - The baseline and coverage phases show progress and write real logs, so a
38
+ slow or failing phase can be diagnosed from the output.
39
+ - When the baseline child is killed by a signal, the message names it
40
+ (`SIGKILL` adds a possible-OOM hint) instead of reporting a bare failure.
41
+
42
+ ### Removed
43
+ - The coarse-survivor gate: a surviving whole-body mutant no longer skips the
44
+ method's fine mutants; every eligible fine mutant runs.
45
+
46
+ ### Fixed
47
+ - Mutations inside private `module_function` helpers are preserved when the
48
+ method is re-evaluated; they previously reverted and reported as false
49
+ survivors.
50
+ - Rails apps that test against in-memory SQLite (`:memory:`) no longer fail
51
+ in forked workers with "Could not find table": saboteur never disconnects,
52
+ renames or copies an in-memory pool. Each in-memory database is
53
+ snapshotted to a temporary file after boot, and every forked child
54
+ restores the snapshot into its own fresh database.
55
+ - With a Rails app under Bootsnap, Coverage no longer sees an empty map:
56
+ saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its own process when
57
+ the variable is unset, because bytecode from Bootsnap's compile cache
58
+ carries no coverage counters.
59
+ - A relative SQLite `database:` path resolves against `Rails.root` (as Rails
60
+ 7.1+ does) for the per-worker copy and its cleanup. The copy uses SQLite's
61
+ backup API after a WAL checkpoint, refuses to run when a reader keeps the
62
+ WAL busy, and a missing source file is an error rather than a skipped copy.
63
+ - Per-worker database copies (`jobs` above 1) are dropped when the run ends,
64
+ best effort: a failed drop is a warning and never changes the outcome. A
65
+ copy left by a killed run is reclaimed by the next run. A failed copy
66
+ names the sessions blocking it.
67
+ - For `rails: true`, the app's `spec/` directory is on the load path before
68
+ `require` is processed, as with rspec, so a boot file such as a dummy app's
69
+ `spec/rails_helper.rb` can require its siblings.
70
+ - When Rails did not load, the error points engine users at the dummy app's
71
+ environment file or helper (`DummyApp.setup` in `spec/rails_helper.rb`).
72
+
73
+ ### Added
74
+ - `rails generate saboteur:install` and a Rails quickstart in the README. The
75
+ generator writes `config/saboteur.rb` and never overwrites without `--force`.
76
+ - The warning about threads alive when a worker forks names those threads,
77
+ falling back to their backtraces for unnamed ones.
78
+ - `--test` no longer requires source operands when `--since` is given: the
79
+ changed files are the sources, so `saboteur run --since origin/main --test
80
+ spec/billing_spec.rb` scopes a pull-request run to chosen specs. A `--since`
81
+ run without `--test` now warns before it starts that the two baseline passes
82
+ run the loaded suite (a cold coverage cache adds one more pass), with the
83
+ spec-file count and how to scope them, instead of going quiet for hours on a
84
+ large app.
85
+ - `config/saboteur_custom_map.rb` declares explicit source-file to spec-file
86
+ mappings (`SABOTEUR_CUSTOM_MAP`): exact paths, globs (the more specific glob
87
+ wins; equal specificity is an error) and Regexps with capture substitution.
88
+ Invalid maps fail the run with the file and key; stale and duplicate keys
89
+ warn on runs, and `saboteur doctor` fails on stale keys. The map is parsed
90
+ and validated only for now - it does not change which specs run yet.
91
+
92
+ - An integration target can declare a known error profile
93
+ (`allow_error_fraction` per strategy in integration/targets.yml): the
94
+ majority-error exit is accepted only when the run's own fresh report
95
+ carries the CLI's `errorMajority` marker, the errored fraction stays
96
+ within the bound, and no mutant hit a resource limit. The JSON report
97
+ records the run's exit cause under `saboteur.exitCause`.
98
+ - A survivor lists the first five tests that covered its line and still passed
99
+ with the mutant applied, in the terminal and the HTML page.
100
+ - In diff mode (`--since`), each survivor names the changed line it relates to
101
+ (`changed line: path:N` in the terminal, `changedLines` in each JSON finding),
102
+ because a multi-line statement is reported at its first line.
103
+ - The score is split into assertion kills and crash kills, in the terminal,
104
+ the JSON report (`saboteur.scores`) and the HTML page. A second score leaves
105
+ out whole-body mutants (`body_nil`, `body_true`, `body_false`, `body_raise`). The headline score and
106
+ `--threshold` are unchanged.
107
+ - `saboteur run --agent` (`--format agent`): one bounded JSON summary (under 8 KiB) on stdout
108
+ with status, verdict, provenance, the first three findings and the commands for full
109
+ detail. Exit 0 means a complete, clean run only: 1 survivors, 2 usage or config error,
110
+ 3 incomplete or failed. `--threshold` cannot be combined with it. The next commands are shell-escaped and keep the
111
+ full selection; the working-tree state is read before the run; target output stays on stderr. `saboteur agent-skill`
112
+ prints an AGENTS.md block for coding agents (also in `docs/agent-skill.md`). Classic
113
+ exit codes are unchanged.
114
+ - Fine `pair_removal` mutants for hash literals and keyword lists, including
115
+ sole associations, with token-based separators and the normal method cap.
116
+ - Fine `argument_removal` mutants: remove one ordinary argument from a
117
+ parenthesized call while preserving keywords and blocks. Token-based comma
118
+ edits respect comments and strings. Existing candidate validation, order and
119
+ per-method caps apply. Unparenthesized calls and heredoc arguments are skipped.
120
+
9
121
  ## [0.3.0] - 2026-10-11
10
122
 
11
123
  ### Added
@@ -64,7 +176,7 @@ First release that does mutation testing. MIT licensed.
64
176
  ### Added
65
177
  - `saboteur run` with `--since`, `--subject`, `--timeout-factor`, `--jobs`,
66
178
  `--reeval`, `--json`, `--no-cache`, `--incremental`, `--config`.
67
- - Subjects found with Prism; mutations are byte-range edits: whole-method
179
+ - Subjects found with Prism; mutations are byte-range edits: whole-body
68
180
  `nil`/`true`/`false`/`raise` first, then comparison, boolean, arithmetic,
69
181
  numeric literal, statement removal, condition, negation, safe navigation
70
182
  and `||=`.
data/README.md CHANGED
@@ -31,14 +31,22 @@ fail?" for many small breakages and lists the ones nobody noticed. Read the
31
31
 
32
32
  ### The unchanged suite is checked first
33
33
 
34
- Every run with selected subjects runs the entire unchanged suite twice in fresh
34
+ Every run with selected subjects runs the loaded unchanged suite twice (the specs chosen with `--test`, if given) in fresh
35
35
  supervised children before coverage lookup or incremental verdict reuse. This
36
36
  costs two suite runs even on a cache hit, plus a coverage-building suite run on
37
37
  a cache miss. Both baseline passes must be green, complete and in the same
38
- RSpec order at the run seed. A failure exits 3, stops mutation and names the log and fix. Each suite pass
38
+ RSpec order at the run seed. A failure exits 3, stops mutation and names the log and fix. Long phases
39
+ print their elapsed time about every 30 seconds, and each phase's log holds
40
+ the suite's progress and full failure detail. Each suite pass
39
41
  uses `coverage_timeout` and configured memory/process limits (the suite memory
40
42
  budget is four times `max_memory_mb`, as for coverage). Older benchmark numbers
41
43
  need a rerun before they describe this extra fixed cost.
44
+ A whole-suite coverage build that records no lines for any included file
45
+ means the setup hid the app from Coverage (Bootsnap's compile cache is the
46
+ common cause), so the run fails with exit 3 instead of testing nothing, and
47
+ the empty map is not cached. With `--test` an empty map instead stays a
48
+ legitimate "no coverage" outcome, because the chosen specs may not touch the
49
+ selected sources.
42
50
  Two passes detect observed instability, not every possible flaky test. See
43
51
  [baseline stability](docs/baseline-stability.md) for limits and hook behaviour.
44
52
 
@@ -57,9 +65,8 @@ bundle exec saboteur run --subject 'YourClass#your_method'
57
65
  ```
58
66
 
59
67
  Outside a Bundler project, `gem install saboteur` installs the same executable.
60
- Rails apps need a Rails block in `.saboteur.yml` (boot file, `app/**/*.rb`,
61
- `rails: true`); see [Rails apps](#rails-apps) below. The defaults are for plain
62
- gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
68
+ Rails app? Follow the [Rails quickstart](#rails-apps) below: the defaults are for
69
+ plain gems (`lib/**/*.rb`, `rails: false`, `jobs: 1`) and are not Rails-aware.
63
70
 
64
71
  Start with one method. You get counts per status, a score, timing, and a
65
72
  unified diff for every survivor, so you can read the exact change your tests
@@ -74,22 +81,24 @@ line coverage touched the changed statement. Tests that killed earlier mutants
74
81
  of the same method go first, then the fastest, and the first failure stops the
75
82
  run. A change no test reaches is reported as "no coverage", not as a survivor.
76
83
 
77
- **It does not waste time on dead ends.** Each method first gets whole-body
78
- mutants (`nil` and `raise`, or `true`/`false` for predicates; a few methods,
79
- such as some that rescue or log, get none). If your suite does not notice
80
- that, saboteur says so and skips the fine-grained mutants for that method. Subjects with no tests
81
- are skipped before any mutants are generated, and each method is capped at
82
- `max_mutants_per_method` mutants (100 by default).
84
+ **It runs coarse mutants first, then fine mutants.** Each method first gets
85
+ whole-body mutants: `nil` (`true`/`false` for predicates) unless the body is empty or
86
+ only logs, and `raise` unless a catch-all rescue would absorb it. Fine-grained mutants run
87
+ regardless of the coarse verdict, including a survivor or a reused verdict.
88
+ Deleting a body unnoticed does not prove its smaller changes are harmless.
89
+ Subjects with no tests are skipped before any mutants are generated, and each
90
+ method is capped at `max_mutants_per_method` mutants (100 by default). Running
91
+ fine mutants after coarse survivors can increase run time and change scores.
83
92
 
84
93
  **It prepares memory before it forks.** Before workers fork, saboteur warms and
85
94
  compacts the Ruby heap ([`ForkReadiness`](lib/saboteur/execution/fork_readiness.rb)),
86
95
  which is meant to let workers share more pages with their parent and copy less.
87
- The physical-memory benefit is not yet established by a published benchmark.
96
+ Observed on the pinned hashie workload (paired runs, below): about 17-18% lower peak PSS at 2 and 6 workers. Not a general guarantee.
88
97
 
89
98
  **It is aimed at pull requests.** `--since origin/main` mutates only the methods
90
99
  your branch changed: fine-grained mutants on the changed lines, plus the
91
100
  whole-body mutants of those methods. On one real change in saboteur's own history, that took a file
92
- from 139 mutants to 7, and median wall time fell 22.9%. The time saving is
101
+ from 139 mutants to 7, and median wall time fell 22.9% (measured at d2b7abb, before the coarse-survivor gate was removed; a rerun is pending). The time saving is
93
102
  smaller than the mutant count because building the coverage map is a fixed
94
103
  cost, and this is one file, not a Rails app, so it is not a general speed
95
104
  claim. The full method, raw timings and
@@ -97,8 +106,8 @@ caveats are in [bench/since](bench/since/README.md).
97
106
 
98
107
  **It checks its own kills.** After a method's mutants run, saboteur re-runs the
99
108
  killing tests against the *unchanged* method. If they fail there too, the kill
100
- is not credited and becomes an error. In our comparison on lobsters, 0 of 20
101
- checked kills were fake. Every mutant runs in a supervised child with deadline,
109
+ is not credited and becomes an error. In one comparison on lobsters, 0 of 20
110
+ randomly sampled kills were fake. Every mutant runs in a supervised child with deadline,
102
111
  memory and process limits, and saboteur tracks and kills the processes it
103
112
  starts.
104
113
 
@@ -124,17 +133,20 @@ Positional operands name the sources to mutate: Ruby files, or directories
124
133
  (all their `.rb` files). Each must exist; a typo or a non-Ruby file exits 2
125
134
  before anything runs. They replace `include` and `exclude`. `--test`
126
135
  (repeatable; spec files or directories, with an optional `:LINE` or `[1:2]`
127
- selector; needs at least one source) replaces only the files RSpec runs. Other
136
+ selector; needs at least one source, unless `--since` is given - the changed
137
+ files are the sources) replaces only the files RSpec runs. Other
128
138
  `rspec_args` and `.rspec` options such as `--require` stay. The coverage map
129
139
  is built from only those specs, each mutant runs only the ones that cover it,
130
140
  and the selection is part of the coverage and `--incremental` cache keys.
131
141
 
132
- `--threshold PERCENT` (0 to 100, or `threshold:` in `.saboteur.yml`) is for CI.
142
+ `--threshold PERCENT` (0 to 100, or `c.threshold` in `config/saboteur.rb`) is for CI.
133
143
  The run exits 1 when the score is below PERCENT and 0 when it is at or above,
134
144
  even with survivors. It fails closed: a run that tested no mutant has no score
135
145
  and exits 1, and a run where any mutant errored or hit a resource limit exits 3,
136
146
  because the score covers only part of the run. Without a threshold, any
137
- unignored survivor exits 1.
147
+ unignored survivor exits 1. Either way, a run where more than half of the
148
+ executed mutants errored exits 3: errored mutants are outside the score,
149
+ so such a run tested almost nothing.
138
150
 
139
151
  Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
140
152
  `--config PATH`, `--integration NAME`, `--reeval class_eval|nesting`,
@@ -142,7 +154,11 @@ Other flags: `--jobs N` (concurrent workers), `--timeout-factor N`,
142
154
 
143
155
  For pull requests, start with `--since` against your target branch. This is a
144
156
  workflow recommendation, not a CLI default: plain `run` still checks the whole
145
- configured project. Keep whole-project runs on main: diff mode does not check every method affected by a
157
+ configured project. To scope the specs as well, pass them with `--test` and no
158
+ source operands: `bundle exec saboteur run --since origin/main --test spec/billing_spec.rb`.
159
+ With `--since` and no `--test`, the run warns before it starts that the two baseline passes
160
+ run the loaded suite (a cold coverage cache adds one more pass), and how to
161
+ scope them. Keep whole-project runs on main: diff mode does not check every method affected by a
146
162
  dependency change. [The pull-request guide](docs/pull-requests.md) covers
147
163
  checkout history, a GitHub Actions example, configuration and limits.
148
164
 
@@ -152,7 +168,7 @@ mutation reports or coverage/verdict caches. Target boot files and tests still
152
168
  run, so their normal side effects remain. See the [doctor guide](docs/guides/doctor.md).
153
169
 
154
170
  `--id` runs exactly one current mutant with its coverage-selected tests, bypassing
155
- per-method caps, coarse gating, ignore suppression and incremental verdict reuse.
171
+ per-method caps, ignore suppression and incremental verdict reuse.
156
172
  It still honors `--subject` and `--since` changed-line selection. See the
157
173
  [replay guide](docs/guides/replay.md) for digest format, checks and limits.
158
174
 
@@ -167,16 +183,42 @@ unchanged. It cannot see a change in a method this one calls, so run without
167
183
  it now and then (and in CI on the main branch).
168
184
 
169
185
  Output: counts per status, the score, timing, and a unified diff for every
170
- survivor. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
186
+ survivor. Under each diff the terminal lists the tests that covered that line
187
+ and still passed with the mutant applied (the first five, likeliest killers
188
+ first; the rest are under `coveredBy` in the JSON report). The missing
189
+ check may belong in one of them. Line coverage can overstate which tests
190
+ exercise a line, so treat the list as a lead. A survivor reused with
191
+ `--incremental` was not run again, and the list says its tests passed in an
192
+ earlier run. The HTML page shows the same list in the
193
+ survivor's notes. The JSON report goes to `.saboteur/report.json` and a self-contained HTML
171
194
  report (one file, no scripts, no network) to `.saboteur/report.html`. Change
172
- the paths with `--json` / `--html` or `json_report` / `html_report` in
173
- `.saboteur.yml`; set either to `null` to turn that report off. The two paths
195
+ the paths with `--json` / `--html` or `c.json_report` / `c.html_report` in
196
+ `config/saboteur.rb`; set either to `nil` to turn that report off. The two paths
174
197
  must differ. The HTML page shows the covered mutation score
175
198
  `(killed + timeout) / (killed + survived + timeout)`, which leaves NoCoverage
176
199
  out and keeps survivors accepted on the ignore list in the denominator. A tool
177
200
  that reads the JSON report may show a different score, because the schema counts
178
201
  NoCoverage as undetected.
179
202
 
203
+ The score is split by how the mutant was killed. A crash kill (a `NoMethodError`,
204
+ a `TypeError`, a parse error) shows the code ran and broke, not that a test
205
+ checked a value. The terminal prints
206
+ `score 100.0 % (assertion 50.0 %, crash 50.0 %)`: each share is a part of
207
+ tested mutants, so the parts add up to the score. Timeouts and kills with no
208
+ recorded cause (a verdict reused from an older run, a test that called `exit`,
209
+ a failure outside any example) are named when there are some. Whole-body
210
+ mutants (`body_nil`, `body_true`, `body_false`, `body_raise`) are killed by
211
+ almost any test that runs the method, so a second line scores the run without
212
+ them, and shows them alone. The headline score and `--threshold` still count every
213
+ mutant. The JSON report carries both splits under `saboteur.scores`
214
+ (`all` and `withoutWholeBodyMutants`), and the HTML page prints them under
215
+ the score. The JSON also records why the run exited the way it did under
216
+ `saboteur.exitCause` (`ok`, `survivors`, `belowThreshold`,
217
+ `errorMajority`, `incompleteUnderThreshold`). In `--agent` mode the cause
218
+ follows the agent verdict, so the report and the process exit classify the
219
+ run the same way: `ok` for clean, `survivors` for survivors, and otherwise
220
+ `errorMajority` when most mutants errored or `incomplete`.
221
+
180
222
  For code-review surfaces, the JSON report also carries a `saboteur.findings`
181
223
  section: a bounded, deterministically ordered list of survivors, coarse
182
224
  mutants first (a surviving coarse mutant hints that no test pins the method's
@@ -188,6 +230,15 @@ Survivors accepted on the ignore list are listed separately with their
188
230
  reason, and `survivorsTruncated` and `acceptedTruncated` count what the bound
189
231
  left out; the full data stays in the schema's `files` section.
190
232
 
233
+ In diff mode (`--since`), a mutant is reported at the line where its statement
234
+ starts. A statement that spans several lines can start a line or two above the
235
+ line that changed. Under each survivor the terminal therefore prints
236
+ `changed line: path:N`, the changed line or lines (the first three) the mutant
237
+ touches, and each finding carries them as `changedLines`. Use `changedLines`,
238
+ not `line`, to put a note on a changed line of the pull-request diff. A
239
+ whole-body mutant of a method whose only change is its signature touches no
240
+ changed line and prints none.
241
+
191
242
  Exit codes: 0 no survivors (or all on the ignore list), or the score meets
192
243
  `--threshold`; 1 survivors, or the score is below `--threshold` or missing;
193
244
  2 bad configuration or a missing source or test; 3 the tool failed (with
@@ -203,38 +254,126 @@ other errors and reused cached kills are not included. See
203
254
 
204
255
  ## ⚙️ Configuration
205
256
 
206
- `.saboteur.yml` in the project root; CLI flags override it. Keep the shared
257
+ `config/saboteur.rb` in the project root; CLI flags override it. Keep the shared
207
258
  config usable for whole-project runs and pass `--since` in the pull-request
208
- job. A PR-only config may set `since: origin/main`; omit `since` (or use
209
- `since: null`) in the config for audits on main. Use your actual target branch.
210
-
211
- ```yaml
212
- include: [lib/**/*.rb]
213
- exclude: [lib/my_gem/version.rb]
214
- require: [] # boot files, e.g. ./config/environment
215
- reeval: nesting # or class_eval; see docs/design.md 6.1
216
- timeout_factor: 3 # per-mutant deadline = factor x its tests' recorded time
217
- timeout_min: 5
218
- max_memory_mb: 2048 # memory growth cap per mutant process tree
219
- max_processes: 64
220
- max_mutants_per_method: 100 # positive integer; raise it for a broader run
221
- ignore_file: .saboteur-ignore.yml
222
- seed: 1 # Ruby's random seed in every test run
259
+ job. A PR-only config may set `c.since = "origin/main"`; omit `since` (or use
260
+ `c.since = nil`) in the config for audits on main. Use your actual target branch.
261
+
262
+ ```ruby
263
+ Saboteur.configure do |c|
264
+ c.include = ["lib/**/*.rb"]
265
+ c.exclude = ["lib/my_gem/version.rb"]
266
+ c.require = [] # boot files, e.g. ./config/environment
267
+ c.reeval = "nesting" # or "class_eval"; see docs/design.md 6.1
268
+ c.timeout_factor = 3 # per-mutant deadline = factor x its tests' recorded time
269
+ c.timeout_min = 5
270
+ c.max_memory_mb = 2048 # memory growth cap per mutant process tree
271
+ c.max_processes = 64
272
+ c.max_mutants_per_method = 100 # positive integer; raise it for a broader run
273
+ c.ignore_file = "config/saboteur_ignore.rb"
274
+ c.seed = 1 # Ruby's random seed in every test run
275
+ c.fingerprint_extra = [] # extra globs whose contents invalidate caches, e.g. ["data/*.csv"]
276
+ end
223
277
  ```
224
278
 
225
279
  ### Rails apps
226
280
 
227
- The defaults target a plain gem, so a Rails app needs this in `.saboteur.yml`:
281
+ Quickstart for a Rails + RSpec app. The defaults target a plain gem, so a Rails
282
+ app needs a few settings; the generator writes them for you.
283
+
284
+ **1. Add the gem**
285
+
286
+ ```sh
287
+ bundle add saboteur --group test
288
+ ```
289
+
290
+ **2. Generate the setup**
291
+
292
+ ```sh
293
+ bundle exec rails generate saboteur:install
294
+ ```
295
+
296
+ This writes `config/saboteur.rb`, a `bin/saboteur` binstub (it sets `RAILS_ENV=test`
297
+ for you) and a `.gitignore` entry for `.saboteur/`. It never overwrites an
298
+ existing file unless you pass `--force`, and never edits your Gemfile.
228
299
 
229
- ```yaml
230
- include: [app/**/*.rb]
231
- require: [./config/environment] # boots the app
232
- rails: true # reconnects ActiveRecord after fork, per-worker databases
233
- jobs: 4 # optional; the default is 1
300
+ Prefer to do it by hand, or on a version without the generator? Put this in
301
+ `config/saboteur.rb`:
302
+
303
+ ```ruby
304
+ Saboteur.configure do |c|
305
+ c.include = ["app/**/*.rb"]
306
+ c.require = ["./config/environment"] # boots the app
307
+ c.rails = true # reconnects ActiveRecord after fork, per-worker databases
308
+ c.jobs = 4 # optional; the default is 1
309
+ end
234
310
  ```
235
311
 
236
- Run with `RAILS_ENV=test` and turn SimpleCov off for the run. An install
237
- generator is planned; until then this block is the supported setup.
312
+ **3. Check the setup**
313
+
314
+ ```sh
315
+ bin/saboteur doctor
316
+ ```
317
+
318
+ Doctor boots your app, checks the config and the test database host, and runs
319
+ your suite twice before any mutation. Fix whatever it reports first. Turn
320
+ SimpleCov off for the run if you use it.
321
+
322
+ **4. Test what your branch changed**
323
+
324
+ ```sh
325
+ bin/saboteur run --since main
326
+ ```
327
+
328
+ It mutates only the methods changed since `main` and reports each survivor with
329
+ its diff: a change your tests did not notice. Write a test that fails on that
330
+ diff, then run again. Use `--since origin/main` in CI. Without the generator, use
331
+ `bundle exec saboteur` and `RAILS_ENV=test` instead of `bin/saboteur`.
332
+
333
+ saboteur puts the app's `spec/` directory on the load path before
334
+ processing `require`, matching rspec, so a boot file like a dummy app's
335
+ `spec/rails_helper.rb` can require its siblings.
336
+
337
+ With `jobs` above 1 on SQLite, each extra worker gets a copy of the test
338
+ database made through SQLite's backup API, so the copy is a consistent
339
+ snapshot. A relative `database:` path resolves against `Rails.root`, the
340
+ same as Rails 7.1+ does, for the copy and for its cleanup. The copy
341
+ checkpoints the source first (`PRAGMA wal_checkpoint(TRUNCATE)`) and
342
+ refuses to run when a reader keeps the WAL busy, because copying on
343
+ would silently miss committed rows. A source file that does not exist is
344
+ an error, not a skipped copy. An in-memory SQLite database (`:memory:`) is
345
+ never copied, renamed or disconnected. After the app boots, each in-memory
346
+ database is snapshotted to a temporary file through SQLite's backup API,
347
+ and every forked child restores the snapshot into its own fresh database -
348
+ a child cannot keep the parent's connection (ActiveRecord discards
349
+ inherited pools and sqlite3 closes inherited writable handles), so without
350
+ the snapshot it would see an empty database.
351
+
352
+ For a Rails engine, point `require` at the dummy app's environment -
353
+ `test/dummy/config/environment.rb`, usually loaded for you by
354
+ `spec/rails_helper.rb`. Only a dynamically created dummy app has no
355
+ environment file; there, require the helper that calls setup instead,
356
+ such as `DummyApp.setup` in `spec/rails_helper.rb`.
357
+
358
+ Bootsnap needs no extra setting: bytecode from its compile cache carries no
359
+ coverage counters, so saboteur sets `DISABLE_BOOTSNAP_COMPILE_CACHE=1` for its
360
+ own process when the variable is not already set (Bootsnap treats any set
361
+ value as off) and runs the eager load with Coverage active. App code is seen
362
+ by Coverage with the app unchanged.
363
+
364
+ With `jobs: 4` (or more), each extra worker gets a copy of every test
365
+ database, named `<database>_saboteur2` and up, made with
366
+ `CREATE DATABASE ... TEMPLATE` on Postgres or a file copy on SQLite.
367
+ saboteur drops the copies when the run ends, on a best-effort basis: a
368
+ drop that fails is reported as a warning and never changes the run's
369
+ outcome. A copy left behind - by a failed drop or a killed run - is
370
+ reclaimed by the next run, whose prepare starts from
371
+ `DROP DATABASE IF EXISTS`. MySQL is not supported for `jobs` above 1.
372
+
373
+ Rails fixtures (`spec/fixtures`, `test/fixtures`), `db/structure.sql`,
374
+ `db/schema.rb` and `config/*.yml` (including `database.yml`) are part of the
375
+ cache fingerprint automatically. Use `fingerprint_extra` for any other file your
376
+ tests read.
238
377
 
239
378
  ### Skipped subjects and the mutant cap
240
379
 
@@ -261,11 +400,13 @@ mutants; the limit does not bound how many proposals a custom source creates.
261
400
 
262
401
  Every survivor is printed with a digest. Accept it with a reason:
263
402
 
264
- ```yaml
265
- # .saboteur-ignore.yml
266
- - subject: "Billing::Invoice#total"
267
- mutation: "1a2b3c4d5e6f7a8b"
268
- reason: "equivalent: amounts are always whole cents"
403
+ ```ruby
404
+ # config/saboteur_ignore.rb
405
+ Saboteur.ignore do |i|
406
+ i.mutation "1a2b3c4d5e6f7a8b",
407
+ subject: "Billing::Invoice#total",
408
+ reason: "equivalent: amounts are always whole cents"
409
+ end
269
410
  ```
270
411
 
271
412
  An entry without a reason is rejected.
@@ -320,6 +461,46 @@ No speed, type-safety or real-fault detection improvement is claimed. See
320
461
  [the literal operator rationale](docs/literal-operators.md) for fault evidence
321
462
  and limits, including string encoding/frozen-value behavior.
322
463
 
464
+ ## Argument removal
465
+
466
+ Builtin removes one ordinary positional argument from a parenthesized call.
467
+ For `Model.includes(:a, :b, c: :d)`, the two proposals remove `:a` or `:b`,
468
+ without deleting the statement or changing `c: :d`. This lets a preload check
469
+ exercise missing associations instead of only a missing return value.
470
+ The operator and source name is `argument_removal`.
471
+
472
+ Only parenthesized calls are supported in this first version. Keyword hashes,
473
+ splats, argument forwarding, bracket indexing/assignment syntax and arguments containing
474
+ heredocs are unchanged. Explicit hash arguments count as positional arguments.
475
+ Blocks and keyword pairs remain intact; separators come from Prism tokens,
476
+ so commas inside strings or comments are not mistaken for separators.
477
+ Arid statements, parameter defaults and nested definitions are skipped.
478
+
479
+ These fine proposals follow existing builtin proposals, with the same validation,
480
+ deduplication and `max_mutants_per_method` cap. Earlier digests and relative order
481
+ stay unchanged for unchanged subjects. Extra proposals can change scores or be
482
+ omitted by the cap. Removing an argument can still raise an arity error; no
483
+ non-crashing guarantee is made. The regression uses a small preload-like receiver
484
+ and checks the missing associations, not an ActiveRecord SQL benchmark. See
485
+ [the validation notes](docs/argument-removal.md) for the pinned snapshot migration.
486
+
487
+ ## Hash and keyword pair removal
488
+
489
+ Builtin removes one key/value association from a hash literal or keyword list.
490
+ `Model.includes(:a, :b, c: :d)` becomes `Model.includes(:a, :b )`, so a nested
491
+ preload guard can be exercised without deleting the whole statement. A sole
492
+ hash pair becomes `{}`. Operator and source name: `pair_removal`.
493
+
494
+ Both parenthesized and command calls are supported. Splats stay intact; pairs
495
+ containing heredocs, parameter defaults, arid statements and nested definitions
496
+ are skipped. Token-based separator edits preserve comments and blocks. Nested
497
+ hash values and callable bodies are visited. Removing a required keyword can
498
+ still crash; the regression uses a preload-like receiver, not an ActiveRecord
499
+ SQL benchmark. These fine proposals follow earlier builtin proposals and use
500
+ the usual validation, deduplication, order and `max_mutants_per_method` cap.
501
+ More proposals can change scores and cap coverage. See
502
+ [the validation notes](docs/pair-removal.md) for snapshot changes.
503
+
323
504
  ## 🔌 Extending
324
505
 
325
506
  Public API, stable across minor versions:
@@ -334,9 +515,10 @@ Public API, stable across minor versions:
334
515
 
335
516
  ## 🌍 Where it has run
336
517
 
337
- An early release (0.3.0). Tested on the RSpec suites of plain gems
518
+ An early release (0.4.0). Tested on the RSpec suites of plain gems
338
519
  (dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
339
- human-essentials on Postgres (4,080 mutants, 6 workers) and lobsters on SQLite,
520
+ human-essentials on Postgres (4,080 mutants, 6 workers; the 2026-10-10 run, before
521
+ the baseline check and the coarse-gate removal) and lobsters on SQLite,
340
522
  both on macOS. Not yet tried on MySQL or older Rails versions. RSpec only, by
341
523
  design. Results: [docs/validation](docs/validation).
342
524
 
@@ -344,17 +526,20 @@ design. Results: [docs/validation](docs/validation).
344
526
 
345
527
  A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
346
528
  1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
347
- ran once on sampled files, so treat the numbers as indicative. mutineer is ahead
529
+ ran once on sampled files, so treat the numbers as indicative. The table was
530
+ measured at 2b6ec02, before the literal, filter and argument operators and
531
+ before the coarse-survivor gate was removed, so mutant counts and times will
532
+ change; a rerun is pending. mutineer is ahead
348
533
  on number of operators and CI extras, and it supports Minitest.
349
534
 
350
535
  | Metric | saboteur | mutineer |
351
536
  |---|---|---|
352
- | Fake kills (lobsters) | 0 of 20 checked | 8 of 8 checked |
537
+ | Fake kills (lobsters) | 0 of 20 sampled at random | 8 of 8 hand-checked (identical mutants, not a random sample) |
353
538
  | Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
354
539
  | Runs on the Postgres app | yes | no |
355
540
  | Time per mutant, hashie | 0.008 s | 0.092 s |
356
541
  | Peak memory, hashie (summed process RSS) | 595 MB | 471 MB |
357
- | Choice of operators, CI features | fixed set, fewer | 20 operators, baseline, HTML, GitHub Action |
542
+ | Choice of operators, CI features | fixed set, fewer | 22 operators, baseline, HTML, GitHub Action |
358
543
 
359
544
  Full results, method and limitations:
360
545
  [benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
@@ -380,12 +565,12 @@ variants, as before.
380
565
 
381
566
  Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not
382
567
  touch less memory, it keeps more of it shared between workers after fork, which
383
- is what PSS measures. On machines running many parallel workers this is the
384
- difference that keeps the box out of swap.
568
+ is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers on a
569
+ 2 GB sandbox. Swap was not measured.
385
570
 
386
- Wall time on uncontended runs: 30-36s for both variants at every job level (no
387
- regression; per-cell minimums 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s
388
- vs 32s at jobs=6, PR first). The shared sandbox had multi-minute CPU-starvation
571
+ Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs
572
+ 32s at jobs=6. These are noisy per-cell minima from historical runs and do not
573
+ show a direction either way. The shared sandbox had multi-minute CPU-starvation
389
574
  windows during this session (walls up to 594s); affected runs are flagged via
390
575
  sampler cadence and excluded from wall analysis. Absolute times are specific to
391
576
  this 2-core sandbox. Environment, method and the contention log are on the
@@ -397,7 +582,7 @@ this 2-core sandbox. Environment, method and the contention log are on the
397
582
  team workflow (GitHub Action, baseline, HTML report), then 1.0. saboteur stays
398
583
  RSpec-only. Design
399
584
  details are in [docs/design.md](docs/design.md), prior art in
400
- [docs/prior-art.md](docs/prior-art.md), and the project checklist in
585
+ [docs/prior-art.md](docs/prior-art.md), and project status in
401
586
  [docs/status.md](docs/status.md).
402
587
 
403
588
  ## 🤝 Contributing
@@ -0,0 +1,21 @@
1
+ # SPDX-License-Identifier: MIT
2
+ # frozen_string_literal: true
3
+
4
+ require "rails/generators"
5
+ require "saboteur/install"
6
+
7
+ module Saboteur
8
+ module Generators
9
+ # rails generate saboteur:install [--force]
10
+ class InstallGenerator < Rails::Generators::Base
11
+ desc "Writes config/saboteur.rb, bin/saboteur and a .gitignore entry for a Rails + RSpec app."
12
+
13
+ def install_saboteur
14
+ installer = Saboteur::Install.new(root: destination_root, force: options[:force])
15
+ installer.call.each { |result| say_status(result.status, result.path) }
16
+ hint = installer.gemfile_hint
17
+ say("\n#{hint}") if hint
18
+ end
19
+ end
20
+ end
21
+ end