evilution 1.5.1 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (227) hide show
  1. checksums.yaml +4 -4
  2. data/.rubocop_todo.yml +30 -0
  3. data/CHANGELOG.md +49 -0
  4. data/README.md +285 -45
  5. data/docs/architecture.md +110 -12
  6. data/docs/ast_pattern_syntax.md +35 -7
  7. data/docs/combined_runs_spike.md +407 -0
  8. data/docs/integrations.md +1 -1
  9. data/docs/isolation.md +63 -2
  10. data/docs/migration-from-mutant.md +16 -11
  11. data/docs/mutant_schemata_spike.md +387 -0
  12. data/docs/operator_yield_study.md +576 -0
  13. data/docs/probe_first_spike.md +264 -0
  14. data/lib/evilution/ast/ancestor_lookup.rb +36 -0
  15. data/lib/evilution/ast/callback_declaration.rb +36 -3
  16. data/lib/evilution/ast/class_body.rb +62 -16
  17. data/lib/evilution/ast/def_region.rb +130 -0
  18. data/lib/evilution/ast/local_declaration.rb +37 -0
  19. data/lib/evilution/ast/parent_index.rb +50 -0
  20. data/lib/evilution/ast/parse_cache.rb +27 -0
  21. data/lib/evilution/ast/parser.rb +8 -3
  22. data/lib/evilution/ast/pattern/filter.rb +53 -12
  23. data/lib/evilution/ast/skip_filter.rb +83 -0
  24. data/lib/evilution/ast/skip_sites/debug_output.rb +67 -0
  25. data/lib/evilution/ast/skip_sites/error_messages.rb +74 -0
  26. data/lib/evilution/ast/skip_sites/logging.rb +93 -0
  27. data/lib/evilution/ast/skip_sites/message.rb +44 -0
  28. data/lib/evilution/ast/skip_sites/nodes.rb +105 -0
  29. data/lib/evilution/ast/skip_sites/presets.rb +31 -0
  30. data/lib/evilution/ast/skip_sites/purity.rb +69 -0
  31. data/lib/evilution/ast/skip_sites/requires.rb +40 -0
  32. data/lib/evilution/ast/skip_sites.rb +9 -0
  33. data/lib/evilution/ast/source_surgeon.rb +20 -2
  34. data/lib/evilution/ast/syntax_check.rb +10 -8
  35. data/lib/evilution/baseline/report.rb +11 -2
  36. data/lib/evilution/baseline.rb +31 -6
  37. data/lib/evilution/cache/keys.rb +140 -0
  38. data/lib/evilution/cache/run_fingerprint.rb +51 -0
  39. data/lib/evilution/cache/skeleton.rb +57 -0
  40. data/lib/evilution/cache/spec_digest.rb +39 -0
  41. data/lib/evilution/cache.rb +143 -50
  42. data/lib/evilution/class_tree.rb +38 -0
  43. data/lib/evilution/cli/command.rb +12 -4
  44. data/lib/evilution/cli/commands/run.rb +12 -3
  45. data/lib/evilution/cli/commands/subjects.rb +5 -2
  46. data/lib/evilution/cli/parser/options_builder.rb +67 -3
  47. data/lib/evilution/cli/printers/environment.rb +34 -4
  48. data/lib/evilution/cli/printers/session_detail.rb +20 -3
  49. data/lib/evilution/compare/categorizer.rb +2 -1
  50. data/lib/evilution/compare/normalizer.rb +3 -2
  51. data/lib/evilution/config/sources.rb +3 -0
  52. data/lib/evilution/config/validators/base.rb +9 -0
  53. data/lib/evilution/config/validators/since.rb +22 -0
  54. data/lib/evilution/config/validators/skip_sites.rb +24 -0
  55. data/lib/evilution/config/validators/timeout_ceiling.rb +10 -0
  56. data/lib/evilution/config/validators/timeout_factor.rb +10 -0
  57. data/lib/evilution/config/validators/timeout_floor.rb +10 -0
  58. data/lib/evilution/config.rb +122 -6
  59. data/lib/evilution/coverage/load_tracker.rb +50 -0
  60. data/lib/evilution/coverage/map.rb +42 -5
  61. data/lib/evilution/coverage/map_builder.rb +123 -22
  62. data/lib/evilution/coverage/map_store.rb +15 -5
  63. data/lib/evilution/coverage/recorder.rb +42 -3
  64. data/lib/evilution/coverage_example_filter.rb +35 -0
  65. data/lib/evilution/equivalent/heuristic/comment_marking.rb +14 -1
  66. data/lib/evilution/example_filter.rb +20 -0
  67. data/lib/evilution/git/changed_lines.rb +103 -0
  68. data/lib/evilution/git/since_scope.rb +77 -0
  69. data/lib/evilution/git/unified_diff.rb +68 -0
  70. data/lib/evilution/infra_crash.rb +26 -0
  71. data/lib/evilution/integration/base.rb +97 -6
  72. data/lib/evilution/integration/crash_detector.rb +46 -6
  73. data/lib/evilution/integration/known_failures.rb +4 -0
  74. data/lib/evilution/integration/loading/body_call_neutralizer.rb +49 -11
  75. data/lib/evilution/integration/loading/callback_equivalence.rb +62 -0
  76. data/lib/evilution/integration/loading/callback_redeclaration.rb +151 -36
  77. data/lib/evilution/integration/loading/concern_redeclaration.rb +16 -12
  78. data/lib/evilution/integration/loading/concern_registry.rb +87 -0
  79. data/lib/evilution/integration/loading/concern_state_cleaner.rb +19 -8
  80. data/lib/evilution/integration/loading/created_modules.rb +56 -0
  81. data/lib/evilution/integration/loading/frozen_namespaces.rb +80 -0
  82. data/lib/evilution/integration/loading/mutation_applier.rb +42 -11
  83. data/lib/evilution/integration/loading/neutralization.rb +41 -0
  84. data/lib/evilution/integration/loading/redefinition_recovery.rb +5 -3
  85. data/lib/evilution/integration/loading/superseded_modules.rb +22 -19
  86. data/lib/evilution/integration/minitest/stop_hook.rb +57 -0
  87. data/lib/evilution/integration/minitest.rb +22 -2
  88. data/lib/evilution/integration/minitest_crash_detector.rb +19 -0
  89. data/lib/evilution/integration/rspec/baseline_runner.rb +3 -2
  90. data/lib/evilution/integration/rspec/crash_detector_lifecycle.rb +6 -1
  91. data/lib/evilution/integration/rspec/example_filter_applier.rb +16 -0
  92. data/lib/evilution/integration/rspec/example_ids.rb +23 -0
  93. data/lib/evilution/integration/rspec/example_order.rb +86 -0
  94. data/lib/evilution/integration/rspec/killer_first.rb +72 -0
  95. data/lib/evilution/integration/rspec/result_builder.rb +7 -0
  96. data/lib/evilution/integration/rspec/state_guard/example_group_descendants.rb +44 -0
  97. data/lib/evilution/integration/rspec/state_guard.rb +2 -2
  98. data/lib/evilution/integration/rspec/target_resolver.rb +42 -0
  99. data/lib/evilution/integration/rspec/test_file_resolver.rb +4 -2
  100. data/lib/evilution/integration/rspec.rb +98 -47
  101. data/lib/evilution/integration/stop_policy.rb +49 -0
  102. data/lib/evilution/integration/test_unit/dispatcher.rb +20 -1
  103. data/lib/evilution/integration/test_unit/subject_class_registry.rb +6 -1
  104. data/lib/evilution/integration/test_unit.rb +28 -2
  105. data/lib/evilution/isolation/fork.rb +58 -13
  106. data/lib/evilution/isolation/in_process.rb +12 -1
  107. data/lib/evilution/line_set.rb +72 -0
  108. data/lib/evilution/mcp/info_tool/actions/environment.rb +12 -16
  109. data/lib/evilution/mcp/info_tool/actions/subjects.rb +3 -5
  110. data/lib/evilution/mcp/info_tool/status_glossary.rb +15 -2
  111. data/lib/evilution/mcp/info_tool.rb +24 -17
  112. data/lib/evilution/mcp/mutate_tool/config_builder.rb +13 -1
  113. data/lib/evilution/mcp/mutate_tool/option_parser.rb +2 -2
  114. data/lib/evilution/mcp/mutate_tool/progress_streamer.rb +1 -1
  115. data/lib/evilution/mcp/mutate_tool/report_trimmer.rb +28 -5
  116. data/lib/evilution/mcp/mutate_tool/survived_enricher.rb +12 -7
  117. data/lib/evilution/mcp/mutate_tool.rb +32 -7
  118. data/lib/evilution/mcp/session_tool.rb +1 -2
  119. data/lib/evilution/mcp.rb +4 -0
  120. data/lib/evilution/mutation/line_diff.rb +111 -0
  121. data/lib/evilution/mutation.rb +17 -13
  122. data/lib/evilution/mutator/base.rb +121 -98
  123. data/lib/evilution/mutator/constant_value_subjects.rb +29 -1
  124. data/lib/evilution/mutator/line_range_filter.rb +27 -0
  125. data/lib/evilution/mutator/load_sources.rb +88 -0
  126. data/lib/evilution/mutator/operator/alias_removal.rb +2 -5
  127. data/lib/evilution/mutator/operator/argument_method_call_replacement.rb +3 -0
  128. data/lib/evilution/mutator/operator/attr_accessor_swap.rb +2 -5
  129. data/lib/evilution/mutator/operator/constant_namespace_strip.rb +3 -3
  130. data/lib/evilution/mutator/operator/freeze_removal.rb +29 -11
  131. data/lib/evilution/mutator/operator/method_call_removal.rb +3 -1
  132. data/lib/evilution/mutator/operator/mixin_removal.rb +2 -5
  133. data/lib/evilution/mutator/operator/superclass_removal.rb +2 -5
  134. data/lib/evilution/mutator/operator/symbol_literal.rb +1 -1
  135. data/lib/evilution/mutator/operator/visibility_removal.rb +2 -5
  136. data/lib/evilution/mutator/registry.rb +11 -5
  137. data/lib/evilution/mutator/restore_source_cache.rb +41 -0
  138. data/lib/evilution/parallel/pool.rb +11 -4
  139. data/lib/evilution/parallel/work_queue/dispatcher/result_stream.rb +40 -0
  140. data/lib/evilution/parallel/work_queue/dispatcher.rb +20 -7
  141. data/lib/evilution/parallel/work_queue/worker/loop.rb +7 -2
  142. data/lib/evilution/parallel/work_queue/worker.rb +10 -3
  143. data/lib/evilution/parallel/work_queue/worker_stat.rb +7 -0
  144. data/lib/evilution/parallel/work_queue.rb +69 -12
  145. data/lib/evilution/parallel_db_warning.rb +39 -12
  146. data/lib/evilution/process_supervisor.rb +22 -2
  147. data/lib/evilution/reporter/cli/item_formatters/coverage_gap.rb +13 -2
  148. data/lib/evilution/reporter/cli/item_formatters/subject_score.rb +5 -1
  149. data/lib/evilution/reporter/cli/line_formatters/mutations.rb +9 -2
  150. data/lib/evilution/reporter/cli/line_formatters/result_line.rb +9 -1
  151. data/lib/evilution/reporter/cli/line_formatters/skipped_sites.rb +14 -0
  152. data/lib/evilution/reporter/cli/line_formatters/uncovered_code.rb +17 -0
  153. data/lib/evilution/reporter/cli/metrics_block.rb +4 -0
  154. data/lib/evilution/reporter/cli/trailer.rb +4 -4
  155. data/lib/evilution/reporter/cli.rb +9 -2
  156. data/lib/evilution/reporter/html/assets/style.css +8 -0
  157. data/lib/evilution/reporter/html/sections/file_section.rb +9 -0
  158. data/lib/evilution/reporter/html/sections/uncovered_details.rb +27 -0
  159. data/lib/evilution/reporter/html/templates/file_section.html.erb +2 -1
  160. data/lib/evilution/reporter/html/templates/summary_cards.html.erb +3 -0
  161. data/lib/evilution/reporter/html/templates/uncovered_details.html.erb +11 -0
  162. data/lib/evilution/reporter/json/mutation_detail.rb +77 -0
  163. data/lib/evilution/reporter/json/subjects.rb +1 -0
  164. data/lib/evilution/reporter/json/timings.rb +42 -0
  165. data/lib/evilution/reporter/json.rb +13 -50
  166. data/lib/evilution/result/coverage_gap.rb +5 -0
  167. data/lib/evilution/result/mutation_result.rb +55 -3
  168. data/lib/evilution/result/mutation_timings.rb +54 -0
  169. data/lib/evilution/result/run_timings.rb +20 -0
  170. data/lib/evilution/result/subject_score.rb +8 -1
  171. data/lib/evilution/result/subject_scorer.rb +5 -3
  172. data/lib/evilution/result/summary.rb +49 -12
  173. data/lib/evilution/runner/baseline_runner.rb +47 -5
  174. data/lib/evilution/runner/diagnostics.rb +0 -13
  175. data/lib/evilution/runner/isolation_resolver.rb +16 -4
  176. data/lib/evilution/runner/mutation_executor/isolated_run.rb +34 -0
  177. data/lib/evilution/runner/mutation_executor/kill_history.rb +66 -0
  178. data/lib/evilution/runner/mutation_executor/mutation_runner.rb +31 -12
  179. data/lib/evilution/runner/mutation_executor/neutralizer/baseline_failed.rb +1 -0
  180. data/lib/evilution/runner/mutation_executor/neutralizer/infra_error.rb +6 -20
  181. data/lib/evilution/runner/mutation_executor/probe_first.rb +99 -0
  182. data/lib/evilution/runner/mutation_executor/probe_first_run.rb +59 -0
  183. data/lib/evilution/runner/mutation_executor/result_cache.rb +45 -14
  184. data/lib/evilution/runner/mutation_executor/result_notifier.rb +1 -1
  185. data/lib/evilution/runner/mutation_executor/result_packer.rb +54 -24
  186. data/lib/evilution/runner/mutation_executor/runless_verdict.rb +39 -0
  187. data/lib/evilution/runner/mutation_executor/strategy/parallel/hinted_items.rb +27 -0
  188. data/lib/evilution/runner/mutation_executor/strategy/parallel/ordered_delivery.rb +97 -0
  189. data/lib/evilution/runner/mutation_executor/strategy/parallel.rb +102 -83
  190. data/lib/evilution/runner/mutation_executor/timed_run.rb +33 -0
  191. data/lib/evilution/runner/mutation_executor/timeout_policy/adaptive.rb +67 -0
  192. data/lib/evilution/runner/mutation_executor/timeout_policy/base.rb +35 -0
  193. data/lib/evilution/runner/mutation_executor/timeout_policy/fixed.rb +17 -0
  194. data/lib/evilution/runner/mutation_executor/timeout_policy.rb +56 -0
  195. data/lib/evilution/runner/mutation_executor.rb +75 -10
  196. data/lib/evilution/runner/mutation_planner.rb +29 -21
  197. data/lib/evilution/runner/phase_clock.rb +22 -0
  198. data/lib/evilution/runner/report_publisher.rb +1 -1
  199. data/lib/evilution/runner/subject_pipeline.rb +18 -8
  200. data/lib/evilution/runner.rb +87 -26
  201. data/lib/evilution/session/diff.rb +9 -2
  202. data/lib/evilution/session/store.rb +17 -2
  203. data/lib/evilution/source_ast_cache.rb +10 -0
  204. data/lib/evilution/spec_ast_cache.rb +15 -0
  205. data/lib/evilution/spec_resolver/constant_references.rb +66 -0
  206. data/lib/evilution/spec_resolver/project_files.rb +50 -0
  207. data/lib/evilution/spec_resolver/sibling_spec_ranker.rb +111 -0
  208. data/lib/evilution/spec_resolver/spec_pattern.rb +19 -0
  209. data/lib/evilution/spec_resolver.rb +76 -65
  210. data/lib/evilution/spec_selector.rb +17 -13
  211. data/lib/evilution/subject.rb +7 -2
  212. data/lib/evilution/version.rb +1 -1
  213. data/lib/evilution/worker_database/copies.rb +73 -0
  214. data/lib/evilution/worker_database/schema.rb +39 -0
  215. data/lib/evilution/worker_database/sqlite_copy.rb +65 -0
  216. data/lib/evilution/worker_database.rb +214 -0
  217. data/schema/evilution.config.schema.json +74 -2
  218. metadata +84 -13
  219. data/lib/evilution/integration/rspec/state_guard/object_space_example_groups.rb +0 -42
  220. data/scripts/benchmark_density +0 -262
  221. data/scripts/benchmark_density.yml +0 -19
  222. data/scripts/canary_manifest.yml +0 -47
  223. data/scripts/compare_mutations +0 -421
  224. data/scripts/compare_mutations.yml +0 -24
  225. data/scripts/compare_targeting +0 -279
  226. data/scripts/compare_targeting.example.yml +0 -24
  227. data/scripts/mutant_json_adapter +0 -227
@@ -0,0 +1,407 @@
1
+ # Combined runs: spike report
2
+
3
+ Status: **no-go**. No prototype in `lib/`. The prototype was a harness
4
+ loaded from outside the gem, kept in `scripts/combined_runs_spike/` (see
5
+ [Reproducing](#reproducing)). The corpus
6
+ gate was met, but a constructed example shows the method can report a
7
+ survivor as killed, and the coverage map cannot rule that out. See
8
+ [Decision](#decision) for what would have to change first.
9
+
10
+ ## Idea
11
+
12
+ Running several mutants in one process only pays if one run gives a verdict
13
+ for each of them. Naive grouping (apply k mutants, run their tests once) does
14
+ not: with 85–95% of mutants killed, a group almost always fails, and nothing
15
+ says which member caused it.
16
+
17
+ The variant tested here uses the per-example coverage map
18
+ (`--example-targeting coverage`, RSpec only). Mutants are combined only when
19
+ the sets of examples that execute their changed lines are disjoint. The union
20
+ of their targeted examples runs once. Each failing example executes the
21
+ changed lines of exactly one member in the original code, so its failure is
22
+ attributed to that member. A member with no failing example is not called a
23
+ survivor on that basis. It falls back to a single run.
24
+
25
+ ## Prototype
26
+
27
+ A harness (`RUBYOPT=-r<harness>`) that prepends `MutationExecutor#run_batch`.
28
+
29
+ **Eligibility.** A mutant can join a group when all of these hold:
30
+
31
+ - it gets no verdict without a run (not unparseable, not `uncovered`, not
32
+ implied by a probe sentinel);
33
+ - its file is built in the map;
34
+ - the examples covering its reported line are known, so its single run would
35
+ be coverage-targeted and not lexical;
36
+ - every changed line that ran at all ran inside some example. A line that
37
+ only ran while the file loaded (a constant, a class-body call, a `def`
38
+ signature) can reach any example, so its mutant is not eligible.
39
+
40
+ Its *affecting set* is the union of the examples covering its changed lines.
41
+
42
+ **Grouping.** Greedy first fit, in the run's own order. A mutant joins the
43
+ first group where it overlaps no member's affecting set and, within the same
44
+ file, no member's changed byte range (a touching range counts as
45
+ overlapping). An optional cap limits members per group. An optional rule
46
+ puts at most one mutant of each subject in a group.
47
+
48
+ **Group run.** One forked process per group, `--jobs` of them at a time, with
49
+ a sandbox directory and the isolated-worker flags as in `Isolation::Fork`.
50
+ In that process:
51
+
52
+ - All members are applied. Members in the same file are merged: their
53
+ changed regions of the neutralized source are spliced into one source,
54
+ which `MutationApplier` evaluates once.
55
+ - The union of the members' targeted examples runs once, ordered
56
+ killer-first as a single run would be. The parent records each group's
57
+ kills in the kill history as the group finishes, so later groups and the
58
+ remaining single runs start with those killers.
59
+ - **Per-member stop on kill.** An `around(:each)` hook skips the remaining
60
+ examples of a member that has already been killed. When every member is
61
+ killed, the run stops.
62
+ - The time limit is the sum of the members' limits.
63
+
64
+ **Fallback.** These cases go back to a normal single run (targeted run,
65
+ then confirmation):
66
+
67
+ - a group that failed to apply, crashed, timed out, or failed outside any
68
+ example (a spec file that did not load, a context hook);
69
+ - a failure whose example belongs to no member;
70
+ - a member whose only failures were infrastructure crashes.
71
+
72
+ A member that passed its targeted examples in a clean group goes straight to
73
+ the confirmation run over its whole resolved spec file ("direct
74
+ confirmation"). Its targeted examples already passed once.
75
+
76
+ Killed members are reported `killed`, with the killing example and their
77
+ share of the group's duration.
78
+
79
+ ## How many mutants combine
80
+
81
+ Measured from the plan on each target. Coverage targeting, `--isolation fork`
82
+ and `--jobs 2` throughout. "Processes" counts the forked processes that
83
+ decide a verdict. *Ideal* assumes every group runs cleanly and only the
84
+ members a single run did not kill fall back.
85
+
86
+ | Target | Mutants (run) | Eligible | Main reasons not eligible | Groups (no cap) | Members grouped | Mean size | Processes: single → ideal |
87
+ |---|---|---|---|---|---|---|---|
88
+ | fixture (3 files) | 440 (440) | 369 | reported line not attributed: 71 | 83 | 343 | 4.1 | 440 → 226 (**−49%**) |
89
+ | dry-monads `maybe.rb` | 610 (561) | 287 | line not attributed: 274; uncovered: 49 | 46 | 267 | 5.8 | 561 → 377 (**−33%**) |
90
+ | httparty `httparty.rb` | 1412 (1203) | 920 | line not attributed: 283; uncovered: 209 | 157 | 818 | 5.2 | 1203 → 656 (**−45%**) |
91
+ | grape `declared_params_handler.rb` | 691 (691) | 618 | line not attributed: 73 | 17 | 34 | 2.0 | 691 → 677 (**−2%**) |
92
+
93
+ "Mutants" leaves out equivalent mutants (1 on dry-monads, 2 on grape), which
94
+ are never run. The verdict table below counts them, hence 611 and 693 there.
95
+
96
+ A cap barely changes these numbers above 8 members. With the ideal process
97
+ count, cap 2 / 4 / 8 / none give −19 / −27 / −31 / −33% on dry-monads and
98
+ −26 / −42 / −45 / −45% on httparty. Grape gains almost nothing: every
99
+ endpoint example of `declared_spec.rb` runs through most of the file, so
100
+ almost no two subjects have disjoint example sets. How much can be combined
101
+ depends on how narrowly the target's examples exercise it, not on the kill
102
+ rate.
103
+
104
+ "Line not attributed" covers mutants reported at a line that no single
105
+ example is credited with: method-body mutants reported at their `def`, and
106
+ endless `def`s. Their single run uses lexical targeting, so they cannot join
107
+ a group.
108
+
109
+ ## Results
110
+
111
+ ### Verdicts
112
+
113
+ Each mutant was matched by file, line, operator and diff against a single-run
114
+ control on the same tree with the same flags. The control was run 2–4 times
115
+ per target.
116
+
117
+ | Target | Combined runs | Divergent verdicts | Control vs control |
118
+ |---|---|---|---|
119
+ | fixture | 5 (one without direct confirmation) | **0 / 440** each | 0 |
120
+ | dry-monads `maybe.rb` | 3 (one without direct confirmation) | **0 / 611** each | 0 |
121
+ | httparty | 6 (also cap 4, distinct subjects, no direct confirmation) | **0 / 1412** each | 0 |
122
+ | grape | 2 | **0 / 693** each | 0 |
123
+
124
+ That is 16 combined runs and 0 divergences on about 13,900 mutant verdicts,
125
+ so **the gate as stated is met**.
126
+
127
+ The first dry-monads run had 16 divergences. They were not caused by
128
+ combining. They came from a bug in coverage targeting, which also produces
129
+ wrong verdicts in the control (see [Known issues found along the way](#known-issues-found-along-the-way)).
130
+ The harness now treats a failure outside any example as a group failure, so
131
+ those groups fall back, and later runs reproduce the control exactly.
132
+
133
+ ### Process count and time
134
+
135
+ Durations are `summary.duration` in seconds. The two modes were alternated
136
+ on the same tree.
137
+
138
+ | Target | Single runs | Combined | Change | Processes (measured) |
139
+ |---|---|---|---|---|
140
+ | fixture | 53.6, 53.4, 54.8 | 50.3, 50.6, 49.9 | **−7%** | 440 → 226 |
141
+ | fixture, without direct confirmation | (same) | 57.2 | +6% | 440 → 226 |
142
+ | dry-monads `maybe.rb` | 45.3, 46.3, 46.2 | 39.4, 39.2 (40.0 without direct confirmation) | **−14%** | 561 → 532 |
143
+ | httparty | 83.8, 85.4, 86.3 | 55.2, 56.0, 56.4 | **−34%** | 1203 → 656 |
144
+ | httparty, cap 4 / distinct subjects / no direct confirmation | (same) | 59.6 / 56.8 / 56.7 | −30% / −33% / −33% | |
145
+ | grape | 54.6, 54.4 | 52.6, 53.4 | −3% (noise) | 691 → 677 |
146
+
147
+ Where the time goes:
148
+
149
+ - **httparty.** A killed mutant costs 128 ms on average in a single run. Of
150
+ that, 107 ms is `test_run`, and only 1.75 examples run per kill. The cost is
151
+ almost all fixed per-process work (fork, sandbox, apply, RSpec setup, reap),
152
+ not examples. Single runs spent 91.3 s of killtime on the 704 mutants that
153
+ groups killed. The 157 group runs that replaced them took 33.3 s, even
154
+ though they went through more examples (5841, skipped ones included, vs 1095). Grouping pays when the fixed
155
+ cost per process outweighs the examples it adds.
156
+ - **fixture.** Each example sleeps 50 ms, so examples dominate. The 83 groups
157
+ cost 44.9–46.0 s, against 39.2–41.0 s for the same 297 kills in single
158
+ runs. Process count halved, examples gone through rose from 490–514 to
159
+ 1846–1858 (skipped ones included), and the groups cost about 14% *more*
160
+ than the single runs they replaced. The whole −7% comes from direct
161
+ confirmation: 46 group survivors skipped their targeted run (49.9–50.0 s
162
+ → 37.2 s). Without it the fixture is 6% *slower*.
163
+ - **dry-monads.** 22 of 46 groups failed outside any example, because of the
164
+ shared-example targeting bug below. 9 more failed to apply: dry-monads turns
165
+ Ruby warnings into errors, and merged sources raised "assigned but unused
166
+ variable" warnings. Only 75 members were killed in groups, saving about
167
+ 6.5 s of killtime. The rest of the measured gain was in mutants that still
168
+ ran singly: in all three combined runs they ran fewer examples (1006–1067
169
+ vs 1092–1128 for the singly killed ones), consistent with groups filling the
170
+ kill history first. The ideal −33% process count is out of reach until that
171
+ bug is fixed.
172
+ - **Ordering.** The kill history matters to groups. With an empty history
173
+ (the first prototype recorded kills only after all groups had run), the
174
+ fixture's group phase took 26.9 s. Recording kills as each group finishes
175
+ brought it down to 22.8 s.
176
+
177
+ ## Failure modes
178
+
179
+ **Shared or global state between subjects (false kill, demonstrated).**
180
+ Disjoint example sets say nothing about state that outlives an example. In a
181
+ constructed project, `Writer#with_entry` deletes its key in `ensure`, and
182
+ `Reader#read` has a survivor (`return :none if false`). In a single run,
183
+ `Writer`'s mutants `STORE.delete!(key)` and `STORE.delete` are killed, but
184
+ they leave the key in the store. Grouped with the `Reader` survivor, the
185
+ leaked key makes `Reader`'s example fail. The failure is attributed to the
186
+ `Reader` mutant, which is reported **killed**. A single run reports it
187
+ survived. Two of the 109 mutants in that project diverged this way.
188
+
189
+ **Control flow into another member's code (false kill, demonstrated).** The
190
+ map is built from the original code. A mutant can send its example into code
191
+ that the example never reached before, including another member's mutated
192
+ code. In the same project, `Router#route` is mutated from `n > 0` to
193
+ `n == 0`, which routes `route(5)` into `Helper#label`. On its own the mutant
194
+ survives, because the original `label(5)` returns `:positive`. Grouped with a
195
+ `Helper#label` mutant that returns `nil`, `Router`'s example fails, and the
196
+ `Router` mutant is reported killed. One more divergence. The
197
+ distinct-subjects rule does not prevent either case: the members here are in
198
+ different classes.
199
+
200
+ Cheaper static guards do not make the method sound either. Requiring the
201
+ members to be in different files excludes neither case: move `STORE`,
202
+ `Writer` and `Reader` into three files, or `Router` and `Helper` into two,
203
+ and the mechanism is the same. Requiring that no two members' bodies name a
204
+ common constant or each other's classes does exclude both constructed cases
205
+ (`Writer` and `Reader` both name `STORE`, `Router` names `Helper`), but one
206
+ step of indirection gets past it: `Writer` and `Reader` reaching the store
207
+ through two different helper classes, or `Router` calling a `Service` that
208
+ calls `Helper`. (These variants were not run; the failing example and its
209
+ attribution are the same as in the originals.) A static rule cannot follow indirect call chains, dynamic dispatch,
210
+ instance variables on shared objects, globals, `ENV`, the filesystem or a
211
+ database.
212
+
213
+ Both cases produce a false kill, which is the worst error for a mutation
214
+ tool to make: it hides a gap in the tests, and no reader can reproduce it.
215
+ The corpus has none, but nothing in the method rules them out. A check that
216
+ each failing example executed only its own member's mutated lines would
217
+ catch the second case. It would not catch the first: the reader's example
218
+ runs only reader code.
219
+
220
+ **Load order and constants redefined, several mutants at once.** Each file
221
+ of a group is evaluated once, with its members merged, through the normal
222
+ `MutationApplier` (constant pinning, concern cleanup, redefinition
223
+ recovery). Groups spanning the fixture's three files gave identical verdicts.
224
+ The risk shows up in projects that turn warnings into errors. A merged source
225
+ can raise a warning that none of its members raises alone (dry-monads: 9 of
226
+ 46 groups). The harness falls back on any apply error, so the cost is time,
227
+ not correctness. A single mutant whose load crashes is reported killed
228
+ (`load_crashed`). Inside a group the same crash only causes a fallback.
229
+
230
+ **Crashes, timeouts and infinite loops.** A group limited by the sum of its
231
+ members' limits loses all of that time when one member loops, and then every
232
+ member runs again on its own. In the constructed project, a `Looper` mutant
233
+ times out after 12 s on its own. Five groups containing one timed out after
234
+ 30–50 s each, before their members re-ran. The run took **128 s instead of
235
+ 39 s**. A tighter group limit (the largest member's limit plus a margin)
236
+ would cut the waste but risks timing out a slow but healthy group. The
237
+ corpus had no timeouts.
238
+
239
+ **Stop on kill.** A single run stops at its first decisive failure. A group
240
+ cannot stop at the first failure without losing the other members' verdicts,
241
+ so the prototype stops per member: it skips the remaining examples of a
242
+ killed member, and stops the run once all members are killed. A survivor in
243
+ a group still runs all of its targeted examples, and with that group's
244
+ duration, then pays its confirmation run as well. On the fixture, groups went
245
+ through 3.7 times as many examples (skipped ones included) as single runs did
246
+ for the same kills.
247
+
248
+ **Killer-first ordering.** Groups need a filled kill history as much as
249
+ single runs do (see the fixture numbers above). The history fills while
250
+ groups run, so the order in which groups are dispatched matters too.
251
+
252
+ **`uncovered` and probe-first.** Mutants given a verdict without a run are
253
+ never grouped. Probe-first was not measured. It calls `run_batch` once per
254
+ batch (sentinels first), and grouping would work within each batch.
255
+
256
+ **Incremental cache.** The prototype ran with the cache off. A real
257
+ implementation would keep cached mutants out of groups, and would store each
258
+ group kill per member, with its killing example. That kill depends on which
259
+ mutants ran beside it, which the cache key does not record.
260
+
261
+ **The map itself.** Attribution is only as sound as the map. A stale map
262
+ must never be used: `MapStore` invalidates by file digest, and the run
263
+ fingerprint covers the helpers. Coverage from code the tests run in a
264
+ forked subprocess is lost, so such examples are missing from affecting sets.
265
+ Shared examples are recorded under the shared file (below).
266
+
267
+ ## Known issues found along the way
268
+
269
+ Both are bugs in coverage targeting as it ships, independent of combined
270
+ runs, and both are filed.
271
+
272
+ - **Coverage targeting runs shared examples by their definition file**
273
+ ([#1973](https://github.com/marinazzio/evilution/issues/1973)). The
274
+ map credits examples defined in a shared group to that group's own
275
+ location (dry-monads: `spec/shared/applicative.rb:5`). A targeted run
276
+ passes that location to RSpec, which loads the shared file on its own,
277
+ redefines the shared group and runs nothing. On dry-monads, where warnings
278
+ are errors, this reports **30 mutants of `maybe.rb` killed with 0 examples
279
+ run**, and 11 more as errors, in the plain single-run control. Lexical
280
+ targeting reports 10 of the line-92 mutants as survivors. In a project
281
+ where the shared file loads cleanly on its own, the targeted run runs
282
+ nothing and the confirmation run over the whole spec file still decides,
283
+ at the cost of a wasted run.
284
+ - **Coverage targeting without `--preload` falls back silently**
285
+ ([#1974](https://github.com/marinazzio/evilution/issues/1974)). On the
286
+ fixture, the map builder child raised `NameError: uninitialized constant
287
+ RSpec` (in `MapBuilder#reset_world`). The parent printed the backtrace,
288
+ wrote no map, and continued with lexical targeting. Passing
289
+ `--preload spec/spec_helper.rb` avoids it.
290
+
291
+ ## Decision
292
+
293
+ **No-go** for now, neither as a default nor as an opt-in, for four reasons:
294
+
295
+ 1. **The method is not sound, and the gate does not show that.** Zero
296
+ divergences on the corpus is the stated gate, and it was met in 16 runs.
297
+ But two mechanisms that turn a survivor into a kill (shared state between
298
+ examples, control flow into another member's code) were shown on a
299
+ constructed project. The coverage map cannot see either one, and the
300
+ result is a false kill, which no reader can reproduce.
301
+ 2. **The gain depends on the target.** −34% on httparty, where a kill costs
302
+ more in process overhead than in examples. −7% on the fixture, all of it
303
+ from skipping survivors' targeted runs; the groups themselves cost more
304
+ than the single runs they replaced. −3% (noise) on grape, whose
305
+ examples cover almost the whole file. −14% on dry-monads, where most
306
+ groups fell back.
307
+ 3. **Failures are expensive.** A looping member costs its group the sum of
308
+ the members' limits, and then everyone runs again (39 s → 128 s on the
309
+ constructed project). Merged sources raise warnings that no single
310
+ mutant raises, and in a project that turns warnings into errors the whole
311
+ group falls back.
312
+ 4. **It is RSpec and coverage-targeting only,** and the shared-example bug
313
+ above currently makes coverage targeting give wrong verdicts on one of
314
+ the corpus targets.
315
+
316
+ What would make it worth revisiting:
317
+
318
+ - the shared-example targeting bug fixed;
319
+ - an attribution check: kill a member only if every failing example
320
+ executed that member's mutated lines and no other member's, which closes
321
+ the control-flow case;
322
+ - groups limited to members whose examples share no mutable state. Nothing
323
+ measured here can tell that; marking specs that run in random order and
324
+ isolation is the closest available signal;
325
+ - a group time limit tighter than the sum of the members' limits.
326
+
327
+ Even then, a group kill would be the only verdict evilution gives that a
328
+ single run cannot reproduce. Re-running each group kill alone to check it
329
+ would cost more than grouping saves: on httparty, 704 re-runs against 547
330
+ processes saved (1203 → 656), each re-run mostly the fixed per-process cost
331
+ (128 ms per kill, 1.75 examples, on average), so the combined run would be
332
+ slower than single runs.
333
+
334
+ ## Reproducing
335
+
336
+ The harness, the comparison scripts and the constructed project are in
337
+ [`scripts/combined_runs_spike/`](../scripts/combined_runs_spike/), whose
338
+ README covers setup and running. They are measurement code, not loaded by
339
+ the gem.
340
+
341
+ Corpus: local copies of the fixture (`spec/support/fixtures/benchmark_project`),
342
+ of the constructed project (`scripts/combined_runs_spike/fm`), and of the
343
+ dry-monads, httparty and grape targets at the commits pinned in
344
+ `scripts/benchmark_perf.yml`. Each copy has a Gemfile adding
345
+ `gem "evilution", path: <tree>`. In httparty, the cwd-relative fixture paths
346
+ are made absolute (`spec/spec_helper.rb` `file_fixture`, and the
347
+ `tiny.gif` path in `spec/httparty_spec.rb`), because mutation processes run
348
+ in a sandbox directory.
349
+
350
+ Harness (`combined.rb`), loaded with `RUBYOPT=-r<file>` (it requires
351
+ `bundler/setup` and `evilution` first):
352
+
353
+ - A module prepended to `Evilution::Runner::MutationExecutor` whose private
354
+ `run_batch(notifier, pipeline, mutations, baseline_result, integration)`
355
+ plans the groups, as described under [Prototype](#prototype). It reads the
356
+ map from the integration's `CoverageExampleFilter`, and it decides
357
+ eligibility from `Coverage::Map#examples_for` / `#executed?` over
358
+ `mutation.changed_lines`. Group runs are forked by the harness, `--jobs`
359
+ at a time. Then `super` handles the rest, and the two `ExecutionResult`s
360
+ are merged.
361
+ - Merging within a file: each member's change is the common prefix and
362
+ suffix of `restore_source` against `eval_source` (and of
363
+ `original_source` against `mutated_source`). Non-touching ranges are
364
+ spliced into one `Evilution::Mutation`, which the integration's
365
+ `MutationApplier` evaluates.
366
+ - A module prepended to `Evilution::Integration::RSpec` whose `run_tests`
367
+ goes straight to `confirm_run` for members that passed in a clean group.
368
+ - `EV_COMBINED=1` runs groups (otherwise plan only). `EV_COMBINED_CAP`,
369
+ `EV_COMBINED_DISTINCT=1` and `EV_COMBINED_DIRECT_CONFIRM=0` select the
370
+ variants. `EV_COMBINED_OUT` writes the plan, the per-cap groupings and the
371
+ per-group outcomes as JSON.
372
+ - Verdict equivalence (`div.rb`): match the two runs' JSON entries by file,
373
+ line, operator and diff, and compare statuses. `analyze.rb` gives the
374
+ group-phase costs under [Process count and time](#process-count-and-time),
375
+ `ideal.rb` the ideal process counts, and `rest.rb` the cost of the mutants
376
+ left to single runs.
377
+
378
+ Commands (in the target dir; the harness via `RUBYOPT`). `run.sh` wraps them:
379
+
380
+ ```
381
+ # fixture
382
+ bundle exec evilution run lib/pricing.rb lib/inventory.rb lib/grade.rb --preload spec/spec_helper.rb \
383
+ --example-targeting coverage --isolation fork --jobs 2 --format json
384
+ # dry-monads
385
+ bundle exec evilution run lib/dry/monads/maybe.rb --spec spec/unit/maybe_spec.rb --integration rspec \
386
+ --preload spec/spec_helper.rb --example-targeting coverage --isolation fork --jobs 2 --format json
387
+ # httparty
388
+ bundle exec evilution run lib/httparty.rb --spec spec/httparty_spec.rb --integration rspec \
389
+ --preload spec/spec_helper.rb --example-targeting coverage --isolation fork --jobs 2 --format json
390
+ # grape
391
+ bundle exec evilution run lib/grape/declared_params_handler.rb --spec spec/grape/endpoint/declared_spec.rb \
392
+ --integration rspec --preload spec/spec_helper.rb --example-targeting coverage --isolation fork \
393
+ --jobs 2 --format json
394
+ ```
395
+
396
+ The constructed project behind the failure modes
397
+ (`scripts/combined_runs_spike/fm`) is one file:
398
+
399
+ - a `STORE` hash;
400
+ - `Writer#with_entry` (set the key, yield, delete it in `ensure`);
401
+ - `Reader#read` (with an untested `nil` guard);
402
+ - `Router#route` (`n > 0 ? :positive : Helper.new.label(n)`);
403
+ - `Helper#label` (`n.positive? ? :positive : :other`);
404
+ - `Looper#count` (`i += 1 while i < n`).
405
+
406
+ It has one example per subject, and two for `Helper`. i18n and Redmine were
407
+ left out: they are Minitest, and the coverage map is RSpec only.
data/docs/integrations.md CHANGED
@@ -129,7 +129,7 @@ The test-unit collaborators are:
129
129
 
130
130
  - `framework_loader.rb` — `require "test-unit"` + disables the at_exit
131
131
  auto-run handler so it doesn't fire on evilution exit.
132
- - `subject_class_registry.rb` — ObjectSpace tracking of newly-loaded
132
+ - `subject_class_registry.rb` — class-tree tracking of newly-loaded
133
133
  `Test::Unit::TestCase` subclasses (test-unit has no public
134
134
  registry-clear analog to `Minitest::Runnable.runnables`).
135
135
  - `dispatcher.rb` — assembles a `Test::Unit::TestSuite` from the new
data/docs/isolation.md CHANGED
@@ -144,6 +144,27 @@ creates and removes their methods when the file is evaluated next, so that
144
144
  method lookup passes through them. Only modules are emptied, not anonymous
145
145
  classes, and only the methods written in that file.
146
146
 
147
+ To see those modules being created without walking the heap (which touches
148
+ every object in the process, twice per evaluation), an `in_process` run
149
+ prepends a hook to `Module#initialize` and `Module#initialize_copy` (for
150
+ copies made with `dup` or `clone`) the first time it evaluates a file. It
151
+ records nothing outside an evaluation. Likewise, a run that loads
152
+ `ActiveSupport::Concern` modules prepends a hook to the concern module's
153
+ `extended`, so the concerns it must clear before each evaluation are listed
154
+ as they are defined. The heap is walked once per process for the ones that
155
+ already existed: in the parent when ActiveSupport is already loaded there
156
+ (by a preload, for instance), otherwise once in each mutation process.
157
+
158
+ A mutant can also freeze a class or module its file declares (`EMPTY =
159
+ Bag.freeze` where the original builds `Bag.new([].freeze).freeze`), and Ruby
160
+ cannot thaw it. Evilution still reports that mutant: it checks the original
161
+ against a fresh copy of the class, then puts the frozen class back, since
162
+ other files may reopen the class or include the module and keep referring to
163
+ the frozen one. Under `in_process` the class therefore stays frozen for the
164
+ rest of the run, and the later mutants of that file are reported as errors.
165
+ A `--isolation fork` run judges them, since every mutant there starts from a
166
+ process the freeze never touched.
167
+
147
168
  Other things a file can add to at load time are not undone: an `at_exit`
148
169
  hook, a `Signal.trap`, a `TracePoint`, a callback pushed into a registry of
149
170
  some library. If the file you are mutating does one of those at the top
@@ -293,14 +314,54 @@ mutation subprocess and any grandchildren it spawned rather than orphaning them,
293
314
  and a terminal interrupt (Ctrl-C) is forwarded to every worker group — an
294
315
  aborted parallel run leaves no stray worker or mutation processes behind.
295
316
 
317
+ ### One database per worker
318
+
319
+ A preloaded Rails app is booted, and its `database.yml` resolved, in the
320
+ parent, so every worker inherits a connection to the same test database. When
321
+ that is a SQLite file, workers would load fixtures into and write to one
322
+ database at once, and a test could fail on another worker's data: a kill that
323
+ depends on timing, not on the mutation. Right after the fork, before its first
324
+ mutation, each worker therefore runs `Evilution::WorkerDatabase`:
325
+
326
+ 1. it re-renders `database.yml` with its own `TEST_ENV_NUMBER` and, when that
327
+ names a different, prepared database, reconnects `ActiveRecord::Base` to
328
+ it (the `parallel_tests` convention);
329
+ 2. otherwise, with `copy_worker_databases` on (the default), it copies the
330
+ preloaded SQLite database through SQLite's backup API into a directory of
331
+ its own (`WorkerDatabase::Copies`) and reconnects to the copy.
332
+
333
+ The per-mutation children forked by the worker inherit the worker's
334
+ connection settings, so all of a worker's mutations run against its database.
335
+ The parent picks the copies directory under the system temp dir before the
336
+ workers fork and removes it when `WorkQueue#map` returns or raises (an
337
+ interrupt reaches the parent as an exception, so it is covered). A worker
338
+ that is recycled, times out or dies is reaped first, and its subdirectory is
339
+ deleted before its replacement forks, so frequent replacement never holds
340
+ more than one copy per worker slot on disk. The
341
+ directory is deliberately not registered with `TempDirTracker`: every forked
342
+ process inherits that tracker's `at_exit` cleanup, and a mutation child dying
343
+ through it would delete the databases of workers still running. See the
344
+ README's "Parallel Runs with SQLite" for what is and is not covered.
345
+
296
346
  ## Related flags
297
347
 
298
348
  - `--timeout N` sets the per-mutation time limit. Under `fork`, this drives
299
349
  SIGKILL. Under `in_process`, this drives `Timeout.timeout` and is subject
300
- to the interrupt-mask hazard described above.
350
+ to the interrupt-mask hazard described above. Without it each mutation's
351
+ limit is derived from the baseline time of its spec files
352
+ (`--timeout-factor`, `--timeout-floor`, `--timeout-ceiling`; see the README's
353
+ "Per-mutation timeout"). A mutation that uses up such a limit while it is
354
+ shorter than the fixed one is run again under the fixed limit before a
355
+ timeout is reported, and the parallel pool's per-worker deadline is twice
356
+ the longest time a mutation of the run may take, both runs together.
301
357
  - `--jobs N` runs N workers in parallel. The parallel pool respects the
302
358
  configured isolation strategy, so `--jobs 4 --isolation fork` uses fork
303
- isolation per-mutation inside each worker.
359
+ isolation per-mutation inside each worker; those workers live for the
360
+ whole run, each taking the next mutation as soon as it is done with the
361
+ last. Under `--isolation in_process` the mutation runs in the worker
362
+ itself, so each worker runs one mutation and is replaced by a fresh one
363
+ forked from the clean parent: nothing one mutation leaves behind reaches
364
+ the next.
304
365
  - `--[no-]canary` toggles the proof-of-life pre-flight check (see above).
305
366
 
306
367
  [rails-handle-interrupt]: https://github.com/rails/rails/blob/main/activesupport/lib/active_support/concurrency/thread_monitor.rb
@@ -15,7 +15,7 @@ precisely so you can line the two tools up and see the difference for yourself.
15
15
 
16
16
  | mutant | Evilution |
17
17
  | --- | --- |
18
- | `mutant run --use rspec -- 'MyApp::Foo*'` | `evilution run lib/my_app/ --target 'MyApp::Foo*'` |
18
+ | `mutant run --use rspec -- 'MyApp::Foo*'` | `evilution run lib/my_app/**/*.rb --target 'MyApp::Foo*'` |
19
19
  | `mutant.yml` | `.evilution.yml` (run `evilution init` to scaffold) |
20
20
  | `mutant-rspec` / `mutant-minitest` gems | built in: `--integration rspec\|minitest\|test-unit` |
21
21
  | "mutation coverage" %, "alive" | "mutation score" 0.0–1.0, "survived" |
@@ -51,8 +51,9 @@ bundle exec mutant run --use rspec -- 'MyApp::Calculator#add'
51
51
  # evilution — point at the file, optionally filter to the method
52
52
  bundle exec evilution run lib/my_app/calculator.rb --target 'MyApp::Calculator#add'
53
53
 
54
- # whole directory, then filter by expression
55
- bundle exec evilution run lib/ --target 'MyApp::Calculator*'
54
+ # every file of a directory (file arguments must be files, so let the shell
55
+ # expand the glob; bash needs `shopt -s globstar`), then filter by expression
56
+ bundle exec evilution run lib/**/*.rb --target 'MyApp::Calculator*'
56
57
  ```
57
58
 
58
59
  `--target` accepts the expression shapes you are used to, plus a source glob:
@@ -81,13 +82,17 @@ bundle exec evilution run lib/ --target 'MyApp::Calculator*'
81
82
  | JSON report | session JSON (always written) | `--format json` (`--save-session` to persist) |
82
83
  | Inline opt-out | `# mutant:disable` | `# evilution:disable` |
83
84
  | Aggressive operators | _(always on)_ | `--profile strict` (or `--strict`) |
85
+ | Changed code only | `--since <git-ref>` | `--since <git-ref>` (lines changed since the ref's merge base with `HEAD`, uncommitted changes included) |
84
86
 
85
- ### Things mutant has that Evilution does not (yet)
87
+ ### `--since` differences
86
88
 
87
- - **`--since <git-ref>`** — mutant can restrict subjects to lines changed since a git
88
- ref. Evilution has no diff-aware selection. The closest tools are `--incremental`
89
- (caches `killed`/`timeout` results and skips unchanged mutations on re-runs — a
90
- different model) and `--target source:<glob>` / explicit file args to scope a run.
89
+ - mutant selects the **subjects** whose source changed and runs all their
90
+ mutations; Evilution runs only the mutations that change a changed line (a
91
+ subject is picked when any of its lines changed, then its unchanged lines are left
92
+ alone). Expect fewer mutations for the same diff.
93
+ - Evilution diffs to the working tree, so uncommitted and untracked files count.
94
+ - Without file arguments Evilution takes the changed `.rb` files under `lib/` and
95
+ `app/`; `--target source:<glob>` cannot be combined with `--since`.
91
96
 
92
97
  ### Things Evilution adds
93
98
 
@@ -193,7 +198,7 @@ whether the same code is being killed:
193
198
 
194
199
  ```sh
195
200
  # produce an evilution report
196
- bundle exec evilution run lib/ --format json --save-session
201
+ bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
197
202
 
198
203
  # line it up against your last mutant session JSON
199
204
  bundle exec evilution compare \
@@ -213,8 +218,8 @@ operator.
213
218
  - **Different mutations.** Evilution ships its own operator registry (`--profile
214
219
  default` / `strict`); it is not mutant's AST-handler set. Counts and specific
215
220
  survivors will differ. Use `compare` to quantify.
216
- - **No diff-based selection** (`--since <ref>`). Use file args, `--target source:`,
217
- or `--incremental`.
221
+ - **Diff-based selection is line-based** (`--since <ref>`): see the `--since`
222
+ differences above.
218
223
  - **File-driven, not subject-driven.** Always pass paths; narrow with `--target`.
219
224
  - **100%-coverage culture.** Evilution does not assume a 100% gate; choose `min_score`
220
225
  deliberately, and note that `:unresolved` mutations are excluded from the score