saboteur 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +32 -0
- data/README.md +37 -3
- data/lib/saboteur/baseline/comparison.rb +1 -0
- data/lib/saboteur/baseline/evidence.rb +11 -1
- data/lib/saboteur/baseline/run_evidence.rb +1 -1
- data/lib/saboteur/cli/run_reporting.rb +1 -1
- data/lib/saboteur/cli.rb +15 -4
- data/lib/saboteur/config.rb +22 -2
- data/lib/saboteur/doctor.rb +24 -1
- data/lib/saboteur/engine/candidates.rb +4 -1
- data/lib/saboteur/engine/run_fingerprint.rb +1 -1
- data/lib/saboteur/engine/test_guess.rb +44 -7
- data/lib/saboteur/engine.rb +11 -3
- data/lib/saboteur/frameworks/database_hosts.rb +30 -0
- data/lib/saboteur/frameworks/rails.rb +8 -10
- data/lib/saboteur/help.rb +2 -1
- data/lib/saboteur/mutation_source/builtin.rb +2 -2
- data/lib/saboteur/mutation_source/operator_examples.rb +35 -0
- data/lib/saboteur/mutation_source/operator_registry.rb +32 -7
- data/lib/saboteur/preview.rb +10 -1
- data/lib/saboteur/reporter/agent_commands.rb +12 -4
- data/lib/saboteur/reporter/agent_summary.rb +1 -1
- data/lib/saboteur/reporter/history_policy.rb +35 -0
- data/lib/saboteur/reporter/history_record.rb +4 -5
- data/lib/saboteur/reporter/score_history.rb +14 -2
- data/lib/saboteur/reporter/stryker.rb +6 -4
- data/lib/saboteur/reporter/terminal.rb +9 -0
- data/lib/saboteur/result.rb +5 -2
- data/lib/saboteur/version.rb +1 -1
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 50217a40a53106f31413eb89d15146944e93d15a83de3d2670c028094ef956b3
|
|
4
|
+
data.tar.gz: 534b1a15255aa3dd4743eb6c5a813529f8b4460423858c4de6e61bc34e2a6c93
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 599ab18ac5ea974b823c6b84d35ac721ab40ceb902155145c9f87311834a767fc66e8c53c88e69a142f35825612a5799579648894eccb833db47b389b566e591
|
|
7
|
+
data.tar.gz: a1952882b5586fa3d846fb123f4700f5b5f0085d8c81dd129afeb4a08f3437da27550f93e7ac3aeb3a59ce682d26e7b31f200734c46da03105f254ac3fa2138d
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,38 @@ All notable changes are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.6.0] - 2026-10-14
|
|
10
|
+
|
|
11
|
+
### Fixed
|
|
12
|
+
- `saboteur doctor`'s boot-file check accepts `require` and `hooks`
|
|
13
|
+
paths with or without the `.rb` extension, so the README's
|
|
14
|
+
`c.require = ["./config/environment"]` quickstart no longer fails
|
|
15
|
+
the check (#257).
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
- `allow_db_hosts` config (#258): hosts you assert are disposable test
|
|
19
|
+
databases. The Rails integration's local-database check accepts them,
|
|
20
|
+
and its refusal names the setting. `saboteur doctor` surfaces the same
|
|
21
|
+
refusal, naming the remote hosts, and after a successful boot prints
|
|
22
|
+
each test database's host. `0.0.0.0` now counts as a local host: compose
|
|
23
|
+
setups write it into database URLs, and connecting to it reaches this
|
|
24
|
+
machine.
|
|
25
|
+
- `--guess-tests` now maps files in packwerk packs (any directory with a
|
|
26
|
+
`package.yml`) to the spec inside the same pack, and maps
|
|
27
|
+
`app/controllers` files to `spec/requests` specs (`users_spec.rb` or
|
|
28
|
+
`users_controller_spec.rb`). It still names the changed files it could
|
|
29
|
+
not match (#259).
|
|
30
|
+
- `script/preflight`: a local pre-review check on one exact base and head
|
|
31
|
+
(#210). It runs the protected-path scope check, sign-off, SPDX headers,
|
|
32
|
+
whitespace, RuboCop and the specs (fast or full mode), writes every harness
|
|
33
|
+
output through a symlink-safe writer under `tmp/preflight/`, and takes
|
|
34
|
+
check reports through harness-owned pipes rather than checkout paths.
|
|
35
|
+
It is not CI and a pass is not an approval.
|
|
36
|
+
|
|
37
|
+
### Changed
|
|
38
|
+
- All built-in operators now run by default, including `range`, `operator_assignment` and `condition_negation`, which were opt-in in 0.5.0. Use `--exclude-operators` (or `c.exclude_operators`) to omit operators, or `--operators` (or `c.operators`) to select a smaller set. Exclusions win; each CLI list replaces its matching config list. This changes the default mutation set: counts, runtime, survivors and scores may change. Review CI thresholds and survivor baselines before adopting the new results. To retain the 0.5.0 default operator set during migration: `bundle exec saboteur run --exclude-operators range,operator_assignment,condition_negation`. To clear an exclusion set in config from the CLI: `bundle exec saboteur run --exclude-operators ''`.
|
|
39
|
+
- Breaking selection policy change: old `core`/`extended` tier names are refused with migration guidance. Policy version 2 binds effective operators and exclusions to cache/verdict identity, reports, score-history v2 and baseline evidence. Old baselines are incompatible and need a fresh reviewed proposal; unchanged mutation IDs stay stable.
|
|
40
|
+
|
|
9
41
|
## [0.5.0] - 2026-10-14
|
|
10
42
|
|
|
11
43
|
### Added
|
data/README.md
CHANGED
|
@@ -500,6 +500,13 @@ Doctor boots your app, checks the config and the test database host, and runs
|
|
|
500
500
|
your suite twice before any mutation. Fix whatever it reports first. Turn
|
|
501
501
|
SimpleCov off for the run if you use it.
|
|
502
502
|
|
|
503
|
+
The database check refuses any test database whose host is not local
|
|
504
|
+
(`localhost`, `127.0.0.1`, `::1`, `0.0.0.0` or a socket), so a remote
|
|
505
|
+
database is never mutated by accident. `0.0.0.0` counts as local: compose
|
|
506
|
+
setups write it into database URLs, and connecting to it reaches this
|
|
507
|
+
machine. For another disposable test database host, assert it yourself:
|
|
508
|
+
`c.allow_db_hosts = ["db.internal"]` in `config/saboteur.rb`.
|
|
509
|
+
|
|
503
510
|
**4. Test what your branch changed**
|
|
504
511
|
|
|
505
512
|
```sh
|
|
@@ -885,6 +892,14 @@ details are in [docs/design.md](docs/design.md), prior art in
|
|
|
885
892
|
|
|
886
893
|
See [CONTRIBUTING.md](CONTRIBUTING.md). Commits need a DCO `Signed-off-by` line.
|
|
887
894
|
|
|
895
|
+
Before you ask for review, run the local preflight check on your exact head:
|
|
896
|
+
|
|
897
|
+
```sh
|
|
898
|
+
ruby script/preflight --base <full main sha> --head <full head sha> --base-ref main --mode full
|
|
899
|
+
```
|
|
900
|
+
|
|
901
|
+
It runs scope, sign-off, SPDX, full RuboCop and the specs, and writes a report under `tmp/preflight/`. A pass is not an approval; fast mode is partial. See [docs/preflight.md](docs/preflight.md).
|
|
902
|
+
|
|
888
903
|
## Acknowledgements
|
|
889
904
|
|
|
890
905
|
Thanks to [Glomo](https://glomo.one) for providing the AI tokens used to build saboteur.
|
|
@@ -932,13 +947,30 @@ The builtin `endpoint_replacement` operator swaps explicit, no-argument `first`/
|
|
|
932
947
|
|
|
933
948
|
JSON findings are ordered by a deterministic [review ranking](docs/survivor-ranking.md): coarse first, then covering-test count, edited-line impact and changed-line proximity, with file/line/digest ties. The terminal groups survivors by file, method and line first; the ranking only orders the survivors on one line, and every survivor is listed, in review mode too. Only consecutive survivors with matching evidence share a test/evidence block; repeated evidence after another group is printed again to keep that order. Ranking does not change scores or execution.
|
|
934
949
|
|
|
950
|
+
Cache and baseline identities include the policy. Re-propose and review old survivor baselines with `baseline propose`; never auto-accept new survivors. Mutation digests for unchanged edits stay stable.
|
|
951
|
+
|
|
935
952
|
### Preview and choose operators
|
|
936
953
|
|
|
937
954
|
`bundle exec saboteur run --dry-run` discovers source methods and generates candidates without booting the app, running tests, forking, or writing reports/caches. It honors `--since`, `--subject` and source arguments. It prints per-method counts and cap omissions. After a matching normal run, a cached estimate shows baseline covering-test work (cached original-test durations times planned mutants; it is not a bound on mutant runtime and excludes boot and fork costs). If any file cannot be parsed, the total is marked INCOMPLETE and no estimate is given.
|
|
938
955
|
|
|
939
956
|
Trust boundary: `--dry-run` never boots the app or runs its tests, but custom `sources` entries (`path.rb:ClassName`) are still `require`d to generate candidates, so their code runs. Only preview with sources you trust.
|
|
940
957
|
|
|
941
|
-
`bundle exec saboteur run --list-operators` lists every builtin name and description. `--operators arithmetic,comparison` or `c.operators = ["arithmetic", "comparison"]` selects
|
|
958
|
+
`bundle exec saboteur run --list-operators` lists every builtin name and description. All builtin operators run by default, including [`range`](docs/operators/range.md), [`operator_assignment`](docs/operators/operator_assignment.md), and [`condition_negation`](docs/operators/condition_negation.md). `--operators arithmetic,comparison` or `c.operators = ["arithmetic", "comparison"]` still selects a subset; `all` selects every name. There is no opt-in tier. `--exclude-operators range,condition_negation` or `c.exclude_operators = ["range", "condition_negation"]` removes named operators from that selection; exclusion wins if a name is in both lists. An empty selection means all, while an empty exclusion means none. CLI lists replace their corresponding config lists. These controls apply to builtin operators, not custom source plugins. Selection and exclusions invalidate cached verdict identity and are carried into agent rerun/replay commands.
|
|
959
|
+
|
|
960
|
+
|
|
961
|
+
All operators run unless you select a smaller set; exclusions always remove operators from that set.
|
|
962
|
+
Repeated `--operators` or `--exclude-operators` flags use the last list, not a union.
|
|
963
|
+
To re-enable an operator excluded in config: `bundle exec saboteur run --operators range --exclude-operators ''`.
|
|
964
|
+
To keep the 0.5.0 default inventory: `bundle exec saboteur run --exclude-operators range,operator_assignment,condition_negation`.
|
|
965
|
+
The old `core` and `extended` names now give a migration error, not a changed selection in disguise.
|
|
966
|
+
An empty effective selection is a usage error before boot. `--list-operators` loads current config and shows on/off plus examples; preview and the run header show the effective set.
|
|
967
|
+
|
|
968
|
+
```ruby
|
|
969
|
+
Saboteur.configure do |c|
|
|
970
|
+
c.operators = [] # all by default; or ["arithmetic", "range"] for a subset
|
|
971
|
+
c.exclude_operators = ["condition_negation"]
|
|
972
|
+
end
|
|
973
|
+
```
|
|
942
974
|
|
|
943
975
|
Inspired by mutineer's --dry-run, --list-operators and --operators tiers (github.com/davidteren/mutineer, MIT, by David Teren).
|
|
944
976
|
|
|
@@ -957,9 +989,11 @@ step; the bundled Action's upload behavior is unchanged.
|
|
|
957
989
|
|
|
958
990
|
Score history now refuses non-regular files, symlink paths and report/history
|
|
959
991
|
inode aliases before running the engine or writing reports. Imported records
|
|
960
|
-
have a 64 KiB per-line bound and strict version
|
|
961
|
-
counts, scores and duplicate keys. History is an output, excluded from baseline
|
|
992
|
+
have a 64 KiB per-line bound and strict validation of supported version 1
|
|
993
|
+
and version 2 records, including counts, scores and duplicate keys. History is an output, excluded from baseline
|
|
962
994
|
fingerprints. See [history safety and comparison limits](docs/scores.md).
|
|
995
|
+
|
|
996
|
+
Operator policy version 2, selected names, excluded names and effective names are in JSON/agent/baseline evidence and fresh score-history v2 rows. Old history v1 rows remain readable but are not compared across policy versions.
|
|
963
997
|
### Inline suppression comments
|
|
964
998
|
|
|
965
999
|
`total = a + b # saboteur:disable-line arithmetic reason="rounding checked elsewhere"` suppresses edits starting on that line, restricted to the named operators. Omit names to suppress all operators. Multiple names are space-separated. Only actual Ruby comments count; directive text in strings does not. A missing or blank reason or unknown operator is a usage error naming file:line. Terminal, JSON and HTML list every suppressed candidate with its reason and digest, outside the score. Moving the commented code keeps the suppression. Multi-line edits use their starting line.
|
|
@@ -19,6 +19,7 @@ module Saboteur
|
|
|
19
19
|
@issues = current.fetch("issues").dup
|
|
20
20
|
@categories = {}
|
|
21
21
|
unless baseline.fetch("compatibility") == current.fetch("compatibility")
|
|
22
|
+
@issues << "baseline inputs or operator policy changed; re-propose with baseline propose"
|
|
22
23
|
@state = "incompatible"
|
|
23
24
|
return
|
|
24
25
|
end
|
|
@@ -56,7 +56,7 @@ module Saboteur
|
|
|
56
56
|
compatibility = document.fetch("compatibility")
|
|
57
57
|
config = compatibility.fetch("config")
|
|
58
58
|
runtime = compatibility.fetch("runtime")
|
|
59
|
-
|
|
59
|
+
validate_operator_policy(config)
|
|
60
60
|
fields!(runtime, %w[engine version platform patchlevel saboteur])
|
|
61
61
|
raise ConfigError, "baseline nonfresh config" if config["since"] || config["incremental"]
|
|
62
62
|
|
|
@@ -69,6 +69,16 @@ module Saboteur
|
|
|
69
69
|
validate_generator(compatibility.fetch("generator"), config, runtime)
|
|
70
70
|
end
|
|
71
71
|
|
|
72
|
+
def validate_operator_policy(config)
|
|
73
|
+
unless config["operator_policy"] && config["operator_policy"]["version"] == MutationSource::OperatorRegistry::POLICY_VERSION
|
|
74
|
+
raise ConfigError, "baseline operator policy is incompatible; re-propose with baseline propose"
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
fields!(config, Config::DEFAULTS.keys - RunEvidence::OUTPUT_SETTINGS + ["operator_policy"])
|
|
78
|
+
expected = Config.new(config.except("operator_policy")).operator_policy
|
|
79
|
+
raise ConfigError, "baseline operator policy mismatch" unless expected == config["operator_policy"]
|
|
80
|
+
end
|
|
81
|
+
|
|
72
82
|
def validate_generator(generator, config, runtime)
|
|
73
83
|
fields!(generator, %w[saboteur sources plugins])
|
|
74
84
|
plugins = document.fetch("snapshot").fetch("inventory").select { |entry| entry["kind"] == "source_plugin" }
|
|
@@ -95,7 +95,7 @@ module Saboteur
|
|
|
95
95
|
entry.kind == "source_plugin"
|
|
96
96
|
end.map(&:to_h) },
|
|
97
97
|
"runtime" => Engine::RunFingerprint.runtime_identity,
|
|
98
|
-
"config" => @config.
|
|
98
|
+
"config" => @config.selection_settings.except(*OUTPUT_SETTINGS),
|
|
99
99
|
"scope" => { "include" => @config["include"], "exclude" => @config["exclude"],
|
|
100
100
|
"subjects" => @config["subjects"] } }
|
|
101
101
|
end
|
|
@@ -66,7 +66,7 @@ module Saboteur
|
|
|
66
66
|
return unless path
|
|
67
67
|
|
|
68
68
|
tree = WorkingTree.new(config.root, ignored: config.output_paths).state
|
|
69
|
-
scope = config.
|
|
69
|
+
scope = config.selection_settings.except("json_report", "html_report", "score_history", "cache_dir")
|
|
70
70
|
message = Reporter::ScoreHistory.new(path).append(report, head: tree.head, scope: scope)
|
|
71
71
|
@err.puts("saboteur: #{message}")
|
|
72
72
|
end
|
data/lib/saboteur/cli.rb
CHANGED
|
@@ -212,10 +212,15 @@ module Saboteur
|
|
|
212
212
|
subjects: Array(config["subjects"]), tests: Array(engine.guessed_tests || options[:tests]),
|
|
213
213
|
sources: @sources || [],
|
|
214
214
|
config_path: options[:config], digest: options[:id], cache: config["cache"],
|
|
215
|
-
incremental: config["incremental"], settings:
|
|
215
|
+
incremental: config["incremental"], settings: agent_settings
|
|
216
216
|
)
|
|
217
217
|
end
|
|
218
218
|
|
|
219
|
+
def agent_settings
|
|
220
|
+
options.slice(*Reporter::Agent::Commands::SETTING_FLAGS.keys)
|
|
221
|
+
.merge(operators: config["operators"], exclude_operators: config["exclude_operators"])
|
|
222
|
+
end
|
|
223
|
+
|
|
219
224
|
def agent_mode
|
|
220
225
|
return "replay" if options[:id]
|
|
221
226
|
return "diff" if config["since"]
|
|
@@ -286,17 +291,23 @@ module Saboteur
|
|
|
286
291
|
raise UsageError, "--dry-run and --list-operators print text, not --agent JSON" if agent?
|
|
287
292
|
raise UsageError, "--id replays one mutant: not with --dry-run or --list-operators" if options[:id]
|
|
288
293
|
|
|
289
|
-
|
|
294
|
+
if options[:list_operators]
|
|
295
|
+
MutationSource::OperatorRegistry.list(@out, names: config["operators"], excluded: config["exclude_operators"])
|
|
296
|
+
return EXIT_OK
|
|
297
|
+
end
|
|
290
298
|
|
|
291
299
|
Preview.new(config, out: @out, err: @err).call
|
|
292
300
|
EXIT_OK
|
|
293
301
|
end
|
|
294
302
|
|
|
295
303
|
def define_selection_options(opts)
|
|
296
|
-
opts.on("--operators NAMES", Array, "Choose operator names
|
|
304
|
+
opts.on("--operators NAMES", Array, "Choose operator names (default: all)") do |v|
|
|
297
305
|
set(:operators, v)
|
|
298
306
|
end
|
|
299
|
-
opts.on("--
|
|
307
|
+
opts.on("--exclude-operators NAMES", Array, "Exclude operator names from the selected set") do |v|
|
|
308
|
+
set(:exclude_operators, v)
|
|
309
|
+
end
|
|
310
|
+
opts.on("--list-operators", "List operator names and descriptions") { set(:list_operators, true) }
|
|
300
311
|
opts.on("--dry-run", "Preview mutants without boot, tests, forks or report writes") { set(:dry_run, true) }
|
|
301
312
|
opts.on("--id DIGEST", "Replay exactly one current mutant by its full digest") { |v| set(:id, v) }
|
|
302
313
|
opts.on("--since REF", "Only mutate subjects changed since git REF") { |v| options[:since] = v }
|
data/lib/saboteur/config.rb
CHANGED
|
@@ -39,8 +39,11 @@ module Saboteur
|
|
|
39
39
|
"custom_map" => "config/saboteur_custom_map.rb",
|
|
40
40
|
"sources" => ["builtin"],
|
|
41
41
|
"operators" => [],
|
|
42
|
+
"exclude_operators" => [],
|
|
42
43
|
"hooks" => [],
|
|
43
44
|
"rails" => false,
|
|
45
|
+
# Hosts the user asserts are disposable test databases (issue #258).
|
|
46
|
+
"allow_db_hosts" => [],
|
|
44
47
|
"reeval" => "nesting",
|
|
45
48
|
"cache_dir" => ".saboteur",
|
|
46
49
|
"json_report" => ".saboteur/report.json",
|
|
@@ -68,7 +71,7 @@ module Saboteur
|
|
|
68
71
|
THRESHOLD_RANGE = (0.0..100.0)
|
|
69
72
|
|
|
70
73
|
LIST_KEYS = %w[require include exclude arid hazard sources hooks rspec_args subjects tests
|
|
71
|
-
fingerprint_extra sandbox_writable operators whole_file].freeze
|
|
74
|
+
fingerprint_extra sandbox_writable operators exclude_operators whole_file allow_db_hosts].freeze
|
|
72
75
|
|
|
73
76
|
attr_reader :root
|
|
74
77
|
|
|
@@ -236,8 +239,25 @@ module Saboteur
|
|
|
236
239
|
validate_operator_config!
|
|
237
240
|
end
|
|
238
241
|
|
|
242
|
+
public
|
|
243
|
+
|
|
244
|
+
def operator_policy
|
|
245
|
+
MutationSource::OperatorRegistry.policy(self["operators"], self["exclude_operators"])
|
|
246
|
+
end
|
|
247
|
+
|
|
248
|
+
# Versioned policy travels with every identity, not with mutation digests.
|
|
249
|
+
def selection_settings
|
|
250
|
+
to_h.merge("operator_policy" => operator_policy)
|
|
251
|
+
end
|
|
252
|
+
|
|
253
|
+
private
|
|
254
|
+
|
|
239
255
|
def validate_operator_config!
|
|
240
|
-
MutationSource::OperatorRegistry.
|
|
256
|
+
effective = MutationSource::OperatorRegistry.select(self["operators"], self["exclude_operators"])
|
|
257
|
+
if effective.empty?
|
|
258
|
+
raise ConfigError,
|
|
259
|
+
"operator selection is empty: enable at least one operator or clear exclusions"
|
|
260
|
+
end
|
|
241
261
|
|
|
242
262
|
self["arid"].each { |pattern| Regexp.new(pattern) }
|
|
243
263
|
rescue RegexpError => e
|
data/lib/saboteur/doctor.rb
CHANGED
|
@@ -25,6 +25,7 @@ module Saboteur
|
|
|
25
25
|
integration = boot_target(hooks)
|
|
26
26
|
@out.puts "Integration: #{@config['integration']} (loaded)"
|
|
27
27
|
@out.puts "Rails: #{@config['rails'] ? 'enabled' : 'disabled'}"
|
|
28
|
+
check_database_hosts
|
|
28
29
|
@out.puts "SimpleCov: #{defined?(::SimpleCov) ? 'present; start suppressed for this process' : 'not loaded'}"
|
|
29
30
|
check_coverage
|
|
30
31
|
check_suite(integration, hooks)
|
|
@@ -105,11 +106,33 @@ module Saboteur
|
|
|
105
106
|
@out.puts "Custom map: #{map.entries.size} entries (loaded)"
|
|
106
107
|
end
|
|
107
108
|
|
|
109
|
+
# Prints the test database hosts the booted app reports, so a non-local
|
|
110
|
+
# refusal names the host here instead of only at target boot (issue #258).
|
|
111
|
+
def check_database_hosts
|
|
112
|
+
return unless @config["rails"]
|
|
113
|
+
return @out.puts("Databases: ActiveRecord did not load; host check skipped") unless defined?(::ActiveRecord::Base)
|
|
114
|
+
|
|
115
|
+
configs = ::ActiveRecord::Base.configurations.configs_for(env_name: "test")
|
|
116
|
+
configs.each do |db|
|
|
117
|
+
host = db.configuration_hash[:host]
|
|
118
|
+
@out.puts "Database: #{db.name} (host #{host.to_s.empty? ? 'local socket' : host})"
|
|
119
|
+
end
|
|
120
|
+
remote = Frameworks::Rails.remote_databases(configs, @config["allow_db_hosts"])
|
|
121
|
+
return if remote.empty?
|
|
122
|
+
|
|
123
|
+
raise ConfigError, "non-local test databases: #{remote.join(', ')}; if they are disposable test databases, " \
|
|
124
|
+
"allow them with c.allow_db_hosts in #{Config::FILE}"
|
|
125
|
+
end
|
|
126
|
+
|
|
108
127
|
def check_boot_files
|
|
109
128
|
%w[require hooks].each do |key|
|
|
110
129
|
@config[key].each do |file|
|
|
111
130
|
path = File.expand_path(file, @config.root)
|
|
112
|
-
|
|
131
|
+
# Resolve the way Kernel#require does: the path as given, or with
|
|
132
|
+
# .rb appended (the Rails quickstart sets ./config/environment, issue #257).
|
|
133
|
+
unless File.file?(path) || File.file?("#{path}.rb")
|
|
134
|
+
raise ConfigError, "#{key} file missing: #{file}; fix config/saboteur.rb"
|
|
135
|
+
end
|
|
113
136
|
|
|
114
137
|
@out.puts "#{key}: #{file} (found)"
|
|
115
138
|
end
|
|
@@ -13,7 +13,10 @@ module Saboteur
|
|
|
13
13
|
def self.build_sources(config)
|
|
14
14
|
root = File.realpath(config.root)
|
|
15
15
|
config["sources"].map do |entry|
|
|
16
|
-
|
|
16
|
+
if entry == "builtin"
|
|
17
|
+
next MutationSource::Builtin.new(arid: config["arid"], operators: config["operators"],
|
|
18
|
+
exclude_operators: config["exclude_operators"])
|
|
19
|
+
end
|
|
17
20
|
|
|
18
21
|
file, const = entry.split(":", 2)
|
|
19
22
|
raise ConfigError, "mutation source #{entry.inspect} must be builtin or path.rb:ClassName" unless const
|
|
@@ -127,7 +127,7 @@ module Saboteur
|
|
|
127
127
|
issues = output_overlap_issues
|
|
128
128
|
entries = inventory(issues).sort_by(&:path).freeze
|
|
129
129
|
issues = issues.uniq.sort
|
|
130
|
-
settings_digest = hash_value(@config.
|
|
130
|
+
settings_digest = hash_value(@config.selection_settings.except(*LOCATION_SETTINGS))
|
|
131
131
|
runtime_digest = hash_value(@runtime_identity)
|
|
132
132
|
digest = hash_value(format: FORMAT_VERSION, files: entries.map(&:to_h),
|
|
133
133
|
settings: settings_digest, runtime: runtime_digest)
|
|
@@ -6,8 +6,9 @@ module Saboteur
|
|
|
6
6
|
# --guess-tests: the specs that sit next to the changed source files by
|
|
7
7
|
# naming convention (lib/a/b.rb -> spec/a/b_spec.rb). A guess, not a
|
|
8
8
|
# coverage fact: a spec that only reaches the file through another one
|
|
9
|
-
# is not found.
|
|
10
|
-
#
|
|
9
|
+
# is not found. Packwerk packs (a directory with a package.yml) map to the
|
|
10
|
+
# spec directory inside the same pack, and controllers also map to request
|
|
11
|
+
# specs. The run scopes to the guesses only when every changed file has one.
|
|
11
12
|
class TestGuess
|
|
12
13
|
Result = Data.define(:specs, :unmatched, :changed)
|
|
13
14
|
|
|
@@ -40,12 +41,48 @@ module Saboteur
|
|
|
40
41
|
end
|
|
41
42
|
|
|
42
43
|
def guess_for(file)
|
|
43
|
-
|
|
44
|
+
candidates_for(file).find { |path| File.file?(File.join(@root, path)) }
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# Candidate specs for a source file, most specific first: the nearest
|
|
48
|
+
# package.yml pack's own spec directory, then the app-wide one.
|
|
49
|
+
def candidates_for(file)
|
|
50
|
+
pack = pack_of(file)
|
|
51
|
+
# pack is nil for files outside packs: "#{nil}/" is "/", which
|
|
52
|
+
# never prefixes a relative path, so delete_prefix is a no-op there.
|
|
53
|
+
base = file.delete_prefix("#{pack}/")
|
|
54
|
+
spec_root = pack ? "#{pack}/spec" : "spec"
|
|
55
|
+
stem = base.delete_suffix(".rb")
|
|
44
56
|
rest = stem.split("/", 2)
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
57
|
+
rooted = rest.size == 2 && SOURCE_ROOTS.include?(rest.first)
|
|
58
|
+
names = rooted ? [rest.last, "lib/#{rest.last}", stem] : [stem]
|
|
59
|
+
names = names.flat_map { |name| [name, *request_names(name)] }.uniq
|
|
60
|
+
names.map { |name| "#{spec_root}/#{name}_spec.rb" }
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Rails tests controllers as request specs: spec/requests/users_spec.rb
|
|
64
|
+
# or spec/requests/users_controller_spec.rb for controllers/users_controller.rb.
|
|
65
|
+
def request_names(name)
|
|
66
|
+
return [] unless name.start_with?("controllers/")
|
|
67
|
+
|
|
68
|
+
path = name.delete_prefix("controllers/")
|
|
69
|
+
["requests/#{path}", "requests/#{path.delete_suffix('_controller')}"].uniq
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# The nearest ancestor directory holding a package.yml (a packwerk
|
|
73
|
+
# pack), relative to the root. The root itself is not a pack here:
|
|
74
|
+
# its specs follow the plain mapping.
|
|
75
|
+
def pack_of(file)
|
|
76
|
+
dir = File.dirname(file)
|
|
77
|
+
until dir == "."
|
|
78
|
+
return dir if pack_dirs.include?(dir)
|
|
79
|
+
|
|
80
|
+
dir = File.dirname(dir)
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def pack_dirs
|
|
85
|
+
@pack_dirs ||= Dir.glob("**/package.yml", base: @root).to_set { |path| File.dirname(path) }
|
|
49
86
|
end
|
|
50
87
|
end
|
|
51
88
|
end
|
data/lib/saboteur/engine.rb
CHANGED
|
@@ -13,7 +13,8 @@ module Saboteur
|
|
|
13
13
|
# engine/; this class only wires them together.
|
|
14
14
|
class Engine
|
|
15
15
|
# Everything besides sources and tests that can change a verdict.
|
|
16
|
-
VERDICT_INPUTS = %w[integration require rspec_args rails reeval sources operators arid
|
|
16
|
+
VERDICT_INPUTS = %w[integration require rspec_args rails reeval sources operators exclude_operators arid
|
|
17
|
+
timeout_factor timeout_min
|
|
17
18
|
max_memory_mb max_processes seed tests whole_file].freeze
|
|
18
19
|
|
|
19
20
|
attr_reader :config, :hooks
|
|
@@ -41,6 +42,7 @@ module Saboteur
|
|
|
41
42
|
refuse_fail_fast_evidence
|
|
42
43
|
started = monotonic
|
|
43
44
|
guess_tests
|
|
45
|
+
announce_operators
|
|
44
46
|
@run_evidence = Baseline::RunEvidence.new(config: config, extra_files: @evidence_files) if @capture_evidence
|
|
45
47
|
ignore_list = @replay ? IgnoreList.new([]) : IgnoreList.load(config.path("ignore_file"))
|
|
46
48
|
warn_custom_map
|
|
@@ -110,6 +112,10 @@ module Saboteur
|
|
|
110
112
|
end
|
|
111
113
|
|
|
112
114
|
# The run's immutable report, built once the last result is in.
|
|
115
|
+
def announce_operators
|
|
116
|
+
@diagnostics.inform("Operators: #{config.operator_policy['effective'].join(', ')}")
|
|
117
|
+
end
|
|
118
|
+
|
|
113
119
|
def build_report(results, subjects, scope, started)
|
|
114
120
|
Report.new(results: results, subjects: subjects, wall_time: monotonic - started, root: @root,
|
|
115
121
|
invalid_mutations: @diagnostics.invalid_mutations, warnings: @diagnostics.warnings,
|
|
@@ -117,7 +123,8 @@ module Saboteur
|
|
|
117
123
|
diff_scope: scope, generation_events: @diagnostics.generation_events,
|
|
118
124
|
omissions: @diagnostics.omissions, run_evidence: complete_evidence(results, subjects),
|
|
119
125
|
hazard_suppressions: @diagnostics.hazard_suppressions, early_stop: @early_stop,
|
|
120
|
-
matrix: config["matrix"], inline_suppressions: @diagnostics.inline_suppressions
|
|
126
|
+
matrix: config["matrix"], inline_suppressions: @diagnostics.inline_suppressions,
|
|
127
|
+
operator_policy: config.operator_policy)
|
|
121
128
|
end
|
|
122
129
|
|
|
123
130
|
def complete_evidence(results, subjects)
|
|
@@ -249,7 +256,8 @@ module Saboteur
|
|
|
249
256
|
return nil if @replay
|
|
250
257
|
return nil unless config["incremental"]
|
|
251
258
|
|
|
252
|
-
inputs = JSON.generate(VERDICT_INPUTS.to_h { |key| [key, config[key]] }
|
|
259
|
+
inputs = JSON.generate(VERDICT_INPUTS.to_h { |key| [key, config[key]] }
|
|
260
|
+
.merge("operator_policy" => config.operator_policy))
|
|
253
261
|
ResultCache.new(path: File.join(config.path("cache_dir"), "results.json"), root: @root, inputs: inputs,
|
|
254
262
|
suite_digest: SuiteFiles.new(config).digest)
|
|
255
263
|
end
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# SPDX-License-Identifier: MIT
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
module Saboteur
|
|
5
|
+
module Frameworks
|
|
6
|
+
# Which database hosts count as local for the Rails integration's refusal
|
|
7
|
+
# to mutate a remote test database (issue #258).
|
|
8
|
+
module DatabaseHosts
|
|
9
|
+
# 0.0.0.0: a listen address, but in a URL it connects to this machine (issue #258).
|
|
10
|
+
LOCAL_HOSTS = ["localhost", "127.0.0.1", "::1", "[::1]", "0.0.0.0"].freeze
|
|
11
|
+
|
|
12
|
+
module_function
|
|
13
|
+
|
|
14
|
+
def local?(host)
|
|
15
|
+
host = host.to_s
|
|
16
|
+
host.empty? || host.start_with?("/") || LOCAL_HOSTS.include?(host.downcase)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# "name (host host)" per config whose host is not local and not in
|
|
20
|
+
# extra_allowed. Shared by the boot refusal and doctor.
|
|
21
|
+
def remote(configs, extra_allowed = [])
|
|
22
|
+
allowed = extra_allowed.map(&:downcase)
|
|
23
|
+
configs.filter_map do |db|
|
|
24
|
+
host = db.configuration_hash[:host]
|
|
25
|
+
"#{db.name} (host #{host})" unless local?(host) || allowed.include?(host.downcase)
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
@@ -7,6 +7,8 @@ require "pathname"
|
|
|
7
7
|
require "English"
|
|
8
8
|
require "tmpdir"
|
|
9
9
|
|
|
10
|
+
require_relative "database_hosts"
|
|
11
|
+
|
|
10
12
|
module Saboteur
|
|
11
13
|
module Frameworks
|
|
12
14
|
# Opt-in Rails support (`c.rails = true` in config/saboteur.rb).
|
|
@@ -16,8 +18,6 @@ module Saboteur
|
|
|
16
18
|
# forking, and drops database connections in the parent so each child opens
|
|
17
19
|
# its own. Prints host names only, never credentials.
|
|
18
20
|
class Rails
|
|
19
|
-
LOCAL_HOSTS = ["localhost", "127.0.0.1", "::1", "[::1]"].freeze
|
|
20
|
-
|
|
21
21
|
# How many blocking session names the template-copy error lists.
|
|
22
22
|
MAX_BLOCKER_NAMES = 5
|
|
23
23
|
|
|
@@ -170,9 +170,9 @@ module Saboteur
|
|
|
170
170
|
db.adapter.to_s.match?(/sqlite/) && db.database.to_s == ":memory:"
|
|
171
171
|
end
|
|
172
172
|
|
|
173
|
-
def self.local_host?(host)
|
|
174
|
-
|
|
175
|
-
|
|
173
|
+
def self.local_host?(host) = DatabaseHosts.local?(host)
|
|
174
|
+
|
|
175
|
+
def self.remote_databases(configs, extra_allowed = []) = DatabaseHosts.remote(configs, extra_allowed)
|
|
176
176
|
|
|
177
177
|
private
|
|
178
178
|
|
|
@@ -197,13 +197,11 @@ module Saboteur
|
|
|
197
197
|
def check_databases
|
|
198
198
|
return unless defined?(::ActiveRecord::Base)
|
|
199
199
|
|
|
200
|
-
remote =
|
|
201
|
-
host = db.configuration_hash[:host]
|
|
202
|
-
"#{db.name} (host #{host})" unless self.class.local_host?(host)
|
|
203
|
-
end
|
|
200
|
+
remote = self.class.remote_databases(test_configs, @config["allow_db_hosts"])
|
|
204
201
|
return if remote.empty?
|
|
205
202
|
|
|
206
|
-
raise Error, "rails integration refuses to run against non-local databases: #{remote.join(', ')}"
|
|
203
|
+
raise Error, "rails integration refuses to run against non-local databases: #{remote.join(', ')}; " \
|
|
204
|
+
"if they are disposable test databases, allow them with c.allow_db_hosts in #{Config::FILE}"
|
|
207
205
|
end
|
|
208
206
|
|
|
209
207
|
def test_configs
|
data/lib/saboteur/help.rb
CHANGED
|
@@ -42,7 +42,8 @@ module Saboteur
|
|
|
42
42
|
"tests that cover each change, and reports survivors (changes no test noticed). Source files or " \
|
|
43
43
|
"directories after run replace include and exclude; --test limits the specs that run; " \
|
|
44
44
|
"--threshold PERCENT replaces the survivor check: exit 0 when the score is at or above " \
|
|
45
|
-
"PERCENT, exit 1 when it is below.
|
|
45
|
+
"PERCENT, exit 1 when it is below. All operators run unless you select a smaller set; " \
|
|
46
|
+
"exclusions always remove operators from that set. Inline comment syntax: " \
|
|
46
47
|
'# saboteur:disable-line [operator ...] reason="...". A reason is required.',
|
|
47
48
|
takes_options: true,
|
|
48
49
|
exit_codes: EXIT_CODE_MEANINGS,
|
|
@@ -53,9 +53,9 @@ module Saboteur
|
|
|
53
53
|
CATCH_ALL = %w[StandardError ::StandardError Exception ::Exception].freeze
|
|
54
54
|
SCOPE_NODES = [Prism::DefNode, Prism::ClassNode, Prism::ModuleNode, Prism::SingletonClassNode].freeze
|
|
55
55
|
|
|
56
|
-
def initialize(arid: DEFAULT_ARID, operators: [])
|
|
56
|
+
def initialize(arid: DEFAULT_ARID, operators: [], exclude_operators: [])
|
|
57
57
|
super()
|
|
58
|
-
@operators = OperatorRegistry.
|
|
58
|
+
@operators = OperatorRegistry.select(operators, exclude_operators)
|
|
59
59
|
@arid = arid.map { |pattern| pattern.is_a?(Regexp) ? pattern : Regexp.new(pattern) }
|
|
60
60
|
@filter_replacement = FilterReplacement.new(arid: arid)
|
|
61
61
|
@literal_replacement = LiteralReplacement.new(arid: arid)
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# SPDX-License-Identifier: MIT
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
module Saboteur
|
|
5
|
+
module MutationSource
|
|
6
|
+
module OperatorExamples
|
|
7
|
+
EXAMPLES = {
|
|
8
|
+
"body_nil" => "def value = 1 -> def value = nil",
|
|
9
|
+
"body_true" => "def ok? = false -> def ok? = true",
|
|
10
|
+
"body_false" => "def ok? = true -> def ok? = false",
|
|
11
|
+
"body_raise" => "def value = 1 -> def value; raise; end",
|
|
12
|
+
"comparison" => "a < b -> a <= b",
|
|
13
|
+
"arithmetic" => "a + b -> a - b",
|
|
14
|
+
"boolean_logic" => "a && b -> a",
|
|
15
|
+
"condition_true" => "if x -> if true",
|
|
16
|
+
"condition_false" => "if x -> if false",
|
|
17
|
+
"negation_removal" => "!x -> x",
|
|
18
|
+
"literal_number" => "1 -> 0",
|
|
19
|
+
"or_assign" => "x ||= 1 -> x = 1",
|
|
20
|
+
"safe_navigation" => "x&.name -> x.name",
|
|
21
|
+
"statement_removal" => "check; value -> nil; value",
|
|
22
|
+
"filter_replacement" => "xs.select { |x| x } -> xs.reject { |x| x }",
|
|
23
|
+
"literal_string" => "\"hello\" -> \"\"",
|
|
24
|
+
"literal_boolean" => "true -> false",
|
|
25
|
+
"argument_removal" => "send(a, b) -> send(a)",
|
|
26
|
+
"pair_removal" => "{a: 1, b: 2} -> {b: 2}",
|
|
27
|
+
"endpoint_replacement" => "xs.first -> xs.last",
|
|
28
|
+
"chain_removal" => "x.first.to_s -> x.to_s",
|
|
29
|
+
"range" => "1..3 -> 1...3",
|
|
30
|
+
"operator_assignment" => "x += 1 -> x -= 1",
|
|
31
|
+
"condition_negation" => "x ? a : b -> !(x) ? a : b"
|
|
32
|
+
}.freeze
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -1,6 +1,9 @@
|
|
|
1
1
|
# SPDX-License-Identifier: MIT
|
|
2
2
|
# frozen_string_literal: true
|
|
3
3
|
|
|
4
|
+
require "did_you_mean"
|
|
5
|
+
require_relative "operator_examples"
|
|
6
|
+
|
|
4
7
|
module Saboteur
|
|
5
8
|
module MutationSource
|
|
6
9
|
# Public names used by selection, help and inline suppression validation.
|
|
@@ -31,24 +34,46 @@ module Saboteur
|
|
|
31
34
|
"endpoint_replacement" => "Swap explicit no-argument first and last",
|
|
32
35
|
"chain_removal" => "Remove one call from a method chain"
|
|
33
36
|
}.freeze
|
|
34
|
-
|
|
35
|
-
TIERS = { "
|
|
36
|
-
"all" => DESCRIPTIONS.keys.freeze }.freeze
|
|
37
|
+
POLICY_VERSION = 2
|
|
38
|
+
TIERS = { "all" => DESCRIPTIONS.keys.freeze }.freeze
|
|
37
39
|
|
|
38
|
-
def self.list(out)
|
|
40
|
+
def self.list(out, names: [], excluded: [])
|
|
41
|
+
active = select(names, excluded)
|
|
39
42
|
DESCRIPTIONS.each do |name, description|
|
|
40
|
-
out.puts("#{name} #{
|
|
43
|
+
out.puts("#{name} #{active.include?(name) ? 'on' : 'off'} #{description} #{OperatorExamples::EXAMPLES.fetch(name)}")
|
|
41
44
|
end
|
|
42
45
|
nil
|
|
43
46
|
end
|
|
44
47
|
|
|
48
|
+
def self.policy(names, excluded)
|
|
49
|
+
{ "version" => POLICY_VERSION, "selected" => names, "excluded" => excluded,
|
|
50
|
+
"effective" => select(names, excluded) }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Exclusion wins over explicit selection; empty exclusions remove nothing.
|
|
54
|
+
def self.select(names, excluded)
|
|
55
|
+
selected = resolve(names)
|
|
56
|
+
excluded.empty? ? selected : selected - resolve(excluded)
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def self.suggestion(name)
|
|
60
|
+
matches = DidYouMean::SpellChecker.new(dictionary: DESCRIPTIONS.keys).correct(name)
|
|
61
|
+
matches.empty? ? "" : "Did you mean #{matches.first}?"
|
|
62
|
+
end
|
|
63
|
+
|
|
45
64
|
def self.resolve(names)
|
|
46
|
-
return TIERS.fetch("
|
|
65
|
+
return TIERS.fetch("all") if names.empty?
|
|
47
66
|
|
|
67
|
+
legacy = names & %w[core extended]
|
|
68
|
+
unless legacy.empty?
|
|
69
|
+
raise ConfigError, "operator tiers #{legacy.join(', ')} were removed: use concrete names or all; " \
|
|
70
|
+
"for the old core set exclude range,operator_assignment,condition_negation"
|
|
71
|
+
end
|
|
48
72
|
unknown = names - DESCRIPTIONS.keys - TIERS.keys
|
|
49
73
|
unless unknown.empty?
|
|
50
74
|
raise ConfigError, "unknown operators: #{unknown.join(', ')}. Valid names: " \
|
|
51
|
-
"#{(DESCRIPTIONS.keys + TIERS.keys).join(', ')}"
|
|
75
|
+
"#{(DESCRIPTIONS.keys + TIERS.keys).join(', ')}. " \
|
|
76
|
+
"#{suggestion(unknown.first)} See --list-operators."
|
|
52
77
|
end
|
|
53
78
|
names.flat_map { |name| TIERS.fetch(name) { [name] } }.uniq
|
|
54
79
|
end
|
data/lib/saboteur/preview.rb
CHANGED
|
@@ -13,6 +13,7 @@ module Saboteur
|
|
|
13
13
|
end
|
|
14
14
|
|
|
15
15
|
def call
|
|
16
|
+
announce_operators
|
|
16
17
|
selection = Engine::SubjectSelection.new(@config, @diagnostics)
|
|
17
18
|
subjects = selection.select(@config.included_files)
|
|
18
19
|
scope = selection.changed_lines && Engine::DiffScope.new(selection.changed_lines)
|
|
@@ -24,12 +25,20 @@ module Saboteur
|
|
|
24
25
|
@out.puts("Total: #{lists.sum(&:size)} mutants; #{@diagnostics.omissions.sum(&:count)} " \
|
|
25
26
|
"dropped by max_mutants_per_method#{incomplete_note(incomplete)}")
|
|
26
27
|
incomplete.zero? ? estimate(subjects, lists) : @out.puts("no estimate: source discovery was incomplete")
|
|
27
|
-
|
|
28
|
+
print_warnings
|
|
28
29
|
nil
|
|
29
30
|
end
|
|
30
31
|
|
|
31
32
|
private
|
|
32
33
|
|
|
34
|
+
def print_warnings
|
|
35
|
+
@diagnostics.warnings.each { |warning| @out.puts("warning: #{warning}") }
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def announce_operators
|
|
39
|
+
@out.puts("Operators: #{@config.operator_policy['effective'].join(', ')}")
|
|
40
|
+
end
|
|
41
|
+
|
|
33
42
|
# Candidate filtering precedes diff restriction and the per-method cap.
|
|
34
43
|
# Name that scope rather than implying these rows were executed or scored.
|
|
35
44
|
def inline_suppressions
|
|
@@ -15,7 +15,8 @@ module Saboteur
|
|
|
15
15
|
|
|
16
16
|
# Options typed on the command line that change how a run behaves.
|
|
17
17
|
SETTING_FLAGS = { reeval: "--reeval", integration: "--integration", jobs: "--jobs",
|
|
18
|
-
timeout_factor: "--timeout-factor", json_report: "--json", html_report: "--html"
|
|
18
|
+
timeout_factor: "--timeout-factor", json_report: "--json", html_report: "--html",
|
|
19
|
+
operators: "--operators", exclude_operators: "--exclude-operators" }.freeze
|
|
19
20
|
|
|
20
21
|
class << self
|
|
21
22
|
# Run the same selection again: ref, subjects, sources, tests, config,
|
|
@@ -46,15 +47,22 @@ module Saboteur
|
|
|
46
47
|
tests.empty? ? [] : ["--no-guess-tests", *tests.flat_map { |test| ["--test", test] }]
|
|
47
48
|
end
|
|
48
49
|
|
|
50
|
+
def setting_words(provenance)
|
|
51
|
+
SETTING_FLAGS.flat_map do |key, flag|
|
|
52
|
+
next [] unless provenance.settings.key?(key)
|
|
53
|
+
|
|
54
|
+
value = provenance.settings[key]
|
|
55
|
+
[flag, value.is_a?(Array) ? value.join(",") : value.to_s]
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
|
|
49
59
|
def selection(provenance, replay: false)
|
|
50
60
|
words = []
|
|
51
61
|
words.push("--config", provenance.config_path) if provenance.config_path
|
|
52
62
|
words.push("--since", provenance.ref) if provenance.ref
|
|
53
63
|
provenance.subjects.each { |subject| words.push("--subject", subject) }
|
|
54
64
|
words.push(*test_words(provenance.tests))
|
|
55
|
-
|
|
56
|
-
words.push(flag, provenance.settings[key].to_s) if provenance.settings.key?(key)
|
|
57
|
-
end
|
|
65
|
+
words.concat(setting_words(provenance))
|
|
58
66
|
words << "--no-cache" if provenance.cache == false
|
|
59
67
|
words << "--incremental" if provenance.incremental
|
|
60
68
|
words.push("--id", provenance.digest) if provenance.digest && !replay
|
|
@@ -22,7 +22,7 @@ module Saboteur
|
|
|
22
22
|
verdict = verdict_of(report, status)
|
|
23
23
|
{
|
|
24
24
|
version: VERSION, status: status, verdict: verdict, exitCode: exit_code_for(verdict),
|
|
25
|
-
provenance: provenance_hash(provenance),
|
|
25
|
+
provenance: provenance_hash(provenance).merge(operatorPolicy: report.operator_policy),
|
|
26
26
|
counts: counts(report), score: score(report),
|
|
27
27
|
scoreDefinition: "(killed + timeout) / (killed + survived + timeout); null when nothing was tested",
|
|
28
28
|
incomplete: incomplete, timeoutCountedAsDetected: report.counts[:timeout],
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# SPDX-License-Identifier: MIT
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
module Saboteur
|
|
5
|
+
module Reporter
|
|
6
|
+
# v1 stays readable, but v2 records require versioned operator evidence.
|
|
7
|
+
module HistoryPolicy
|
|
8
|
+
def validate_header(record, fields)
|
|
9
|
+
version = record.is_a?(Hash) && record["version"]
|
|
10
|
+
raise Error, "score history contains an invalid version" unless version.is_a?(Integer) && [1,
|
|
11
|
+
2].include?(version)
|
|
12
|
+
|
|
13
|
+
fields(record, version == 2 ? fields + ["operatorPolicy"] : fields)
|
|
14
|
+
validate_policy(record["operatorPolicy"]) if version == 2
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def validate_policy(value)
|
|
18
|
+
raise Error, "score history contains invalid operator policy" unless value.is_a?(Hash)
|
|
19
|
+
|
|
20
|
+
fields(value, %w[version selected excluded effective])
|
|
21
|
+
raise Error, "score history contains invalid operator policy" if value["effective"].empty?
|
|
22
|
+
|
|
23
|
+
lists = value.values_at("selected", "excluded", "effective")
|
|
24
|
+
unless lists.all? { |list| list.is_a?(Array) && list.all?(String) }
|
|
25
|
+
raise Error, "score history contains invalid operator policy"
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
expected = MutationSource::OperatorRegistry.policy(value["selected"], value["excluded"])
|
|
29
|
+
raise Error, "score history contains invalid operator policy" unless value == expected
|
|
30
|
+
rescue ConfigError, NoMethodError, TypeError
|
|
31
|
+
raise Error, "score history contains invalid operator policy"
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
|
|
4
4
|
require "json"
|
|
5
5
|
require "time"
|
|
6
|
+
require_relative "history_policy"
|
|
6
7
|
|
|
7
8
|
module Saboteur
|
|
8
9
|
module Reporter
|
|
@@ -21,6 +22,8 @@ module Saboteur
|
|
|
21
22
|
end
|
|
22
23
|
end
|
|
23
24
|
|
|
25
|
+
extend HistoryPolicy
|
|
26
|
+
|
|
24
27
|
module_function
|
|
25
28
|
|
|
26
29
|
def parse(line)
|
|
@@ -32,11 +35,7 @@ module Saboteur
|
|
|
32
35
|
end
|
|
33
36
|
|
|
34
37
|
def validate(record)
|
|
35
|
-
|
|
36
|
-
unless record.fetch("version").is_a?(Integer) && record.fetch("version") == 1
|
|
37
|
-
raise Error,
|
|
38
|
-
"score history contains an invalid version"
|
|
39
|
-
end
|
|
38
|
+
validate_header(record, FIELDS)
|
|
40
39
|
|
|
41
40
|
date(record.fetch("date"))
|
|
42
41
|
commit = record.fetch("commit")
|
|
@@ -13,6 +13,7 @@ module Saboteur
|
|
|
13
13
|
# Opt-in append-only score observations, not a baseline or a CI gate.
|
|
14
14
|
class ScoreHistory
|
|
15
15
|
DIFFERENT_RUN = "score history: change unavailable (different selection/inventory or incomplete run)"
|
|
16
|
+
DIFFERENT_POLICY = "score history: change unavailable (different record version or operator policy)"
|
|
16
17
|
MISSING_SCORE = "score history: change unavailable (no tested mutants)"
|
|
17
18
|
MAX_RECORD_BYTES = 64 * 1024
|
|
18
19
|
OPEN_FLAGS = File::RDWR | File::CREAT | File::NONBLOCK | File::NOFOLLOW
|
|
@@ -58,13 +59,13 @@ module Saboteur
|
|
|
58
59
|
private
|
|
59
60
|
|
|
60
61
|
def build(report, head, scope)
|
|
61
|
-
{ "version" =>
|
|
62
|
+
{ "version" => 2, "date" => Time.now.utc.iso8601, "commit" => head,
|
|
62
63
|
"selection" => Digest::SHA256.hexdigest(JSON.generate(scope.sort_by { |key, _| key.to_s }.to_h)),
|
|
63
64
|
"inventory" => Digest::SHA256.hexdigest(JSON.generate(report.results.map(&:digest).sort)),
|
|
64
65
|
"score" => overall_score(report), "coveredScore" => report.score,
|
|
65
66
|
"counts" => report.counts.transform_keys(&:to_s),
|
|
66
67
|
"breakdown" => report.score_breakdown.to_h.transform_keys(&:to_s),
|
|
67
|
-
"incomplete" => incompleteness(report) }
|
|
68
|
+
"incomplete" => incompleteness(report), "operatorPolicy" => report.operator_policy }
|
|
68
69
|
end
|
|
69
70
|
|
|
70
71
|
def overall_score(report)
|
|
@@ -107,11 +108,22 @@ module Saboteur
|
|
|
107
108
|
previous
|
|
108
109
|
end
|
|
109
110
|
|
|
111
|
+
# Only one record version and one non-null operator policy compare:
|
|
112
|
+
# scores never mix across v1/v2 or across policy configurations.
|
|
113
|
+
# A non-null previous policy forces a non-null equal current one, so
|
|
114
|
+
# only the previous record needs the null guard.
|
|
115
|
+
def comparable_policy?(previous, current)
|
|
116
|
+
previous["version"] == current["version"] &&
|
|
117
|
+
!previous["operatorPolicy"].nil? &&
|
|
118
|
+
previous["operatorPolicy"] == current["operatorPolicy"]
|
|
119
|
+
end
|
|
120
|
+
|
|
110
121
|
def change(previous, current)
|
|
111
122
|
return "score history: first record" unless previous
|
|
112
123
|
return DIFFERENT_RUN unless
|
|
113
124
|
previous["selection"] == current["selection"] && previous["inventory"] == current["inventory"] &&
|
|
114
125
|
previous["incomplete"].empty? && current["incomplete"].empty?
|
|
126
|
+
return DIFFERENT_POLICY unless comparable_policy?(previous, current)
|
|
115
127
|
return MISSING_SCORE if
|
|
116
128
|
previous["coveredScore"].nil? || current["coveredScore"].nil?
|
|
117
129
|
|
|
@@ -40,13 +40,15 @@ module Saboteur
|
|
|
40
40
|
warnings: report.warnings,
|
|
41
41
|
framework: { name: "saboteur", version: VERSION },
|
|
42
42
|
saboteur: { findings: Findings.build(report), scores: scores(report),
|
|
43
|
-
inlineSuppressions: report
|
|
44
|
-
|
|
45
|
-
end }
|
|
46
|
-
.merge(exit_cause ? { exitCause: exit_cause } : {})
|
|
43
|
+
inlineSuppressions: inline_audit(report) }
|
|
44
|
+
.merge(operatorPolicy: report.operator_policy).merge(exit_cause ? { exitCause: exit_cause } : {})
|
|
47
45
|
}.merge(report.seed ? { config: { seed: report.seed } } : {})
|
|
48
46
|
end
|
|
49
47
|
|
|
48
|
+
def self.inline_audit(report)
|
|
49
|
+
report.inline_suppressions.map { |entry| entry.merge(file: relative(entry[:file], report.root)) }
|
|
50
|
+
end
|
|
51
|
+
|
|
50
52
|
# Score with its assertion and crash shares, for all mutants and without
|
|
51
53
|
# the whole-body ones (body_nil, body_true, body_false, body_raise).
|
|
52
54
|
def self.scores(report)
|
|
@@ -34,11 +34,20 @@ module Saboteur
|
|
|
34
34
|
warnings(report)
|
|
35
35
|
early_stop(report)
|
|
36
36
|
test_matrix(report)
|
|
37
|
+
operator_policy(report)
|
|
37
38
|
summary(report)
|
|
38
39
|
end
|
|
39
40
|
|
|
40
41
|
private
|
|
41
42
|
|
|
43
|
+
def operator_policy(report)
|
|
44
|
+
policy = report.operator_policy
|
|
45
|
+
return unless policy && policy["excluded"].any?
|
|
46
|
+
|
|
47
|
+
@out.puts("Excluded operators: #{policy['excluded'].join(', ')} " \
|
|
48
|
+
"(policy #{policy['version']}; #{policy['effective'].size} enabled)")
|
|
49
|
+
end
|
|
50
|
+
|
|
42
51
|
def test_matrix(report)
|
|
43
52
|
return unless report.matrix?
|
|
44
53
|
|
data/lib/saboteur/result.rb
CHANGED
|
@@ -112,13 +112,15 @@ module Saboteur
|
|
|
112
112
|
MAX_ERROR_FRACTION = 0.5
|
|
113
113
|
|
|
114
114
|
attr_reader :results, :subjects, :wall_time, :invalid_mutations, :warnings, :root, :seed, :skipped_subjects,
|
|
115
|
-
:generation_events, :omissions, :run_evidence, :hazard_suppressions, :early_stop, :inline_suppressions
|
|
115
|
+
:generation_events, :omissions, :run_evidence, :hazard_suppressions, :early_stop, :inline_suppressions,
|
|
116
|
+
:operator_policy
|
|
116
117
|
|
|
117
118
|
# REVIEW: diff mode; the terminal then shows one survivor per line.
|
|
118
119
|
# seed: the random seed every test run started from (nil when unknown).
|
|
119
120
|
def initialize(results:, subjects:, wall_time:, root:, invalid_mutations: 0, warnings: [], review: false,
|
|
120
121
|
seed: nil, skipped_subjects: {}, diff_scope: nil, generation_events: [], omissions: [],
|
|
121
|
-
run_evidence: nil, hazard_suppressions: 0, early_stop: nil, matrix: false, inline_suppressions: []
|
|
122
|
+
run_evidence: nil, hazard_suppressions: 0, early_stop: nil, matrix: false, inline_suppressions: [],
|
|
123
|
+
operator_policy: nil)
|
|
122
124
|
@results = results
|
|
123
125
|
@subjects = subjects
|
|
124
126
|
@wall_time = wall_time
|
|
@@ -136,6 +138,7 @@ module Saboteur
|
|
|
136
138
|
@early_stop = early_stop
|
|
137
139
|
@matrix = matrix
|
|
138
140
|
@inline_suppressions = inline_suppressions.dup.freeze
|
|
141
|
+
@operator_policy = operator_policy
|
|
139
142
|
end
|
|
140
143
|
|
|
141
144
|
# A --matrix run: every covering test ran per mutant.
|
data/lib/saboteur/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: saboteur
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Vamsi Pavan Mahesh Gunturu
|
|
@@ -110,6 +110,7 @@ files:
|
|
|
110
110
|
- lib/saboteur/execution/supervisor.rb
|
|
111
111
|
- lib/saboteur/execution/worker_pool.rb
|
|
112
112
|
- lib/saboteur/execution/wrapper.rb
|
|
113
|
+
- lib/saboteur/frameworks/database_hosts.rb
|
|
113
114
|
- lib/saboteur/frameworks/rails.rb
|
|
114
115
|
- lib/saboteur/gates.rb
|
|
115
116
|
- lib/saboteur/help.rb
|
|
@@ -132,6 +133,7 @@ files:
|
|
|
132
133
|
- lib/saboteur/mutation_source/literal_replacement.rb
|
|
133
134
|
- lib/saboteur/mutation_source/mutation.rb
|
|
134
135
|
- lib/saboteur/mutation_source/operator_assignment.rb
|
|
136
|
+
- lib/saboteur/mutation_source/operator_examples.rb
|
|
135
137
|
- lib/saboteur/mutation_source/operator_registry.rb
|
|
136
138
|
- lib/saboteur/mutation_source/pair_removal.rb
|
|
137
139
|
- lib/saboteur/mutation_source/range_replacement.rb
|
|
@@ -145,6 +147,7 @@ files:
|
|
|
145
147
|
- lib/saboteur/reporter/diff.rb
|
|
146
148
|
- lib/saboteur/reporter/findings.rb
|
|
147
149
|
- lib/saboteur/reporter/history_paths.rb
|
|
150
|
+
- lib/saboteur/reporter/history_policy.rb
|
|
148
151
|
- lib/saboteur/reporter/history_record.rb
|
|
149
152
|
- lib/saboteur/reporter/html.rb
|
|
150
153
|
- lib/saboteur/reporter/mutation_text.rb
|