mutineer 1.8.2 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +611 -0
- data/README.md +84 -49
- data/lib/mutineer/changed_lines.rb +2 -1
- data/lib/mutineer/child_cause.rb +149 -0
- data/lib/mutineer/child_stdout.rb +12 -1
- data/lib/mutineer/cli.rb +536 -58
- data/lib/mutineer/command_redact.rb +38 -0
- data/lib/mutineer/config.rb +112 -6
- data/lib/mutineer/coverage_map.rb +1345 -156
- data/lib/mutineer/daemon_backend.rb +40 -17
- data/lib/mutineer/daemon_client.rb +63 -15
- data/lib/mutineer/daemon_server.rb +75 -5
- data/lib/mutineer/database_urls.rb +189 -0
- data/lib/mutineer/external_backend.rb +47 -1
- data/lib/mutineer/fs_path.rb +125 -0
- data/lib/mutineer/isolation.rb +374 -20
- data/lib/mutineer/job_plan.rb +349 -39
- data/lib/mutineer/minitest_integration/file_order.rb +56 -0
- data/lib/mutineer/minitest_integration/in_test_database_error.rb +243 -19
- data/lib/mutineer/minitest_integration.rb +26 -58
- data/lib/mutineer/mutant_id.rb +4 -1
- data/lib/mutineer/mutation.rb +6 -1
- data/lib/mutineer/mutator_registry.rb +6 -3
- data/lib/mutineer/mutators/collection_method.rb +195 -6
- data/lib/mutineer/mutators/literal_mutation.rb +33 -3
- data/lib/mutineer/mutators/range_literal.rb +78 -29
- data/lib/mutineer/mutators/rescue_body.rb +126 -0
- data/lib/mutineer/mutators/return_nil.rb +23 -6
- data/lib/mutineer/output_filter.rb +451 -0
- data/lib/mutineer/pairing.rb +463 -31
- data/lib/mutineer/parser.rb +14 -0
- data/lib/mutineer/project_path.rb +15 -8
- data/lib/mutineer/rails_worker_db.rb +356 -6
- data/lib/mutineer/reach_probe.rb +229 -0
- data/lib/mutineer/redaction.rb +264 -32
- data/lib/mutineer/reload_guards.rb +460 -39
- data/lib/mutineer/reporter.rb +465 -57
- data/lib/mutineer/result.rb +6 -7
- data/lib/mutineer/run_seed.rb +16 -0
- data/lib/mutineer/runner.rb +175 -51
- data/lib/mutineer/source_bytes.rb +187 -0
- data/lib/mutineer/statement_lines.rb +108 -14
- data/lib/mutineer/test_runners/rspec.rb +118 -10
- data/lib/mutineer/test_runners/rspec_seed.rb +151 -0
- data/lib/mutineer/test_runners.rb +3 -2
- data/lib/mutineer/triage.rb +55 -10
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +1 -0
- data/skills/mutineer/SKILL.md +12 -4
- metadata +12 -1
|
@@ -27,31 +27,125 @@ module Mutineer
|
|
|
27
27
|
# counts when the file loads, and `x if done; y(` counts when `y` does
|
|
28
28
|
# not run.
|
|
29
29
|
def self.for(def_node, source, offset, loose: false)
|
|
30
|
-
|
|
30
|
+
statement = statement_at(def_node, offset, conditional: !loose)
|
|
31
|
+
return [] unless statement
|
|
32
|
+
|
|
33
|
+
location = statement.location
|
|
34
|
+
lines = (location.start_line..last_line(statement)).to_a
|
|
35
|
+
return lines - [def_node.location.start_line] if loose
|
|
36
|
+
|
|
37
|
+
strict_lines(source, location, lines)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Lines of the statement that {for} drops because they share the line with
|
|
41
|
+
# other code, when that drop leaves no line. Empty when {for} keeps a line,
|
|
42
|
+
# when no statement holds the offset, or when the code runs only sometimes
|
|
43
|
+
# (a branch, the right side of `&&`, the body of `x if c`): those lines can
|
|
44
|
+
# count without this code (#187, #254).
|
|
45
|
+
#
|
|
46
|
+
# A one-line block on a call that ran at load has no strict line of its own.
|
|
47
|
+
# The call's line can count without the block: `each` on an empty array
|
|
48
|
+
# still counts the line. {JobPlan.ran_at_load?} does not treat that as the
|
|
49
|
+
# block having run. It uses these lines only for a `define_method` or
|
|
50
|
+
# `define_singleton_method` call (#435).
|
|
51
|
+
#
|
|
52
|
+
# @param def_node [Prism::DefNode] the method that holds the position.
|
|
53
|
+
# @param source [String] the source text of the file.
|
|
54
|
+
# @param offset [Integer] byte offset of the position.
|
|
55
|
+
# @return [Array<Integer>] the dropped lines, or an empty array.
|
|
56
|
+
def self.shared_lines(def_node, source, offset)
|
|
57
|
+
statement = statement_at(def_node, offset, conditional: true)
|
|
58
|
+
return [] unless statement
|
|
59
|
+
|
|
60
|
+
location = statement.location
|
|
61
|
+
lines = (location.start_line..last_line(statement)).to_a
|
|
62
|
+
return [] unless strict_lines(source, location, lines.dup).empty?
|
|
63
|
+
|
|
64
|
+
lines
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
# The statement that holds `offset`, or nil when none does. With
|
|
68
|
+
# `conditional`, also nil when the code runs only sometimes ({sometimes?}
|
|
69
|
+
# or a modifier such as `x if c`).
|
|
70
|
+
#
|
|
71
|
+
# @param def_node [Prism::DefNode, nil] the method that holds the position.
|
|
72
|
+
# @param offset [Integer] byte offset of the position.
|
|
73
|
+
# @param conditional [Boolean] drop code that runs only sometimes.
|
|
74
|
+
# @return [Prism::Node, nil]
|
|
75
|
+
def self.statement_at(def_node, offset, conditional:)
|
|
76
|
+
return nil unless def_node&.body
|
|
31
77
|
|
|
32
78
|
path = path_to(def_node.body, offset) || []
|
|
33
79
|
index = path.rindex(&:newline?)
|
|
34
|
-
return
|
|
80
|
+
return nil unless index
|
|
81
|
+
return nil if conditional && conditional_code?(path, index)
|
|
35
82
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
# the statement need not have run it.
|
|
40
|
-
return [] if path[index..].each_cons(2).any? { |parent, child| sometimes?(parent, child) }
|
|
83
|
+
path[index]
|
|
84
|
+
end
|
|
85
|
+
private_class_method :statement_at
|
|
41
86
|
|
|
87
|
+
# Whether the statement at `index` runs only sometimes when its parent runs.
|
|
88
|
+
#
|
|
89
|
+
# @param path [Array<Prism::Node>] nodes from the method body down to the offset.
|
|
90
|
+
# @param index [Integer] index of the statement on `path`.
|
|
91
|
+
# @return [Boolean]
|
|
92
|
+
def self.conditional_code?(path, index)
|
|
93
|
+
# Code that runs only sometimes inside the statement: the tests that ran
|
|
94
|
+
# the statement need not have run it.
|
|
95
|
+
path[index..].each_cons(2).any? { |parent, child| sometimes?(parent, child) } ||
|
|
42
96
|
# Ruby counts the line of `x while c` or `x if c` when it checks `c`,
|
|
43
97
|
# also when `x` never runs. The path is the modifier, its body, statement.
|
|
44
|
-
|
|
98
|
+
(index >= 2 && modifier?(path[index - 2]))
|
|
99
|
+
end
|
|
100
|
+
private_class_method :conditional_code?
|
|
101
|
+
|
|
102
|
+
# `lines` with a first or last line removed when that line holds other code.
|
|
103
|
+
# Ruby can count the line without this statement: the `def` line counts when
|
|
104
|
+
# the file loads, and `x if done; y(` counts when `y` does not run.
|
|
105
|
+
#
|
|
106
|
+
# @param source [String] the source text of the file.
|
|
107
|
+
# @param location [Prism::Location] the statement's location.
|
|
108
|
+
# @param lines [Array<Integer>] the statement's lines; modified in place.
|
|
109
|
+
# @return [Array<Integer>] `lines`.
|
|
110
|
+
def self.strict_lines(source, location, lines)
|
|
111
|
+
lines.delete(location.start_line) unless before(source, location).strip.empty?
|
|
112
|
+
rest = after(source, location).strip
|
|
113
|
+
lines.delete(location.end_line) unless rest.empty? || rest.start_with?("#")
|
|
114
|
+
lines
|
|
115
|
+
end
|
|
116
|
+
private_class_method :strict_lines
|
|
117
|
+
|
|
118
|
+
# Calls that store a block as a method. The block does not run when the
|
|
119
|
+
# call does, so a mutant in the call or in the block is the original method
|
|
120
|
+
# once that call has run at load (#435).
|
|
121
|
+
STORED_METHOD_CALLS = %i[define_method define_singleton_method].freeze
|
|
122
|
+
|
|
123
|
+
# Lines of the nearest {STORED_METHOD_CALLS} call that holds `offset`: the
|
|
124
|
+
# call's strict lines, or the line it shares with other code when strict
|
|
125
|
+
# mode drops every line. Empty when `offset` is not in such a call, or when
|
|
126
|
+
# that call runs only sometimes. A line here ran only if the caller recorded
|
|
127
|
+
# it; an empty `each` on the same line can count without this call (#286).
|
|
128
|
+
#
|
|
129
|
+
# @param def_node [Prism::DefNode] the method that holds the position.
|
|
130
|
+
# @param source [String] the source text of the file.
|
|
131
|
+
# @param offset [Integer] byte offset of the position.
|
|
132
|
+
# @return [Array<Integer>]
|
|
133
|
+
def self.stored_method_lines(def_node, source, offset)
|
|
134
|
+
return [] unless def_node&.body
|
|
135
|
+
|
|
136
|
+
path = path_to(def_node.body, offset)
|
|
137
|
+
call = path&.reverse_each&.find do |node|
|
|
138
|
+
node.is_a?(Prism::CallNode) && STORED_METHOD_CALLS.include?(node.name)
|
|
45
139
|
end
|
|
140
|
+
return [] unless call
|
|
141
|
+
|
|
142
|
+
statement = statement_at(def_node, call.location.start_offset, conditional: true)
|
|
143
|
+
return [] unless statement
|
|
46
144
|
|
|
47
145
|
location = statement.location
|
|
48
146
|
lines = (location.start_line..last_line(statement)).to_a
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
lines.delete(location.start_line) unless before(source, location).strip.empty?
|
|
52
|
-
after = after(source, location).strip
|
|
53
|
-
lines.delete(location.end_line) unless after.empty? || after.start_with?("#")
|
|
54
|
-
lines
|
|
147
|
+
kept = strict_lines(source, location, lines.dup)
|
|
148
|
+
kept.empty? ? lines : kept
|
|
55
149
|
end
|
|
56
150
|
|
|
57
151
|
# Whether the code at `offset` runs each time the method runs: it is in
|
|
@@ -1,7 +1,11 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "stringio"
|
|
4
|
+
require_relative "../fs_path"
|
|
5
|
+
require_relative "../child_cause"
|
|
4
6
|
require_relative "../kill_channel"
|
|
7
|
+
require_relative "rspec_seed"
|
|
8
|
+
require_relative "../minitest_integration/in_test_database_error"
|
|
5
9
|
|
|
6
10
|
module Mutineer
|
|
7
11
|
# Raised when the target project asks for a framework whose gem isn't present.
|
|
@@ -12,9 +16,18 @@ module Mutineer
|
|
|
12
16
|
# Child-process-only RSpec runner.
|
|
13
17
|
#
|
|
14
18
|
# Mirrors MinitestIntegration's contract: run the given spec files and
|
|
15
|
-
# return 0 (all passed), 1 (an example failed) or 2 (
|
|
16
|
-
# outside examples, such as a spec file that raised while
|
|
19
|
+
# return 0 (all passed), 1 (an example failed) or 2 (no verdict). Exit 2
|
|
20
|
+
# is an error outside examples, such as a spec file that raised while
|
|
21
|
+
# loading, or a run whose first failed example failed with an Active
|
|
22
|
+
# Record connection or lock error (#347).
|
|
17
23
|
module RSpec
|
|
24
|
+
# The sink text from the most recent {.run} (#360 clean-failure detail).
|
|
25
|
+
#
|
|
26
|
+
# @return [String, nil]
|
|
27
|
+
def self.last_output
|
|
28
|
+
@last_output
|
|
29
|
+
end
|
|
30
|
+
|
|
18
31
|
# Runs the given RSpec files.
|
|
19
32
|
#
|
|
20
33
|
# @param spec_files [String, Array<String>] one file or many files.
|
|
@@ -30,6 +43,7 @@ module Mutineer
|
|
|
30
43
|
end
|
|
31
44
|
|
|
32
45
|
require_rspec!
|
|
46
|
+
RSpecSeed.install!
|
|
33
47
|
|
|
34
48
|
::RSpec::Core::Runner.disable_autorun!
|
|
35
49
|
# Not RSpec.reset: it also drops the settings that gems added when Bundler
|
|
@@ -45,36 +59,109 @@ module Mutineer
|
|
|
45
59
|
# silenced here: the fork boundary that calls this method has already
|
|
46
60
|
# pointed stdout at File::NULL (see ChildStdout).
|
|
47
61
|
sink = StringIO.new
|
|
48
|
-
|
|
62
|
+
# RSpec matches the spec path with a UTF-8 regexp. A binary path
|
|
63
|
+
# under LC_ALL=C raises before the example runs (#346).
|
|
64
|
+
files = Array(spec_files).map { |file| FsPath.read(file) }
|
|
65
|
+
# Clean checks pass no hook. They still use this file order, so they
|
|
66
|
+
# share it with the mutant run (#282).
|
|
67
|
+
keep_file_order(files)
|
|
49
68
|
args = ["--no-color"]
|
|
50
69
|
args << "--fail-fast" if stop_at_first_failure
|
|
51
70
|
status = with_fail_fast_off(record_to) do
|
|
52
|
-
::RSpec::Core::Runner.run([*args, *
|
|
71
|
+
::RSpec::Core::Runner.run([*args, *files], sink, sink)
|
|
53
72
|
end
|
|
73
|
+
@last_output = sink.string
|
|
54
74
|
# Reached only when the suite returned: an example that exits the
|
|
55
75
|
# process skips it, and the parent then keeps the row incomplete. A
|
|
56
76
|
# return alone proves nothing: fail-fast turned on mid-run stops RSpec
|
|
57
77
|
# early, so `end` needs a reported outcome for every planned example.
|
|
58
78
|
KillChannel.write_end(record_to) if record_to && KillFormatter.last&.saw_every_example?
|
|
59
79
|
|
|
60
|
-
status.zero?
|
|
80
|
+
return 0 if status.zero?
|
|
81
|
+
|
|
82
|
+
code = failure_code
|
|
83
|
+
# Fixed cause when RSpec failed outside examples (#417 / KTD5). Status 2
|
|
84
|
+
# is also an in-example database error (#347), so test the predicate,
|
|
85
|
+
# not the status. Write here so failure_code stays a pure status helper.
|
|
86
|
+
ChildCause.write_text(ChildCause::RSPEC_OUTSIDE_EXAMPLES) if outside_examples?
|
|
87
|
+
code
|
|
61
88
|
ensure
|
|
62
89
|
FileOrder.rank = nil
|
|
63
90
|
end
|
|
64
91
|
|
|
65
92
|
# The exit status of a failed run: 2 when no example failed and RSpec
|
|
66
|
-
# recorded an error outside examples,
|
|
93
|
+
# recorded an error outside examples, or when the first failed example's
|
|
94
|
+
# own exception is an Active Record connection or lock error (#347).
|
|
95
|
+
# Otherwise 1.
|
|
96
|
+
#
|
|
97
|
+
# A later `after` hook is not the example's own exception. RSpec keeps
|
|
98
|
+
# the first one and wraps a later hook error around it, the same way
|
|
99
|
+
# Minitest records a teardown error after the setup or body error.
|
|
67
100
|
#
|
|
68
101
|
# @api private
|
|
69
102
|
# @return [Integer] 1 or 2.
|
|
70
103
|
def self.failure_code
|
|
104
|
+
return 2 if outside_examples?
|
|
105
|
+
|
|
106
|
+
if (error = first_failed_database_error)
|
|
107
|
+
MinitestIntegration::InTestDatabaseError.warn_database_error(error)
|
|
108
|
+
return 2
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
1
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
# True when RSpec recorded an error outside examples and no example failed.
|
|
115
|
+
#
|
|
116
|
+
# Exit status 2 is also an in-example database error ({.failure_code}).
|
|
117
|
+
# A cause line for the outside-examples case has to test this predicate,
|
|
118
|
+
# not a status of 2. Pull request #443 writes that cause when the status
|
|
119
|
+
# is 2; after this change that condition would label a database error as
|
|
120
|
+
# an error outside examples (#347, #417).
|
|
121
|
+
#
|
|
122
|
+
# @api private
|
|
123
|
+
# @return [Boolean]
|
|
124
|
+
def self.outside_examples?
|
|
71
125
|
world = ::RSpec.world
|
|
72
|
-
|
|
73
|
-
|
|
126
|
+
outside = world.respond_to?(:non_example_failure) && world.non_example_failure
|
|
127
|
+
!!(outside && ::RSpec.configuration.reporter.failed_examples.empty?)
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
# The database error of the first failed example, in run order, or nil
|
|
131
|
+
# when that example failed for another reason.
|
|
132
|
+
#
|
|
133
|
+
# @api private
|
|
134
|
+
# @return [Exception, nil]
|
|
135
|
+
def self.first_failed_database_error
|
|
136
|
+
example = ::RSpec.configuration.reporter.failed_examples.first
|
|
137
|
+
return unless example
|
|
138
|
+
|
|
139
|
+
error = primary_exception(example.exception)
|
|
140
|
+
error if MinitestIntegration::InTestDatabaseError.database_exception?(error)
|
|
141
|
+
rescue StandardError
|
|
142
|
+
nil
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
# The exception that ended the example, before a later hook error.
|
|
146
|
+
# `RSpec::Core::MultipleExceptionError` keeps that first exception at
|
|
147
|
+
# the front of `all_exceptions` and appends each later hook error.
|
|
148
|
+
#
|
|
149
|
+
# @api private
|
|
150
|
+
# @param error [Exception, nil]
|
|
151
|
+
# @return [Exception, nil]
|
|
152
|
+
def self.primary_exception(error)
|
|
153
|
+
nested = ::RSpec::Core::MultipleExceptionError
|
|
154
|
+
return error.all_exceptions.first if error.is_a?(nested) && error.respond_to?(:all_exceptions)
|
|
155
|
+
|
|
156
|
+
error
|
|
157
|
+
rescue StandardError
|
|
158
|
+
error
|
|
74
159
|
end
|
|
75
160
|
|
|
76
161
|
# Runs the top-level example groups in the order of `spec_files` (#203),
|
|
77
162
|
# after RSpec's own ordering, which then decides the order within a file.
|
|
163
|
+
# Every run uses it, including the clean check, so a mutant run and the
|
|
164
|
+
# clean check of the same files share that order (#282).
|
|
78
165
|
#
|
|
79
166
|
# @api private
|
|
80
167
|
# @param spec_files [String, Array<String>] the files, in run order.
|
|
@@ -238,7 +325,8 @@ module Mutineer
|
|
|
238
325
|
# @param notification [RSpec::Core::Notifications::FailedExampleNotification]
|
|
239
326
|
# @return [void]
|
|
240
327
|
def example_failed(notification)
|
|
241
|
-
|
|
328
|
+
example = notification.example
|
|
329
|
+
send_event(example_event(example), example)
|
|
242
330
|
example_finished
|
|
243
331
|
@failed = true
|
|
244
332
|
end
|
|
@@ -302,9 +390,29 @@ module Mutineer
|
|
|
302
390
|
KillChannel.write_unskip(@channel)
|
|
303
391
|
end
|
|
304
392
|
|
|
393
|
+
# {KillChannel::UNAVAILABLE} when the example's own exception is a
|
|
394
|
+
# database error, otherwise {KillChannel::KILL}. A fault here is a
|
|
395
|
+
# kill: recording nothing would hide a failure, and raising would
|
|
396
|
+
# change the verdict.
|
|
397
|
+
#
|
|
398
|
+
# @param example [RSpec::Core::Example] the failed example.
|
|
399
|
+
# @return [String]
|
|
400
|
+
def example_event(example)
|
|
401
|
+
error = example.exception if example.respond_to?(:exception)
|
|
402
|
+
error = Mutineer::TestRunners::RSpec.primary_exception(error)
|
|
403
|
+
if error && Mutineer::MinitestIntegration::InTestDatabaseError.database_exception?(error)
|
|
404
|
+
KillChannel::UNAVAILABLE
|
|
405
|
+
else
|
|
406
|
+
KillChannel::KILL
|
|
407
|
+
end
|
|
408
|
+
rescue StandardError
|
|
409
|
+
KillChannel::KILL
|
|
410
|
+
end
|
|
411
|
+
|
|
305
412
|
# Writes one test line for `example`.
|
|
306
413
|
#
|
|
307
|
-
# @param event [String] {KillChannel::PASS}
|
|
414
|
+
# @param event [String] {KillChannel::PASS}, {KillChannel::KILL}, or
|
|
415
|
+
# {KillChannel::UNAVAILABLE}.
|
|
308
416
|
# @param example [RSpec::Core::Example] the example.
|
|
309
417
|
# @return [void]
|
|
310
418
|
def send_event(event, example)
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../run_seed"
|
|
4
|
+
|
|
5
|
+
module Mutineer
|
|
6
|
+
module TestRunners
|
|
7
|
+
# Gives a random RSpec order the same seed on the clean check and the
|
|
8
|
+
# mutant runs (#282). A defined order is left as it is: assigning
|
|
9
|
+
# `config.seed` would switch the suite to random. A seed the suite sets
|
|
10
|
+
# (`--seed`, `--order rand:N`, `SPEC_OPTS`, `.rspec`, or `config.seed=`)
|
|
11
|
+
# wins. The hooks install before boot, `--require` files, and spec files
|
|
12
|
+
# load, so a `config.seed=` in any of those is visible. The seed is
|
|
13
|
+
# written after spec files load and before RSpec orders examples.
|
|
14
|
+
#
|
|
15
|
+
# A spec helper that only reads `config.seed` (the generated
|
|
16
|
+
# `Kernel.srand config.seed`) does not count as setting it. The read
|
|
17
|
+
# still sees RSpec's temporary seed; the examples then run under the
|
|
18
|
+
# pinned seed.
|
|
19
|
+
module RSpecSeed
|
|
20
|
+
# Installs the hooks. Idempotent. Requires `rspec/core`, which the
|
|
21
|
+
# caller has already loaded.
|
|
22
|
+
#
|
|
23
|
+
# @api private
|
|
24
|
+
# @return [void]
|
|
25
|
+
def self.install!
|
|
26
|
+
return if ::RSpec::Core::Configuration.ancestors.include?(Watch)
|
|
27
|
+
|
|
28
|
+
::RSpec::Core::Configuration.prepend(Watch)
|
|
29
|
+
::RSpec::Core::World.prepend(PinBeforeOrder)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# True while {pin!} is assigning the shared seed, so that assignment
|
|
33
|
+
# is not recorded as the suite's own seed.
|
|
34
|
+
#
|
|
35
|
+
# @api private
|
|
36
|
+
# @return [Boolean]
|
|
37
|
+
def self.pinning?
|
|
38
|
+
@pinning == true
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# True when `value` is an order string that carries a seed, such as
|
|
42
|
+
# `"rand:12"` or `"random:12"`. RSpec stores `--seed N` as
|
|
43
|
+
# `"rand:N"`.
|
|
44
|
+
#
|
|
45
|
+
# @api private
|
|
46
|
+
# @param value [Object]
|
|
47
|
+
# @return [Boolean]
|
|
48
|
+
def self.embedded_seed?(value)
|
|
49
|
+
_order, seed = value.to_s.split(":", 2)
|
|
50
|
+
!seed.nil? && !seed.empty? && seed.match?(/\A\d+\z/)
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# The seed named in the not-green message. `SPEC_OPTS` is what the
|
|
54
|
+
# parent can see. A `.rspec` file or `config.seed` is applied in the
|
|
55
|
+
# child and is not part of this string.
|
|
56
|
+
#
|
|
57
|
+
# @api private
|
|
58
|
+
# @return [String]
|
|
59
|
+
def self.advertised_seed
|
|
60
|
+
opts = ENV.fetch("SPEC_OPTS", "")
|
|
61
|
+
found = []
|
|
62
|
+
opts.scan(/--seed(?:=|\s+)(\d+)|--order(?:=|\s+)rand(?:om)?:(\d+)/) do |seed, order|
|
|
63
|
+
found << (seed || order)
|
|
64
|
+
end
|
|
65
|
+
found.last || RunSeed::PINNED_SEED.to_s
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Sets the shared seed when the loaded suite's order is random and the
|
|
69
|
+
# suite did not set a seed. Re-seeds Ruby's global generator so
|
|
70
|
+
# `rand` in examples matches, after a spec helper's `Kernel.srand`.
|
|
71
|
+
#
|
|
72
|
+
# @api private
|
|
73
|
+
# @param config [RSpec::Core::Configuration]
|
|
74
|
+
# @return [void]
|
|
75
|
+
def self.pin!(config)
|
|
76
|
+
return if config.instance_variable_get(:@mutineer_seed_explicit)
|
|
77
|
+
return unless random_order?(config)
|
|
78
|
+
|
|
79
|
+
@pinning = true
|
|
80
|
+
begin
|
|
81
|
+
config.seed = RunSeed::PINNED_SEED
|
|
82
|
+
Kernel.srand(config.seed)
|
|
83
|
+
ensure
|
|
84
|
+
@pinning = false
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# True when the global ordering strategy is RSpec's random one.
|
|
89
|
+
#
|
|
90
|
+
# @api private
|
|
91
|
+
# @param config [RSpec::Core::Configuration]
|
|
92
|
+
# @return [Boolean]
|
|
93
|
+
def self.random_order?(config)
|
|
94
|
+
registry = config.ordering_registry
|
|
95
|
+
registry.fetch(:global).equal?(registry.fetch(:random))
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
# Prepended on `RSpec::Core::Configuration`. Records a seed the suite
|
|
99
|
+
# sets, from the command, `SPEC_OPTS`, `.rspec`, or the spec helper.
|
|
100
|
+
module Watch
|
|
101
|
+
# CLI, `SPEC_OPTS` and `.rspec` arrive here. `--seed` is passed as
|
|
102
|
+
# an order of `rand:N`.
|
|
103
|
+
#
|
|
104
|
+
# @param hash [Hash]
|
|
105
|
+
# @return [void]
|
|
106
|
+
def force(hash)
|
|
107
|
+
mark_seed_explicit if hash.key?(:seed) || RSpecSeed.embedded_seed?(hash[:order])
|
|
108
|
+
super
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
# A spec helper's `config.seed =`.
|
|
112
|
+
#
|
|
113
|
+
# @param value [Object]
|
|
114
|
+
# @return [void]
|
|
115
|
+
def seed=(value)
|
|
116
|
+
mark_seed_explicit unless RSpecSeed.pinning?
|
|
117
|
+
super
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# A spec helper's `config.order = "rand:N"`.
|
|
121
|
+
#
|
|
122
|
+
# @param value [Object]
|
|
123
|
+
# @return [void]
|
|
124
|
+
def order=(value)
|
|
125
|
+
mark_seed_explicit if RSpecSeed.embedded_seed?(value)
|
|
126
|
+
super
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# Records that this configuration object set its own seed.
|
|
130
|
+
#
|
|
131
|
+
# @api private
|
|
132
|
+
# @return [void]
|
|
133
|
+
def mark_seed_explicit
|
|
134
|
+
@mutineer_seed_explicit = true
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
# Prepended on `RSpec::Core::World`. Pins the seed after spec files
|
|
139
|
+
# load and before examples are ordered.
|
|
140
|
+
module PinBeforeOrder
|
|
141
|
+
# The groups RSpec is about to run.
|
|
142
|
+
#
|
|
143
|
+
# @return [Array<RSpec::Core::ExampleGroup>]
|
|
144
|
+
def ordered_example_groups
|
|
145
|
+
RSpecSeed.pin!(::RSpec.configuration)
|
|
146
|
+
super
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
end
|
|
150
|
+
end
|
|
151
|
+
end
|
|
@@ -9,8 +9,9 @@ module Mutineer
|
|
|
9
9
|
# Each runner responds to
|
|
10
10
|
# `.run(files, stop_at_first_failure: false, record_to: nil) -> 0/1/2`
|
|
11
11
|
# (0 = all passed, 1 = any failure, 2 = no verdict: an error outside tests,
|
|
12
|
-
# which the RSpec runner reports, or a
|
|
13
|
-
#
|
|
12
|
+
# which the RSpec runner reports, or a run whose first failure was an Active
|
|
13
|
+
# Record connection or lock error, on Minitest and on RSpec, #296, #347) and
|
|
14
|
+
# is called only inside
|
|
14
15
|
# a forked child.
|
|
15
16
|
# Only the mutant path passes `stop_at_first_failure: true`; a `--matrix`
|
|
16
17
|
# mutant run passes `record_to:` instead (a {KillChannel} pipe) and runs every
|
data/lib/mutineer/triage.rb
CHANGED
|
@@ -5,9 +5,10 @@ require "optparse"
|
|
|
5
5
|
require "yaml"
|
|
6
6
|
|
|
7
7
|
module Mutineer
|
|
8
|
-
# Prints `.mutineer.yml` `ignore:` entries for survivors in a JSON report
|
|
9
|
-
#
|
|
10
|
-
#
|
|
8
|
+
# Prints `.mutineer.yml` `ignore:` entries for survivors in a JSON report,
|
|
9
|
+
# and for each copy dropped as a repeat of one of those survivors (#374).
|
|
10
|
+
# An entry still ignores only its own id. The command writes to stdout and
|
|
11
|
+
# does not edit files. The user pastes the lines under `ignore:`.
|
|
11
12
|
module Triage
|
|
12
13
|
# Usage line for a bad invocation.
|
|
13
14
|
USAGE = "Usage: mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)"
|
|
@@ -59,17 +60,35 @@ module Mutineer
|
|
|
59
60
|
return 2
|
|
60
61
|
end
|
|
61
62
|
|
|
62
|
-
|
|
63
|
-
return 2 if
|
|
63
|
+
groups = survivor_groups(report, err)
|
|
64
|
+
return 2 if groups.nil?
|
|
64
65
|
|
|
65
|
-
|
|
66
|
-
|
|
66
|
+
by_id = groups.to_h
|
|
67
|
+
copy_owner = {}
|
|
68
|
+
groups.each { |id, copies| copies.each { |copy| copy_owner[copy] ||= id } }
|
|
69
|
+
|
|
70
|
+
chosen = all ? groups.map(&:first) : ids
|
|
71
|
+
missing = chosen.reject { |id| by_id.key?(id) || copy_owner.key?(id) }
|
|
67
72
|
unless missing.empty?
|
|
68
|
-
err.puts "mutineer: triage id is not a survivor: #{missing.join(', ')}"
|
|
73
|
+
err.puts "mutineer: triage id is not a survivor or one of its copies: #{missing.join(', ')}"
|
|
69
74
|
return 2
|
|
70
75
|
end
|
|
71
76
|
|
|
72
|
-
|
|
77
|
+
seen_owners = {}
|
|
78
|
+
seen_entry_ids = {}
|
|
79
|
+
entries = []
|
|
80
|
+
chosen.uniq.each do |id|
|
|
81
|
+
owner = by_id.key?(id) ? id : copy_owner[id]
|
|
82
|
+
next if seen_owners[owner]
|
|
83
|
+
|
|
84
|
+
seen_owners[owner] = true
|
|
85
|
+
([owner] + by_id[owner]).each do |entry_id|
|
|
86
|
+
next if seen_entry_ids[entry_id]
|
|
87
|
+
|
|
88
|
+
seen_entry_ids[entry_id] = true
|
|
89
|
+
entries << { "id" => entry_id, "reason" => text }
|
|
90
|
+
end
|
|
91
|
+
end
|
|
73
92
|
if entries.empty?
|
|
74
93
|
err.puts "mutineer: triage has nothing to paste under ignore:"
|
|
75
94
|
return 0
|
|
@@ -87,6 +106,18 @@ module Mutineer
|
|
|
87
106
|
# @param err [IO] usage errors.
|
|
88
107
|
# @return [Array<String>, nil]
|
|
89
108
|
def self.survivor_ids(path, err)
|
|
109
|
+
groups = survivor_groups(path, err)
|
|
110
|
+
groups&.map(&:first)
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
# Each survivor id and the dropped-copy ids listed on it, in report order.
|
|
114
|
+
# A `copies` value that is not a list counts as none. Nil when the file
|
|
115
|
+
# cannot be used, after a message on `err`.
|
|
116
|
+
#
|
|
117
|
+
# @param path [String] report path.
|
|
118
|
+
# @param err [IO] usage errors.
|
|
119
|
+
# @return [Array<Array(String, Array<String>)>, nil]
|
|
120
|
+
def self.survivor_groups(path, err)
|
|
90
121
|
unless File.file?(path)
|
|
91
122
|
err.puts "mutineer: triage report not found: #{path}"
|
|
92
123
|
return nil
|
|
@@ -99,7 +130,7 @@ module Mutineer
|
|
|
99
130
|
return nil
|
|
100
131
|
end
|
|
101
132
|
|
|
102
|
-
rows.filter_map { |row|
|
|
133
|
+
rows.filter_map { |row| group_for(row) }
|
|
103
134
|
rescue JSON::ParserError => e
|
|
104
135
|
err.puts "mutineer: triage report is not JSON: #{e.message}"
|
|
105
136
|
nil
|
|
@@ -107,5 +138,19 @@ module Mutineer
|
|
|
107
138
|
err.puts "mutineer: triage report cannot be read: #{path}"
|
|
108
139
|
nil
|
|
109
140
|
end
|
|
141
|
+
|
|
142
|
+
# One survivor and its copy ids, or nil when the row has no id.
|
|
143
|
+
#
|
|
144
|
+
# @api private
|
|
145
|
+
# @param row [Object] one `survivors[]` element.
|
|
146
|
+
# @return [Array(String, Array<String>), nil]
|
|
147
|
+
def self.group_for(row)
|
|
148
|
+
return unless row.is_a?(Hash) && row["id"].is_a?(String) && !row["id"].empty?
|
|
149
|
+
|
|
150
|
+
raw = row["copies"]
|
|
151
|
+
copies = raw.is_a?(Array) ? raw : []
|
|
152
|
+
copies = copies.select { |id| id.is_a?(String) && !id.empty? && id != row["id"] }.uniq
|
|
153
|
+
[row["id"], copies]
|
|
154
|
+
end
|
|
110
155
|
end
|
|
111
156
|
end
|
data/lib/mutineer/version.rb
CHANGED
data/lib/mutineer.rb
CHANGED
|
@@ -34,6 +34,7 @@ require_relative "mutineer/mutators/operand_removal"
|
|
|
34
34
|
require_relative "mutineer/mutators/array_literal"
|
|
35
35
|
require_relative "mutineer/mutators/condition_forcing"
|
|
36
36
|
require_relative "mutineer/mutators/operator_assignment"
|
|
37
|
+
require_relative "mutineer/mutators/rescue_body"
|
|
37
38
|
require_relative "mutineer/mutator_registry"
|
|
38
39
|
require_relative "mutineer/worker_pool"
|
|
39
40
|
require_relative "mutineer/progress"
|
data/skills/mutineer/SKILL.md
CHANGED
|
@@ -18,7 +18,7 @@ gem install mutineer
|
|
|
18
18
|
## Run
|
|
19
19
|
|
|
20
20
|
```sh
|
|
21
|
-
mutineer run <source...> [--test <test>...] [options] # no --test: pairs each source with its test by convention; unmatched sources are skipped; --dry-run with --rails/--boot needs --test
|
|
21
|
+
mutineer run <source...> [--test <test>...] [options] # no --test: pairs each source with its test by convention; unmatched sources are skipped and counted in the report; --dry-run with --rails/--boot needs --test
|
|
22
22
|
```
|
|
23
23
|
|
|
24
24
|
Example — mutate `lib/calculator.rb`, gate CI below 90%:
|
|
@@ -27,6 +27,11 @@ Example — mutate `lib/calculator.rb`, gate CI below 90%:
|
|
|
27
27
|
mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
+
A source that raises `NameError` (including `NoMethodError`) while it loads, with no boot file, needs `--boot` or `boot:` pointed at an app entry point or a gem's `lib/<name>.rb`.
|
|
31
|
+
That file loads before the sources.
|
|
32
|
+
Without a boot file, `require:` files load after the sources, so a source cannot use them while it loads.
|
|
33
|
+
Listing the entry file as an earlier source also loads it first.
|
|
34
|
+
|
|
30
35
|
## Agent loop
|
|
31
36
|
|
|
32
37
|
1. Edit code and tests on a branch. Fetch `origin/main` before using it below.
|
|
@@ -48,7 +53,10 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
48
53
|
Stop after a few unsuccessful attempts and ask a person to review the survivor.
|
|
49
54
|
Failed attempts do not prove that a mutant is equivalent. To
|
|
50
55
|
suppress it, `mutineer triage .mutineer/run.json --reason TEXT --id ID` prints an
|
|
51
|
-
`ignore:` entry for `.mutineer.yml
|
|
56
|
+
`ignore:` entry for `.mutineer.yml`, and one entry for each id in that
|
|
57
|
+
survivor's `copies`. Paste every entry. An entry ignores only its own id
|
|
58
|
+
(Mutineer 1.7 or later reads it; `copies` is on reports from the version that
|
|
59
|
+
lists them). See
|
|
52
60
|
[Suppressing equivalent mutants](https://github.com/davidteren/mutineer/blob/main/docs/results.md#suppressing-equivalent-mutants).
|
|
53
61
|
|
|
54
62
|
Keep `fail_fast` disabled (the default). If the active `.mutineer.yml` sets it to
|
|
@@ -59,7 +67,7 @@ and its score and exit code do not cover the full scope.
|
|
|
59
67
|
Zero survivors alone is not success: an empty or fully suppressed scope, or a run
|
|
60
68
|
with no usable verdicts, also has zero survivors. If the score is null, stop and
|
|
61
69
|
report “no score”. Check `no_coverage[]` for test gaps and `no_verdict[]` for
|
|
62
|
-
harness failures, and fix them before starting a new run. A positive threshold fails when nothing can be scored and something
|
|
70
|
+
harness failures, and fix them before starting a new run. Also check `summary.unpaired_mutants` (Mutineer 1.9.0 or later). Above 0, a source with no paired test was skipped. Its mutants are not in the score or in `no_verdict[]`, so they cannot fail `--threshold`. They count in the unrun-share warning on stderr, except under `--fail-fast`. `unpaired_sources[]` names each skipped source. Add a test for each entry whose `mutants` is above 0, then run again. `--test` replaces pairing, so if you pass it, list every test file, the new ones included. Under `--since`, an entry with `mutants` 0 has no mutant on a changed line. Also look for an entry whose `mutants` is `null`. That count failed, and `summary.unpaired_mutants` leaves it out. Report that run as incomplete. Do not accept it. A positive threshold fails when nothing can be scored and something
|
|
63
71
|
broke, or when more than one mutant has no verdict and they exceed 10% of those
|
|
64
72
|
attempted; it does not require zero errors. Set the threshold to your target.
|
|
65
73
|
|
|
@@ -71,7 +79,7 @@ An empty run (`summary.total` is 0) exits 0 in 1.x and prints a warning on stder
|
|
|
71
79
|
| Code | Meaning |
|
|
72
80
|
|------|---------|
|
|
73
81
|
| `0` | Score ≥ threshold (or no gate) **and** no baseline regression. |
|
|
74
|
-
| `1` | Score below `--threshold`, OR nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted, OR a `--baseline` regression, OR a runtime error, including a boot, source or `require:` file that raises while loading. |
|
|
82
|
+
| `1` | Score below `--threshold`, OR nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted, OR a `--baseline` regression, OR a runtime error, including a boot, source or `require:` file that raises while loading (except an in-process `--since` run with no mutant on a changed line, which checks that those files exist but does not load them). |
|
|
75
83
|
| `2` | Usage / invalid-flag error, including a mistyped flag, a bad path (such as a missing `--boot` or `require:` file), or an unreadable baseline. |
|
|
76
84
|
<!-- /contract:exit-codes -->
|
|
77
85
|
|