mutaruby 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +32 -0
- data/README.md +67 -12
- data/lib/mutaruby/cli.rb +3 -0
- data/lib/mutaruby/command.rb +18 -5
- data/lib/mutaruby/coverage_map.rb +51 -0
- data/lib/mutaruby/generator.rb +63 -9
- data/lib/mutaruby/platform.rb +9 -0
- data/lib/mutaruby/probe/mutaruby_probe.rb +169 -0
- data/lib/mutaruby/render.rb +28 -0
- data/lib/mutaruby/reporters.rb +6 -2
- data/lib/mutaruby/runner.rb +51 -21
- data/lib/mutaruby/sandbox.rb +6 -1
- data/lib/mutaruby/selection.rb +21 -3
- data/lib/mutaruby/version.rb +1 -1
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9b140915c577e60f56e81a1bec2d530a3791a82794bcb1e68de429f53fd5074d
|
|
4
|
+
data.tar.gz: 55cd843abdd670e427fc4a76b891cd75503623b160bc30dce514c90308966de4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1e6660338016f557eb61693a675c7fb672c8ed3f04bff6858412936ed470f4479278e9bc86da140bedfd7434141ba9699395d6bf9fa84831d18d14e4cd1c472f
|
|
7
|
+
data.tar.gz: 3b5aed66653a95656ef93303e76d981065cf68ed88116e3a4ffdedb097f706052189acb3b70457c1f06c617d49e135f3b3e530d9f7556e878b0dd5ea1a8c6e7f
|
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,37 @@
|
|
|
3
3
|
All notable changes to this project are documented here. The format follows
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses [Semantic Versioning](https://semver.org/).
|
|
5
5
|
|
|
6
|
+
## [0.3.0] - 2026-10-12
|
|
7
|
+
|
|
8
|
+
### Added
|
|
9
|
+
|
|
10
|
+
- `--coverage`: coverage-based test selection. The baseline run records, with Ruby's Coverage API and the test
|
|
11
|
+
framework's per-test hooks, which tests execute which lines; each mutant then runs only those tests (Minitest and
|
|
12
|
+
RSpec), and falls back to the whole command for load-time code, unreached lines and other test commands. The
|
|
13
|
+
summary and the JSON report show how many mutants ran a subset.
|
|
14
|
+
- `--subject`: select methods by their place in the code (`Foo#bar`, `Foo.bar`, `Foo#*`, `Foo.*`, `Foo` with its
|
|
15
|
+
nested namespaces, `Foo*`).
|
|
16
|
+
- Eight operators: `block-removal`, `method-swap`, `element-removal`, `regexp-never-match`, `yield-removal`,
|
|
17
|
+
`super-removal`, `when-removal`, `rescue-removal`.
|
|
18
|
+
- `--unparser`: render rebuilt nodes (`element-removal`) with the MIT-licensed `unparser` gem, an optional extra.
|
|
19
|
+
Text splicing, with its self-check, stays the default.
|
|
20
|
+
|
|
21
|
+
### Fixed
|
|
22
|
+
|
|
23
|
+
- An interrupted run starts no new mutant and waits for its workers to finish before it removes their sandboxes.
|
|
24
|
+
- A test run interrupted while mutaruby waits for it is stopped (with its process tree) before the interrupt goes
|
|
25
|
+
on, so its output file and sandbox are free to be removed.
|
|
26
|
+
- On Windows, a forcibly stopped run is given time to exit for good before its sandbox is removed (the directory
|
|
27
|
+
stays locked until then). A sandbox that still cannot be removed no longer hides the real error (Ctrl-C, a
|
|
28
|
+
failing baseline); removal is also retried when the directory is not yet empty.
|
|
29
|
+
- With `--coverage`, Ruby processes started by the tests themselves (a test that runs another suite) are neither
|
|
30
|
+
recorded nor filtered, and launcher processes (`bundle exec`, `rake`) do not overwrite the map.
|
|
31
|
+
|
|
32
|
+
### Changed
|
|
33
|
+
|
|
34
|
+
- The README's comparison credits mutant as the mature reference implementation and states that mutaruby is an
|
|
35
|
+
independent, clean-room implementation that shares no code with it.
|
|
36
|
+
|
|
6
37
|
## [0.2.0] - 2026-10-12
|
|
7
38
|
|
|
8
39
|
### Added
|
|
@@ -71,6 +102,7 @@ First release.
|
|
|
71
102
|
reason, ambiguity and staleness checks, `--strict-ignore` and `--suggest-ignore`.
|
|
72
103
|
- Human, JSON and JUnit XML reports, a mutation score and `--fail-under` for CI.
|
|
73
104
|
|
|
105
|
+
[0.3.0]: https://github.com/Atlas-Authorization/mutaruby/releases/tag/v0.3.0
|
|
74
106
|
[0.2.0]: https://github.com/Atlas-Authorization/mutaruby/releases/tag/v0.2.0
|
|
75
107
|
[0.1.1]: https://github.com/Atlas-Authorization/mutaruby/releases/tag/v0.1.1
|
|
76
108
|
[0.1.0]: https://github.com/Atlas-Authorization/mutaruby/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -5,10 +5,10 @@
|
|
|
5
5
|
|
|
6
6
|
Free, MIT-licensed mutation testing for Ruby.
|
|
7
7
|
|
|
8
|
-
mutaruby is mutation-tested with itself. In version 0.
|
|
9
|
-
|
|
10
|
-
[`.mutaruby-ignore.yml`](.mutaruby-ignore.yml) with a reason you can check. `bundle exec rake mutate`
|
|
11
|
-
run; it fails on any surviving mutant or stale ignore entry.
|
|
8
|
+
mutaruby is mutation-tested with itself. In version 0.3.0, all 4,968 mutants of its own code are accounted for:
|
|
9
|
+
4,962 are killed by its tests (573 of them by timeout), and the other 6 are true equivalents, each listed in
|
|
10
|
+
[`.mutaruby-ignore.yml`](.mutaruby-ignore.yml) with a reason you can check. `bundle exec rake mutate` (or the sharded
|
|
11
|
+
`Self-mutation` workflow) reproduces the run; it fails on any surviving mutant or stale ignore entry.
|
|
12
12
|
|
|
13
13
|
mutaruby makes small, deliberate changes to your code, one at a time, and runs your test command against each change.
|
|
14
14
|
If your tests still pass, the change (the *mutant*) **survived**, and that points to code your tests do not really check.
|
|
@@ -43,6 +43,7 @@ The survivor above tells you that no test checks a total of exactly 100.
|
|
|
43
43
|
- [Quickstart](#quickstart)
|
|
44
44
|
- [CLI reference](#cli-reference)
|
|
45
45
|
- [Mutation operators](#mutation-operators)
|
|
46
|
+
- [Coverage-based test selection](#coverage-based-test-selection)
|
|
46
47
|
- [Equivalent mutants: the ignore file](#equivalent-mutants-the-ignore-file)
|
|
47
48
|
- [Using it in CI](#using-it-in-ci)
|
|
48
49
|
- [Output formats](#output-formats)
|
|
@@ -138,6 +139,8 @@ Positional arguments are files or globs relative to the project root. The defaul
|
|
|
138
139
|
| `-t`, `--test COMMAND` | **Required.** Shell command that runs your tests: run with `/bin/sh -c` on Linux and macOS, with `cmd.exe` on Windows. Exit 0 means the mutant survived, non-zero means killed. |
|
|
139
140
|
| `-C`, `--root DIR` | Project root (default: the current directory). This is the directory copied into each sandbox, and the command runs there. |
|
|
140
141
|
| `-j`, `--jobs N` | Parallel workers (default: the number of CPUs). Each worker has its own copy of the project. |
|
|
142
|
+
| `--coverage` | Run, for each mutant, only the tests that execute its lines (Minitest and RSpec). See [Coverage-based test selection](#coverage-based-test-selection). |
|
|
143
|
+
| `--unparser` | Render rebuilt nodes (the `element-removal` operator) with the [`unparser`](https://github.com/mbj/unparser) gem instead of from the source text. Needs `gem install unparser`. |
|
|
141
144
|
| `--timeout SECONDS` | Per-mutant time limit. A mutant that exceeds it (an infinite loop, say) is killed together with every process it started and counted as **killed, by timeout**. The default is `baseline time x factor + 5` seconds. |
|
|
142
145
|
| `--timeout-factor F` | The factor for the default timeout (default: 3). |
|
|
143
146
|
| `--copy-exclude GLOB` | Do not copy matching paths into the sandboxes (repeatable). `.git` and `coverage` are never copied. Matches against the relative path and against the base name, e.g. `--copy-exclude node_modules --copy-exclude 'log/*'`. |
|
|
@@ -150,6 +153,7 @@ The command gets these environment variables: `MUTARUBY=1`, and while a mutant r
|
|
|
150
153
|
| Option | Meaning |
|
|
151
154
|
|---|---|
|
|
152
155
|
| `--since REF` | Only mutate files changed since a git ref: committed, staged and unstaged changes plus new untracked files (`git diff --name-only REF` and `git ls-files --others --exclude-standard`), intersected with the file arguments. |
|
|
156
|
+
| `--subject PATTERN` | Only mutants in these methods (repeatable): `Foo#bar` (instance method), `Foo.bar` (singleton method), `Foo#*` / `Foo.*` (all of one kind), `Foo` or `Foo::Bar` (every method of that namespace and of the namespaces nested in it), `Foo*` (a glob over namespaces). |
|
|
153
157
|
| `--method PATTERN` | Only mutants inside matching methods (repeatable). The pattern is a glob matched against the qualified name (`Pricing#discount` for an instance method, `Pricing.build` for a singleton method) or the bare name (`discount`). `Pricing#*` selects every instance method of `Pricing`. |
|
|
154
158
|
| `--exclude-method PATTERN` | Skip mutants inside matching methods (repeatable). |
|
|
155
159
|
| `--line SPEC` | Only mutants starting on these lines (repeatable). `SPEC` is `[FILE:]LINE` or `[FILE:]FROM-TO`, e.g. `lib/a.rb:10-20` or `42`. |
|
|
@@ -237,10 +241,56 @@ mutant's, are dropped before anything runs. `--list-operators` prints this list.
|
|
|
237
241
|
| `op-assign` | swaps compound assignments: `+=` and `-=`, `*=` and `/=`, `\|=` and `&=`, `<<=` and `>>=` | `i += 1` to `i -= 1` |
|
|
238
242
|
| `or-assign` | `\|\|=` becomes `=` | `@x \|\|= load` to `@x = load` |
|
|
239
243
|
| `and-assign` | `&&=` becomes `=` | `a &&= b` to `a = b` |
|
|
244
|
+
| `block-removal` | a call loses its block | `list.map { \|x\| f(x) }` to `list.map` |
|
|
245
|
+
| `method-swap` | a method becomes its counterpart: `first`/`last`, `min`/`max`, `min_by`/`max_by`, `any?`/`all?`, `select`/`reject` (and `filter`), `upcase`/`downcase`, `floor`/`ceil`, `succ`/`pred`, `keys`/`values`, `start_with?`/`end_with?`, `take`/`drop`, `take_while`/`drop_while`, `push`/`unshift`, `pop`/`shift` | `items.first` to `items.last` |
|
|
246
|
+
| `element-removal` | one element of a bracketed array or braced hash removed (literals with 2 to 6 elements) | `[a, b]` to `[b]` and `[a]` |
|
|
247
|
+
| `regexp-never-match` | a regexp literal becomes one that matches nothing | `/\A\d+\z/` to `/(?!)/` |
|
|
248
|
+
| `yield-removal` | `yield` becomes `nil` | `yield(item)` to `nil` |
|
|
249
|
+
| `super-removal` | `super` becomes `nil` | `super(name)` to `nil` |
|
|
250
|
+
| `when-removal` | one `when` branch of a `case` removed | |
|
|
251
|
+
| `rescue-removal` | one `rescue` clause removed | |
|
|
240
252
|
|
|
241
253
|
Strings inside interpolations, heredocs, `%w[]` arrays and the arguments of `require`-style declarations are not
|
|
242
254
|
mutated, and neither are symbols used as hash keys (`a: 1`).
|
|
243
255
|
|
|
256
|
+
## Coverage-based test selection
|
|
257
|
+
|
|
258
|
+
Most mutants are reached by only a few tests, yet by default every mutant runs the whole test command. With
|
|
259
|
+
`--coverage`, mutaruby first finds out which tests execute which lines, then runs each mutant against just those
|
|
260
|
+
tests:
|
|
261
|
+
|
|
262
|
+
```sh
|
|
263
|
+
mutaruby -t "bundle exec rspec" --coverage
|
|
264
|
+
mutaruby -t "bundle exec rake test" --coverage
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
How it works:
|
|
268
|
+
|
|
269
|
+
1. The baseline run (which has to pass anyway) is made with a small probe loaded into the test process
|
|
270
|
+
(`RUBYLIB` plus `RUBYOPT=-rmutaruby_probe`). The probe uses Ruby's `Coverage` API and the test framework's
|
|
271
|
+
per-test hooks to record, for every test, the lines of the project it executes. Tests are identified as
|
|
272
|
+
`Class#test_name` for Minitest and by the example id (`./spec/a_spec.rb[1:2]`) for RSpec.
|
|
273
|
+
2. For each mutant, the tests that execute any of its lines are written to a file, and the run loads the probe in
|
|
274
|
+
selection mode, which tells Minitest or RSpec to run only those tests. The test command itself is unchanged.
|
|
275
|
+
3. A mutant runs the **whole** command instead when one of its lines ran while the code was being loaded (a class
|
|
276
|
+
body, a constant, a method definition line), when no test reaches its lines, or when the baseline recorded no
|
|
277
|
+
Minitest or RSpec tests at all (mutaruby then prints a warning).
|
|
278
|
+
|
|
279
|
+
What it saves depends on the suite. Measured on GitHub Actions (`script/benchmark-coverage.rb`, 4 workers), with the
|
|
280
|
+
same verdicts both ways:
|
|
281
|
+
|
|
282
|
+
| Project | Mutants | Whole test command | `--coverage` |
|
|
283
|
+
|---|---|---|---|
|
|
284
|
+
| mutaruby's own unit suite (a 1/25 sample of its mutants) | 199 | 3,788 s | 2,276 s (1.7x faster) |
|
|
285
|
+
| `test/fixtures/calc` (5 tests, Minitest or RSpec) | 43 | 13 s | 13 s |
|
|
286
|
+
|
|
287
|
+
A small suite gains nothing: starting Ruby costs more than its tests. The larger and slower the suite, and the fewer
|
|
288
|
+
tests reach each line, the bigger the win.
|
|
289
|
+
|
|
290
|
+
The summary shows how many mutants ran a subset, and the JSON report has, per mutant, the number of tests it ran
|
|
291
|
+
(`tests`, or `null` for the whole command). Selection assumes that tests are independent of each other: a test that
|
|
292
|
+
only passes after another one has run can make a mutant look killed.
|
|
293
|
+
|
|
244
294
|
## Equivalent mutants: the ignore file
|
|
245
295
|
|
|
246
296
|
`.mutaruby-ignore.yml` (or the file given to `--ignore-file`) is a YAML list. Each entry names one **mutation site** and
|
|
@@ -342,17 +392,21 @@ survivors as failed tests.
|
|
|
342
392
|
|
|
343
393
|
## How it compares with mutant
|
|
344
394
|
|
|
345
|
-
[mutant](https://github.com/mbj/mutant) is the
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
source projects).
|
|
395
|
+
[mutant](https://github.com/mbj/mutant) is the mature reference implementation of mutation testing for Ruby, and the
|
|
396
|
+
tool mutaruby is most often compared with. It is more capable: many more operators, semantic knowledge of Ruby's core
|
|
397
|
+
library, precise subject and test selection, and years of use on large code bases. It is distributed under its own
|
|
398
|
+
commercial licence (free for open source projects).
|
|
399
|
+
|
|
400
|
+
**mutaruby shares no code with mutant.** It is an independent, clean-room implementation, written from general,
|
|
401
|
+
publicly known ideas about mutation testing (operators, coverage-based test selection, equivalent-mutant handling),
|
|
402
|
+
and it is MIT-licensed. Its only runtime dependency is `parser`; `unparser`, an MIT-licensed gem, is an optional
|
|
403
|
+
extra for `--unparser`.
|
|
349
404
|
|
|
350
405
|
mutaruby aims to be **MIT-licensed and simple**:
|
|
351
406
|
|
|
352
407
|
- It works with **any test command** (rspec, minitest, rake, a Makefile target), because the oracle is just an exit
|
|
353
|
-
status.
|
|
354
|
-
|
|
355
|
-
- It is small enough to read in an afternoon, and it has no dependency besides `parser`.
|
|
408
|
+
status. With `--coverage` it also runs only the tests that reach each mutant, for Minitest and RSpec suites.
|
|
409
|
+
- It is small enough to read in an afternoon.
|
|
356
410
|
- It runs each mutant in a **separate process** in a **copy of the project**, so it cannot leave your working tree
|
|
357
411
|
modified.
|
|
358
412
|
|
|
@@ -419,7 +473,8 @@ mutaruby works the same way on Windows, with these differences:
|
|
|
419
473
|
|
|
420
474
|
- On Windows, processes that a test run leaves running after it exits normally are not stopped (see
|
|
421
475
|
[Windows notes](#windows-notes)).
|
|
422
|
-
-
|
|
476
|
+
- Per-test selection (`--coverage`) understands Minitest and RSpec; any other command runs in full for every mutant.
|
|
477
|
+
Selection assumes the tests do not depend on each other's order or side effects.
|
|
423
478
|
- The parser grammar is chosen to match the running Ruby, up to the newest grammar the installed `parser` gem knows.
|
|
424
479
|
Files that `parser` cannot read (for example syntax newer than it supports) are skipped with a warning.
|
|
425
480
|
- Mutants are syntactic. Some are equivalent by nature; that is what the ignore file is for.
|
data/lib/mutaruby/cli.rb
CHANGED
|
@@ -71,6 +71,8 @@ module Mutaruby
|
|
|
71
71
|
op.on('-j', '--jobs N', Integer, "Parallel workers (default: #{Etc.nprocessors})") { |v| o.jobs = v }
|
|
72
72
|
op.on('--timeout SECONDS', Float, 'Per-mutant timeout (default: baseline time x factor + 5s)') { |v| o.timeout = v }
|
|
73
73
|
op.on('--timeout-factor F', Float, 'Multiplier on the baseline time for the default timeout (default: 3)') { |v| o.timeout_factor = v }
|
|
74
|
+
op.on('--coverage', 'Run only the tests that cover each mutant (Minitest and RSpec; mapped in the baseline run)') { o.coverage = true }
|
|
75
|
+
op.on('--unparser', 'Render rebuilt nodes with the unparser gem (element-removal; needs gem install unparser)') { o.renderer = :unparser }
|
|
74
76
|
op.on('--copy-exclude GLOB', 'Do not copy matching paths into the sandboxes (repeatable; .git and coverage are always skipped)') { |v| o.copy_excludes += [v] }
|
|
75
77
|
op.separator ''
|
|
76
78
|
op.separator 'Selection:'
|
|
@@ -79,6 +81,7 @@ module Mutaruby
|
|
|
79
81
|
op.on('--exclude-method PATTERN', 'Skip mutants inside matching methods (repeatable)') { |v| o.exclude_methods += [v] }
|
|
80
82
|
op.on('--line SPEC', 'Only mutants on these lines: [FILE:]LINE or [FILE:]FROM-TO (repeatable)') { |v| o.lines += [Selection.parse_line_spec(v)] }
|
|
81
83
|
op.on('--exclude-line SPEC', 'Skip mutants on these lines (repeatable)') { |v| o.exclude_lines += [Selection.parse_line_spec(v)] }
|
|
84
|
+
op.on('--subject PATTERN', 'Only mutants in these methods: "Foo#bar", "Foo.bar", "Foo#*", "Foo" (with nested namespaces), "Foo*" (repeatable)') { |v| o.subjects += [v] }
|
|
82
85
|
op.on('--operators LIST', Array, 'Only these operators (comma-separated; globs allowed, e.g. condition-*)') { |v| @include_ops += v }
|
|
83
86
|
op.on('--exclude-operators LIST', Array, 'Skip these operators (comma-separated; globs allowed)') { |v| @exclude_ops += v }
|
|
84
87
|
op.on('--shard I/N', 'Run only shard I of N (1-based) of the selected mutants') { |v| o.shard = Selection.parse_shard(v) }
|
data/lib/mutaruby/command.rb
CHANGED
|
@@ -52,11 +52,14 @@ module Mutaruby
|
|
|
52
52
|
chdir: dir, in: File::NULL, out: out, err: out, **@platform.spawn_options)
|
|
53
53
|
waiter = Process.detach(pid)
|
|
54
54
|
@lock.synchronize { @live[pid] = waiter }
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
55
|
+
begin
|
|
56
|
+
timed_out = wait(pid, waiter, timeout)
|
|
57
|
+
# A timed-out run, and (where the platform allows) anything a finished run left behind, must not outlive it.
|
|
58
|
+
kill_group(pid, waiter) if timed_out || @platform.stop_leftovers?
|
|
59
|
+
st = waiter.value
|
|
60
|
+
ensure
|
|
61
|
+
@lock.synchronize { @live.delete(pid) }
|
|
62
|
+
end
|
|
60
63
|
status = if timed_out then :timeout
|
|
61
64
|
elsif st.success? then :passed
|
|
62
65
|
else :failed
|
|
@@ -64,6 +67,15 @@ module Mutaruby
|
|
|
64
67
|
Result.new(status: status, exitstatus: st.exitstatus, seconds: monotonic - started)
|
|
65
68
|
end
|
|
66
69
|
|
|
70
|
+
# True when the run timed out. Interrupted while waiting (Ctrl-C), the run is stopped before the interrupt goes on:
|
|
71
|
+
# its output file, its sandbox and its process group must not outlive it.
|
|
72
|
+
def wait(pid, waiter, timeout)
|
|
73
|
+
waiter.join(timeout).nil?
|
|
74
|
+
rescue Interrupt
|
|
75
|
+
kill_group(pid, waiter)
|
|
76
|
+
raise
|
|
77
|
+
end
|
|
78
|
+
|
|
67
79
|
# Bundler exports the ORIGINAL Gemfile path to child processes. Left as is, `bundle exec` in the sandbox would
|
|
68
80
|
# resolve the Gemfile (and a `gemspec` line's lib/ directory) from the user's checkout and test the unmutated code.
|
|
69
81
|
# The per-mutant variables are always set or cleared explicitly, so a mutaruby run nested inside another one's
|
|
@@ -86,6 +98,7 @@ module Mutaruby
|
|
|
86
98
|
deadline = monotonic + @kill_grace
|
|
87
99
|
sleep POLL while monotonic < deadline && @platform.alive?(pid, waiter)
|
|
88
100
|
@platform.stop_tree(pid, graceful: false)
|
|
101
|
+
@platform.settle(waiter, @kill_grace)
|
|
89
102
|
end
|
|
90
103
|
|
|
91
104
|
def tail(bytes)
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'json'
|
|
4
|
+
|
|
5
|
+
module Mutaruby
|
|
6
|
+
# Which tests execute which lines of the project, recorded in the baseline run by the probe
|
|
7
|
+
# (lib/mutaruby/probe/mutaruby_probe.rb) through Ruby's Coverage API. Used to run, for each mutant, only the tests
|
|
8
|
+
# that reach its lines.
|
|
9
|
+
class CoverageMap
|
|
10
|
+
PROBE_DIR = File.expand_path('probe', __dir__).freeze
|
|
11
|
+
|
|
12
|
+
# Environment that loads the probe into the test process, plus `extra` (the probe's own variables). RUBYLIB is a
|
|
13
|
+
# path list (spaces are fine there); RUBYOPT names only the feature, so no path has to survive its splitting.
|
|
14
|
+
def self.probe_env(extra)
|
|
15
|
+
lib = [PROBE_DIR, ENV.fetch('RUBYLIB', '')].reject(&:empty?).join(File::PATH_SEPARATOR)
|
|
16
|
+
opt = [ENV.fetch('RUBYOPT', ''), '-rmutaruby_probe'].reject(&:empty?).join(' ')
|
|
17
|
+
{ 'RUBYLIB' => lib, 'RUBYOPT' => opt }.merge(extra)
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def self.load(path)
|
|
21
|
+
new(JSON.parse(File.read(path, encoding: 'UTF-8')))
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
attr_reader :framework
|
|
25
|
+
|
|
26
|
+
# data: { "framework" => "minitest" | "rspec" | nil, "tests" => { id => { file => [line, ...] } },
|
|
27
|
+
# "load" => { file => [line, ...] } }
|
|
28
|
+
def initialize(data)
|
|
29
|
+
@framework = data['framework']
|
|
30
|
+
@tests = Hash.new { |h, k| h[k] = [] }
|
|
31
|
+
data['tests'].each do |id, files|
|
|
32
|
+
files.each { |file, lines| lines.each { |l| @tests[[file, l]] << id } }
|
|
33
|
+
end
|
|
34
|
+
@load = data['load'].flat_map { |file, lines| lines.map { |l| [file, l] } }
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# True when the run recorded tests of a supported framework.
|
|
38
|
+
def usable?
|
|
39
|
+
!@framework.nil? && @tests.any?
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# The ids of the tests that execute any of `lines` of `file`, sorted; nil when the whole suite has to run: a line
|
|
43
|
+
# that ran while the code was loaded (a class body, a method definition), or a line no test reaches.
|
|
44
|
+
def tests_for(file, lines)
|
|
45
|
+
return nil if lines.any? { |l| @load.include?([file, l]) }
|
|
46
|
+
|
|
47
|
+
ids = lines.flat_map { |l| @tests.fetch([file, l], []) }.uniq.sort
|
|
48
|
+
ids.empty? ? nil : ids
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
data/lib/mutaruby/generator.rb
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
require_relative 'parsing'
|
|
4
4
|
require_relative 'mutation'
|
|
5
5
|
require_relative 'splicer'
|
|
6
|
+
require_relative 'render'
|
|
6
7
|
|
|
7
8
|
module Mutaruby
|
|
8
9
|
# Walks a file's AST (the `parser` gem) and produces every mutant of it. Each mutant is ONE splice of the original
|
|
@@ -46,7 +47,24 @@ module Mutaruby
|
|
|
46
47
|
'raise-removal' => 'raise X -> nil (also fail)',
|
|
47
48
|
'op-assign' => 'swap compound assignment (+= <-> -=, *= <-> /=, |= <-> &=, <<= <-> >>=)',
|
|
48
49
|
'or-assign' => 'a ||= b -> a = b',
|
|
49
|
-
'and-assign' => 'a &&= b -> a = b'
|
|
50
|
+
'and-assign' => 'a &&= b -> a = b',
|
|
51
|
+
'block-removal' => 'list.map { |x| f(x) } -> list.map (the call without its block)',
|
|
52
|
+
'method-swap' => 'swap a method for its counterpart (first <-> last, min <-> max, select <-> reject, ...)',
|
|
53
|
+
'element-removal' => '[a, b] -> [b], [a]; { k: 1, j: 2 } -> { j: 2 }, { k: 1 } (one element removed at a time)',
|
|
54
|
+
'regexp-never-match' => '/pattern/ -> /(?!)/ (a regexp that matches nothing)',
|
|
55
|
+
'yield-removal' => 'yield x -> nil',
|
|
56
|
+
'super-removal' => 'super / super(x) -> nil',
|
|
57
|
+
'when-removal' => 'one `when` branch of a case removed',
|
|
58
|
+
'rescue-removal' => 'one `rescue` clause removed'
|
|
59
|
+
}.freeze
|
|
60
|
+
|
|
61
|
+
# method-swap pairs: each method and the one it is replaced with.
|
|
62
|
+
METHOD_SWAPS = {
|
|
63
|
+
first: :last, last: :first, min: :max, max: :min, min_by: :max_by, max_by: :min_by, any?: :all?, all?: :any?,
|
|
64
|
+
select: :reject, reject: :select, filter: :reject, upcase: :downcase, downcase: :upcase, floor: :ceil,
|
|
65
|
+
ceil: :floor, succ: :pred, pred: :succ, keys: :values, values: :keys, start_with?: :end_with?,
|
|
66
|
+
end_with?: :start_with?, take: :drop, drop: :take, take_while: :drop_while, drop_while: :take_while,
|
|
67
|
+
push: :unshift, unshift: :push, pop: :shift, shift: :pop
|
|
50
68
|
}.freeze
|
|
51
69
|
|
|
52
70
|
SWAPS = {
|
|
@@ -75,8 +93,11 @@ module Mutaruby
|
|
|
75
93
|
|
|
76
94
|
# path: file path for messages; rel: path relative to the project root (stored on the mutants).
|
|
77
95
|
# operators: nil (all) or a list of operator ids.
|
|
78
|
-
|
|
96
|
+
# renderer: :text (default) builds replacements from the source text; any other value (:unparser) renders the
|
|
97
|
+
# operators that rebuild a node (element-removal) with the `unparser` gem instead, which must be installed.
|
|
98
|
+
def initialize(source, path:, rel: path, operators: nil, renderer: :text)
|
|
79
99
|
@src = source
|
|
100
|
+
@renderer = renderer
|
|
80
101
|
@path = path
|
|
81
102
|
@rel = rel
|
|
82
103
|
@operators = operators
|
|
@@ -226,10 +247,8 @@ module Mutaruby
|
|
|
226
247
|
when :true then add(n.loc.expression, 'false', 'literal-boolean')
|
|
227
248
|
when :false then add(n.loc.expression, 'true', 'literal-boolean')
|
|
228
249
|
when :nil then add(n.loc.expression, 'true', 'literal-nil')
|
|
229
|
-
when :array
|
|
230
|
-
|
|
231
|
-
when :hash
|
|
232
|
-
add(n.loc.expression, '{}', 'literal-hash') if n.children.any? && n.loc.begin&.source == '{'
|
|
250
|
+
when :array then visit_collection(n, '[', '[]', 'literal-array')
|
|
251
|
+
when :hash then visit_collection(n, '{', '{}', 'literal-hash')
|
|
233
252
|
when :irange, :erange
|
|
234
253
|
# an endless range (a..) means the same with either operator
|
|
235
254
|
add(n.loc.operator, n.type == :irange ? '...' : '..', 'range-swap') unless n.children[1].nil?
|
|
@@ -242,6 +261,39 @@ module Mutaruby
|
|
|
242
261
|
when :and_asgn then add(n.loc.operator, '=', 'and-assign')
|
|
243
262
|
when :begin, :kwbegin then visit_body(n)
|
|
244
263
|
when :def, :defs then visit_def(n)
|
|
264
|
+
when :block, :numblock then add(n.loc.expression, src_of(n.children.first), 'block-removal')
|
|
265
|
+
when :regexp then add(n.loc.expression, '/(?!)/', 'regexp-never-match')
|
|
266
|
+
when :yield then add(n.loc.expression, 'nil', 'yield-removal')
|
|
267
|
+
when :super, :zsuper then add(n.loc.expression, 'nil', 'super-removal')
|
|
268
|
+
when :case then remove_clauses(n.children[1...-1], 'when-removal')
|
|
269
|
+
when :rescue then remove_clauses(n.children[1...-1], 'rescue-removal')
|
|
270
|
+
end
|
|
271
|
+
end
|
|
272
|
+
|
|
273
|
+
# A bracketed array / braced hash: emptied, and each element removed in turn.
|
|
274
|
+
def visit_collection(n, open, empty, op)
|
|
275
|
+
return unless n.loc.begin&.source == open
|
|
276
|
+
|
|
277
|
+
add(n.loc.expression, empty, op) if n.children.any?
|
|
278
|
+
remove_elements(n) if n.children.size.between?(2, MAX_ARGS)
|
|
279
|
+
end
|
|
280
|
+
|
|
281
|
+
# Each clause (a `when`, a `rescue`) deleted in turn; the validity check drops what would not parse.
|
|
282
|
+
def remove_clauses(clauses, op)
|
|
283
|
+
clauses.each { |c| add(c.loc.expression, '', op) }
|
|
284
|
+
end
|
|
285
|
+
|
|
286
|
+
# [a, b, c] -> [b, c], [a, c], [a, b] (and the same for the pairs of a braced hash).
|
|
287
|
+
def remove_elements(n)
|
|
288
|
+
kids = n.children
|
|
289
|
+
kids.each_index do |i|
|
|
290
|
+
rest = kids.reject.with_index { |_, j| j == i }
|
|
291
|
+
if @renderer == :text
|
|
292
|
+
add(kids.first.loc.expression.join(kids.last.loc.expression), rest.map { |k| src_of(k) }.join(', '),
|
|
293
|
+
'element-removal')
|
|
294
|
+
else
|
|
295
|
+
add(n.loc.expression, Render.unparse(src_of(n), i), 'element-removal')
|
|
296
|
+
end
|
|
245
297
|
end
|
|
246
298
|
end
|
|
247
299
|
|
|
@@ -250,6 +302,7 @@ module Mutaruby
|
|
|
250
302
|
SWAPS.each do |op, table|
|
|
251
303
|
add(n.loc.selector, table[meth].to_s, op) if table.key?(meth) && args.size == 1
|
|
252
304
|
end
|
|
305
|
+
add(n.loc.selector, METHOD_SWAPS[meth].to_s, 'method-swap') if recv && METHOD_SWAPS.key?(meth)
|
|
253
306
|
add(n.loc.expression, src_of(recv), 'negation-removal') if meth == :!
|
|
254
307
|
add(n.loc.expression, src_of(recv), 'unary-minus-removal') if meth == :-@
|
|
255
308
|
add(n.loc.dot, '.', 'safe-navigation') if n.type == :csend
|
|
@@ -298,12 +351,12 @@ module Mutaruby
|
|
|
298
351
|
end
|
|
299
352
|
|
|
300
353
|
def visit_int(n)
|
|
301
|
-
v = n.children
|
|
354
|
+
v, = n.children
|
|
302
355
|
[v + 1, v - 1, 0, 1].each { |x| add(n.loc.expression, x.to_s, 'literal-int') }
|
|
303
356
|
end
|
|
304
357
|
|
|
305
358
|
def visit_float(n)
|
|
306
|
-
v = n.children
|
|
359
|
+
v, = n.children
|
|
307
360
|
add(n.loc.expression, '0.0', 'literal-float') unless v.zero?
|
|
308
361
|
add(n.loc.expression, (v + 1.0).to_s, 'literal-float') if v.finite?
|
|
309
362
|
end
|
|
@@ -317,7 +370,8 @@ module Mutaruby
|
|
|
317
370
|
close = n.loc.end if open
|
|
318
371
|
return unless close
|
|
319
372
|
|
|
320
|
-
|
|
373
|
+
text, = n.children
|
|
374
|
+
fill = text.empty? ? 'mutaruby' : ''
|
|
321
375
|
add(n.loc.expression, "#{open.source}#{fill}#{close.source}", 'literal-string')
|
|
322
376
|
end
|
|
323
377
|
|
data/lib/mutaruby/platform.rb
CHANGED
|
@@ -20,6 +20,9 @@ module Mutaruby
|
|
|
20
20
|
# After a run ends, whatever is left of its process group is stopped too.
|
|
21
21
|
def stop_leftovers? = true
|
|
22
22
|
|
|
23
|
+
# Nothing to wait for: the group is gone once signalled with SIGKILL.
|
|
24
|
+
def settle(_waiter, _seconds); end
|
|
25
|
+
|
|
23
26
|
# graceful: SIGTERM, otherwise SIGKILL, to the process group led by pid.
|
|
24
27
|
def stop_tree(pid, graceful:)
|
|
25
28
|
Process.kill(graceful ? 'TERM' : 'KILL', -pid)
|
|
@@ -57,6 +60,12 @@ module Mutaruby
|
|
|
57
60
|
# already have reused its id for an unrelated process.
|
|
58
61
|
def stop_leftovers? = false
|
|
59
62
|
|
|
63
|
+
# After taskkill /F, give the process up to `seconds` to exit for good: until then Windows keeps its files and
|
|
64
|
+
# working directory (the sandbox) locked.
|
|
65
|
+
def settle(waiter, seconds)
|
|
66
|
+
waiter.join(seconds)
|
|
67
|
+
end
|
|
68
|
+
|
|
60
69
|
def stop_tree(pid, graceful:)
|
|
61
70
|
@system.call('taskkill', '/T', *(graceful ? [] : ['/F']), '/PID', pid.to_s)
|
|
62
71
|
end
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Loaded into the user's test process by mutaruby (RUBYLIB=<this directory>, RUBYOPT=-rmutaruby_probe) when
|
|
4
|
+
# coverage-based test selection is on. It depends on nothing from mutaruby itself and does nothing unless one of its
|
|
5
|
+
# environment variables is set:
|
|
6
|
+
#
|
|
7
|
+
# MUTARUBY_COVERAGE_OUT=FILE record, for every test, the lines of the project it executes (Ruby's Coverage API),
|
|
8
|
+
# and write them to FILE as JSON when the process exits
|
|
9
|
+
# MUTARUBY_COVERAGE_ROOT=DIR the project directory; only files below it are recorded
|
|
10
|
+
# MUTARUBY_SELECT=FILE run only the tests whose ids FILE lists, one per line
|
|
11
|
+
#
|
|
12
|
+
# The test command may start several Ruby processes (bundle exec -> rake -> ruby running the tests). Each loads the
|
|
13
|
+
# probe, but only the process that loads a test framework records or selects: launchers write nothing. That process
|
|
14
|
+
# then removes the variables, so Ruby processes started by the tests themselves run normally.
|
|
15
|
+
#
|
|
16
|
+
# Test ids: "Class#test_method" for Minitest, the example id ("./spec/a_spec.rb[1:2]") for RSpec.
|
|
17
|
+
module MutarubyProbe
|
|
18
|
+
class << self
|
|
19
|
+
def setup
|
|
20
|
+
@out = ENV.fetch('MUTARUBY_COVERAGE_OUT', '')
|
|
21
|
+
@select = ENV.fetch('MUTARUBY_SELECT', '')
|
|
22
|
+
@installed = []
|
|
23
|
+
return if @out.empty? && @select.empty?
|
|
24
|
+
|
|
25
|
+
@select.empty? ? start_recording : load_selection
|
|
26
|
+
Object.prepend(RequireHook)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def recording? = !@out.empty?
|
|
30
|
+
|
|
31
|
+
# --- recording -------------------------------------------------------------------------------------------------
|
|
32
|
+
|
|
33
|
+
def start_recording
|
|
34
|
+
require 'coverage'
|
|
35
|
+
require 'json'
|
|
36
|
+
root = ENV.fetch('MUTARUBY_COVERAGE_ROOT')
|
|
37
|
+
@root = "#{File.realpath(root)}/"
|
|
38
|
+
@tests = Hash.new { |h, id| h[id] = {} }
|
|
39
|
+
@load = {}
|
|
40
|
+
Coverage.start(lines: true)
|
|
41
|
+
at_exit { write if @installed.any? }
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
# Lines run before a test starts (loading code, defining methods) count as load-time lines.
|
|
45
|
+
def test_started(id)
|
|
46
|
+
collect(@load)
|
|
47
|
+
@current = id
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def test_finished
|
|
51
|
+
collect(@tests[@current])
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def collect(into)
|
|
55
|
+
Coverage.result(stop: false, clear: true).each do |path, data|
|
|
56
|
+
rel = relative(path)
|
|
57
|
+
next unless rel
|
|
58
|
+
|
|
59
|
+
lines = data[:lines]
|
|
60
|
+
hit = lines.each_index.select { |i| lines[i].to_i.positive? }.map { |i| i + 1 }
|
|
61
|
+
into[rel] = (into.fetch(rel, []) | hit).sort
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
# Path relative to the project, or nil for files outside it. Coverage reports the path a file was loaded from; it
|
|
66
|
+
# is compared by its real path.
|
|
67
|
+
def relative(path)
|
|
68
|
+
real = real_path(path)
|
|
69
|
+
real.delete_prefix(@root) if real.start_with?(@root)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# Ruby records required files by their real path already; a file deleted since it was loaded keeps that path.
|
|
73
|
+
def real_path(path)
|
|
74
|
+
File.realpath(path)
|
|
75
|
+
rescue SystemCallError
|
|
76
|
+
path
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# Lines run after the last test (at_exit and after_run hooks) count as load-time lines too.
|
|
80
|
+
def write
|
|
81
|
+
collect(@load)
|
|
82
|
+
File.write(@out, JSON.generate(framework: @installed.first, tests: @tests, load: @load))
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
# --- selection -------------------------------------------------------------------------------------------------
|
|
86
|
+
|
|
87
|
+
def load_selection
|
|
88
|
+
@selected = File.readlines(@select, chomp: true)
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def selected?(id) = @selected.include?(id)
|
|
92
|
+
|
|
93
|
+
# --- frameworks ------------------------------------------------------------------------------------------------
|
|
94
|
+
|
|
95
|
+
def install_hooks
|
|
96
|
+
install_minitest if !@installed.include?('minitest') && defined?(::Minitest::Test)
|
|
97
|
+
install_rspec if !@installed.include?('rspec') && defined?(::RSpec) && ::RSpec.respond_to?(:configure)
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
# The first framework found: this is the test process. Its children must not record or select.
|
|
101
|
+
def claim
|
|
102
|
+
%w[MUTARUBY_COVERAGE_OUT MUTARUBY_COVERAGE_ROOT MUTARUBY_SELECT].each { |k| ENV.delete(k) }
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def install_minitest
|
|
106
|
+
claim
|
|
107
|
+
@installed << 'minitest'
|
|
108
|
+
if recording?
|
|
109
|
+
::Minitest::Test.prepend(MinitestRecording)
|
|
110
|
+
else
|
|
111
|
+
::Minitest::Test.singleton_class.prepend(MinitestSelection)
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def install_rspec
|
|
116
|
+
claim
|
|
117
|
+
@installed << 'rspec'
|
|
118
|
+
if recording?
|
|
119
|
+
::RSpec.configure do |c|
|
|
120
|
+
c.around do |ex| # (an example hook)
|
|
121
|
+
MutarubyProbe.test_started(ex.id)
|
|
122
|
+
ex.run
|
|
123
|
+
ensure
|
|
124
|
+
MutarubyProbe.test_finished
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
else
|
|
128
|
+
::RSpec.configure do |c|
|
|
129
|
+
c.define_derived_metadata do |meta|
|
|
130
|
+
meta[:mutaruby_selected] = true if MutarubyProbe.selected?("#{meta[:rerun_file_path]}[#{meta[:scoped_id]}]")
|
|
131
|
+
end
|
|
132
|
+
c.filter_run_including(mutaruby_selected: true)
|
|
133
|
+
end
|
|
134
|
+
end
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
# Re-checks for test frameworks after every require: they are loaded after this file.
|
|
139
|
+
module RequireHook
|
|
140
|
+
private
|
|
141
|
+
|
|
142
|
+
def require(name)
|
|
143
|
+
super
|
|
144
|
+
ensure
|
|
145
|
+
MutarubyProbe.install_hooks
|
|
146
|
+
end
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
module MinitestRecording
|
|
150
|
+
def before_setup
|
|
151
|
+
MutarubyProbe.test_started("#{self.class}##{name}")
|
|
152
|
+
super
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
def after_teardown
|
|
156
|
+
super
|
|
157
|
+
ensure
|
|
158
|
+
MutarubyProbe.test_finished
|
|
159
|
+
end
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
module MinitestSelection
|
|
163
|
+
def runnable_methods
|
|
164
|
+
super.select { |m| MutarubyProbe.selected?("#{name}##{m}") }
|
|
165
|
+
end
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
MutarubyProbe.setup
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'error'
|
|
4
|
+
|
|
5
|
+
module Mutaruby
|
|
6
|
+
# Renders rebuilt nodes back to Ruby source with the MIT-licensed `unparser` gem (optional: `gem install unparser`).
|
|
7
|
+
# Text splicing stays the default; this path is used, with --unparser, for operators that rebuild a node. The
|
|
8
|
+
# rendered text is still spliced over the node's own range, so the splice self-check applies as usual.
|
|
9
|
+
module Render
|
|
10
|
+
class Unavailable < Error; end
|
|
11
|
+
|
|
12
|
+
module_function
|
|
13
|
+
|
|
14
|
+
def load!
|
|
15
|
+
require 'unparser'
|
|
16
|
+
rescue LoadError
|
|
17
|
+
raise Unavailable, '--unparser needs the unparser gem (gem install unparser)'
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# `source` (an array or hash literal) without its element number `index`, rendered by unparser. The node is
|
|
21
|
+
# parsed again by unparser itself, so its AST is in the form unparser expects.
|
|
22
|
+
def unparse(source, index)
|
|
23
|
+
load!
|
|
24
|
+
node = Unparser.parse(source)
|
|
25
|
+
Unparser.unparse(node.updated(nil, node.children.reject.with_index { |_, j| j == index }))
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
data/lib/mutaruby/reporters.rb
CHANGED
|
@@ -43,6 +43,10 @@ module Mutaruby
|
|
|
43
43
|
io.puts " survived: #{report.count(:survived)}"
|
|
44
44
|
io.puts " equivalent: #{report.count(:equivalent)} (ignore file)"
|
|
45
45
|
io.puts " score: #{score_text(report)} = killed / (mutants - equivalent)"
|
|
46
|
+
if report.coverage
|
|
47
|
+
subset = report.outcomes.count(&:tests)
|
|
48
|
+
io.puts " coverage: #{report.coverage}: #{subset} of #{report.outcomes.size} mutants ran only the tests covering them"
|
|
49
|
+
end
|
|
46
50
|
io.puts format(' time: %.1fs (baseline %.2fs, timeout %ss, jobs %d)', report.seconds,
|
|
47
51
|
report.baseline_seconds, report.timeout, report.jobs)
|
|
48
52
|
end
|
|
@@ -50,12 +54,12 @@ module Mutaruby
|
|
|
50
54
|
def json_data(report)
|
|
51
55
|
{
|
|
52
56
|
tool: 'mutaruby', version: VERSION, command: report.command,
|
|
53
|
-
score: report.score&.round(4), complete: report.complete,
|
|
57
|
+
score: report.score&.round(4), complete: report.complete, coverage: report.coverage,
|
|
54
58
|
counts: { mutants: report.outcomes.size, killed: report.killed, timeout: report.count(:timeout),
|
|
55
59
|
survived: report.count(:survived), equivalent: report.count(:equivalent) },
|
|
56
60
|
files: report.files, baseline_seconds: secs(report.baseline_seconds), timeout: report.timeout,
|
|
57
61
|
mutants: report.outcomes.map do |o|
|
|
58
|
-
o.mutation.to_h.merge(status: o.status, seconds: secs(o.seconds), ignore_reason: o.ignore&.reason)
|
|
62
|
+
o.mutation.to_h.merge(status: o.status, seconds: secs(o.seconds), ignore_reason: o.ignore&.reason, tests: o.tests)
|
|
59
63
|
end,
|
|
60
64
|
stale_ignores: report.stale.map { |e| e.to_h.compact.except(:index) },
|
|
61
65
|
warnings: report.warnings
|
data/lib/mutaruby/runner.rb
CHANGED
|
@@ -4,19 +4,21 @@ require_relative 'ignore_file'
|
|
|
4
4
|
require_relative 'sandbox'
|
|
5
5
|
require_relative 'command'
|
|
6
6
|
require_relative 'selection'
|
|
7
|
+
require_relative 'coverage_map'
|
|
7
8
|
|
|
8
9
|
module Mutaruby
|
|
9
10
|
class BaselineFailed < Error; end
|
|
10
11
|
|
|
11
12
|
# Generates the mutants, checks the baseline, runs every mutant and collects the verdicts.
|
|
12
13
|
class Runner
|
|
13
|
-
|
|
14
|
+
# tests: how many tests the mutant ran with coverage-based selection; nil when it ran the whole command.
|
|
15
|
+
Outcome = Struct.new(:mutation, :status, :seconds, :ignore, :tests, keyword_init: true) do
|
|
14
16
|
# :killed, :timeout (also a kill), :survived, :equivalent (survived, listed in the ignore file)
|
|
15
17
|
def killed? = %i[killed timeout].include?(status)
|
|
16
18
|
end
|
|
17
19
|
|
|
18
20
|
Report = Struct.new(:root, :command, :files, :outcomes, :sources, :stale, :warnings, :baseline_seconds, :timeout,
|
|
19
|
-
:jobs, :seconds, :complete, keyword_init: true) do
|
|
21
|
+
:jobs, :seconds, :complete, :coverage, keyword_init: true) do
|
|
20
22
|
def count(status) = outcomes.count { |o| o.status == status }
|
|
21
23
|
def killed = count(:killed) + count(:timeout)
|
|
22
24
|
|
|
@@ -29,10 +31,11 @@ module Mutaruby
|
|
|
29
31
|
|
|
30
32
|
Options = Struct.new(:root, :command, :patterns, :since, :jobs, :timeout, :timeout_factor, :methods,
|
|
31
33
|
:exclude_methods, :lines, :exclude_lines, :shard, :ignore_file, :operators, :copy_excludes,
|
|
32
|
-
:progress, keyword_init: true) do
|
|
34
|
+
:progress, :subjects, :coverage, :renderer, keyword_init: true) do
|
|
33
35
|
def self.build(**kw)
|
|
34
36
|
new(root: '.', patterns: [], jobs: 1, timeout_factor: 3.0, methods: [], exclude_methods: [], lines: [],
|
|
35
|
-
exclude_lines: [], copy_excludes: [], progress: false,
|
|
37
|
+
exclude_lines: [], copy_excludes: [], progress: false, subjects: [], coverage: false, renderer: :text,
|
|
38
|
+
**kw)
|
|
36
39
|
end
|
|
37
40
|
end
|
|
38
41
|
|
|
@@ -52,7 +55,7 @@ module Mutaruby
|
|
|
52
55
|
sources = files.to_h { |f| [f, File.binread(File.join(@root, f)).force_encoding(Encoding::UTF_8)] }
|
|
53
56
|
warnings = []
|
|
54
57
|
all = files.flat_map do |f|
|
|
55
|
-
g = Generator.new(sources[f], path: File.join(@root, f), rel: f, operators: @o.operators)
|
|
58
|
+
g = Generator.new(sources[f], path: File.join(@root, f), rel: f, operators: @o.operators, renderer: @o.renderer)
|
|
56
59
|
ms = g.mutations
|
|
57
60
|
warnings.concat(g.errors)
|
|
58
61
|
ms
|
|
@@ -73,28 +76,45 @@ module Mutaruby
|
|
|
73
76
|
p = plan
|
|
74
77
|
first = Sandbox.new(@root, excludes: @o.copy_excludes)
|
|
75
78
|
sandboxes << first
|
|
76
|
-
|
|
79
|
+
coverage_out = first.scratch('coverage.json')
|
|
80
|
+
base_env = @o.coverage ? CoverageMap.probe_env('MUTARUBY_COVERAGE_OUT' => coverage_out,
|
|
81
|
+
'MUTARUBY_COVERAGE_ROOT' => first.dir) : {}
|
|
82
|
+
base = cmd.run(first.dir, capture: true, env: base_env)
|
|
77
83
|
unless base.passed?
|
|
78
84
|
raise BaselineFailed, "the test command fails on the UNMUTATED code (exit #{base.exitstatus}); every mutant " \
|
|
79
85
|
"would count as killed, so nothing would be measured.\ncommand: #{@o.command}\n" \
|
|
80
86
|
"--- output (tail) ---\n#{base.output}"
|
|
81
87
|
end
|
|
82
88
|
timeout = @o.timeout || ((base.seconds * @o.timeout_factor) + 5).ceil
|
|
89
|
+
p[:coverage] = coverage_map(coverage_out, p[:warnings]) if @o.coverage
|
|
83
90
|
outcomes = execute(p, cmd, sandboxes, timeout)
|
|
84
91
|
matched = outcomes.select { |o| o.status == :equivalent }.map(&:ignore)
|
|
85
92
|
stale = p[:ignore] && p[:complete] ? p[:ignore].stale(matched, p[:files]) : []
|
|
86
93
|
Report.new(root: @root, command: @o.command, files: p[:files], outcomes: outcomes, sources: p[:sources],
|
|
87
94
|
stale: stale, warnings: p[:warnings], baseline_seconds: base.seconds, timeout: timeout,
|
|
88
|
-
jobs: jobs, seconds: monotonic - t0, complete: p[:complete])
|
|
95
|
+
jobs: jobs, seconds: monotonic - t0, complete: p[:complete], coverage: p[:coverage]&.framework)
|
|
89
96
|
rescue Interrupt
|
|
97
|
+
# Workers see @stop and start no new mutant; the runs in flight are killed; then the workers finish, before
|
|
98
|
+
# their sandboxes go.
|
|
99
|
+
@stop = true
|
|
90
100
|
cmd.kill_all
|
|
101
|
+
@threads&.each(&:join)
|
|
91
102
|
raise
|
|
92
103
|
ensure
|
|
93
|
-
sandboxes
|
|
104
|
+
remove(sandboxes, $!)
|
|
94
105
|
end
|
|
95
106
|
|
|
96
107
|
private
|
|
97
108
|
|
|
109
|
+
# Removes the sandboxes. When the run is already failing (an interrupt, a failing baseline), a sandbox that cannot
|
|
110
|
+
# be removed is left behind rather than replacing that error: on Windows a process that is still exiting keeps its
|
|
111
|
+
# directory locked for a moment.
|
|
112
|
+
def remove(sandboxes, failing)
|
|
113
|
+
sandboxes.each(&:cleanup)
|
|
114
|
+
rescue SystemCallError
|
|
115
|
+
raise unless failing
|
|
116
|
+
end
|
|
117
|
+
|
|
98
118
|
def jobs
|
|
99
119
|
[@o.jobs, 1].max
|
|
100
120
|
end
|
|
@@ -102,7 +122,17 @@ module Mutaruby
|
|
|
102
122
|
def selection
|
|
103
123
|
Selection.new(root: @root, patterns: @o.patterns, since: @o.since, methods: @o.methods,
|
|
104
124
|
exclude_methods: @o.exclude_methods, lines: @o.lines, exclude_lines: @o.exclude_lines,
|
|
105
|
-
shard: @o.shard)
|
|
125
|
+
shard: @o.shard, subjects: @o.subjects)
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
# The coverage map recorded by the baseline run, or nil (with a warning) when no supported test framework ran.
|
|
129
|
+
def coverage_map(path, warnings)
|
|
130
|
+
map = File.exist?(path) ? CoverageMap.load(path) : nil
|
|
131
|
+
return map if map&.usable?
|
|
132
|
+
|
|
133
|
+
warnings << 'coverage: no Minitest or RSpec tests were recorded in the baseline run; every mutant runs the ' \
|
|
134
|
+
'whole test command'
|
|
135
|
+
nil
|
|
106
136
|
end
|
|
107
137
|
|
|
108
138
|
def load_ignore
|
|
@@ -123,9 +153,9 @@ module Mutaruby
|
|
|
123
153
|
sandboxes << Sandbox.new(@root, excludes: @o.copy_excludes) while sandboxes.size < n
|
|
124
154
|
results = []
|
|
125
155
|
lock = Mutex.new
|
|
126
|
-
threads = sandboxes.map do |box|
|
|
156
|
+
@threads = sandboxes.map do |box|
|
|
127
157
|
Thread.new do
|
|
128
|
-
while (m = next_mutation(queue))
|
|
158
|
+
while !@stop && (m = next_mutation(queue))
|
|
129
159
|
o = evaluate(m, box, plan, cmd, timeout)
|
|
130
160
|
lock.synchronize do
|
|
131
161
|
results << o
|
|
@@ -134,33 +164,33 @@ module Mutaruby
|
|
|
134
164
|
end
|
|
135
165
|
end
|
|
136
166
|
end
|
|
137
|
-
|
|
138
|
-
threads.each(&:join)
|
|
139
|
-
rescue Interrupt
|
|
140
|
-
threads.each(&:kill)
|
|
141
|
-
raise
|
|
142
|
-
end
|
|
167
|
+
@threads.each(&:join)
|
|
143
168
|
@out.puts if @o.progress && results.any?
|
|
144
169
|
results.sort_by { |o| o.mutation.id }
|
|
145
170
|
end
|
|
146
171
|
|
|
147
172
|
def next_mutation(queue)
|
|
148
|
-
queue.
|
|
173
|
+
queue.deq(true)
|
|
149
174
|
rescue ThreadError
|
|
150
175
|
nil
|
|
151
176
|
end
|
|
152
177
|
|
|
153
178
|
def evaluate(m, box, plan, cmd, timeout)
|
|
154
|
-
|
|
155
|
-
|
|
179
|
+
env = { 'MUTARUBY_MUTANT_ID' => m.id.to_s, 'MUTARUBY_FILE' => m.file }
|
|
180
|
+
tests = plan[:coverage]&.tests_for(m.file, m.lineno..(m.lineno + m.orig.count("\n")))
|
|
181
|
+
if tests
|
|
182
|
+
list = box.scratch('tests.txt')
|
|
183
|
+
File.write(list, tests.map { |t| "#{t}\n" }.join)
|
|
184
|
+
env.merge!(CoverageMap.probe_env('MUTARUBY_SELECT' => list))
|
|
156
185
|
end
|
|
186
|
+
r = box.with_file(m.file, m.text, plan[:sources][m.file]) { cmd.run(box.dir, timeout: timeout, env: env) }
|
|
157
187
|
entry = plan[:ignore]&.match(m)
|
|
158
188
|
status = if r.timeout? then :timeout
|
|
159
189
|
elsif !r.passed? then :killed
|
|
160
190
|
elsif entry then :equivalent
|
|
161
191
|
else :survived
|
|
162
192
|
end
|
|
163
|
-
Outcome.new(mutation: m, status: status, seconds: r.seconds, ignore: entry)
|
|
193
|
+
Outcome.new(mutation: m, status: status, seconds: r.seconds, ignore: entry, tests: tests&.size)
|
|
164
194
|
end
|
|
165
195
|
|
|
166
196
|
def progress(outcome)
|
data/lib/mutaruby/sandbox.rb
CHANGED
|
@@ -17,7 +17,7 @@ module Mutaruby
|
|
|
17
17
|
# process that is still exiting, or a virus scanner, can hold it for a moment).
|
|
18
18
|
RETRIES = 10
|
|
19
19
|
RETRY_DELAY = 0.1
|
|
20
|
-
LOCKED = [Errno::EACCES, Errno::EBUSY, Errno::EPERM].freeze
|
|
20
|
+
LOCKED = [Errno::EACCES, Errno::EBUSY, Errno::EPERM, Errno::ENOTEMPTY].freeze
|
|
21
21
|
|
|
22
22
|
# Runs the block, retrying while it raises one of the LOCKED errors.
|
|
23
23
|
def self.retrying(tries: RETRIES, delay: RETRY_DELAY)
|
|
@@ -51,6 +51,11 @@ module Mutaruby
|
|
|
51
51
|
File.join(@dir, rel)
|
|
52
52
|
end
|
|
53
53
|
|
|
54
|
+
# A path for mutaruby's own files: next to the project copy, not inside it.
|
|
55
|
+
def scratch(name)
|
|
56
|
+
File.join(@base, name)
|
|
57
|
+
end
|
|
58
|
+
|
|
54
59
|
# Writes `text` to `rel` inside the sandbox, yields, and restores the original bytes afterwards (also on error).
|
|
55
60
|
def with_file(rel, text, original)
|
|
56
61
|
target = path(rel)
|
data/lib/mutaruby/selection.rb
CHANGED
|
@@ -34,7 +34,7 @@ module Mutaruby
|
|
|
34
34
|
|
|
35
35
|
# lines / exclude_lines: LineSpec objects (see parse_line_spec). shard: [i, n] or nil.
|
|
36
36
|
def initialize(root:, patterns: [], since: nil, methods: [], exclude_methods: [], lines: [], exclude_lines: [],
|
|
37
|
-
shard: nil)
|
|
37
|
+
shard: nil, subjects: [])
|
|
38
38
|
@root = File.expand_path(root)
|
|
39
39
|
@patterns = patterns.empty? ? DEFAULT_PATTERNS : patterns
|
|
40
40
|
@since = since
|
|
@@ -43,11 +43,13 @@ module Mutaruby
|
|
|
43
43
|
@lines = lines
|
|
44
44
|
@exclude_lines = exclude_lines
|
|
45
45
|
@shard = shard
|
|
46
|
+
@subjects = subjects
|
|
46
47
|
end
|
|
47
48
|
|
|
48
49
|
# True when every mutant of each selected file is run (so stale ignore entries can be detected).
|
|
49
50
|
def complete?
|
|
50
|
-
@methods.empty? && @exclude_methods.empty? && @lines.empty? && @exclude_lines.empty? && @shard.nil?
|
|
51
|
+
@methods.empty? && @exclude_methods.empty? && @lines.empty? && @exclude_lines.empty? && @shard.nil? &&
|
|
52
|
+
@subjects.empty?
|
|
51
53
|
end
|
|
52
54
|
|
|
53
55
|
# Sorted paths of the matching files, relative to the root.
|
|
@@ -64,7 +66,8 @@ module Mutaruby
|
|
|
64
66
|
|
|
65
67
|
def filter(mutations)
|
|
66
68
|
kept = mutations.select do |m|
|
|
67
|
-
(@
|
|
69
|
+
(@subjects.empty? || @subjects.any? { |s| subject_match?(s, m.method) }) &&
|
|
70
|
+
(@methods.empty? || @methods.any? { |p| method_match?(p, m.method) }) &&
|
|
68
71
|
@exclude_methods.none? { |p| method_match?(p, m.method) } &&
|
|
69
72
|
(@lines.empty? || @lines.any? { |s| s.cover?(m) }) &&
|
|
70
73
|
@exclude_lines.none? { |s| s.cover?(m) }
|
|
@@ -83,6 +86,21 @@ module Mutaruby
|
|
|
83
86
|
name ? File.fnmatch?(pattern, name) || File.fnmatch?(pattern, name.split(/[#.]/).last) : false
|
|
84
87
|
end
|
|
85
88
|
|
|
89
|
+
# A subject names methods by their place in the code:
|
|
90
|
+
# "Foo#bar", "Foo.bar" one method (instance, singleton); globs allowed: "Foo#*", "Foo.*", "Foo#to_*"
|
|
91
|
+
# "Foo", "Foo::Bar" every method of that namespace and of the namespaces nested in it
|
|
92
|
+
# "Foo*" a glob over namespaces ("Foo*" matches Foo, FooBar and Foo::Baz)
|
|
93
|
+
def subject_match?(subject, name)
|
|
94
|
+
name ? subject_name_match?(subject, name) : false
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def subject_name_match?(subject, name)
|
|
98
|
+
return File.fnmatch?(subject, name) if subject.match?(/[#.]/)
|
|
99
|
+
|
|
100
|
+
namespace = name.split(/[#.]/).first
|
|
101
|
+
namespace.start_with?("#{subject}::") || File.fnmatch?(subject, namespace)
|
|
102
|
+
end
|
|
103
|
+
|
|
86
104
|
# Files changed relative to a git ref: committed, staged and unstaged edits, plus new untracked files.
|
|
87
105
|
def changed_files
|
|
88
106
|
git('diff', '--name-only', '--relative', @since, '--') + git('ls-files', '--others', '--exclude-standard')
|
data/lib/mutaruby/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutaruby
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Atlas Authorization
|
|
@@ -46,12 +46,15 @@ files:
|
|
|
46
46
|
- lib/mutaruby.rb
|
|
47
47
|
- lib/mutaruby/cli.rb
|
|
48
48
|
- lib/mutaruby/command.rb
|
|
49
|
+
- lib/mutaruby/coverage_map.rb
|
|
49
50
|
- lib/mutaruby/error.rb
|
|
50
51
|
- lib/mutaruby/generator.rb
|
|
51
52
|
- lib/mutaruby/ignore_file.rb
|
|
52
53
|
- lib/mutaruby/mutation.rb
|
|
53
54
|
- lib/mutaruby/parsing.rb
|
|
54
55
|
- lib/mutaruby/platform.rb
|
|
56
|
+
- lib/mutaruby/probe/mutaruby_probe.rb
|
|
57
|
+
- lib/mutaruby/render.rb
|
|
55
58
|
- lib/mutaruby/reporters.rb
|
|
56
59
|
- lib/mutaruby/runner.rb
|
|
57
60
|
- lib/mutaruby/sandbox.rb
|