supercov 2.0.1 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -22
- package/docs/agent-loop.md +8 -13
- package/docs/assertions.md +161 -75
- package/docs/cli.md +111 -59
- package/docs/coverage-model.md +5 -6
- package/docs/evidence.md +0 -11
- package/docs/getting-started.md +1 -1
- package/docs/performance.md +6 -14
- package/docs/quality.md +1 -1
- package/docs/security.md +43 -12
- package/docs/supported-suites.md +44 -48
- package/docs/troubleshooting.md +12 -5
- package/docs/workspace-isolation.md +23 -2
- package/package.json +33 -21
- package/runtime/javascript/atomic.mjs +46 -2
- package/runtime/javascript/ava.mjs +82 -0
- package/runtime/javascript/bootstrap.cjs +43 -0
- package/runtime/javascript/jest.cjs +52 -15
- package/runtime/javascript/jest.config.mjs +9 -1
- package/runtime/javascript/jestReporter.mjs +5 -6
- package/runtime/javascript/launchSupervisor.mjs +127 -15
- package/runtime/javascript/mocha.mjs +180 -0
- package/runtime/javascript/nodeTest.mjs +57 -3
- package/runtime/javascript/playwright.mjs +241 -32
- package/runtime/javascript/provenance.mjs +1 -1
- package/runtime/javascript/register.mjs +290 -2
- package/runtime/javascript/resolve-loader.mjs +40 -1
- package/runtime/javascript/runnerEvidence.mjs +3 -5
- package/runtime/javascript/runtime.mjs +551 -61
- package/runtime/javascript/tap.mjs +130 -0
- package/runtime/javascript/transport.mjs +7 -2
- package/runtime/javascript/vitest.mjs +5 -8
- package/runtime/javascript/vitestReporter.mjs +3 -10
- package/docs/assertion-agent.md +0 -150
- package/docs/assertion-evidence.md +0 -103
- package/docs/assertion-maps.md +0 -316
- package/schemas/assertions.schema.json +0 -276
package/README.md
CHANGED
|
@@ -12,7 +12,7 @@ Measure code quality with npx supercov.
|
|
|
12
12
|
|
|
13
13
|
Scoring needs a [TypeSafe AI](https://typesafe.ai) API key, and your coding agent will usually ask you for it. It costs about a cent per megabyte of source. Coverage needs no account, config file, import, custom reporter, or hosted service. Supercov is local, free, open source, and MIT licensed.
|
|
14
14
|
|
|
15
|
-
[Website](https://supercov.com) · [Documentation](https://supercov.com/docs) · [npm](https://www.npmjs.com/package/supercov) · [GitHub](https://github.com/supercorp-ai/supercov)
|
|
15
|
+
[Website](https://supercov.com) · [Documentation](https://supercov.com/docs) · [npm](https://www.npmjs.com/package/supercov) · [GitHub](https://github.com/supercorp-ai/supercov) · [Discord](https://discord.gg/CudcAH53yF)
|
|
16
16
|
|
|
17
17
|
Supported languages: [JavaScript](https://supercov.com/docs/javascript) · [TypeScript](https://supercov.com/docs/javascript) · [Rust](https://supercov.com/docs/rust) · [Python](https://supercov.com/docs/python) · [Ruby](https://supercov.com/docs/ruby) · [Go](https://supercov.com/docs/go) · [Java](https://supercov.com/docs/java) · [Kotlin](https://supercov.com/docs/kotlin).
|
|
18
18
|
|
|
@@ -141,30 +141,21 @@ npx supercov -- python -m unittest
|
|
|
141
141
|
npx supercov -- bundle exec rspec
|
|
142
142
|
```
|
|
143
143
|
|
|
144
|
-
##
|
|
144
|
+
## See what the tests assert
|
|
145
145
|
|
|
146
|
-
|
|
147
|
-
|
|
146
|
+
Coverage says which code ran; assertion coverage says which of it the tests
|
|
147
|
+
would catch breaking. Supercov asks [Jev](https://typesafe.ai), test by test,
|
|
148
|
+
whether changing each statement the test ran would make it fail:
|
|
148
149
|
|
|
149
150
|
```sh
|
|
150
|
-
npx supercov
|
|
151
|
-
|
|
152
|
-
npx supercov runs
|
|
153
|
-
npx supercov runs <run> source src/example.ts
|
|
154
|
-
# Edit assertions.json, then validate and acknowledge:
|
|
155
|
-
npx supercov runs <run> assertions validate --json
|
|
156
|
-
# Copy examined expectedBasis tokens into assertions.json; save again.
|
|
157
|
-
npx supercov runs <run> assertions check --require-mappings --json
|
|
158
|
-
npx supercov runs <run> # includes the assertion percentage in the regular report
|
|
151
|
+
npx supercov -- npm test
|
|
152
|
+
TYPESAFE_API_KEY=... npx supercov runs latest assertions assess
|
|
153
|
+
npx supercov runs latest assertions # the share asserted and what is not
|
|
159
154
|
```
|
|
160
155
|
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
The score is agent-assessed and separate from MC/DC. See
|
|
165
|
-
[assertion maps](docs/assertion-maps.md) for the format and JS/TS limits, or run
|
|
166
|
-
`supercov docs assertion-agent` for the agent workflow. `supercov assertions schema`
|
|
167
|
-
exports the editor schema; `assertions validate --file <path>` checks JSON syntax.
|
|
156
|
+
Answers are reused across runs whenever the question is the same, so after a
|
|
157
|
+
commit only what changed is asked again. Every language Supercov measures. See
|
|
158
|
+
[assertion coverage](docs/assertions.md).
|
|
168
159
|
|
|
169
160
|
## Give Supercov a job
|
|
170
161
|
|
|
@@ -313,9 +304,11 @@ Supercov uses exact per-test attribution where an adapter is available. For othe
|
|
|
313
304
|
| --- | --- |
|
|
314
305
|
| Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
|
|
315
306
|
| Vitest | Exact per test, with setup execution kept separate |
|
|
316
|
-
| Jest | Exact per test, including parameterized tests
|
|
307
|
+
| Jest | Exact per test, including parameterized tests |
|
|
317
308
|
| `node:test` | Exact per test |
|
|
318
|
-
|
|
|
309
|
+
| Mocha | Exact per test and retry, `--parallel` included |
|
|
310
|
+
| AVA | Exact per test, concurrent tests included |
|
|
311
|
+
| node-tap 15 and newer | Exact per subtest |
|
|
319
312
|
| Cargo's standard libtest runner | Exact test, attempt, and passing-assertion identity |
|
|
320
313
|
| rustdoc doctests | Exact per doctest |
|
|
321
314
|
| cargo-nextest | Exact test, attempt, retry, and binary identity |
|
|
@@ -385,6 +378,10 @@ npx supercov runs clean # remove all runs and the build cache
|
|
|
385
378
|
|
|
386
379
|
[MIT licensed](LICENSE). Inspect, extend, and run it anywhere.
|
|
387
380
|
|
|
381
|
+
## Community
|
|
382
|
+
|
|
383
|
+
Ask questions, share what you measured, and talk to the people building Supercov on [Discord](https://discord.gg/CudcAH53yF).
|
|
384
|
+
|
|
388
385
|
## Contributors
|
|
389
386
|
|
|
390
387
|
- [@DaVinciTachyon](https://github.com/DaVinciTachyon)
|
package/docs/agent-loop.md
CHANGED
|
@@ -55,24 +55,19 @@ duplicate.
|
|
|
55
55
|
|
|
56
56
|
## Add assertion coverage
|
|
57
57
|
|
|
58
|
-
For JavaScript and TypeScript projects,
|
|
59
|
-
|
|
60
|
-
map and carries forward compatible work from earlier runs of the same command.
|
|
58
|
+
For JavaScript and TypeScript projects, ask which executed statements no test
|
|
59
|
+
would catch breaking, then add checks for them:
|
|
61
60
|
|
|
62
61
|
```sh supercov
|
|
62
|
+
npx supercov runs latest assertions assess
|
|
63
63
|
npx supercov runs latest assertions
|
|
64
|
-
npx supercov
|
|
64
|
+
npx supercov runs latest assertions <path>
|
|
65
65
|
```
|
|
66
66
|
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
Start with [Understanding assertion coverage](assertions.md) for a copyable
|
|
73
|
-
prompt. [Mapping assertions with an agent](assertion-agent.md) gives the complete
|
|
74
|
-
editing and checking loop. Use the evidence and recorded observations alongside
|
|
75
|
-
MC/DC when deciding which tests need stronger checks.
|
|
67
|
+
`assess` needs `TYPESAFE_API_KEY`; reading does not. A statement that is not
|
|
68
|
+
asserted runs in some test, but nothing that test checks depends on it. After
|
|
69
|
+
adding a check, rerun the suite and assess again: only the questions the change
|
|
70
|
+
touched are asked. See [Assertion coverage](assertions.md).
|
|
76
71
|
|
|
77
72
|
## Example
|
|
78
73
|
|
package/docs/assertions.md
CHANGED
|
@@ -1,17 +1,7 @@
|
|
|
1
1
|
# Assertion coverage
|
|
2
2
|
|
|
3
|
-
Line coverage shows which code ran. Assertion coverage
|
|
4
|
-
tests
|
|
5
|
-
Supercov checks those links against recorded execution from the same test.
|
|
6
|
-
|
|
7
|
-
Assertion coverage measures JavaScript, TypeScript, Python, Ruby and Rust. The
|
|
8
|
-
map format is the same for every one of them: an assertion is identified by its
|
|
9
|
-
file, line and column, so a project written in more than one language keeps a
|
|
10
|
-
single map.
|
|
11
|
-
|
|
12
|
-
Python and Ruby report an assertion's line but not its column, so two
|
|
13
|
-
assertions written on one line cannot be told apart and neither is credited.
|
|
14
|
-
Put them on separate lines.
|
|
3
|
+
Line coverage shows which code ran. Assertion coverage shows which of that code
|
|
4
|
+
the tests would catch breaking.
|
|
15
5
|
|
|
16
6
|
## A passing test can miss a wrong result
|
|
17
7
|
|
|
@@ -37,75 +27,171 @@ test('confirms an order', () => {
|
|
|
37
27
|
});
|
|
38
28
|
```
|
|
39
29
|
|
|
40
|
-
Every line
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
##
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
The
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
30
|
+
Every line runs and the test passes. It would also pass if `total` were
|
|
31
|
+
`undefined`, so the line computing it is covered but not asserted.
|
|
32
|
+
|
|
33
|
+
## What counts as asserted
|
|
34
|
+
|
|
35
|
+
A statement is asserted when changing it would make at least one passing test
|
|
36
|
+
that runs it fail: an assertion fails, test or helper code throws, the test
|
|
37
|
+
times out, or the process crashes. The change depends on the statement:
|
|
38
|
+
|
|
39
|
+
| Statement | Change |
|
|
40
|
+
| --- | --- |
|
|
41
|
+
| `if` (and `elif`, `elsif`, `unless`, `else if`) | the condition inverted, each way separately (asserted if either is caught); a Rust `if let` is made not to match |
|
|
42
|
+
| `return x`, and a block's value in Ruby, Rust and Kotlin | returns a stand-in for nothing, without evaluating `x` |
|
|
43
|
+
| a `when`, `switch`, `try`, `loop` or Kotlin `throw` a function ends on | every value it returns becomes that stand-in (skipping it would leave the function nothing to return) |
|
|
44
|
+
| declaration or assignment with a value | the value becomes that stand-in, without being evaluated |
|
|
45
|
+
| JSX expression | the value becomes undefined |
|
|
46
|
+
| anything else | skipped: the statement does not run |
|
|
47
|
+
|
|
48
|
+
The stand-in is `undefined` in JavaScript and TypeScript, `None` in Python and
|
|
49
|
+
`nil` in Ruby. Go and the JVM languages replace a value after it is computed,
|
|
50
|
+
as their mutation testers do: the zero value in Go, and `null`, `0` or `false`
|
|
51
|
+
in Java and Kotlin. Rust has no value that fits every type, so it takes a
|
|
52
|
+
different value of the same type, from the function's signature or the
|
|
53
|
+
`let`'s annotation where there is one: a boolean negated, an `Ok` turned into
|
|
54
|
+
an `Err` and back, a literal swapped (`false` for `true`, `1` for `0`), an
|
|
55
|
+
`Ordering` reversed, `None`, `""` or `0`. Where nothing is annotated, the type
|
|
56
|
+
comes from the crate's own signatures, the parameters and `let`s in scope, or
|
|
57
|
+
a `match`'s arms (`let (major, text) = numeric(text)?;` takes `numeric`'s
|
|
58
|
+
return type): a tuple gets one element changed, an enum another of its
|
|
59
|
+
variants, a struct literal one field, a closure its result, integer
|
|
60
|
+
arithmetic its lowest bit flipped, a comparison its order reversed, a slice
|
|
61
|
+
or split the empty one, and `let Some(x) = .. else` gets `None`. A value of a
|
|
62
|
+
type the crate defines is built from its own constants, constructors and
|
|
63
|
+
enums (`Err(Error::new(ErrorKind::Empty))`, `Ok(Version::new(0, 0, 0))`).
|
|
64
|
+
`Default::default()` is left only where nothing says what the type is.
|
|
65
|
+
|
|
66
|
+
Imports, declarations without a value, and function, class and type
|
|
67
|
+
declarations are not assessed; in Python neither are `pass`, `global`,
|
|
68
|
+
`nonlocal` and docstrings, and in Ruby neither are the calls that declare
|
|
69
|
+
(`require`, `attr_reader`, `private`, `include` ...). **Assertions** is the
|
|
70
|
+
share of the executed statements that are asserted.
|
|
71
|
+
|
|
72
|
+
## Assess and read
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
npx supercov -- npm test
|
|
76
|
+
npx supercov runs latest assertions assess
|
|
77
|
+
npx supercov runs latest assertions
|
|
62
78
|
```
|
|
63
79
|
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
80
|
+
`assertions assess` works it out and saves the result to the run. For each test,
|
|
81
|
+
Supercov renders what the test ran -- its code, its helpers, and the source it
|
|
82
|
+
executed -- and asks [Jev](https://typesafe.ai) whether the test would fail for
|
|
83
|
+
each statement's change. It prints an estimate first; `--dry-run` sends nothing.
|
|
84
|
+
It needs a TypeSafe AI API key in `TYPESAFE_API_KEY`.
|
|
69
85
|
|
|
70
|
-
|
|
71
|
-
|
|
86
|
+
`assertions` reads the saved result with no network and no key, a page at a
|
|
87
|
+
time, from the run down to one statement:
|
|
72
88
|
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
and update the map. Show the test change, before-and-after assertion
|
|
79
|
-
coverage, and any uncertainty or missing evidence.
|
|
89
|
+
```bash
|
|
90
|
+
npx supercov runs latest assertions # the share, and the files by statements not asserted
|
|
91
|
+
npx supercov runs latest assertions src/adapters.ts # that file's statements not asserted
|
|
92
|
+
npx supercov runs latest assertions src/adapters.ts:100 # one statement: each asked test's answer
|
|
93
|
+
npx supercov runs latest assertions --test "closes the stream" # what one test catches, and what it misses
|
|
80
94
|
```
|
|
81
95
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
compatible mappings and flags explanations that need another look.
|
|
85
|
-
|
|
86
|
-
## Read the result
|
|
87
|
-
|
|
88
|
-
The agent can show the summary and the statement-level report for a file:
|
|
96
|
+
```text
|
|
97
|
+
Run run_4f2a: 93.8% asserted (2731 of 2913 executed statements)
|
|
89
98
|
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
99
|
+
Files, most statements not asserted first: a statement is not asserted when no test that runs it was judged to fail if it changed.
|
|
100
|
+
NOT ASSERTED ASSERTED FILE
|
|
101
|
+
41 388/429 src/adapters.ts
|
|
102
|
+
...
|
|
103
|
+
showing 1-20 of 57
|
|
104
|
+
next page: npx supercov runs 'run_4f2a' assertions --offset 20
|
|
105
|
+
inspect a file: npx supercov runs 'run_4f2a' assertions 'src/adapters.ts'
|
|
93
106
|
```
|
|
94
107
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
108
|
+
Each listing ends with the command for its next page and for the next step
|
|
109
|
+
down, so an agent reads only as much as it needs. `runs latest file` and
|
|
110
|
+
`runs latest line` show the same verdicts beside the coverage, and
|
|
111
|
+
`npx supercov runs latest` and the HTML report show the same share.
|
|
112
|
+
|
|
113
|
+
A statement that is not asserted is a place to add a check: a test that runs it
|
|
114
|
+
exists, but nothing it asserts depends on what the statement does.
|
|
115
|
+
|
|
116
|
+
## Which tests check a change
|
|
117
|
+
|
|
118
|
+
`npx supercov runs latest tests affected` uses the assessment to say which of
|
|
119
|
+
the tests a change reaches were judged to catch it, and lists them first;
|
|
120
|
+
`assertions assess --changed` asks every test that ran the changed code. See
|
|
121
|
+
[Find the tests a change affects](cli.md#find-the-tests-a-change-affects).
|
|
122
|
+
|
|
123
|
+
## Later runs ask only about what changed
|
|
124
|
+
|
|
125
|
+
Answers are kept in `.supercov/assertions/`. Each is stored under the exact text
|
|
126
|
+
Jev saw for that statement and test, and reused only when a new run would ask
|
|
127
|
+
the same question with the same text. Nothing decides whether a change could
|
|
128
|
+
matter: a change to the statement, to code around it that the test ran, to the
|
|
129
|
+
test or its helpers, or to which tests run it, asks again. A dependency or
|
|
130
|
+
configuration change asks everything again. After a typical commit only a few
|
|
131
|
+
dozen questions are asked; the rest are reused.
|
|
132
|
+
|
|
133
|
+
## Cost, speed and accuracy
|
|
134
|
+
|
|
135
|
+
- **Cost:** about 0.006 cents per statement for a first assessment: 14 projects
|
|
136
|
+
in six languages (8,356 statements) cost $0.48 together. A project of 3,000
|
|
137
|
+
statements costs about $0.18; a later commit usually well under a cent.
|
|
138
|
+
- **Speed:** about 20 seconds for 1,500 statements with 24 requests in flight
|
|
139
|
+
(`--workers`, default 24; it changes how long a pass takes, not what it
|
|
140
|
+
costs).
|
|
141
|
+
|
|
142
|
+
Measured on four JavaScript and TypeScript projects (7,393 executed statements):
|
|
143
|
+
|
|
144
|
+
- **Accuracy:** against ground truth from applying each change and running the
|
|
145
|
+
tests, on a fresh sample of 78 statements: 69 right; of the 13 not asserted it
|
|
146
|
+
found 8, and 8 of the 12 it flagged were truly not asserted. The share it
|
|
147
|
+
reported was 84.6% against a true 83.3%.
|
|
148
|
+
- **Python:** on h11 (532 executed statements, pytest), every statement checked
|
|
149
|
+
the same way: 502 right; of the 67 not asserted it found 45, and 45 of the 53
|
|
150
|
+
it flagged were truly not asserted. The share it reported was 90.0% against a
|
|
151
|
+
true 87.4%, for about $0.04.
|
|
152
|
+
- **Go, Rust, Java, Ruby and Kotlin**, checked the same way. These were used
|
|
153
|
+
to shape how changes and tests are shown to Jev, so they are flattering:
|
|
154
|
+
|
|
155
|
+
| Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
|
|
156
|
+
| --- | --- | --- | --- | --- |
|
|
157
|
+
| google/uuid (Go) | 339 | 298 | 34 (28) | 90.0% / 81.4% |
|
|
158
|
+
| hashicorp/go-version (Go) | 257 | 253 | 21 (19) | 91.8% / 91.8% |
|
|
159
|
+
| dtolnay/semver (Rust) | 200 of a sample of 200 | 190 | 7 (2) | 96.5% / 96.5% |
|
|
160
|
+
| apache/commons-cli (Java) | 196 of a sample of 200 | 167 | 49 (33) | 75.0% / 76.5% |
|
|
161
|
+
| dry-rb/dry-inflector (Ruby 3.4) | 157 | 148 | 13 (8) | 91.7% / 92.4% |
|
|
162
|
+
| sporkmonger/addressable (Ruby) | 198 of a sample of 200 | 182 | 25 (14) | 87.4% / 90.4% |
|
|
163
|
+
| square/moshi (Kotlin) | 60 of a sample of 60 | 52 | 9 (6) | 85.0% / 81.7% |
|
|
164
|
+
|
|
165
|
+
- **Held out:** five projects nothing was tuned on, each assessed once by the
|
|
166
|
+
shipped version and its verdicts frozen before any change was run, scored on
|
|
167
|
+
a pre-registered sample of 60 statements (and on every statement where the
|
|
168
|
+
suite is fast enough):
|
|
169
|
+
|
|
170
|
+
| Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
|
|
171
|
+
| --- | --- | --- | --- | --- |
|
|
172
|
+
| pallets/itsdangerous (Python) | 60 of 60 | 56 | 3 (2) | 95.0% / 91.7% |
|
|
173
|
+
| — every statement | 238 of 238 | 224 | 13 (9) | 94.5% / 92.0% |
|
|
174
|
+
| Masterminds/semver (Go) | 60 of 60 | 55 | 7 (7) | 88.3% / 80.0% |
|
|
175
|
+
| — every statement | 497 of 498 | 472 | 46 (44) | 90.7% / 86.5% |
|
|
176
|
+
| hashie/hashie (Ruby 3.4) | 60 of 60 | 57 | 3 (1) | 95.0% / 96.7% |
|
|
177
|
+
| — every statement | 700 of 701 | 655 | 26 (10) | 96.3% / 94.4% |
|
|
178
|
+
| FasterXML/java-classmate (Java) | 60 of 60 | 58 | 9 (7) | 85.0% / 88.3% |
|
|
179
|
+
| bluss/arrayvec (Rust) | 54 of 60 | 48 | 1 (1) | 98.1% / 87.0% |
|
|
180
|
+
|
|
181
|
+
Assessing all five (2,582 statements) cost $0.15, about 0.006 cents per
|
|
182
|
+
statement. Go reads high: Jev takes a dropped error (`return false,
|
|
183
|
+
fmt.Errorf(..)` becoming `return false, nil`) as caught where the tests check
|
|
184
|
+
only the boolean. Rust reads high too on arrayvec, whose code is generic over
|
|
185
|
+
its element type; six of its sampled changes cannot be made, because no value
|
|
186
|
+
of an unknown `T` can be written.
|
|
187
|
+
|
|
188
|
+
The share is a good measure of a project; a single statement's verdict is a
|
|
189
|
+
judgment worth checking before acting on it. Supercov does not run mutated code
|
|
190
|
+
for this: nothing is executed, and your tests are never slowed.
|
|
191
|
+
|
|
192
|
+
## Requirements
|
|
193
|
+
|
|
194
|
+
Runs of any language Supercov measures: JavaScript and TypeScript, Python
|
|
195
|
+
(pytest, unittest), Ruby (RSpec, Minitest, test-unit, Cucumber), Go, Rust
|
|
196
|
+
(libtest, nextest, doctests), and Java and Kotlin (JUnit, TestNG); and
|
|
197
|
+
`TYPESAFE_API_KEY` for `assess`. Reading a saved result needs neither.
|
package/docs/cli.md
CHANGED
|
@@ -17,8 +17,11 @@ npx supercov --help
|
|
|
17
17
|
| Read the newest run | `npx supercov runs latest` |
|
|
18
18
|
| Find useful gaps | `npx supercov runs latest gaps` |
|
|
19
19
|
| Inspect one file | `npx supercov runs latest file <path>` |
|
|
20
|
-
|
|
|
21
|
-
|
|
|
20
|
+
| See how much executed code the tests assert | `npx supercov runs latest assertions` |
|
|
21
|
+
| Assess it with Jev | `npx supercov runs latest assertions assess` |
|
|
22
|
+
| Find what is not asserted in one file | `npx supercov runs latest assertions <path>` |
|
|
23
|
+
| See each test's answer for one statement | `npx supercov runs latest assertions <path>:<line>` |
|
|
24
|
+
| See what one test is judged to catch | `npx supercov runs latest assertions --test <name>` |
|
|
22
25
|
| Read matching current source code | `npx supercov runs latest source <path>` |
|
|
23
26
|
| Compare two runs | `npx supercov diff <older> <newer>` |
|
|
24
27
|
| Open an interactive report | `npx supercov report` |
|
|
@@ -36,17 +39,17 @@ npx supercov --help
|
|
|
36
39
|
## Assess source quality
|
|
37
40
|
|
|
38
41
|
```sh supercov-example
|
|
39
|
-
supercov quality # this repository
|
|
40
|
-
supercov quality src/ # one directory
|
|
41
|
-
supercov quality gaps # only files something fired on
|
|
42
|
-
supercov quality file src/a.ts # one file, every check
|
|
43
|
-
supercov quality scope # which files are assessed, and why
|
|
44
|
-
supercov quality snapshots # saved assessments
|
|
45
|
-
supercov quality diff <older> <newer>
|
|
46
|
-
supercov security # twelve security checks, every file
|
|
47
|
-
supercov security gaps # only files something fired on
|
|
48
|
-
supercov security patch --base origin/main
|
|
49
|
-
supercov security --run latest # flagged files no test exercises
|
|
42
|
+
npx supercov quality # this repository
|
|
43
|
+
npx supercov quality src/ # one directory
|
|
44
|
+
npx supercov quality gaps # only files something fired on
|
|
45
|
+
npx supercov quality file src/a.ts # one file, every check
|
|
46
|
+
npx supercov quality scope # which files are assessed, and why
|
|
47
|
+
npx supercov quality snapshots # saved assessments
|
|
48
|
+
npx supercov quality diff <older> <newer>
|
|
49
|
+
npx supercov security # twelve security checks, every file
|
|
50
|
+
npx supercov security gaps # only files something fired on
|
|
51
|
+
npx supercov security patch --base origin/main
|
|
52
|
+
npx supercov security --run latest # flagged files no test exercises
|
|
50
53
|
```
|
|
51
54
|
|
|
52
55
|
With no argument the subject is the repository you are standing in. Every
|
|
@@ -74,9 +77,9 @@ number is worth and which files get assessed.
|
|
|
74
77
|
## Review what a change introduced
|
|
75
78
|
|
|
76
79
|
```sh supercov-example
|
|
77
|
-
supercov quality patch
|
|
78
|
-
supercov quality patch --base origin/main
|
|
79
|
-
supercov quality patch --base origin/main --annotate github --run latest
|
|
80
|
+
npx supercov quality patch
|
|
81
|
+
npx supercov quality patch --base origin/main
|
|
82
|
+
npx supercov quality patch --base origin/main --annotate github --run latest
|
|
80
83
|
```
|
|
81
84
|
|
|
82
85
|
The same twelve properties asked of a change, plus six risk checks that only
|
|
@@ -146,8 +149,7 @@ npx supercov runs <run-id> [query] [options]
|
|
|
146
149
|
| `kinds` | Group coverage by test level, such as unit or E2E |
|
|
147
150
|
| `runners` | Group coverage by test runner |
|
|
148
151
|
| `scope` | Review included, excluded, and ambiguous source files |
|
|
149
|
-
| `assertions` |
|
|
150
|
-
| `assertion <id>` | Inspect one assertion and its authored flows |
|
|
152
|
+
| `assertions` | Read the share of executed statements the tests assert, and the ones they do not; `assertions assess` works it out |
|
|
151
153
|
| `source <path>` | Read matching current project source with line numbers |
|
|
152
154
|
| `minimize` | Find a small test subset that preserves a coverage target |
|
|
153
155
|
|
|
@@ -169,46 +171,34 @@ npx supercov runs latest file --help
|
|
|
169
171
|
npx supercov runs latest assertions --help
|
|
170
172
|
```
|
|
171
173
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
source changes to inherit the map into a new run.
|
|
174
|
+
`source <path>` reads the matching current file directly. It prints source code
|
|
175
|
+
with line numbers, preserving indentation; add `--json` only when you want
|
|
176
|
+
structured `{line, text}` items. `--offset` is zero-based and `--limit` controls
|
|
177
|
+
the number of source lines. The file must still match the run; rerun the suite
|
|
178
|
+
after source changes.
|
|
178
179
|
|
|
179
180
|
### Assertion coverage
|
|
180
181
|
|
|
181
|
-
The regular run summary includes assertion coverage when a map has been
|
|
182
|
-
assessed. JSON reports expose it under `data.assertionCoverage`. Start with
|
|
183
|
-
[Understanding assertion coverage](assertions.md), or use these commands to
|
|
184
|
-
inspect and check a map:
|
|
185
|
-
|
|
186
182
|
```sh supercov-example
|
|
187
|
-
npx supercov runs <run-id> assertions --
|
|
188
|
-
npx supercov runs <run-id>
|
|
189
|
-
npx supercov runs <run-id> assertions
|
|
190
|
-
npx supercov runs <run-id> assertions
|
|
191
|
-
npx supercov runs <run-id> assertions validate --json
|
|
192
|
-
npx supercov runs <run-id> assertions check --require-mappings
|
|
183
|
+
npx supercov runs <run-id> assertions assess --dry-run
|
|
184
|
+
npx supercov runs <run-id> assertions assess
|
|
185
|
+
npx supercov runs <run-id> assertions
|
|
186
|
+
npx supercov runs <run-id> assertions --all --json
|
|
193
187
|
```
|
|
194
188
|
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
to the assertion detail command. `--compact` omits repeated source text from the
|
|
203
|
-
report. Follow the printed next-page command or JSON `pagination.nextOffset`.
|
|
204
|
-
Validation supports `--view flows`, `--view changes` and `--view errors` for large
|
|
205
|
-
maps. The [map reference](assertion-maps.md) describes all fields and gates.
|
|
189
|
+
`assertions assess` asks Jev, test by test, whether each statement the test ran
|
|
190
|
+
would make it fail if changed, and saves the result to the run; it needs
|
|
191
|
+
`TYPESAFE_API_KEY` for questions not answered before. `assertions` reads the
|
|
192
|
+
saved result without the network. The regular run summary shows the same share,
|
|
193
|
+
and JSON reports expose it under `data.assertionCoverage`. See
|
|
194
|
+
[Assertion coverage](assertions.md) for what counts as asserted and how answers
|
|
195
|
+
are reused across runs.
|
|
206
196
|
|
|
207
197
|
## Fail CI below a coverage floor
|
|
208
198
|
|
|
209
199
|
```sh supercov-example
|
|
210
|
-
supercov runs check --min-lines 90 --min-branches 80 --min-mcdc 80
|
|
211
|
-
supercov runs check --min-lines 100 --per-file --json
|
|
200
|
+
npx supercov runs check --min-lines 90 --min-branches 80 --min-mcdc 80
|
|
201
|
+
npx supercov runs check --min-lines 100 --per-file --json
|
|
212
202
|
```
|
|
213
203
|
|
|
214
204
|
`check` reads a recorded run; it never runs tests again. Give a floor per metric
|
|
@@ -231,16 +221,11 @@ rather than a pass or a failure:
|
|
|
231
221
|
- a requested metric left obligations unmeasured, so no exact judgement exists
|
|
232
222
|
- a requested metric is not recorded by the language adapter
|
|
233
223
|
|
|
234
|
-
Assertion coverage keeps its own check. Whether a test *examines* what it
|
|
235
|
-
executes is a different question from whether a line ran, and
|
|
236
|
-
`runs <id> assertions check` carries the freshness and acknowledgement rules
|
|
237
|
-
that answer needs.
|
|
238
|
-
|
|
239
224
|
## Check the lines a change touches
|
|
240
225
|
|
|
241
226
|
```sh supercov-example
|
|
242
|
-
supercov runs patch --base origin/main --min-lines 100
|
|
243
|
-
supercov runs patch --base origin/main --annotate github
|
|
227
|
+
npx supercov runs patch --base origin/main --min-lines 100
|
|
228
|
+
npx supercov runs patch --base origin/main --annotate github
|
|
244
229
|
```
|
|
245
230
|
|
|
246
231
|
`patch` answers whether the lines this change added or modified are tested. It
|
|
@@ -267,9 +252,9 @@ needs no token and posts no comment.
|
|
|
267
252
|
## Export for other tools
|
|
268
253
|
|
|
269
254
|
```sh supercov-example
|
|
270
|
-
supercov runs report --format lcov --output coverage/lcov.info
|
|
271
|
-
supercov runs report --format cobertura --output coverage/cobertura.xml
|
|
272
|
-
supercov runs report --format html --output coverage/report
|
|
255
|
+
npx supercov runs report --format lcov --output coverage/lcov.info
|
|
256
|
+
npx supercov runs report --format cobertura --output coverage/cobertura.xml
|
|
257
|
+
npx supercov runs report --format html --output coverage/report
|
|
273
258
|
```
|
|
274
259
|
|
|
275
260
|
Both are written from the same view `check` and `patch` read, so a viewer,
|
|
@@ -401,11 +386,72 @@ neither do comments, blank lines or trailing whitespace. A test that did not
|
|
|
401
386
|
pass in the run is listed regardless.
|
|
402
387
|
|
|
403
388
|
`--names` prints one affected test name per line and `--files` one test file
|
|
404
|
-
per line, for a runner's filter. A
|
|
389
|
+
per line, for a runner's filter. A Vitest, Playwright or node:test test inside
|
|
390
|
+
`describe` blocks is printed as the blocks and the test joined by spaces
|
|
391
|
+
(`small charges`), which is what `vitest -t`, `--grep` and
|
|
392
|
+
`--test-name-pattern` match. A dependency, lockfile, configuration or
|
|
405
393
|
toolchain change affects every test and is reported as such. A source file
|
|
406
394
|
added since the run is outside every test's record; the working-tree check
|
|
407
395
|
says so, and the suite should run in full.
|
|
408
396
|
|
|
397
|
+
### Which affected tests check the change
|
|
398
|
+
|
|
399
|
+
When the run has an [assertion assessment](assertions.md), `tests affected`
|
|
400
|
+
also says, for each affected test, whether it was judged to catch a change to
|
|
401
|
+
the statements that changed in code it ran, and lists those tests first. A run
|
|
402
|
+
without an assessment answers from coverage alone, as above.
|
|
403
|
+
|
|
404
|
+
```sh supercov
|
|
405
|
+
npx supercov runs latest assertions assess # once, on the run
|
|
406
|
+
# ... edit code ...
|
|
407
|
+
npx supercov runs latest assertions assess --changed # ask every test that ran the changed code
|
|
408
|
+
npx supercov runs latest tests affected --names --ran-changed
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
The assessment stops asking about a statement once one test catches it, so on
|
|
412
|
+
its own it knows only some tests' answers. `assess --changed` asks, for the
|
|
413
|
+
statements in code changed since the run, every test that ran them: typically
|
|
414
|
+
a few requests per commit. After it, `--ran-changed` leaves out the affected
|
|
415
|
+
tests that ran the changed declaration but none of the changed statements, and
|
|
416
|
+
`--asserting` also those judged to catch no change to them. Tests whose own
|
|
417
|
+
file changed or that did not pass stay, and so does any test never asked. With
|
|
418
|
+
an assessment or without, `--names` and `--files` list first the tests whose
|
|
419
|
+
own file or name shares words with a changed file (`test_receivebuffer.py` for
|
|
420
|
+
`_receivebuffer.py`), the verdicts breaking ties.
|
|
421
|
+
|
|
422
|
+
Measured by applying random changes and running the suite:
|
|
423
|
+
|
|
424
|
+
| Project | Breaking changes (failing tests) | Coverage's set | `--ran-changed` | `--asserting` | A failing test in the first three |
|
|
425
|
+
| --- | --- | --- | --- | --- | --- |
|
|
426
|
+
| h11 (Python, 78 tests) | 23 (153) | all 153 in 28 tests | all 153 in 23.5 | 141 in 19.9 | 18 of 23 |
|
|
427
|
+
| go-version (Go, 31 tests) | 36 (140) | all 140 in 10.1 | all 140 in 9.8 | 133 in 9.4 | 32 of 36 |
|
|
428
|
+
|
|
429
|
+
`--ran-changed` never dropped a failing test; `--asserting` dropped 5 to 8% of
|
|
430
|
+
them. `assess --changed` cost about a tenth of a cent per change.
|
|
431
|
+
|
|
432
|
+
Measured on real history: at each of a project's recent commits, the source
|
|
433
|
+
the commit replaced was put back -- someone reintroducing what it fixed -- and
|
|
434
|
+
the commit's own tests run against it:
|
|
435
|
+
|
|
436
|
+
| Project | Commits (breaking) | Failing tests | Kept by coverage's set | Kept by `--asserting` | Tests selected (coverage / `--asserting`) of all |
|
|
437
|
+
| --- | --- | --- | --- | --- | --- |
|
|
438
|
+
| h11 (Python) | 60 (13) | 27 | 27 | 27 | 25.2 / 24.9 of 75 |
|
|
439
|
+
| go-version (Go) | 24 (2) | 3 | 3 | 3 | 15.2 / 14.1 of 25 |
|
|
440
|
+
| dry-inflector (Ruby) | 12 (7) | 26 | 26 | 26 | 813 / 718 of 1,109 |
|
|
441
|
+
| uuid (TypeScript, node:test) | 4 (3) | 6 | 6 | 6 | 13.2 / 11.5 of 64 |
|
|
442
|
+
| dtolnay/semver (Rust) | 20 (0) | 0 | -- | -- | 27.9 / 25.9 of 38 |
|
|
443
|
+
| commons-cli (Java) | 15 (7) | 12 | 12 | 12 | 159 / 143 of 985 |
|
|
444
|
+
|
|
445
|
+
No failing test was left out of the set to run. On real commits the
|
|
446
|
+
assessment narrows little -- Jev judges most tests that ran a changed line to
|
|
447
|
+
catch the change -- and its verdicts alone ranked a failing test in the first
|
|
448
|
+
three for only 6 of 32 breaking commits. Ranking by name, with the verdicts
|
|
449
|
+
breaking ties, put one there for 10 of those 32, and for 15 of 25 breaking
|
|
450
|
+
commits in four projects it was not designed on (hashie, Masterminds/semver,
|
|
451
|
+
itsdangerous, java-classmate), where file order managed 11; without an
|
|
452
|
+
assessment, name alone managed 8 and 12. After a project's first
|
|
453
|
+
assessment, each commit cost under a cent.
|
|
454
|
+
|
|
409
455
|
## Combine shards
|
|
410
456
|
|
|
411
457
|
```sh supercov
|
|
@@ -461,6 +507,7 @@ terminal or offline environment after the package has been downloaded.
|
|
|
461
507
|
| --- | --- |
|
|
462
508
|
| `SUPERCOV_SOURCE_ROOTS` | Comma-separated directories or files that hold your own code, in any language; everything else is left out |
|
|
463
509
|
| `SUPERCOV_TEST_KIND` | Label the wrapped command as a test level such as `unit` or `e2e` |
|
|
510
|
+
| `SUPERCOV_BUILD_COMMAND` | The build a JavaScript project's tests need, as words or a JSON array, such as `yarn workspace web build` in a monorepo whose root `build` builds every package. Without it Supercov runs the project's `build` script, through the package manager that started the tests, when the tests need built output |
|
|
464
511
|
|
|
465
512
|
Examples:
|
|
466
513
|
|
|
@@ -477,3 +524,8 @@ SUPERCOV_TEST_KIND=e2e npx supercov -- npx playwright test
|
|
|
477
524
|
| Wrapped command's code | The test command failed and Supercov preserved its status |
|
|
478
525
|
| `1` | A valid measurement failed a policy you set, such as a coverage floor |
|
|
479
526
|
| `2` | Supercov could not complete the request, or the evidence cannot answer it |
|
|
527
|
+
|
|
528
|
+
`quality` and `security` exit `0` for a complete report in which some files
|
|
529
|
+
could not be assessed, such as a file larger than the model accepts; each is
|
|
530
|
+
listed in the report with its reason, and stderr says how many. They exit `2`
|
|
531
|
+
when no file could be assessed.
|