supercov 2.0.1 → 3.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/README.md +19 -22
  2. package/docs/agent-loop.md +8 -13
  3. package/docs/assertions.md +161 -75
  4. package/docs/cli.md +111 -59
  5. package/docs/coverage-model.md +5 -6
  6. package/docs/evidence.md +0 -11
  7. package/docs/getting-started.md +1 -1
  8. package/docs/performance.md +6 -14
  9. package/docs/quality.md +1 -1
  10. package/docs/security.md +43 -12
  11. package/docs/supported-suites.md +44 -48
  12. package/docs/troubleshooting.md +12 -5
  13. package/docs/workspace-isolation.md +23 -2
  14. package/package.json +33 -21
  15. package/runtime/javascript/atomic.mjs +46 -2
  16. package/runtime/javascript/ava.mjs +82 -0
  17. package/runtime/javascript/bootstrap.cjs +43 -0
  18. package/runtime/javascript/jest.cjs +52 -15
  19. package/runtime/javascript/jest.config.mjs +9 -1
  20. package/runtime/javascript/jestReporter.mjs +5 -6
  21. package/runtime/javascript/launchSupervisor.mjs +127 -15
  22. package/runtime/javascript/mocha.mjs +180 -0
  23. package/runtime/javascript/nodeTest.mjs +57 -3
  24. package/runtime/javascript/playwright.mjs +241 -32
  25. package/runtime/javascript/provenance.mjs +1 -1
  26. package/runtime/javascript/register.mjs +290 -2
  27. package/runtime/javascript/resolve-loader.mjs +40 -1
  28. package/runtime/javascript/runnerEvidence.mjs +3 -5
  29. package/runtime/javascript/runtime.mjs +551 -61
  30. package/runtime/javascript/tap.mjs +130 -0
  31. package/runtime/javascript/transport.mjs +7 -2
  32. package/runtime/javascript/vitest.mjs +5 -8
  33. package/runtime/javascript/vitestReporter.mjs +3 -10
  34. package/docs/assertion-agent.md +0 -150
  35. package/docs/assertion-evidence.md +0 -103
  36. package/docs/assertion-maps.md +0 -316
  37. package/schemas/assertions.schema.json +0 -276
package/README.md CHANGED
@@ -12,7 +12,7 @@ Measure code quality with npx supercov.
12
12
 
13
13
  Scoring needs a [TypeSafe AI](https://typesafe.ai) API key, and your coding agent will usually ask you for it. It costs about a cent per megabyte of source. Coverage needs no account, config file, import, custom reporter, or hosted service. Supercov is local, free, open source, and MIT licensed.
14
14
 
15
- [Website](https://supercov.com) · [Documentation](https://supercov.com/docs) · [npm](https://www.npmjs.com/package/supercov) · [GitHub](https://github.com/supercorp-ai/supercov)
15
+ [Website](https://supercov.com) · [Documentation](https://supercov.com/docs) · [npm](https://www.npmjs.com/package/supercov) · [GitHub](https://github.com/supercorp-ai/supercov) · [Discord](https://discord.gg/CudcAH53yF)
16
16
 
17
17
  Supported languages: [JavaScript](https://supercov.com/docs/javascript) · [TypeScript](https://supercov.com/docs/javascript) · [Rust](https://supercov.com/docs/rust) · [Python](https://supercov.com/docs/python) · [Ruby](https://supercov.com/docs/ruby) · [Go](https://supercov.com/docs/go) · [Java](https://supercov.com/docs/java) · [Kotlin](https://supercov.com/docs/kotlin).
18
18
 
@@ -141,30 +141,21 @@ npx supercov -- python -m unittest
141
141
  npx supercov -- bundle exec rspec
142
142
  ```
143
143
 
144
- ## Map what assertions check
144
+ ## See what the tests assert
145
145
 
146
- Each normal test run creates an assertion map automatically. An agent can
147
- optionally fill in what the assertions check:
146
+ Coverage says which code ran; assertion coverage says which of it the tests
147
+ would catch breaking. Supercov asks [Jev](https://typesafe.ai), test by test,
148
+ whether changing each statement the test ran would make it fail:
148
149
 
149
150
  ```sh
150
- npx supercov runs latest assertions --json
151
- # Pin the returned run ID. Inspect assertions and the source that ran:
152
- npx supercov runs <run> assertion <assertion-id>
153
- npx supercov runs <run> source src/example.ts
154
- # Edit assertions.json, then validate and acknowledge:
155
- npx supercov runs <run> assertions validate --json
156
- # Copy examined expectedBasis tokens into assertions.json; save again.
157
- npx supercov runs <run> assertions check --require-mappings --json
158
- npx supercov runs <run> # includes the assertion percentage in the regular report
151
+ npx supercov -- npm test
152
+ TYPESAFE_API_KEY=... npx supercov runs latest assertions assess
153
+ npx supercov runs latest assertions # the share asserted and what is not
159
154
  ```
160
155
 
161
- After code or tests change, run the same test command again. Supercov reuses the
162
- newest available map for that command and language and identifies dirty flows. Rust owns
163
- validation, change tracking and reporting; the agent supplies semantic reasoning.
164
- The score is agent-assessed and separate from MC/DC. See
165
- [assertion maps](docs/assertion-maps.md) for the format and JS/TS limits, or run
166
- `supercov docs assertion-agent` for the agent workflow. `supercov assertions schema`
167
- exports the editor schema; `assertions validate --file <path>` checks JSON syntax.
156
+ Answers are reused across runs whenever the question is the same, so after a
157
+ commit only what changed is asked again. Every language Supercov measures. See
158
+ [assertion coverage](docs/assertions.md).
168
159
 
169
160
  ## Give Supercov a job
170
161
 
@@ -313,9 +304,11 @@ Supercov uses exact per-test attribution where an adapter is available. For othe
313
304
  | --- | --- |
314
305
  | Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
315
306
  | Vitest | Exact per test, with setup execution kept separate |
316
- | Jest | Exact per test, including parameterized tests; exact `expect` occurrence identity for assertion maps |
307
+ | Jest | Exact per test, including parameterized tests |
317
308
  | `node:test` | Exact per test |
318
- | AVA and Mocha | Aggregate structural coverage |
309
+ | Mocha | Exact per test and retry, `--parallel` included |
310
+ | AVA | Exact per test, concurrent tests included |
311
+ | node-tap 15 and newer | Exact per subtest |
319
312
  | Cargo's standard libtest runner | Exact test, attempt, and passing-assertion identity |
320
313
  | rustdoc doctests | Exact per doctest |
321
314
  | cargo-nextest | Exact test, attempt, retry, and binary identity |
@@ -385,6 +378,10 @@ npx supercov runs clean # remove all runs and the build cache
385
378
 
386
379
  [MIT licensed](LICENSE). Inspect, extend, and run it anywhere.
387
380
 
381
+ ## Community
382
+
383
+ Ask questions, share what you measured, and talk to the people building Supercov on [Discord](https://discord.gg/CudcAH53yF).
384
+
388
385
  ## Contributors
389
386
 
390
387
  - [@DaVinciTachyon](https://github.com/DaVinciTachyon)
@@ -55,24 +55,19 @@ duplicate.
55
55
 
56
56
  ## Add assertion coverage
57
57
 
58
- For JavaScript and TypeScript projects, you can also ask the agent to explain
59
- what each test assertion checks. Every normal run creates an `assertions.json`
60
- map and carries forward compatible work from earlier runs of the same command.
58
+ For JavaScript and TypeScript projects, ask which executed statements no test
59
+ would catch breaking, then add checks for them:
61
60
 
62
61
  ```sh supercov
62
+ npx supercov runs latest assertions assess
63
63
  npx supercov runs latest assertions
64
- npx supercov docs assertion-agent
64
+ npx supercov runs latest assertions <path>
65
65
  ```
66
66
 
67
- Keep the printed run ID fixed while editing. The agent reads matching current
68
- source, updates the map, validates its references and saves review tokens for
69
- the explanations it examined. The regular run report then shows the assertion
70
- percentage. It does not need another test run just to read the edited map.
71
-
72
- Start with [Understanding assertion coverage](assertions.md) for a copyable
73
- prompt. [Mapping assertions with an agent](assertion-agent.md) gives the complete
74
- editing and checking loop. Use the evidence and recorded observations alongside
75
- MC/DC when deciding which tests need stronger checks.
67
+ `assess` needs `TYPESAFE_API_KEY`; reading does not. A statement that is not
68
+ asserted runs in some test, but nothing that test checks depends on it. After
69
+ adding a check, rerun the suite and assess again: only the questions the change
70
+ touched are asked. See [Assertion coverage](assertions.md).
76
71
 
77
72
  ## Example
78
73
 
@@ -1,17 +1,7 @@
1
1
  # Assertion coverage
2
2
 
3
- Line coverage shows which code ran. Assertion coverage helps you see what the
4
- tests checked. Your coding agent traces assertions back to the source, and
5
- Supercov checks those links against recorded execution from the same test.
6
-
7
- Assertion coverage measures JavaScript, TypeScript, Python, Ruby and Rust. The
8
- map format is the same for every one of them: an assertion is identified by its
9
- file, line and column, so a project written in more than one language keeps a
10
- single map.
11
-
12
- Python and Ruby report an assertion's line but not its column, so two
13
- assertions written on one line cannot be told apart and neither is credited.
14
- Put them on separate lines.
3
+ Line coverage shows which code ran. Assertion coverage shows which of that code
4
+ the tests would catch breaking.
15
5
 
16
6
  ## A passing test can miss a wrong result
17
7
 
@@ -37,75 +27,171 @@ test('confirms an order', () => {
37
27
  });
38
28
  ```
39
29
 
40
- Every line in the function runs. The test passes. But it would also pass if the
41
- total were `49` instead of `50`.
42
-
43
- ## How the agent finds the gap
44
-
45
- 1. **Supercov records execution:** which statements ran in each test and which
46
- assertions passed.
47
- 2. **Your agent traces each assertion** through the test and source to explain
48
- what it checks. It saves those links in the run's `assertions.json` map.
49
- 3. **Supercov checks the map against the run.** A statement needs a current
50
- explanation, execution, and a passing assertion in the same test to count.
51
-
52
- In this example, the agent follows the status assertion back to the returned
53
- order. It finds no check that reads the total:
54
-
55
- - `order.status` → must equal `'confirmed'`.
56
- - `order.total` → no assertion checks the calculated total.
57
-
58
- The agent adds the missing assertion to the existing test:
59
-
60
- ```js
61
- assert.equal(order.total, 50);
30
+ Every line runs and the test passes. It would also pass if `total` were
31
+ `undefined`, so the line computing it is covered but not asserted.
32
+
33
+ ## What counts as asserted
34
+
35
+ A statement is asserted when changing it would make at least one passing test
36
+ that runs it fail: an assertion fails, test or helper code throws, the test
37
+ times out, or the process crashes. The change depends on the statement:
38
+
39
+ | Statement | Change |
40
+ | --- | --- |
41
+ | `if` (and `elif`, `elsif`, `unless`, `else if`) | the condition inverted, each way separately (asserted if either is caught); a Rust `if let` is made not to match |
42
+ | `return x`, and a block's value in Ruby, Rust and Kotlin | returns a stand-in for nothing, without evaluating `x` |
43
+ | a `when`, `switch`, `try`, `loop` or Kotlin `throw` a function ends on | every value it returns becomes that stand-in (skipping it would leave the function nothing to return) |
44
+ | declaration or assignment with a value | the value becomes that stand-in, without being evaluated |
45
+ | JSX expression | the value becomes undefined |
46
+ | anything else | skipped: the statement does not run |
47
+
48
+ The stand-in is `undefined` in JavaScript and TypeScript, `None` in Python and
49
+ `nil` in Ruby. Go and the JVM languages replace a value after it is computed,
50
+ as their mutation testers do: the zero value in Go, and `null`, `0` or `false`
51
+ in Java and Kotlin. Rust has no value that fits every type, so it takes a
52
+ different value of the same type, from the function's signature or the
53
+ `let`'s annotation where there is one: a boolean negated, an `Ok` turned into
54
+ an `Err` and back, a literal swapped (`false` for `true`, `1` for `0`), an
55
+ `Ordering` reversed, `None`, `""` or `0`. Where nothing is annotated, the type
56
+ comes from the crate's own signatures, the parameters and `let`s in scope, or
57
+ a `match`'s arms (`let (major, text) = numeric(text)?;` takes `numeric`'s
58
+ return type): a tuple gets one element changed, an enum another of its
59
+ variants, a struct literal one field, a closure its result, integer
60
+ arithmetic its lowest bit flipped, a comparison its order reversed, a slice
61
+ or split the empty one, and `let Some(x) = .. else` gets `None`. A value of a
62
+ type the crate defines is built from its own constants, constructors and
63
+ enums (`Err(Error::new(ErrorKind::Empty))`, `Ok(Version::new(0, 0, 0))`).
64
+ `Default::default()` is left only where nothing says what the type is.
65
+
66
+ Imports, declarations without a value, and function, class and type
67
+ declarations are not assessed; in Python neither are `pass`, `global`,
68
+ `nonlocal` and docstrings, and in Ruby neither are the calls that declare
69
+ (`require`, `attr_reader`, `private`, `include` ...). **Assertions** is the
70
+ share of the executed statements that are asserted.
71
+
72
+ ## Assess and read
73
+
74
+ ```bash
75
+ npx supercov -- npm test
76
+ npx supercov runs latest assertions assess
77
+ npx supercov runs latest assertions
62
78
  ```
63
79
 
64
- It reruns the full suite and updates the map. A total of `49` now fails the
65
- test: two items at $25 must total $50. Line coverage has not changed, but the
66
- test now checks the calculation.
67
-
68
- ## Try it in your project
80
+ `assertions assess` works it out and saves the result to the run. For each test,
81
+ Supercov renders what the test ran -- its code, its helpers, and the source it
82
+ executed -- and asks [Jev](https://typesafe.ai) whether the test would fail for
83
+ each statement's change. It prints an estimate first; `--dry-run` sends nothing.
84
+ It needs a TypeSafe AI API key in `TYPESAFE_API_KEY`.
69
85
 
70
- Open your project in your usual coding agent and paste this prompt. The agent
71
- can install Supercov and run the commands for you.
86
+ `assertions` reads the saved result with no network and no key, a page at a
87
+ time, from the run down to one statement:
72
88
 
73
- ```text supercov-prompt
74
- Read npx supercov docs assertion-agent. Run the full test suite through
75
- Supercov, then build and validate its assertion map. Find one useful
76
- missing check and add an assertion for the expected behavior.
77
- Only change tests; do not weaken existing checks. Rerun the same suite
78
- and update the map. Show the test change, before-and-after assertion
79
- coverage, and any uncertainty or missing evidence.
89
+ ```bash
90
+ npx supercov runs latest assertions # the share, and the files by statements not asserted
91
+ npx supercov runs latest assertions src/adapters.ts # that file's statements not asserted
92
+ npx supercov runs latest assertions src/adapters.ts:100 # one statement: each asked test's answer
93
+ npx supercov runs latest assertions --test "closes the stream" # what one test catches, and what it misses
80
94
  ```
81
95
 
82
- The agent uses your actual test command after `--`, for example
83
- `npx supercov -- npm test`. On later runs of the same command, Supercov reuses
84
- compatible mappings and flags explanations that need another look.
85
-
86
- ## Read the result
87
-
88
- The agent can show the summary and the statement-level report for a file:
96
+ ```text
97
+ Run run_4f2a: 93.8% asserted (2731 of 2913 executed statements)
89
98
 
90
- ```sh
91
- npx supercov runs <run-id>
92
- npx supercov runs <run-id> assertions report --view statements --file checkout.js
99
+ Files, most statements not asserted first: a statement is not asserted when no test that runs it was judged to fail if it changed.
100
+ NOT ASSERTED ASSERTED FILE
101
+ 41 388/429 src/adapters.ts
102
+ ...
103
+ showing 1-20 of 57
104
+ next page: npx supercov runs 'run_4f2a' assertions --offset 20
105
+ inspect a file: npx supercov runs 'run_4f2a' assertions 'src/adapters.ts'
93
106
  ```
94
107
 
95
- **Assertions** is the percentage of measured source statements linked to
96
- passing assertions by the agent's map. It is not a count of assertions. Each
97
- statement counts once; statements that never ran remain in the total.
98
-
99
- An unmapped statement is a place to investigate, not proof of a missing test:
100
- the agent may not have mapped its existing check yet.
101
-
102
- The strength of the check still matters. `assert.ok(order.total)` accepts both
103
- `49` and `50`; `assert.equal(order.total, 50)` distinguishes them. Review the
104
- expected behavior, not just the percentage.
105
-
106
- Supercov does not generate or execute mutated code for this assessment. Unlike
107
- mutation testing, it does not test whether deliberately introduced bugs are
108
- caught. The agent's explanations still need review.
109
-
110
- For the detailed workflow, see [Mapping assertions with an agent](assertion-agent.md).
111
- For a result you cannot explain, see [Investigating assertion evidence](assertion-evidence.md).
108
+ Each listing ends with the command for its next page and for the next step
109
+ down, so an agent reads only as much as it needs. `runs latest file` and
110
+ `runs latest line` show the same verdicts beside the coverage, and
111
+ `npx supercov runs latest` and the HTML report show the same share.
112
+
113
+ A statement that is not asserted is a place to add a check: a test that runs it
114
+ exists, but nothing it asserts depends on what the statement does.
115
+
116
+ ## Which tests check a change
117
+
118
+ `npx supercov runs latest tests affected` uses the assessment to say which of
119
+ the tests a change reaches were judged to catch it, and lists them first;
120
+ `assertions assess --changed` asks every test that ran the changed code. See
121
+ [Find the tests a change affects](cli.md#find-the-tests-a-change-affects).
122
+
123
+ ## Later runs ask only about what changed
124
+
125
+ Answers are kept in `.supercov/assertions/`. Each is stored under the exact text
126
+ Jev saw for that statement and test, and reused only when a new run would ask
127
+ the same question with the same text. Nothing decides whether a change could
128
+ matter: a change to the statement, to code around it that the test ran, to the
129
+ test or its helpers, or to which tests run it, asks again. A dependency or
130
+ configuration change asks everything again. After a typical commit only a few
131
+ dozen questions are asked; the rest are reused.
132
+
133
+ ## Cost, speed and accuracy
134
+
135
+ - **Cost:** about 0.006 cents per statement for a first assessment: 14 projects
136
+ in six languages (8,356 statements) cost $0.48 together. A project of 3,000
137
+ statements costs about $0.18; a later commit usually well under a cent.
138
+ - **Speed:** about 20 seconds for 1,500 statements with 24 requests in flight
139
+ (`--workers`, default 24; it changes how long a pass takes, not what it
140
+ costs).
141
+
142
+ Measured on four JavaScript and TypeScript projects (7,393 executed statements):
143
+
144
+ - **Accuracy:** against ground truth from applying each change and running the
145
+ tests, on a fresh sample of 78 statements: 69 right; of the 13 not asserted it
146
+ found 8, and 8 of the 12 it flagged were truly not asserted. The share it
147
+ reported was 84.6% against a true 83.3%.
148
+ - **Python:** on h11 (532 executed statements, pytest), every statement checked
149
+ the same way: 502 right; of the 67 not asserted it found 45, and 45 of the 53
150
+ it flagged were truly not asserted. The share it reported was 90.0% against a
151
+ true 87.4%, for about $0.04.
152
+ - **Go, Rust, Java, Ruby and Kotlin**, checked the same way. These were used
153
+ to shape how changes and tests are shown to Jev, so they are flattering:
154
+
155
+ | Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
156
+ | --- | --- | --- | --- | --- |
157
+ | google/uuid (Go) | 339 | 298 | 34 (28) | 90.0% / 81.4% |
158
+ | hashicorp/go-version (Go) | 257 | 253 | 21 (19) | 91.8% / 91.8% |
159
+ | dtolnay/semver (Rust) | 200 of a sample of 200 | 190 | 7 (2) | 96.5% / 96.5% |
160
+ | apache/commons-cli (Java) | 196 of a sample of 200 | 167 | 49 (33) | 75.0% / 76.5% |
161
+ | dry-rb/dry-inflector (Ruby 3.4) | 157 | 148 | 13 (8) | 91.7% / 92.4% |
162
+ | sporkmonger/addressable (Ruby) | 198 of a sample of 200 | 182 | 25 (14) | 87.4% / 90.4% |
163
+ | square/moshi (Kotlin) | 60 of a sample of 60 | 52 | 9 (6) | 85.0% / 81.7% |
164
+
165
+ - **Held out:** five projects nothing was tuned on, each assessed once by the
166
+ shipped version and its verdicts frozen before any change was run, scored on
167
+ a pre-registered sample of 60 statements (and on every statement where the
168
+ suite is fast enough):
169
+
170
+ | Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
171
+ | --- | --- | --- | --- | --- |
172
+ | pallets/itsdangerous (Python) | 60 of 60 | 56 | 3 (2) | 95.0% / 91.7% |
173
+ | — every statement | 238 of 238 | 224 | 13 (9) | 94.5% / 92.0% |
174
+ | Masterminds/semver (Go) | 60 of 60 | 55 | 7 (7) | 88.3% / 80.0% |
175
+ | — every statement | 497 of 498 | 472 | 46 (44) | 90.7% / 86.5% |
176
+ | hashie/hashie (Ruby 3.4) | 60 of 60 | 57 | 3 (1) | 95.0% / 96.7% |
177
+ | — every statement | 700 of 701 | 655 | 26 (10) | 96.3% / 94.4% |
178
+ | FasterXML/java-classmate (Java) | 60 of 60 | 58 | 9 (7) | 85.0% / 88.3% |
179
+ | bluss/arrayvec (Rust) | 54 of 60 | 48 | 1 (1) | 98.1% / 87.0% |
180
+
181
+ Assessing all five (2,582 statements) cost $0.15, about 0.006 cents per
182
+ statement. Go reads high: Jev takes a dropped error (`return false,
183
+ fmt.Errorf(..)` becoming `return false, nil`) as caught where the tests check
184
+ only the boolean. Rust reads high too on arrayvec, whose code is generic over
185
+ its element type; six of its sampled changes cannot be made, because no value
186
+ of an unknown `T` can be written.
187
+
188
+ The share is a good measure of a project; a single statement's verdict is a
189
+ judgment worth checking before acting on it. Supercov does not run mutated code
190
+ for this: nothing is executed, and your tests are never slowed.
191
+
192
+ ## Requirements
193
+
194
+ Runs of any language Supercov measures: JavaScript and TypeScript, Python
195
+ (pytest, unittest), Ruby (RSpec, Minitest, test-unit, Cucumber), Go, Rust
196
+ (libtest, nextest, doctests), and Java and Kotlin (JUnit, TestNG); and
197
+ `TYPESAFE_API_KEY` for `assess`. Reading a saved result needs neither.
package/docs/cli.md CHANGED
@@ -17,8 +17,11 @@ npx supercov --help
17
17
  | Read the newest run | `npx supercov runs latest` |
18
18
  | Find useful gaps | `npx supercov runs latest gaps` |
19
19
  | Inspect one file | `npx supercov runs latest file <path>` |
20
- | List assertions and their status | `npx supercov runs latest assertions` |
21
- | Inspect one assertion and its flows | `npx supercov runs latest assertion <id>` |
20
+ | See how much executed code the tests assert | `npx supercov runs latest assertions` |
21
+ | Assess it with Jev | `npx supercov runs latest assertions assess` |
22
+ | Find what is not asserted in one file | `npx supercov runs latest assertions <path>` |
23
+ | See each test's answer for one statement | `npx supercov runs latest assertions <path>:<line>` |
24
+ | See what one test is judged to catch | `npx supercov runs latest assertions --test <name>` |
22
25
  | Read matching current source code | `npx supercov runs latest source <path>` |
23
26
  | Compare two runs | `npx supercov diff <older> <newer>` |
24
27
  | Open an interactive report | `npx supercov report` |
@@ -36,17 +39,17 @@ npx supercov --help
36
39
  ## Assess source quality
37
40
 
38
41
  ```sh supercov-example
39
- supercov quality # this repository
40
- supercov quality src/ # one directory
41
- supercov quality gaps # only files something fired on
42
- supercov quality file src/a.ts # one file, every check
43
- supercov quality scope # which files are assessed, and why
44
- supercov quality snapshots # saved assessments
45
- supercov quality diff <older> <newer>
46
- supercov security # twelve security checks, every file
47
- supercov security gaps # only files something fired on
48
- supercov security patch --base origin/main
49
- supercov security --run latest # flagged files no test exercises
42
+ npx supercov quality # this repository
43
+ npx supercov quality src/ # one directory
44
+ npx supercov quality gaps # only files something fired on
45
+ npx supercov quality file src/a.ts # one file, every check
46
+ npx supercov quality scope # which files are assessed, and why
47
+ npx supercov quality snapshots # saved assessments
48
+ npx supercov quality diff <older> <newer>
49
+ npx supercov security # twelve security checks, every file
50
+ npx supercov security gaps # only files something fired on
51
+ npx supercov security patch --base origin/main
52
+ npx supercov security --run latest # flagged files no test exercises
50
53
  ```
51
54
 
52
55
  With no argument the subject is the repository you are standing in. Every
@@ -74,9 +77,9 @@ number is worth and which files get assessed.
74
77
  ## Review what a change introduced
75
78
 
76
79
  ```sh supercov-example
77
- supercov quality patch
78
- supercov quality patch --base origin/main
79
- supercov quality patch --base origin/main --annotate github --run latest
80
+ npx supercov quality patch
81
+ npx supercov quality patch --base origin/main
82
+ npx supercov quality patch --base origin/main --annotate github --run latest
80
83
  ```
81
84
 
82
85
  The same twelve properties asked of a change, plus six risk checks that only
@@ -146,8 +149,7 @@ npx supercov runs <run-id> [query] [options]
146
149
  | `kinds` | Group coverage by test level, such as unit or E2E |
147
150
  | `runners` | Group coverage by test runner |
148
151
  | `scope` | Review included, excluded, and ambiguous source files |
149
- | `assertions` | List assertions, including sites without flows, with freshness and execution status |
150
- | `assertion <id>` | Inspect one assertion and its authored flows |
152
+ | `assertions` | Read the share of executed statements the tests assert, and the ones they do not; `assertions assess` works it out |
151
153
  | `source <path>` | Read matching current project source with line numbers |
152
154
  | `minimize` | Find a small test subset that preserves a coverage target |
153
155
 
@@ -169,46 +171,34 @@ npx supercov runs latest file --help
169
171
  npx supercov runs latest assertions --help
170
172
  ```
171
173
 
172
- Assertion queries read the run-owned map. `source <path>` reads the matching current
173
- file directly. It prints source code with line numbers, preserving indentation;
174
- add `--json` only when you want structured `{line, text}` items. `--offset` is
175
- zero-based and `--limit` controls the number of source lines. Source and assertion
176
- investigation require current files that match the run. Rerun the suite after
177
- source changes to inherit the map into a new run.
174
+ `source <path>` reads the matching current file directly. It prints source code
175
+ with line numbers, preserving indentation; add `--json` only when you want
176
+ structured `{line, text}` items. `--offset` is zero-based and `--limit` controls
177
+ the number of source lines. The file must still match the run; rerun the suite
178
+ after source changes.
178
179
 
179
180
  ### Assertion coverage
180
181
 
181
- The regular run summary includes assertion coverage when a map has been
182
- assessed. JSON reports expose it under `data.assertionCoverage`. Start with
183
- [Understanding assertion coverage](assertions.md), or use these commands to
184
- inspect and check a map:
185
-
186
182
  ```sh supercov-example
187
- npx supercov runs <run-id> assertions --needs-attention
188
- npx supercov runs <run-id> assertion <assertion-id>
189
- npx supercov runs <run-id> assertions report --view statements --file src/shipping.js
190
- npx supercov runs <run-id> assertions report --view excludedStatements
191
- npx supercov runs <run-id> assertions validate --json
192
- npx supercov runs <run-id> assertions check --require-mappings
183
+ npx supercov runs <run-id> assertions assess --dry-run
184
+ npx supercov runs <run-id> assertions assess
185
+ npx supercov runs <run-id> assertions
186
+ npx supercov runs <run-id> assertions --all --json
193
187
  ```
194
188
 
195
- Edit the file shown by `assertions`. Validation returns `expectedBasis` tokens;
196
- after examining a flow, save its token in the map before running `check`.
197
- `--require-mappings` requires explanations for recognized assertions observed
198
- passing. Add `--require-observed` when every mapped site and selector should have
199
- passing evidence, or `--min <percentage>` for a chosen target.
200
-
201
- To inspect one large flow, add `--flow <flow-id> --view nodes` or `--view edges`
202
- to the assertion detail command. `--compact` omits repeated source text from the
203
- report. Follow the printed next-page command or JSON `pagination.nextOffset`.
204
- Validation supports `--view flows`, `--view changes` and `--view errors` for large
205
- maps. The [map reference](assertion-maps.md) describes all fields and gates.
189
+ `assertions assess` asks Jev, test by test, whether each statement the test ran
190
+ would make it fail if changed, and saves the result to the run; it needs
191
+ `TYPESAFE_API_KEY` for questions not answered before. `assertions` reads the
192
+ saved result without the network. The regular run summary shows the same share,
193
+ and JSON reports expose it under `data.assertionCoverage`. See
194
+ [Assertion coverage](assertions.md) for what counts as asserted and how answers
195
+ are reused across runs.
206
196
 
207
197
  ## Fail CI below a coverage floor
208
198
 
209
199
  ```sh supercov-example
210
- supercov runs check --min-lines 90 --min-branches 80 --min-mcdc 80
211
- supercov runs check --min-lines 100 --per-file --json
200
+ npx supercov runs check --min-lines 90 --min-branches 80 --min-mcdc 80
201
+ npx supercov runs check --min-lines 100 --per-file --json
212
202
  ```
213
203
 
214
204
  `check` reads a recorded run; it never runs tests again. Give a floor per metric
@@ -231,16 +221,11 @@ rather than a pass or a failure:
231
221
  - a requested metric left obligations unmeasured, so no exact judgement exists
232
222
  - a requested metric is not recorded by the language adapter
233
223
 
234
- Assertion coverage keeps its own check. Whether a test *examines* what it
235
- executes is a different question from whether a line ran, and
236
- `runs <id> assertions check` carries the freshness and acknowledgement rules
237
- that answer needs.
238
-
239
224
  ## Check the lines a change touches
240
225
 
241
226
  ```sh supercov-example
242
- supercov runs patch --base origin/main --min-lines 100
243
- supercov runs patch --base origin/main --annotate github
227
+ npx supercov runs patch --base origin/main --min-lines 100
228
+ npx supercov runs patch --base origin/main --annotate github
244
229
  ```
245
230
 
246
231
  `patch` answers whether the lines this change added or modified are tested. It
@@ -267,9 +252,9 @@ needs no token and posts no comment.
267
252
  ## Export for other tools
268
253
 
269
254
  ```sh supercov-example
270
- supercov runs report --format lcov --output coverage/lcov.info
271
- supercov runs report --format cobertura --output coverage/cobertura.xml
272
- supercov runs report --format html --output coverage/report
255
+ npx supercov runs report --format lcov --output coverage/lcov.info
256
+ npx supercov runs report --format cobertura --output coverage/cobertura.xml
257
+ npx supercov runs report --format html --output coverage/report
273
258
  ```
274
259
 
275
260
  Both are written from the same view `check` and `patch` read, so a viewer,
@@ -401,11 +386,72 @@ neither do comments, blank lines or trailing whitespace. A test that did not
401
386
  pass in the run is listed regardless.
402
387
 
403
388
  `--names` prints one affected test name per line and `--files` one test file
404
- per line, for a runner's filter. A dependency, lockfile, configuration or
389
+ per line, for a runner's filter. A Vitest, Playwright or node:test test inside
390
+ `describe` blocks is printed as the blocks and the test joined by spaces
391
+ (`small charges`), which is what `vitest -t`, `--grep` and
392
+ `--test-name-pattern` match. A dependency, lockfile, configuration or
405
393
  toolchain change affects every test and is reported as such. A source file
406
394
  added since the run is outside every test's record; the working-tree check
407
395
  says so, and the suite should run in full.
408
396
 
397
+ ### Which affected tests check the change
398
+
399
+ When the run has an [assertion assessment](assertions.md), `tests affected`
400
+ also says, for each affected test, whether it was judged to catch a change to
401
+ the statements that changed in code it ran, and lists those tests first. A run
402
+ without an assessment answers from coverage alone, as above.
403
+
404
+ ```sh supercov
405
+ npx supercov runs latest assertions assess # once, on the run
406
+ # ... edit code ...
407
+ npx supercov runs latest assertions assess --changed # ask every test that ran the changed code
408
+ npx supercov runs latest tests affected --names --ran-changed
409
+ ```
410
+
411
+ The assessment stops asking about a statement once one test catches it, so on
412
+ its own it knows only some tests' answers. `assess --changed` asks, for the
413
+ statements in code changed since the run, every test that ran them: typically
414
+ a few requests per commit. After it, `--ran-changed` leaves out the affected
415
+ tests that ran the changed declaration but none of the changed statements, and
416
+ `--asserting` also those judged to catch no change to them. Tests whose own
417
+ file changed or that did not pass stay, and so does any test never asked. With
418
+ an assessment or without, `--names` and `--files` list first the tests whose
419
+ own file or name shares words with a changed file (`test_receivebuffer.py` for
420
+ `_receivebuffer.py`), the verdicts breaking ties.
421
+
422
+ Measured by applying random changes and running the suite:
423
+
424
+ | Project | Breaking changes (failing tests) | Coverage's set | `--ran-changed` | `--asserting` | A failing test in the first three |
425
+ | --- | --- | --- | --- | --- | --- |
426
+ | h11 (Python, 78 tests) | 23 (153) | all 153 in 28 tests | all 153 in 23.5 | 141 in 19.9 | 18 of 23 |
427
+ | go-version (Go, 31 tests) | 36 (140) | all 140 in 10.1 | all 140 in 9.8 | 133 in 9.4 | 32 of 36 |
428
+
429
+ `--ran-changed` never dropped a failing test; `--asserting` dropped 5 to 8% of
430
+ them. `assess --changed` cost about a tenth of a cent per change.
431
+
432
+ Measured on real history: at each of a project's recent commits, the source
433
+ the commit replaced was put back -- someone reintroducing what it fixed -- and
434
+ the commit's own tests run against it:
435
+
436
+ | Project | Commits (breaking) | Failing tests | Kept by coverage's set | Kept by `--asserting` | Tests selected (coverage / `--asserting`) of all |
437
+ | --- | --- | --- | --- | --- | --- |
438
+ | h11 (Python) | 60 (13) | 27 | 27 | 27 | 25.2 / 24.9 of 75 |
439
+ | go-version (Go) | 24 (2) | 3 | 3 | 3 | 15.2 / 14.1 of 25 |
440
+ | dry-inflector (Ruby) | 12 (7) | 26 | 26 | 26 | 813 / 718 of 1,109 |
441
+ | uuid (TypeScript, node:test) | 4 (3) | 6 | 6 | 6 | 13.2 / 11.5 of 64 |
442
+ | dtolnay/semver (Rust) | 20 (0) | 0 | -- | -- | 27.9 / 25.9 of 38 |
443
+ | commons-cli (Java) | 15 (7) | 12 | 12 | 12 | 159 / 143 of 985 |
444
+
445
+ No failing test was left out of the set to run. On real commits the
446
+ assessment narrows little -- Jev judges most tests that ran a changed line to
447
+ catch the change -- and its verdicts alone ranked a failing test in the first
448
+ three for only 6 of 32 breaking commits. Ranking by name, with the verdicts
449
+ breaking ties, put one there for 10 of those 32, and for 15 of 25 breaking
450
+ commits in four projects it was not designed on (hashie, Masterminds/semver,
451
+ itsdangerous, java-classmate), where file order managed 11; without an
452
+ assessment, name alone managed 8 and 12. After a project's first
453
+ assessment, each commit cost under a cent.
454
+
409
455
  ## Combine shards
410
456
 
411
457
  ```sh supercov
@@ -461,6 +507,7 @@ terminal or offline environment after the package has been downloaded.
461
507
  | --- | --- |
462
508
  | `SUPERCOV_SOURCE_ROOTS` | Comma-separated directories or files that hold your own code, in any language; everything else is left out |
463
509
  | `SUPERCOV_TEST_KIND` | Label the wrapped command as a test level such as `unit` or `e2e` |
510
+ | `SUPERCOV_BUILD_COMMAND` | The build a JavaScript project's tests need, as words or a JSON array, such as `yarn workspace web build` in a monorepo whose root `build` builds every package. Without it Supercov runs the project's `build` script, through the package manager that started the tests, when the tests need built output |
464
511
 
465
512
  Examples:
466
513
 
@@ -477,3 +524,8 @@ SUPERCOV_TEST_KIND=e2e npx supercov -- npx playwright test
477
524
  | Wrapped command's code | The test command failed and Supercov preserved its status |
478
525
  | `1` | A valid measurement failed a policy you set, such as a coverage floor |
479
526
  | `2` | Supercov could not complete the request, or the evidence cannot answer it |
527
+
528
+ `quality` and `security` exit `0` for a complete report in which some files
529
+ could not be assessed, such as a file larger than the model accepts; each is
530
+ listed in the report with its reason, and stderr says how many. They exit `2`
531
+ when no file could be assessed.