supercov 0.0.46 → 0.0.47
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/assertion-maps.md +21 -0
- package/docs/assertions.md +79 -100
- package/package.json +9 -9
package/docs/assertion-maps.md
CHANGED
|
@@ -105,6 +105,27 @@ After examining a flow, copy its returned `expectedBasis` into that flow's
|
|
|
105
105
|
do not generate it yourself. Editing a claim or its dependencies makes the old
|
|
106
106
|
token stale.
|
|
107
107
|
|
|
108
|
+
### What makes a review token stale
|
|
109
|
+
|
|
110
|
+
A flow needs a fresh review when its claim changes, when a file it watches or a
|
|
111
|
+
test it selects changes, or when the project's dependencies, configuration or
|
|
112
|
+
the instrumenter change. The ambient environment does not count: running from
|
|
113
|
+
another directory, a new terminal session, a different package manager or
|
|
114
|
+
another Node installation leaves current flows current. A behavioural
|
|
115
|
+
difference that matters still shows up on its own, because credit requires a
|
|
116
|
+
passing assertion occurrence and execution of the claimed statement in the same
|
|
117
|
+
selected test.
|
|
118
|
+
|
|
119
|
+
If your suite genuinely depends on particular variables, name them; only the
|
|
120
|
+
ones you name participate, and an unset variable is recorded as absent.
|
|
121
|
+
|
|
122
|
+
```sh supercov-example
|
|
123
|
+
SUPERCOV_ASSERTION_CONTEXT_ENV=TZ,LANG npx supercov -- npm test
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Use the same list for every run. Changing it changes the recorded context and
|
|
127
|
+
asks for a fresh review.
|
|
128
|
+
|
|
108
129
|
```sh supercov-example
|
|
109
130
|
npx supercov runs <run-id> assertions check --require-mappings
|
|
110
131
|
npx supercov runs <run-id>
|
package/docs/assertions.md
CHANGED
|
@@ -1,132 +1,111 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Assertion coverage
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
run's execution evidence and shows an **Assertions** percentage alongside Lines,
|
|
7
|
-
Branches and MC/DC.
|
|
3
|
+
Line coverage shows which code ran. Assertion coverage helps you see what the
|
|
4
|
+
tests checked. Your coding agent traces assertions back to the source, and
|
|
5
|
+
Supercov checks those links against recorded execution from the same test.
|
|
8
6
|
|
|
9
|
-
|
|
7
|
+
Assertion coverage measures JavaScript, TypeScript, Python, Ruby and Rust. The
|
|
8
|
+
map format is the same for every one of them: an assertion is identified by its
|
|
9
|
+
file, line and column, so a project written in more than one language keeps a
|
|
10
|
+
single map.
|
|
10
11
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
Use your project's test command after `--`. Each run creates its own map and
|
|
17
|
-
reuses compatible work from earlier runs of the same command. The assertions
|
|
18
|
-
list shows the map's path and the assertions that still need attention.
|
|
12
|
+
Python and Ruby report an assertion's line but not its column, so two
|
|
13
|
+
assertions written on one line cannot be told apart and neither is credited.
|
|
14
|
+
Put them on separate lines.
|
|
19
15
|
|
|
20
|
-
|
|
21
|
-
work on the same run even if another test run finishes. You can give it this
|
|
22
|
-
prompt, replacing `<run-id>` with that ID:
|
|
23
|
-
|
|
24
|
-
```text
|
|
25
|
-
Read the instructions from npx supercov docs assertion-agent. Build or update
|
|
26
|
-
assertion coverage for run <run-id>. Investigate the tests and current source,
|
|
27
|
-
then edit that run's assertions.json. Preserve reusable flows and explain
|
|
28
|
-
uncertainty. Do not change application code or tests. Validate the map and
|
|
29
|
-
show the assertion percentage, remaining gaps, and any missing evidence.
|
|
30
|
-
```
|
|
16
|
+
## A passing test can miss a wrong result
|
|
31
17
|
|
|
32
|
-
|
|
33
|
-
model or make model calls itself.
|
|
18
|
+
This function confirms an order and calculates its total:
|
|
34
19
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
20
|
+
```js
|
|
21
|
+
export function checkout(price, quantity) {
|
|
22
|
+
const total = price * quantity;
|
|
23
|
+
return { status: 'confirmed', total };
|
|
24
|
+
}
|
|
39
25
|
```
|
|
40
26
|
|
|
41
|
-
|
|
27
|
+
The test checks the status, but not the total:
|
|
42
28
|
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
|
|
29
|
+
```js
|
|
30
|
+
import assert from 'node:assert/strict';
|
|
31
|
+
import test from 'node:test';
|
|
32
|
+
import { checkout } from './checkout.js';
|
|
33
|
+
|
|
34
|
+
test('confirms an order', () => {
|
|
35
|
+
const order = checkout(25, 2);
|
|
36
|
+
assert.equal(order.status, 'confirmed');
|
|
37
|
+
});
|
|
46
38
|
```
|
|
47
39
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
statement counts once, even when several assertions check it. Unexecuted code
|
|
51
|
-
stays in the denominator. TypeScript imports that are known to disappear during
|
|
52
|
-
compilation are excluded.
|
|
40
|
+
Every line in the function runs. The test passes. But it would also pass if the
|
|
41
|
+
total were `49` instead of `50`.
|
|
53
42
|
|
|
54
|
-
|
|
55
|
-
not change the assertion percentage. You do not need to rerun tests just to
|
|
56
|
-
recalculate a map: save it and query the run again.
|
|
43
|
+
## How the agent finds the gap
|
|
57
44
|
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
45
|
+
1. **Supercov records execution:** which statements ran in each test and which
|
|
46
|
+
assertions passed.
|
|
47
|
+
2. **Your agent traces each assertion** through the test and source to explain
|
|
48
|
+
what it checks. It saves those links in the run's `assertions.json` map.
|
|
49
|
+
3. **Supercov checks the map against the run.** A statement needs a current
|
|
50
|
+
explanation, execution, and a passing assertion in the same test to count.
|
|
64
51
|
|
|
65
|
-
|
|
66
|
-
|
|
52
|
+
In this example, the agent follows the status assertion back to the returned
|
|
53
|
+
order. It finds no check that reads the total:
|
|
67
54
|
|
|
68
|
-
|
|
55
|
+
- `order.status` → must equal `'confirmed'`.
|
|
56
|
+
- `order.total` → no assertion checks the calculated total.
|
|
69
57
|
|
|
70
|
-
|
|
58
|
+
The agent adds the missing assertion to the existing test:
|
|
71
59
|
|
|
72
60
|
```js
|
|
73
|
-
|
|
74
|
-
return 4;
|
|
75
|
-
}
|
|
61
|
+
assert.equal(order.total, 50);
|
|
76
62
|
```
|
|
77
63
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
| Assertion | What it checks |
|
|
82
|
-
| --- | --- |
|
|
83
|
-
| `assert.equal(shippingCost(), 4)` | The cost is exactly `4`. Returning `5` would fail. |
|
|
84
|
-
| `assert.ok(shippingCost())` | The cost is truthy. Returning `5` would still pass. |
|
|
85
|
-
| `shippingCost(); assert.equal(7, 7)` | Nothing about the returned cost. The assertion compares constants. |
|
|
64
|
+
It reruns the full suite and updates the map. A total of `49` now fails the
|
|
65
|
+
test: two items at $25 must total $50. Line coverage has not changed, but the
|
|
66
|
+
test now checks the calculation.
|
|
86
67
|
|
|
87
|
-
|
|
88
|
-
change without failing a test when the new value still satisfies the assertion.
|
|
89
|
-
Read the mapped observation when deciding whether a test protects your change.
|
|
68
|
+
## Try it in your project
|
|
90
69
|
|
|
91
|
-
|
|
70
|
+
Open your project in your usual coding agent and paste this prompt. The agent
|
|
71
|
+
can install Supercov and run the commands for you.
|
|
92
72
|
|
|
93
|
-
```
|
|
94
|
-
npx supercov
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
73
|
+
```text supercov-prompt
|
|
74
|
+
Read npx supercov docs assertion-agent. Run the full test suite through
|
|
75
|
+
Supercov, then build and validate its assertion map. Find one useful
|
|
76
|
+
missing check and add an assertion for the expected behavior.
|
|
77
|
+
Only change tests; do not weaken existing checks. Rerun the same suite
|
|
78
|
+
and update the map. Show the test change, before-and-after assertion
|
|
79
|
+
coverage, and any uncertainty or missing evidence.
|
|
98
80
|
```
|
|
99
81
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
See [Investigating assertion evidence](assertion-evidence.md) when execution
|
|
104
|
-
or a passing assertion is missing.
|
|
82
|
+
The agent uses your actual test command after `--`, for example
|
|
83
|
+
`npx supercov -- npm test`. On later runs of the same command, Supercov reuses
|
|
84
|
+
compatible mappings and flags explanations that need another look.
|
|
105
85
|
|
|
106
|
-
|
|
107
|
-
crediting a message-sending statement that never ran. Its map should explain
|
|
108
|
-
what was observed, when observation ended and why that was sufficient.
|
|
86
|
+
## Read the result
|
|
109
87
|
|
|
110
|
-
|
|
88
|
+
The agent can show the summary and the statement-level report for a file:
|
|
89
|
+
|
|
90
|
+
```sh
|
|
91
|
+
npx supercov runs <run-id>
|
|
92
|
+
npx supercov runs <run-id> assertions report --view statements --file checkout.js
|
|
93
|
+
```
|
|
111
94
|
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
can affect all its flows. Your agent can update those parts instead of starting
|
|
116
|
-
over.
|
|
95
|
+
**Assertions** is the percentage of measured source statements linked to
|
|
96
|
+
passing assertions by the agent's map. It is not a count of assertions. Each
|
|
97
|
+
statement counts once; statements that never ran remain in the total.
|
|
117
98
|
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
longer matches, rerun tests before continuing the map.
|
|
99
|
+
An unmapped statement is a place to investigate, not proof of a missing test:
|
|
100
|
+
the agent may not have mapped its existing check yet.
|
|
121
101
|
|
|
122
|
-
|
|
102
|
+
The strength of the check still matters. `assert.ok(order.total)` accepts both
|
|
103
|
+
`49` and `50`; `assert.equal(order.total, 50)` distinguishes them. Review the
|
|
104
|
+
expected behavior, not just the percentage.
|
|
123
105
|
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
106
|
+
Supercov does not generate or execute mutated code for this assessment. Unlike
|
|
107
|
+
mutation testing, it does not test whether deliberately introduced bugs are
|
|
108
|
+
caught. The agent's explanations still need review.
|
|
127
109
|
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
the behavior you intend to preserve. See [Assertion map format](assertion-maps.md)
|
|
131
|
-
for editing and validation, or [Trusting results](verification.md) for the
|
|
132
|
-
broader test-review workflow.
|
|
110
|
+
For the detailed workflow, see [Mapping assertions with an agent](assertion-agent.md).
|
|
111
|
+
For a result you cannot explain, see [Investigating assertion evidence](assertion-evidence.md).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "supercov",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.47",
|
|
4
4
|
"description": "Coverage for coding agents and software factories \ud83c\udf19",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -94,14 +94,14 @@
|
|
|
94
94
|
"docs:check": "node scripts/sync-docs.mjs --check"
|
|
95
95
|
},
|
|
96
96
|
"optionalDependencies": {
|
|
97
|
-
"@supercov/cli-darwin-arm64": "0.0.
|
|
98
|
-
"@supercov/cli-darwin-x64": "0.0.
|
|
99
|
-
"@supercov/cli-linux-arm64-gnu": "0.0.
|
|
100
|
-
"@supercov/cli-linux-arm64-musl": "0.0.
|
|
101
|
-
"@supercov/cli-linux-x64-gnu": "0.0.
|
|
102
|
-
"@supercov/cli-linux-x64-musl": "0.0.
|
|
103
|
-
"@supercov/cli-win32-arm64": "0.0.
|
|
104
|
-
"@supercov/cli-win32-x64": "0.0.
|
|
97
|
+
"@supercov/cli-darwin-arm64": "0.0.47",
|
|
98
|
+
"@supercov/cli-darwin-x64": "0.0.47",
|
|
99
|
+
"@supercov/cli-linux-arm64-gnu": "0.0.47",
|
|
100
|
+
"@supercov/cli-linux-arm64-musl": "0.0.47",
|
|
101
|
+
"@supercov/cli-linux-x64-gnu": "0.0.47",
|
|
102
|
+
"@supercov/cli-linux-x64-musl": "0.0.47",
|
|
103
|
+
"@supercov/cli-win32-arm64": "0.0.47",
|
|
104
|
+
"@supercov/cli-win32-x64": "0.0.47"
|
|
105
105
|
},
|
|
106
106
|
"peerDependencies": {
|
|
107
107
|
"@playwright/test": ">=1.55.0",
|