supercov 0.0.45 → 0.0.46
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/docs/agent-loop.md +17 -13
- package/docs/assertion-agent.md +134 -144
- package/docs/assertion-evidence.md +103 -9
- package/docs/assertion-maps.md +188 -204
- package/docs/assertions.md +99 -49
- package/docs/cli.md +31 -14
- package/docs/coverage-model.md +6 -0
- package/docs/evidence.md +28 -0
- package/docs/performance.md +16 -0
- package/docs/supported-suites.md +64 -147
- package/package.json +14 -12
- package/runtime/javascript/launchSupervisor.mjs +5 -0
- package/runtime/javascript/nodeTest.mjs +57 -41
- package/runtime/javascript/provenance.mjs +4 -1
- package/runtime/javascript/register.mjs +9 -5
- package/runtime/javascript/runnerEvidence.mjs +4 -1
- package/runtime/javascript/runtime.mjs +44 -3
package/docs/assertion-maps.md
CHANGED
|
@@ -1,30 +1,26 @@
|
|
|
1
|
-
# Assertion
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
supercov -- npm test
|
|
13
|
-
supercov runs latest assertions --json
|
|
14
|
-
# Pin the returned run ID; edit data.map using matching current project files.
|
|
15
|
-
supercov runs <run> assertions report --view changes --json
|
|
16
|
-
supercov runs <run> assertions validate --json
|
|
17
|
-
# After investigating, copy expectedBasis tokens into the corresponding entries.
|
|
18
|
-
supercov runs <run> assertions check --require-mappings --require-observed --json
|
|
19
|
-
supercov runs <run>
|
|
1
|
+
# Assertion map format
|
|
2
|
+
|
|
3
|
+
Edit a run's `assertions.json` to record which assertions check which source
|
|
4
|
+
statements. Start with [Understanding assertion coverage](assertions.md) if you
|
|
5
|
+
want an agent to create the map for you. This reference covers the fields and
|
|
6
|
+
commands used to inspect, update and validate it.
|
|
7
|
+
|
|
8
|
+
## Open the run's map
|
|
9
|
+
|
|
10
|
+
```sh supercov
|
|
11
|
+
npx supercov runs latest assertions --json
|
|
20
12
|
```
|
|
21
13
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
particular claim. It is not proof, a signature, or a model confidence score.
|
|
14
|
+
The response gives you `data.run` and `data.map`. Use that run ID throughout the
|
|
15
|
+
edit. A normal run creates the file automatically and inherits compatible work
|
|
16
|
+
from earlier runs of the same test command and language.
|
|
26
17
|
|
|
27
|
-
|
|
18
|
+
Only edit `assertions.json`. Keep `assertions.state.json` and the run's evidence
|
|
19
|
+
unchanged. Use current project files that match the run when investigating.
|
|
20
|
+
|
|
21
|
+
## Describe an assertion and its flow
|
|
22
|
+
|
|
23
|
+
This illustrative draft maps an equality check to a return statement:
|
|
28
24
|
|
|
29
25
|
```json
|
|
30
26
|
{
|
|
@@ -32,7 +28,9 @@ particular claim. It is not proof, a signature, or a model confidence score.
|
|
|
32
28
|
"assertions": [{
|
|
33
29
|
"id": "a_example",
|
|
34
30
|
"at": {
|
|
35
|
-
"file": "tests/value.test.ts",
|
|
31
|
+
"file": "tests/value.test.ts",
|
|
32
|
+
"line": 5,
|
|
33
|
+
"column": 3,
|
|
36
34
|
"text": "assert.equal(value(), 1)"
|
|
37
35
|
},
|
|
38
36
|
"observes": ["The returned number equals one."],
|
|
@@ -40,10 +38,15 @@ particular claim. It is not proof, a signature, or a model confidence score.
|
|
|
40
38
|
"id": "return-value",
|
|
41
39
|
"basis": null,
|
|
42
40
|
"appliesTo": [{ "file": "tests/value.test.ts", "name": "value" }],
|
|
43
|
-
"explanation": "
|
|
41
|
+
"explanation": "value() returns the number compared by the assertion.",
|
|
44
42
|
"nodes": [{
|
|
45
43
|
"id": "return",
|
|
46
|
-
"at": {
|
|
44
|
+
"at": {
|
|
45
|
+
"file": "src/value.ts",
|
|
46
|
+
"line": 2,
|
|
47
|
+
"column": 3,
|
|
48
|
+
"text": "return 1;"
|
|
49
|
+
}
|
|
47
50
|
}],
|
|
48
51
|
"edges": [{ "from": "return", "to": "$assertion", "kind": "data" }],
|
|
49
52
|
"countsAsAsserted": ["return"],
|
|
@@ -53,156 +56,152 @@ particular claim. It is not proof, a signature, or a model confidence score.
|
|
|
53
56
|
}
|
|
54
57
|
```
|
|
55
58
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
types as the parser. The published editor schema is `schemas/assertions.schema.json`.
|
|
60
|
-
`supercov assertions validate --file <path> --json` checks JSON shape without a run.
|
|
59
|
+
Use the actual assertion IDs, test names and source locations from your run.
|
|
60
|
+
The example remains a draft until its references match and its flow has been
|
|
61
|
+
examined and acknowledged.
|
|
61
62
|
|
|
62
|
-
| Field |
|
|
63
|
+
| Field | What to write |
|
|
63
64
|
| --- | --- |
|
|
64
|
-
| `id`, `at` |
|
|
65
|
-
| `observes` |
|
|
66
|
-
| `flows` |
|
|
67
|
-
| `
|
|
68
|
-
| Flow `
|
|
69
|
-
|
|
|
70
|
-
| `
|
|
71
|
-
| `
|
|
72
|
-
| `
|
|
73
|
-
| `
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
Every counted node needs a path through the
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
##
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
the
|
|
104
|
-
|
|
105
|
-
|
|
65
|
+
| Assertion `id`, `at` | Preserve the assertion's ID and exact test expression. |
|
|
66
|
+
| `observes` | Describe the property checked, such as an exact value, length or substring. |
|
|
67
|
+
| `flows` | Explain the routes from relevant source to this assertion. An empty array means no explanations are recorded. |
|
|
68
|
+
| Flow `id` | Choose a stable name within the assertion, such as `return-value`. |
|
|
69
|
+
| Flow `basis` | Start with `null`. After review, copy the token returned by validation. |
|
|
70
|
+
| `appliesTo` | Select tests by project-relative file and exact displayed test name. |
|
|
71
|
+
| `nodes`, `edges` | Record source locations and relationships ending at `$assertion`. |
|
|
72
|
+
| `countsAsAsserted` | List the node IDs you judge to be checked by the assertion. |
|
|
73
|
+
| `watch` | List additional files the explanation depends on, such as helpers or configuration. |
|
|
74
|
+
| `questions` | Record unresolved investigation questions. Questions inside a flow block its credit. |
|
|
75
|
+
|
|
76
|
+
Source anchors use project-relative paths with `/`, one-based lines and one-based
|
|
77
|
+
UTF-8 byte columns. Preserve exact text, including multiline expressions. IDs
|
|
78
|
+
use letters, digits, `_`, `-` or `.`. Edge `basis` is optional explanatory text;
|
|
79
|
+
it is separate from the flow's review token.
|
|
80
|
+
|
|
81
|
+
Every counted node needs a path through the recorded edges to `$assertion` and
|
|
82
|
+
must match a measured statement exactly. Counting an `if` statement does not
|
|
83
|
+
also count every statement inside it. Nodes that only provide context can stay
|
|
84
|
+
in the graph without appearing in `countsAsAsserted`.
|
|
85
|
+
|
|
86
|
+
Test selectors must be unambiguous. An empty `appliesTo` supplies no execution
|
|
87
|
+
credit. For a shared assertion, select the applicable test cases explicitly.
|
|
88
|
+
An absence or fixture-only check can use `countsAsAsserted: []`; explain what
|
|
89
|
+
was observed and, for absence, the ordering and observation window.
|
|
90
|
+
|
|
91
|
+
## Validate and acknowledge your edits
|
|
92
|
+
|
|
93
|
+
```sh supercov-example
|
|
94
|
+
npx supercov assertions schema --json
|
|
95
|
+
npx supercov assertions validate --file <map-path> --json
|
|
96
|
+
npx supercov runs <run-id> assertions validate --json
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
The first command exports the editor schema. File validation checks JSON shape.
|
|
100
|
+
Run validation also checks IDs, source anchors, graph links, selected tests and
|
|
101
|
+
changed inputs. It does not edit the map or prove the explanation.
|
|
102
|
+
|
|
103
|
+
After examining a flow, copy its returned `expectedBasis` into that flow's
|
|
104
|
+
`basis`, save the file, and run `check`. Treat the token as an opaque value;
|
|
105
|
+
do not generate it yourself. Editing a claim or its dependencies makes the old
|
|
106
|
+
token stale.
|
|
107
|
+
|
|
108
|
+
```sh supercov-example
|
|
109
|
+
npx supercov runs <run-id> assertions check --require-mappings
|
|
110
|
+
npx supercov runs <run-id>
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
| Check option | Requirement |
|
|
106
114
|
| --- | --- |
|
|
107
|
-
|
|
|
108
|
-
| `
|
|
109
|
-
| `
|
|
110
|
-
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
115
|
+
| No extra option | Authored claims and references are valid and current, questions and change assessments are resolved, and the run passed. Untouched assertions without flows are allowed. |
|
|
116
|
+
| `--require-mappings` | Every recognized assertion observed passing has a current explanation, including legitimate zero-credit explanations. |
|
|
117
|
+
| `--require-observed` | Every listed assertion and explicit test selector has matching passing evidence. |
|
|
118
|
+
| `--min <percentage>` | The available assertion percentage meets your target. |
|
|
119
|
+
|
|
120
|
+
Use `--require-observed` when every mapped site is expected to run. Skipped,
|
|
121
|
+
TODO and untaken cases can make it fail even when their explanations are useful.
|
|
122
|
+
Neither gate establishes that every possible flow has been found.
|
|
123
|
+
|
|
124
|
+
`validate` exits with code 2 for syntax or reference errors. A null or stale
|
|
125
|
+
review token alone is not a syntax error. `check` exits with code 2 when a
|
|
126
|
+
requested requirement is unmet.
|
|
127
|
+
|
|
128
|
+
## Understand statement credit
|
|
129
|
+
|
|
130
|
+
The percentage counts the union of explicitly claimed measured statements with
|
|
131
|
+
current flows, a passing assertion and execution in the same selected passing
|
|
132
|
+
test. Duplicate claims count once. Unexecuted statements remain in the
|
|
133
|
+
denominator. A line receives assertion credit only when all measured statements
|
|
134
|
+
on it receive credit.
|
|
135
|
+
|
|
136
|
+
TypeScript imports known to disappear during compilation are excluded. Inspect
|
|
137
|
+
their locations with:
|
|
138
|
+
|
|
139
|
+
```sh supercov-example
|
|
140
|
+
npx supercov runs <run-id> assertions report --view excludedStatements
|
|
120
141
|
```
|
|
121
142
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
the
|
|
143
|
+
This includes explicit `import type` and, for supported tsc/tsx/ts-node settings,
|
|
144
|
+
imports used only as types. Runtime, mixed and side-effect imports remain.
|
|
145
|
+
Preserving compiler settings and configurations that Supercov cannot resolve
|
|
146
|
+
keep ambiguous imports in the denominator. Older runs keep their recorded totals.
|
|
147
|
+
|
|
148
|
+
An assertion detail explains each node's credit decision. In JSON, computed
|
|
149
|
+
`nodeCredit` entries include the decision, reason codes, matched statement IDs
|
|
150
|
+
and matching tests. Keep these report fields out of the editable map. See
|
|
151
|
+
[Investigating assertion evidence](assertion-evidence.md) for missing execution,
|
|
152
|
+
shared setup and asynchronous cases.
|
|
129
153
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
it describes the whole run, as its label says. MC/DC remains a separate metric.
|
|
135
|
-
`revision` binds each response to the map, managed state and run evidence.
|
|
154
|
+
A numeric result can come from a partial map. `current` means the flow has no
|
|
155
|
+
freshness, reference or question blockers; it does not mean all source nodes
|
|
156
|
+
receive credit or every relevant flow is known. The normal report shows counts
|
|
157
|
+
of assertions without flows and flows still needing attention beside the score.
|
|
136
158
|
|
|
137
|
-
|
|
159
|
+
JSON regular reports put the summary at `data.assertionCoverage.summary`.
|
|
160
|
+
Assertion reports use `data.summary`. A `summary.status` of `available` supplies
|
|
161
|
+
a percentage; `notAssessed`, `pending`, `unavailable` and `notApplicable` do not.
|
|
162
|
+
The assertion summary always covers the whole run, even when other coverage
|
|
163
|
+
metrics or the returned items are filtered.
|
|
138
164
|
|
|
139
|
-
|
|
165
|
+
## Find the next part to investigate
|
|
166
|
+
|
|
167
|
+
| Command after `supercov runs <run-id>` | Shows |
|
|
140
168
|
| --- | --- |
|
|
141
|
-
| `
|
|
142
|
-
| `
|
|
143
|
-
| `
|
|
144
|
-
| `
|
|
145
|
-
| `
|
|
146
|
-
| `
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
usable map for that exact command and language. A malformed newer candidate is
|
|
178
|
-
reported under `inheritance.skipped`; fallback claims require fresh acknowledgement.
|
|
179
|
-
Publication writes evidence, map and managed state together atomically. Previous
|
|
180
|
-
runs stay untouched. Finish editing the newest map before starting its successor.
|
|
181
|
-
|
|
182
|
-
The run stores a hash manifest and assertion identities, not complete source
|
|
183
|
-
files. Current project files supply the source for investigation. Managed
|
|
184
|
-
`assertions.state.json` (version 3) binds evidence and input identities and stores
|
|
185
|
-
invalidation generations, outstanding change records and inheritance metadata.
|
|
186
|
-
It contains no second semantic graph. Only `assertions.json` is agent-editable.
|
|
187
|
-
No query changes either file.
|
|
188
|
-
|
|
189
|
-
A flow depends on the assertion file, each selected test file, every node file,
|
|
190
|
-
and extra `watch` files. Any byte change in those files, including comments or
|
|
191
|
-
blank lines, invalidates that flow. Other sibling flows can remain current.
|
|
192
|
-
Changes to the assertion/observation affect all its flows. Context/dependency/
|
|
193
|
-
instrumenter changes invalidate inherited claims conservatively. Flow tokens
|
|
194
|
-
bind the claim (excluding their own token), context, sorted dependency hashes,
|
|
195
|
-
and effective invalidation generation; they exclude run IDs and runtime events.
|
|
196
|
-
|
|
197
|
-
Exact identities retain IDs. Unique snippets or unique identical-file renames
|
|
198
|
-
can relocate as suggestions. A sole unmatched old/new assertion in the same file
|
|
199
|
-
can retain its ID as a changed candidate. Ambiguous/removed assertions remain
|
|
200
|
-
under `retiredAssertions` with explanations. Unresolved changes stay stale across
|
|
201
|
-
reruns and reverts until acknowledged. Every new run supplies fresh evidence.
|
|
202
|
-
|
|
203
|
-
Every added, edited or removed file in the captured analysis scope enters the
|
|
204
|
-
change queue, even if a flow already watches it. Known dependencies are a starting
|
|
205
|
-
point, not proof that other flows are unaffected. Read `--view changes` and edit:
|
|
169
|
+
| `assertions` | Assertions, including those without recorded flows. |
|
|
170
|
+
| `assertions --needs-attention` | Missing explanations, questions, stale or draft flows, and claimed nodes without credit. |
|
|
171
|
+
| `assertion <id>` | One assertion, its flows and node-credit reasons. |
|
|
172
|
+
| `source <path>` | Matching current code with line numbers. |
|
|
173
|
+
| `assertions files` | Captured input paths, sizes and hashes, even if the checkout is stale. |
|
|
174
|
+
| `assertions report --view <view>` | `summary`, `assertions`, `statements`, `tests`, `changes`, `creditedLines`, `unassertedLines` or `excludedStatements`. |
|
|
175
|
+
|
|
176
|
+
Use `--file <path>` to filter applicable list views; the summary still describes
|
|
177
|
+
the whole run. Lists support `--offset`, `--limit` and `--json`. Follow the
|
|
178
|
+
printed next-page command or JSON `pagination.nextOffset` until it is null.
|
|
179
|
+
Restart a paged read if `revision` changes while you are reading it.
|
|
180
|
+
|
|
181
|
+
An assertion detail pages whole flows by default. For a large individual flow,
|
|
182
|
+
use `--flow <flow-id> --view nodes` and `--view edges`. Add `--compact` to omit
|
|
183
|
+
repeated source text while retaining locations. Read that text with `source`;
|
|
184
|
+
do not save compact report objects into the map.
|
|
185
|
+
|
|
186
|
+
Large validation results support `--view flows`, `--view changes` or
|
|
187
|
+
`--view errors`. Each page still reports validity for the entire map. The source
|
|
188
|
+
command pages lines; JSON source output uses `{line, text}` objects for integrations.
|
|
189
|
+
|
|
190
|
+
## Update the map after a change
|
|
191
|
+
|
|
192
|
+
Run the same test command again, then edit the new run's map. Supercov carries
|
|
193
|
+
forward compatible mappings and leaves the previous run unchanged. If a newer
|
|
194
|
+
map cannot be reused, `inheritance.skipped` explains the fallback.
|
|
195
|
+
|
|
196
|
+
Each flow depends on its assertion file, selected test files, node files and
|
|
197
|
+
extra `watch` files. A change to any of those files requires another look at that
|
|
198
|
+
flow, even if only a comment changed. Independent sibling flows can stay current.
|
|
199
|
+
Changing the assertion or its observation affects all its flows. Dependency,
|
|
200
|
+
configuration and instrumentation changes can affect many flows.
|
|
201
|
+
|
|
202
|
+
Start with `assertions report --view changes`. Investigate every listed change,
|
|
203
|
+
including effects on flows that did not yet watch the changed file. Add a
|
|
204
|
+
response in the map's top-level `changeAssessments` array:
|
|
206
205
|
|
|
207
206
|
```json
|
|
208
207
|
{
|
|
@@ -210,43 +209,28 @@ point, not proof that other flows are unaffected. Read `--view changes` and edit
|
|
|
210
209
|
"id": "c_copy_from_changes_view",
|
|
211
210
|
"basis": null,
|
|
212
211
|
"affectedFlows": ["a_example/return-value"],
|
|
213
|
-
"explanation": "The edit affects the returned value
|
|
212
|
+
"explanation": "The edit affects the returned value. The independent sibling calculation is unchanged."
|
|
214
213
|
}]
|
|
215
214
|
}
|
|
216
215
|
```
|
|
217
216
|
|
|
218
|
-
This is an excerpt
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
The JS/TS verification matrix covers Node ESM/CommonJS/native TypeScript, Vitest
|
|
240
|
-
TypeScript, Jest CommonJS and Playwright's Node-side TypeScript assertions,
|
|
241
|
-
including aliases, shared parameterized sites and async rejection matchers.
|
|
242
|
-
Unsupported/custom sites can be documented, but JS/TS credit requires the exact
|
|
243
|
-
recognized expression and operation in runtime evidence. This does not infer
|
|
244
|
-
browser assertion identity or data flows through external services.
|
|
245
|
-
|
|
246
|
-
The map is an agent assessment. A credited statement can be changed without
|
|
247
|
-
failing a test when the change preserves the observed property. Neither 100%
|
|
248
|
-
assertion coverage nor 100% MC/DC proves mutation resistance or safety for every
|
|
249
|
-
change. Use the observation and graph to select tests and identify missing
|
|
250
|
-
checks; sample mutation audits can independently evaluate the authored claims.
|
|
251
|
-
See [the agent workflow](assertion-agent.md), [assertion evidence](assertion-evidence.md),
|
|
252
|
-
and [verification](verification.md).
|
|
217
|
+
This is an excerpt to add to your existing map. Include every entry in `knownFlows`
|
|
218
|
+
that still exists and any additional affected flows. An empty list needs an
|
|
219
|
+
explanation of why existing claims are unaffected; it does not mean the changed
|
|
220
|
+
code is tested.
|
|
221
|
+
|
|
222
|
+
Save the assessments and graph edits, validate, and copy the examined change
|
|
223
|
+
tokens. Save again, then validate once more before copying final flow tokens.
|
|
224
|
+
Assessing a change can make another flow stale, so the order matters. Removing
|
|
225
|
+
an assessment or reverting a file does not by itself clear an unresolved change.
|
|
226
|
+
|
|
227
|
+
Keep removed or ambiguous assertion records for review under `retiredAssertions`.
|
|
228
|
+
Follow the supplied questions when inheriting older maps. Do not invent source
|
|
229
|
+
links or evidence to clear a warning.
|
|
230
|
+
|
|
231
|
+
## Keep queries fast
|
|
232
|
+
|
|
233
|
+
Reports reuse a disposable assessment cache while checking that sources still
|
|
234
|
+
match. Editing the map invalidates the cached assessment automatically. Queries
|
|
235
|
+
never acknowledge flows or change the authored map. See [Speed and storage](performance.md)
|
|
236
|
+
for keeping repeated investigation fast.
|