supercov 0.0.44 → 0.0.45
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -12
- package/docs/agent-loop.md +12 -8
- package/docs/assertion-agent.md +156 -0
- package/docs/assertion-evidence.md +9 -694
- package/docs/assertion-maps.md +252 -0
- package/docs/assertions.md +82 -0
- package/docs/cli.md +23 -8
- package/docs/coverage-model.md +12 -0
- package/package.json +34 -35
- package/runtime/javascript/runtime.mjs +18 -33
- package/schemas/assertions.schema.json +276 -0
- package/analyzers/typescript/README.md +0 -59
- package/analyzers/typescript/bin/compiler-identity.mjs +0 -78
- package/analyzers/typescript/bin/identity.mjs +0 -71
- package/analyzers/typescript/bin/query.mjs +0 -29
- package/analyzers/typescript/dist/analyze.js +0 -5273
- package/analyzers/typescript/dist/archive.js +0 -337
- package/analyzers/typescript/dist/awaited-observations.js +0 -376
- package/analyzers/typescript/dist/build-identity.json +0 -1
- package/analyzers/typescript/dist/compiler.js +0 -32
- package/analyzers/typescript/dist/frontend.js +0 -75
- package/analyzers/typescript/dist/mock-counts.js +0 -2517
- package/analyzers/typescript/dist/native-frontend.js +0 -271
- package/analyzers/typescript/dist/pragmas.js +0 -186
- package/analyzers/typescript/dist/types.js +0 -1
- package/analyzers/typescript/package.json +0 -27
- package/analyzers/typescript/src/analyze.ts +0 -6180
- package/analyzers/typescript/src/archive.ts +0 -471
- package/analyzers/typescript/src/awaited-observations.ts +0 -561
- package/analyzers/typescript/src/compiler.ts +0 -49
- package/analyzers/typescript/src/frontend.ts +0 -136
- package/analyzers/typescript/src/mock-counts.ts +0 -3219
- package/analyzers/typescript/src/native-frontend.ts +0 -315
- package/analyzers/typescript/src/pragmas.ts +0 -284
- package/analyzers/typescript/src/types.ts +0 -45
- package/analyzers/typescript/tsconfig.json +0 -12
- package/docs/code-verification.md +0 -4
package/README.md
CHANGED
|
@@ -46,22 +46,30 @@ npx supercov -- python -m unittest
|
|
|
46
46
|
npx supercov -- bundle exec rspec
|
|
47
47
|
```
|
|
48
48
|
|
|
49
|
-
##
|
|
49
|
+
## Map what assertions check
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+
Each normal test run creates an assertion map automatically. An agent can
|
|
52
|
+
optionally fill in what the assertions check:
|
|
52
53
|
|
|
53
54
|
```sh
|
|
54
|
-
npx supercov runs latest assertions --
|
|
55
|
-
|
|
56
|
-
npx supercov runs
|
|
55
|
+
npx supercov runs latest assertions --json
|
|
56
|
+
# Pin the returned run ID. Inspect assertions and the source that ran:
|
|
57
|
+
npx supercov runs <run> assertion <assertion-id>
|
|
58
|
+
npx supercov runs <run> source src/example.ts
|
|
59
|
+
# Edit assertions.json, then validate and acknowledge:
|
|
60
|
+
npx supercov runs <run> assertions validate --json
|
|
61
|
+
# Copy examined expectedBasis tokens into assertions.json; save again.
|
|
62
|
+
npx supercov runs <run> assertions check --require-mappings --require-observed --json
|
|
63
|
+
npx supercov runs <run> # includes the assertion percentage in the regular report
|
|
57
64
|
```
|
|
58
65
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
[
|
|
64
|
-
assertion
|
|
66
|
+
After code or tests change, run the same test command again. Supercov reuses the
|
|
67
|
+
newest available map for that command and language and identifies dirty flows. Rust owns
|
|
68
|
+
validation, change tracking and reporting; the agent supplies semantic reasoning.
|
|
69
|
+
The score is agent-assessed and separate from MC/DC. See
|
|
70
|
+
[assertion maps](docs/assertion-maps.md) for the format and JS/TS limits, or run
|
|
71
|
+
`supercov docs assertion-agent` for the agent workflow. `supercov assertions schema`
|
|
72
|
+
exports the editor schema; `assertions validate --file <path>` checks JSON syntax.
|
|
65
73
|
|
|
66
74
|
## Give Supercov a job
|
|
67
75
|
|
|
@@ -170,7 +178,7 @@ Supercov uses exact per-test attribution where an adapter is available. For othe
|
|
|
170
178
|
| --- | --- |
|
|
171
179
|
| Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
|
|
172
180
|
| Vitest | Exact per test, with setup execution kept separate |
|
|
173
|
-
| Jest | Exact per test, including parameterized tests; `expect`
|
|
181
|
+
| Jest | Exact per test, including parameterized tests; exact `expect` occurrence identity for assertion maps |
|
|
174
182
|
| `node:test` | Exact per test |
|
|
175
183
|
| AVA and Mocha | Aggregate structural coverage |
|
|
176
184
|
| Cargo's standard libtest runner | Exact test, attempt, and passing-assertion identity |
|
|
@@ -223,6 +231,7 @@ npx supercov clean # remove all runs and the build cache
|
|
|
223
231
|
|
|
224
232
|
- [Getting started](https://supercov.com/docs/getting-started)
|
|
225
233
|
- [Agent workflow](https://supercov.com/docs/agent-loop)
|
|
234
|
+
- [Understanding assertions](docs/assertions.md)
|
|
226
235
|
- [Troubleshooting](https://supercov.com/docs/troubleshooting)
|
|
227
236
|
- [CLI reference](https://supercov.com/docs/cli)
|
|
228
237
|
- [Supported languages and test suites](https://supercov.com/docs/supported-suites)
|
package/docs/agent-loop.md
CHANGED
|
@@ -53,18 +53,22 @@ The `line` query is useful before writing a test because it shows which tests
|
|
|
53
53
|
already reach that line. Extending a nearby test is often better than adding a
|
|
54
54
|
duplicate.
|
|
55
55
|
|
|
56
|
-
|
|
56
|
+
Every new test run creates `assertions.json`. To investigate what tests assert:
|
|
57
57
|
|
|
58
58
|
```sh
|
|
59
|
-
npx supercov runs latest assertions --
|
|
60
|
-
|
|
59
|
+
npx supercov runs latest assertions --json
|
|
60
|
+
# Pin data.run and edit the file at data.map.
|
|
61
61
|
```
|
|
62
62
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
63
|
+
Inspect an entry with `runs <run> assertion <id>` and read archived code with
|
|
64
|
+
`runs <run> source <path>`. Edit its `assertions.json`, declare credited nodes
|
|
65
|
+
and broader watch inputs, then obtain expectedBasis tokens with `assertions validate --json` and save the examined tokens in the map or repeated `--flow ASSERTION/FLOW`. After the next run of
|
|
66
|
+
the same command, its map automatically inherits prior work; repair affected entries. Supercov
|
|
67
|
+
validates references and freshness, while the agent owns semantic meaning.
|
|
68
|
+
Run `supercov docs assertion-agent` for the full mapping instructions. Validate
|
|
69
|
+
with `runs <run> assertions validate`, save examined expectedBasis tokens in the map, then use
|
|
70
|
+
`runs <run> assertions check --require-mappings --require-observed` as the
|
|
71
|
+
completion gate. See [assertion maps](assertion-maps.md) for the contract and JS/TS limits.
|
|
68
72
|
|
|
69
73
|
## Example
|
|
70
74
|
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Agent workflow for assertion maps
|
|
2
|
+
|
|
3
|
+
You are the semantic author. Supercov supplies source identities, execution
|
|
4
|
+
records, file hashes and a JSON validator. It does not infer or prove your
|
|
5
|
+
explanations. Use your ordinary source-reading tools and edit one run-owned
|
|
6
|
+
`assertions.json`; no model is embedded in Supercov.
|
|
7
|
+
|
|
8
|
+
## Start with a real run
|
|
9
|
+
|
|
10
|
+
1. Run the intended suite through `supercov -- <test command>` if no matching
|
|
11
|
+
current run exists. Use the full suite for a suite-wide result.
|
|
12
|
+
2. Read `supercov runs --json`, pin its concrete run ID, then read
|
|
13
|
+
`supercov runs <run> assertions --json`. Edit `data.map`; the run creates it
|
|
14
|
+
automatically and inherits compatible prior work. Check `data.inheritance`
|
|
15
|
+
for fallback errors. Keep old runs and their maps.
|
|
16
|
+
3. Read `supercov docs assertion-maps`. Page the assertion, statement, test and
|
|
17
|
+
change views. Do not mistake one page for the whole suite:
|
|
18
|
+
|
|
19
|
+
```sh
|
|
20
|
+
supercov runs <run> assertions --limit 100 --json
|
|
21
|
+
supercov runs <run> assertions report --view statements --limit 100 --json
|
|
22
|
+
supercov runs <run> assertions report --view tests --limit 100 --json
|
|
23
|
+
supercov runs <run> assertions report --view changes --limit 100 --json
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Follow `pagination.nextOffset` with `--offset`; restart if `revision` changes.
|
|
27
|
+
Use a single map writer. Save edits atomically where your editor supports it.
|
|
28
|
+
The outer object contains `schemaVersion: 2` and `assertions`; do not insert run IDs.
|
|
29
|
+
|
|
30
|
+
## Investigate precise claims
|
|
31
|
+
|
|
32
|
+
Read current test setup, inputs, mocks, callbacks, branches, callees, helper
|
|
33
|
+
modules and relevant configuration. Current files must match the pinned run.
|
|
34
|
+
Supercov retains hashes and identities, not a source checkout. Optional
|
|
35
|
+
`runs <run> source <path> --offset 0 --limit 100` prints matching code with line
|
|
36
|
+
numbers; `assertions files` lists input hashes even when files no longer match.
|
|
37
|
+
|
|
38
|
+
Use `runs <run> assertion <id>` for the exact site and graph. Preserve its ID and
|
|
39
|
+
full `at` anchor, usually the complete assertion call without a trailing semicolon.
|
|
40
|
+
Use one-based UTF-8 byte columns, not character counts. An `inMap: false` site
|
|
41
|
+
was recognized but is missing from the map; restore it when investigating.
|
|
42
|
+
|
|
43
|
+
For each assertion:
|
|
44
|
+
|
|
45
|
+
- Describe exactly what its predicate distinguishes in `observes`. Truthiness,
|
|
46
|
+
existence, length or substring checks do not imply equality of every field.
|
|
47
|
+
- Write independently maintainable flows with stable IDs, `basis: null`, a
|
|
48
|
+
concrete explanation, exact nodes and authored edges to `$assertion`.
|
|
49
|
+
- Use explicit `appliesTo: [{file, name}]` from the test view. File and name
|
|
50
|
+
must resolve unambiguously. Empty means no credit. Runtime IDs never belong
|
|
51
|
+
in selectors. Shared/parameterized sites may need multiple cases or flows.
|
|
52
|
+
- Take production statement anchors from the `statements` view when possible.
|
|
53
|
+
Put only nodes you judge asserted in `countsAsAsserted`. Each counted node
|
|
54
|
+
needs an authored path to `$assertion`; a block does not credit nested code.
|
|
55
|
+
- Put additional dependency paths in `watch`, as whole-file strings. Assertion,
|
|
56
|
+
selected-test and node files are already dependencies. Include setup, guards,
|
|
57
|
+
alternate paths and helpers your explanation relies on, even without nodes.
|
|
58
|
+
- Preserve uncertainty in `questions`. Flow questions block that flow's credit.
|
|
59
|
+
Assertion questions record unfinished exploration without implying a known
|
|
60
|
+
total. There is no `analysis`, `mapped`, or `complete` flag.
|
|
61
|
+
- A fixture-only or absence explanation can have `countsAsAsserted: []`. Explain
|
|
62
|
+
the absence, including ordering/barriers and the observation window. Never
|
|
63
|
+
credit an unexecuted body just because its execution would violate a check.
|
|
64
|
+
|
|
65
|
+
Inspect `selectors`, `blockers`, `reasons`, empty `observedPassingTests`, and
|
|
66
|
+
statement `at: null` entries. They may represent skipped tests, unsupported
|
|
67
|
+
syntax, ambiguous test names or missing evidence. Do not borrow another site's
|
|
68
|
+
identity, fabricate events, delete inconvenient entries or edit managed state.
|
|
69
|
+
Zero-credit explanations are useful; invented credit is not.
|
|
70
|
+
|
|
71
|
+
## Account for changes before finalizing flow tokens
|
|
72
|
+
|
|
73
|
+
When inheriting a map, inspect **every** item in `--view changes`, including files
|
|
74
|
+
already watched by some flows. Dependencies may be missing. In root
|
|
75
|
+
`changeAssessments`, write one response per managed change ID:
|
|
76
|
+
|
|
77
|
+
```json
|
|
78
|
+
{
|
|
79
|
+
"id": "c_copy_from_changes_view",
|
|
80
|
+
"basis": null,
|
|
81
|
+
"affectedFlows": ["assertion-id/flow-id"],
|
|
82
|
+
"explanation": "What changed, which claims it affects, and why other existing claims remain valid."
|
|
83
|
+
}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Include all known dependent flows still present, plus any others affected.
|
|
87
|
+
Repair watches if the change reveals a missing dependency. An empty list needs
|
|
88
|
+
an actual explanation. Removing a response does not clear the outstanding change.
|
|
89
|
+
A changed test/assertion can affect all its flows; a production file used by one
|
|
90
|
+
sibling may affect only that flow. No machine can tell you an expected flow total.
|
|
91
|
+
|
|
92
|
+
Save the responses and graph edits. Run:
|
|
93
|
+
|
|
94
|
+
```sh
|
|
95
|
+
supercov assertions validate --file <map-path> --json
|
|
96
|
+
supercov runs <run> assertions validate --json
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
For large maps, page `validate --view changes`, `validate --view flows`, or
|
|
100
|
+
`validate --view errors` with `--offset` and `--limit`; tokens appear under
|
|
101
|
+
`items` and overall validity still covers the whole map. Pin `revision` across pages.
|
|
102
|
+
|
|
103
|
+
Validation checks shape, IDs, graph links, anchors and state binding. It returns
|
|
104
|
+
`changes[].expectedBasis` and `flows[].expectedBasis`; null/stale tokens alone
|
|
105
|
+
are unfinished work, not syntax errors. Copy change tokens **only after inspecting
|
|
106
|
+
those impact assessments**, then save. Validate again and copy flow tokens only
|
|
107
|
+
for claims you have examined. Save the map again. Change acknowledgements can
|
|
108
|
+
invalidate additional flows, so obtain final flow tokens after change tokens.
|
|
109
|
+
Changing a graph, selector, observation or watch after obtaining its token makes
|
|
110
|
+
that token stale. Do not implement the hashing algorithm or manufacture tokens.
|
|
111
|
+
|
|
112
|
+
There is no `review` command. All commands are read-only; acknowledgement is your
|
|
113
|
+
explicit file edit. Passing reference validation does not prove your reasoning.
|
|
114
|
+
|
|
115
|
+
## Check and report
|
|
116
|
+
|
|
117
|
+
```sh
|
|
118
|
+
supercov runs <run> assertions check --require-mappings --require-observed --json
|
|
119
|
+
supercov runs <run> assertions report --view statements --limit 100 --json
|
|
120
|
+
supercov runs <run>
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Basic `check` permits untouched sites without flows, but fails on invalid or
|
|
124
|
+
unfinished authored claims, open questions and outstanding change impact.
|
|
125
|
+
`--require-mappings` requires at least one current observed explanation for every
|
|
126
|
+
recognized passing site, including legitimate zero-credit explanations.
|
|
127
|
+
`--require-observed` checks every site and selector. Neither means every semantic
|
|
128
|
+
flow was found. Use `--min <percentage>` only for the project's chosen target.
|
|
129
|
+
Do not lower a target or hide unknowns to make a check pass.
|
|
130
|
+
|
|
131
|
+
The normal report includes the percentage immediately from the edited file.
|
|
132
|
+
`notAssessed`, `pending`, `unavailable` and `notApplicable` have null percentages,
|
|
133
|
+
not fabricated zeroes. A numeric partial-map result still reports counts of
|
|
134
|
+
assertions without flows and claims needing work. Describe the score as
|
|
135
|
+
agent-assessed statements; successful same-test coexecution is evidence, not
|
|
136
|
+
proof of causality or mutation resistance.
|
|
137
|
+
|
|
138
|
+
## Continue after edits
|
|
139
|
+
|
|
140
|
+
Run the same suite command again. The new run carries IDs, explanations and
|
|
141
|
+
unchanged acknowledgements using prior file hashes, without needing old source.
|
|
142
|
+
Use `assertions --needs-attention` and the change view. Repair only affected
|
|
143
|
+
claims; keep current siblings. Any dependency byte edit, including comments,
|
|
144
|
+
requires rechecking. Unique relocation preserves identity as a suggestion;
|
|
145
|
+
ambiguous/removed sites remain in `retiredAssertions`. Old v1 flows are imported
|
|
146
|
+
as drafts with questions about selectors and graph paths; inspect them explicitly.
|
|
147
|
+
|
|
148
|
+
Do not edit `assertions.state.json`. Dirty generations persist across reruns and
|
|
149
|
+
reverts until current tokens are recorded. Resolved change responses are folded
|
|
150
|
+
and retired on the next publication. Runtime evidence never carries forward.
|
|
151
|
+
If source differs from the pinned run, rerun tests before continuing.
|
|
152
|
+
|
|
153
|
+
Finish with the pinned run ID, status, numeric percentage/counts when available,
|
|
154
|
+
assertions without flows, remaining questions/stale or unobserved claims, and
|
|
155
|
+
verification commands used. Explain remaining limits. The property in `observes`
|
|
156
|
+
matters: changing a credited line while preserving that property can still pass.
|