supercov 0.0.43 → 0.0.45
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +23 -14
- package/docs/agent-loop.md +128 -39
- package/docs/assertion-agent.md +156 -0
- package/docs/assertion-evidence.md +9 -135
- package/docs/assertion-maps.md +252 -0
- package/docs/assertions.md +82 -0
- package/docs/cli.md +38 -23
- package/docs/coverage-model.md +18 -6
- package/docs/evidence.md +6 -6
- package/docs/getting-started.md +60 -72
- package/docs/performance.md +4 -4
- package/docs/troubleshooting.md +8 -8
- package/docs/verification.md +2 -2
- package/docs/workspace-isolation.md +1 -1
- package/package.json +34 -33
- package/runtime/javascript/nodeAssertAdapter.mjs +32 -8
- package/runtime/javascript/nodeTest.mjs +13 -5
- package/runtime/javascript/runnerEvidence.mjs +33 -11
- package/runtime/javascript/runtime.mjs +27 -23
- package/schemas/assertions.schema.json +276 -0
- package/analyzers/typescript/README.md +0 -55
- package/analyzers/typescript/bin/compiler-identity.mjs +0 -78
- package/analyzers/typescript/bin/identity.mjs +0 -67
- package/analyzers/typescript/bin/query.mjs +0 -29
- package/analyzers/typescript/dist/analyze.js +0 -3972
- package/analyzers/typescript/dist/archive.js +0 -309
- package/analyzers/typescript/dist/build-identity.json +0 -1
- package/analyzers/typescript/dist/compiler.js +0 -32
- package/analyzers/typescript/dist/frontend.js +0 -75
- package/analyzers/typescript/dist/native-frontend.js +0 -271
- package/analyzers/typescript/dist/pragmas.js +0 -143
- package/analyzers/typescript/dist/types.js +0 -1
- package/analyzers/typescript/package.json +0 -27
- package/analyzers/typescript/src/analyze.ts +0 -4538
- package/analyzers/typescript/src/archive.ts +0 -438
- package/analyzers/typescript/src/compiler.ts +0 -49
- package/analyzers/typescript/src/frontend.ts +0 -136
- package/analyzers/typescript/src/native-frontend.ts +0 -315
- package/analyzers/typescript/src/pragmas.ts +0 -218
- package/analyzers/typescript/src/types.ts +0 -45
- package/analyzers/typescript/tsconfig.json +0 -12
- package/docs/code-verification.md +0 -182
package/README.md
CHANGED
|
@@ -10,8 +10,8 @@ No account, config file, import, custom reporter, or hosted service is required.
|
|
|
10
10
|
|
|
11
11
|
Supported by [Supercorp](https://supercorp.ai).
|
|
12
12
|
|
|
13
|
-
[
|
|
14
|
-
|
|
13
|
+
[Agent workflow](https://supercov.com/docs/agent-loop): ask your coding agent
|
|
14
|
+
to add a test in your own project, with a recorded example to follow along.
|
|
15
15
|
|
|
16
16
|
## Start with the suite you already have
|
|
17
17
|
|
|
@@ -46,22 +46,30 @@ npx supercov -- python -m unittest
|
|
|
46
46
|
npx supercov -- bundle exec rspec
|
|
47
47
|
```
|
|
48
48
|
|
|
49
|
-
##
|
|
49
|
+
## Map what assertions check
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+
Each normal test run creates an assertion map automatically. An agent can
|
|
52
|
+
optionally fill in what the assertions check:
|
|
52
53
|
|
|
53
54
|
```sh
|
|
54
|
-
npx supercov runs latest assertions --
|
|
55
|
-
|
|
56
|
-
npx supercov runs
|
|
55
|
+
npx supercov runs latest assertions --json
|
|
56
|
+
# Pin the returned run ID. Inspect assertions and the source that ran:
|
|
57
|
+
npx supercov runs <run> assertion <assertion-id>
|
|
58
|
+
npx supercov runs <run> source src/example.ts
|
|
59
|
+
# Edit assertions.json, then validate and acknowledge:
|
|
60
|
+
npx supercov runs <run> assertions validate --json
|
|
61
|
+
# Copy examined expectedBasis tokens into assertions.json; save again.
|
|
62
|
+
npx supercov runs <run> assertions check --require-mappings --require-observed --json
|
|
63
|
+
npx supercov runs <run> # includes the assertion percentage in the regular report
|
|
57
64
|
```
|
|
58
65
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
[
|
|
64
|
-
assertion
|
|
66
|
+
After code or tests change, run the same test command again. Supercov reuses the
|
|
67
|
+
newest available map for that command and language and identifies dirty flows. Rust owns
|
|
68
|
+
validation, change tracking and reporting; the agent supplies semantic reasoning.
|
|
69
|
+
The score is agent-assessed and separate from MC/DC. See
|
|
70
|
+
[assertion maps](docs/assertion-maps.md) for the format and JS/TS limits, or run
|
|
71
|
+
`supercov docs assertion-agent` for the agent workflow. `supercov assertions schema`
|
|
72
|
+
exports the editor schema; `assertions validate --file <path>` checks JSON syntax.
|
|
65
73
|
|
|
66
74
|
## Give Supercov a job
|
|
67
75
|
|
|
@@ -170,7 +178,7 @@ Supercov uses exact per-test attribution where an adapter is available. For othe
|
|
|
170
178
|
| --- | --- |
|
|
171
179
|
| Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
|
|
172
180
|
| Vitest | Exact per test, with setup execution kept separate |
|
|
173
|
-
| Jest | Exact per test, including parameterized tests; `expect`
|
|
181
|
+
| Jest | Exact per test, including parameterized tests; exact `expect` occurrence identity for assertion maps |
|
|
174
182
|
| `node:test` | Exact per test |
|
|
175
183
|
| AVA and Mocha | Aggregate structural coverage |
|
|
176
184
|
| Cargo's standard libtest runner | Exact test, attempt, and passing-assertion identity |
|
|
@@ -223,6 +231,7 @@ npx supercov clean # remove all runs and the build cache
|
|
|
223
231
|
|
|
224
232
|
- [Getting started](https://supercov.com/docs/getting-started)
|
|
225
233
|
- [Agent workflow](https://supercov.com/docs/agent-loop)
|
|
234
|
+
- [Understanding assertions](docs/assertions.md)
|
|
226
235
|
- [Troubleshooting](https://supercov.com/docs/troubleshooting)
|
|
227
236
|
- [CLI reference](https://supercov.com/docs/cli)
|
|
228
237
|
- [Supported languages and test suites](https://supercov.com/docs/supported-suites)
|
package/docs/agent-loop.md
CHANGED
|
@@ -1,41 +1,33 @@
|
|
|
1
1
|
# Agent workflow
|
|
2
2
|
|
|
3
|
-
Supercov
|
|
4
|
-
|
|
3
|
+
Use Supercov with your coding agent and the test suite you already have.
|
|
4
|
+
Supercov reports coverage and gaps. Your agent writes a test, reruns the suite,
|
|
5
|
+
and checks what improved.
|
|
5
6
|
|
|
6
|
-
|
|
7
|
-
run the suite → choose a gap → write one test → rerun → compare
|
|
8
|
-
↑ |
|
|
9
|
-
└────────────────────────────────────────────────────────────┘
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
Supercov supplies the coverage signal and evidence. Your coding agent writes
|
|
13
|
-
the tests.
|
|
7
|
+
## Start with one test
|
|
14
8
|
|
|
15
|
-
|
|
9
|
+
Open your own repository in your coding agent and paste this prompt. You don't
|
|
10
|
+
need to install Supercov first; the agent can handle that.
|
|
16
11
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
on coverage. Only edit tests. Rerun the complete suite and report what improved.
|
|
12
|
+
```text supercov-prompt
|
|
13
|
+
Measure code coverage with npx supercov and write one missing test.
|
|
14
|
+
Only change tests. Rerun the full test suite and show me the test you
|
|
15
|
+
added and the before-and-after coverage.
|
|
22
16
|
```
|
|
23
17
|
|
|
24
|
-
|
|
18
|
+
If the project has several test commands, tell the agent which full suite to
|
|
19
|
+
use. Let it run the commands and edit the tests, approving those actions if
|
|
20
|
+
your agent asks.
|
|
25
21
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
useful gaps remain. Never weaken assertions or change application code to make
|
|
29
|
-
coverage easier. Stop at a measurement limit, unreachable behavior, or the end
|
|
30
|
-
of the available time budget. Report the run ids compared and what improved.
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
The second prompt is intentionally open-ended, but 100% is a direction rather
|
|
34
|
-
than permission to write meaningless tests or reshape application code.
|
|
22
|
+
The result is a normal test-file change and a coverage comparison in the
|
|
23
|
+
conversation. Ask separately if you want a commit or pull request.
|
|
35
24
|
|
|
36
25
|
## One safe pass
|
|
37
26
|
|
|
38
|
-
|
|
27
|
+
The agent should run the suite, inspect a gap, write a test, then rerun the
|
|
28
|
+
same suite and compare. These are the commands it can use:
|
|
29
|
+
|
|
30
|
+
```sh supercov-example
|
|
39
31
|
# 1. Establish a baseline.
|
|
40
32
|
npx supercov -- npm test
|
|
41
33
|
|
|
@@ -52,29 +44,126 @@ npx supercov -- npm test
|
|
|
52
44
|
npx supercov diff <previous-run-id> latest
|
|
53
45
|
```
|
|
54
46
|
|
|
55
|
-
|
|
56
|
-
and
|
|
47
|
+
Everything after `--` is your project's test command. Use your actual command
|
|
48
|
+
and file paths in place of the examples. For example,
|
|
49
|
+
Rust projects can use `cargo test`, Python projects `pytest`, and Ruby projects
|
|
50
|
+
`bundle exec rspec`. Keep the baseline and verification commands identical.
|
|
57
51
|
|
|
58
52
|
The `line` query is useful before writing a test because it shows which tests
|
|
59
53
|
already reach that line. Extending a nearby test is often better than adding a
|
|
60
54
|
duplicate.
|
|
61
55
|
|
|
62
|
-
|
|
56
|
+
Every new test run creates `assertions.json`. To investigate what tests assert:
|
|
63
57
|
|
|
64
58
|
```sh
|
|
65
|
-
npx supercov runs latest assertions --
|
|
66
|
-
|
|
59
|
+
npx supercov runs latest assertions --json
|
|
60
|
+
# Pin data.run and edit the file at data.map.
|
|
67
61
|
```
|
|
68
62
|
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
63
|
+
Inspect an entry with `runs <run> assertion <id>` and read archived code with
|
|
64
|
+
`runs <run> source <path>`. Edit its `assertions.json`, declare credited nodes
|
|
65
|
+
and broader watch inputs, then obtain expectedBasis tokens with `assertions validate --json` and save the examined tokens in the map or repeated `--flow ASSERTION/FLOW`. After the next run of
|
|
66
|
+
the same command, its map automatically inherits prior work; repair affected entries. Supercov
|
|
67
|
+
validates references and freshness, while the agent owns semantic meaning.
|
|
68
|
+
Run `supercov docs assertion-agent` for the full mapping instructions. Validate
|
|
69
|
+
with `runs <run> assertions validate`, save examined expectedBasis tokens in the map, then use
|
|
70
|
+
`runs <run> assertions check --require-mappings --require-observed` as the
|
|
71
|
+
completion gate. See [assertion maps](assertion-maps.md) for the contract and JS/TS limits.
|
|
72
|
+
|
|
73
|
+
## Example
|
|
74
|
+
|
|
75
|
+
Here's a recorded Codex run in a JavaScript project, using the first prompt.
|
|
76
|
+
The files are from our checkout example; you don't need to add them to your
|
|
77
|
+
project.
|
|
78
|
+
|
|
79
|
+
[`src/session.js`](https://github.com/supercorp-ai/supercov/blob/main/examples/checkout-verification/starter/src/session.js)
|
|
80
|
+
allows checkout only when the customer is signed in and their session has
|
|
81
|
+
not expired:
|
|
82
|
+
|
|
83
|
+
```js
|
|
84
|
+
export function canCheckout(signedIn, expired) {
|
|
85
|
+
if (signedIn && !expired) return true;
|
|
86
|
+
return false;
|
|
87
|
+
}
|
|
88
|
+
```
|
|
74
89
|
|
|
75
|
-
|
|
90
|
+
The two tests in
|
|
91
|
+
[`tests/session.test.js`](https://github.com/supercorp-ai/supercov/blob/main/examples/checkout-verification/starter/tests/session.test.js)
|
|
92
|
+
check a valid session and a signed-out visitor:
|
|
93
|
+
|
|
94
|
+
```js
|
|
95
|
+
assert.equal(canCheckout(true, false), true);
|
|
96
|
+
assert.equal(canCheckout(false, false), false);
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
The agent ran `npx supercov -- npm test`. Both tests passed, and the summary
|
|
100
|
+
from `npx supercov runs latest` showed:
|
|
76
101
|
|
|
77
102
|
```text
|
|
103
|
+
Coverage
|
|
104
|
+
Lines 100.00% (3/3)
|
|
105
|
+
Branches 100.00% (2/2)
|
|
106
|
+
MC/DC 50.00% (1/2)
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
It listed the gaps and inspected the file. You can open those views with:
|
|
110
|
+
|
|
111
|
+
```sh
|
|
112
|
+
npx supercov runs latest gaps
|
|
113
|
+
npx supercov runs latest file src/session.js
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
The file query explained the gap:
|
|
117
|
+
|
|
118
|
+
```text
|
|
119
|
+
LINE STATUS SOURCE
|
|
120
|
+
2 PARTIAL signedIn && !expired
|
|
121
|
+
Unobserved: no witness pair shows `!expired` independently changing the decision result
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Both return paths had run, but neither test checked an expired session. MC/DC
|
|
125
|
+
checks whether each condition has been shown to affect the decision
|
|
126
|
+
independently. Here, `signedIn` had; `!expired` had not.
|
|
127
|
+
|
|
128
|
+
The agent added one test to `tests/session.test.js`, leaving the application
|
|
129
|
+
code and existing tests unchanged:
|
|
130
|
+
|
|
131
|
+
```js
|
|
132
|
+
test('a signed-in visitor with an expired session cannot check out', () => {
|
|
133
|
+
assert.equal(canCheckout(true, true), false);
|
|
134
|
+
});
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
It reran the same full suite. All three tests passed, and MC/DC reached 100%.
|
|
138
|
+
The comparison from `npx supercov diff <before-run-id> latest` showed:
|
|
139
|
+
|
|
140
|
+
```text
|
|
141
|
+
lines +0pp, branches +0pp, MC/DC +50pp
|
|
142
|
+
gained: 0 lines, 0 branches, 1 MC/DC conditions
|
|
143
|
+
lost: 0 lines, 0 branches, 0 MC/DC conditions
|
|
144
|
+
+ MC/DC src/session.js:2 C2 !expired
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
The new assertion checks that checkout is denied when a signed-in customer's
|
|
148
|
+
session has expired. Removing the expiry check makes this test fail; the
|
|
149
|
+
original two tests still pass.
|
|
150
|
+
|
|
151
|
+
In your project, look for the same evidence: the test checks the behavior the
|
|
152
|
+
agent identified, and the full suite passes. One useful test won't necessarily
|
|
153
|
+
take coverage to 100%.
|
|
154
|
+
|
|
155
|
+
To try these exact files, [download the starter](https://supercov.com/downloads/supercov-tutorial.zip),
|
|
156
|
+
extract it, open the `supercov-tutorial` folder in your agent, and run `npm ci`.
|
|
157
|
+
Then use the JavaScript prompt above. The completed test is not included in the
|
|
158
|
+
download. The [recorded run](https://github.com/supercorp-ai/supercov/tree/main/examples/checkout-verification/agent-run)
|
|
159
|
+
includes the commands, full output, and completed test.
|
|
160
|
+
|
|
161
|
+
## A complete prompt for longer runs
|
|
162
|
+
|
|
163
|
+
Once you've reviewed the first test, use this prompt to continue through
|
|
164
|
+
useful gaps—for example, during an overnight run:
|
|
165
|
+
|
|
166
|
+
```text supercov-prompt
|
|
78
167
|
Use `npx supercov` to improve coverage. Only write tests. Keep going while
|
|
79
168
|
useful gaps remain.
|
|
80
169
|
|
|
@@ -110,7 +199,7 @@ reason to manufacture a test.
|
|
|
110
199
|
|
|
111
200
|
If the repository separates test levels, narrow the view:
|
|
112
201
|
|
|
113
|
-
```sh
|
|
202
|
+
```sh supercov
|
|
114
203
|
npx supercov runs latest gaps --kind e2e --limit 10
|
|
115
204
|
```
|
|
116
205
|
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Agent workflow for assertion maps
|
|
2
|
+
|
|
3
|
+
You are the semantic author. Supercov supplies source identities, execution
|
|
4
|
+
records, file hashes and a JSON validator. It does not infer or prove your
|
|
5
|
+
explanations. Use your ordinary source-reading tools and edit one run-owned
|
|
6
|
+
`assertions.json`; no model is embedded in Supercov.
|
|
7
|
+
|
|
8
|
+
## Start with a real run
|
|
9
|
+
|
|
10
|
+
1. Run the intended suite through `supercov -- <test command>` if no matching
|
|
11
|
+
current run exists. Use the full suite for a suite-wide result.
|
|
12
|
+
2. Read `supercov runs --json`, pin its concrete run ID, then read
|
|
13
|
+
`supercov runs <run> assertions --json`. Edit `data.map`; the run creates it
|
|
14
|
+
automatically and inherits compatible prior work. Check `data.inheritance`
|
|
15
|
+
for fallback errors. Keep old runs and their maps.
|
|
16
|
+
3. Read `supercov docs assertion-maps`. Page the assertion, statement, test and
|
|
17
|
+
change views. Do not mistake one page for the whole suite:
|
|
18
|
+
|
|
19
|
+
```sh
|
|
20
|
+
supercov runs <run> assertions --limit 100 --json
|
|
21
|
+
supercov runs <run> assertions report --view statements --limit 100 --json
|
|
22
|
+
supercov runs <run> assertions report --view tests --limit 100 --json
|
|
23
|
+
supercov runs <run> assertions report --view changes --limit 100 --json
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Follow `pagination.nextOffset` with `--offset`; restart if `revision` changes.
|
|
27
|
+
Use a single map writer. Save edits atomically where your editor supports it.
|
|
28
|
+
The outer object contains `schemaVersion: 2` and `assertions`; do not insert run IDs.
|
|
29
|
+
|
|
30
|
+
## Investigate precise claims
|
|
31
|
+
|
|
32
|
+
Read current test setup, inputs, mocks, callbacks, branches, callees, helper
|
|
33
|
+
modules and relevant configuration. Current files must match the pinned run.
|
|
34
|
+
Supercov retains hashes and identities, not a source checkout. Optional
|
|
35
|
+
`runs <run> source <path> --offset 0 --limit 100` prints matching code with line
|
|
36
|
+
numbers; `assertions files` lists input hashes even when files no longer match.
|
|
37
|
+
|
|
38
|
+
Use `runs <run> assertion <id>` for the exact site and graph. Preserve its ID and
|
|
39
|
+
full `at` anchor, usually the complete assertion call without a trailing semicolon.
|
|
40
|
+
Use one-based UTF-8 byte columns, not character counts. An `inMap: false` site
|
|
41
|
+
was recognized but is missing from the map; restore it when investigating.
|
|
42
|
+
|
|
43
|
+
For each assertion:
|
|
44
|
+
|
|
45
|
+
- Describe exactly what its predicate distinguishes in `observes`. Truthiness,
|
|
46
|
+
existence, length or substring checks do not imply equality of every field.
|
|
47
|
+
- Write independently maintainable flows with stable IDs, `basis: null`, a
|
|
48
|
+
concrete explanation, exact nodes and authored edges to `$assertion`.
|
|
49
|
+
- Use explicit `appliesTo: [{file, name}]` from the test view. File and name
|
|
50
|
+
must resolve unambiguously. Empty means no credit. Runtime IDs never belong
|
|
51
|
+
in selectors. Shared/parameterized sites may need multiple cases or flows.
|
|
52
|
+
- Take production statement anchors from the `statements` view when possible.
|
|
53
|
+
Put only nodes you judge asserted in `countsAsAsserted`. Each counted node
|
|
54
|
+
needs an authored path to `$assertion`; a block does not credit nested code.
|
|
55
|
+
- Put additional dependency paths in `watch`, as whole-file strings. Assertion,
|
|
56
|
+
selected-test and node files are already dependencies. Include setup, guards,
|
|
57
|
+
alternate paths and helpers your explanation relies on, even without nodes.
|
|
58
|
+
- Preserve uncertainty in `questions`. Flow questions block that flow's credit.
|
|
59
|
+
Assertion questions record unfinished exploration without implying a known
|
|
60
|
+
total. There is no `analysis`, `mapped`, or `complete` flag.
|
|
61
|
+
- A fixture-only or absence explanation can have `countsAsAsserted: []`. Explain
|
|
62
|
+
the absence, including ordering/barriers and the observation window. Never
|
|
63
|
+
credit an unexecuted body just because its execution would violate a check.
|
|
64
|
+
|
|
65
|
+
Inspect `selectors`, `blockers`, `reasons`, empty `observedPassingTests`, and
|
|
66
|
+
statement `at: null` entries. They may represent skipped tests, unsupported
|
|
67
|
+
syntax, ambiguous test names or missing evidence. Do not borrow another site's
|
|
68
|
+
identity, fabricate events, delete inconvenient entries or edit managed state.
|
|
69
|
+
Zero-credit explanations are useful; invented credit is not.
|
|
70
|
+
|
|
71
|
+
## Account for changes before finalizing flow tokens
|
|
72
|
+
|
|
73
|
+
When inheriting a map, inspect **every** item in `--view changes`, including files
|
|
74
|
+
already watched by some flows. Dependencies may be missing. In root
|
|
75
|
+
`changeAssessments`, write one response per managed change ID:
|
|
76
|
+
|
|
77
|
+
```json
|
|
78
|
+
{
|
|
79
|
+
"id": "c_copy_from_changes_view",
|
|
80
|
+
"basis": null,
|
|
81
|
+
"affectedFlows": ["assertion-id/flow-id"],
|
|
82
|
+
"explanation": "What changed, which claims it affects, and why other existing claims remain valid."
|
|
83
|
+
}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Include all known dependent flows still present, plus any others affected.
|
|
87
|
+
Repair watches if the change reveals a missing dependency. An empty list needs
|
|
88
|
+
an actual explanation. Removing a response does not clear the outstanding change.
|
|
89
|
+
A changed test/assertion can affect all its flows; a production file used by one
|
|
90
|
+
sibling may affect only that flow. No machine can tell you an expected flow total.
|
|
91
|
+
|
|
92
|
+
Save the responses and graph edits. Run:
|
|
93
|
+
|
|
94
|
+
```sh
|
|
95
|
+
supercov assertions validate --file <map-path> --json
|
|
96
|
+
supercov runs <run> assertions validate --json
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
For large maps, page `validate --view changes`, `validate --view flows`, or
|
|
100
|
+
`validate --view errors` with `--offset` and `--limit`; tokens appear under
|
|
101
|
+
`items` and overall validity still covers the whole map. Pin `revision` across pages.
|
|
102
|
+
|
|
103
|
+
Validation checks shape, IDs, graph links, anchors and state binding. It returns
|
|
104
|
+
`changes[].expectedBasis` and `flows[].expectedBasis`; null/stale tokens alone
|
|
105
|
+
are unfinished work, not syntax errors. Copy change tokens **only after inspecting
|
|
106
|
+
those impact assessments**, then save. Validate again and copy flow tokens only
|
|
107
|
+
for claims you have examined. Save the map again. Change acknowledgements can
|
|
108
|
+
invalidate additional flows, so obtain final flow tokens after change tokens.
|
|
109
|
+
Changing a graph, selector, observation or watch after obtaining its token makes
|
|
110
|
+
that token stale. Do not implement the hashing algorithm or manufacture tokens.
|
|
111
|
+
|
|
112
|
+
There is no `review` command. All commands are read-only; acknowledgement is your
|
|
113
|
+
explicit file edit. Passing reference validation does not prove your reasoning.
|
|
114
|
+
|
|
115
|
+
## Check and report
|
|
116
|
+
|
|
117
|
+
```sh
|
|
118
|
+
supercov runs <run> assertions check --require-mappings --require-observed --json
|
|
119
|
+
supercov runs <run> assertions report --view statements --limit 100 --json
|
|
120
|
+
supercov runs <run>
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Basic `check` permits untouched sites without flows, but fails on invalid or
|
|
124
|
+
unfinished authored claims, open questions and outstanding change impact.
|
|
125
|
+
`--require-mappings` requires at least one current observed explanation for every
|
|
126
|
+
recognized passing site, including legitimate zero-credit explanations.
|
|
127
|
+
`--require-observed` checks every site and selector. Neither means every semantic
|
|
128
|
+
flow was found. Use `--min <percentage>` only for the project's chosen target.
|
|
129
|
+
Do not lower a target or hide unknowns to make a check pass.
|
|
130
|
+
|
|
131
|
+
The normal report includes the percentage immediately from the edited file.
|
|
132
|
+
`notAssessed`, `pending`, `unavailable` and `notApplicable` have null percentages,
|
|
133
|
+
not fabricated zeroes. A numeric partial-map result still reports counts of
|
|
134
|
+
assertions without flows and claims needing work. Describe the score as
|
|
135
|
+
agent-assessed statements; successful same-test coexecution is evidence, not
|
|
136
|
+
proof of causality or mutation resistance.
|
|
137
|
+
|
|
138
|
+
## Continue after edits
|
|
139
|
+
|
|
140
|
+
Run the same suite command again. The new run carries IDs, explanations and
|
|
141
|
+
unchanged acknowledgements using prior file hashes, without needing old source.
|
|
142
|
+
Use `assertions --needs-attention` and the change view. Repair only affected
|
|
143
|
+
claims; keep current siblings. Any dependency byte edit, including comments,
|
|
144
|
+
requires rechecking. Unique relocation preserves identity as a suggestion;
|
|
145
|
+
ambiguous/removed sites remain in `retiredAssertions`. Old v1 flows are imported
|
|
146
|
+
as drafts with questions about selectors and graph paths; inspect them explicitly.
|
|
147
|
+
|
|
148
|
+
Do not edit `assertions.state.json`. Dirty generations persist across reruns and
|
|
149
|
+
reverts until current tokens are recorded. Resolved change responses are folded
|
|
150
|
+
and retired on the next publication. Runtime evidence never carries forward.
|
|
151
|
+
If source differs from the pinned run, rerun tests before continuing.
|
|
152
|
+
|
|
153
|
+
Finish with the pinned run ID, status, numeric percentage/counts when available,
|
|
154
|
+
assertions without flows, remaining questions/stale or unobserved claims, and
|
|
155
|
+
verification commands used. Explain remaining limits. The property in `observes`
|
|
156
|
+
matters: changing a credited line while preserving that property can still pass.
|
|
@@ -1,135 +1,9 @@
|
|
|
1
|
-
# Assertion evidence
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
are different things; do not interpret a passing assertion nearby as protection.
|
|
11
|
-
|
|
12
|
-
## Requirements
|
|
13
|
-
|
|
14
|
-
Use the **npm Supercov launcher**, which supplies the installed analyzer assets.
|
|
15
|
-
The current JS/TS adapter is exercised with Node's test runner and Vitest.
|
|
16
|
-
Browser, background, merged, retried and ambiguously attributed records have
|
|
17
|
-
explicit limitations; other language assertion analyzers are not enabled here.
|
|
18
|
-
|
|
19
|
-
The analyzed project must provide a TypeScript compiler API, **even for a
|
|
20
|
-
JavaScript project**. For example, add TypeScript as a development dependency
|
|
21
|
-
before recording the run. The analyzer uses that project's compiler, not a
|
|
22
|
-
silently substituted global or bundled version. Installing it afterward changes
|
|
23
|
-
the dependency fingerprint, so rerun the suite. Missing/incompatible compiler APIs
|
|
24
|
-
produce an error. **TypeScript 5.8.3 and native 7.0.2 are tested.** Version 7.0.2
|
|
25
|
-
uses its own native parser/checker, not a fallback to TypeScript 5. Install its
|
|
26
|
-
platform-specific optional dependency too; the report hashes both the JS client
|
|
27
|
-
and the native compiler package, including its standard libraries. This backend
|
|
28
|
-
requires Node 22.12 or newer (Node 24 tested), and uses original-source evidence
|
|
29
|
-
from ordinary Supercov archives, not legacy ts-node/V8 generated-line coverage.
|
|
30
|
-
Native module resolution currently follows actual import/export references;
|
|
31
|
-
unresolved helper-only specifiers remain visible as compiler limitations.
|
|
32
|
-
Other native compiler versions are not enabled until separately calibrated.
|
|
33
|
-
Do not downgrade an application's compiler just to improve an assertion report.
|
|
34
|
-
|
|
35
|
-
Compiler compatibility and regression checks do not prove general correctness.
|
|
36
|
-
All candidates remain unverified and `assertionScore` remains null. Query-side
|
|
37
|
-
native compiler work adds no test-time instrumentation.
|
|
38
|
-
|
|
39
|
-
Standalone native/Python/Ruby distributions do not currently bundle this JS/TS
|
|
40
|
-
analyzer. Use the npm launcher, or explicitly set `SUPERCOV_PACKAGE_ROOT` to an
|
|
41
|
-
installed npm package directory. Normal coverage commands are unaffected.
|
|
42
|
-
|
|
43
|
-
## Run, inspect, follow evidence
|
|
44
|
-
|
|
45
|
-
`assertions` analyzes source behaviors and their assertion evidence; it does not
|
|
46
|
-
just count or list assertion calls. Use `npx supercov docs assertion-evidence`
|
|
47
|
-
to read this guide from the installed package.
|
|
48
|
-
|
|
49
|
-
```sh
|
|
50
|
-
npx supercov -- npm test
|
|
51
|
-
npx supercov runs latest assertions --limit 5
|
|
52
|
-
npx supercov runs latest assertions --file src/core.ts --json
|
|
53
|
-
npx supercov runs latest assertions --site '<site-id>' --json
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
The summary reports candidate counts and the site denominator, without claiming
|
|
57
|
-
a verified assertion percentage. Each ordinary site row contains its source,
|
|
58
|
-
candidate classification, available facts, and an `evidence.pointer` for full
|
|
59
|
-
details. Global tests, attempts, execution links, diagnostics and source-scope
|
|
60
|
-
limits are referenced under `evidence`, not repeated inside every site page.
|
|
61
|
-
|
|
62
|
-
```sh
|
|
63
|
-
npx supercov runs latest assertions --evidence /tests --limit 5 --json
|
|
64
|
-
npx supercov runs latest assertions --evidence /diagnostics --json
|
|
65
|
-
npx supercov runs latest assertions --evidence /sites/0/facts --json
|
|
66
|
-
```
|
|
67
|
-
|
|
68
|
-
Pointers use JSON Pointer syntax: escape `/` in a property name as `~1` and `~`
|
|
69
|
-
as `~0`. Use an empty pointer (`--evidence ''`) for the document root. Follow
|
|
70
|
-
returned pointers rather than constructing indices from filtered page offsets:
|
|
71
|
-
site pointers index the complete, stable inventory.
|
|
72
|
-
|
|
73
|
-
## Pagination and oversized records
|
|
74
|
-
|
|
75
|
-
All JSON responses stay within the normal response budget. Pages may contain
|
|
76
|
-
fewer than `--limit` results: **follow `pagination.nextOffset`**, not offset plus
|
|
77
|
-
the requested limit. `hasMore: false` marks the end.
|
|
78
|
-
|
|
79
|
-
An oversized site/hint is returned as `detailOnly: true` with an
|
|
80
|
-
`evidence.pointer`. Nothing has been discarded. An evidence page returns immediate
|
|
81
|
-
object members or array entries as `items`. Small entries contain `value`; large
|
|
82
|
-
ones contain `detailOnly: true` and another `pointer` to inspect. String leaves
|
|
83
|
-
return `text` chunks, with offsets/counts measured in Unicode scalar values,
|
|
84
|
-
not bytes. This allows reading one large observation or diagnostic completely.
|
|
85
|
-
|
|
86
|
-
```sh
|
|
87
|
-
npx supercov runs '<run-id>' assertions --evidence '<returned-pointer>' \
|
|
88
|
-
--offset 0 --limit 4000 --analysis '<analysisId>' --json
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
Every page carries an `analysisId` hashing the complete derived document and its
|
|
92
|
-
provenance. Pass the first page's id as `--analysis` on follow-ups to reject mixed
|
|
93
|
-
analyses if the compiler, analyzer, or derived results change. Use a fixed run id
|
|
94
|
-
instead of `latest` while paging. Source/run freshness checks still run on every
|
|
95
|
-
query. Derived results are not cached or written into the archive.
|
|
96
|
-
|
|
97
|
-
JSON reports use `reportSchema: 2`. Shared evidence is accessed through the
|
|
98
|
-
evidence pointers above.
|
|
99
|
-
Success and error JSON envelopes identify this query as `coverage.assertions`.
|
|
100
|
-
|
|
101
|
-
## Optional assertion hints
|
|
102
|
-
|
|
103
|
-
```ts
|
|
104
|
-
// observes: src/core.ts#compute return value
|
|
105
|
-
assert.equal(compute(), 4);
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
The comment must precede a statement containing exactly one recognized assertion.
|
|
109
|
-
The project-relative file, exact function/owner name, and optional literal source
|
|
110
|
-
substring must select one inventory site. Optional `via ...` is explanation only.
|
|
111
|
-
|
|
112
|
-
```sh
|
|
113
|
-
npx supercov runs latest assertions --pragmas --json
|
|
114
|
-
```
|
|
115
|
-
|
|
116
|
-
Origin is `user-suggested`; validation is separately `analyzer-supported`,
|
|
117
|
-
`unresolved`, or `invalid`. Support requires the named assertion's exact passing
|
|
118
|
-
witness and a connection supported by the ordinary effect rules. Strength remains
|
|
119
|
-
presence/value/total; presence does not mean value protection. Unsupported
|
|
120
|
-
decision, absence and internal-state paths remain unresolved. A missing inventory
|
|
121
|
-
match may reflect unsupported source rather than a bad declaration.
|
|
122
|
-
|
|
123
|
-
Hints cannot inject observations, borrow another assertion's evidence, inflate
|
|
124
|
-
coverage or remove sites from the denominator. Editing comments changes test
|
|
125
|
-
source and requires a new run. Supported hints retain the analyzer's limitations;
|
|
126
|
-
they are not formal proofs.
|
|
127
|
-
|
|
128
|
-
## Agent workflow
|
|
129
|
-
|
|
130
|
-
Read coverage first, then inspect assertion candidates and their evidence before
|
|
131
|
-
writing another test. An execution gap may need a reachable scenario; an assertion
|
|
132
|
-
gap may need a stronger check. An analysis limit needs investigation—not a test
|
|
133
|
-
written just to satisfy the analyzer. Keep the original denominator and unresolved
|
|
134
|
-
work visible. Never automatically treat an `evident` candidate as permission to
|
|
135
|
-
change application behavior.
|
|
1
|
+
# Assertion evidence
|
|
2
|
+
|
|
3
|
+
The earlier automatic JS/TS analyzer has been replaced by agent-authored
|
|
4
|
+
assertion maps. Use [Understanding assertion coverage](assertions.md) to get
|
|
5
|
+
started, [Agent-authored assertion maps](assertion-maps.md) for the file format
|
|
6
|
+
and CLI reference, and [the agent instructions](assertion-agent.md) to build or
|
|
7
|
+
update a map. Existing structural runs remain readable; mapping requires a new
|
|
8
|
+
run with an assertion input manifest. Old `--pragmas`, `--analysis` and `--evidence`
|
|
9
|
+
options do not apply to the map workflow.
|