supercov 0.0.26 → 0.0.28
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +102 -90
- package/docs/agent-loop.md +78 -39
- package/docs/cli.md +108 -46
- package/docs/coverage-model.md +81 -37
- package/docs/evidence.md +63 -48
- package/docs/getting-started.md +75 -55
- package/docs/performance.md +53 -43
- package/docs/supported-suites.md +75 -57
- package/docs/troubleshooting.md +143 -0
- package/docs/verification.md +59 -45
- package/docs/workspace-isolation.md +59 -38
- package/package.json +7 -7
package/docs/verification.md
CHANGED
|
@@ -1,39 +1,38 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Trusting a result
|
|
2
2
|
|
|
3
|
-
Coverage is useful only
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
Coverage is useful only when the test command still behaves normally and the
|
|
4
|
+
report distinguishes what was covered from what could not be measured.
|
|
5
|
+
Supercov is designed to fail clearly rather than turn uncertainty into a clean
|
|
6
|
+
percentage.
|
|
6
7
|
|
|
7
|
-
##
|
|
8
|
+
## Review a test written by an agent
|
|
8
9
|
|
|
9
|
-
|
|
10
|
+
Before merging, check three things:
|
|
10
11
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
- supported JavaScript, TypeScript, and Rust runner contracts;
|
|
18
|
-
- Chromium, Firefox, and WebKit browser execution;
|
|
19
|
-
- source isolation, interrupted-run recovery, and atomic publication; and
|
|
20
|
-
- package installation and execution from a clean project.
|
|
12
|
+
1. **The complete wrapped test command passes.** A focused test command is useful
|
|
13
|
+
during iteration, but it is not the final gate.
|
|
14
|
+
2. **The diff shows the expected gain.** Look for unexplained coverage losses or
|
|
15
|
+
a changed source boundary.
|
|
16
|
+
3. **The test protects behavior.** It should make a meaningful assertion without
|
|
17
|
+
weakening existing assertions or changing application code for the metric.
|
|
21
18
|
|
|
22
|
-
|
|
23
|
-
|
|
19
|
+
```sh
|
|
20
|
+
npx supercov diff <baseline-run> <new-run>
|
|
21
|
+
npx supercov runs <new-run> test "new test name"
|
|
22
|
+
```
|
|
24
23
|
|
|
25
|
-
|
|
24
|
+
Keep the compared run ids in the pull request or agent summary when the result
|
|
25
|
+
needs to be reviewed later.
|
|
26
26
|
|
|
27
|
-
|
|
28
|
-
text or creates source dynamically without a stable denominator. It leaves the
|
|
29
|
-
affected behavior uninstrumented and records a completeness blocker with the
|
|
30
|
-
reason and location.
|
|
27
|
+
## Read the status honestly
|
|
31
28
|
|
|
32
|
-
|
|
33
|
-
reported as aggregate, unattributed, or missing. It is not assigned to a test
|
|
34
|
-
that may not have caused it.
|
|
29
|
+
Supercov separates four states:
|
|
35
30
|
|
|
36
|
-
|
|
31
|
+
- **covered** — the obligation was measured and observed;
|
|
32
|
+
- **uncovered** — it was measured but not observed, so a test may close it;
|
|
33
|
+
- **measurement limit** — Supercov could not establish a complete boundary;
|
|
34
|
+
- **stale** — the run is valid history but no longer describes the current
|
|
35
|
+
workspace.
|
|
37
36
|
|
|
38
37
|
```sh
|
|
39
38
|
npx supercov runs latest
|
|
@@ -41,28 +40,43 @@ npx supercov runs latest scope
|
|
|
41
40
|
npx supercov runs latest gaps
|
|
42
41
|
```
|
|
43
42
|
|
|
44
|
-
|
|
43
|
+
Do not treat a measurement limit as a test-writing task. Read its reason, fix
|
|
44
|
+
source scope or configuration when possible, and otherwise report the boundary.
|
|
45
45
|
|
|
46
|
-
|
|
46
|
+
## How Supercov avoids false confidence
|
|
47
47
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
48
|
+
- The coverage denominator is derived before the suite runs, so adding tests
|
|
49
|
+
cannot silently redefine 100%.
|
|
50
|
+
- Instrumentation runs in an isolated copy instead of rewriting your source.
|
|
51
|
+
- Evidence is attributed to a test only when the runner exposes a reliable
|
|
52
|
+
identity; everything else remains aggregate or background evidence.
|
|
53
|
+
- Completed runs are immutable and checked against the current workspace.
|
|
54
|
+
- Missing, corrupt, contradictory, or unsupported evidence is surfaced as a
|
|
55
|
+
limitation instead of being guessed or discarded.
|
|
52
56
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
+
## What happens when code cannot be measured safely
|
|
58
|
+
|
|
59
|
+
Some code observes its own source text, creates source dynamically, or executes
|
|
60
|
+
behind a launcher Supercov cannot see. Forcing ordinary instrumentation through
|
|
61
|
+
those boundaries could change behavior or invent a denominator.
|
|
62
|
+
|
|
63
|
+
Supercov leaves the affected behavior uninstrumented and reports the location
|
|
64
|
+
and reason. Likewise, coverage from a runner without exact test identity stays
|
|
65
|
+
aggregate instead of being assigned to whichever test happened to be nearby.
|
|
66
|
+
|
|
67
|
+
## How releases are checked
|
|
57
68
|
|
|
58
|
-
|
|
59
|
-
|
|
69
|
+
Release checks cover program behavior before and after instrumentation, line and
|
|
70
|
+
branch results, decision vectors and MC/DC, supported JavaScript, TypeScript,
|
|
71
|
+
and Rust runner contracts, Chromium, Firefox, and WebKit execution, clean
|
|
72
|
+
installation, interrupted-run recovery, and isolated publication.
|
|
60
73
|
|
|
61
|
-
|
|
74
|
+
JavaScript behavior is exercised against a pinned TC39 Test262 corpus. MC/DC
|
|
75
|
+
cases are also compared with an independent LLVM implementation so a
|
|
76
|
+
self-consistent calculation error does not pass unnoticed.
|
|
62
77
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
incompatible is surfaced as such instead of being opened as a plausible report.
|
|
78
|
+
These checks reduce risk; they do not replace reviewing the assertions and
|
|
79
|
+
behavior protected by a new test.
|
|
66
80
|
|
|
67
|
-
See [
|
|
68
|
-
|
|
81
|
+
See [Runs and evidence](evidence.md) for immutable run ids, comparisons, and
|
|
82
|
+
retention.
|
|
@@ -1,25 +1,18 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Files, privacy, and cleanup
|
|
2
2
|
|
|
3
3
|
Supercov measures an instrumented copy of the project. It does not rewrite the
|
|
4
|
-
source tree you edit or
|
|
4
|
+
source tree you edit or send your source and coverage evidence to a hosted
|
|
5
|
+
Supercov service.
|
|
5
6
|
|
|
6
|
-
## What
|
|
7
|
+
## What uses the network
|
|
7
8
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
| `.supercov/runs/<run-id>/` | Immutable completed runs |
|
|
11
|
-
| `.supercov/work/` | In-progress state and evidence staging |
|
|
12
|
-
| `.supercov/locks/` | Prevents overlapping run and cleanup operations |
|
|
13
|
-
| `supercov/workspace/<project>/` | Marker-protected isolated source and build cache |
|
|
14
|
-
|
|
15
|
-
Supercov owns these locations only when its exact marker is present. If the
|
|
16
|
-
project already contains a user-created `supercov/` directory, Supercov does
|
|
17
|
-
not adopt or delete it; it chooses a deterministic fallback location instead.
|
|
9
|
+
The first `npx supercov` invocation may contact the npm registry to download the
|
|
10
|
+
package. Your wrapped test command may also use the network if it normally does.
|
|
18
11
|
|
|
19
|
-
The
|
|
20
|
-
|
|
12
|
+
The Supercov CLI does not need a Supercov account or upload a coverage run to
|
|
13
|
+
Supercov. Run evidence and query indexes stay on the machine.
|
|
21
14
|
|
|
22
|
-
## What
|
|
15
|
+
## What stays untouched
|
|
23
16
|
|
|
24
17
|
Supercov does not intentionally edit:
|
|
25
18
|
|
|
@@ -27,33 +20,57 @@ Supercov does not intentionally edit:
|
|
|
27
20
|
- imports or dependency declarations;
|
|
28
21
|
- test-runner configuration or reporter lists;
|
|
29
22
|
- the project's ordinary build output; or
|
|
30
|
-
- files outside
|
|
23
|
+
- files outside marker-owned Supercov storage.
|
|
24
|
+
|
|
25
|
+
The wrapped command still has its normal side effects. If `npm test` writes
|
|
26
|
+
snapshots, calls a service, or changes a database, wrapping it does not remove
|
|
27
|
+
that behavior. The isolation guarantee applies to Supercov's instrumentation
|
|
28
|
+
and evidence work.
|
|
31
29
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
30
|
+
Files the wrapped command creates or changes inside the isolated workspace are
|
|
31
|
+
synced back to the project after the run, so `supercov -- npm test -- -u`
|
|
32
|
+
updates snapshots in the repository exactly as `npm test -- -u` would. Two
|
|
33
|
+
exceptions are reported instead of applied: changes the command makes to
|
|
34
|
+
instrumented source files (the instrumented copies must never overwrite your
|
|
35
|
+
sources) and deletions (never propagated automatically).
|
|
35
36
|
|
|
36
|
-
##
|
|
37
|
+
## Files Supercov creates
|
|
38
|
+
|
|
39
|
+
| Location | What it is for |
|
|
40
|
+
| --- | --- |
|
|
41
|
+
| `.supercov/runs/<run-id>/` | Completed immutable runs |
|
|
42
|
+
| `.supercov/work/` | Temporary state while a run is being prepared |
|
|
43
|
+
| `.supercov/locks/` | Prevents two operations from racing |
|
|
44
|
+
| `.supercov-workspace/workspace/<project>/` | Isolated source and reusable instrumented build cache (safe to delete) |
|
|
37
45
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
force an unrelated application rebuild.
|
|
46
|
+
Managed directories include Git ignore rules so run evidence and instrumented
|
|
47
|
+
builds do not become ordinary repository changes.
|
|
41
48
|
|
|
42
|
-
|
|
43
|
-
|
|
49
|
+
Supercov owns a directory only when its exact marker is present. If the project
|
|
50
|
+
already has a user-created `supercov/` directory, the CLI chooses a deterministic
|
|
51
|
+
fallback instead of adopting or deleting it.
|
|
44
52
|
|
|
45
|
-
##
|
|
53
|
+
## Repeated runs and the build cache
|
|
46
54
|
|
|
47
|
-
|
|
55
|
+
When source, dependencies, configuration, toolchain, and build mode still
|
|
56
|
+
match, Supercov can reuse the isolated instrumented build. Test-only changes do
|
|
57
|
+
not force an unrelated application rebuild.
|
|
58
|
+
|
|
59
|
+
Workspace refreshes are prepared separately and become active only when
|
|
60
|
+
complete. An interrupted refresh does not replace the last complete cache with
|
|
61
|
+
partial output.
|
|
62
|
+
|
|
63
|
+
## Interrupted and overlapping commands
|
|
64
|
+
|
|
65
|
+
One project can run one coverage or cleanup transaction at a time. A second
|
|
48
66
|
operation fails clearly instead of racing the first.
|
|
49
67
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
completed runs are published atomically and remain immutable.
|
|
68
|
+
After interruption or host restart, the next command recovers unpublished
|
|
69
|
+
staging state. Completed runs remain immutable.
|
|
53
70
|
|
|
54
|
-
##
|
|
71
|
+
## Clean up local data
|
|
55
72
|
|
|
56
|
-
Preview cleanup before
|
|
73
|
+
Preview cleanup before removing anything:
|
|
57
74
|
|
|
58
75
|
```sh
|
|
59
76
|
npx supercov clean --dry-run
|
|
@@ -61,12 +78,16 @@ npx supercov clean --keep 20
|
|
|
61
78
|
npx supercov clean
|
|
62
79
|
```
|
|
63
80
|
|
|
64
|
-
|
|
65
|
-
|
|
81
|
+
The final command removes all runs and the isolated build cache. `--keep 20`
|
|
82
|
+
retains the 20 newest runs. Cleanup follows marker ownership, waits for the
|
|
83
|
+
project lock, and does not scan for similarly named directories.
|
|
66
84
|
|
|
67
85
|
## Containers and remote workspaces
|
|
68
86
|
|
|
69
87
|
When a suite launches a container or VM from a mounted workspace, Supercov uses
|
|
70
|
-
the isolated workspace as the source presented to that environment.
|
|
71
|
-
|
|
72
|
-
|
|
88
|
+
the isolated workspace as the source presented to that environment. The runtime
|
|
89
|
+
must be able to cross the launch boundary and return evidence.
|
|
90
|
+
|
|
91
|
+
If a remote executor hides the launch or mount boundary, Supercov reports the
|
|
92
|
+
limitation instead of claiming unseen code was measured. See
|
|
93
|
+
[Supported suites](supported-suites.md) for the current boundary.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "supercov",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.28",
|
|
4
4
|
"description": "Zero-edit, runner-aware coverage completeness for JavaScript test suites",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -68,12 +68,12 @@
|
|
|
68
68
|
"prepublishOnly": "npm run release:check"
|
|
69
69
|
},
|
|
70
70
|
"optionalDependencies": {
|
|
71
|
-
"@supercov/cli-darwin-arm64": "0.0.
|
|
72
|
-
"@supercov/cli-darwin-x64": "0.0.
|
|
73
|
-
"@supercov/cli-linux-arm64-gnu": "0.0.
|
|
74
|
-
"@supercov/cli-linux-arm64-musl": "0.0.
|
|
75
|
-
"@supercov/cli-linux-x64-gnu": "0.0.
|
|
76
|
-
"@supercov/cli-linux-x64-musl": "0.0.
|
|
71
|
+
"@supercov/cli-darwin-arm64": "0.0.28",
|
|
72
|
+
"@supercov/cli-darwin-x64": "0.0.28",
|
|
73
|
+
"@supercov/cli-linux-arm64-gnu": "0.0.28",
|
|
74
|
+
"@supercov/cli-linux-arm64-musl": "0.0.28",
|
|
75
|
+
"@supercov/cli-linux-x64-gnu": "0.0.28",
|
|
76
|
+
"@supercov/cli-linux-x64-musl": "0.0.28"
|
|
77
77
|
},
|
|
78
78
|
"peerDependencies": {
|
|
79
79
|
"@playwright/test": ">=1.55.0",
|