supercov 0.0.22 → 0.0.24

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. package/README.md +147 -402
  2. package/docs/agent-loop.md +66 -129
  3. package/docs/cli.md +74 -87
  4. package/docs/coverage-model.md +79 -129
  5. package/docs/evidence.md +60 -91
  6. package/docs/getting-started.md +71 -97
  7. package/docs/performance.md +57 -106
  8. package/docs/supported-suites.md +82 -103
  9. package/docs/verification.md +49 -83
  10. package/docs/workspace-isolation.md +67 -115
  11. package/package.json +7 -7
  12. package/runtime/javascript/capability.js +19 -0
  13. package/runtime/javascript/register.mjs +5 -1
package/docs/agent-loop.md CHANGED
@@ -1,162 +1,99 @@
1
1
  # Agent loop
2
2
 
3
- Use Supercov in a repeatable agent loop: run the suite, query uncovered
4
- obligations, add a focused test, rerun, and compare results. This page covers
5
- the loop, the recommended prompt, and failure handling.
6
-
7
- ## The shape of the loop
3
+ Use Supercov in a simple loop: run the suite, choose one useful gap, write one
4
+ test, rerun, and prove what improved.
8
5
 
9
6
  ```text
10
- run the suite -> ask what is open -> write one test -> re-run -> diff
11
- ^ |
12
- +-------------------------------------------------------------------+
7
+ run the suite → choose a gap → write one test → rerun → compare
8
+ ↑ |
9
+ └────────────────────────────────────────────────────────────┘
13
10
  ```
14
11
 
15
- Each pass should close a small number of related obligations and end with
16
- evidence that it did. An agent that writes ten tests before re-running has no
17
- way to attribute the outcome; an agent that re-runs after every trivial edit
18
- spends its budget on test execution instead of thinking.
19
-
20
- ## One pass, in commands
12
+ ## One pass
21
13
 
22
14
  ```sh
23
- # 1. Establish a baseline. Only needed once per session.
15
+ # 1. Establish a baseline.
24
16
  npx supercov -- npm test
25
17
 
26
- # 2. Orient without loading a report into context.
27
- npx supercov runs latest --json
28
- npx supercov runs latest gaps --limit 5 --json
29
-
30
- # 3. Understand one target.
31
- npx supercov runs latest file app/checkout/session.ts --json
32
- npx supercov runs latest decision app/checkout/session.ts:64 --json
18
+ # 2. Choose a useful target without loading a large report.
19
+ npx supercov runs latest gaps --limit 5
33
20
 
34
- # 4. Check what already exercises that line, to avoid writing a duplicate.
35
- npx supercov runs latest line app/checkout/session.ts:64 --json
21
+ # 3. Understand the target and what already reaches it.
22
+ npx supercov runs latest file app/checkout/session.ts
23
+ npx supercov runs latest decision app/checkout/session.ts:64
24
+ npx supercov runs latest line app/checkout/session.ts:64
36
25
 
37
- # 5. Write one test. Then re-run and prove the gain.
26
+ # 4. Write one focused test, then rerun and prove the gain.
38
27
  npx supercov -- npm test
39
- npx supercov diff <previous-run-id> latest --json
28
+ npx supercov diff <previous-run-id> latest
40
29
  ```
41
30
 
42
- Step 4 is the one agents skip and should not. `line` answers "what already
43
- executes this line", which usually reveals either an existing test to extend or
44
- the exact reason nothing reaches it.
31
+ For Rust, replace `npm test` with `cargo test` or `cargo nextest run` in both
32
+ runs. Keep the command identical between the baseline and comparison.
45
33
 
46
- ## A prompt you can paste
34
+ The `line` query is useful before writing a test: it shows what already
35
+ executes that line, which can reveal an existing test to extend instead of a
36
+ duplicate to add.
37
+
38
+ ## Prompt for a coding agent
47
39
 
48
40
  ```text
49
- You are improving test coverage for this repository using Supercov.
50
-
51
- Baseline:
52
- npx supercov -- npm test
53
-
54
- Then repeat this loop until coverage completeness stops improving, the target
55
- is met, or you run out of time:
56
-
57
- 1. npx supercov runs latest gaps --limit 5 --json
58
- 2. Pick the file with the highest-value open obligations.
59
- 3. npx supercov runs latest file <path> --json
60
- npx supercov runs latest decision <path>:<line> --json
61
- npx supercov runs latest line <path>:<line> --json
62
- 4. Write ONE focused test that closes the specific obligations you just read.
63
- The assertion must be meaningful on its own; never assert something trivial
64
- just to execute a line.
65
- 5. npx supercov -- npm test
66
- 6. npx supercov diff <previous-run-id> latest --json
67
- If the diff shows no gain, revert the test rather than keeping it.
68
-
69
- Rules:
70
- - Never contort live application source to make coverage easier. Deleting
71
- provably dead code is the opposite case: coverage found real cruft, and
72
- removing it (with the project owner's normal review) is the improvement.
73
- - Never weaken or delete an existing assertion.
74
- - If a decision cannot be reached from any public entry point, say so and move
75
- on instead of exporting internals to reach it.
76
- - Report the run ids you compared and the obligations you closed.
41
+ Use `npx supercov` to improve coverage. Only write tests. Keep going while
42
+ useful gaps remain.
43
+
44
+ Run the repository's complete test command through Supercov. Then repeat:
45
+ 1. Run `npx supercov runs latest gaps --limit 5`.
46
+ 2. Choose one useful uncovered behavior.
47
+ 3. Inspect it with the `file`, `decision`, or `line` query.
48
+ 4. Write one focused test with meaningful assertions.
49
+ 5. Rerun the same complete suite through Supercov.
50
+ 6. Run `npx supercov diff <previous-run-id> latest` to prove the gain.
51
+
52
+ Only edit tests. Never weaken assertions or change application code to make
53
+ coverage easier. Stop when no useful gaps remain, a gap is not reachable
54
+ through a public behavior, or the time budget is exhausted. Report the run ids
55
+ you compared and what improved.
77
56
  ```
78
57
 
79
- The last rule matters more than it looks. An unattended agent that cannot reach
80
- a branch will otherwise start reshaping the code so it can, which is exactly
81
- the failure mode that gives coverage targets a bad name. The first rule cuts
82
- the other way just as deliberately: when an obligation is unreachable because
83
- the code is dead, the honest fix is deletion, not an exclusion that leaves the
84
- cruft sitting behind a clean number.
85
-
86
- ## Budgeting an overnight session
58
+ ## Choose valuable gaps
87
59
 
88
- Test execution dominates the wall clock, so the number of passes is roughly the
89
- time budget divided by suite duration. Two adjustments help:
60
+ `gaps` ranks unresolved obligations, but coverage count is not the same as
61
+ product value. Prefer code that protects user-facing behavior, permissions,
62
+ payments, state transitions, error recovery, and other high-consequence paths.
90
63
 
91
- - Narrow the command while iterating. `npx supercov -- npx vitest run
92
- app/checkout` produces a valid run over a smaller denominator; use the full
93
- suite for the baseline and the final verification.
94
- - Let the build cache work. When the source, configuration and toolchain
95
- fingerprint is unchanged, the instrumented build is reused and that phase
96
- costs approximately nothing. Changing a dependency or a build config in the
97
- middle of a session throws that away.
64
+ Useful checks before writing a test:
98
65
 
99
- ## Choosing what to attack
66
+ - Is this behavior reachable through a public API or user action?
67
+ - Does an existing test almost cover it?
68
+ - Can the test make a meaningful assertion rather than merely execute a line?
69
+ - Is the path actually dead code that should be reported for human review?
100
70
 
101
- `gaps` is ordered to be useful, but not every open obligation deserves a test.
102
- For a project that prefers end-to-end evidence, start with the existing
103
- projection rather than inventing a new test taxonomy:
71
+ If the project separates test levels, focus the view:
104
72
 
105
73
  ```sh
106
- npx supercov runs latest gaps --kind e2e
74
+ npx supercov runs latest gaps --kind e2e --limit 10
107
75
  ```
108
76
 
109
- Each file distinguishes obligations covered by another test kind from those
110
- uncovered everywhere. The former are candidates for stronger E2E coverage;
111
- the latter are gaps in the combined suite. When an error path cannot be reached
112
- through E2E, first check whether the test double can express that failure before
113
- falling back to a narrower unit test.
114
-
115
- Two queries help an agent argue about value rather than count:
77
+ ## Keep the loop efficient
116
78
 
117
- ```sh
118
- # What does the suite prove today, minus redundancy?
119
- npx supercov runs latest minimize --filter passed
120
-
121
- # Reach a target with the smallest possible subset.
122
- npx supercov runs latest minimize --filter passed --metric mcdc --target 80
123
- ```
79
+ - Begin and end with the complete test command.
80
+ - While iterating, a narrower test command is fine if its smaller denominator
81
+ is understood.
82
+ - Write one related test at a time, then rerun. Large batches make failures and
83
+ coverage gains harder to attribute.
84
+ - Use immutable run ids when work spans several sessions. `latest` is a
85
+ convenience for interactive use.
86
+ - Treat a stale run as history when the source has changed since it was made.
124
87
 
125
- `minimize` is an exact branch-and-bound solver, not a greedy approximation: the
126
- subset it returns is a proved minimum. It refuses to answer for a view that
127
- contains background or unattributed evidence, because there is no honest way to
128
- name an exact subset of tests when the runner never exposed test boundaries.
88
+ ## Know when to stop
129
89
 
130
- ## Reading a run that is not the newest
90
+ Stop instead of grinding when:
131
91
 
132
- `latest` is a convenience for interactive use. An agent that resumes work later,
133
- or that compares across a session, should use the immutable run id:
134
-
135
- ```sh
136
- npx supercov runs --limit 10 --json
137
- npx supercov runs run_0123456789abcdef gaps --json
138
- ```
92
+ - no useful uncovered behavior remains;
93
+ - the open path cannot be reached through a supported public behavior;
94
+ - source scope is ambiguous and needs `SUPERCOV_SOURCE_ROOTS`;
95
+ - execution belongs to an unsupported or unattributed runner; or
96
+ - Supercov reports a completeness blocker rather than an ordinary test gap.
139
97
 
140
- Queries compare the stored fingerprint with the current workspace and mark a run
141
- stale when the code has moved on. Treat a stale run as history, not as a
142
- description of the working tree.
143
-
144
- ## What to do about honest gaps
145
-
146
- Some obligations are open because the tooling says so, not because a test is
147
- missing:
148
-
149
- - **Background or unattributed evidence.** An unsupported runner, or work that
150
- arrived without a carrier, is recorded under a first-class background scope.
151
- It appears in the default all-attempt view and is excluded from per-test
152
- passed-only coverage. Writing more tests will not move it; adding runner
153
- support will.
154
- - **Ambiguous source scope.** A candidate file that Supercov could not
155
- confidently classify as first-party blocks a complete verdict. Inspect with
156
- `coverage scope` and set `SUPERCOV_SOURCE_ROOTS` to declare the authoritative
157
- scope.
158
- - **Semantic-safety blockers.** A function whose source is coerced or reflected
159
- on at runtime is left uninstrumented on purpose, and direct `eval` cannot have
160
- a stable denominator at all. Both are recorded with their exact location.
161
-
162
- An agent should surface these rather than grind against them.
98
+ These states are reported explicitly so an agent does not reshape application
99
+ code merely to reach a number.
package/docs/cli.md CHANGED
@@ -1,145 +1,132 @@
1
1
  # CLI reference
2
2
 
3
- Every command is local. Nothing is uploaded, and no command runs your test
4
- suite unless you ask it to.
3
+ Supercov runs locally. No command uploads source or coverage evidence.
5
4
 
6
5
  ```sh
7
- supercov --help
6
+ npx supercov --help
8
7
  ```
9
8
 
10
- ## Creating a run
9
+ ## Measure a test command
11
10
 
12
11
  ```sh
13
- supercov -- <test command>
12
+ npx supercov -- <test command>
14
13
  ```
15
14
 
16
- Everything after `--` is executed as written. Supercov propagates coverage
17
- through every Node child process the command launches, then publishes one
18
- immutable run.
15
+ Everything after `--` is the command Supercov measures.
19
16
 
20
17
  ```sh
21
18
  npx supercov -- npm test
22
19
  npx supercov -- npx playwright test --project=chromium
23
- npx supercov -- npx vitest run app/checkout
20
+ npx supercov -- cargo test
21
+ npx supercov -- cargo nextest run
24
22
  ```
25
23
 
26
- A per-project lock rejects overlapping runs before either can build.
24
+ A coverage run exits with the test command's own status, so the wrapped command
25
+ can remain a CI gate.
27
26
 
28
- ## Listing runs
27
+ ## List runs
29
28
 
30
29
  ```sh
31
- supercov runs [--limit N] [--json]
30
+ npx supercov runs [--limit N] [--json]
32
31
  ```
33
32
 
34
- Runs are listed newest first with their id, duration, phase timings and
35
- integrity state. Use the id — not `latest` — when work spans a session.
33
+ Runs are listed newest first. Use an immutable run id when work spans a session;
34
+ use `latest` for interactive work.
36
35
 
37
- ## Coverage queries
38
-
39
- All coverage queries take the form:
36
+ ## Query one run
40
37
 
41
38
  ```sh
42
- supercov runs <run-id> [query] [options]
39
+ npx supercov runs <run-id> [query] [options]
43
40
  ```
44
41
 
45
- `<run-id>` is positional because every coverage view belongs to exactly one
46
- immutable run. `latest` selects the newest local run.
47
-
48
- | Query | Answers |
42
+ | Query | What it answers |
49
43
  | --- | --- |
50
- | no query | Overall completeness for the selected view |
51
- | `kinds` | Completeness split by semantic level (`unit`, `e2e`, …) |
52
- | `runners` | Completeness split by executing runner |
53
- | `scope` | Which source files are included, excluded or ambiguous |
54
- | `files` | Every included source file, ranked |
55
- | `gaps` | Only files with unresolved obligations or measurement limits |
56
- | `file <path>` | Every open obligation in one file |
57
- | `decision <id \| path:line>` | Observed vectors and missing witnesses for one decision |
58
- | `line <path:line>` | Line state, nested obligations, covering tests, and phases |
59
- | `test <id \| name fragment>` | What one test contributes |
60
- | `minimize` | The smallest test subset that preserves coverage |
61
-
62
- ### Options
63
-
64
- | Option | Applies to | Meaning |
65
- | --- | --- | --- |
66
- | `--filter all \| passed \| failed` | most queries | Which attempts contribute. `all` is the default and matches conventional tools. |
67
- | `--kind <kind>` | most queries | Restrict to a semantic level, for example `--kind e2e`. |
68
- | `--runner <runner>` | summary | Restrict to one executing runner, for example `--runner playwright`. |
69
- | `--metric all \| lines \| statements \| functions \| branches \| mcdc` | `minimize` | Which obligations the solver must preserve. |
70
- | `--target 0..100` | `minimize` | Stop once the metric reaches this level. |
71
- | `--limit N`, `--offset N` | collections | Pagination. Collections default to 20 items and print a copyable next-page command. |
72
- | `--json` | every query | The stable machine format. |
73
-
74
- ### Examples
44
+ | no query | Overall completeness and measurement limits |
45
+ | `kinds` | Coverage by semantic level, such as unit or E2E |
46
+ | `runners` | Coverage by test runner |
47
+ | `scope` | Included, excluded, and ambiguous source files |
48
+ | `files` | All included files, ranked |
49
+ | `gaps` | Files with useful open obligations or measurement limits |
50
+ | `file <path>` | Open obligations in one file |
51
+ | `decision <id \| path:line>` | Observed decision vectors and missing witnesses |
52
+ | `line <path:line>` | Line state, nested obligations, and covering tests |
53
+ | `test <id \| name>` | What one test contributes |
54
+ | `minimize` | The smallest test subset that preserves selected coverage |
55
+
56
+ Common examples:
75
57
 
76
58
  ```sh
77
- # Orient in a few lines.
78
59
  npx supercov runs latest
79
- npx supercov runs latest --filter passed
80
- npx supercov runs latest kinds
81
-
82
- # Find and open one target.
83
- npx supercov runs latest gaps --kind e2e --limit 10
84
- npx supercov runs latest file app/routes/example.ts
85
- npx supercov runs latest decision app/routes/example.ts:42
86
- npx supercov runs latest line app/routes/example.ts:57
87
-
88
- # Understand contribution and redundancy.
60
+ npx supercov runs latest gaps --limit 10
61
+ npx supercov runs latest file app/routes/checkout.ts
62
+ npx supercov runs latest decision app/routes/checkout.ts:42
63
+ npx supercov runs latest line app/routes/checkout.ts:57
89
64
  npx supercov runs latest test "checkout retry"
90
- npx supercov runs latest minimize --filter passed
91
- npx supercov runs latest minimize --filter passed --metric mcdc --target 80
92
65
  ```
93
66
 
94
- With `--kind`, gap and file queries additionally distinguish obligations covered
95
- only by other test levels from obligations uncovered everywhere. On a combined
96
- unit/E2E run, the default summary also prints the line count reached by other
97
- test kinds but not by E2E, followed by the exact `gaps --kind e2e` query.
67
+ ## Query options
68
+
69
+ | Option | Meaning |
70
+ | --- | --- |
71
+ | `--filter all \| passed \| failed` | Choose which test attempts contribute. `all` is the default. |
72
+ | `--kind <kind>` | Restrict to a test level such as `unit`, `integration`, or `e2e`. |
73
+ | `--runner <runner>` | Restrict a summary to one runner. |
74
+ | `--limit N`, `--offset N` | Page through collection results. |
75
+ | `--metric all \| lines \| statements \| functions \| branches \| mcdc` | Choose the obligations preserved by `minimize`. |
76
+ | `--target 0..100` | Stop `minimize` when the selected metric reaches the target. |
77
+ | `--json` | Return the stable machine-readable form when an integration needs it. |
98
78
 
99
- ## Comparing runs
79
+ Collections print a copyable next-page command. Ordinary text output is intended
80
+ to work well for both people and coding agents.
81
+
82
+ ## Compare runs
100
83
 
101
84
  ```sh
102
- supercov diff <older-run> <newer-run> [--limit N] [--json]
85
+ npx supercov diff <older-run> <newer-run> [--limit N] [--json]
103
86
  ```
104
87
 
105
- Reports what the newer run covers that the older one did not, and what it lost.
106
- Both runs remain untouched.
88
+ `diff` shows both gains and losses. Neither input run is changed.
107
89
 
108
- ## Combining shards
90
+ ## Merge shards
109
91
 
110
92
  ```sh
111
- supercov merge <run-id> <run-id> [...]
93
+ npx supercov merge <run-id> <run-id> [...]
112
94
  ```
113
95
 
114
- Accepts only runs with identical source, test, dependency, configuration,
115
- instrumenter, schema and denominator fingerprints. It rewrites the run scope
116
- inside every evidence record, namespaces shard paths, and publishes a new
117
- immutable run atomically. Input runs are never modified. Incompatible shards
118
- fail clearly rather than producing a plausible but invalid aggregate; the
119
- error names each exact fingerprint domain that differs.
96
+ `merge` creates a new run from compatible shards. If source, configuration,
97
+ toolchain, schema, or denominator fingerprints differ, it fails clearly rather
98
+ than producing an invalid aggregate.
120
99
 
121
- ## Retention
100
+ ## Clean local data
122
101
 
123
102
  ```sh
124
- supercov clean [--keep N] [--dry-run]
103
+ npx supercov clean --dry-run
104
+ npx supercov clean --keep 20
105
+ npx supercov clean
125
106
  ```
126
107
 
127
- `clean` removes all history and the isolated build workspace by default.
128
- `--keep N` preserves the N newest runs. It never runs automatically, takes the
129
- same lock as a coverage run, refuses to race an active run, and deletes only
130
- exactly marker-owned Supercov storage.
108
+ By default, `clean` removes all stored runs and the isolated build cache.
109
+ `--keep N` preserves the newest N runs. Cleanup only removes marker-owned
110
+ Supercov storage.
111
+
112
+ ## Read bundled documentation
113
+
114
+ ```sh
115
+ npx supercov docs
116
+ npx supercov docs getting-started
117
+ ```
131
118
 
132
119
  ## Environment variables
133
120
 
134
- | Variable | Effect |
121
+ | Variable | Use |
135
122
  | --- | --- |
136
- | `SUPERCOV_SOURCE_ROOTS` | Declares the authoritative first-party source scope, resolving ambiguity that would otherwise block a complete verdict. |
137
- | `SUPERCOV_TEST_KIND` | Declares the semantic level of the tests in this command, overriding every inference. |
123
+ | `SUPERCOV_SOURCE_ROOTS` | Declare the authoritative first-party source roots when automatic scope is ambiguous. |
124
+ | `SUPERCOV_TEST_KIND` | Declare the semantic level of the tests in the wrapped command. |
138
125
 
139
126
  ## Exit codes
140
127
 
141
128
  | Code | Meaning |
142
129
  | --- | --- |
143
130
  | `0` | The run or query succeeded. |
144
- | The test command's own code | A coverage run exits with the status of your command, so `supercov -- npm test` remains usable as a CI gate. |
145
- | `2` | Supercov itself failed: an unknown command, an unreadable run, an incompatible merge, or a lock conflict. |
131
+ | Test command's status | A coverage run preserves the wrapped command's exit status. |
132
+ | `2` | Supercov itself could not complete the request. |