supercov 2.0.1 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -22
- package/docs/agent-loop.md +8 -13
- package/docs/assertions.md +161 -75
- package/docs/cli.md +111 -59
- package/docs/coverage-model.md +5 -6
- package/docs/evidence.md +0 -11
- package/docs/getting-started.md +1 -1
- package/docs/performance.md +6 -14
- package/docs/quality.md +1 -1
- package/docs/security.md +43 -12
- package/docs/supported-suites.md +44 -48
- package/docs/troubleshooting.md +12 -5
- package/docs/workspace-isolation.md +23 -2
- package/package.json +33 -21
- package/runtime/javascript/atomic.mjs +46 -2
- package/runtime/javascript/ava.mjs +82 -0
- package/runtime/javascript/bootstrap.cjs +43 -0
- package/runtime/javascript/jest.cjs +52 -15
- package/runtime/javascript/jest.config.mjs +9 -1
- package/runtime/javascript/jestReporter.mjs +5 -6
- package/runtime/javascript/launchSupervisor.mjs +127 -15
- package/runtime/javascript/mocha.mjs +180 -0
- package/runtime/javascript/nodeTest.mjs +57 -3
- package/runtime/javascript/playwright.mjs +241 -32
- package/runtime/javascript/provenance.mjs +1 -1
- package/runtime/javascript/register.mjs +290 -2
- package/runtime/javascript/resolve-loader.mjs +40 -1
- package/runtime/javascript/runnerEvidence.mjs +3 -5
- package/runtime/javascript/runtime.mjs +551 -61
- package/runtime/javascript/tap.mjs +130 -0
- package/runtime/javascript/transport.mjs +7 -2
- package/runtime/javascript/vitest.mjs +5 -8
- package/runtime/javascript/vitestReporter.mjs +3 -10
- package/docs/assertion-agent.md +0 -150
- package/docs/assertion-evidence.md +0 -103
- package/docs/assertion-maps.md +0 -316
- package/schemas/assertions.schema.json +0 -276
package/docs/coverage-model.md
CHANGED
|
@@ -150,9 +150,8 @@ or generated output merely to make a warning disappear.
|
|
|
150
150
|
|
|
151
151
|
## Assertion percentage
|
|
152
152
|
|
|
153
|
-
When a run has
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
independently of `--filter`, `--kind` and `--runner
|
|
157
|
-
structural metrics above. See [assertion
|
|
158
|
-
freshness and incomplete-map rules.
|
|
153
|
+
When a run has been assessed with `assertions assess`, its regular summary also
|
|
154
|
+
shows assertion coverage: executed statements a test is judged to catch
|
|
155
|
+
breaking, divided by all assessed statements. This row always describes the
|
|
156
|
+
whole run, independently of `--filter`, `--kind` and `--runner`, and is separate
|
|
157
|
+
from the structural metrics above. See [assertion coverage](assertions.md).
|
package/docs/evidence.md
CHANGED
|
@@ -134,14 +134,3 @@ SUPERCOV_TEST_KIND=integration npx supercov -- npm test
|
|
|
134
134
|
|
|
135
135
|
Supported values are `unit`, `component`, `integration` and `e2e`. Use separate
|
|
136
136
|
runs when different suites need different classifications.
|
|
137
|
-
|
|
138
|
-
## When assertion evidence is missing
|
|
139
|
-
|
|
140
|
-
A test can make assertions without executing measured application code. It
|
|
141
|
-
might check static data or a dependency, or use work performed in shared setup.
|
|
142
|
-
Missing attribution across an asynchronous boundary can also leave a gap.
|
|
143
|
-
|
|
144
|
-
The assertion detail explains why a mapped node did or did not receive credit.
|
|
145
|
-
Use [Investigating assertion evidence](assertion-evidence.md) to distinguish
|
|
146
|
-
these cases. The report's runtime action-phase counts are separate from the
|
|
147
|
-
assertion map; zero action-phase lines does not mean zero asserted statements.
|
package/docs/getting-started.md
CHANGED
|
@@ -139,7 +139,7 @@ if anything is missing.
|
|
|
139
139
|
- for Go, Go 1.22 or newer;
|
|
140
140
|
- for Java and Kotlin, JDK 17 or newer, with Maven or Gradle;
|
|
141
141
|
- for Python, CPython 3.9 or newer with pytest or unittest;
|
|
142
|
-
- for Ruby, Ruby 3.
|
|
142
|
+
- for Ruby, Ruby 3.3 or newer with RSpec, Minitest, test-unit or Cucumber.
|
|
143
143
|
|
|
144
144
|
The CLI is a native binary. `npx supercov` picks the build for your operating
|
|
145
145
|
system and architecture, and nothing is compiled on install; the same binary is
|
package/docs/performance.md
CHANGED
|
@@ -130,18 +130,10 @@ Use `--dry-run` to preview cleanup. Keep enough run history for active reviews
|
|
|
130
130
|
and automation; remove the cache only when reclaiming space matters more than a
|
|
131
131
|
faster next run.
|
|
132
132
|
|
|
133
|
-
## Keep assertion
|
|
133
|
+
## Keep assertion assessments fast and cheap
|
|
134
134
|
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
the
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
For large maps, ask for a short page instead of the whole graph. Within one
|
|
141
|
-
flow, `--view nodes` or `--view edges` pages the details; `--compact` omits repeated
|
|
142
|
-
source text while keeping locations and credit reasons. See
|
|
143
|
-
[Investigating assertion evidence](assertion-evidence.md#read-a-large-flow).
|
|
144
|
-
|
|
145
|
-
The disposable `assertions.report.cache.json` and `assertions.summary.cache.json`
|
|
146
|
-
files live beside the map. A missing or damaged cache is rebuilt. Removing it affects the next query's speed, not
|
|
147
|
-
your saved explanations.
|
|
135
|
+
`assertions assess` asks only questions it has not answered before; answers live
|
|
136
|
+
in `.supercov/assertions/` and are reused across runs whenever the question is
|
|
137
|
+
the same. `--workers N` raises the requests in flight (default 8) for a faster,
|
|
138
|
+
slightly costlier pass. `--dry-run` prints the estimate without sending
|
|
139
|
+
anything.
|
package/docs/quality.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Understanding quality
|
|
2
2
|
|
|
3
|
-
`supercov quality` tells you what is in your code: twelve named properties,
|
|
3
|
+
`npx supercov quality` tells you what is in your code: twelve named properties,
|
|
4
4
|
checked file by file, so a score is never a number you have to take on faith.
|
|
5
5
|
|
|
6
6
|
Judgments come from [Jev](https://typesafe.ai), which answers typed questions
|
package/docs/security.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Security surface
|
|
2
2
|
|
|
3
|
-
`supercov security` tells you what security-relevant surface each file shows:
|
|
3
|
+
`npx supercov security` tells you what security-relevant surface each file shows:
|
|
4
4
|
twelve named checks, asked file by file, each mapped to the weakness classes it
|
|
5
5
|
stands for. There is no score. A file is clean, or it names what fired.
|
|
6
6
|
|
|
7
|
-
Judgments come from [Jev](https://typesafe.ai), the same way `supercov quality`
|
|
7
|
+
Judgments come from [Jev](https://typesafe.ai), the same way `npx supercov quality`
|
|
8
8
|
gets them. Set `TYPESAFE_API_KEY`, and optionally another provider, as described
|
|
9
9
|
in [quality](quality.md); reading a saved assessment never needs a key.
|
|
10
10
|
|
|
@@ -50,16 +50,47 @@ attempt measured on the labelled corpus fired on unguarded handlers the
|
|
|
50
50
|
application leaves open by design as often as on the ones it should not.
|
|
51
51
|
|
|
52
52
|
|
|
53
|
-
Measured with
|
|
54
|
-
140-repository labelled corpus,
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
53
|
+
Measured with 2.0.1 on the 72 held-out repositories of RealVuln, a
|
|
54
|
+
140-repository labelled corpus, scored by its own rule: F1 0.42 with 42%
|
|
55
|
+
precision and 43% recall, in a median of 10 seconds and about 5 cents a
|
|
56
|
+
repository. The strongest agent-based scanners score 0.69 to 0.71 but take
|
|
57
|
+
5 to 9 minutes and about $4 a repository; Semgrep's default rules score 0.11.
|
|
58
|
+
Redirects, injection, unsafe code execution and paths taken from a request
|
|
59
|
+
are found at 79 to 89% recall; authorisation (11%) and authentication (24%)
|
|
60
|
+
are the weakest, because the guard that decides them lives in another file.
|
|
61
|
+
Every scanner's results are at
|
|
62
|
+
[supercov.com/docs/security-benchmark](https://supercov.com/docs/security-benchmark).
|
|
63
|
+
A file too large for one request is read in windows at declaration
|
|
64
|
+
boundaries, or at line boundaries where Supercov has no parser for it, such as
|
|
65
|
+
a template.
|
|
66
|
+
|
|
67
|
+
## What is read
|
|
68
|
+
|
|
69
|
+
Security reads every source file, template and configuration file that the
|
|
70
|
+
conventions do not exclude, including files under no recognised source root,
|
|
71
|
+
because code a project did not declare is deployed as often as not.
|
|
72
|
+
`npx supercov security scope` lists exactly the files a run reads and why the
|
|
73
|
+
rest were left out. Three kinds are left out because they are not the code
|
|
74
|
+
under review:
|
|
75
|
+
|
|
76
|
+
- **Tests and generated output**, as for `quality`.
|
|
77
|
+
- **Vendored and minified JavaScript.** A minified file anywhere, and in a
|
|
78
|
+
served or vendor directory (`static/`, `public/`, `assets/`, `plugins/`,
|
|
79
|
+
`vendor/`, `lib/`) a file with a licence banner or a well-known library's
|
|
80
|
+
name, such as `jquery.flot.js` or `swagger-ui-bundle.js`.
|
|
81
|
+
- **JSON data.** A JSON file is read when its name or directory says it
|
|
82
|
+
configures something (`config.json`, `proxy.conf.json`, `appsettings.json`,
|
|
83
|
+
`secrets.json`, anything under `config/`), and not when it holds data such
|
|
84
|
+
as translations, schema snapshots or scanner output.
|
|
85
|
+
|
|
86
|
+
A directory called `target/` is build output only when a build that writes
|
|
87
|
+
there (Cargo, Maven, Gradle, sbt, Leiningen) is declared beside it; otherwise
|
|
88
|
+
it is read like any other. Name a file directly to read it whatever the scope
|
|
89
|
+
says: `npx supercov security static/js/app.min.js`.
|
|
90
|
+
|
|
91
|
+
A run that could not assess a few files still exits 0: each one is in the
|
|
92
|
+
report with its reason, and stderr says how many. It exits 2 only when no
|
|
93
|
+
file could be assessed.
|
|
63
94
|
## Across files
|
|
64
95
|
|
|
65
96
|
The second pass also labels every function of every file, on the same
|
package/docs/supported-suites.md
CHANGED
|
@@ -50,9 +50,11 @@ Supercov reports the level it actually observed. It does not guess.
|
|
|
50
50
|
| --- | --- |
|
|
51
51
|
| Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
|
|
52
52
|
| Vitest | Exact per test, with setup execution kept separate; Browser Mode included |
|
|
53
|
-
| Jest | Exact per test, including parameterized tests, with the user's own configuration, setup files and reporters kept
|
|
53
|
+
| Jest | Exact per test, including parameterized tests, with the user's own configuration, setup files and reporters kept |
|
|
54
54
|
| `node:test` | Exact per test |
|
|
55
|
-
|
|
|
55
|
+
| Mocha | Exact per test and retry, `--parallel` included, with setup execution kept separate |
|
|
56
|
+
| AVA | Exact per test, concurrent tests included, in worker threads or child processes |
|
|
57
|
+
| node-tap 15 and newer | Exact per subtest, nested and parallel (`t.jobs`) subtests included |
|
|
56
58
|
| Other Node-based runners | Aggregate when their processes remain visible to Supercov |
|
|
57
59
|
| Browser component runners without an adapter | Aggregate structural coverage |
|
|
58
60
|
|
|
@@ -81,9 +83,6 @@ obligations. `{value}` is reached exactly when the tree is, and
|
|
|
81
83
|
`onClick={() => save()}` is measured where the handler is called rather than
|
|
82
84
|
where it is created.
|
|
83
85
|
|
|
84
|
-
`expect.element(...)`, `expect.soft(...)` and `expect.poll(...)` are recognised
|
|
85
|
-
as assertions, so their passing occurrences are available to assertion maps.
|
|
86
|
-
|
|
87
86
|
### React and React Native
|
|
88
87
|
|
|
89
88
|
React components can use Testing Library with Vitest/jsdom, Vitest Browser Mode,
|
|
@@ -93,11 +92,10 @@ it does not measure Hermes, native modules, simulator/device execution, Detox or
|
|
|
93
92
|
Maestro. React SSR hydration is exercised in jsdom and Chromium; this does not
|
|
94
93
|
establish Next.js Server Components, streaming SSR or server actions.
|
|
95
94
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
checked. Keep your existing runner and matchers.
|
|
95
|
+
Assertion coverage treats a JSX expression as a statement: it asks whether a
|
|
96
|
+
test would notice the displayed value, accessible name, disabled state or error
|
|
97
|
+
message becoming undefined. A button being rendered does not establish that its
|
|
98
|
+
disabled state was checked. Keep your existing runner and matchers.
|
|
101
99
|
|
|
102
100
|
The [React verification example](https://github.com/supercorp-ai/supercov/tree/main/examples/react-verification)
|
|
103
101
|
shows a fully executed checkout whose weak tests accept four UI regressions,
|
|
@@ -118,11 +116,6 @@ Supercov runs the project's build inside the isolated copy before testing.
|
|
|
118
116
|
Keep using your normal test and build commands. Instrumentation does not
|
|
119
117
|
require changing the project's TypeScript settings.
|
|
120
118
|
|
|
121
|
-
For assertion maps, Node, Vitest, Jest and Playwright's Node-side assertions
|
|
122
|
-
supply supported passing-occurrence evidence. Custom assertion wrappers and
|
|
123
|
-
browser-side checks can have additional observation limits. See
|
|
124
|
-
[Assertion evidence](assertion-evidence.md) before interpreting a missing occurrence.
|
|
125
|
-
|
|
126
119
|
### Browsers, servers, and child processes
|
|
127
120
|
|
|
128
121
|
Playwright support includes Chromium, Firefox, and WebKit, along with pages,
|
|
@@ -169,6 +162,19 @@ modules. Statements, functions, branches, boolean decisions, loops and error
|
|
|
169
162
|
propagation are measured. Const contexts and macro expansions remain visible
|
|
170
163
|
with explicit measurement limitations.
|
|
171
164
|
|
|
165
|
+
Test code is not measured, as cargo-llvm-cov and cargo-tarpaulin leave it out:
|
|
166
|
+
files under a `tests`, `examples` or `benches` directory, files named
|
|
167
|
+
`tests.rs`, `*_tests.rs` or `*-tests.rs`, everything a test target or a
|
|
168
|
+
`#[cfg(test)]` module declares, and items marked `#[cfg(test)]`, `#[test]` or
|
|
169
|
+
`#[bench]`. Its assertions still mark which evidence a passing assertion
|
|
170
|
+
witnessed.
|
|
171
|
+
|
|
172
|
+
Neither is code the compiler never builds for the target: a `#[cfg(windows)]`
|
|
173
|
+
function on Linux, or one behind `#[cfg(not(target_os = "macos"))]` on a Mac,
|
|
174
|
+
is not counted. Only predicates the target settles (`unix`, `windows`,
|
|
175
|
+
`target_*`) are decided; an item behind a feature or a custom cfg is counted
|
|
176
|
+
either way.
|
|
177
|
+
|
|
172
178
|
Use the repository's normal flags after the wrapped command:
|
|
173
179
|
|
|
174
180
|
```sh
|
|
@@ -200,7 +206,8 @@ comprehensions, short-circuit operators, `match` cases and exception paths.
|
|
|
200
206
|
Child interpreters, threads and thread pools can retain the calling test's
|
|
201
207
|
identity. The report also distinguishes execution before a passing assertion
|
|
202
208
|
from later execution. These phase records alone do not prove which values the
|
|
203
|
-
assertion checks.
|
|
209
|
+
assertion checks; [assertion coverage](assertions.md) measures which executed
|
|
210
|
+
statements the pytest and unittest tests would catch breaking.
|
|
204
211
|
|
|
205
212
|
Interpreters launched with `-I`, `-E` or `-S` ignore the required startup hook
|
|
206
213
|
and are not measured. Code compiled from strings at runtime has no source
|
|
@@ -222,23 +229,21 @@ npx supercov -- python -m unittest
|
|
|
222
229
|
| Runner | Attribution | Current requirement |
|
|
223
230
|
| --- | --- | --- |
|
|
224
231
|
| RSpec | Exact example and before/example/after phase identity | Ruby 3.4 or newer for full measurement; run with `npx supercov -- rspec` or `bundle exec rspec` |
|
|
225
|
-
| Minitest (including Minitest::Spec and ActiveSupport::TestCase) | Exact test and setup/test/teardown identity;
|
|
232
|
+
| Minitest (including Minitest::Spec and ActiveSupport::TestCase) | Exact test and setup/test/teardown identity; skips recorded | `ruby -Itest ...`, `rake test`, `rails test` |
|
|
226
233
|
| test-unit | Exact test and setup/test/teardown identity; omissions and pendings recorded | `ruby -Itest ...`, `rake test` |
|
|
227
234
|
| parallel_tests, Rails process workers | Exact per worker process | Workers inherit the run through `RUBYOPT`; verified on a Rails app with bootsnap, Zeitwerk and two forked workers |
|
|
228
235
|
| Thread-parallel Minitest (`parallelize_me!`, `parallelize(with: :threads)`) | Probe observations exact per test; line, method and simple-branch observations made while phases overlapped go to the run, declared | |
|
|
229
236
|
| Cucumber | Exact scenario identity (`features/x.feature:LINE`), hook steps as setup/teardown | `cucumber`, `bundle exec cucumber` |
|
|
230
237
|
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
cannot settle the question as **undetermined** rather than unaffected, so
|
|
241
|
-
`--names` includes it in the set to run.
|
|
238
|
+
Every test is credited with every line it runs: each statement of an
|
|
239
|
+
instrumented file carries a probe. Code Supercov cannot instrument -- a file
|
|
240
|
+
it could not compile with its probes, or a `Ractor.new` block, where no probe
|
|
241
|
+
can run -- falls back to Ruby's one-shot line coverage, which reports a line
|
|
242
|
+
the first time it executes and never again. There a test is credited with the
|
|
243
|
+
lines it was first to reach; what it is not credited with is not evidence it
|
|
244
|
+
did not run the code. The run says so, per-test numbers there read "at least",
|
|
245
|
+
and `tests affected` reports a test whose own record cannot settle the
|
|
246
|
+
question as **undetermined**, so `--names` includes it in the set to run.
|
|
242
247
|
|
|
243
248
|
Your project runs in place with its own interpreter and bundle. Supercov loads
|
|
244
249
|
through `RUBYOPT`; application files on disk and their backtrace line numbers
|
|
@@ -246,11 +251,9 @@ stay unchanged. RSpec, Minitest and test-unit assertions can identify execution
|
|
|
246
251
|
before a passing assertion. That timing evidence alone does not show which
|
|
247
252
|
values the assertion checks.
|
|
248
253
|
|
|
249
|
-
Ruby 3.
|
|
254
|
+
Ruby 3.3 and newer support statement, method, branch and MC/DC measurement,
|
|
250
255
|
including loops, iterator blocks, short-circuit operators, pattern matching,
|
|
251
|
-
optional calls and exception paths.
|
|
252
|
-
and branch coverage; obligations requiring additional instrumentation are
|
|
253
|
-
reported as measurement limits.
|
|
256
|
+
optional calls and exception paths.
|
|
254
257
|
|
|
255
258
|
Some constructs have narrower coverage. Code in a non-main Ractor keeps line
|
|
256
259
|
coverage but may lack other observations. Certain nested-return expressions
|
|
@@ -300,7 +303,7 @@ which costs whatever its parallelism was worth — on sixteen I/O-bound JUnit
|
|
|
300
303
|
tests across fifteen cores, 3.6x. Either way the coverage totals are the same;
|
|
301
304
|
what changes is whether a line can be traced to the test that ran it.
|
|
302
305
|
|
|
303
|
-
`supercov runs latest` reports which you got, under `Attribution`.
|
|
306
|
+
`npx supercov runs latest` reports which you got, under `Attribution`.
|
|
304
307
|
|
|
305
308
|
## Go
|
|
306
309
|
|
|
@@ -332,7 +335,7 @@ measured the same way: `go test` runs both, and what they reach is real
|
|
|
332
335
|
coverage no test can claim.
|
|
333
336
|
|
|
334
337
|
Such a test is still recorded as having run, with its outcome, and credited
|
|
335
|
-
with no coverage. `supercov runs <id> test <name>` says that rather than
|
|
338
|
+
with no coverage. `npx supercov runs <id> test <name>` says that rather than
|
|
336
339
|
reporting zeroes, coverage percentages leave it out, and `tests affected` lists
|
|
337
340
|
it as **undetermined**: nothing can say a change missed it, so `--names`
|
|
338
341
|
includes it in the set to run. A package where every test calls `t.Parallel()`
|
|
@@ -347,10 +350,6 @@ no way to run code on `os.Exit`, and a `TestMain` need not reach the `m.Run()`
|
|
|
347
350
|
call Supercov wraps — `goleak.VerifyTestMain(m)` runs the suite and exits
|
|
348
351
|
itself. Without periodic writes such a run recorded nothing at all.
|
|
349
352
|
|
|
350
|
-
For assertion maps, `t.Error`, `t.Errorf`, `t.Fatal`, `t.Fatalf` and testify's
|
|
351
|
-
`assert` and `require` are inventoried. A Go test states its claim with an `if`
|
|
352
|
-
and reports the violation, so the report is the site.
|
|
353
|
-
|
|
354
353
|
A repository with several modules works either way it is laid out. A directory
|
|
355
354
|
with a `go.mod` of its own that no `go.work` names is a different module, and
|
|
356
355
|
`go test ./...` walks past it, so Supercov leaves it alone. A `go.work`
|
|
@@ -372,7 +371,7 @@ npx supercov -- go test ./core/... ./app/...
|
|
|
372
371
|
| Kotest | Exact per test, under the names Kotest itself reports | — |
|
|
373
372
|
| Spock | Exact per feature, under the names Spock itself reports | — |
|
|
374
373
|
| TestNG | Exact per test, each data-provider invocation its own | — |
|
|
375
|
-
| JUnit 4 alone | Exact per test, through Vintage — see below | Maven |
|
|
374
|
+
| JUnit 4 alone | Exact per test, through Vintage — see below | Maven or Gradle |
|
|
376
375
|
|
|
377
376
|
Multi-module builds are measured module by module: each compiles its own source
|
|
378
377
|
set and forks its own JVM, so each gets a runtime and records evidence of its
|
|
@@ -391,10 +390,11 @@ report name the same thing.
|
|
|
391
390
|
JUnit 4 on its own is not a platform engine and does not run on one. Maven and
|
|
392
391
|
Gradle choose a test provider from what is on the classpath, so putting the
|
|
393
392
|
platform there makes the build pick a provider that finds no engine and fail.
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
393
|
+
Supercov adds `junit-vintage-engine` to the copy — the platform's own way of
|
|
394
|
+
running exactly those JUnit 4 tests through the lifecycle it listens to — and
|
|
395
|
+
measures them; your own build still runs JUnit 4 as it did. In a Gradle build
|
|
396
|
+
the copy also switches each JUnit 4 module's test tasks to `useJUnitPlatform()`,
|
|
397
|
+
leaving any task the build already put on the platform or on TestNG as it was.
|
|
398
398
|
|
|
399
399
|
Supercov instruments an isolated copy and leaves your build file alone. In the
|
|
400
400
|
copy it adds a test-scoped `junit-platform-launcher`, because the listener is
|
|
@@ -423,15 +423,11 @@ same goes for a loop over a constant, `while (true)`, which can only go one
|
|
|
423
423
|
way. A Kotlin `contract { }` has to stay the first statement of its function,
|
|
424
424
|
so the probe that records the function being entered is written after it.
|
|
425
425
|
|
|
426
|
-
Every one of these is named in the run: ask for `supercov runs latest
|
|
426
|
+
Every one of these is named in the run: ask for `npx supercov runs latest
|
|
427
427
|
limitations` and each appears with its file, its line, and why it was left
|
|
428
428
|
alone. A source file the parser cannot read is declared there too, so a hole in
|
|
429
429
|
the denominator stays visible after the build log is gone.
|
|
430
430
|
|
|
431
|
-
For assertion maps, forms spelled `assertSomething`, `assertThat` or `fail` are
|
|
432
|
-
inventoried, which covers JUnit, TestNG, AssertJ, Hamcrest and kotlin.test.
|
|
433
|
-
Kotest's infix matchers are not.
|
|
434
|
-
|
|
435
431
|
```sh
|
|
436
432
|
npx supercov -- mvn test
|
|
437
433
|
npx supercov -- ./gradlew test
|
package/docs/troubleshooting.md
CHANGED
|
@@ -93,11 +93,18 @@ outside that loader. Supercov declares this boundary on the affected file,
|
|
|
93
93
|
including on Python 3.9–3.11. It does not silently rewrite the child command,
|
|
94
94
|
which could change imports, `sys.argv`, or process behavior.
|
|
95
95
|
|
|
96
|
-
A file
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
96
|
+
A file that tests also import stays measured: what the imports ran is
|
|
97
|
+
counted, and the file carries the limitation, because lines it shows as
|
|
98
|
+
uncovered may have run in the direct execution. Its numbers are a lower bound,
|
|
99
|
+
and the run reports its measurement as incomplete. A file that is only ever
|
|
100
|
+
run directly has nothing observed, and is left out of the measured denominator
|
|
101
|
+
and the uncovered-lines list; inspect its measurement limitations before
|
|
102
|
+
treating a zero denominator as complete coverage.
|
|
103
|
+
|
|
104
|
+
To have a script measured, run it through the import system:
|
|
105
|
+
`python -m package.module`, or, for a file whose name is not a module name
|
|
106
|
+
such as `bin/anti-drift-gate.py`, `runpy.run_path("bin/anti-drift-gate.py",
|
|
107
|
+
run_name="__main__")` from a test.
|
|
101
108
|
|
|
102
109
|
## Python subprocess output or timeout assertions fail
|
|
103
110
|
|
|
@@ -27,6 +27,19 @@ snapshots, calls a service, or changes a database, wrapping it does not remove
|
|
|
27
27
|
that behavior. The isolation guarantee applies to Supercov's instrumentation
|
|
28
28
|
and evidence work.
|
|
29
29
|
|
|
30
|
+
## Tools that judge the code
|
|
31
|
+
|
|
32
|
+
Some tools a test command runs read or measure the code instead of running it,
|
|
33
|
+
and the instrumented copy is not what you wrote:
|
|
34
|
+
|
|
35
|
+
- Linters, formatters and `tsc --noEmit` (ESLint, Prettier, standard, xo and
|
|
36
|
+
the like) read each file as you wrote it, and don't see `.supercov`.
|
|
37
|
+
- Coverage tools the command runs itself (tap, c8, nyc, Jest's and Vitest's
|
|
38
|
+
`--coverage`) still collect and report coverage. They measure the
|
|
39
|
+
instrumented copy, and report close to what they report without Supercov,
|
|
40
|
+
but not always exactly, so their thresholds are not checked, and the run
|
|
41
|
+
prints a line saying so. Supercov's own report has the run's coverage.
|
|
42
|
+
|
|
30
43
|
Files the wrapped command creates or changes inside the isolated workspace are
|
|
31
44
|
synced back to the project after the run, so `supercov -- npm test -- -u`
|
|
32
45
|
updates snapshots in the repository exactly as `npm test -- -u` would. Two
|
|
@@ -87,8 +100,16 @@ project lock, and does not scan for similarly named directories.
|
|
|
87
100
|
## Containers and remote workspaces
|
|
88
101
|
|
|
89
102
|
When a suite launches a container or VM from a mounted workspace, Supercov uses
|
|
90
|
-
the isolated workspace as the source presented to that environment.
|
|
91
|
-
|
|
103
|
+
the isolated workspace as the source presented to that environment. Where it
|
|
104
|
+
sees the launch call -- a command and its environment passed as an `env`,
|
|
105
|
+
`environment` or `envs` option, as a plain map after the command, or to a
|
|
106
|
+
builder method such as Testcontainers' `withEnvironment` -- it hands the guest
|
|
107
|
+
its settings with paths moved to the guest's mount, and the guest's coverage is
|
|
108
|
+
credited to the test that launched it. A guest started any other way, with none
|
|
109
|
+
of those settings, still runs the instrumented code: it reads the run's
|
|
110
|
+
settings from the mounted workspace itself, and its coverage counts for the run
|
|
111
|
+
as background coverage, credited to no test. The run summary's `Guests` line
|
|
112
|
+
says how many processes did that.
|
|
92
113
|
|
|
93
114
|
Dependencies stay out of the instrumented copy. The root `node_modules` is
|
|
94
115
|
linked entry by entry to the project's own, so an environment that mounts the
|
package/package.json
CHANGED
|
@@ -1,7 +1,24 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "supercov",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "3.0.1",
|
|
4
4
|
"description": "Coverage, security and code quality for coding agents",
|
|
5
|
+
"keywords": [
|
|
6
|
+
"coverage",
|
|
7
|
+
"code-coverage",
|
|
8
|
+
"test-coverage",
|
|
9
|
+
"branch-coverage",
|
|
10
|
+
"mcdc",
|
|
11
|
+
"vitest",
|
|
12
|
+
"jest",
|
|
13
|
+
"playwright",
|
|
14
|
+
"pytest",
|
|
15
|
+
"security",
|
|
16
|
+
"code-quality",
|
|
17
|
+
"coding-agents",
|
|
18
|
+
"claude-code",
|
|
19
|
+
"codex",
|
|
20
|
+
"cli"
|
|
21
|
+
],
|
|
5
22
|
"license": "MIT",
|
|
6
23
|
"repository": {
|
|
7
24
|
"type": "git",
|
|
@@ -20,9 +37,6 @@
|
|
|
20
37
|
"docs/assertions.md",
|
|
21
38
|
"docs/quality.md",
|
|
22
39
|
"docs/security.md",
|
|
23
|
-
"docs/assertion-evidence.md",
|
|
24
|
-
"docs/assertion-maps.md",
|
|
25
|
-
"docs/assertion-agent.md",
|
|
26
40
|
"docs/troubleshooting.md",
|
|
27
41
|
"docs/cli.md",
|
|
28
42
|
"docs/coverage-model.md",
|
|
@@ -31,7 +45,6 @@
|
|
|
31
45
|
"docs/verification.md",
|
|
32
46
|
"docs/performance.md",
|
|
33
47
|
"docs/workspace-isolation.md",
|
|
34
|
-
"schemas",
|
|
35
48
|
"README.md"
|
|
36
49
|
],
|
|
37
50
|
"engines": {
|
|
@@ -52,9 +65,6 @@
|
|
|
52
65
|
"build": "cargo build -p supercov",
|
|
53
66
|
"sync:rust-assets": "node scripts/sync-rust-package-assets.mjs",
|
|
54
67
|
"test:rust-assets": "node scripts/sync-rust-package-assets.mjs --check",
|
|
55
|
-
"check:rustc-backend-spike": "(cd spikes/rustc-backend && RUSTC_BOOTSTRAP=1 RUSTUP_TOOLCHAIN=1.95.0 cargo fmt --check && RUSTC_BOOTSTRAP=1 RUSTUP_TOOLCHAIN=1.95.0 cargo clippy --all-targets -- -D warnings)",
|
|
56
|
-
"test:rustc-backend-spike": "node scripts/rustc-backend-spike.mjs",
|
|
57
|
-
"test:rust-compiler-spikes": "cargo build -p supercov && node scripts/rust-libtest-companion-spike.mjs && node scripts/rust-async-attribution-spike.mjs && node scripts/rust-subprocess-attribution-spike.mjs && node scripts/rust-custom-harness-spike.mjs && node scripts/rust-libtest-builder-lifecycle-spike.mjs",
|
|
58
68
|
"test": "cargo test --workspace",
|
|
59
69
|
"test:react": "cargo build -p supercov && node scripts/rust-vitest-projects-integration.mjs && npm --prefix examples/react-verification ci && npm --prefix examples/react-verification run demo && node scripts/react-hydration-compatibility.mjs",
|
|
60
70
|
"test:release-tooling": "node --test tests/release/*.test.mjs",
|
|
@@ -71,7 +81,7 @@
|
|
|
71
81
|
"eval:skill": "claude plugin eval plugins/supercov --scaffold --trust-plugin --no-publish --model claude-sonnet-5 --judge-model claude-haiku-4-5 --threshold 0.66 --tag behavior trigger --allow-tools Bash Edit Write --max-cost-usd 15",
|
|
72
82
|
"eval:skill-triggers": "claude plugin eval plugins/supercov --scaffold --trust-plugin --no-publish --model claude-sonnet-5 --judge-model claude-haiku-4-5 --threshold 0.66 --tag trigger --ablation none --max-cost-usd 5",
|
|
73
83
|
"test:engine": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace && npm run test:runtime && npm run test:rust-assets && npm run test:engine-integrations",
|
|
74
|
-
"test:engine-integrations": "cargo build -p supercov && node scripts/html-report-integration.mjs && node scripts/rust-process-supervision.mjs && node scripts/rust-direct-node-integration.mjs && node scripts/rust-public-run-integration.mjs && node scripts/rust-embedded-runtime-integration.mjs && node scripts/rust-direct-vitest-integration.mjs && node scripts/rust-direct-jest-integration.mjs && node scripts/rust-direct-playwright-integration.mjs && node scripts/rust-custom-browser-playwright-integration.mjs && node scripts/rust-generic-esbuild-integration.mjs && node scripts/rust-generic-tsc-integration.mjs && node scripts/rust-generic-build-matrix.mjs && node scripts/rust-vite-playwright-integration.mjs",
|
|
84
|
+
"test:engine-integrations": "cargo build -p supercov && node scripts/html-report-integration.mjs && node scripts/rust-process-supervision.mjs && node scripts/rust-direct-node-integration.mjs && node scripts/rust-public-run-integration.mjs && node scripts/rust-embedded-runtime-integration.mjs && node scripts/rust-direct-vitest-integration.mjs && node scripts/rust-direct-jest-integration.mjs && node scripts/rust-direct-playwright-integration.mjs && node scripts/rust-custom-browser-playwright-integration.mjs && node scripts/rust-classic-browser-integration.mjs && node scripts/rust-top-packages-integration.mjs && node scripts/rust-generic-esbuild-integration.mjs && node scripts/rust-generic-tsc-integration.mjs && node scripts/rust-generic-build-matrix.mjs && node scripts/rust-vite-playwright-integration.mjs",
|
|
75
85
|
"test:platform": "cargo test --workspace && cargo build -p supercov && node scripts/rust-process-supervision.mjs && node scripts/workspace-crash-integration.mjs",
|
|
76
86
|
"test:native-package": "cargo build --release -p supercov && node scripts/native-package-integration.mjs && node scripts/native-release-set-integration.mjs",
|
|
77
87
|
"test:pypi-wheel": "node scripts/pypi-wheel-integration.mjs",
|
|
@@ -84,7 +94,7 @@
|
|
|
84
94
|
"test:test262": "cargo build --release -p supercov && node scripts/test262-equivalence.mjs",
|
|
85
95
|
"benchmark:check": "cargo build --release -p supercov && node scripts/rust-transform-benchmark.mjs",
|
|
86
96
|
"benchmark:python": "cargo build -p supercov && node scripts/python-benchmark.mjs",
|
|
87
|
-
"check": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && npm run test && npm run test:release-tooling && npm run test:runtime && npm run test:rust-assets && npm run test:agent-commands && npm run test:
|
|
97
|
+
"check": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && npm run test && npm run test:release-tooling && npm run test:runtime && npm run test:rust-assets && npm run test:agent-commands && npm run test:assertions && npm run test:sweep-target && npm run test:skip-guard && npm run test:windows-tls && node scripts/package-preflight.mjs && node scripts/verify-binstall-metadata.mjs --offline",
|
|
88
98
|
"release:check": "cargo clean && node scripts/sweep-target.mjs && npm run check && npm run test:engine && npm run test:go && npm run test:jvm && npm run test:launcher && npm run test:fixture && npm run test:watchdog && npm run test:engine-contract && npm run test:agent && npm run test:child-attribution && npm run test:host-loader && npm run test:python && npm run test:ruby-coverage && npm run test:rust-public-cargo && npm run test:packed-npx && npm run test:clang-mcdc && npm run benchmark:check",
|
|
89
99
|
"release:bump": "node scripts/bump-version.mjs",
|
|
90
100
|
"sweep": "node scripts/sweep-target.mjs",
|
|
@@ -98,9 +108,7 @@
|
|
|
98
108
|
"test:go": "cargo test --no-fail-fast -p supercov-engine --test go_frontend --test go_run",
|
|
99
109
|
"test:jvm": "cargo test --no-fail-fast -p supercov-engine --test jvm_frontend --test jvm_run -- --include-ignored --test-threads=1",
|
|
100
110
|
"test:rust-public-cargo": "cargo build -p supercov && node scripts/rust-public-cargo-integration.mjs",
|
|
101
|
-
"test:
|
|
102
|
-
"test:assertion-maps:js": "cargo build -p supercov && node scripts/assertion-map-schema.mjs --check && node scripts/assertion-map-js-integration.mjs && node scripts/assertion-dogfood-integration.mjs",
|
|
103
|
-
"sync:assertion-schema": "cargo build -p supercov && node scripts/assertion-map-schema.mjs",
|
|
111
|
+
"test:assertions": "cargo build -p supercov && node scripts/assertions-integration.mjs",
|
|
104
112
|
"docs:sync": "node scripts/sync-docs.mjs",
|
|
105
113
|
"docs:check": "node scripts/sync-docs.mjs --check",
|
|
106
114
|
"test:launcher": "go vet ./cmd/... && go test ./cmd/...",
|
|
@@ -110,14 +118,14 @@
|
|
|
110
118
|
"test:windows-tls": "node scripts/windows-tls-test.mjs"
|
|
111
119
|
},
|
|
112
120
|
"optionalDependencies": {
|
|
113
|
-
"@supercov/cli-darwin-arm64": "
|
|
114
|
-
"@supercov/cli-darwin-x64": "
|
|
115
|
-
"@supercov/cli-linux-arm64-gnu": "
|
|
116
|
-
"@supercov/cli-linux-arm64-musl": "
|
|
117
|
-
"@supercov/cli-linux-x64-gnu": "
|
|
118
|
-
"@supercov/cli-linux-x64-musl": "
|
|
119
|
-
"@supercov/cli-win32-arm64": "
|
|
120
|
-
"@supercov/cli-win32-x64": "
|
|
121
|
+
"@supercov/cli-darwin-arm64": "3.0.1",
|
|
122
|
+
"@supercov/cli-darwin-x64": "3.0.1",
|
|
123
|
+
"@supercov/cli-linux-arm64-gnu": "3.0.1",
|
|
124
|
+
"@supercov/cli-linux-arm64-musl": "3.0.1",
|
|
125
|
+
"@supercov/cli-linux-x64-gnu": "3.0.1",
|
|
126
|
+
"@supercov/cli-linux-x64-musl": "3.0.1",
|
|
127
|
+
"@supercov/cli-win32-arm64": "3.0.1",
|
|
128
|
+
"@supercov/cli-win32-x64": "3.0.1"
|
|
121
129
|
},
|
|
122
130
|
"peerDependencies": {
|
|
123
131
|
"@playwright/test": ">=1.55.0",
|
|
@@ -140,9 +148,13 @@
|
|
|
140
148
|
"@swc/core": "^1.16.1",
|
|
141
149
|
"@types/node": "^24.0.0",
|
|
142
150
|
"@vitest/browser-playwright": "4.1.11",
|
|
151
|
+
"@vitest/coverage-v8": "4.1.11",
|
|
152
|
+
"ava": "8.0.1",
|
|
143
153
|
"esbuild": "^0.28.2",
|
|
144
154
|
"expect": "30.4.1",
|
|
145
155
|
"jest": "30.5.1",
|
|
156
|
+
"libtap": "1.4.1",
|
|
157
|
+
"mocha": "12.0.3",
|
|
146
158
|
"next": "^16.3.2",
|
|
147
159
|
"react": "^19.2.4",
|
|
148
160
|
"react-dom": "^19.2.4",
|
|
@@ -1,6 +1,7 @@
|
|
|
1
|
-
import { appendFileSync, existsSync, mkdirSync, openSync, closeSync, fsyncSync, renameSync, rmSync, writeFileSync, } from "node:fs";
|
|
1
|
+
import { appendFileSync, existsSync, mkdirSync, openSync, closeSync, fsyncSync, renameSync, rmSync, statSync, writeFileSync, } from "node:fs";
|
|
2
2
|
import { randomUUID } from "node:crypto";
|
|
3
|
-
import { dirname } from "node:path";
|
|
3
|
+
import { dirname, resolve } from "node:path";
|
|
4
|
+
import { Buffer } from "node:buffer";
|
|
4
5
|
function fsyncDirectory(path) {
|
|
5
6
|
try {
|
|
6
7
|
const directory = openSync(path, "r");
|
|
@@ -64,3 +65,46 @@ export function atomicRenameSync(source, destination) {
|
|
|
64
65
|
renameSync(source, destination);
|
|
65
66
|
fsyncDirectory(dirname(destination));
|
|
66
67
|
}
|
|
68
|
+
/**
|
|
69
|
+
* Append one test's evidence to this writer's own journal.
|
|
70
|
+
*
|
|
71
|
+
* A file per test cost a directory, a temporary file, two fsyncs and a rename:
|
|
72
|
+
* jshttp/cookie's 63,740 tests had written 26,600 of them after seven
|
|
73
|
+
* minutes, where the suite alone takes two seconds. An append is one write the
|
|
74
|
+
* kernel keeps when the process is killed, which is the durability the rest of
|
|
75
|
+
* the workspace has; set SUPERCOV_DURABLE_EVIDENCE_EACH_TEST=1 to fsync each
|
|
76
|
+
* record as well. The reader already takes `*.mcdc.jsonl` journals, one record
|
|
77
|
+
* per line, as Playwright writes them.
|
|
78
|
+
*
|
|
79
|
+
* A journal is named by a token, never by the pid alone: worker threads share
|
|
80
|
+
* a pid, and VMs restored from one snapshot share the token too, so a journal
|
|
81
|
+
* that grew by anything but this writer's own appends is left to its other
|
|
82
|
+
* writer and a new one is started.
|
|
83
|
+
*/
|
|
84
|
+
const journals = new Map();
|
|
85
|
+
export function appendEvidenceRecord(directory, prefix, payload) {
|
|
86
|
+
const line = `${JSON.stringify(payload)}\n`;
|
|
87
|
+
let journal = journals.get(prefix);
|
|
88
|
+
if (journal) {
|
|
89
|
+
let size = -1;
|
|
90
|
+
try {
|
|
91
|
+
size = statSync(journal.path).size;
|
|
92
|
+
}
|
|
93
|
+
catch {
|
|
94
|
+
// A missing journal is started again below.
|
|
95
|
+
}
|
|
96
|
+
if (size !== journal.size)
|
|
97
|
+
journal = undefined;
|
|
98
|
+
}
|
|
99
|
+
if (!journal) {
|
|
100
|
+
const writer = (process.env.SUPERCOV_EXECUTION_LOG_SHARD ?? `pid-${process.pid}`)
|
|
101
|
+
.replace(/[^A-Za-z0-9_-]/g, "_");
|
|
102
|
+
journal = { path: resolve(directory, `${prefix}-${writer}-${randomUUID()}.mcdc.jsonl`), size: 0 };
|
|
103
|
+
journals.set(prefix, journal);
|
|
104
|
+
}
|
|
105
|
+
const append = process.env.SUPERCOV_DURABLE_EVIDENCE_EACH_TEST === "1"
|
|
106
|
+
? appendJsonLineDurableSync
|
|
107
|
+
: appendJsonLineSync;
|
|
108
|
+
append(journal.path, line);
|
|
109
|
+
journal.size += Buffer.byteLength(line);
|
|
110
|
+
}
|