supercov 2.0.1 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/README.md +19 -22
  2. package/docs/agent-loop.md +8 -13
  3. package/docs/assertions.md +161 -75
  4. package/docs/cli.md +111 -59
  5. package/docs/coverage-model.md +5 -6
  6. package/docs/evidence.md +0 -11
  7. package/docs/getting-started.md +1 -1
  8. package/docs/performance.md +6 -14
  9. package/docs/quality.md +1 -1
  10. package/docs/security.md +43 -12
  11. package/docs/supported-suites.md +44 -48
  12. package/docs/troubleshooting.md +12 -5
  13. package/docs/workspace-isolation.md +23 -2
  14. package/package.json +33 -21
  15. package/runtime/javascript/atomic.mjs +46 -2
  16. package/runtime/javascript/ava.mjs +82 -0
  17. package/runtime/javascript/bootstrap.cjs +43 -0
  18. package/runtime/javascript/jest.cjs +52 -15
  19. package/runtime/javascript/jest.config.mjs +9 -1
  20. package/runtime/javascript/jestReporter.mjs +5 -6
  21. package/runtime/javascript/launchSupervisor.mjs +127 -15
  22. package/runtime/javascript/mocha.mjs +180 -0
  23. package/runtime/javascript/nodeTest.mjs +57 -3
  24. package/runtime/javascript/playwright.mjs +241 -32
  25. package/runtime/javascript/provenance.mjs +1 -1
  26. package/runtime/javascript/register.mjs +290 -2
  27. package/runtime/javascript/resolve-loader.mjs +40 -1
  28. package/runtime/javascript/runnerEvidence.mjs +3 -5
  29. package/runtime/javascript/runtime.mjs +551 -61
  30. package/runtime/javascript/tap.mjs +130 -0
  31. package/runtime/javascript/transport.mjs +7 -2
  32. package/runtime/javascript/vitest.mjs +5 -8
  33. package/runtime/javascript/vitestReporter.mjs +3 -10
  34. package/docs/assertion-agent.md +0 -150
  35. package/docs/assertion-evidence.md +0 -103
  36. package/docs/assertion-maps.md +0 -316
  37. package/schemas/assertions.schema.json +0 -276
@@ -150,9 +150,8 @@ or generated output merely to make a warning disappear.
150
150
 
151
151
  ## Assertion percentage
152
152
 
153
- When a run has an `assertions.json` map, its regular summary also shows
154
- agent-assessed assertion coverage: credited measured statements divided by all
155
- measured statements. This row always describes the whole archived run,
156
- independently of `--filter`, `--kind` and `--runner`. It is separate from the
157
- structural metrics above. See [assertion maps](assertion-maps.md) for credit,
158
- freshness and incomplete-map rules.
153
+ When a run has been assessed with `assertions assess`, its regular summary also
154
+ shows assertion coverage: executed statements a test is judged to catch
155
+ breaking, divided by all assessed statements. This row always describes the
156
+ whole run, independently of `--filter`, `--kind` and `--runner`, and is separate
157
+ from the structural metrics above. See [assertion coverage](assertions.md).
package/docs/evidence.md CHANGED
@@ -134,14 +134,3 @@ SUPERCOV_TEST_KIND=integration npx supercov -- npm test
134
134
 
135
135
  Supported values are `unit`, `component`, `integration` and `e2e`. Use separate
136
136
  runs when different suites need different classifications.
137
-
138
- ## When assertion evidence is missing
139
-
140
- A test can make assertions without executing measured application code. It
141
- might check static data or a dependency, or use work performed in shared setup.
142
- Missing attribution across an asynchronous boundary can also leave a gap.
143
-
144
- The assertion detail explains why a mapped node did or did not receive credit.
145
- Use [Investigating assertion evidence](assertion-evidence.md) to distinguish
146
- these cases. The report's runtime action-phase counts are separate from the
147
- assertion map; zero action-phase lines does not mean zero asserted statements.
@@ -139,7 +139,7 @@ if anything is missing.
139
139
  - for Go, Go 1.22 or newer;
140
140
  - for Java and Kotlin, JDK 17 or newer, with Maven or Gradle;
141
141
  - for Python, CPython 3.9 or newer with pytest or unittest;
142
- - for Ruby, Ruby 3.4 or newer with RSpec, Minitest, test-unit or Cucumber (3.3 measures lines, methods and simple branches only).
142
+ - for Ruby, Ruby 3.3 or newer with RSpec, Minitest, test-unit or Cucumber.
143
143
 
144
144
  The CLI is a native binary. `npx supercov` picks the build for your operating
145
145
  system and architecture, and nothing is compiled on install; the same binary is
@@ -130,18 +130,10 @@ Use `--dry-run` to preview cleanup. Keep enough run history for active reviews
130
130
  and automation; remove the cache only when reclaiming space matters more than a
131
131
  faster next run.
132
132
 
133
- ## Keep assertion investigation fast
133
+ ## Keep assertion assessments fast and cheap
134
134
 
135
- Save `assertions.json` and query the run to see the updated score. Supercov does
136
- not rerun tests or call a model to calculate the report. Repeated queries reuse
137
- the previous assessment while checking that your source still matches the run.
138
- Editing the map automatically refreshes that assessment.
139
-
140
- For large maps, ask for a short page instead of the whole graph. Within one
141
- flow, `--view nodes` or `--view edges` pages the details; `--compact` omits repeated
142
- source text while keeping locations and credit reasons. See
143
- [Investigating assertion evidence](assertion-evidence.md#read-a-large-flow).
144
-
145
- The disposable `assertions.report.cache.json` and `assertions.summary.cache.json`
146
- files live beside the map. A missing or damaged cache is rebuilt. Removing it affects the next query's speed, not
147
- your saved explanations.
135
+ `assertions assess` asks only questions it has not answered before; answers live
136
+ in `.supercov/assertions/` and are reused across runs whenever the question is
137
+ the same. `--workers N` raises the requests in flight (default 8) for a faster,
138
+ slightly costlier pass. `--dry-run` prints the estimate without sending
139
+ anything.
package/docs/quality.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Understanding quality
2
2
 
3
- `supercov quality` tells you what is in your code: twelve named properties,
3
+ `npx supercov quality` tells you what is in your code: twelve named properties,
4
4
  checked file by file, so a score is never a number you have to take on faith.
5
5
 
6
6
  Judgments come from [Jev](https://typesafe.ai), which answers typed questions
package/docs/security.md CHANGED
@@ -1,10 +1,10 @@
1
1
  # Security surface
2
2
 
3
- `supercov security` tells you what security-relevant surface each file shows:
3
+ `npx supercov security` tells you what security-relevant surface each file shows:
4
4
  twelve named checks, asked file by file, each mapped to the weakness classes it
5
5
  stands for. There is no score. A file is clean, or it names what fired.
6
6
 
7
- Judgments come from [Jev](https://typesafe.ai), the same way `supercov quality`
7
+ Judgments come from [Jev](https://typesafe.ai), the same way `npx supercov quality`
8
8
  gets them. Set `TYPESAFE_API_KEY`, and optionally another provider, as described
9
9
  in [quality](quality.md); reading a saved assessment never needs a key.
10
10
 
@@ -50,16 +50,47 @@ attempt measured on the labelled corpus fired on unguarded handlers the
50
50
  application leaves open by design as often as on the ones it should not.
51
51
 
52
52
 
53
- Measured with this command on fifteen held-out repositories of a
54
- 140-repository labelled corpus, whole repositories, at finding level: F1
55
- 0.47 with 54% precision and 41% recall, where Semgrep scores 0.14, the
56
- general agentic LLM scanners 0.50 to 0.60, and the two leaders 0.76 and
57
- 0.77, at about two cents per repository against 30 cents to 4 dollars for
58
- the agentic scanners. Injection, secrets, path, redirect and mass
59
- assignment are found at 70 to 90% recall; authorisation and authentication
60
- at 10%, because the guard that decides them lives in another file. A file
61
- too large for one request is read in windows at declaration boundaries.
62
-
53
+ Measured with 2.0.1 on the 72 held-out repositories of RealVuln, a
54
+ 140-repository labelled corpus, scored by its own rule: F1 0.42 with 42%
55
+ precision and 43% recall, in a median of 10 seconds and about 5 cents a
56
+ repository. The strongest agent-based scanners score 0.69 to 0.71 but take
57
+ 5 to 9 minutes and about $4 a repository; Semgrep's default rules score 0.11.
58
+ Redirects, injection, unsafe code execution and paths taken from a request
59
+ are found at 79 to 89% recall; authorisation (11%) and authentication (24%)
60
+ are the weakest, because the guard that decides them lives in another file.
61
+ Every scanner's results are at
62
+ [supercov.com/docs/security-benchmark](https://supercov.com/docs/security-benchmark).
63
+ A file too large for one request is read in windows at declaration
64
+ boundaries, or at line boundaries where Supercov has no parser for it, such as
65
+ a template.
66
+
67
+ ## What is read
68
+
69
+ Security reads every source file, template and configuration file that the
70
+ conventions do not exclude, including files under no recognised source root,
71
+ because code a project did not declare is deployed as often as not.
72
+ `npx supercov security scope` lists exactly the files a run reads and why the
73
+ rest were left out. Three kinds are left out because they are not the code
74
+ under review:
75
+
76
+ - **Tests and generated output**, as for `quality`.
77
+ - **Vendored and minified JavaScript.** A minified file anywhere, and in a
78
+ served or vendor directory (`static/`, `public/`, `assets/`, `plugins/`,
79
+ `vendor/`, `lib/`) a file with a licence banner or a well-known library's
80
+ name, such as `jquery.flot.js` or `swagger-ui-bundle.js`.
81
+ - **JSON data.** A JSON file is read when its name or directory says it
82
+ configures something (`config.json`, `proxy.conf.json`, `appsettings.json`,
83
+ `secrets.json`, anything under `config/`), and not when it holds data such
84
+ as translations, schema snapshots or scanner output.
85
+
86
+ A directory called `target/` is build output only when a build that writes
87
+ there (Cargo, Maven, Gradle, sbt, Leiningen) is declared beside it; otherwise
88
+ it is read like any other. Name a file directly to read it whatever the scope
89
+ says: `npx supercov security static/js/app.min.js`.
90
+
91
+ A run that could not assess a few files still exits 0: each one is in the
92
+ report with its reason, and stderr says how many. It exits 2 only when no
93
+ file could be assessed.
63
94
  ## Across files
64
95
 
65
96
  The second pass also labels every function of every file, on the same
@@ -50,9 +50,11 @@ Supercov reports the level it actually observed. It does not guess.
50
50
  | --- | --- |
51
51
  | Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
52
52
  | Vitest | Exact per test, with setup execution kept separate; Browser Mode included |
53
- | Jest | Exact per test, including parameterized tests, with the user's own configuration, setup files and reporters kept; passing `expect` occurrences are identified for assertion maps |
53
+ | Jest | Exact per test, including parameterized tests, with the user's own configuration, setup files and reporters kept |
54
54
  | `node:test` | Exact per test |
55
- | AVA and Mocha | Aggregate structural coverage |
55
+ | Mocha | Exact per test and retry, `--parallel` included, with setup execution kept separate |
56
+ | AVA | Exact per test, concurrent tests included, in worker threads or child processes |
57
+ | node-tap 15 and newer | Exact per subtest, nested and parallel (`t.jobs`) subtests included |
56
58
  | Other Node-based runners | Aggregate when their processes remain visible to Supercov |
57
59
  | Browser component runners without an adapter | Aggregate structural coverage |
58
60
 
@@ -81,9 +83,6 @@ obligations. `{value}` is reached exactly when the tree is, and
81
83
  `onClick={() => save()}` is measured where the handler is called rather than
82
84
  where it is created.
83
85
 
84
- `expect.element(...)`, `expect.soft(...)` and `expect.poll(...)` are recognised
85
- as assertions, so their passing occurrences are available to assertion maps.
86
-
87
86
  ### React and React Native
88
87
 
89
88
  React components can use Testing Library with Vitest/jsdom, Vitest Browser Mode,
@@ -93,11 +92,10 @@ it does not measure Hermes, native modules, simulator/device execution, Detox or
93
92
  Maestro. React SSR hydration is exercised in jsdom and Chromium; this does not
94
93
  establish Next.js Server Components, streaming SSR or server actions.
95
94
 
96
- An assertion map can explain which displayed value, accessible name, disabled
97
- state or error message a test checks. JSX expression coverage and a passing
98
- assertion are evidence for reviewing that explanation, not automatic semantic
99
- proof. A button being rendered does not establish that its disabled state was
100
- checked. Keep your existing runner and matchers.
95
+ Assertion coverage treats a JSX expression as a statement: it asks whether a
96
+ test would notice the displayed value, accessible name, disabled state or error
97
+ message becoming undefined. A button being rendered does not establish that its
98
+ disabled state was checked. Keep your existing runner and matchers.
101
99
 
102
100
  The [React verification example](https://github.com/supercorp-ai/supercov/tree/main/examples/react-verification)
103
101
  shows a fully executed checkout whose weak tests accept four UI regressions,
@@ -118,11 +116,6 @@ Supercov runs the project's build inside the isolated copy before testing.
118
116
  Keep using your normal test and build commands. Instrumentation does not
119
117
  require changing the project's TypeScript settings.
120
118
 
121
- For assertion maps, Node, Vitest, Jest and Playwright's Node-side assertions
122
- supply supported passing-occurrence evidence. Custom assertion wrappers and
123
- browser-side checks can have additional observation limits. See
124
- [Assertion evidence](assertion-evidence.md) before interpreting a missing occurrence.
125
-
126
119
  ### Browsers, servers, and child processes
127
120
 
128
121
  Playwright support includes Chromium, Firefox, and WebKit, along with pages,
@@ -169,6 +162,19 @@ modules. Statements, functions, branches, boolean decisions, loops and error
169
162
  propagation are measured. Const contexts and macro expansions remain visible
170
163
  with explicit measurement limitations.
171
164
 
165
+ Test code is not measured, as cargo-llvm-cov and cargo-tarpaulin leave it out:
166
+ files under a `tests`, `examples` or `benches` directory, files named
167
+ `tests.rs`, `*_tests.rs` or `*-tests.rs`, everything a test target or a
168
+ `#[cfg(test)]` module declares, and items marked `#[cfg(test)]`, `#[test]` or
169
+ `#[bench]`. Its assertions still mark which evidence a passing assertion
170
+ witnessed.
171
+
172
+ Neither is code the compiler never builds for the target: a `#[cfg(windows)]`
173
+ function on Linux, or one behind `#[cfg(not(target_os = "macos"))]` on a Mac,
174
+ is not counted. Only predicates the target settles (`unix`, `windows`,
175
+ `target_*`) are decided; an item behind a feature or a custom cfg is counted
176
+ either way.
177
+
172
178
  Use the repository's normal flags after the wrapped command:
173
179
 
174
180
  ```sh
@@ -200,7 +206,8 @@ comprehensions, short-circuit operators, `match` cases and exception paths.
200
206
  Child interpreters, threads and thread pools can retain the calling test's
201
207
  identity. The report also distinguishes execution before a passing assertion
202
208
  from later execution. These phase records alone do not prove which values the
203
- assertion checks.
209
+ assertion checks; [assertion coverage](assertions.md) measures which executed
210
+ statements the pytest and unittest tests would catch breaking.
204
211
 
205
212
  Interpreters launched with `-I`, `-E` or `-S` ignore the required startup hook
206
213
  and are not measured. Code compiled from strings at runtime has no source
@@ -222,23 +229,21 @@ npx supercov -- python -m unittest
222
229
  | Runner | Attribution | Current requirement |
223
230
  | --- | --- | --- |
224
231
  | RSpec | Exact example and before/example/after phase identity | Ruby 3.4 or newer for full measurement; run with `npx supercov -- rspec` or `bundle exec rspec` |
225
- | Minitest (including Minitest::Spec and ActiveSupport::TestCase) | Exact test and setup/test/teardown identity; a test's coverage is a lower bound (see below); skips recorded | `ruby -Itest ...`, `rake test`, `rails test` |
232
+ | Minitest (including Minitest::Spec and ActiveSupport::TestCase) | Exact test and setup/test/teardown identity; skips recorded | `ruby -Itest ...`, `rake test`, `rails test` |
226
233
  | test-unit | Exact test and setup/test/teardown identity; omissions and pendings recorded | `ruby -Itest ...`, `rake test` |
227
234
  | parallel_tests, Rails process workers | Exact per worker process | Workers inherit the run through `RUBYOPT`; verified on a Rails app with bootsnap, Zeitwerk and two forked workers |
228
235
  | Thread-parallel Minitest (`parallelize_me!`, `parallelize(with: :threads)`) | Probe observations exact per test; line, method and simple-branch observations made while phases overlapped go to the run, declared | |
229
236
  | Cucumber | Exact scenario identity (`features/x.feature:LINE`), hook steps as setup/teardown | `cucumber`, `bundle exec cucumber` |
230
237
 
231
- Ruby reports a line the first time it executes and never again, which is what
232
- makes collecting coverage cheap enough to leave on. So a test is credited with
233
- the lines it was first to reach, and a later test running the same lines is
234
- credited with none of them: what a test is credited with is really its own, and
235
- what it is not credited with is not evidence it did not run the code.
236
-
237
- Totals are unaffected — every line is credited to exactly one test. What this
238
- changes is per-test reporting: `supercov runs <id> test <name>` gives its
239
- numbers as "at least", and `tests affected` reports a test whose own record
240
- cannot settle the question as **undetermined** rather than unaffected, so
241
- `--names` includes it in the set to run.
238
+ Every test is credited with every line it runs: each statement of an
239
+ instrumented file carries a probe. Code Supercov cannot instrument -- a file
240
+ it could not compile with its probes, or a `Ractor.new` block, where no probe
241
+ can run -- falls back to Ruby's one-shot line coverage, which reports a line
242
+ the first time it executes and never again. There a test is credited with the
243
+ lines it was first to reach; what it is not credited with is not evidence it
244
+ did not run the code. The run says so, per-test numbers there read "at least",
245
+ and `tests affected` reports a test whose own record cannot settle the
246
+ question as **undetermined**, so `--names` includes it in the set to run.
242
247
 
243
248
  Your project runs in place with its own interpreter and bundle. Supercov loads
244
249
  through `RUBYOPT`; application files on disk and their backtrace line numbers
@@ -246,11 +251,9 @@ stay unchanged. RSpec, Minitest and test-unit assertions can identify execution
246
251
  before a passing assertion. That timing evidence alone does not show which
247
252
  values the assertion checks.
248
253
 
249
- Ruby 3.4 and newer support statement, method, branch and MC/DC measurement,
254
+ Ruby 3.3 and newer support statement, method, branch and MC/DC measurement,
250
255
  including loops, iterator blocks, short-circuit operators, pattern matching,
251
- optional calls and exception paths. Ruby 3.3 supplies Ruby's own line, method
252
- and branch coverage; obligations requiring additional instrumentation are
253
- reported as measurement limits.
256
+ optional calls and exception paths.
254
257
 
255
258
  Some constructs have narrower coverage. Code in a non-main Ractor keeps line
256
259
  coverage but may lack other observations. Certain nested-return expressions
@@ -300,7 +303,7 @@ which costs whatever its parallelism was worth — on sixteen I/O-bound JUnit
300
303
  tests across fifteen cores, 3.6x. Either way the coverage totals are the same;
301
304
  what changes is whether a line can be traced to the test that ran it.
302
305
 
303
- `supercov runs latest` reports which you got, under `Attribution`.
306
+ `npx supercov runs latest` reports which you got, under `Attribution`.
304
307
 
305
308
  ## Go
306
309
 
@@ -332,7 +335,7 @@ measured the same way: `go test` runs both, and what they reach is real
332
335
  coverage no test can claim.
333
336
 
334
337
  Such a test is still recorded as having run, with its outcome, and credited
335
- with no coverage. `supercov runs <id> test <name>` says that rather than
338
+ with no coverage. `npx supercov runs <id> test <name>` says that rather than
336
339
  reporting zeroes, coverage percentages leave it out, and `tests affected` lists
337
340
  it as **undetermined**: nothing can say a change missed it, so `--names`
338
341
  includes it in the set to run. A package where every test calls `t.Parallel()`
@@ -347,10 +350,6 @@ no way to run code on `os.Exit`, and a `TestMain` need not reach the `m.Run()`
347
350
  call Supercov wraps — `goleak.VerifyTestMain(m)` runs the suite and exits
348
351
  itself. Without periodic writes such a run recorded nothing at all.
349
352
 
350
- For assertion maps, `t.Error`, `t.Errorf`, `t.Fatal`, `t.Fatalf` and testify's
351
- `assert` and `require` are inventoried. A Go test states its claim with an `if`
352
- and reports the violation, so the report is the site.
353
-
354
353
  A repository with several modules works either way it is laid out. A directory
355
354
  with a `go.mod` of its own that no `go.work` names is a different module, and
356
355
  `go test ./...` walks past it, so Supercov leaves it alone. A `go.work`
@@ -372,7 +371,7 @@ npx supercov -- go test ./core/... ./app/...
372
371
  | Kotest | Exact per test, under the names Kotest itself reports | — |
373
372
  | Spock | Exact per feature, under the names Spock itself reports | — |
374
373
  | TestNG | Exact per test, each data-provider invocation its own | — |
375
- | JUnit 4 alone | Exact per test, through Vintage — see below | Maven |
374
+ | JUnit 4 alone | Exact per test, through Vintage — see below | Maven or Gradle |
376
375
 
377
376
  Multi-module builds are measured module by module: each compiles its own source
378
377
  set and forks its own JVM, so each gets a runtime and records evidence of its
@@ -391,10 +390,11 @@ report name the same thing.
391
390
  JUnit 4 on its own is not a platform engine and does not run on one. Maven and
392
391
  Gradle choose a test provider from what is on the classpath, so putting the
393
392
  platform there makes the build pick a provider that finds no engine and fail.
394
- For a Maven module, Supercov adds `junit-vintage-engine` to the copy — the
395
- platform's own way of running exactly those JUnit 4 tests through the lifecycle
396
- it listens to — and measures them; your own build still runs JUnit 4 as it did.
397
- A Gradle module is left alone and told about rather than broken.
393
+ Supercov adds `junit-vintage-engine` to the copy — the platform's own way of
394
+ running exactly those JUnit 4 tests through the lifecycle it listens to — and
395
+ measures them; your own build still runs JUnit 4 as it did. In a Gradle build
396
+ the copy also switches each JUnit 4 module's test tasks to `useJUnitPlatform()`,
397
+ leaving any task the build already put on the platform or on TestNG as it was.
398
398
 
399
399
  Supercov instruments an isolated copy and leaves your build file alone. In the
400
400
  copy it adds a test-scoped `junit-platform-launcher`, because the listener is
@@ -423,15 +423,11 @@ same goes for a loop over a constant, `while (true)`, which can only go one
423
423
  way. A Kotlin `contract { }` has to stay the first statement of its function,
424
424
  so the probe that records the function being entered is written after it.
425
425
 
426
- Every one of these is named in the run: ask for `supercov runs latest
426
+ Every one of these is named in the run: ask for `npx supercov runs latest
427
427
  limitations` and each appears with its file, its line, and why it was left
428
428
  alone. A source file the parser cannot read is declared there too, so a hole in
429
429
  the denominator stays visible after the build log is gone.
430
430
 
431
- For assertion maps, forms spelled `assertSomething`, `assertThat` or `fail` are
432
- inventoried, which covers JUnit, TestNG, AssertJ, Hamcrest and kotlin.test.
433
- Kotest's infix matchers are not.
434
-
435
431
  ```sh
436
432
  npx supercov -- mvn test
437
433
  npx supercov -- ./gradlew test
@@ -93,11 +93,18 @@ outside that loader. Supercov declares this boundary on the affected file,
93
93
  including on Python 3.9–3.11. It does not silently rewrite the child command,
94
94
  which could change imports, `sys.argv`, or process behavior.
95
95
 
96
- A file can therefore have tests that touched it through an import, while a
97
- separate direct execution makes its overall coverage unmeasured. Those lines
98
- are excluded from both the measured denominator and the uncovered-lines list.
99
- Inspect the file's measurement limitations before treating a zero denominator
100
- as complete coverage.
96
+ A file that tests also import stays measured: what the imports ran is
97
+ counted, and the file carries the limitation, because lines it shows as
98
+ uncovered may have run in the direct execution. Its numbers are a lower bound,
99
+ and the run reports its measurement as incomplete. A file that is only ever
100
+ run directly has nothing observed, and is left out of the measured denominator
101
+ and the uncovered-lines list; inspect its measurement limitations before
102
+ treating a zero denominator as complete coverage.
103
+
104
+ To have a script measured, run it through the import system:
105
+ `python -m package.module`, or, for a file whose name is not a module name
106
+ such as `bin/anti-drift-gate.py`, `runpy.run_path("bin/anti-drift-gate.py",
107
+ run_name="__main__")` from a test.
101
108
 
102
109
  ## Python subprocess output or timeout assertions fail
103
110
 
@@ -27,6 +27,19 @@ snapshots, calls a service, or changes a database, wrapping it does not remove
27
27
  that behavior. The isolation guarantee applies to Supercov's instrumentation
28
28
  and evidence work.
29
29
 
30
+ ## Tools that judge the code
31
+
32
+ Some tools a test command runs read or measure the code instead of running it,
33
+ and the instrumented copy is not what you wrote:
34
+
35
+ - Linters, formatters and `tsc --noEmit` (ESLint, Prettier, standard, xo and
36
+ the like) read each file as you wrote it, and don't see `.supercov`.
37
+ - Coverage tools the command runs itself (tap, c8, nyc, Jest's and Vitest's
38
+ `--coverage`) still collect and report coverage. They measure the
39
+ instrumented copy, and report close to what they report without Supercov,
40
+ but not always exactly, so their thresholds are not checked, and the run
41
+ prints a line saying so. Supercov's own report has the run's coverage.
42
+
30
43
  Files the wrapped command creates or changes inside the isolated workspace are
31
44
  synced back to the project after the run, so `supercov -- npm test -- -u`
32
45
  updates snapshots in the repository exactly as `npm test -- -u` would. Two
@@ -87,8 +100,16 @@ project lock, and does not scan for similarly named directories.
87
100
  ## Containers and remote workspaces
88
101
 
89
102
  When a suite launches a container or VM from a mounted workspace, Supercov uses
90
- the isolated workspace as the source presented to that environment. The runtime
91
- must be able to cross the launch boundary and return evidence.
103
+ the isolated workspace as the source presented to that environment. Where it
104
+ sees the launch call -- a command and its environment passed as an `env`,
105
+ `environment` or `envs` option, as a plain map after the command, or to a
106
+ builder method such as Testcontainers' `withEnvironment` -- it hands the guest
107
+ its settings with paths moved to the guest's mount, and the guest's coverage is
108
+ credited to the test that launched it. A guest started any other way, with none
109
+ of those settings, still runs the instrumented code: it reads the run's
110
+ settings from the mounted workspace itself, and its coverage counts for the run
111
+ as background coverage, credited to no test. The run summary's `Guests` line
112
+ says how many processes did that.
92
113
 
93
114
  Dependencies stay out of the instrumented copy. The root `node_modules` is
94
115
  linked entry by entry to the project's own, so an environment that mounts the
package/package.json CHANGED
@@ -1,7 +1,24 @@
1
1
  {
2
2
  "name": "supercov",
3
- "version": "2.0.1",
3
+ "version": "3.0.0",
4
4
  "description": "Coverage, security and code quality for coding agents",
5
+ "keywords": [
6
+ "coverage",
7
+ "code-coverage",
8
+ "test-coverage",
9
+ "branch-coverage",
10
+ "mcdc",
11
+ "vitest",
12
+ "jest",
13
+ "playwright",
14
+ "pytest",
15
+ "security",
16
+ "code-quality",
17
+ "coding-agents",
18
+ "claude-code",
19
+ "codex",
20
+ "cli"
21
+ ],
5
22
  "license": "MIT",
6
23
  "repository": {
7
24
  "type": "git",
@@ -20,9 +37,6 @@
20
37
  "docs/assertions.md",
21
38
  "docs/quality.md",
22
39
  "docs/security.md",
23
- "docs/assertion-evidence.md",
24
- "docs/assertion-maps.md",
25
- "docs/assertion-agent.md",
26
40
  "docs/troubleshooting.md",
27
41
  "docs/cli.md",
28
42
  "docs/coverage-model.md",
@@ -31,7 +45,6 @@
31
45
  "docs/verification.md",
32
46
  "docs/performance.md",
33
47
  "docs/workspace-isolation.md",
34
- "schemas",
35
48
  "README.md"
36
49
  ],
37
50
  "engines": {
@@ -52,9 +65,6 @@
52
65
  "build": "cargo build -p supercov",
53
66
  "sync:rust-assets": "node scripts/sync-rust-package-assets.mjs",
54
67
  "test:rust-assets": "node scripts/sync-rust-package-assets.mjs --check",
55
- "check:rustc-backend-spike": "(cd spikes/rustc-backend && RUSTC_BOOTSTRAP=1 RUSTUP_TOOLCHAIN=1.95.0 cargo fmt --check && RUSTC_BOOTSTRAP=1 RUSTUP_TOOLCHAIN=1.95.0 cargo clippy --all-targets -- -D warnings)",
56
- "test:rustc-backend-spike": "node scripts/rustc-backend-spike.mjs",
57
- "test:rust-compiler-spikes": "cargo build -p supercov && node scripts/rust-libtest-companion-spike.mjs && node scripts/rust-async-attribution-spike.mjs && node scripts/rust-subprocess-attribution-spike.mjs && node scripts/rust-custom-harness-spike.mjs && node scripts/rust-libtest-builder-lifecycle-spike.mjs",
58
68
  "test": "cargo test --workspace",
59
69
  "test:react": "cargo build -p supercov && node scripts/rust-vitest-projects-integration.mjs && npm --prefix examples/react-verification ci && npm --prefix examples/react-verification run demo && node scripts/react-hydration-compatibility.mjs",
60
70
  "test:release-tooling": "node --test tests/release/*.test.mjs",
@@ -71,7 +81,7 @@
71
81
  "eval:skill": "claude plugin eval plugins/supercov --scaffold --trust-plugin --no-publish --model claude-sonnet-5 --judge-model claude-haiku-4-5 --threshold 0.66 --tag behavior trigger --allow-tools Bash Edit Write --max-cost-usd 15",
72
82
  "eval:skill-triggers": "claude plugin eval plugins/supercov --scaffold --trust-plugin --no-publish --model claude-sonnet-5 --judge-model claude-haiku-4-5 --threshold 0.66 --tag trigger --ablation none --max-cost-usd 5",
73
83
  "test:engine": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace && npm run test:runtime && npm run test:rust-assets && npm run test:engine-integrations",
74
- "test:engine-integrations": "cargo build -p supercov && node scripts/html-report-integration.mjs && node scripts/rust-process-supervision.mjs && node scripts/rust-direct-node-integration.mjs && node scripts/rust-public-run-integration.mjs && node scripts/rust-embedded-runtime-integration.mjs && node scripts/rust-direct-vitest-integration.mjs && node scripts/rust-direct-jest-integration.mjs && node scripts/rust-direct-playwright-integration.mjs && node scripts/rust-custom-browser-playwright-integration.mjs && node scripts/rust-generic-esbuild-integration.mjs && node scripts/rust-generic-tsc-integration.mjs && node scripts/rust-generic-build-matrix.mjs && node scripts/rust-vite-playwright-integration.mjs",
84
+ "test:engine-integrations": "cargo build -p supercov && node scripts/html-report-integration.mjs && node scripts/rust-process-supervision.mjs && node scripts/rust-direct-node-integration.mjs && node scripts/rust-public-run-integration.mjs && node scripts/rust-embedded-runtime-integration.mjs && node scripts/rust-direct-vitest-integration.mjs && node scripts/rust-direct-jest-integration.mjs && node scripts/rust-direct-playwright-integration.mjs && node scripts/rust-custom-browser-playwright-integration.mjs && node scripts/rust-classic-browser-integration.mjs && node scripts/rust-top-packages-integration.mjs && node scripts/rust-generic-esbuild-integration.mjs && node scripts/rust-generic-tsc-integration.mjs && node scripts/rust-generic-build-matrix.mjs && node scripts/rust-vite-playwright-integration.mjs",
75
85
  "test:platform": "cargo test --workspace && cargo build -p supercov && node scripts/rust-process-supervision.mjs && node scripts/workspace-crash-integration.mjs",
76
86
  "test:native-package": "cargo build --release -p supercov && node scripts/native-package-integration.mjs && node scripts/native-release-set-integration.mjs",
77
87
  "test:pypi-wheel": "node scripts/pypi-wheel-integration.mjs",
@@ -84,7 +94,7 @@
84
94
  "test:test262": "cargo build --release -p supercov && node scripts/test262-equivalence.mjs",
85
95
  "benchmark:check": "cargo build --release -p supercov && node scripts/rust-transform-benchmark.mjs",
86
96
  "benchmark:python": "cargo build -p supercov && node scripts/python-benchmark.mjs",
87
- "check": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && npm run test && npm run test:release-tooling && npm run test:runtime && npm run test:rust-assets && npm run test:agent-commands && npm run test:assertion-maps && npm run test:sweep-target && npm run test:skip-guard && npm run test:windows-tls && node scripts/package-preflight.mjs && node scripts/verify-binstall-metadata.mjs --offline",
97
+ "check": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && npm run test && npm run test:release-tooling && npm run test:runtime && npm run test:rust-assets && npm run test:agent-commands && npm run test:assertions && npm run test:sweep-target && npm run test:skip-guard && npm run test:windows-tls && node scripts/package-preflight.mjs && node scripts/verify-binstall-metadata.mjs --offline",
88
98
  "release:check": "cargo clean && node scripts/sweep-target.mjs && npm run check && npm run test:engine && npm run test:go && npm run test:jvm && npm run test:launcher && npm run test:fixture && npm run test:watchdog && npm run test:engine-contract && npm run test:agent && npm run test:child-attribution && npm run test:host-loader && npm run test:python && npm run test:ruby-coverage && npm run test:rust-public-cargo && npm run test:packed-npx && npm run test:clang-mcdc && npm run benchmark:check",
89
99
  "release:bump": "node scripts/bump-version.mjs",
90
100
  "sweep": "node scripts/sweep-target.mjs",
@@ -98,9 +108,7 @@
98
108
  "test:go": "cargo test --no-fail-fast -p supercov-engine --test go_frontend --test go_run",
99
109
  "test:jvm": "cargo test --no-fail-fast -p supercov-engine --test jvm_frontend --test jvm_run -- --include-ignored --test-threads=1",
100
110
  "test:rust-public-cargo": "cargo build -p supercov && node scripts/rust-public-cargo-integration.mjs",
101
- "test:assertion-maps": "cargo build -p supercov && node scripts/assertion-map-schema.mjs --check && node scripts/assertion-map-integration.mjs && node scripts/assertion-map-js-integration.mjs && node scripts/assertion-dogfood-integration.mjs",
102
- "test:assertion-maps:js": "cargo build -p supercov && node scripts/assertion-map-schema.mjs --check && node scripts/assertion-map-js-integration.mjs && node scripts/assertion-dogfood-integration.mjs",
103
- "sync:assertion-schema": "cargo build -p supercov && node scripts/assertion-map-schema.mjs",
111
+ "test:assertions": "cargo build -p supercov && node scripts/assertions-integration.mjs",
104
112
  "docs:sync": "node scripts/sync-docs.mjs",
105
113
  "docs:check": "node scripts/sync-docs.mjs --check",
106
114
  "test:launcher": "go vet ./cmd/... && go test ./cmd/...",
@@ -110,14 +118,14 @@
110
118
  "test:windows-tls": "node scripts/windows-tls-test.mjs"
111
119
  },
112
120
  "optionalDependencies": {
113
- "@supercov/cli-darwin-arm64": "2.0.1",
114
- "@supercov/cli-darwin-x64": "2.0.1",
115
- "@supercov/cli-linux-arm64-gnu": "2.0.1",
116
- "@supercov/cli-linux-arm64-musl": "2.0.1",
117
- "@supercov/cli-linux-x64-gnu": "2.0.1",
118
- "@supercov/cli-linux-x64-musl": "2.0.1",
119
- "@supercov/cli-win32-arm64": "2.0.1",
120
- "@supercov/cli-win32-x64": "2.0.1"
121
+ "@supercov/cli-darwin-arm64": "3.0.0",
122
+ "@supercov/cli-darwin-x64": "3.0.0",
123
+ "@supercov/cli-linux-arm64-gnu": "3.0.0",
124
+ "@supercov/cli-linux-arm64-musl": "3.0.0",
125
+ "@supercov/cli-linux-x64-gnu": "3.0.0",
126
+ "@supercov/cli-linux-x64-musl": "3.0.0",
127
+ "@supercov/cli-win32-arm64": "3.0.0",
128
+ "@supercov/cli-win32-x64": "3.0.0"
121
129
  },
122
130
  "peerDependencies": {
123
131
  "@playwright/test": ">=1.55.0",
@@ -140,9 +148,13 @@
140
148
  "@swc/core": "^1.16.1",
141
149
  "@types/node": "^24.0.0",
142
150
  "@vitest/browser-playwright": "4.1.11",
151
+ "@vitest/coverage-v8": "4.1.11",
152
+ "ava": "8.0.1",
143
153
  "esbuild": "^0.28.2",
144
154
  "expect": "30.4.1",
145
155
  "jest": "30.5.1",
156
+ "libtap": "1.4.1",
157
+ "mocha": "12.0.3",
146
158
  "next": "^16.3.2",
147
159
  "react": "^19.2.4",
148
160
  "react-dom": "^19.2.4",
@@ -1,6 +1,7 @@
1
- import { appendFileSync, existsSync, mkdirSync, openSync, closeSync, fsyncSync, renameSync, rmSync, writeFileSync, } from "node:fs";
1
+ import { appendFileSync, existsSync, mkdirSync, openSync, closeSync, fsyncSync, renameSync, rmSync, statSync, writeFileSync, } from "node:fs";
2
2
  import { randomUUID } from "node:crypto";
3
- import { dirname } from "node:path";
3
+ import { dirname, resolve } from "node:path";
4
+ import { Buffer } from "node:buffer";
4
5
  function fsyncDirectory(path) {
5
6
  try {
6
7
  const directory = openSync(path, "r");
@@ -64,3 +65,46 @@ export function atomicRenameSync(source, destination) {
64
65
  renameSync(source, destination);
65
66
  fsyncDirectory(dirname(destination));
66
67
  }
68
+ /**
69
+ * Append one test's evidence to this writer's own journal.
70
+ *
71
+ * A file per test cost a directory, a temporary file, two fsyncs and a rename:
72
+ * jshttp/cookie's 63,740 tests had written 26,600 of them after seven
73
+ * minutes, where the suite alone takes two seconds. An append is one write the
74
+ * kernel keeps when the process is killed, which is the durability the rest of
75
+ * the workspace has; set SUPERCOV_DURABLE_EVIDENCE_EACH_TEST=1 to fsync each
76
+ * record as well. The reader already takes `*.mcdc.jsonl` journals, one record
77
+ * per line, as Playwright writes them.
78
+ *
79
+ * A journal is named by a token, never by the pid alone: worker threads share
80
+ * a pid, and VMs restored from one snapshot share the token too, so a journal
81
+ * that grew by anything but this writer's own appends is left to its other
82
+ * writer and a new one is started.
83
+ */
84
+ const journals = new Map();
85
+ export function appendEvidenceRecord(directory, prefix, payload) {
86
+ const line = `${JSON.stringify(payload)}\n`;
87
+ let journal = journals.get(prefix);
88
+ if (journal) {
89
+ let size = -1;
90
+ try {
91
+ size = statSync(journal.path).size;
92
+ }
93
+ catch {
94
+ // A missing journal is started again below.
95
+ }
96
+ if (size !== journal.size)
97
+ journal = undefined;
98
+ }
99
+ if (!journal) {
100
+ const writer = (process.env.SUPERCOV_EXECUTION_LOG_SHARD ?? `pid-${process.pid}`)
101
+ .replace(/[^A-Za-z0-9_-]/g, "_");
102
+ journal = { path: resolve(directory, `${prefix}-${writer}-${randomUUID()}.mcdc.jsonl`), size: 0 };
103
+ journals.set(prefix, journal);
104
+ }
105
+ const append = process.env.SUPERCOV_DURABLE_EVIDENCE_EACH_TEST === "1"
106
+ ? appendJsonLineDurableSync
107
+ : appendJsonLineSync;
108
+ append(journal.path, line);
109
+ journal.size += Buffer.byteLength(line);
110
+ }