supercov 0.0.21 → 0.0.23

package/docs/performance.md

@@ -1,123 +1,74 @@
 # Performance and storage
 
-Supercov separates timings that it can measure safely from overhead that
-requires an explicit control run. It never runs a user's command a second time
-automatically: an arbitrary test command can write data, call paid services, or
-be intentionally non-repeatable.
+Test execution usually dominates a Supercov run. Supercov records the other
+phases separately so you can see whether time is going into workspace setup,
+instrumentation, the test command, or evidence publication.
 
-## Per-run measurements
+## Read run timings
 
-Every coverage run prints and stores monotonic durations for:
+```sh
+npx supercov runs --limit 5
+npx supercov runs latest
+```
+
+Each run records:
 
 | Phase | Includes |
 | --- | --- |
-| `initializationMs` | recovery, locking, project discovery, and integrity fingerprints |
-| `workspacePreparationMs` | transactional refresh of the isolated namespace |
-| `adapterSetupMs` | generated adapters, configs, manifests, and runtime files |
-| `instrumentedBuildMs` | the coverage-aware build/direct pass, or near-zero when an exact-fingerprint build is reused |
-| `testCommandMs` | the user's unchanged command, including any runner or remote infrastructure latency |
-| `evidencePublicationMs` | evidence collection, validation/summary analysis, lossless archive generation, and atomic run staging |
+| Initialization | Recovery, locking, project discovery, and fingerprints |
+| Workspace preparation | Refreshing the isolated project workspace |
+| Adapter setup | Preparing runner integration and runtime files |
+| Instrumented build | Building instrumented source, or near-zero on an exact cache hit |
+| Test command | The wrapped command, including runner and remote latency |
+| Evidence publication | Validation, archive creation, summary analysis, and atomic publication |
 
-The fields are stored in `.supercov/runs/<run-id>/run.json` and returned by
-`supercov runs --json`. Total duration is stored separately as `durationMs`.
+The same fields are available in `run.json` and in `runs --json` when an
+integration needs machine-readable timings.
 
-The non-test phases are not automatically labelled “overhead.” For example, a
-test script that ordinarily performs its own build may overlap work with the
-instrumented-build phase. True end-to-end overhead must compare equivalent
-cold runs or equivalent warm runs of the same command.
+## Keep repeated runs fast
 
-## Reproducible comparison
+- Use the same complete command for the baseline and final verification.
+- Let the isolated build cache survive between passes.
+- Avoid changing dependencies, build configuration, or toolchains in the
+  middle of a coverage loop unless the test requires it.
+- Use a focused test command while iterating, then finish with the full suite.
+- Query the stored run instead of rerunning merely to inspect a different view.
 
-Run the command without and with Supercov under the same cache state. Use at
-least three alternating pairs and report the medians. Never compare a cold VM,
-browser, package-manager, or build-cache run with a warm one.
+Supercov reuses an instrumented build only when the relevant source,
+configuration, dependencies, toolchain, build mode, and instrumenter identity
+match exactly. A mismatch causes a fresh build rather than risking stale
+coverage.
+
+## Measure end-to-end overhead
+
+Supercov never runs the test command a second time automatically because tests
+may write data, call paid services, or be intentionally non-repeatable. To
+measure overhead, compare the original and wrapped command under the same cache
+state:
 
 ```sh
 /usr/bin/time -p npm test
 /usr/bin/time -p npx supercov -- npm test
 ```
 
-Package acquisition is a separate user-interface cost. A cold `npx` download
-depends on the registry and network; a cached `npx` resolution should be
-reported separately from Supercov's recorded phases.
-
-## Reference measurement, not a guarantee
-
-On 2026-08-26, the 30-test Essential SEO Playwright suite running across seven
-cycle-restored VMs produced a 43.02 s uncovered control and two valid warm
-Supercov measurements of 46.30 s and 40.20 s. The slower covered sample is
-1.076x the control and therefore remains below the current 1.1x realistic-suite
-target; the faster result illustrates why multiple alternating pairs are still
-required for a statistically stable benchmark. The 40.20 s run spent 38.63 s
-inside the unchanged test command and 1.04 s total on initialization, workspace
-refresh, adapter restoration and evidence publication. It attributed 6,727
-server records across 30 remote launches with zero corrupt evidence records.
-
-VM-image creation is not included in these warm comparisons. A changed engine
-fingerprint intentionally caused a one-time 119 s rebake before the first
-post-change run; comparing that cold run with a warm control would be invalid.
-
-On 2026-08-24, the 29-test Essential SEO offline suite on the development Mac
-produced this warm pair:
-
-| Measurement | Duration |
-| --- | ---: |
-| unchanged command | 39.57 s |
-| Supercov total | 45.38 s |
-| end-to-end difference | +5.81 s (+14.7%) |
-| initialization | 0.06 s |
-| workspace preparation | 0.35 s |
-| adapter setup | 0.05 s |
-| instrumented build | 4.87 s |
-| test command inside Supercov | 39.60 s |
-| evidence/report preparation (historical format) | 0.40 s |
-
-The test-command durations were effectively identical in this pair. The extra
-instrumented build accounted for about 84% of the measured difference. A
-seven-sample isolated workspace refresh had a 270 ms median and 447 ms maximum.
-The built output grew from 2,781,273 to 3,094,366 logical bytes (+11.3%). A
-cached `npx supercov help` added a 686 ms median over direct CLI startup; the
-first observed `npx` resolution took 2.22 s.
-
-Cold VM-image runs were 170.34 s without Supercov and 175.44 s with Supercov in
-the same session, but a single cold pair is too noisy for a general percentage.
-Both spent approximately 124 seconds preparing their VM image.
-
-Before raw-evidence-only storage, the reference run retained 4.5 MB of reports
-and 1.7 MB across 178 loose evidence files. Its canonical compressed JSON was
-0.9 MB. The execution evidence alone packed to about 121 KiB; current archives
-also embed the exact denominator manifest and are the sole coverage artifact.
-Every CLI query derives its view from the archive. The first query builds a
-disposable integrity-bound binary index; later queries reuse it while the run
-identity remains valid. The index is an implementation detail and can always be
-deleted and reconstructed.
-These numbers are application- and filesystem-specific optimization baselines.
-An exact matching Vite build is also reused across runs, removing the measured
-4.87-second repeated build; any source/configuration/toolchain-key change falls
-back to a fresh isolated build.
-
-The evidence-only Essential SEO validation packed the exact manifest plus 178
-execution-evidence files (2.66 MB uncompressed) into a 248 KiB archive. With
-the 4 KiB `run.json`, the complete immutable run occupies 252 KiB and contains
-no derived report. Its identical warm 29-test run recorded 0 ms for the build
-phase and 40.49 seconds total: 0.07 seconds initialization, 0.36 seconds
-workspace refresh, 0.05 seconds adapter setup, 39.84 seconds in the unchanged
-test command, and 0.12 seconds evidence validation/archive publication. Fresh
-process queries for summary, files, and gaps each took 0.16–0.20 seconds on
-this run before the reusable query index was introduced. Test execution is
-still the dominant and naturally variable part of the total.
-
-## Isolation strategy trade-offs
-
-| Strategy | Arbitrary-runner compatibility | Failure isolation | Startup/storage |
-| --- | --- | --- | --- |
-| Transactional physical namespace | Highest; ordinary filesystem consumers and opaque mounts see real files | Strong when staging, publication, recovery, locking, and same-filesystem renames are enforced | Recreates directory entries and may copy bytes when reflinks are unavailable |
-| Node loader/Vite plugins | High for observed Node and bundler graphs, incomplete for native readers and hidden remote mounts | Strong because transformed source need not be persisted | Lowest retained storage and usually fastest |
-| FUSE/OS overlay | Potentially broad local read interception, but not portable or zero-install | Adds mount, privilege, kernel/extension, and teardown failure boundaries | Low duplicated storage but operationally expensive |
-| Adaptive hybrid | Fast path where capability is proven; transactional namespace otherwise | Inherits the physical fallback's guarantees when detection is conservative | Best practical balance; more implementation paths must be tested |
-
-The safe default remains the transactional physical namespace. The intended
-optimization is an adaptive hybrid that selects a proven loader/plugin path
-and lazily materializes the same transactional fallback whenever an opaque
-runner needs real files. FUSE is not an appropriate portable default for a
-zero-install `npx` tool.
+Use several alternating pairs and compare medians. Do not compare a cold
+package, browser, build, or VM cache with a warm one. A first `npx` download is
+package-acquisition time, not coverage-engine time.
+
+## Storage
+
+Each completed run stores compressed raw evidence and a small metadata file
+under `.supercov/runs/<run-id>/`. Query views are derived from that evidence;
+Supercov does not retain a separate full report for every filter.
+
+The isolated workspace can be larger than a run because it may contain an
+instrumented build cache. Control retention explicitly:
+
+```sh
+npx supercov clean --dry-run
+npx supercov clean --keep 20
+npx supercov clean
+```
+
+Supercov never prunes runs in the background. Cleanup is explicit so historical
+evidence does not disappear during an unattended agent session.

package/docs/supported-suites.md

@@ -1,119 +1,98 @@
 # Supported suites
 
-Supercov wraps a test command and instruments the processes it launches.
-Runner support differs by attribution level: exact per-test attribution or
-aggregate coverage.
+Supercov supports JavaScript, TypeScript, and Rust today. Support is exact when
+Supercov can identify individual test attempts; otherwise it reports aggregate
+coverage without guessing which test caused a hit.
 
-## Attribution by runner
+## Languages
 
-| Runner | Attribution | Notes |
+| Language | Status | Supported commands |
 | --- | --- | --- |
-| Playwright | Exact per test | Test, worker, retry and outcome scopes. ESM and CommonJS specs in arbitrary directories, plus project-owned fixture packages. |
-| Vitest | Exact per test | Module-import and setup execution is kept as a separate setup scope. |
-| Jest | Exact per test | Including concurrent and parameterized tests. |
-| `node:test` | Exact per test | Through the generated adapter. |
-| AVA, Mocha, other runners | Aggregate only | First-party structural coverage through inherited process instrumentation. Hits are recorded as background rather than guessed onto tests. |
-| Browser component runners without an adapter | Aggregate only | Same boundary, made explicit in the report. |
+| JavaScript | Available | Existing Node-based test commands |
+| TypeScript | Available | Existing Node-based test commands and build pipelines |
+| Rust | Available | `cargo test`, `cargo nextest run` |
+| Python | Coming soon | — |
+| Zig | Coming soon | — |
+| PHP | Coming soon | — |
+| C | Coming soon | — |
 
-Adapters are generated into the isolated workspace. Your test imports, reporter
-list and runner configuration are not modified.
+More languages will follow. The current npm-distributed CLI requires Node.js 22
+or newer for every language.
 
-A single command may collect several runners into one run. Each test is then
-labelled with the runner that executed it and the semantic kind it belongs to.
+## JavaScript and TypeScript runners
 
-## Builds
-
-| Project shape | How instrumentation is applied |
+| Runner | Attribution |
 | --- | --- |
-| Vite or Vitest | Through the existing Vite graph |
-| Next, Turbopack, Webpack, esbuild, SWC, other build commands | Applied to the disposable source copy, then your unchanged build command runs against it |
-| No build step (ESM or CommonJS) | Direct instrumentation of the disposable source copy |
+| Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
+| Vitest | Exact per test, with setup execution kept separate |
+| Jest | Exact per test, including concurrent and parameterized tests |
+| `node:test` | Exact per test |
+| AVA, Mocha, and other Node runners | Aggregate structural coverage |
+| Browser component runners without an adapter | Aggregate structural coverage |
+
+Aggregate evidence is still included in the full-run view. It is labelled as
+background rather than being assigned to a test that may not have caused it.
+
+A single command may launch several runners. Supercov combines their evidence
+into one run and keeps the runner identity where exact attribution is available.
+
+## Rust runners
 
-The ordinary application build is never read as an input, overwritten, or
-rebuilt afterwards.
+| Runner | Attribution | Current boundary |
+| --- | --- | --- |
+| Cargo's standard libtest runner | Exact test and attempt identity | Run with `npx supercov -- cargo test` |
+| cargo-nextest | Exact test, attempt, retry, and binary identity | Run with cargo-nextest 0.9.138 or 0.9.140 |
+
+Rust support currently follows the Rust 1.95 toolchain and preserves Cargo's
+test selection, scheduling, fail-fast behavior, environment, and exit status.
+`cross` is not supported yet. Unsupported command shapes fail clearly rather
+than falling back to plausible but inaccurate attribution.
 
-When the complete source, configuration and toolchain fingerprint is unchanged
-between runs, the previous instrumented output and manifest are carried into the
-refreshed workspace and the build is skipped entirely.
+## Builds and source formats
 
-### Build-only environment flags
+JavaScript and TypeScript projects can use Vite, Vitest, Next, Turbopack,
+Webpack, esbuild, SWC, `tsc`, or no build step. ESM and CommonJS are supported,
+along with modern JavaScript, JSX, TypeScript, and TSX syntax.
 
-Before the isolated build, Supercov compares the invoked npm, pnpm, yarn or bun
-script with explicit string-valued `process.env` checks in the project's build
-configuration. A semantic match — a `test:preview` script and a
-`process.env.TEST_PREVIEW === "true"` check, for example — activates that
-build-only flag, and the decision is printed before the build. Values are never
-guessed for unrelated environment variables.
+Supercov instruments an isolated workspace. It does not add imports, reporters,
+or plugins to the authored project, and it does not overwrite the project's
+ordinary build output.
 
 ## Browsers
 
-The compatibility workflow exercises Chromium, Firefox and WebKit, along with
-Node 22, 24 and 25, Playwright 1.55 and current, Vite 5 and current, Vitest 2
-and current, and modern JavaScript, JSX, TypeScript and TSX syntax fixtures.
-
-The Playwright adapter covers the `page` and `request` fixtures, API request
-contexts, user-created browser contexts and pages, popups and all of their
-frames, dedicated and service workers, WebSocket handshake headers, and
-test-spawned child processes.
-
-For Chromium documents exposed through the page target, a pre-document probe
-installs the action phase before application JavaScript starts. A newly created
-cross-origin iframe may run in a separate target that cannot be safely paused
-during navigation; its earliest probes use a timing fallback until the frame is
-live. This affects action-level causal precision only — never structural
-coverage or test attribution.
-
-## Servers, background work and child processes
-
-Server-side coverage is safe when Playwright runs multiple workers against one
-application server. Every routed request carries a run, worker, test and retry
-scope; Node async context retains that scope and its current phase across
-awaited work; and each worker writes to a distinct attempt path that only its
-own collecting fixture will accept.
-
-Detached work is never dropped silently or guessed onto whichever test is
-active:
-
-- HTTP callbacks inherit the carrier automatically.
-- Child processes inherit it through their environment.
-- Exported queue helpers cover BullMQ, Bee-Queue, pg-boss, Agenda and
-  in-process schedulers.
-- Anything that still arrives without a carrier is persisted under the
-  background scope.
-
-## Remote and containerised execution
-
-Discovery is structural rather than provider-specific. The preload and a
-narrowly gated ESM transform look for a static `build(options)` capability,
-activate only when those options contain a host-to-guest mount that includes the
-isolated project, scope any existing cache or snapshot identity to the run's
-source fingerprint, and follow the returned object graph. A method whose options
-contain `argv`, `cmd` or `command` receives guest-translated Supercov paths and
-a guest-valid Node preload.
-
-The execution log records this process and capability graph, but hashes long or
-multiline arguments so embedded shell bodies and credentials are never
-persisted.
-
-The boundary is explicit: Supercov follows Node child processes, not arbitrary
-non-Node supervisors, and not a remote control plane that never exposes its
-launches to the local process. CommonJS and pure-ESM executor SDKs,
-object-shaped and positional execution APIs, and opaque returned object graphs
-are all covered when a discoverable build capability exposes the workspace mount
-and an execution capability accepts an environment. Anything that hides all
-launch state behind an out-of-process RPC needs a dedicated adapter, and
-Supercov reports missing evidence rather than claiming those paths are covered.
-
-The public regression suite includes provider-neutral CommonJS and pure-ESM
-opaque executors. CI requires Supercov to discover that structure, scope the
-cache identity, translate paths and the Node preload into the guest, run nested
-Vitest and Playwright commands, parse every concurrent trace shard, and produce
-100% fixture coverage.
-
-## Distributed runs
-
-Each shard produces its own immutable run. `supercov merge` combines runs whose
-source, test, dependency, configuration, instrumenter, schema and denominator
-fingerprints match exactly, publishes a new immutable run atomically, and leaves
-every input untouched. Incompatible shards fail with a clear reason instead of
-being averaged together.
+Playwright coverage supports Chromium, Firefox, and WebKit. It follows pages,
+frames, popups, workers, request contexts, WebSockets, and test-spawned child
+processes where the runner exposes the required identity.
+
+## Background processes and servers
+
+Node child processes inherit coverage automatically. Long-running servers are
+given a short drain window after the test command finishes so buffered evidence
+can arrive before the run is published.
+
+If work arrives without a reliable test identity, Supercov records it as
+background evidence. The default whole-run view includes it; passed-only and
+per-test views do not pretend it belongs to a particular test.
+
+## Containers, VMs, and remote execution
+
+Supercov can collect from supported processes launched through a container, VM,
+or remote executor when the command exposes a discoverable launch boundary and
+the Supercov runtime can be carried into that environment. Mounted workspaces
+and local child-process launchers are the most direct path.
+
+If the remote boundary hides how code is launched or cannot return evidence,
+Supercov reports the missing coverage boundary. It does not silently treat
+remote execution as measured.
+
+## Distributed suites
+
+Run shards separately, then merge compatible run ids:
+
+```sh
+npx supercov merge <shard-a> <shard-b> <shard-c>
+```
+
+All shards must describe the same source, configuration, toolchain, schema, and
+coverage denominator. Incompatible shards are rejected with the mismatched
+domains listed.

package/docs/verification.md

@@ -1,102 +1,68 @@
 # Verification
 
-Supercov instruments source before the test command runs. Seven release gates
-check that the instrumented program matches the original program and that
-coverage calculations are correct. Any failed gate blocks publication.
+Coverage is useful only if instrumentation preserves program behavior and the
+reported obligations match what actually executed. Supercov fails closed when
+it cannot establish either condition.
 
-## 1. Semantic differential execution
+## What release checks cover
 
-Original and instrumented programs are executed in isolated scopes and compared
-on three axes: return values, thrown errors, and the observable order of side
-effects.
+Every release is checked for:
 
-The fixtures deliberately target the places where a naive transform breaks:
-getters, proxies, optional calls and `this` binding, computed logical
-assignments, parameter defaults, `try`/`catch`/`finally`, iterator closing,
-switch fallthrough, labelled loops, async functions and generators.
+- identical return values, thrown errors, and side-effect order before and
+  after instrumentation;
+- short-circuiting, getters, proxies, optional calls, `this` binding, defaults,
+  exceptions, loops, async functions, and generators;
+- exact line, branch, decision-vector, and MC/DC results;
+- JavaScript behavior across a pinned TC39 Test262 corpus;
+- supported JavaScript, TypeScript, and Rust runner contracts;
+- Chromium, Firefox, and WebKit browser execution;
+- source isolation, interrupted-run recovery, and atomic publication; and
+- package installation and execution from a clean project.
 
-## 2. Deterministic generated corpus
+MC/DC cases are also compared with an independent LLVM implementation so a
+self-consistent error in Supercov's own calculation does not pass unnoticed.
 
-A generated corpus exercises 160 nested combinations of short-circuiting,
-ternaries, coercion and thrown expressions on every run. It is deterministic, so
-a regression reproduces exactly rather than appearing once in CI and never
-again.
+## What happens when code cannot be measured safely
 
-## 3. Property testing
+Supercov does not force a transform through code that observes its own source
+text or creates source dynamically without a stable denominator. It leaves the
+affected behavior uninstrumented and records a completeness blocker with the
+reason and location.
 
-Seeded `fast-check` properties generate a further 500 nested expressions and 300
-control-flow executions per run, with shrinking and a reproducible seed printed
-on failure.
+Similarly, evidence from an unsupported runner or hidden remote boundary is
+reported as aggregate, unattributed, or missing. It is not assigned to a test
+that may not have caused it.
 
-## 4. Coverage oracles
+Inspect these states with:
 
-Behaviour equivalence is not enough — the numbers have to be right too. Separate
-oracles assert exact decision vectors, MC/DC witnesses and branch alternatives
-independently of what the program does.
-
-## 5. An independent MC/DC implementation
-
-The same three-condition masking-MC/DC golden cases must produce identical
-verdicts under Supercov and under Clang/LLVM source-based MC/DC: 100% for a
-complete witness set and 33.33% for an incomplete one.
-
-This is the gate that matters most. MC/DC has enough subtlety — masking versus
-unique-cause, short-circuit evaluation, compound conditions — that agreement
-with an independently implemented, widely audited toolchain is far stronger
-evidence than any self-consistent test suite.
-
-## 6. TC39 Test262
-
-Release CI shards the pinned Test262 corpus across 16 workers, runs the official
-harness against original and instrumented sources, and rejects any scenario that
-passes originally but fails after transformation.
-
-Some categories are excluded by construction, with reason counts printed for
-every shard:
-
-| Excluded | Why |
-| --- | --- |
-| Module, async and raw tests | Not comparable under the source-rewrite harness |
-| Parse and resolution negatives | The transform never runs on unparseable input |
-| Annex B sloppy-script extensions | Does not apply to the application modules Supercov instruments |
-| `Function.prototype.toString` and function-source coercion | Exact source reflection necessarily observes a source transform |
-
-The last category is handled in the product, not hidden: when application code
-directly coerces or observes a function's source, Supercov leaves that body
-uninstrumented and records a visible `semantic-safety` completeness blocker.
-Dedicated differential fixtures cover the async and generator cases that Test262
-cannot compare.
-
-## 7. Performance budgets
+```sh
+npx supercov runs latest
+npx supercov runs latest scope
+npx supercov runs latest gaps
+```
 
-Transform latency, transactional workspace preparation, output expansion and
-runtime probe overhead are each checked against explicit budgets. A change that
-makes instrumentation correct but unusably slow fails the same way a wrong
-answer does.
+## How to review a coverage change
 
-## Cross-platform and compatibility gates
+For a test added by a coding agent, check three things:
 
-Alongside the seven correctness gates, the compatibility workflow runs Node 22,
-24 and 25, Playwright 1.55 and current, Vite 5 and current, Vitest 2 and
-current, Chromium, Firefox and WebKit, and modern JavaScript, JSX, TypeScript
-and TSX syntax fixtures.
+1. The wrapped test command still passes.
+2. `diff` shows the expected obligations gained and no unexplained loss.
+3. The test contains meaningful assertions and does not weaken application
+   behavior merely to improve a percentage.
 
-Filesystem publication, symlink handling, copy fallback, `ENOSPC`, failed
-rename and forced-termination recovery are all exercised on Ubuntu, macOS and
-Windows.
+```sh
+npx supercov diff <baseline-run> <new-run>
+npx supercov runs <new-run> test "new test name"
+```
 
-A clean-room gate packs the npm tarball, invokes it through `npx` in a project
-with no build step, and asserts that not a single source or configuration file
-changed.
+Store the compared run ids in the review or agent summary when the evidence
+needs to be reproducible later.
 
-## Running the gates yourself
+## Integrity of stored runs
 
-```sh
-npm test
-npm run test:clang-mcdc
-npm run benchmark:check
-TEST262_DIR=/path/to/test262 npm run test:test262
-```
+Completed runs are immutable and integrity-bound to their evidence,
+fingerprints, and schema. A run whose bytes are missing, corrupt, stale, or
+incompatible is surfaced as such instead of being opened as a plausible report.
 
-The Clang/LLVM oracle requires `clang` and `llvm` to be installed. The Test262
-gate requires a checkout of the pinned corpus.
+See [Evidence and runs](/docs/evidence) for retention, comparison, and shard
+merging.