supercov 2.0.0 → 3.0.0

package/README.md

@@ -12,10 +12,12 @@ Measure code quality with npx supercov.
 
 Scoring needs a [TypeSafe AI](https://typesafe.ai) API key, and your coding agent will usually ask you for it. It costs about a cent per megabyte of source. Coverage needs no account, config file, import, custom reporter, or hosted service. Supercov is local, free, open source, and MIT licensed.
 
-[Website](https://supercov.com) · [Documentation](https://supercov.com/docs) · [npm](https://www.npmjs.com/package/supercov) · [GitHub](https://github.com/supercorp-ai/supercov)
+[Website](https://supercov.com) · [Documentation](https://supercov.com/docs) · [npm](https://www.npmjs.com/package/supercov) · [GitHub](https://github.com/supercorp-ai/supercov) · [Discord](https://discord.gg/CudcAH53yF)
 
 Supported languages: [JavaScript](https://supercov.com/docs/javascript) · [TypeScript](https://supercov.com/docs/javascript) · [Rust](https://supercov.com/docs/rust) · [Python](https://supercov.com/docs/python) · [Ruby](https://supercov.com/docs/ruby) · [Go](https://supercov.com/docs/go) · [Java](https://supercov.com/docs/java) · [Kotlin](https://supercov.com/docs/kotlin).
 
+In Claude Code, Codex and Gemini CLI you can also [install it as a plugin](#give-supercov-a-job).
+
 Supported by [Supercorp](https://supercorp.ai).
 
 [Agent workflow](https://supercov.com/docs/agent-loop): ask your coding agent
@@ -139,35 +141,57 @@ npx supercov -- python -m unittest
 npx supercov -- bundle exec rspec
 ```
 
-## Map what assertions check
+## See what the tests assert
 
-Each normal test run creates an assertion map automatically. An agent can
-optionally fill in what the assertions check:
+Coverage says which code ran; assertion coverage says which of it the tests
+would catch breaking. Supercov asks [Jev](https://typesafe.ai), test by test,
+whether changing each statement the test ran would make it fail:
 
 ```sh
-npx supercov runs latest assertions --json
-# Pin the returned run ID. Inspect assertions and the source that ran:
-npx supercov runs <run> assertion <assertion-id>
-npx supercov runs <run> source src/example.ts
-# Edit assertions.json, then validate and acknowledge:
-npx supercov runs <run> assertions validate --json
-# Copy examined expectedBasis tokens into assertions.json; save again.
-npx supercov runs <run> assertions check --require-mappings --json
-npx supercov runs <run>  # includes the assertion percentage in the regular report
+npx supercov -- npm test
+TYPESAFE_API_KEY=... npx supercov runs latest assertions assess
+npx supercov runs latest assertions   # the share asserted and what is not
 ```
 
-After code or tests change, run the same test command again. Supercov reuses the
-newest available map for that command and language and identifies dirty flows. Rust owns
-validation, change tracking and reporting; the agent supplies semantic reasoning.
-The score is agent-assessed and separate from MC/DC. See
-[assertion maps](docs/assertion-maps.md) for the format and JS/TS limits, or run
-`supercov docs assertion-agent` for the agent workflow. `supercov assertions schema`
-exports the editor schema; `assertions validate --file <path>` checks JSON syntax.
+Answers are reused across runs whenever the question is the same, so after a
+commit only what changed is asked again. Every language Supercov measures. See
+[assertion coverage](docs/assertions.md).
 
 ## Give Supercov a job
 
 Paste one of these prompts into Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, or any coding agent that can run terminal commands.
 
+Or install the agent skill, and your agent reaches for Supercov on its own when you ask for tests, coverage, refactoring or a security scan. In Claude Code:
+
+```text
+/plugin marketplace add supercorp-ai/supercov
+/plugin install supercov@supercov
+```
+
+In Codex:
+
+```bash
+codex plugin marketplace add supercorp-ai/supercov
+codex plugin add supercov@supercov
+```
+
+In Gemini CLI:
+
+```bash
+gemini extensions install https://github.com/supercorp-ai/supercov
+```
+
+For Cursor, GitHub Copilot and other agents that read skills: `npx skills add supercorp-ai/supercov`.
+
+The Claude Code plugin and the Gemini CLI extension also add four commands you can type:
+
+| Command | What it does |
+| --- | --- |
+| `/supercov:coverage [test command]` | Writes one test for code no test reaches, and reports coverage before and after |
+| `/supercov:security [patch]` | Scans the repository, or a change, for security vulnerabilities |
+| `/supercov:quality [patch]` | Ranks files by code smells, or checks what a change introduced |
+| `/supercov:review [--base ref]` | Checks a branch or uncommitted change for security and quality problems |
+
 ### Write the first useful test
 
 ```text
@@ -280,9 +304,11 @@ Supercov uses exact per-test attribution where an adapter is available. For othe
 | --- | --- |
 | Playwright | Exact per test, worker, retry, outcome, action, and assertion phase |
 | Vitest | Exact per test, with setup execution kept separate |
-| Jest | Exact per test, including parameterized tests; exact `expect` occurrence identity for assertion maps |
+| Jest | Exact per test, including parameterized tests |
 | `node:test` | Exact per test |
-| AVA and Mocha | Aggregate structural coverage |
+| Mocha | Exact per test and retry, `--parallel` included |
+| AVA | Exact per test, concurrent tests included |
+| node-tap 15 and newer | Exact per subtest |
 | Cargo's standard libtest runner | Exact test, attempt, and passing-assertion identity |
 | rustdoc doctests | Exact per doctest |
 | cargo-nextest | Exact test, attempt, retry, and binary identity |
@@ -352,6 +378,10 @@ npx supercov runs clean             # remove all runs and the build cache
 
 [MIT licensed](LICENSE). Inspect, extend, and run it anywhere.
 
+## Community
+
+Ask questions, share what you measured, and talk to the people building Supercov on [Discord](https://discord.gg/CudcAH53yF).
+
 ## Contributors
 
 - [@DaVinciTachyon](https://github.com/DaVinciTachyon)

package/docs/agent-loop.md

@@ -55,24 +55,19 @@ duplicate.
 
 ## Add assertion coverage
 
-For JavaScript and TypeScript projects, you can also ask the agent to explain
-what each test assertion checks. Every normal run creates an `assertions.json`
-map and carries forward compatible work from earlier runs of the same command.
+For JavaScript and TypeScript projects, ask which executed statements no test
+would catch breaking, then add checks for them:
 
 ```sh supercov
+npx supercov runs latest assertions assess
 npx supercov runs latest assertions
-npx supercov docs assertion-agent
+npx supercov runs latest assertions <path>
 ```
 
-Keep the printed run ID fixed while editing. The agent reads matching current
-source, updates the map, validates its references and saves review tokens for
-the explanations it examined. The regular run report then shows the assertion
-percentage. It does not need another test run just to read the edited map.
-
-Start with [Understanding assertion coverage](assertions.md) for a copyable
-prompt. [Mapping assertions with an agent](assertion-agent.md) gives the complete
-editing and checking loop. Use the evidence and recorded observations alongside
-MC/DC when deciding which tests need stronger checks.
+`assess` needs `TYPESAFE_API_KEY`; reading does not. A statement that is not
+asserted runs in some test, but nothing that test checks depends on it. After
+adding a check, rerun the suite and assess again: only the questions the change
+touched are asked. See [Assertion coverage](assertions.md).
 
 ## Example
 

package/docs/assertions.md

@@ -1,17 +1,7 @@
 # Assertion coverage
 
-Line coverage shows which code ran. Assertion coverage helps you see what the
-tests checked. Your coding agent traces assertions back to the source, and
-Supercov checks those links against recorded execution from the same test.
-
-Assertion coverage measures JavaScript, TypeScript, Python, Ruby and Rust. The
-map format is the same for every one of them: an assertion is identified by its
-file, line and column, so a project written in more than one language keeps a
-single map.
-
-Python and Ruby report an assertion's line but not its column, so two
-assertions written on one line cannot be told apart and neither is credited.
-Put them on separate lines.
+Line coverage shows which code ran. Assertion coverage shows which of that code
+the tests would catch breaking.
 
 ## A passing test can miss a wrong result
 
@@ -37,75 +27,171 @@ test('confirms an order', () => {
 });
 ```
 
-Every line in the function runs. The test passes. But it would also pass if the
-total were `49` instead of `50`.
-
-## How the agent finds the gap
-
-1. **Supercov records execution:** which statements ran in each test and which
-   assertions passed.
-2. **Your agent traces each assertion** through the test and source to explain
-   what it checks. It saves those links in the run's `assertions.json` map.
-3. **Supercov checks the map against the run.** A statement needs a current
-   explanation, execution, and a passing assertion in the same test to count.
-
-In this example, the agent follows the status assertion back to the returned
-order. It finds no check that reads the total:
-
-- `order.status` → must equal `'confirmed'`.
-- `order.total` → no assertion checks the calculated total.
-
-The agent adds the missing assertion to the existing test:
-
-```js
-assert.equal(order.total, 50);
+Every line runs and the test passes. It would also pass if `total` were
+`undefined`, so the line computing it is covered but not asserted.
+
+## What counts as asserted
+
+A statement is asserted when changing it would make at least one passing test
+that runs it fail: an assertion fails, test or helper code throws, the test
+times out, or the process crashes. The change depends on the statement:
+
+| Statement | Change |
+| --- | --- |
+| `if` (and `elif`, `elsif`, `unless`, `else if`) | the condition inverted, each way separately (asserted if either is caught); a Rust `if let` is made not to match |
+| `return x`, and a block's value in Ruby, Rust and Kotlin | returns a stand-in for nothing, without evaluating `x` |
+| a `when`, `switch`, `try`, `loop` or Kotlin `throw` a function ends on | every value it returns becomes that stand-in (skipping it would leave the function nothing to return) |
+| declaration or assignment with a value | the value becomes that stand-in, without being evaluated |
+| JSX expression | the value becomes undefined |
+| anything else | skipped: the statement does not run |
+
+The stand-in is `undefined` in JavaScript and TypeScript, `None` in Python and
+`nil` in Ruby. Go and the JVM languages replace a value after it is computed,
+as their mutation testers do: the zero value in Go, and `null`, `0` or `false`
+in Java and Kotlin. Rust has no value that fits every type, so it takes a
+different value of the same type, from the function's signature or the
+`let`'s annotation where there is one: a boolean negated, an `Ok` turned into
+an `Err` and back, a literal swapped (`false` for `true`, `1` for `0`), an
+`Ordering` reversed, `None`, `""` or `0`. Where nothing is annotated, the type
+comes from the crate's own signatures, the parameters and `let`s in scope, or
+a `match`'s arms (`let (major, text) = numeric(text)?;` takes `numeric`'s
+return type): a tuple gets one element changed, an enum another of its
+variants, a struct literal one field, a closure its result, integer
+arithmetic its lowest bit flipped, a comparison its order reversed, a slice
+or split the empty one, and `let Some(x) = .. else` gets `None`. A value of a
+type the crate defines is built from its own constants, constructors and
+enums (`Err(Error::new(ErrorKind::Empty))`, `Ok(Version::new(0, 0, 0))`).
+`Default::default()` is left only where nothing says what the type is.
+
+Imports, declarations without a value, and function, class and type
+declarations are not assessed; in Python neither are `pass`, `global`,
+`nonlocal` and docstrings, and in Ruby neither are the calls that declare
+(`require`, `attr_reader`, `private`, `include` ...). **Assertions** is the
+share of the executed statements that are asserted.
+
+## Assess and read
+
+```bash
+npx supercov -- npm test
+npx supercov runs latest assertions assess
+npx supercov runs latest assertions
 ```
 
-It reruns the full suite and updates the map. A total of `49` now fails the
-test: two items at $25 must total $50. Line coverage has not changed, but the
-test now checks the calculation.
-
-## Try it in your project
+`assertions assess` works it out and saves the result to the run. For each test,
+Supercov renders what the test ran -- its code, its helpers, and the source it
+executed -- and asks [Jev](https://typesafe.ai) whether the test would fail for
+each statement's change. It prints an estimate first; `--dry-run` sends nothing.
+It needs a TypeSafe AI API key in `TYPESAFE_API_KEY`.
 
-Open your project in your usual coding agent and paste this prompt. The agent
-can install Supercov and run the commands for you.
+`assertions` reads the saved result with no network and no key, a page at a
+time, from the run down to one statement:
 
-```text supercov-prompt
-Read npx supercov docs assertion-agent. Run the full test suite through
-Supercov, then build and validate its assertion map. Find one useful
-missing check and add an assertion for the expected behavior.
-Only change tests; do not weaken existing checks. Rerun the same suite
-and update the map. Show the test change, before-and-after assertion
-coverage, and any uncertainty or missing evidence.
+```bash
+npx supercov runs latest assertions                       # the share, and the files by statements not asserted
+npx supercov runs latest assertions src/adapters.ts       # that file's statements not asserted
+npx supercov runs latest assertions src/adapters.ts:100   # one statement: each asked test's answer
+npx supercov runs latest assertions --test "closes the stream"  # what one test catches, and what it misses
 ```
 
-The agent uses your actual test command after `--`, for example
-`npx supercov -- npm test`. On later runs of the same command, Supercov reuses
-compatible mappings and flags explanations that need another look.
-
-## Read the result
-
-The agent can show the summary and the statement-level report for a file:
+```text
+Run run_4f2a: 93.8% asserted (2731 of 2913 executed statements)
 
-```sh
-npx supercov runs <run-id>
-npx supercov runs <run-id> assertions report --view statements --file checkout.js
+Files, most statements not asserted first: a statement is not asserted when no test that runs it was judged to fail if it changed.
+ NOT ASSERTED  ASSERTED  FILE
+           41   388/429  src/adapters.ts
+           ...
+showing 1-20 of 57
+next page: npx supercov runs 'run_4f2a' assertions --offset 20
+inspect a file: npx supercov runs 'run_4f2a' assertions 'src/adapters.ts'
 ```
 
-**Assertions** is the percentage of measured source statements linked to
-passing assertions by the agent's map. It is not a count of assertions. Each
-statement counts once; statements that never ran remain in the total.
-
-An unmapped statement is a place to investigate, not proof of a missing test:
-the agent may not have mapped its existing check yet.
-
-The strength of the check still matters. `assert.ok(order.total)` accepts both
-`49` and `50`; `assert.equal(order.total, 50)` distinguishes them. Review the
-expected behavior, not just the percentage.
-
-Supercov does not generate or execute mutated code for this assessment. Unlike
-mutation testing, it does not test whether deliberately introduced bugs are
-caught. The agent's explanations still need review.
-
-For the detailed workflow, see [Mapping assertions with an agent](assertion-agent.md).
-For a result you cannot explain, see [Investigating assertion evidence](assertion-evidence.md).
+Each listing ends with the command for its next page and for the next step
+down, so an agent reads only as much as it needs. `runs latest file` and
+`runs latest line` show the same verdicts beside the coverage, and
+`npx supercov runs latest` and the HTML report show the same share.
+
+A statement that is not asserted is a place to add a check: a test that runs it
+exists, but nothing it asserts depends on what the statement does.
+
+## Which tests check a change
+
+`npx supercov runs latest tests affected` uses the assessment to say which of
+the tests a change reaches were judged to catch it, and lists them first;
+`assertions assess --changed` asks every test that ran the changed code. See
+[Find the tests a change affects](cli.md#find-the-tests-a-change-affects).
+
+## Later runs ask only about what changed
+
+Answers are kept in `.supercov/assertions/`. Each is stored under the exact text
+Jev saw for that statement and test, and reused only when a new run would ask
+the same question with the same text. Nothing decides whether a change could
+matter: a change to the statement, to code around it that the test ran, to the
+test or its helpers, or to which tests run it, asks again. A dependency or
+configuration change asks everything again. After a typical commit only a few
+dozen questions are asked; the rest are reused.
+
+## Cost, speed and accuracy
+
+- **Cost:** about 0.006 cents per statement for a first assessment: 14 projects
+  in six languages (8,356 statements) cost $0.48 together. A project of 3,000
+  statements costs about $0.18; a later commit usually well under a cent.
+- **Speed:** about 20 seconds for 1,500 statements with 24 requests in flight
+  (`--workers`, default 24; it changes how long a pass takes, not what it
+  costs).
+
+Measured on four JavaScript and TypeScript projects (7,393 executed statements):
+
+- **Accuracy:** against ground truth from applying each change and running the
+  tests, on a fresh sample of 78 statements: 69 right; of the 13 not asserted it
+  found 8, and 8 of the 12 it flagged were truly not asserted. The share it
+  reported was 84.6% against a true 83.3%.
+- **Python:** on h11 (532 executed statements, pytest), every statement checked
+  the same way: 502 right; of the 67 not asserted it found 45, and 45 of the 53
+  it flagged were truly not asserted. The share it reported was 90.0% against a
+  true 87.4%, for about $0.04.
+- **Go, Rust, Java, Ruby and Kotlin**, checked the same way. These were used
+  to shape how changes and tests are shown to Jev, so they are flattering:
+
+  | Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
+  | --- | --- | --- | --- | --- |
+  | google/uuid (Go) | 339 | 298 | 34 (28) | 90.0% / 81.4% |
+  | hashicorp/go-version (Go) | 257 | 253 | 21 (19) | 91.8% / 91.8% |
+  | dtolnay/semver (Rust) | 200 of a sample of 200 | 190 | 7 (2) | 96.5% / 96.5% |
+  | apache/commons-cli (Java) | 196 of a sample of 200 | 167 | 49 (33) | 75.0% / 76.5% |
+  | dry-rb/dry-inflector (Ruby 3.4) | 157 | 148 | 13 (8) | 91.7% / 92.4% |
+  | sporkmonger/addressable (Ruby) | 198 of a sample of 200 | 182 | 25 (14) | 87.4% / 90.4% |
+  | square/moshi (Kotlin) | 60 of a sample of 60 | 52 | 9 (6) | 85.0% / 81.7% |
+
+- **Held out:** five projects nothing was tuned on, each assessed once by the
+  shipped version and its verdicts frozen before any change was run, scored on
+  a pre-registered sample of 60 statements (and on every statement where the
+  suite is fast enough):
+
+  | Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
+  | --- | --- | --- | --- | --- |
+  | pallets/itsdangerous (Python) | 60 of 60 | 56 | 3 (2) | 95.0% / 91.7% |
+  | — every statement | 238 of 238 | 224 | 13 (9) | 94.5% / 92.0% |
+  | Masterminds/semver (Go) | 60 of 60 | 55 | 7 (7) | 88.3% / 80.0% |
+  | — every statement | 497 of 498 | 472 | 46 (44) | 90.7% / 86.5% |
+  | hashie/hashie (Ruby 3.4) | 60 of 60 | 57 | 3 (1) | 95.0% / 96.7% |
+  | — every statement | 700 of 701 | 655 | 26 (10) | 96.3% / 94.4% |
+  | FasterXML/java-classmate (Java) | 60 of 60 | 58 | 9 (7) | 85.0% / 88.3% |
+  | bluss/arrayvec (Rust) | 54 of 60 | 48 | 1 (1) | 98.1% / 87.0% |
+
+  Assessing all five (2,582 statements) cost $0.15, about 0.006 cents per
+  statement. Go reads high: Jev takes a dropped error (`return false,
+  fmt.Errorf(..)` becoming `return false, nil`) as caught where the tests check
+  only the boolean. Rust reads high too on arrayvec, whose code is generic over
+  its element type; six of its sampled changes cannot be made, because no value
+  of an unknown `T` can be written.
+
+The share is a good measure of a project; a single statement's verdict is a
+judgment worth checking before acting on it. Supercov does not run mutated code
+for this: nothing is executed, and your tests are never slowed.
+
+## Requirements
+
+Runs of any language Supercov measures: JavaScript and TypeScript, Python
+(pytest, unittest), Ruby (RSpec, Minitest, test-unit, Cucumber), Go, Rust
+(libtest, nextest, doctests), and Java and Kotlin (JUnit, TestNG); and
+`TYPESAFE_API_KEY` for `assess`. Reading a saved result needs neither.