supercov 0.0.50 → 0.0.52
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/assertion-agent.md +9 -5
- package/docs/assertion-maps.md +53 -13
- package/docs/cli.md +22 -0
- package/package.json +12 -11
- package/runtime/javascript/provenance.mjs +71 -12
- package/schemas/assertions.schema.json +2 -2
package/docs/assertion-agent.md
CHANGED
|
@@ -59,9 +59,10 @@ For each assertion:
|
|
|
59
59
|
- Put a node in `countsAsAsserted` only when its behavior is checked by this
|
|
60
60
|
assertion. Every counted node needs a recorded path to `$assertion`.
|
|
61
61
|
Counting a guard or block does not count the nested body automatically.
|
|
62
|
-
- Add helper, configuration and other dependency files to `watch`.
|
|
63
|
-
|
|
64
|
-
and
|
|
62
|
+
- Add helper, configuration and other dependency files to `watch`. The
|
|
63
|
+
declarations holding your nodes, the top level of their files, the assertion
|
|
64
|
+
file and the selected test files are already dependencies. Include relevant
|
|
65
|
+
guards and alternatives even when they do not appear as graph nodes.
|
|
65
66
|
- Keep uncertainty in `questions`. A flow's unresolved questions block its
|
|
66
67
|
credit. Assertion-level questions record broader unfinished investigation.
|
|
67
68
|
|
|
@@ -97,8 +98,11 @@ back into the map.
|
|
|
97
98
|
|
|
98
99
|
## Assess changes before renewing flows
|
|
99
100
|
|
|
100
|
-
When inheriting a map, read every entry in `--view changes`.
|
|
101
|
-
|
|
101
|
+
When inheriting a map, read every entry in `--view changes`. Each names the
|
|
102
|
+
flows it already made stale (`knownFlows`) and, under `exposed`, the tests that
|
|
103
|
+
ran the changed code and how many flows they carry; start with those, and
|
|
104
|
+
investigate whether other flows are affected too. A flow's own `notices` list changes near its
|
|
105
|
+
nodes that could not have reached it.
|
|
102
106
|
|
|
103
107
|
Add a `changeAssessments` entry for each managed change ID. Include all listed
|
|
104
108
|
`knownFlows` that still exist, plus any other affected flows. Explain why the
|
package/docs/assertion-maps.md
CHANGED
|
@@ -70,7 +70,7 @@ examined and acknowledged.
|
|
|
70
70
|
| `appliesTo` | Select tests by project-relative file and exact displayed test name. |
|
|
71
71
|
| `nodes`, `edges` | Record source locations and relationships ending at `$assertion`. |
|
|
72
72
|
| `countsAsAsserted` | List the node IDs you judge to be checked by the assertion. |
|
|
73
|
-
| `watch` | List additional files the explanation depends on, such as helpers or configuration. Manifests, lockfiles and runner configuration are already tracked for the whole run; naming one
|
|
73
|
+
| `watch` | List additional files the explanation depends on, such as helpers or configuration. A watched file is depended on **as a whole**: any change to it that is not a comment is a review. Use it for a file your claim reasons about but holds no node of yours -- including claims about what a file does *not* contain, which no node can anchor. Naming a file that already holds this flow's nodes widens the flow from those declarations to the whole file, so a neighbouring function's body becomes a review again; that is sometimes what you mean, and the report says when you have done it. Manifests, lockfiles and runner configuration are already tracked for the whole run; naming one catches nothing, and the report says so. A redundant entry is free to take back out: it is not part of what the acknowledgement rests on, so removing one keeps the flow's credit. |
|
|
74
74
|
| `questions` | Record unresolved investigation questions. Questions inside a flow block its credit. |
|
|
75
75
|
|
|
76
76
|
Source anchors use project-relative paths with `/`, one-based lines and one-based
|
|
@@ -107,13 +107,36 @@ token stale.
|
|
|
107
107
|
|
|
108
108
|
### What makes a review token stale
|
|
109
109
|
|
|
110
|
-
A flow needs a fresh review when its claim changes
|
|
111
|
-
|
|
112
|
-
|
|
110
|
+
A flow needs a fresh review when its claim changes; when a declaration holding
|
|
111
|
+
one of its nodes changes -- the function, method or class the node sits in, or
|
|
112
|
+
the top level of that file, its imports and constants; when that file's set of
|
|
113
|
+
declarations changes, one added, removed or renamed; when a file it watches, a
|
|
114
|
+
test it selects or its assertion's file changes; or when the configuration
|
|
115
|
+
that decides what executes changes: a transpiler, a test runner, an
|
|
116
|
+
interpreter pin.
|
|
117
|
+
|
|
118
|
+
Each reason names what moved -- `src/server.js: Server.start (line 12)
|
|
119
|
+
changed (holds this flow's return:31)` -- so you can look rather than reread.
|
|
113
120
|
|
|
114
121
|
Several things that sound like they should count do not, because an
|
|
115
122
|
acknowledgement demanded for all of them at once stops being read.
|
|
116
123
|
|
|
124
|
+
Comments do not, nor blank lines or trailing whitespace: no program can tell.
|
|
125
|
+
A comment the language itself reads is the exception and does count -- a Go
|
|
126
|
+
`//go:embed` directive, a Ruby magic comment, a Rust doctest.
|
|
127
|
+
|
|
128
|
+
A change to another declaration in a node's file does not. The claim rests on
|
|
129
|
+
the code it names; the rest of the file is the author's to name in `watch` if
|
|
130
|
+
it matters. Such a change is a notice on the flow (`notices` in the report),
|
|
131
|
+
not a review, and it is asked about once, as a change to assess.
|
|
132
|
+
|
|
133
|
+
Code the flow's test ran elsewhere does not make the flow stale either. A claim
|
|
134
|
+
does not pass through every function its test happened to execute. What each
|
|
135
|
+
test ran is recorded, and a changed file's change record says which tests ran
|
|
136
|
+
the changed code and how many flows that exposes, so the one assessment the
|
|
137
|
+
change asks for is asked of the right people -- and a change no selected test
|
|
138
|
+
ran is not asked about at all.
|
|
139
|
+
|
|
117
140
|
Cutting a release does not. A manifest is fingerprinted by what it declares, so
|
|
118
141
|
a version number moving in `package.json`, `Cargo.toml`, `pyproject.toml` or a
|
|
119
142
|
lockfile changes nothing. Neither does reformatting one.
|
|
@@ -233,11 +256,13 @@ Run the same test command again, then edit the new run's map. Supercov carries
|
|
|
233
256
|
forward compatible mappings and leaves the previous run unchanged. If a newer
|
|
234
257
|
map cannot be reused, `inheritance.skipped` explains the fallback.
|
|
235
258
|
|
|
236
|
-
Each flow depends on
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
259
|
+
Each flow depends on the declarations holding its nodes and the top level of
|
|
260
|
+
their files, on its assertion file, its selected test files and its extra
|
|
261
|
+
`watch` files. A change to any of those requires another look at that flow;
|
|
262
|
+
a comment, a blank line or another declaration's body does not. Independent
|
|
263
|
+
sibling flows stay current. Changing the assertion or its observation affects
|
|
264
|
+
all its flows. Dependency, configuration and instrumentation changes can
|
|
265
|
+
affect many flows.
|
|
241
266
|
|
|
242
267
|
Start with `assertions report --view changes`. Investigate every listed change,
|
|
243
268
|
including effects on flows that did not yet watch the changed file. Add a
|
|
@@ -254,10 +279,25 @@ response in the map's top-level `changeAssessments` array:
|
|
|
254
279
|
}
|
|
255
280
|
```
|
|
256
281
|
|
|
257
|
-
This is an excerpt to add to your existing map.
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
282
|
+
This is an excerpt to add to your existing map. `affectedFlows` is your
|
|
283
|
+
judgement: the flows this change invalidates, and only those. It is not a
|
|
284
|
+
restatement of `knownFlows` -- Supercov already holds that list and does not
|
|
285
|
+
ask you to retype it. **Every flow you name loses its acknowledgement and has
|
|
286
|
+
to be reread**, so naming a flow is a decision, and naming all of them is the
|
|
287
|
+
expensive one.
|
|
288
|
+
|
|
289
|
+
To make that judgement, read what the change view gives you. `knownFlows`
|
|
290
|
+
counts the flows the change has already made stale and samples them. `exposed`
|
|
291
|
+
counts the flows whose selected tests ran the changed code and names those
|
|
292
|
+
tests -- not stale for it, but the ones most worth thinking about. Neither is
|
|
293
|
+
a verdict: under an integration suite every test runs most of the code, so
|
|
294
|
+
exposure is a place to look, not a list to copy.
|
|
295
|
+
|
|
296
|
+
An empty list is the normal outcome for a change that touches no claim, and it
|
|
297
|
+
is the contract this channel exists to provide: one explanation answers for the
|
|
298
|
+
change and every dependent keeps its credit. It still needs an explanation of
|
|
299
|
+
why existing claims are unaffected, and it does not mean the changed code is
|
|
300
|
+
tested.
|
|
261
301
|
|
|
262
302
|
Save the assessments and graph edits, validate, and copy the examined change
|
|
263
303
|
tokens. Save again, then validate once more before copying final flow tokens.
|
package/docs/cli.md
CHANGED
|
@@ -21,6 +21,7 @@ npx supercov --help
|
|
|
21
21
|
| Inspect one assertion and its flows | `npx supercov runs latest assertion <id>` |
|
|
22
22
|
| Read matching current source code | `npx supercov runs latest source <path>` |
|
|
23
23
|
| Compare two runs | `npx supercov diff <older> <newer>` |
|
|
24
|
+
| Find the tests a change affects | `npx supercov runs latest tests affected` |
|
|
24
25
|
| Combine shards | `npx supercov merge <id> <id> [...]` |
|
|
25
26
|
| Remove local data | `npx supercov clean` |
|
|
26
27
|
| Read bundled guides | `npx supercov docs` |
|
|
@@ -289,6 +290,27 @@ target. It does not edit, delete, or skip tests for you. Treat the result as an
|
|
|
289
290
|
analysis aid, not permission to remove tests that protect behavior outside the
|
|
290
291
|
selected metric.
|
|
291
292
|
|
|
293
|
+
## Find the tests a change affects
|
|
294
|
+
|
|
295
|
+
```sh supercov
|
|
296
|
+
npx supercov runs latest tests affected
|
|
297
|
+
npx supercov runs latest tests affected --files
|
|
298
|
+
npx supercov runs latest tests affected --json
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
`tests affected` names the tests of a run whose recorded execution the changes
|
|
302
|
+
since that run could have reached: a change in code the test ran, in its test
|
|
303
|
+
file, or in the shape of a file it ran code in -- a declaration added, removed
|
|
304
|
+
or renamed. A change confined to code the test never ran does not count, and
|
|
305
|
+
neither do comments, blank lines or trailing whitespace. A test that did not
|
|
306
|
+
pass in the run is listed regardless.
|
|
307
|
+
|
|
308
|
+
`--names` prints one affected test name per line and `--files` one test file
|
|
309
|
+
per line, for a runner's filter. A dependency, lockfile, configuration or
|
|
310
|
+
toolchain change affects every test and is reported as such. A source file
|
|
311
|
+
added since the run is outside every test's record; the working-tree check
|
|
312
|
+
says so, and the suite should run in full.
|
|
313
|
+
|
|
292
314
|
## Combine shards
|
|
293
315
|
|
|
294
316
|
```sh supercov
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "supercov",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.52",
|
|
4
4
|
"description": "Coverage for coding agents and software factories \ud83c\udf19",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -76,7 +76,7 @@
|
|
|
76
76
|
"benchmark:check": "cargo build --release -p supercov && node scripts/rust-transform-benchmark.mjs",
|
|
77
77
|
"benchmark:python-monitoring": "cargo build -p supercov && node scripts/python-monitoring-benchmark.mjs",
|
|
78
78
|
"check": "cargo fmt --all -- --check && cargo clippy --workspace --all-targets -- -D warnings && npm run test && npm run test:runtime && npm run test:rust-assets && npm run test:assertion-maps && node scripts/package-preflight.mjs && node scripts/verify-binstall-metadata.mjs --offline",
|
|
79
|
-
"release:check": "cargo clean && node scripts/sweep-target.mjs && npm run check && npm run test:engine && npm run test:go && npm run test:jvm && npm run test:fixture && npm run test:watchdog && npm run test:engine-contract && npm run test:agent && npm run test:child-attribution && npm run test:host-loader && npm run test:python-monitoring && npm run test:ruby-coverage && npm run test:rust-public-cargo && npm run test:packed-npx && npm run test:clang-mcdc && npm run benchmark:check",
|
|
79
|
+
"release:check": "cargo clean && node scripts/sweep-target.mjs && npm run check && npm run test:engine && npm run test:go && npm run test:jvm && npm run test:launcher && npm run test:fixture && npm run test:watchdog && npm run test:engine-contract && npm run test:agent && npm run test:child-attribution && npm run test:host-loader && npm run test:python-monitoring && npm run test:ruby-coverage && npm run test:rust-public-cargo && npm run test:packed-npx && npm run test:clang-mcdc && npm run benchmark:check",
|
|
80
80
|
"release:bump": "node scripts/bump-version.mjs",
|
|
81
81
|
"sweep": "node scripts/sweep-target.mjs",
|
|
82
82
|
"oracle:rust": "cargo build --release -p supercov && node scripts/rust-coverage-oracle.mjs",
|
|
@@ -93,17 +93,18 @@
|
|
|
93
93
|
"test:assertion-maps:js": "cargo build -p supercov && node scripts/assertion-map-schema.mjs --check && node scripts/assertion-map-js-integration.mjs && node scripts/assertion-dogfood-integration.mjs",
|
|
94
94
|
"sync:assertion-schema": "cargo build -p supercov && node scripts/assertion-map-schema.mjs",
|
|
95
95
|
"docs:sync": "node scripts/sync-docs.mjs",
|
|
96
|
-
"docs:check": "node scripts/sync-docs.mjs --check"
|
|
96
|
+
"docs:check": "node scripts/sync-docs.mjs --check",
|
|
97
|
+
"test:launcher": "go vet ./cmd/... && go test ./cmd/..."
|
|
97
98
|
},
|
|
98
99
|
"optionalDependencies": {
|
|
99
|
-
"@supercov/cli-darwin-arm64": "0.0.
|
|
100
|
-
"@supercov/cli-darwin-x64": "0.0.
|
|
101
|
-
"@supercov/cli-linux-arm64-gnu": "0.0.
|
|
102
|
-
"@supercov/cli-linux-arm64-musl": "0.0.
|
|
103
|
-
"@supercov/cli-linux-x64-gnu": "0.0.
|
|
104
|
-
"@supercov/cli-linux-x64-musl": "0.0.
|
|
105
|
-
"@supercov/cli-win32-arm64": "0.0.
|
|
106
|
-
"@supercov/cli-win32-x64": "0.0.
|
|
100
|
+
"@supercov/cli-darwin-arm64": "0.0.52",
|
|
101
|
+
"@supercov/cli-darwin-x64": "0.0.52",
|
|
102
|
+
"@supercov/cli-linux-arm64-gnu": "0.0.52",
|
|
103
|
+
"@supercov/cli-linux-arm64-musl": "0.0.52",
|
|
104
|
+
"@supercov/cli-linux-x64-gnu": "0.0.52",
|
|
105
|
+
"@supercov/cli-linux-x64-musl": "0.0.52",
|
|
106
|
+
"@supercov/cli-win32-arm64": "0.0.52",
|
|
107
|
+
"@supercov/cli-win32-x64": "0.0.52"
|
|
107
108
|
},
|
|
108
109
|
"peerDependencies": {
|
|
109
110
|
"@playwright/test": ">=1.55.0",
|
|
@@ -4,13 +4,50 @@ const NORMALIZED_KINDS = [
|
|
|
4
4
|
["integration", /(^|[/_.-])(integration|int)([/_.-]|$)/i],
|
|
5
5
|
["e2e", /(^|[/_.-])(e2e|end-to-end|offline|online)([/_.-]|$)/i],
|
|
6
6
|
];
|
|
7
|
-
|
|
8
|
-
|
|
7
|
+
// A runner that can only drive the whole system through its external interface
|
|
8
|
+
// is evidence, not a guess. Nothing else here is: a token in a filename is a
|
|
9
|
+
// convention someone may not have followed.
|
|
10
|
+
const STRONG_RUNNER_KINDS = { playwright: "e2e" };
|
|
11
|
+
// The most specific token in a segment wins, not the first one this list
|
|
12
|
+
// happens to hold. `checkout-integration-e2e` is an e2e spec whose name notes
|
|
13
|
+
// what it integrates; ordering by array position called it an integration.
|
|
14
|
+
function segmentKind(segment) {
|
|
15
|
+
if (!segment)
|
|
9
16
|
return undefined;
|
|
10
17
|
// gatewayE2e.test.ts and responseIntegration.test.ts are conventional
|
|
11
18
|
// camel-case paths too. Do not infer kinds from test titles or API usage.
|
|
12
|
-
const words =
|
|
13
|
-
|
|
19
|
+
const words = segment.replace(/([a-z0-9])([A-Z])/g, "$1-$2");
|
|
20
|
+
let best;
|
|
21
|
+
for (const [kind, pattern] of NORMALIZED_KINDS) {
|
|
22
|
+
const found = words.search(pattern);
|
|
23
|
+
if (found >= 0 && (best === undefined || found > best.at))
|
|
24
|
+
best = { kind, at: found };
|
|
25
|
+
}
|
|
26
|
+
return best?.kind;
|
|
27
|
+
}
|
|
28
|
+
function classifiedKind(value) {
|
|
29
|
+
if (!value)
|
|
30
|
+
return undefined;
|
|
31
|
+
return segmentKind(value);
|
|
32
|
+
}
|
|
33
|
+
// A directory is a deliberate choice about where a suite lives; a filename is
|
|
34
|
+
// often just a description of the thing under test. Read directories from the
|
|
35
|
+
// deepest inwards, so the nearest enclosing suite wins.
|
|
36
|
+
function directoryKind(file) {
|
|
37
|
+
if (!file)
|
|
38
|
+
return undefined;
|
|
39
|
+
const segments = file.split(/[/\\]/).slice(0, -1);
|
|
40
|
+
for (let index = segments.length - 1; index >= 0; index -= 1) {
|
|
41
|
+
const kind = segmentKind(segments[index]);
|
|
42
|
+
if (kind)
|
|
43
|
+
return kind;
|
|
44
|
+
}
|
|
45
|
+
return undefined;
|
|
46
|
+
}
|
|
47
|
+
function basenameKind(file) {
|
|
48
|
+
if (!file)
|
|
49
|
+
return undefined;
|
|
50
|
+
return segmentKind(file.split(/[/\\]/).pop());
|
|
14
51
|
}
|
|
15
52
|
export function inferTestProvenance({ runner, file, project, explicitKind, }) {
|
|
16
53
|
if (explicitKind?.trim()) {
|
|
@@ -30,20 +67,42 @@ export function inferTestProvenance({ runner, file, project, explicitKind, }) {
|
|
|
30
67
|
source: "project",
|
|
31
68
|
};
|
|
32
69
|
}
|
|
33
|
-
|
|
34
|
-
|
|
70
|
+
// A directory outranks the runner: putting a Playwright spec under
|
|
71
|
+
// `tests/integration/` is a statement about that suite.
|
|
72
|
+
const directory = directoryKind(file);
|
|
73
|
+
if (directory) {
|
|
74
|
+
return {
|
|
75
|
+
runner,
|
|
76
|
+
kind: directory,
|
|
77
|
+
...(project ? { project } : {}),
|
|
78
|
+
source: "path",
|
|
79
|
+
};
|
|
80
|
+
}
|
|
81
|
+
// A filename token does not. `storefront-empire-integration.spec.ts` drove
|
|
82
|
+
// a real browser; reading `integration` out of its name reported 102 lines
|
|
83
|
+
// as untouched by E2E while a browser had rendered them. The runner knows
|
|
84
|
+
// better than the filename here, so it is asked first.
|
|
85
|
+
const strong = STRONG_RUNNER_KINDS[runner];
|
|
86
|
+
if (strong) {
|
|
87
|
+
return {
|
|
88
|
+
runner,
|
|
89
|
+
kind: strong,
|
|
90
|
+
...(project ? { project } : {}),
|
|
91
|
+
source: "runner-default",
|
|
92
|
+
};
|
|
93
|
+
}
|
|
94
|
+
const basename = basenameKind(file);
|
|
95
|
+
if (basename) {
|
|
35
96
|
return {
|
|
36
97
|
runner,
|
|
37
|
-
kind:
|
|
98
|
+
kind: basename,
|
|
38
99
|
...(project ? { project } : {}),
|
|
39
100
|
source: "path",
|
|
40
101
|
};
|
|
41
102
|
}
|
|
42
|
-
const defaultKind = runner === "
|
|
43
|
-
? "
|
|
44
|
-
:
|
|
45
|
-
? "unit"
|
|
46
|
-
: "unknown";
|
|
103
|
+
const defaultKind = runner === "vitest" || runner === "jest" || runner === "node:test"
|
|
104
|
+
? "unit"
|
|
105
|
+
: "unknown";
|
|
47
106
|
return {
|
|
48
107
|
runner,
|
|
49
108
|
kind: defaultKind,
|
|
@@ -110,7 +110,7 @@
|
|
|
110
110
|
"string",
|
|
111
111
|
"null"
|
|
112
112
|
],
|
|
113
|
-
"pattern": "^
|
|
113
|
+
"pattern": "^scov[23]:[0-9a-f]{64}$"
|
|
114
114
|
},
|
|
115
115
|
"explanation": {
|
|
116
116
|
"type": "string"
|
|
@@ -236,7 +236,7 @@
|
|
|
236
236
|
"string",
|
|
237
237
|
"null"
|
|
238
238
|
],
|
|
239
|
-
"pattern": "^
|
|
239
|
+
"pattern": "^scov[23]:[0-9a-f]{64}$"
|
|
240
240
|
},
|
|
241
241
|
"affectedFlows": {
|
|
242
242
|
"type": "array",
|