testguard-cli 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.3] - 2026-09-17
11
+
12
+ From a second field report on a real codebase (456 tests, 27 claims, 35
13
+ faults; 13 survived on the first run, two critical claims with zero coverage).
14
+
15
+ ### Fixed
16
+
17
+ - **Worktree mode probed HEAD while reading the claims file from the working
18
+ tree**, so uncommitted defender changes were silently ignored — the same
19
+ survivors came back with no hint why. `probe` now refuses (exit 2) when any
20
+ resolved defender or fault target has uncommitted changes, naming the files
21
+ and the commit it would have probed. Every summary names the commit probed.
22
+ - `killed-by-undeclared-tests` never said which tests killed the fault; the
23
+ author could not fix `defendedBy` without grepping the suite. Evidence now
24
+ carries `detail.undeclaredKillers` and the CLI names the files.
25
+ - `anchor-ambiguous` did not say how many hits; `detail.anchor { hits,
26
+ expected }` is recorded and printed.
27
+ - A replacement with an unbalanced paren was `suite-failed-to-load`, not
28
+ `replacement-does-not-compile`: esbuild/vitest wording is now matched.
29
+ - The claims schema promised defender discovery for an absent `defendedBy`;
30
+ the tool answered `nocover`. Discovery is implemented: the test files that
31
+ import the fault's target (relative or alias), recorded as
32
+ `defenders.discovered`. `nocover` now means exactly "no test file imports
33
+ this source".
34
+ - `baseline.json` records `dirty`, as evidence already did.
35
+
36
+ ### Added
37
+
38
+ - `--include-dirty`: snapshot the working tree (tracked edits and untracked,
39
+ non-ignored files) into a throwaway commit and probe that. HEAD, index and
40
+ the user's tree are never touched; `run.repo.snapshot` records the commit.
41
+ - A one-line progress indicator on stderr (TTY only) so a minute of silence
42
+ is not mistaken for a hang.
43
+ - Fixture: a claim with no `defendedBy` whose defender is discovered.
44
+
45
+ ### Changed
46
+
47
+ - **Default output shows only unproven faults plus a killed count.**
48
+ `--verbose` restores the full stream.
49
+ - The `survived` hint reminds the author to check that the fault is
50
+ observable at all before writing a test for it.
51
+ - README: `$schema` path for consumers, `min-release-age` note, the
52
+ worktree-vs-working-tree rule, `brief` writes `brief.json` by default.
53
+
10
54
  ## [0.1.2] - 2026-09-17
11
55
 
12
56
  From a field report on a real codebase (63 test files, 458 tests, 39 faults).
package/README.md CHANGED
@@ -52,9 +52,13 @@ tests were written against the survivors, 39/39 were killed.
52
52
  | npm | `npm i -D testguard-cli` then `npx testguard probe` |
53
53
  | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
54
54
  | Homebrew | `brew tap raccioly/tap && brew install testguard` |
55
- | GitHub Action | `uses: raccioly/testguard@v0.1.2` — see [`action.yml`](./action.yml) |
55
+ | GitHub Action | `uses: raccioly/testguard@v0.1.3` — see [`action.yml`](./action.yml) |
56
56
  | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
57
57
 
58
+ Projects that set `min-release-age` in `.npmrc` cannot see a version published
59
+ less than that many days ago (`ENOVERSIONS`); install that one with
60
+ `npm i -D testguard-cli --min-release-age=0`.
61
+
58
62
  ## How it works
59
63
 
60
64
  ```bash
@@ -64,9 +68,11 @@ npx testguard-cli baseline # freeze today's unproven findings; from now on on
64
68
  npx testguard-cli brief # tell the agent where the suite is blind, before it writes
65
69
  ```
66
70
 
67
- 1. **Claims** live in `testguard.claims.json`: a statement, where it comes
68
- from, which tests supposedly defend it, and one or more *faults* — each a
69
- deterministic source change that would make the statement false. Every
71
+ 1. **Claims** live in `testguard.claims.json` (editors validate it against
72
+ `"$schema": "./node_modules/testguard-cli/spec/schemas/claims.schema.json"`):
73
+ a statement, where it comes from, which tests supposedly defend it, and
74
+ one or more *faults* — each a deterministic source change that would make
75
+ the statement false. Every
70
76
  claim and every fault records who produced it. `testguard claims`
71
77
  validates the file and reports drift against `@claim <ID>` annotations in
72
78
  source. Test files are deliberately not scanned — a claim asserted by a test is the authorship trap the tool exists for — and annotation ids must contain a hyphen so prose is never mistaken for one.
@@ -91,10 +97,23 @@ npx testguard-cli brief # tell the agent where the suite is blind, before
91
97
  written to `.testguard/evidence.json` — validated against the spec before
92
98
  it is written.
93
99
 
100
+ Worktree mode probes a **commit**. If a defender or target file has
101
+ uncommitted changes, `probe` refuses and says so — otherwise your new
102
+ tests would be silently absent and the same survivors would come back
103
+ with no hint why. `--include-dirty` snapshots the working tree (tracked
104
+ edits and new files) into a throwaway commit and probes that; your tree,
105
+ HEAD and index are never touched. Every summary names the commit probed.
106
+
107
+ A claim with no `defendedBy` has its defenders **discovered**: the test
108
+ files that import the fault's target, by relative path or resolved alias.
109
+ `NOCOVER` then means exactly "no test file imports this source".
110
+
94
111
  Practical loop: first pass `--no-escalate` (escalation re-runs the whole
95
- suite N times per survivor); iterate on one claim with
96
- `--claim <ID> --in-place` only the files faults are applied to must be
97
- committed, test files may be dirty; final pass with defaults. A custom
112
+ suite N times per survivor); iterate on one claim with `--claim <ID>` and
113
+ either `--include-dirty` or `--in-place` (only fault target files must be
114
+ clean there; test files may be dirty); final pass with defaults. By default
115
+ the stream shows only unproven faults plus a killed count — `--verbose`
116
+ shows every fault. A custom
98
117
  runner (`pnpm --filter`, a specific config) goes in
99
118
  `--runner-cmd "<cmd> {files} … {out}"`; if the scratch worktree cannot
100
119
  see your `node_modules`, pass `--node-modules <dir>`.
@@ -103,7 +122,8 @@ npx testguard-cli brief # tell the agent where the suite is blind, before
103
122
  source and defenders are unchanged reuse their prior verdict, so a probe
104
123
  in CI costs only what changed.
105
124
  4. **Brief** turns evidence plus baseline into a ranked, capped
106
- `## TEST BLINDSPOT CONTEXT` block. Wire it into an agent's session start
125
+ `## TEST BLINDSPOT CONTEXT` block, printed and also written to
126
+ `.testguard/brief.json` (`--text` prints only). Wire it into an agent's session start
107
127
  — for Claude Code, in `.claude/settings.json`:
108
128
 
109
129
  ```json
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "testguard-cli",
3
- "version": "0.1.2",
3
+ "version": "0.1.3",
4
4
  "description": "Proves a test suite defends the claims a project makes: injects the faults those claims forbid and reports every one the tests fail to detect.",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -35,10 +35,16 @@ Rules that follow from the table:
35
35
  reporting `survived` would be wrong the other way. It is `flaky-defender`.
36
36
  5. **Escalation never upgrades a verdict.** A fault that survives its declared
37
37
  defenders may be re-run against the whole suite. If the wider suite kills
38
- it, the verdict stays `survived` with reason `killed-by-undeclared-tests`:
39
- the claim's stated evidence chain is broken even though the suite is not
40
- blind. It gates, and ranks below a true survivor.
41
- 6. **Never a single global score.** Output is per claim, ranked. Blindness is
38
+ it, the verdict stays `survived` with reason `killed-by-undeclared-tests`
39
+ and `detail.undeclaredKillers` names the tests, so the author can fix
40
+ `defendedBy`. The claim's stated evidence chain is broken even though the
41
+ suite is not blind. It gates, and ranks below a true survivor.
42
+ 6. **A verdict names the commit it is about.** Evidence records `repo.head`;
43
+ when the working tree was probed instead, `repo.snapshot` holds the
44
+ throwaway commit that captured it. A tool must refuse to probe a commit
45
+ while defenders or targets have uncommitted changes, unless told to
46
+ snapshot the working tree — otherwise the answer looks right and is not.
47
+ 7. **Never a single global score.** Output is per claim, ranked. Blindness is
42
48
  concentrated, and one number hides where.
43
49
 
44
50
  ## Baseline and delta
@@ -72,6 +72,9 @@ const semantic = {
72
72
  if (r.verdict === 'nocover' && !r.defenders.nocover) {
73
73
  errors.push({ path: `${p}/defenders/nocover`, message: 'nocover verdict requires defenders.nocover = true' });
74
74
  }
75
+ if (r.detail.undeclaredKillers && r.detail.reason !== 'killed-by-undeclared-tests') {
76
+ errors.push({ path: `${p}/detail/undeclaredKillers`, message: 'undeclaredKillers is only meaningful with reason killed-by-undeclared-tests' });
77
+ }
75
78
  if (r.verdict === 'unverifiable' && !r.detail.reason) {
76
79
  errors.push({ path: `${p}/detail/reason`, message: 'unverifiable requires a reason (e.g. anchor-missing, anchor-ambiguous)' });
77
80
  }
@@ -11,6 +11,10 @@
11
11
  "tool": { "$ref": "urn:guard-spec:v1:common#/$defs/tool" },
12
12
  "createdAt": { "$ref": "urn:guard-spec:v1:common#/$defs/isoDateTime" },
13
13
  "head": { "$ref": "urn:guard-spec:v1:common#/$defs/gitSha" },
14
+ "dirty": {
15
+ "description": "Whether the working tree had uncommitted changes when the evidence behind this baseline was taken.",
16
+ "type": "boolean"
17
+ },
14
18
  "fingerprints": {
15
19
  "type": "object",
16
20
  "propertyNames": { "pattern": "^[a-f0-9]{64}$" },
@@ -42,7 +42,7 @@
42
42
  "severity": { "$ref": "urn:guard-spec:v1:common#/$defs/severity" },
43
43
  "producedBy": { "$ref": "urn:guard-spec:v1:common#/$defs/provenance" },
44
44
  "defendedBy": {
45
- "description": "Globs (repo-relative) for the test files that supposedly defend this claim. Empty or absent means: let the runner discover related tests. If the globs resolve to zero files the verdict is `nocover`.",
45
+ "description": "Globs (project-relative) for the test files that supposedly defend this claim. Empty or absent means: discover defenders as the test files that import each fault's target file (relative paths and resolved aliases). If nothing resolves, the verdict is `nocover`.",
46
46
  "type": "array",
47
47
  "items": { "type": "string", "minLength": 1 },
48
48
  "uniqueItems": true
@@ -21,7 +21,11 @@
21
21
  "required": ["head", "dirty"],
22
22
  "properties": {
23
23
  "head": { "$ref": "urn:guard-spec:v1:common#/$defs/gitSha" },
24
- "dirty": { "type": "boolean" }
24
+ "dirty": { "type": "boolean" },
25
+ "snapshot": {
26
+ "description": "When the working tree was probed instead of `head`: the dangling commit that captured it. Not a ref; may be garbage-collected.",
27
+ "$ref": "urn:guard-spec:v1:common#/$defs/gitSha"
28
+ }
25
29
  },
26
30
  "additionalProperties": false
27
31
  },
@@ -149,6 +153,22 @@
149
153
  "escalationRuns": {
150
154
  "type": "array",
151
155
  "items": { "$ref": "#/$defs/testRun" }
156
+ },
157
+ "undeclaredKillers": {
158
+ "description": "With reason `killed-by-undeclared-tests`: the tests (as `file::name`) that failed in every escalation run. Names what to add to `defendedBy`.",
159
+ "type": "array",
160
+ "minItems": 1,
161
+ "items": { "type": "string", "minLength": 1, "maxLength": 1024 }
162
+ },
163
+ "anchor": {
164
+ "description": "With an anchor-related reason: how many times `find` occurred versus how many were expected.",
165
+ "type": "object",
166
+ "required": ["hits", "expected"],
167
+ "properties": {
168
+ "hits": { "type": "integer", "minimum": 0 },
169
+ "expected": { "type": "integer", "minimum": 1 }
170
+ },
171
+ "additionalProperties": false
152
172
  }
153
173
  },
154
174
  "additionalProperties": false
@@ -159,7 +179,11 @@
159
179
  "properties": {
160
180
  "requested": { "type": "array", "items": { "type": "string" } },
161
181
  "resolved": { "type": "array", "items": { "$ref": "urn:guard-spec:v1:common#/$defs/repoPath" } },
162
- "nocover": { "type": "boolean" }
182
+ "nocover": { "type": "boolean" },
183
+ "discovered": {
184
+ "description": "True when no defenders were declared and `resolved` was found by static import of the subject's file.",
185
+ "type": "boolean"
186
+ }
163
187
  },
164
188
  "additionalProperties": false
165
189
  },
@@ -7,7 +7,7 @@ export function buildBaseline(evidence, { createdAt = new Date().toISOString() }
7
7
  if (r.verdict === 'killed') continue;
8
8
  fingerprints[r.fingerprint] = (fingerprints[r.fingerprint] ?? 0) + 1;
9
9
  }
10
- return { schemaVersion: 1, tool: evidence.tool, createdAt, head: evidence.run.repo.head, fingerprints };
10
+ return { schemaVersion: 1, tool: evidence.tool, createdAt, head: evidence.run.repo.head, dirty: evidence.run.repo.dirty, fingerprints };
11
11
  }
12
12
 
13
13
  /**
@@ -11,7 +11,7 @@ export function hintFor(r) {
11
11
  case 'survived':
12
12
  return r.detail.reason === 'killed-by-undeclared-tests'
13
13
  ? `Only tests outside its declared defenders (${defenders}) catch this; fix the claim's defendedBy or move the assertion.`
14
- : `${defenders} stayed green with this fault applied; add an assertion that fails on it and passes on HEAD.`;
14
+ : `${defenders} stayed green with this fault applied; add an assertion that fails on it and passes on HEAD. If no test's outcome can change, first check the fault is observable at all.`;
15
15
  case 'nocover':
16
16
  return `No test file matches ${r.defenders.requested.join(', ') || '(no defenders declared)'}; nothing defends this claim.`;
17
17
  case 'unverifiable':
package/src/cli.mjs CHANGED
@@ -29,6 +29,8 @@ probe
29
29
  --severity <level> gate only at or above (default: low)
30
30
  --ref <commit> probe this commit in the scratch worktree (default: HEAD)
31
31
  --claim <ID,ID> probe only these claims; writes .testguard/evidence-partial.json
32
+ --include-dirty probe the working tree (a snapshot commit) instead of HEAD; uncommitted tests count
33
+ --verbose also print each killed fault (default: only unproven ones, plus a count)
32
34
  --runner-cmd "<cmd>" custom runner; must contain {files} and {out}, e.g. "pnpm vitest run {files} --reporter=json --outputFile={out}"
33
35
  --node-modules <dir> node_modules to link into the scratch worktree (or TESTGUARD_NODE_MODULES)
34
36
  --in-place mutate the working tree instead of a scratch worktree
@@ -62,6 +64,8 @@ export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n
62
64
  max: { type: 'string', default: '20' },
63
65
  ref: { type: 'string', default: 'HEAD' },
64
66
  claim: { type: 'string' },
67
+ 'include-dirty': { type: 'boolean', default: false },
68
+ verbose: { type: 'boolean', default: false },
65
69
  'runner-cmd': { type: 'string' },
66
70
  'node-modules': { type: 'string' },
67
71
  'in-place': { type: 'boolean', default: false },
@@ -15,7 +15,7 @@ export async function baselineCommand({ projectDir, values }, io) {
15
15
  const outPath = values.out ? resolve(values.out) : baselinePath(projectDir);
16
16
  writeSpecDoc('baseline', outPath, baseline);
17
17
  const n = Object.values(baseline.fingerprints).reduce((a, b) => a + b, 0);
18
- io.out(`baseline: ${n} unproven finding${n === 1 ? '' : 's'} frozen at ${baseline.head.slice(0, 12)} → ${outPath}`);
18
+ io.out(`baseline: ${n} unproven finding${n === 1 ? '' : 's'} frozen at ${baseline.head.slice(0, 12)}${baseline.dirty ? ' (working tree was dirty)' : ''} → ${outPath}`);
19
19
  io.out('Commit this file; from now on only new findings gate. Ignore the regenerated ones — add to .gitignore:');
20
20
  io.out(' .testguard/evidence.json');
21
21
  io.out(' .testguard/brief.json');
@@ -2,6 +2,7 @@ import { resolve } from 'node:path';
2
2
  import { loadClaims, defaultClaimsPath } from '../claims/load.mjs';
3
3
  import { scanAnnotations, reconcile } from '../claims/annotations.mjs';
4
4
  import { resolveDefenders } from '../probe/runner-vitest.mjs';
5
+ import { discoverDefenders } from '../probe/discover.mjs';
5
6
 
6
7
  export async function claimsCommand({ projectDir, values }, io) {
7
8
  const path = values.claims ? resolve(values.claims) : defaultClaimsPath(projectDir);
@@ -16,8 +17,9 @@ export async function claimsCommand({ projectDir, values }, io) {
16
17
  io.out(`${claims.claims.length} claims in ${path} — ${annotated.size} carry a @claim annotation in source (test files are not scanned)`);
17
18
  io.out('');
18
19
  for (const c of claims.claims) {
19
- const defenders = resolveDefenders(projectDir, c.defendedBy);
20
- const cover = defenders.length ? `${defenders.length} defender${defenders.length === 1 ? '' : 's'}` : 'NO DEFENDER';
20
+ const declared = c.defendedBy?.length > 0;
21
+ const defenders = declared ? resolveDefenders(projectDir, c.defendedBy) : [...new Set(c.faults.flatMap((f) => discoverDefenders(projectDir, f.file)))];
22
+ const cover = defenders.length ? `${defenders.length} ${declared ? 'defender' : 'discovered'}${defenders.length === 1 ? '' : 's'}` : 'NO DEFENDER';
21
23
  io.out(`${annotated.has(c.id) ? '@ ' : ' '}${c.id.padEnd(14)} ${c.severity.padEnd(8)} ${c.source.kind.padEnd(10)} ${String(c.faults.length).padStart(2)} fault${c.faults.length === 1 ? ' ' : 's'} ${cover.padEnd(12)} ${c.statement}`);
22
24
  }
23
25
  if (drift.undeclared.length || drift.stale.length) io.out('');
@@ -41,7 +41,12 @@ export async function probeCommand({ projectDir, values, version }, io) {
41
41
  only,
42
42
  escalate: !values['no-escalate'],
43
43
  toolVersion: version,
44
- onProgress: values.quiet ? undefined : (r) => io.out(renderRecord(r) + (r.reusedFrom ? ' (reused)' : '')),
44
+ includeDirty: values['include-dirty'],
45
+ onStage: !values.quiet && process.stderr.isTTY ? ({ claimId, faultId, stage, i, n }) => process.stderr.write(`\r\x1b[K … ${claimId}/${faultId} ${stage} ${i}/${n}`) : undefined,
46
+ onProgress: values.quiet ? undefined : (r) => {
47
+ if (process.stderr.isTTY) process.stderr.write('\r\x1b[K');
48
+ if (values.verbose || r.verdict !== 'killed') io.out(renderRecord(r) + (r.reusedFrom ? ' (reused)' : ''));
49
+ },
45
50
  });
46
51
  writeSpecDoc('evidence', outPath, evidence);
47
52
 
@@ -52,7 +57,11 @@ export async function probeCommand({ projectDir, values, version }, io) {
52
57
  for (const r of sortForReport(evidence.records).filter((x) => x.verdict !== 'killed')) io.out(' ' + tag(r) + renderRecord(r));
53
58
  }
54
59
  io.out('');
55
- io.out(renderSummary(evidence.records) + (baseline ? ` ${g.new.length} new since baseline, ${g.baselined.length} baselined.` : ' No baseline.'));
60
+ if (!values.quiet && !values.verbose) {
61
+ const killed = evidence.records.filter((r) => r.verdict === 'killed').length;
62
+ if (killed) io.out(` ${killed} killed (not listed; --verbose to see them)`);
63
+ }
64
+ io.out(renderSummary(evidence.records, evidence.run) + (baseline ? ` ${g.new.length} new since baseline, ${g.baselined.length} baselined.` : ' No baseline.'));
56
65
  io.out(`evidence: ${outPath}${only ? ` (partial: --claim ${only.join(',')}; not the canonical evidence file)` : ''}`);
57
66
  return g.new.length > 0 ? 1 : 0;
58
67
  }
package/src/git.mjs CHANGED
@@ -1,10 +1,13 @@
1
1
  import { spawnSync } from 'node:child_process';
2
2
  import { rmSync } from 'node:fs';
3
+ import { tmpdir } from 'node:os';
4
+ import { join } from 'node:path';
5
+ import { randomBytes } from 'node:crypto';
3
6
 
4
7
  export class GitError extends Error {}
5
8
 
6
- export function git(args, cwd) {
7
- const r = spawnSync('git', args, { cwd, encoding: 'utf8' });
9
+ export function git(args, cwd, env) {
10
+ const r = spawnSync('git', args, { cwd, encoding: 'utf8', env: env ? { ...process.env, ...env } : process.env });
8
11
  if (r.status !== 0) throw new GitError(`git ${args.join(' ')}: ${(r.stderr || r.stdout).trim()}`);
9
12
  return r.stdout.trim();
10
13
  }
@@ -25,6 +28,29 @@ export const isDirty = (dir, paths = []) => git(['status', '--porcelain', '--',
25
28
 
26
29
  export const addWorktree = (repo, dest, ref = 'HEAD') => git(['worktree', 'add', '--detach', dest, ref], repo);
27
30
 
31
+ /**
32
+ * A dangling commit holding the working tree exactly as it is — tracked
33
+ * changes AND untracked (non-ignored) files — without touching HEAD, the
34
+ * index, or any ref. Built through a temporary index so the user's staging
35
+ * area is never read or written.
36
+ */
37
+ export function snapshotWorkingTree(repo) {
38
+ const index = join(tmpdir(), `testguard-index-${randomBytes(6).toString('hex')}`);
39
+ const env = {
40
+ GIT_INDEX_FILE: index,
41
+ GIT_AUTHOR_NAME: 'testguard', GIT_AUTHOR_EMAIL: 'testguard@localhost',
42
+ GIT_COMMITTER_NAME: 'testguard', GIT_COMMITTER_EMAIL: 'testguard@localhost',
43
+ };
44
+ try {
45
+ git(['read-tree', 'HEAD'], repo, env);
46
+ git(['add', '-A', '--', '.'], repo, env);
47
+ const tree = git(['write-tree'], repo, env);
48
+ return git(['commit-tree', tree, '-p', 'HEAD', '-m', 'testguard: working-tree snapshot (not a ref)'], repo, env);
49
+ } finally {
50
+ rmSync(index, { force: true });
51
+ }
52
+ }
53
+
28
54
  export function removeWorktree(repo, dest) {
29
55
  spawnSync('git', ['worktree', 'remove', '--force', dest], { cwd: repo });
30
56
  rmSync(dest, { recursive: true, force: true });
@@ -15,7 +15,8 @@ export function classify({ defenders, anchor, baselineRuns, probeRuns, confirmRu
15
15
  }
16
16
  const loadError = probeRuns.find((r) => r.outcome === 'error');
17
17
  if (loadError) {
18
- const parseError = /syntax|parse/i.test(loadError.loadMessage ?? '');
18
+ // esbuild/vitest wording: "Transform failed with 1 error", `Expected ")" but found ";"`, "Unexpected token"
19
+ const parseError = /syntax|parse|transform failed|expected .+ but found|unexpected token/i.test(loadError.loadMessage ?? '');
19
20
  return { verdict: 'fault-invalid', reason: parseError ? 'replacement-does-not-compile' : 'suite-failed-to-load' };
20
21
  }
21
22
  if (probeRuns.some((r) => r.outcome === 'timeout' || r.timeouts > 0)) return { verdict: 'timeout', reason: 'test-timed-out' };
@@ -0,0 +1,12 @@
1
+ import { join } from 'node:path';
2
+ import { listTestFiles } from './runner-vitest.mjs';
3
+ import { fileImports } from './rank.mjs';
4
+
5
+ /**
6
+ * When a claim declares no defenders: the test files that import the fault's
7
+ * target file, directly, by relative path or through a resolved alias. This
8
+ * is what `nocover` measures against — no test file even imports the source.
9
+ */
10
+ export function discoverDefenders(projectDir, targetRel) {
11
+ return listTestFiles(projectDir).filter((t) => fileImports(projectDir, join(projectDir, t), targetRel));
12
+ }
@@ -1,7 +1,9 @@
1
- import { existsSync, readFileSync } from 'node:fs';
1
+ import { existsSync, readFileSync, realpathSync } from 'node:fs';
2
+ import { git } from '../git.mjs';
2
3
  import { createRequire } from 'node:module';
3
4
  import { join, relative, resolve } from 'node:path';
4
- import { repoRoot as gitRoot, headSha, isDirty } from '../git.mjs';
5
+ import { repoRoot as gitRoot, headSha, isDirty, snapshotWorkingTree } from '../git.mjs';
6
+ import { discoverDefenders } from './discover.mjs';
5
7
  import { createScratch, inPlace, PreconditionError } from './worktree.mjs';
6
8
  import { applyFault, locate } from './inject.mjs';
7
9
  import * as vitest from './runner-vitest.mjs';
@@ -34,13 +36,17 @@ export async function probe({
34
36
  runnerCommand,
35
37
  nodeModules,
36
38
  only,
39
+ includeDirty = false,
40
+ onStage = () => {},
37
41
  escalate = true,
38
42
  scratchBase,
39
43
  toolVersion = '0.0.0',
40
44
  previous,
41
45
  onProgress = () => {},
42
46
  }) {
43
- projectDir = resolve(projectDir);
47
+ // realpath: git reports the repository root by its real path (/private/var
48
+ // on macOS, not /var); every relative() below must start from the same place.
49
+ projectDir = realpathSync(resolve(projectDir));
44
50
  const root = gitRoot(projectDir);
45
51
  if (mode === 'in-place' && ref !== 'HEAD') throw new PreconditionError('--ref needs a scratch worktree; drop --in-place');
46
52
  const head = headSha(root, ref);
@@ -50,6 +56,8 @@ export async function probe({
50
56
  if (mode === 'in-place' && isDirty(root, targets)) {
51
57
  throw new PreconditionError(`uncommitted changes in fault target files (${targets.join(', ')}); commit or stash them, or drop --in-place. Only the files faults are applied to must be clean — test files may be dirty, which is what makes --in-place usable while writing tests.`);
52
58
  }
59
+ if (includeDirty && mode !== 'worktree') throw new PreconditionError('--include-dirty applies to worktree mode; drop --in-place');
60
+ if (includeDirty && ref !== 'HEAD') throw new PreconditionError('--include-dirty snapshots the working tree; it cannot be combined with --ref');
53
61
  const selected = only ? new Set(only) : null;
54
62
  if (selected) {
55
63
  const known = new Set(claims.claims.map((c) => c.id));
@@ -57,8 +65,28 @@ export async function probe({
57
65
  if (unknown.length) throw new PreconditionError(`--claim: unknown claim id(s) ${unknown.join(', ')}`);
58
66
  }
59
67
 
68
+ // Worktree mode probes a commit, not the working tree. Uncommitted defender
69
+ // or target edits would be silently absent — the same survivors, no hint why.
70
+ let snapshot;
71
+ if (mode === 'worktree') {
72
+ if (includeDirty) {
73
+ if (isDirty(root)) snapshot = snapshotWorkingTree(root);
74
+ } else if (ref === 'HEAD') {
75
+ const watched = new Set(targets);
76
+ for (const claim of claims.claims) {
77
+ if (selected && !selected.has(claim.id)) continue;
78
+ const declared = claim.defendedBy?.length ? vitest.resolveDefenders(projectDir, claim.defendedBy) : claim.faults.flatMap((f) => discoverDefenders(projectDir, f.file));
79
+ for (const d of declared) watched.add(relative(root, join(projectDir, d)));
80
+ for (const g of claim.defendedBy ?? []) if (!g.includes('*')) watched.add(relative(root, join(projectDir, g)));
81
+ }
82
+ const dirty = git(['status', '--porcelain', '--', ...watched], root).split('\n').filter(Boolean).map((l) => l.replace(/^[ MADRCU?!]{1,2}\s+/, '').replace(/^.* -> /, ''));
83
+ if (dirty.length) {
84
+ throw new PreconditionError(`${dirty.length} defender/target file${dirty.length === 1 ? ' has' : 's have'} uncommitted changes (${dirty.join(', ')}); worktree mode probes HEAD (${head.slice(0, 7)}), so those changes would be silently ignored. Commit them, run with --include-dirty to probe the working tree, or use --in-place.`);
85
+ }
86
+ }
87
+ }
60
88
  const startedAt = new Date().toISOString();
61
- const iso = mode === 'worktree' ? createScratch({ repoRoot: root, projectDir, ref, scratchBase, nodeModules }) : inPlace({ repoRoot: root, projectDir });
89
+ const iso = mode === 'worktree' ? createScratch({ repoRoot: root, projectDir, ref: snapshot ?? ref, scratchBase, nodeModules }) : inPlace({ repoRoot: root, projectDir });
62
90
  const records = [];
63
91
  let runnerVersion;
64
92
  try {
@@ -81,9 +109,11 @@ export async function probe({
81
109
 
82
110
  for (const claim of claims.claims) {
83
111
  if (selected && !selected.has(claim.id)) continue;
84
- const defenders = vitest.resolveDefenders(iso.projectDir, claim.defendedBy);
112
+ const declared = claim.defendedBy?.length ? vitest.resolveDefenders(iso.projectDir, claim.defendedBy) : null;
85
113
  for (const fault of claim.faults) {
86
- const record = await probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, escalate, baselineCache, runDefenders, prior: prior.get(`${claim.id}/${fault.id}`), priorRunId: previous?.run.id });
114
+ const defenders = declared ?? discoverDefenders(iso.projectDir, fault.file);
115
+ const stage = (name, i, n) => onStage({ claimId: claim.id, faultId: fault.id, stage: name, i, n });
116
+ const record = await probeOne({ claim, fault, defenders, discovered: declared === null, allTests, iso, confirmRuns, escalate, baselineCache, runDefenders, stage, prior: prior.get(`${claim.id}/${fault.id}`), priorRunId: previous?.run.id });
87
117
  records.push(record);
88
118
  onProgress(record);
89
119
  }
@@ -99,7 +129,7 @@ export async function probe({
99
129
  id: `run-${startedAt.replace(/[-:.]/g, '').slice(0, 15)}`,
100
130
  startedAt,
101
131
  finishedAt: new Date().toISOString(),
102
- repo: { head, dirty: isDirty(root) },
132
+ repo: { head, dirty: isDirty(root), ...(snapshot ? { snapshot } : {}) },
103
133
  runner: { name: vitest.name, ...((runnerVersion ?? readRunnerVersion(projectDir)) ? { version: runnerVersion ?? readRunnerVersion(projectDir) } : {}) },
104
134
  confirmRuns,
105
135
  mode,
@@ -108,7 +138,7 @@ export async function probe({
108
138
  };
109
139
  }
110
140
 
111
- async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, escalate, baselineCache, runDefenders, prior, priorRunId }) {
141
+ async function probeOne({ claim, fault, defenders, discovered, allTests, iso, confirmRuns, escalate, baselineCache, runDefenders, stage, prior, priorRunId }) {
112
142
  const targetPath = join(iso.projectDir, fault.file);
113
143
  const targetExists = existsSync(targetPath);
114
144
  const inputs = {
@@ -134,6 +164,7 @@ async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, e
134
164
  const runs = [];
135
165
  let loadMessage;
136
166
  for (let i = 0; i < confirmRuns; i++) {
167
+ stage('baseline', i + 1, confirmRuns);
137
168
  const res = await runDefenders(defenders);
138
169
  runs.push(res.run);
139
170
  if (res.run.outcome !== 'pass') {
@@ -156,6 +187,7 @@ async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, e
156
187
  try {
157
188
  const probeRuns = rawProbeRuns;
158
189
  for (let i = 0; i < confirmRuns; i++) {
190
+ stage('probe', i + 1, confirmRuns);
159
191
  const { run, timeouts, loadMessage } = await runDefenders(defenders);
160
192
  probeRuns.push({ ...run, timeouts, loadMessage });
161
193
  if (shouldStopEarly(probeRuns)) break;
@@ -171,6 +203,7 @@ async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, e
171
203
  const runs = [];
172
204
  let killers = null;
173
205
  for (let i = 0; i < confirmRuns; i++) {
206
+ stage('escalation', i + 1, confirmRuns);
174
207
  const { run, failedTests } = await runDefenders(allTests);
175
208
  runs.push(run);
176
209
  killers = killers === null ? new Set(failedTests) : new Set(failedTests.filter((t) => killers.has(t)));
@@ -178,7 +211,10 @@ async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, e
178
211
  }
179
212
  detail.escalated = true;
180
213
  detail.escalationRuns = runs;
181
- if (killers.size > 0 && runs.length === confirmRuns && runs.every(isKill)) detail.reason = 'killed-by-undeclared-tests';
214
+ if (killers.size > 0 && runs.length === confirmRuns && runs.every(isKill)) {
215
+ detail.reason = 'killed-by-undeclared-tests';
216
+ detail.undeclaredKillers = [...killers].sort();
217
+ }
182
218
  }
183
219
  } finally {
184
220
  mutation.restore();
@@ -189,6 +225,7 @@ async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, e
189
225
 
190
226
  const { verdict, reason } = classify({ defenders, anchor, baselineRuns: detail.baselineRuns, probeRuns: rawProbeRuns, confirmRuns });
191
227
  if (reason && !detail.reason) detail.reason = reason;
228
+ if (anchor && anchor.status !== 'ok' && anchor.status !== 'file-missing' && anchor.status !== 'defenders-failed-to-load') detail.anchor = { hits: anchor.hits, expected: anchor.expected };
192
229
 
193
230
  const blast = targetExists ? blastRadius(iso.projectDir, fault.file) : 0;
194
231
 
@@ -198,7 +235,7 @@ async function probeOne({ claim, fault, defenders, allTests, iso, confirmRuns, e
198
235
  subject: { kind: 'fault', id: fault.id, description: fault.description, file: fault.file, faultClass: fault.faultClass, producedBy: fault.producedBy },
199
236
  verdict,
200
237
  detail,
201
- defenders: { requested: claim.defendedBy ?? [], resolved: defenders, nocover: defenders.length === 0 },
238
+ defenders: { requested: claim.defendedBy ?? [], resolved: defenders, nocover: defenders.length === 0, ...(discovered ? { discovered: true } : {}) },
202
239
  inputs,
203
240
  rank: rank({ severity: claim.severity, sourceKind: claim.source.kind, blast }),
204
241
  };
@@ -94,6 +94,20 @@ function resolvesTo(fromFile, specifier, targetAbs, aliases) {
94
94
  return bases.some((b) => expandBase(b).includes(targetAbs));
95
95
  }
96
96
 
97
+ /** Does the file at `absFile` import `targetRel` (relative or alias-resolved)? */
98
+ export function fileImports(projectDir, absFile, targetRel) {
99
+ const targetAbs = resolve(projectDir, targetRel);
100
+ const aliases = loadAliases(projectDir);
101
+ let src;
102
+ try {
103
+ src = readFileSync(absFile, 'utf8');
104
+ } catch {
105
+ return false;
106
+ }
107
+ for (const m of src.matchAll(IMPORT_RE)) if (resolvesTo(absFile, m[1], targetAbs, aliases)) return true;
108
+ return false;
109
+ }
110
+
97
111
  /**
98
112
  * Number of non-test source files that import `targetRel`: relative specifiers,
99
113
  * tsconfig/jsconfig `paths` aliases and package.json `imports` are resolved;
package/src/render.mjs CHANGED
@@ -5,7 +5,16 @@ export const formatVerdict = (v) => (v === 'killed' ? 'killed' : v.toUpperCase()
5
5
 
6
6
  export function renderRecord(r) {
7
7
  const head = `${formatVerdict(r.verdict).padEnd(15)} ${r.claim.id}/${r.subject.id}`.padEnd(38);
8
- const why = r.detail.reason ? ` [${r.detail.reason}]` : '';
8
+ let why = '';
9
+ if (r.detail.reason === 'killed-by-undeclared-tests' && r.detail.undeclaredKillers?.length) {
10
+ const files = [...new Set(r.detail.undeclaredKillers.map((k) => k.split('::')[0]))];
11
+ why = ` [killed-by-undeclared-tests: ${files.join(', ')}]`;
12
+ } else if (r.detail.reason === 'anchor-ambiguous' && r.detail.anchor) {
13
+ why = ` [anchor-ambiguous: ${r.detail.anchor.hits} hits, expected ${r.detail.anchor.expected}]`;
14
+ } else if (r.detail.reason) {
15
+ why = ` [${r.detail.reason}]`;
16
+ }
17
+ if (r.defenders.discovered) why += ' (defenders discovered by import)';
9
18
  return `${head} ${r.claim.severity.padEnd(8)} ${r.subject.file} ${r.subject.description}${why}`;
10
19
  }
11
20
 
@@ -15,12 +24,13 @@ export function summarize(records) {
15
24
  return byVerdict;
16
25
  }
17
26
 
18
- export function renderSummary(records) {
27
+ export function renderSummary(records, run) {
19
28
  const byVerdict = summarize(records);
20
29
  const parts = ORDER.filter((v) => byVerdict[v]).map((v) => `${byVerdict[v]} ${formatVerdict(v)}`);
21
30
  const unproven = records.filter((r) => r.verdict !== 'killed');
22
31
  const claims = new Set(unproven.map((r) => r.claim.id)).size;
23
- return `${records.length} faults probed: ${parts.join(', ')}. ${unproven.length} unproven fault${unproven.length === 1 ? '' : 's'} across ${claims} claim${claims === 1 ? '' : 's'}.`;
32
+ const where = run ? ` Probed ${run.repo.snapshot ? `working tree (snapshot ${run.repo.snapshot.slice(0, 7)} of ${run.repo.head.slice(0, 7)})` : run.mode === 'in-place' ? `in place at ${run.repo.head.slice(0, 7)}${run.repo.dirty ? ' (dirty)' : ''}` : run.repo.head.slice(0, 7)}.` : '';
33
+ return `${records.length} faults probed: ${parts.join(', ')}. ${unproven.length} unproven fault${unproven.length === 1 ? '' : 's'} across ${claims} claim${claims === 1 ? '' : 's'}.${where}`;
24
34
  }
25
35
 
26
36
  /** Survivors first, then by rank score; killed last. */