canary-test-cli 7.1.0 → 8.0.0

package/agents/skills/claude-code/canary-cassandra/SKILL.md

@@ -0,0 +1,187 @@
+---
+name: canary-cassandra
+description: >
+  Vacuous-test detection — finds tests that PASS WITHOUT PROVING ANYTHING: an
+  assertion that compares a value with itself, a test that never invokes the
+  target it claims to cover, and a test whose every assertion is an absence, or
+  a trivially true presence check, observed on a bystander rather than on the
+  code under test. Use when the user says "why did this pass against the bug",
+  "are these tests actually testing anything", "audit my suite for vacuous
+  tests", "green but worthless", or after a bug shipped through a green suite.
+  Advisory and deterministic — no LLM, no execution. NOT for tests with zero
+  assertions (that is `canary review-test`'s LINT-006), NOT for flaky tests
+  (canary-flake-hunter), and NOT a coverage tool — a vacuous test has coverage,
+  which is exactly why coverage never caught it.
+cli: scripts/cli.mjs
+requires: [node>=20]
+---
+
+# Canary Cassandra
+
+> Cassandra Cain reads the fake. So does this skill.
+
+A test with no assertions is easy to find and everybody already looks for it.
+The dangerous test is the one with assertions that **cannot fail** — it has
+coverage, it goes green, and it goes green identically against the bug it was
+written to catch. Three of those shipped in this project in a single cycle
+(recorded in [#486]) and every gate the repo owned read all three as healthy.
+
+Cassandra is Tier-0: deterministic, no LLM, no network, no execution.
+
+## What it finds
+
+| Rule      | Severity | Fires on                                                                                                                                                          |
+| --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `VAC-001` | critical | An assertion whose expectation is identical to the value it checks — `expect(true).toBe(true)`, `assert x == x`                                                   |
+| `VAC-002` | warning  | The test never references the target it claims to cover                                                                                                           |
+| `VAC-003` | warning  | Every assertion in the test asserts an _absence_, and none of them observes the target — so nothing proves it ran                                                 |
+| `VAC-005` | warning  | Every assertion is a trivially true _presence_ check (`toBeDefined`, `toBeTruthy`, `assert x is not None`) on a value the test built itself before the target ran |
+
+`VAC-001` is deterministic, hence `critical`: no implementation can fail it.
+`VAC-005` abstains (no finding) whenever it cannot prove the subject is a
+bystander, such as a name bound in a hook or a multi-line initialiser. `VAC-004`
+is reserved for the self-excusing-skip rule
+(`docs/changes/vac-004-self-excusing-skip/`). `VAC-002` and `VAC-003` depend on
+resolving a target, which is inference, so they are `warning` and carry a
+fidelity tier.
+
+## Run it
+
+Two doors, one detector. Both run the same engine rules, so they cannot disagree
+about a finding or about the denominator.
+
+```bash
+canary vacuity-check tests/            # human-readable, skips counted per reason
+canary vacuity-check tests/ --verbose  # ...plus every skipped test at file:line
+canary vacuity-check tests/ --json     # verdict + denominator + skips
+canary vacuity-check tests/a.test.ts   # one file
+```
+
+As a skill, for an orchestrator or a CI step composing all four Tier-0
+detectors:
+
+```bash
+canary skills run canary-cassandra -- tests/
+canary skills run canary-cassandra -- tests/ --json --strict
+```
+
+Usage, flags, and the full rule list — this one is runnable verbatim, needs no
+fixtures, and is what CI executes to prove the doc still matches the CLI:
+
+```bash
+canary skills run canary-cassandra -- --help
+```
+
+The `--json` envelope matches `canary-savant` / `canary-blackhawk` /
+`canary-katana` — `schema_version`, a `findings` array of
+`{file, line, rule_id, severity, snippet, why}`, and a `summary` — so findings
+from all four merge without special-casing one. Cassandra adds `suggestion` and
+`fidelity` per finding, and `tests_checked` to the summary, because its
+denominator is tests rather than files.
+
+**Advisory by design.** Findings exit **0**. This is the repo's established
+shape for a new detector — advisory first, ratchet to strict only after triage
+(see the dogfooding jobs, [#485]). Do not wire it as a required check on the day
+you adopt it; run it, triage the count, then decide.
+
+A collapsed denominator is **not** advisory and exits **3**. See below.
+
+## The fidelity ladder (the part that decides whether to trust a finding)
+
+A test's "declared target" is declared nowhere, so `VAC-002`/`VAC-003` have to
+resolve one. Three rungs, and the finding says which one it used:
+
+| Tier              | How the target was resolved                                                                   | Trust                                 |
+| ----------------- | --------------------------------------------------------------------------------------------- | ------------------------------------- |
+| `annotated`       | The author wrote `// @covers <symbol>`. That exact symbol is checked.                         | High — the author stated the contract |
+| `import-inferred` | The symbols imported from first-party (relative) modules, closed over local helpers           | Medium — read the test before acting  |
+| _(skipped)_       | Neither available. Reported as a skip with its reason; the test is **not** reported as clean. | None — the check did not run          |
+
+`import-inferred` reads four binding forms, not one: a named or default import,
+a **namespace** import (`import * as store from './store.js'`), a **dynamic**
+import (`const { save } = await import('./store.js')`), and a handle bound to a
+first-party **script path** that a `spawnSync`/`execFileSync`-family call then
+runs. The last is what a subprocess test has instead of a symbol; the
+discriminator is the path, so spawning a bare command (`spawnSync('git', …)`)
+still counts as reaching nothing. A test that launches a script path written
+inline, or that carries a bare `await import('./x.js')`, is likewise read as
+reaching its target — but `VAC-003` stays dark for it and says so in the skip
+list, because "did an assertion observe the target" needs a symbol that a
+subprocess boundary does not provide (#705).
+
+To upgrade a finding from inferred to annotated, add the annotation above the
+test:
+
+```ts
+// @covers resolveOverlay
+it('falls back to the tracked overlay', () => {
+  /* ... */
+});
+```
+
+That is also the fix for a false positive: if the target is reached several
+frames deep and the inference cannot see it, `@covers` tells the check what to
+look for instead of arguing with the heuristic.
+
+## Why a zero denominator exits 3
+
+The whole point of this skill is that **green can mean nothing**. A vacuity
+detector that reported its own silence as success would be the joke telling
+itself, so:
+
+- **No test file matched** → exit 3, `Abstained — verified zero items`.
+- **Files matched but held zero tests** → also exit 3. This is the subtler zero:
+  the file-level count looks healthy, and a scanner that only guarded that one
+  prints a clean tick.
+- **A test whose target could not be resolved** → a `skipped` entry naming the
+  test and the reason, rendered in every summary line.
+
+`checked` in the JSON payload is the number of **tests** read, not files. Always
+read it before believing a zero.
+
+## Reading the output
+
+```text
+[CRITICAL] tests/cli.test.ts:41 (VAC-001)
+  --help does not write the ledger: Assertion compares a value with itself; no implementation can fail it.
+  → Assert the value the code under test should have produced, not the input.
+
+3 finding(s) across 214 checked (11 skipped: VAC-002/VAC-003 (adds two numbers) [target unresolvable: ...])
+```
+
+The summary line always carries the denominator and the skips. A finding count
+with no denominator is not a result.
+
+## Rationalizations to reject
+
+| Rationalization                                          | Why it is wrong                                                                                                                                   |
+| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
+| "It's covered, so it's tested"                           | A vacuous test executes the line and asserts nothing about it. Coverage is the reason this class survives — it satisfies the metric perfectly.    |
+| "VAC-002 flagged a test I know is correct, so it's junk" | Add `// @covers <symbol>`. That converts a guess into a checked claim, permanently, for every future run.                                         |
+| "0 findings, we're clean"                                | Only if `checked` is greater than zero. Read the denominator; exit 3 means the check never ran.                                                   |
+| "We'll make it a required check right away"              | A new detector lands advisory. Triage the existing count first, then ratchet — otherwise the first red build teaches the team to bypass the gate. |
+| "The absence assertion is fine, the test passes"         | That is the `VAC-003` defect exactly: it also passed when the code exited before doing anything. One positive assertion makes it load-bearing.    |
+
+## Escalation
+
+- **A VAC-001 finding is never a false positive.** It is a comparison of a value
+  with itself. Fix the test.
+- **A large VAC-002 count on first run usually means dynamic imports.** Tests
+  that `await import(...)` their target have no static import for the inference
+  to read. Annotate them, or accept the tier and move on — do not rewrite the
+  suite to satisfy a heuristic.
+- **A finding you disagree with is a doc gap, not an argument.** Write the
+  `@covers` annotation; the next reader gets the answer for free.
+
+## Related
+
+- `canary review-test` — `LINT-006` (zero assertions) and `SOUND-001/002/003` (a
+  test that pins a value no correct implementation must produce). Cassandra
+  deliberately does not duplicate either.
+- `canary-promote-test` — consumes these findings as a structured verdict and
+  blocks promotion on the deterministic ones.
+- `canary-katana`, `canary-savant`, `canary-blackhawk` — the other Tier-0
+  deterministic scanners.
+
+[#485]: https://github.com/bop-clocktower/canary/issues/485
+[#486]: https://github.com/bop-clocktower/canary/issues/486

package/agents/skills/claude-code/canary-cassandra/scripts/cli.mjs

@@ -0,0 +1,270 @@
+#!/usr/bin/env node
+// canary-cassandra -- vacuous-test detection (#755).
+//
+// Finds tests that PASS WITHOUT PROVING ANYTHING: an assertion that compares a
+// value with itself (VAC-001), a test that never invokes the target it claims
+// to cover (VAC-002), and a test whose every assertion is an absence observed
+// on a bystander (VAC-003), and a test whose every assertion is a trivially
+// true presence check on a value the test built itself (VAC-005).
+//
+//   <paths>    files or directories to scan (default: the current directory).
+//   --json     emit machine-readable findings instead of human text.
+//   --strict   exit 1 when there are findings (default is advisory: exit 0).
+//
+// Deterministic: no LLM, no network, no test execution. The rules themselves
+// live in the engine (`core/vacuity-scanner`), which is also what `canary
+// vacuity-check` and the promotion gate run -- see engine.mjs for why this one
+// skill delegates where its three siblings self-host.
+//
+// Invoked via `canary skills run canary-cassandra -- [paths] [--json] [--strict]`.
+
+import fs from 'node:fs';
+
+import {
+  createParser,
+  formatUsageError,
+  EXIT_USAGE,
+} from '../../../lib/parse-args.mjs';
+import { loadEngine } from './engine.mjs';
+
+export const SCHEMA_VERSION = 1;
+
+const PREFIX = 'canary-cassandra:';
+
+/** Reserved CLI-wide: exit 3 means "abstained -- verified zero items". */
+const EXIT_ABSTAINED = 3;
+
+// U+2192 written as an escape so this source stays ASCII, matching the family.
+const ARROW = '\u{2192}';
+
+const USAGE =
+  'usage: canary-cassandra [-h] [--json] [--strict] [--] [path ...]\n' +
+  '\n' +
+  'Vacuous-test detection: finds tests that pass without proving anything.\n' +
+  '\n' +
+  'positional arguments:\n' +
+  '  path        files or directories to scan (default: the current directory)\n' +
+  '\n' +
+  'options:\n' +
+  '  -h, --help  show this help message and exit\n' +
+  '  --json      emit machine-readable findings instead of human text\n' +
+  '  --strict    exit 1 when there are findings (default is advisory: exit 0)\n' +
+  '\n' +
+  'rules:\n' +
+  '  VAC-001 (critical)  assertion compares a value with itself\n' +
+  '  VAC-002 (warning)   the test never invokes the target it covers\n' +
+  '  VAC-003 (warning)   every assertion is an absence, on a bystander\n' +
+  '  VAC-005 (warning)   every assertion is trivial presence, on a bystander\n' +
+  '\n' +
+  'The denominator is TESTS read, not files. A zero denominator exits 3 under\n' +
+  '--strict; it is never reported as a clean scan.';
+
+/**
+ * The `--` terminator and a lone `-` come with declaring positionals, so a file
+ * literally named `--json` stays reachable. Shared with the other skill CLIs
+ * via `lib/parse-args.mjs`; `test/skill-cli-conformance.test.ts` asserts this
+ * export exists so a hand-rolled parser cannot land.
+ */
+export const CLI_SPEC = {
+  prog: 'canary-cassandra',
+  booleans: { '--json': 'json', '--strict': 'strict' },
+  positionals: { key: 'paths', defaults: ['.'] },
+};
+
+const parseArgs = createParser(CLI_SPEC);
+
+// Resolved once, at load, so `main` stays synchronous like every sibling's --
+// the conformance suite calls `main(argv)` and reads a number back. A failed
+// resolution is carried, not thrown: `--help` must answer even with no engine.
+const ENGINE = await loadEngine();
+
+/** Every test file the given paths contribute, de-duplicated and ordered. */
+function collectFiles(paths, engine) {
+  const seen = new Set();
+  const files = [];
+  for (const entry of paths) {
+    const found = engine.isDir(entry)
+      ? engine.collectTestFiles(entry)
+      : [entry];
+    for (const file of found) {
+      if (seen.has(file)) continue;
+      seen.add(file);
+      files.push(file);
+    }
+  }
+  return files;
+}
+
+/** Run the engine scanner over each file, accumulating one gate result. */
+function scanFiles(files, engine) {
+  const findings = [];
+  const skipped = [];
+  let checked = 0;
+  for (const file of files) {
+    const r = engine.scanVacuity(file);
+    checked += r.checked;
+    findings.push(...r.findings);
+    if (r.skipped) skipped.push(...r.skipped);
+  }
+  return { checked, findings, skipped };
+}
+
+/** The sibling finding envelope, plus the two fields only cassandra has. */
+function toJson(f) {
+  return {
+    file: f.file,
+    line: f.line,
+    rule_id: f.rule,
+    severity: f.severity,
+    // `snippet` in the sibling envelope is "the locus, in one line". For a
+    // vacuity finding that is the test, not the source line: the defect is the
+    // test as a whole, and a single line of it reads as a lint hit.
+    snippet: f.test,
+    why: f.message,
+    suggestion: f.suggestion,
+    fidelity: f.fidelity ?? null,
+  };
+}
+
+function summary(result, filesScanned, outcome) {
+  const bySeverity = {};
+  for (const f of result.findings) {
+    bySeverity[f.severity] = (bySeverity[f.severity] || 0) + 1;
+  }
+  return {
+    files_scanned: filesScanned,
+    // The denominator that matters. A healthy `files_scanned` over zero tests
+    // is the subtler zero, and the one a file-counting scanner prints a clean
+    // tick on.
+    tests_checked: result.checked,
+    abstained: outcome.abstained,
+    findings: result.findings.length,
+    by_severity: bySeverity,
+    skipped: result.skipped.length,
+  };
+}
+
+function renderText(result, outcome) {
+  const lines = [];
+  for (const f of result.findings) {
+    const tier = f.fidelity ? ` [${f.fidelity}]` : '';
+    lines.push(
+      `[${f.severity.toUpperCase()}] ${f.file}:${f.line} (${f.rule})${tier}`,
+    );
+    lines.push(`  ${f.test}: ${f.message}`);
+    lines.push(`  ${ARROW} ${f.suggestion}`);
+    lines.push('');
+  }
+  lines.push(outcome.summaryLine);
+  if (outcome.abstained) {
+    lines.push(
+      'No test was read, so nothing here is proven. Point at a directory ' +
+        'holding test files, or pass one directly.',
+    );
+  } else if (!result.findings.length) {
+    lines.push(
+      'Advisory by default. Re-run with --strict to fail on findings.',
+    );
+  }
+  return lines.join('\n');
+}
+
+/** Exit 1 with a named reason. "Cannot verify" is a finding, not a skip. */
+function fail(message) {
+  console.error(`${PREFIX} ${message}`);
+  return 1;
+}
+
+/** Emit the result in the caller's mode. */
+function report(result, files, outcome, json) {
+  if (!json) {
+    console.log(renderText(result, outcome));
+    return;
+  }
+  console.log(
+    JSON.stringify(
+      {
+        schema_version: SCHEMA_VERSION,
+        findings: result.findings.map(toJson),
+        skipped: result.skipped,
+        summary: summary(result, files.length, outcome),
+      },
+      null,
+      2,
+    ),
+  );
+}
+
+/**
+ * Scan the given paths.
+ *
+ * Returns the scan plus its gate outcome, or a numeric exit code when nothing
+ * could be scanned at all -- an unresolvable engine and a missing path are
+ * FAILURES, never a clean result with an empty finding list.
+ */
+function scan(paths) {
+  if (!ENGINE.ok) return fail(ENGINE.error);
+  for (const entry of paths) {
+    if (!fs.existsSync(entry)) return fail(`path not found: ${entry}`);
+  }
+  const files = collectFiles(paths, ENGINE);
+  const result = scanFiles(files, ENGINE);
+  if (files.length === 0) {
+    result.skipped.push({
+      name: paths.join(', '),
+      reason: `no test file matched (looked for ${ENGINE.SCANNABLE_DESC})`,
+    });
+  }
+  // The same helper the engine's own surfaces use, rather than a hand-copied
+  // abstention line: this CLI already imports the engine, so the doctrine is
+  // enforced by the code instead of by convention.
+  const outcome = ENGINE.gateOutcome(
+    {
+      checked: result.checked,
+      findings: result.findings,
+      skipped: result.skipped,
+    },
+    'advisory',
+    { noun: 'test(s)' },
+  );
+  return { files, result, outcome };
+}
+
+export function main(argv = []) {
+  const { positionals: paths, opts, help, error } = parseArgs(argv);
+
+  // Usage resolves before any filesystem or engine work, so `--help` answers
+  // even in an install where the engine never resolved.
+  if (help) {
+    console.log(USAGE);
+    return 0;
+  }
+  if (error) {
+    console.error(formatUsageError(CLI_SPEC.prog, error));
+    return EXIT_USAGE;
+  }
+
+  const scanned = scan(paths);
+  if (typeof scanned === 'number') return scanned;
+  const { files, result, outcome } = scanned;
+  report(result, files, outcome, opts.json);
+
+  // Advisory by default (D3): findings are loud, the exit is not. Under
+  // --strict the exit-code contract applies, and a collapsed denominator takes
+  // EXIT_ABSTAINED (3) -- distinct from 1, "found something real".
+  if (!opts.strict) return 0;
+  if (outcome.abstained) return EXIT_ABSTAINED;
+  return result.findings.length ? 1 : 0;
+}
+
+// Direct execution (the skill runner execs this file via its shebang).
+//
+// `process.exitCode` rather than `process.exit()`: a `--json` payload over a
+// large suite exceeds the pipe buffer, and `process.exit` tears the process
+// down mid-write, truncating it to ~64KB. Truncated JSON that still exits 0 is
+// a machine-readable result a consumer cannot parse but a shell reads as
+// success -- the exact class of quiet failure the whole family guards against.
+// Setting the code lets node drain stdout and exit with the same status.
+if (import.meta.url === `file://${process.argv[1]}`) {
+  process.exitCode = main(process.argv.slice(2));
+}

package/agents/skills/claude-code/canary-cassandra/scripts/engine.mjs

@@ -0,0 +1,95 @@
+// Engine resolution for the cassandra skill CLI (#755).
+//
+// WHY THIS SKILL IMPORTS THE ENGINE AND ITS SIBLINGS DO NOT
+//
+// savant, blackhawk and katana each own their detection outright, so their CLIs
+// are self-contained by construction. Cassandra's detection is NOT its own: it
+// already lives in `ts/src/core/vacuity-scanner.ts` and is already exercised by
+// `canary vacuity-check` and by the promotion gate. That module's docstring
+// states the position deliberately -- #605 accepted that `static_linter` and
+// `quality_scorer` already overlap and a third half-enforcer would be the real
+// defect -- so shipping a second, hand-copied vacuity scanner inside this skill
+// would trade #755's asymmetry for a worse one: two detectors that disagree.
+//
+// So the missing piece was never the detection. It was the `cli:` entry point,
+// and this file is what lets one exist without forking the rules.
+//
+// The engine sits three-plus levels above this script in both layouts canary
+// ships, and the lookup is a fixed, ordered list rather than a search: an
+// ambiguous resolution would make which rules ran depend on the install.
+//
+//   repo checkout : <root>/ts/dist/core/...        (after `npm --prefix ts run build`)
+//   npm package   : <pkg>/dist/engine/core/...     (staged by npm/scripts/build-engine.mjs)
+//
+// A failure to resolve is reported as a failure. It is never absorbed into a
+// clean scan -- "the detector could not load" and "the detector found nothing"
+// must not print the same thing.
+
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath, pathToFileURL } from 'node:url';
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+
+// scripts -> canary-cassandra -> claude-code -> skills -> agents -> <root>
+const ROOT = path.resolve(HERE, '..', '..', '..', '..', '..');
+
+/** Candidate engine directories, highest priority first. */
+export function engineCandidates(root = ROOT, env = process.env) {
+  const candidates = [];
+  // An explicit override exists for overlay installs, where the skill is
+  // deployed away from the engine that owns its rules.
+  if (env.CANARY_ENGINE_DIR)
+    candidates.push(path.resolve(env.CANARY_ENGINE_DIR));
+  candidates.push(path.join(root, 'ts', 'dist'));
+  candidates.push(path.join(root, 'dist', 'engine'));
+  return candidates;
+}
+
+const MODULES = [
+  ['core', 'vacuity-scanner.js'],
+  ['core', 'test-files.js'],
+  ['core', 'gate-result.js'],
+];
+
+/** The first candidate directory holding every module we need, or null. */
+export function resolveEngineDir(candidates = engineCandidates()) {
+  for (const dir of candidates) {
+    if (MODULES.every((parts) => fs.existsSync(path.join(dir, ...parts)))) {
+      return dir;
+    }
+  }
+  return null;
+}
+
+/**
+ * Load the engine halves this CLI delegates to.
+ *
+ * @returns `{ ok: true, ... }` on success, or `{ ok: false, error }` naming
+ *   every directory that was tried -- a zero-denominator report of its own.
+ */
+export async function loadEngine(candidates = engineCandidates()) {
+  const dir = resolveEngineDir(candidates);
+  if (dir === null) {
+    return {
+      ok: false,
+      error:
+        'the vacuity engine could not be located, so nothing was scanned. ' +
+        `Tried: ${candidates.join(', ')}. In a Canary checkout run ` +
+        '`npm --prefix ts run build`; otherwise reinstall canary-test-cli, or ' +
+        'set CANARY_ENGINE_DIR to the directory holding core/.',
+    };
+  }
+  const load = (parts) => import(pathToFileURL(path.join(dir, ...parts)).href);
+  const [vacuity, testFiles, gate] = await Promise.all(MODULES.map(load));
+  return {
+    ok: true,
+    dir,
+    scanVacuity: vacuity.scanVacuity,
+    collectTestFiles: testFiles.collectTestFiles,
+    isDir: testFiles.isDir,
+    SCANNABLE_DESC: testFiles.SCANNABLE_DESC,
+    gateOutcome: gate.gateOutcome,
+    EXIT_ABSTAINED: gate.EXIT_ABSTAINED,
+  };
+}