canary-test-cli 7.1.0 → 8.0.0

package/agents/skills/claude-code/canary-savant/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: canary-savant
+description:
+  Order-dependence and isolation detector for test suites. A deterministic
+  static scan flags the shared-state smells which predict order-dependent tests
+  - a module-level mutable a test writes to, a setup with no matching teardown,
+  a mutated process singleton, an order-coupled name - with no test execution,
+  so it runs anywhere node does and on every PR. An opt-in confirming pass
+  (--confirm) shuffles the suite under a pinned seed and bisects the prefix to
+  name the polluting test. Advisory by default; pytest and vitest idioms.
+cli: scripts/cli.mjs
+requires: [node>=20]
+---
+
+# Canary Savant
+
+A test that only passes because of the tests that ran before it is a lie that
+passes CI. Savant finds shared-state leakage and names the culprit. The **static
+pass** is the cheap half that runs on every PR and points at the _suspects_. The
+**confirming pass** (`--confirm`, opt-in) proves a leak by shuffling the suite
+under a pinned seed and bisecting the prefix to name the polluter — not just the
+victim.
+
+Tier-0 by the definition in
+[ADR 0015](../../../../docs/knowledge/decisions/0015-skill-capability-vocabulary.md):
+deterministic, no network, no agent or LLM. Also no secrets and no dependency on
+any other skill. Both passes qualify — `--confirm` runs the suite locally, which
+needs no network and no model.
+
+## Rules (static pass — suspects)
+
+| Rule                              | Severity | Fires on                                                                                                                                                                                                                                                                                                                                                                                                            |
+| --------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `SV001-module-mutable-global`     | medium   | A module-scope mutable (`= {}`, `= []`, `set()`, `dict()`, `list()`, or a top-level JS `let`/`var`/`const` object/array) that some line later mutates in place (`.append`/`.add`/`[...] =`/`+=`/`.attr =`). Fires on the **declaration**, the leak's source.                                                                                                                                                        |
+| `SV002-missing-teardown`          | medium   | A **class/all-scoped** setup whose matching teardown is absent: pytest `setup_class`/`setUpClass`, or vitest/jest `beforeAll`. Per-test setup (`setUp`/`setup_method`/`beforeEach`) is excluded - it rebuilds state each test, so it does not leak.                                                                                                                                                                 |
+| `SV003-shared-singleton-mutation` | low      | A process-global singleton assigned without restore: `os.environ[...] =`, `sys.modules[...] =`, `process.env.X =`. Reads and `==` comparisons never fire, and neither does a file that demonstrably restores the global — a same-key (or computed-loop) restore in `afterEach`/`afterAll`/teardown/post-`yield` fixture code or an in-test `try`/`finally`, or a write-back from a snapshot saved from that global. |
+| `SV004-order-coupled-name`        | low      | A test name or comment that encodes ordering: `test_1_…`, a **terminal** ordinal (`test_first()`, `test_last()` - not `test_first_match_wins`), `must run before …`, `it('… run first')`.                                                                                                                                                                                                                           |
+
+A finding is a **suspect, not a verdict.** A module dict that is only ever read
+is a legitimate constant and does not fire; only a _mutated_ one does.
+
+## Framework conditioning
+
+The setup/teardown idioms (`SV002`) differ by ecosystem, so the rule is
+conditioned on the file: Python files are read with pytest/unittest markers, JS
+and TS with vitest/jest markers. The idioms do not collide across languages, so
+each file is judged by its own ecosystem's conventions.
+
+## Fidelity limits (AST-lite, on purpose)
+
+The static pass is a scanner with no parser dependency, so it ships anywhere
+`node` does. The cost, stated plainly:
+
+- **`SV001` mutation is file-scoped, not flow-scoped.** Any in-place mutation of
+  a module-level name anywhere in the file indicts the declaration, even if the
+  mutation sits in a helper rather than a test body. A shared-state leak is a
+  shared-state leak regardless of which function does the writing.
+- **`SV002` is presence-based, not pairing-based.** A file with one `setUpClass`
+  and one `tearDownClass` is considered balanced even if a _second_ class lacks
+  teardown. It also only judges class/all-scoped setup, so a genuinely leaky
+  per-test setup (rare) is missed - the deliberate false-positive/false-negative
+  trade from dogfooding.
+- **Comment-blind for code rules.** `SV003` skips commented-out lines, and
+  `SV002` judges both halves of its pair against a code-only projection of the
+  file (comments dropped, string contents blanked) - so neither a prose mention
+  of `afterAll` nor a fixture string containing one can stand in for the real
+  teardown (#732). `SV004` deliberately is _not_ comment-blind, because an
+  ordering note in a comment is exactly the self-reported dependence it looks
+  for.
+- **String-aware for code anchors.** An `SV003` (or `SV004` name-pattern) match
+  that _starts_ inside a string literal on its own line is rejected as fixture
+  data. `SV004`'s directive-text alternatives stay unfiltered on purpose: their
+  signal (test titles, docstrings) legitimately lives inside strings.
+- **Restoration check is file-level.** `SV003` suppression cannot verify that a
+  fixture actually applies to the mutating test, and a computed-key loop restore
+  is assumed to cover the whole family — the same file-wide trade blackhawk
+  makes for frozen clocks. `vi.unstubAllEnvs`/`monkeypatch` never suppress a
+  direct assignment: they only undo their own mutations.
+- **Line-scoped.** A declaration or call split across lines can be missed.
+- **A missed suspect costs less than a false one** — the same bias as
+  canary-blackhawk.
+
+## Inline suppression (per-line escape hatch)
+
+A suite that legitimately contains a suspect line — most commonly a fixture
+string feeding a tool that _tests_ these detections — can suppress a single
+finding with an inline pragma, same dialect as `blackhawk-ignore`:
+
+```py
+# savant-ignore SV004 -- fixture: directive-text input the rule under test must detect
+```
+
+- **Reason required** (the `-- reason` tail) — keeps suppressions honest and
+  greppable, like `eslint-disable-next-line` / `# noqa`. A bare
+  `savant-ignore SV004` suppresses nothing.
+- **Rule-scoped** (`SV004`, or the full `SV004-order-coupled-name`) so it never
+  blanket-silences the line; another rule firing on the same line still fires.
+- **Placement:** the pragma may trail the offending line or sit on the line
+  directly above it. Comma-separate ids to suppress several (`SV003,SV004`).
+- **Counted separately:** suppressed findings are reported as
+  `N suppressed (inline savant-ignore)` and in the JSON `summary.suppressed`,
+  out of the actionable total — so a genuinely clean suite can read zero
+  findings while the known-OK fixtures stay visible.
+- **String-guarded:** pragma text that itself sits _inside_ a string literal is
+  fixture data, not a directive — it neither suppresses its own line nor the
+  next one.
+
+## Which files get scanned
+
+A directory walk only visits **test** files — `*.test.*`, `*.spec.*`,
+`test_*.py`, `*_test.py`, or any supported source under `tests/`, `test/`,
+`__tests__/`, `e2e/`, `spec/`. A file named explicitly on the command line is
+always scanned. Supported suffixes: `.py`, `.js`, `.jsx`, `.ts`, `.tsx`, `.mjs`,
+`.cjs`. Dependency directories (`node_modules`, `.venv`, …) are never walked.
+
+## Invocation
+
+```bash
+# Scan the repo's test files (advisory - always exits 0):
+canary skills run canary-savant
+
+# Scan a specific suite:
+canary skills run canary-savant -- tests/unit
+
+# Machine-readable findings:
+canary skills run canary-savant -- tests --json
+
+# Fail the step on any suspect:
+canary skills run canary-savant -- tests --strict
+
+# Usage, options, and the full rule list (exits 0):
+canary skills run canary-savant -- --help
+```
+
+An unknown flag is rejected with `unrecognized arguments: <flag>` and exit 2.
+`--seed` is validated as a determinism flag rather than being allowed to decay
+into a random seed. All three failures exit 2:
+
+| Input                              | Error                                                        |
+| ---------------------------------- | ------------------------------------------------------------ |
+| missing value, or a following flag | `argument --seed: expected one argument`                     |
+| `abc`, `3.7`, `1e999`, `0x10`, ``  | `argument --seed: invalid int value: '<value>'`              |
+| beyond ±(2^53 − 1)                 | `argument --seed: seed out of safe integer range: '<value>'` |
+
+The last one matters because `Number()` rounds silently above 2^53 − 1, so the
+seed actually used would differ from the seed asked for — the same lie the whole
+flag exists to prevent.
+
+`--seed 42` and `--seed=42` are equivalent, and both accept negative seeds
+(`--seed -5`, `--seed=-5`). Use `--` to end option parsing when a path itself
+starts with a dash.
+
+### The confirming pass — dynamic confirmation (`--confirm`, opt-in)
+
+`--confirm` runs the suite in declared order, re-runs it shuffled under a pinned
+seed, and for each order-dependent victim runs it alone and **bisects the prefix
+to name the polluter** — the earlier test whose state leaked. Opt-in because a
+shuffled re-run at least doubles wall-clock; always prints the seed and a
+copy-pasteable reproduce command.
+
+```bash
+# Confirm order-dependence, pinning the seed for reproducibility:
+canary skills run canary-savant -- tests --confirm --seed 424242
+```
+
+**pytest and vitest** are both supported; savant auto-detects from the target
+(extensions, then a directory scan, then config files). pytest needs a shuffle
+plugin (`pytest-randomly` or `pytest-random-order`) and declines loudly if none
+is installed; vitest's shuffle is built in (`--sequence.shuffle`), so no plugin
+is required. Node drives the project's _own_ runner — savant orchestrates it, it
+does not run in the target's language.
+
+**Polluter bisect is pytest-only.** vitest has no CLI-driven ordered per-test
+execution, so a vitest target gets victim _detection_ (which tests break under
+shuffle) but not culprit _naming_. For pytest, node ids are captured via
+`--collect-only`, so class-based layouts (`file.py::Class::test`) re-run
+correctly.
+
+`--json` shape:
+
+```json
+{
+  "schema_version": 1,
+  "findings": [
+    {
+      "file": "tests/test_cache.py",
+      "line": 3,
+      "rule_id": "SV001-module-mutable-global",
+      "severity": "medium",
+      "snippet": "_CACHE = {}",
+      "why": "a module-level mutable is written by a test, so state leaks into whatever test runs next"
+    }
+  ],
+  "summary": {
+    "files_scanned": 8,
+    "findings": 1,
+    "by_severity": { "medium": 1 },
+    "suppressed": 0
+  }
+}
+```
+
+## Fixing what it finds
+
+| Finding | Fix                                                                                                                                             |
+| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| `SV001` | Move the mutable into a fixture that rebuilds it per test, or reset it in teardown. Module-level mutable state shared across tests is the leak. |
+| `SV002` | Add the matching teardown (`teardown_method`, `afterEach`, a `yield` fixture) so acquired state is released.                                    |
+| `SV003` | Use a restoring helper — pytest `monkeypatch.setenv`, or save/restore around the test — instead of assigning the global directly.               |
+| `SV004` | Make the test self-contained so order stops mattering, then drop the ordering hint from the name/comment.                                       |
+
+## Dogfooding and the `--strict` promotion path
+
+canary runs savant's static pass over its **own** test suite on every PR
+(`.github/workflows/harness-quality.yml`, the `Skills (JS)` job), **advisory**:
+it prints suspects to the log and always exits 0. Tuning the rules against that
+real suite dropped the backlog from 37 findings to a handful of genuine
+suspects; the structural residue (savant's own suite testing SV004's
+directive-text detection, #496) is suppressed with per-site `savant-ignore`
+reasons and stays visible in the suppressed count. Promote to blocking by
+appending `--strict` to that step once the remaining suspects are triaged (fixed
+or confirmed benign) - the same advisory-first path every canary gate takes.
+
+## Roadmap
+
+- **Shipped:** the static pass; the dynamic confirming pass (baseline → shuffle
+  → classify) with isolation + polluter bisect (pytest); vitest as a
+  confirming-pass classify target; pytest node-id capture for class-based
+  layouts; advisory CI gate dogfooded on canary's own suite (rules tuned to kill
+  the dominant false positives).
+- **Remaining:** flip the advisory gate to `--strict` once the suspect backlog
+  is triaged; vitest polluter naming is out of scope until vitest gains ordered
+  per-test execution. See `docs/changes/canary-savant/proposal.md`.

package/agents/skills/claude-code/canary-savant/scripts/cli.mjs

@@ -0,0 +1,274 @@
+#!/usr/bin/env node
+// canary-savant -- order-dependence and isolation detector (Tier-1 static scan).
+//
+// Phase 1 ships the always-on static "suspect" tier: an AST-lite scan that
+// flags the shared-state smells that predict order-dependent tests -- module-
+// level mutables written by tests, setup without teardown, mutated process
+// singletons, order-coupled names -- with no test execution. The opt-in
+// dynamic confirmer (--confirm) lands in a later phase.
+//
+//   <paths>    files or directories to scan (default: the current directory).
+//   --json     emit machine-readable findings instead of human text.
+//   --strict   exit 1 when there are findings (default is advisory: exit 0).
+//
+// Tier-0 in the real sense -- no LLM, no network, no secrets, no dependency on
+// any other skill.
+//
+// Invoked via `canary skills run canary-savant -- [paths] [--json] [--strict]`.
+
+import fs from 'node:fs';
+import { scanPaths, toJson } from './scanner.mjs';
+import { confirm, locatePolluters, realPolluterSeams } from './runner.mjs';
+import { RULES } from './rules.mjs';
+import {
+  createParser,
+  formatUsageError,
+  EXIT_USAGE,
+} from '../../../lib/parse-args.mjs';
+
+export const SCHEMA_VERSION = 1;
+
+// --- no-silent-abstention (#508 D2, skill-CLI convention half) ---------------
+//
+// Skill CLIs are deliberately self-contained -- no engine import, no shared
+// module -- so they cannot call `gateOutcome`. They honour the doctrine by
+// CONVENTION instead, emitting the same greppable line the engine helper does.
+// The skill-layer conformance registry (agents/skills/test/gate-conformance.
+// test.ts) is what holds them to it: a row whose fixture collapses the
+// denominator and asserts the loud outcome.
+//
+// U+26A0 / U+2014 are written as escapes so this source stays ASCII, matching
+// ts/src/core/gate-result.ts.
+const ABSTAINED_LINE =
+  '\u{26A0} Abstained \u{2014} verified zero items; this is not a pass.';
+
+const PREFIX = 'canary-savant:';
+
+// The rules block is GENERATED from RULES, never hand-typed, so a new rule
+// appears in --help the moment it is registered.
+const USAGE =
+  'usage: canary-savant [-h] [--json] [--strict] [--confirm] [--seed N] [--]\n' +
+  '                     [path ...]\n' +
+  '\n' +
+  'Order-dependence and isolation detector: flags the shared-state smells that\n' +
+  'predict order-dependent tests, and optionally confirms them dynamically.\n' +
+  '\n' +
+  'positional arguments:\n' +
+  '  path        files or directories to scan (default: the current directory)\n' +
+  '\n' +
+  'options:\n' +
+  '  -h, --help  show this help message and exit\n' +
+  '  --json      emit machine-readable findings instead of human text\n' +
+  '  --strict    exit 1 when there are findings (default is advisory: exit 0)\n' +
+  '  --confirm   run the opt-in Tier-2 dynamic confirmer (executes the suite)\n' +
+  '  --seed N    shuffle seed for --confirm (default: random)\n' +
+  '\n' +
+  'rules:\n' +
+  RULES.map((r) => `  ${r.ruleId} (${r.severity})`).join('\n');
+
+function summary(result) {
+  const bySeverity = {};
+  for (const f of result.findings) {
+    bySeverity[f.severity] = (bySeverity[f.severity] || 0) + 1;
+  }
+  return {
+    files_scanned: result.filesScanned,
+    findings: result.findings.length,
+    by_severity: bySeverity,
+    suppressed: result.suppressed ?? 0,
+  };
+}
+
+// A trailing "N suppressed" note keeps inline-ignored lines visible but out of
+// the actionable total - the pattern canary-blackhawk (#393) and the
+// PR-guardian sticky comment use.
+function suppressedNote(result) {
+  const n = result.suppressed ?? 0;
+  return n ? `\n${n} suppressed (inline savant-ignore).` : '';
+}
+
+function renderText(result) {
+  const count = result.findings.length;
+  const files = result.filesScanned;
+  const fp = files === 1 ? '' : 's';
+  // #508: zero suspects over zero scanned files is an ABSENT result, not a
+  // clean one. Findings outrank abstention, so this is the no-findings path.
+  if (!count && !files) {
+    return (
+      `${ABSTAINED_LINE} No file matched the given paths, so there is ` +
+      'nothing to report. Point at a directory that holds test files, or ' +
+      'pass a file directly.'
+    );
+  }
+  if (!count) {
+    return (
+      `No order-dependence suspects (${files} file${fp} scanned).` +
+      suppressedNote(result)
+    );
+  }
+  const sp = count === 1 ? '' : 's';
+  const lines = [
+    `${count} order-dependence suspect${sp} in ${files} file${fp}:`,
+    '',
+  ];
+  for (const f of result.findings) {
+    lines.push(`  ${f.file}:${f.line}  [${f.severity}] ${f.ruleId}`);
+    lines.push(`      ${f.snippet}`);
+    lines.push(`      why: ${f.why}`);
+  }
+  lines.push('');
+  lines.push(
+    'Advisory by default. Re-run with --strict to fail the step on findings.',
+  );
+  return lines.join('\n') + suppressedNote(result);
+}
+
+/**
+ * savant takes paths, so it gets the `--` end-of-options terminator and treats
+ * a lone `-` as a positional, as argparse does.
+ *
+ * `--seed` is the flag that made the shared int type worth having: a bad value
+ * used to decay to `Math.floor(Math.random() * 1e6)` at exit 0, silently
+ * randomizing the one flag whose entire purpose is reproducibility. The shared
+ * parser rejects a non-integer AND an integer past the safe range (#479).
+ */
+export const CLI_SPEC = {
+  prog: 'canary-savant',
+  booleans: {
+    '--json': 'json',
+    '--strict': 'strict',
+    '--confirm': 'confirm',
+  },
+  values: { '--seed': { key: 'seed', type: 'int' } },
+  positionals: { key: 'paths', defaults: ['.'] },
+};
+
+const parseArgs = createParser(CLI_SPEC);
+
+export function renderConfirm(dyn) {
+  if (
+    dyn.status === 'no_plugin' ||
+    dyn.status === 'baseline_red' ||
+    dyn.status === 'unknown_framework'
+  ) {
+    return `\nTier 2 (dynamic): skipped - ${dyn.message}`;
+  }
+  const lines = [`\nTier 2 (dynamic): seed ${dyn.seed}`];
+  if (!dyn.victims.length) {
+    lines.push('  No order-dependence confirmed under this seed.');
+  } else {
+    let namedAny = false;
+    for (const v of dyn.victims) {
+      lines.push(`  order-dependent: ${v.victim}`);
+      if (v.polluter) {
+        lines.push(`    polluted by: ${v.polluter}`);
+        if (v.reproduce) lines.push(`    reproduce: ${v.reproduce}`);
+        namedAny = true;
+      } else if (v.note) {
+        lines.push(`    ${v.note}`);
+      } else if (v.exhausted) {
+        lines.push(
+          `    no single culprit isolated; smallest reproducing prefix: ` +
+            `${v.minimalPrefix?.length ?? '?'} test(s)`,
+        );
+      }
+    }
+    if (!namedAny && dyn.reproduce) {
+      lines.push(`  reproduce: ${dyn.reproduce}`);
+    }
+    if (dyn.framework === 'vitest') {
+      lines.push(
+        '  (vitest: victims detected; polluter bisect is pytest-only)',
+      );
+    }
+  }
+  if (dyn.nondeterministic.length) {
+    lines.push(
+      `  (${dyn.nondeterministic.length} nondeterministic flake(s) - not order, handed off)`,
+    );
+  }
+  return lines.join('\n');
+}
+
+export function main(argv = []) {
+  const { positionals: paths, opts, help, error } = parseArgs(argv);
+
+  // Usage and parse errors resolve before any filesystem work, so `--help`
+  // never reports a missing path and a typo never half-runs a scan.
+  if (help) {
+    console.log(USAGE);
+    return 0;
+  }
+  if (error) {
+    console.error(formatUsageError(CLI_SPEC.prog, error));
+    return EXIT_USAGE;
+  }
+
+  for (const entry of paths) {
+    if (!fs.existsSync(entry)) {
+      console.error(`${PREFIX} path not found: ${entry}`);
+      return 1;
+    }
+  }
+
+  const result = scanPaths(paths);
+
+  let dyn;
+  if (opts.confirm) {
+    const seed = Number.isFinite(opts.seed)
+      ? opts.seed
+      : Math.floor(Math.random() * 1e6);
+    dyn = confirm(paths, { seed });
+    // Phase 3: name the polluter behind each confirmed victim. pytest only -
+    // vitest has no CLI-driven ordered per-test execution, so it gets victim
+    // detection (Phase 2) but not polluter bisection.
+    if (
+      dyn.status === 'ok' &&
+      dyn.victims.length &&
+      dyn.framework === 'pytest'
+    ) {
+      dyn.victims = locatePolluters(
+        dyn.victims,
+        dyn.order,
+        realPolluterSeams(paths),
+      );
+    }
+  }
+
+  if (opts.json) {
+    const payload = {
+      schema_version: SCHEMA_VERSION,
+      findings: result.findings.map(toJson),
+      summary: summary(result),
+    };
+    if (dyn) {
+      payload.dynamic = {
+        status: dyn.status,
+        seed: dyn.seed,
+        victims: dyn.victims,
+        nondeterministic: dyn.nondeterministic,
+        reproduce: dyn.reproduce,
+        ...(dyn.message ? { message: dyn.message } : {}),
+      };
+    }
+    console.log(JSON.stringify(payload, null, 2));
+  } else {
+    console.log(renderText(result) + (dyn ? renderConfirm(dyn) : ''));
+  }
+
+  const hasViolation =
+    result.findings.length > 0 || (dyn?.victims.length ?? 0) > 0;
+  // Advisory by default (D3); --strict inherits EXIT_ABSTAINED (3) on a
+  // collapsed denominator, distinct from 1 ("found something real").
+  if (opts.strict && !result.filesScanned) return 3;
+  return opts.strict && hasViolation ? 1 : 0;
+}
+
+// Direct execution (the skill runner execs this file via its shebang).
+//
+// `process.exitCode`, not `process.exit()`: a large `--json` payload exceeds
+// the pipe buffer, and `process.exit` tears the process down mid-write, leaving
+// truncated JSON that still exits 0 (#791).
+if (import.meta.url === `file://${process.argv[1]}`) {
+  process.exitCode = main(process.argv.slice(2));
+}