canary-test-cli 7.0.0 → 7.2.0

package/agents/skills/claude-code/canary-promote-test/SKILL.md

@@ -0,0 +1,228 @@
+---
+name: canary-promote-test
+description: >
+  Move a generated test from `tests/generated/` into the committed test suite —
+  reviews it for correctness, drops the generation header, relocates it to the
+  matching suite directory, and confirms it runs in the project's normal test
+  flow. Use for "promote this test", "commit this generated test", "move this
+  test into the suite", or "keep this test" — always after the generated test
+  has been validated against the SUT, never before. Not for tests needing
+  substantial rewriting (regenerate instead) or throwaway investigation tests
+  (leave in `tests/generated/`).
+---
+
+# Canary: Promote Test
+
+> Move a generated test from `tests/generated/` into the committed test suite.
+> Reviews the test for correctness, drops the generation header, relocates it
+> under the appropriate suite directory, and confirms it runs in the project's
+> normal test flow.
+
+## When to Use
+
+- After `canary-generate-test` produced a file that has been validated against
+  the SUT
+- When a user explicitly asks to "commit", "save", or "keep" a generated test
+- When extending an existing suite with a new case the team has agreed to
+  maintain
+- NOT before the generated test has been executed and reviewed — promotion is
+  the _last_ step, not the first
+- NOT for tests that need substantial rewriting — regenerate with a better
+  prompt instead of hand-patching
+- NOT for tests targeting throwaway investigations (perf spikes, ad-hoc bug
+  triage) — leave those in `tests/generated/`
+
+## Process
+
+### Phase 0: GATE — Get the Structured Verdict First
+
+Run this before reading the test. It is deterministic, takes no API key, and it
+will refuse the drafts that are not worth your review time.
+
+```bash
+canary promote-check tests/generated/api/orders_post.py --json
+```
+
+| Exit | Verdict   | What it means                                                |
+| ---- | --------- | ------------------------------------------------------------ |
+| `0`  | `promote` | No gating defect. Advisory findings may remain — your call.  |
+| `1`  | `block`   | A gating defect. **Do not promote.** Regenerate or fix.      |
+| `3`  | `abstain` | No verdict could be produced. Promotion is **not** approved. |
+
+**Which axes gate, and why only those.** Gating on all eight axes of a quality
+critique would block nearly every promotion, so only deterministic defects do:
+
+| Axis              | Rules                          | Gates | Reason                                                 |
+| ----------------- | ------------------------------ | ----- | ------------------------------------------------------ |
+| `soundness`       | `SOUND-001/002/003`            | yes   | Pins a value no correct implementation must produce    |
+| `assertions`      | `LINT-006`                     | yes   | A test that asserts nothing always passes              |
+| `flakiness`       | `FLAKE-001/002`                | yes   | A hardcoded sleep in a committed suite is a future red |
+| `vacuity`         | `VAC-001/003`, annotated `002` | yes   | Cannot fail, or contradicts a declared `@covers`       |
+| `selectors`       | `LINT-001/002/003`             | no    | Brittle, not wrong — a reviewer's call                 |
+| `maintainability` | `LINT-005`, `FLAKE-003/004`    | no    | Style and softer signals                               |
+
+**`VAC-002` gates only at `annotated` fidelity.** At `import-inferred` it is an
+inference about which symbol the test meant to exercise, and a heuristic must
+not be load-bearing on a promotion gate. If you want the gate to check the real
+target, add the annotation to the generated test:
+
+```ts
+// @covers resolveOverlay
+```
+
+**An `abstain` is not a pass.** Exit 3 means the checker had no subject — an
+unparseable extension, or a file with no test declarations. Promotion falls back
+to the manual review below and is **not** approved by silence. Do not read a
+missing verdict as either stricter or looser than one.
+
+**An LLM judgement never gates.** `harness:test-craft` runs an 8-axis per-test
+critique and remains exactly what step 5 below calls it: an optional deeper
+audit for a human. Everything that blocks in this repo is deterministic, and
+`promote-check` keeps it that way — the verdict has no field an LLM opinion
+could arrive in.
+
+### Phase 1: REVIEW — Confirm the Test Is Worth Keeping
+
+1. **Read the test end-to-end.** Treat it like any other code review — naming,
+   assertions, hardcoded values, missing edge cases. Generated tests are drafts,
+   not finished artifacts.
+2. **Run the test against the real SUT** (not just the env it was generated
+   against). A test that only passes in one environment is a fixture-bound test,
+   not a regression test.
+3. **Check assertion strength.** A test that only asserts "status code 200" is
+   weak; promote it only if that's genuinely the contract. If the SUT returns
+   structured data, the test should assert on shape.
+4. **Confirm no hardcoded secrets or environment-specific URLs.** Replace with
+   fixtures or env-driven config before promoting.
+5. **Optional: run `harness:test-craft` for a deeper quality audit.**
+   `test-craft` runs an 8-axis per-test LLM critique (assertion density,
+   flakiness risk, contract vs implementation, etc.). Use it when the generated
+   test is substantial or when the team wants a second opinion before
+   committing. Not required for simple happy-path tests, and **never a blocker**
+   — see Phase 0.
+6. **Triage the advisory findings** `promote-check` reported. They did not
+   block; deciding whether they matter here is the review's job.
+7. **Decide: promote, regenerate, or discard.** If review reveals more than ~3
+   small fixes, regenerate with a sharper prompt instead.
+
+### Phase 2: RELOCATE — Move into the Suite
+
+1. **Identify the destination directory.** Mirror the suite's structure:
+   - `tests/generated/api/foo.py` → `tests/api/foo.py`
+   - `tests/generated/e2e/checkout.spec.ts` → `tests/e2e/checkout.spec.ts`
+   - `tests/generated/unit/validator.py` → `tests/unit/validator.py`
+2. **Match suite conventions.** Look at neighboring files for:
+   - Import style (relative vs absolute)
+   - Fixture/setup imports (most suites have a `conftest.py` or shared setup)
+   - Naming conventions (`test_<feature>_<case>.py`, `<feature>.spec.ts`)
+3. **Move with `git mv`** so the history is preserved if anyone later runs
+   `git log --follow`.
+4. **Update imports** if the file referenced anything by relative path from
+   `tests/generated/`.
+
+### Phase 3: CLEAN — Drop Generation Artifacts
+
+1. **Remove the timestamped generation header.** The "Generated by Canary on
+   [date]" comment is useful in scratch space — meaningless in a committed test
+   and rots immediately.
+2. **Remove any placeholder TODOs.** The generator sometimes emits
+   `# TODO: adjust selector` or similar — either resolve them or stop the
+   promotion and regenerate.
+3. **Tighten formatting.** Run the project's formatter (`black`, `prettier`,
+   etc.) so the file matches surrounding style.
+4. **Strip dead code.** Imports the generator added "just in case" but the test
+   doesn't use.
+
+### Phase 4: VERIFY — Run in the Project's Normal Flow
+
+1. **Run the suite that owns this test:**
+   - Python: `pytest tests/api/test_orders_post.py`
+   - Playwright: `npx playwright test tests/e2e/checkout.spec.ts`
+   - Match whatever CI runs.
+2. **Run the full suite** to confirm no collateral failure (shared fixtures,
+   port conflicts, ordering issues).
+3. **Confirm CI configuration picks it up.** If the suite has a glob in CI
+   config, verify the new path matches. If not, add it.
+4. **Log the promotion.** Append a one-line entry to `docs/CANARY_STATE.md` so
+   the project ledger tracks which generated tests have been promoted.
+
+## Canary Integration
+
+- **`tests/generated/`** — Source for promotion. Gitignored; nothing here is
+  ever a final artifact.
+- **`tests/<suite>/`** — Destination. Each suite has its own conventions; never
+  invent a new top-level dir during promotion.
+- **`docs/CANARY_STATE.md`** — Append a promotion entry: requirement, generated
+  path, promoted path, date.
+
+## Success Criteria
+
+- The promoted test passes when run via the project's normal test command
+- The full suite passes (no collateral breakage)
+- The file matches the surrounding code style (formatter clean, lint clean)
+- No generation artifacts remain (timestamp header, placeholder TODOs, unused
+  imports)
+- CI picks up the new test on the next push
+
+## Rationalizations to Reject
+
+| Rationalization                                                           | Why It Is Wrong                                                                                                                                                             |
+| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| "`promote-check` exited 3, so nothing was wrong"                          | Exit 3 is an abstention — the checker had no subject at all. It is the one outcome that proves nothing, so it can never stand in for approval.                              |
+| "`promote-check` blocked on VAC-002, but the test is fine"                | Only an `annotated` VAC-002 blocks, which means the file's own `@covers` names a symbol the test never touches. Either the annotation is wrong or the test is.              |
+| "I'll wire `harness:test-craft` into the gate for better coverage"        | An 8-axis LLM critique gating a promotion would block nearly everything and would make a judgement load-bearing. Deterministic verdicts gate; critiques inform.             |
+| "The test works, I'll skip the review and just commit it"                 | Generated code looks plausible but commonly has weak assertions or hardcoded values. Review is the whole point — promotion without review just imports debt into the suite. |
+| "I'll fix the 6 issues I found in the review by editing the test"         | Six issues means the prompt was wrong. Regenerate; hand-edits won't transfer to the next similar test.                                                                      |
+| "I'll leave the timestamp header so we know when it was generated"        | Git history already records when the file was committed. The header rots and creates noise.                                                                                 |
+| "It passes against staging, that's good enough"                           | If the test only passes in one environment, it's a fixture-bound smoke check, not a regression test. Either parametrize the env or don't promote it.                        |
+| "I'll commit it without running the full suite — I only changed one test" | New tests can break shared fixtures, conflict on ports, leak state. Always run the full suite once before commit.                                                           |
+
+## Examples
+
+### Example: Clean promotion
+
+**Source:** `tests/generated/api/orders_post_201.py`, validated, single status
+assertion is exactly the contract.
+
+**Action:**
+
+1. `git mv tests/generated/api/orders_post_201.py tests/api/test_orders_post_201.py`
+2. Drop the `# Generated by Canary...` header.
+3. Adjust import to match `tests/api/conftest.py` fixtures.
+4. Run `pytest tests/api/` → passes.
+5. Append to `CANARY_STATE.md`: promoted `orders_post_201.py` →
+   `tests/api/test_orders_post_201.py` on [date].
+
+### Example: Promotion abandoned — regenerate instead
+
+**Source:** `tests/generated/e2e/checkout.spec.ts`. Review finds: hardcoded test
+user creds, no wait for navigation, weak assertion (`expect(true).toBe(true)`),
+TODO comments left in three places.
+
+**Action:** Stop. Regenerate with a sharper prompt that names the real fixture
+user, specifies the wait condition, and asserts the actual checkout success
+state. Do not commit the broken draft.
+
+### Example: Test for throwaway investigation
+
+**Source:** `tests/generated/performance/spike_search_50rps.js`. Used once to
+confirm a single perf hypothesis. No ongoing value.
+
+**Action:** Do not promote. Leave in `tests/generated/`. If the team wants
+ongoing perf monitoring, generate a _new_ test with a sustainable RPS profile
+and promote that.
+
+## Escalation
+
+- **When the review reveals the SUT itself is broken:** Don't promote the test
+  that passes against the broken SUT. File a bug; only promote the test once the
+  SUT is fixed and the assertion has a real contract behind it.
+- **When the suite has no existing convention to match:** This usually means the
+  test belongs in a new sub-suite. Ask the user to confirm the suite structure
+  before inventing one.
+- **When CI doesn't pick up the new path:** Don't merge until CI config matches.
+  A test that exists but isn't run is worse than no test — it implies coverage
+  that doesn't exist.
+- **When the promoted test starts flaking after merge:** Treat as a real
+  regression in the test (or the SUT). Don't quarantine in `tests/generated/` —
+  fix or delete.

package/agents/skills/claude-code/canary-savant/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: canary-savant
+description:
+  Order-dependence and isolation detector for test suites. A deterministic
+  static scan flags the shared-state smells which predict order-dependent tests
+  - a module-level mutable a test writes to, a setup with no matching teardown,
+  a mutated process singleton, an order-coupled name - with no test execution,
+  so it runs anywhere node does and on every PR. An opt-in confirming pass
+  (--confirm) shuffles the suite under a pinned seed and bisects the prefix to
+  name the polluting test. Advisory by default; pytest and vitest idioms.
+cli: scripts/cli.mjs
+requires: [node>=20]
+---
+
+# Canary Savant
+
+A test that only passes because of the tests that ran before it is a lie that
+passes CI. Savant finds shared-state leakage and names the culprit. The **static
+pass** is the cheap half that runs on every PR and points at the _suspects_. The
+**confirming pass** (`--confirm`, opt-in) proves a leak by shuffling the suite
+under a pinned seed and bisecting the prefix to name the polluter — not just the
+victim.
+
+Tier-0 by the definition in
+[ADR 0015](../../../../docs/knowledge/decisions/0015-skill-capability-vocabulary.md):
+deterministic, no network, no agent or LLM. Also no secrets and no dependency on
+any other skill. Both passes qualify — `--confirm` runs the suite locally, which
+needs no network and no model.
+
+## Rules (static pass — suspects)
+
+| Rule                              | Severity | Fires on                                                                                                                                                                                                                                                                                                                                                                                                            |
+| --------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `SV001-module-mutable-global`     | medium   | A module-scope mutable (`= {}`, `= []`, `set()`, `dict()`, `list()`, or a top-level JS `let`/`var`/`const` object/array) that some line later mutates in place (`.append`/`.add`/`[...] =`/`+=`/`.attr =`). Fires on the **declaration**, the leak's source.                                                                                                                                                        |
+| `SV002-missing-teardown`          | medium   | A **class/all-scoped** setup whose matching teardown is absent: pytest `setup_class`/`setUpClass`, or vitest/jest `beforeAll`. Per-test setup (`setUp`/`setup_method`/`beforeEach`) is excluded - it rebuilds state each test, so it does not leak.                                                                                                                                                                 |
+| `SV003-shared-singleton-mutation` | low      | A process-global singleton assigned without restore: `os.environ[...] =`, `sys.modules[...] =`, `process.env.X =`. Reads and `==` comparisons never fire, and neither does a file that demonstrably restores the global — a same-key (or computed-loop) restore in `afterEach`/`afterAll`/teardown/post-`yield` fixture code or an in-test `try`/`finally`, or a write-back from a snapshot saved from that global. |
+| `SV004-order-coupled-name`        | low      | A test name or comment that encodes ordering: `test_1_…`, a **terminal** ordinal (`test_first()`, `test_last()` - not `test_first_match_wins`), `must run before …`, `it('… run first')`.                                                                                                                                                                                                                           |
+
+A finding is a **suspect, not a verdict.** A module dict that is only ever read
+is a legitimate constant and does not fire; only a _mutated_ one does.
+
+## Framework conditioning
+
+The setup/teardown idioms (`SV002`) differ by ecosystem, so the rule is
+conditioned on the file: Python files are read with pytest/unittest markers, JS
+and TS with vitest/jest markers. The idioms do not collide across languages, so
+each file is judged by its own ecosystem's conventions.
+
+## Fidelity limits (AST-lite, on purpose)
+
+The static pass is a scanner with no parser dependency, so it ships anywhere
+`node` does. The cost, stated plainly:
+
+- **`SV001` mutation is file-scoped, not flow-scoped.** Any in-place mutation of
+  a module-level name anywhere in the file indicts the declaration, even if the
+  mutation sits in a helper rather than a test body. A shared-state leak is a
+  shared-state leak regardless of which function does the writing.
+- **`SV002` is presence-based, not pairing-based.** A file with one `setUpClass`
+  and one `tearDownClass` is considered balanced even if a _second_ class lacks
+  teardown. It also only judges class/all-scoped setup, so a genuinely leaky
+  per-test setup (rare) is missed - the deliberate false-positive/false-negative
+  trade from dogfooding.
+- **Comment-blind for code rules.** `SV003` skips commented-out lines, and
+  `SV002` judges both halves of its pair against a code-only projection of the
+  file (comments dropped, string contents blanked) - so neither a prose mention
+  of `afterAll` nor a fixture string containing one can stand in for the real
+  teardown (#732). `SV004` deliberately is _not_ comment-blind, because an
+  ordering note in a comment is exactly the self-reported dependence it looks
+  for.
+- **String-aware for code anchors.** An `SV003` (or `SV004` name-pattern) match
+  that _starts_ inside a string literal on its own line is rejected as fixture
+  data. `SV004`'s directive-text alternatives stay unfiltered on purpose: their
+  signal (test titles, docstrings) legitimately lives inside strings.
+- **Restoration check is file-level.** `SV003` suppression cannot verify that a
+  fixture actually applies to the mutating test, and a computed-key loop restore
+  is assumed to cover the whole family — the same file-wide trade blackhawk
+  makes for frozen clocks. `vi.unstubAllEnvs`/`monkeypatch` never suppress a
+  direct assignment: they only undo their own mutations.
+- **Line-scoped.** A declaration or call split across lines can be missed.
+- **A missed suspect costs less than a false one** — the same bias as
+  canary-blackhawk.
+
+## Inline suppression (per-line escape hatch)
+
+A suite that legitimately contains a suspect line — most commonly a fixture
+string feeding a tool that _tests_ these detections — can suppress a single
+finding with an inline pragma, same dialect as `blackhawk-ignore`:
+
+```py
+# savant-ignore SV004 -- fixture: directive-text input the rule under test must detect
+```
+
+- **Reason required** (the `-- reason` tail) — keeps suppressions honest and
+  greppable, like `eslint-disable-next-line` / `# noqa`. A bare
+  `savant-ignore SV004` suppresses nothing.
+- **Rule-scoped** (`SV004`, or the full `SV004-order-coupled-name`) so it never
+  blanket-silences the line; another rule firing on the same line still fires.
+- **Placement:** the pragma may trail the offending line or sit on the line
+  directly above it. Comma-separate ids to suppress several (`SV003,SV004`).
+- **Counted separately:** suppressed findings are reported as
+  `N suppressed (inline savant-ignore)` and in the JSON `summary.suppressed`,
+  out of the actionable total — so a genuinely clean suite can read zero
+  findings while the known-OK fixtures stay visible.
+- **String-guarded:** pragma text that itself sits _inside_ a string literal is
+  fixture data, not a directive — it neither suppresses its own line nor the
+  next one.
+
+## Which files get scanned
+
+A directory walk only visits **test** files — `*.test.*`, `*.spec.*`,
+`test_*.py`, `*_test.py`, or any supported source under `tests/`, `test/`,
+`__tests__/`, `e2e/`, `spec/`. A file named explicitly on the command line is
+always scanned. Supported suffixes: `.py`, `.js`, `.jsx`, `.ts`, `.tsx`, `.mjs`,
+`.cjs`. Dependency directories (`node_modules`, `.venv`, …) are never walked.
+
+## Invocation
+
+```bash
+# Scan the repo's test files (advisory - always exits 0):
+canary skills run canary-savant
+
+# Scan a specific suite:
+canary skills run canary-savant -- tests/unit
+
+# Machine-readable findings:
+canary skills run canary-savant -- tests --json
+
+# Fail the step on any suspect:
+canary skills run canary-savant -- tests --strict
+
+# Usage, options, and the full rule list (exits 0):
+canary skills run canary-savant -- --help
+```
+
+An unknown flag is rejected with `unrecognized arguments: <flag>` and exit 2.
+`--seed` is validated as a determinism flag rather than being allowed to decay
+into a random seed. All three failures exit 2:
+
+| Input                              | Error                                                        |
+| ---------------------------------- | ------------------------------------------------------------ |
+| missing value, or a following flag | `argument --seed: expected one argument`                     |
+| `abc`, `3.7`, `1e999`, `0x10`, ``  | `argument --seed: invalid int value: '<value>'`              |
+| beyond ±(2^53 − 1)                 | `argument --seed: seed out of safe integer range: '<value>'` |
+
+The last one matters because `Number()` rounds silently above 2^53 − 1, so the
+seed actually used would differ from the seed asked for — the same lie the whole
+flag exists to prevent.
+
+`--seed 42` and `--seed=42` are equivalent, and both accept negative seeds
+(`--seed -5`, `--seed=-5`). Use `--` to end option parsing when a path itself
+starts with a dash.
+
+### The confirming pass — dynamic confirmation (`--confirm`, opt-in)
+
+`--confirm` runs the suite in declared order, re-runs it shuffled under a pinned
+seed, and for each order-dependent victim runs it alone and **bisects the prefix
+to name the polluter** — the earlier test whose state leaked. Opt-in because a
+shuffled re-run at least doubles wall-clock; always prints the seed and a
+copy-pasteable reproduce command.
+
+```bash
+# Confirm order-dependence, pinning the seed for reproducibility:
+canary skills run canary-savant -- tests --confirm --seed 424242
+```
+
+**pytest and vitest** are both supported; savant auto-detects from the target
+(extensions, then a directory scan, then config files). pytest needs a shuffle
+plugin (`pytest-randomly` or `pytest-random-order`) and declines loudly if none
+is installed; vitest's shuffle is built in (`--sequence.shuffle`), so no plugin
+is required. Node drives the project's _own_ runner — savant orchestrates it, it
+does not run in the target's language.
+
+**Polluter bisect is pytest-only.** vitest has no CLI-driven ordered per-test
+execution, so a vitest target gets victim _detection_ (which tests break under
+shuffle) but not culprit _naming_. For pytest, node ids are captured via
+`--collect-only`, so class-based layouts (`file.py::Class::test`) re-run
+correctly.
+
+`--json` shape:
+
+```json
+{
+  "schema_version": 1,
+  "findings": [
+    {
+      "file": "tests/test_cache.py",
+      "line": 3,
+      "rule_id": "SV001-module-mutable-global",
+      "severity": "medium",
+      "snippet": "_CACHE = {}",
+      "why": "a module-level mutable is written by a test, so state leaks into whatever test runs next"
+    }
+  ],
+  "summary": {
+    "files_scanned": 8,
+    "findings": 1,
+    "by_severity": { "medium": 1 },
+    "suppressed": 0
+  }
+}
+```
+
+## Fixing what it finds
+
+| Finding | Fix                                                                                                                                             |
+| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| `SV001` | Move the mutable into a fixture that rebuilds it per test, or reset it in teardown. Module-level mutable state shared across tests is the leak. |
+| `SV002` | Add the matching teardown (`teardown_method`, `afterEach`, a `yield` fixture) so acquired state is released.                                    |
+| `SV003` | Use a restoring helper — pytest `monkeypatch.setenv`, or save/restore around the test — instead of assigning the global directly.               |
+| `SV004` | Make the test self-contained so order stops mattering, then drop the ordering hint from the name/comment.                                       |
+
+## Dogfooding and the `--strict` promotion path
+
+canary runs savant's static pass over its **own** test suite on every PR
+(`.github/workflows/harness-quality.yml`, the `Skills (JS)` job), **advisory**:
+it prints suspects to the log and always exits 0. Tuning the rules against that
+real suite dropped the backlog from 37 findings to a handful of genuine
+suspects; the structural residue (savant's own suite testing SV004's
+directive-text detection, #496) is suppressed with per-site `savant-ignore`
+reasons and stays visible in the suppressed count. Promote to blocking by
+appending `--strict` to that step once the remaining suspects are triaged (fixed
+or confirmed benign) - the same advisory-first path every canary gate takes.
+
+## Roadmap
+
+- **Shipped:** the static pass; the dynamic confirming pass (baseline → shuffle
+  → classify) with isolation + polluter bisect (pytest); vitest as a
+  confirming-pass classify target; pytest node-id capture for class-based
+  layouts; advisory CI gate dogfooded on canary's own suite (rules tuned to kill
+  the dominant false positives).
+- **Remaining:** flip the advisory gate to `--strict` once the suspect backlog
+  is triaged; vitest polluter naming is out of scope until vitest gains ordered
+  per-test execution. See `docs/changes/canary-savant/proposal.md`.