canary-test-cli 7.1.0 → 8.0.0

package/agents/skills/claude-code/canary-add-framework/SKILL.md

@@ -0,0 +1,248 @@
+---
+name: canary-add-framework
+description: >
+  Add a new testing framework to Canary's registry end-to-end — aligns the
+  classifier↔registry contract, authors the
+  `ts/src/data/frameworks/registry.json` entry, validates the execution command,
+  and ensures every classifier `test_type` still resolves to a non-null
+  framework. Use for "add support for a new framework", "add k6", "support
+  cypress", "we need Locust", or when a classifier `test_type` exists with no
+  framework backing it (registry gap). Not for choosing between existing
+  frameworks at runtime (that's the recommender's job) or adding a one-off
+  framework name with no real CLI.
+---
+
+# Canary: Add Framework
+
+> Add a new testing framework to Canary's registry end-to-end. Confirms the
+> classifier↔registry contract, authors the registry entry, validates the
+> execution command, and ensures every classifier `test_type` still resolves to
+> a non-null framework.
+
+## When to Use
+
+- When the user asks to add support for a new framework ("add k6", "support
+  cypress", "we need Locust")
+- When a classifier `test_type` exists but has no framework backing it (registry
+  gap)
+- When migrating from a deprecated framework — adding the new entry before
+  removing the old one
+- NOT for choosing between existing frameworks at runtime — that's the
+  recommender's job
+- NOT for adding a one-off LLM-generated framework name that doesn't exist as a
+  real tool — frameworks must be real, runnable CLIs
+
+## Process
+
+### Phase 1: SCOPE — Confirm the Framework Belongs in the Registry
+
+1. **Verify the framework is real and installable.** It needs a runnable CLI
+   invocation (the `execution_command` template). If you can't write a one-line
+   shell command that runs a single test file, it doesn't fit Canary's model.
+2. **Identify the category.** Match to an existing `category` value (`e2e_ui`,
+   `python_unit`, `api`, `performance`, `frontend_unit`, etc.). New categories
+   require a coordinated classifier update — see Phase 2.
+3. **Check for overlap.** Is there already a `preferred` framework in this
+   category? If yes, decide intentionally: is the new framework a replacement
+   (demote the old to `legacy`), a peer (add as `supported`), or the new
+   preference (demote the old, mark new as `preferred`)?
+4. **Confirm the user wants this maintained.** Each registry entry is a
+   maintenance commitment. A framework added casually but never validated rots
+   into a routing trap.
+
+### Phase 2: ALIGN — Classifier ↔ Registry Contract
+
+1. **Find the classifier rule that emits this `test_type`.** Open
+   `ts/src/core/classifier.ts` and locate the heuristic block that produces the
+   target category.
+2. **If no classifier rule exists for the category:** Add one _before_ the
+   registry entry. A framework with no routing path is a dead entry. Add a
+   heuristic that maps natural-language signals (keywords, phrases) to the new
+   `test_type`.
+3. **If the category exists but the classifier never emits it confidently:**
+   Strengthen the rule's heuristics. Confidence below 0.7 should trigger a
+   clarification, not a generation.
+4. **Run the contract tests:**
+
+   ```bash
+   pytest tests/unit/test_orchestrator.py
+   ```
+
+   Every `test_type` the classifier can emit must resolve to ≥1 framework via
+   `get_by_category` — the orchestrator tests assert non-null framework
+   resolution per category. This is the gate.
+
+### Phase 3: AUTHOR — Write the Registry Entry
+
+1. **Open `ts/src/data/frameworks/registry.json`.** Add a new object to
+   `frameworks[]`. Required fields:
+   - `name` (unique slug)
+   - `display_name`
+   - `category` (must match the classifier emission)
+   - `languages`
+   - `file_extensions`
+   - `execution_command` (with `{file}` placeholder)
+   - `status` (`preferred` / `supported` / `legacy`)
+2. **Add recommended metadata.** `maturity`, `community_size`,
+   `recommended_for`, `strengths`, `avoid_when`, `ecosystems`. These show up in
+   the recommender's reasoning string — sparse entries produce sparse
+   recommendations.
+3. **Validate the `execution_command`.** It must run a single test file when
+   `{file}` is substituted. Test manually:
+
+   ```bash
+   echo '<minimal test>' > /tmp/sample.<ext>
+   <execution_command with /tmp/sample.<ext> substituted>
+   ```
+
+   Confirm exit code 0 (or expected non-zero) and no missing-config errors.
+
+4. **Confirm the `{file}` placeholder is the only substitution.** The executor
+   tokenizes the template with `shlex.split` before substituting `{file}`, so
+   the file path stays a single argv element; paths with spaces must remain
+   intact.
+
+### Phase 4: VERIFY — Generate, Execute, Promote-Dry
+
+1. **Generate a test that should route to the new framework.** Run the
+   `/canary-write-test` slash command in Claude Code with a prompt that should
+   hit the new `test_type`.
+
+2. **Check the trace.** Classification matches the expected category,
+   recommendation picks the new framework (or the existing preferred — confirm
+   this matches your intent), output file has the right extension.
+3. **Execute with `--execute`.** Confirm the framework's CLI actually runs the
+   generated file. A passing execution validates the `execution_command`
+   template under the real Canary invocation path.
+4. **Don't promote.** This is a registry-validation run, not a real test
+   promotion. Leave the artifact in `tests/generated/`.
+
+### Phase 5: DOCUMENT — Update Guides and State
+
+1. **Update `docs/guides/framework-registry.md`** if you added a new category,
+   changed the contract surface, or introduced a new optional field schema.
+2. **Update the LLM-providers guide** if the new framework only works with
+   specific provider output styles (rare; flag this as a smell if it's true).
+3. **Log to `docs/CANARY_STATE.md`.** One line: framework name, category,
+   status, date added. This is the project ledger; future maintainers grep here
+   first.
+4. **Open a PR.** Title: `feat(registry): add <framework-name> support`. Body
+   should include the validation steps you ran and the generated/executed
+   sample.
+
+## Canary Integration
+
+- **`ts/src/data/frameworks/registry.json`** — Source of truth for entries.
+- **`ts/src/core/framework-registry.ts`** — Loader and lookup methods. Don't add
+  lookup helpers here unless the existing five (`get_all_frameworks`,
+  `get_by_category`, `get_preferred_by_category`, `find_by_name`,
+  `match_by_language`) genuinely don't fit.
+- **`ts/src/core/classifier.ts`** — Routing rules. Coordinate changes with
+  registry entries.
+- **`tests/unit/test_orchestrator.py`** — Contract enforcement. Non-null
+  resolution per `test_type` (api → pytest, e2e_ui → playwright, performance →
+  k6, etc.) is asserted here. `tests/unit/test_factory.py` separately enforces
+  the LLM provider matrix.
+
+## Success Criteria
+
+- New entry validates against the contract test (no orphaned `test_type` in
+  classifier, no null framework resolution in registry)
+- The `/canary-write-test` slash command with a routing prompt resolves to the
+  new framework
+- `--execute` runs the generated file successfully via the new
+  `execution_command`
+- `framework-registry.md` documents any new category or schema field
+- `CANARY_STATE.md` ledger entry is present
+- PR includes the validation transcript
+
+## Rationalizations to Reject
+
+| Rationalization                                                           | Why It Is Wrong                                                                                                                                                                                           |
+| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| "I'll add the registry entry now and update the classifier later"         | Orphaned registry entries are routing traps. Either both land together or neither does.                                                                                                                   |
+| "Marking it `preferred` is fine, no need to demote the existing entry"    | Multiple `preferred` entries in one category means `get_preferred_by_category` picks by registry order, not by merit. Demote the loser explicitly.                                                        |
+| "I tested the command with my path — Canary's substitution will work too" | Test the substituted command with a path that contains spaces. The executor's `shlex.split`-then-substitute order exists specifically to handle this case; a broken template will silently corrupt argv.  |
+| "It only needs to work for the happy-path prompt I tried"                 | If the classifier emits this `test_type` with confidence ≥0.7 for _any_ phrasing, the framework must handle the resulting generated file. Test at least two distinct prompts that route to the new entry. |
+| "Sparse metadata is fine, we can fill it in later"                        | The recommender's reasoning string is read by humans and agents. Sparse metadata produces unhelpful recommendations — fill it in now while you have the context.                                          |
+
+## Examples
+
+### Example: Adding Cypress as a peer to Playwright
+
+**Scope:** Cypress already has community demand and a CI image. Category
+`e2e_ui` already has Playwright as `preferred`. Decision: add Cypress as
+`supported`, keep Playwright as `preferred`.
+
+**Classifier:** Already emits `test_type=e2e_ui` on UI-test prompts. No
+classifier change needed.
+
+**Entry (abridged):**
+
+```json
+{
+  "name": "cypress",
+  "display_name": "Cypress",
+  "category": "e2e_ui",
+  "languages": ["javascript", "typescript"],
+  "file_extensions": ["cy.ts", "cy.js"],
+  "execution_command": "npx --yes cypress run --spec {file}",
+  "status": "supported",
+  "strengths": ["Time-travel debugging", "Strong DX for UI tests"],
+  "avoid_when": ["Cross-browser parity required (Playwright is stronger)"]
+}
+```
+
+**Validation:** Generated a UI test prompt, confirmed recommender still picked
+Playwright (correct — `preferred` wins). Verified Cypress is accessible by
+routing a prompt with explicit Cypress mention through a future
+explicit-framework override.
+
+### Example: Adding k6 to a new performance category
+
+**Scope:** Performance category doesn't exist in the registry yet. Classifier
+emits `test_type=performance` on load/stress prompts, but the registry has no
+matching entry — this is the orphan case.
+
+**Phase 2 first:** Confirmed `ts/src/core/classifier.ts` already has the
+performance rule (`if "performance" in p or "load test" in p ...`). No
+classifier change needed.
+
+**Phase 3:**
+
+```json
+{
+  "name": "k6",
+  "display_name": "k6",
+  "category": "performance",
+  "languages": ["javascript"],
+  "file_extensions": ["js"],
+  "execution_command": "k6 run {file}",
+  "status": "preferred",
+  "recommended_for": ["HTTP load testing", "Stress and spike tests"],
+  "avoid_when": ["Browser-level performance (use Playwright tracing)"]
+}
+```
+
+**Validation:** Ran the `/canary-write-test` slash command with "Load test
+/v1/search at 200 RPS for 5 minutes". Confirmed classifier emits
+`test_type=performance` at 0.95 confidence, recommender picks k6, k6 CLI runs
+the generated file without error.
+
+## Escalation
+
+- **When the framework requires a non-trivial config file (e.g.,
+  `playwright.config.ts`):** Document the setup in `avoid_when` or in
+  `recommended_for`. If Canary is expected to generate the config too, that's a
+  scaffolder change, not just a registry change.
+- **When two frameworks legitimately should both be `preferred` for different
+  sub-cases:** Split the category. A single category should have a single
+  `preferred`. If the split isn't clean, surface this to the user as a design
+  decision rather than fudging the registry.
+- **When the classifier doesn't reliably emit the target `test_type`:** Don't
+  add the registry entry yet. Strengthen the classifier first — an unreachable
+  entry is dead weight.
+- **When the framework's CLI doesn't accept a single-file argument:** Canary's
+  model is one-test-per-file generation. Frameworks that only run by directory
+  or by tag don't fit cleanly; raise this with the user before forcing a
+  workaround.

package/agents/skills/claude-code/canary-batwoman/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: canary-batwoman
+description:
+  Closure auditing — reports whether the files a closed issue's fix changed have
+  actually EXECUTED since that fix merged. GitHub closes an issue on a keyword
+  match in a PR body, which checks neither that the fix works nor that it ever
+  ran; batwoman answers only the second question, per changed file, and names
+  the files it could not answer for. Use when the user asks "did that fix
+  actually run", "is this issue really done", "audit a closed issue", or after a
+  batch of merges. Advisory and read-only — it never asserts correctness, never
+  fails a job, and never reopens an issue. NOT a test runner, NOT a coverage
+  tool (a dormant workflow has whatever coverage it always had), and NOT a
+  correctness check.
+cli: canary batwoman
+requires: [node>=20, gh]
+---
+
+# Canary Batwoman
+
+GitHub closes an issue when a merged PR body matches `Closes #N`. That is a
+**string match with no denominator**: nothing checks that the fix works, and
+nothing checks that the fix ever ran. The issue moves to `CLOSED` on the
+strength of a keyword.
+
+Batwoman audits the claim. For each file the closing PR changed, it reports
+whether that artifact has executed since the merge — and, just as importantly,
+which files it could not decide about.
+
+## The founding case
+
+canary#749 fixed a false-green in `.github/workflows/refresh-arch-baseline.yml`.
+It merged as `1e0c05b` and auto-closed on the keyword. Verified afterwards:
+
+- 11 script unit tests passed
+- 97 static workflow assertions passed
+- the workflow itself **had not run since twelve days before the fix merged** —
+  it is label-triggered, so it stayed dormant until someone applied the label
+
+Every gate the repo owns read that as done. The fix was real and well-tested;
+its end-to-end path was unproven. **Verified and exercised are different
+properties**, and that gap is what batwoman makes visible.
+
+## Usage
+
+```bash
+canary batwoman --help
+```
+
+That one needs no credentials, no network and no fixtures. A real audit needs
+the repository named and `gh` authenticated:
+
+```bash
+export GITHUB_REPOSITORY=owner/name   # never inferred from a git remote
+canary batwoman --issue 749           # the report
+canary batwoman --issue 749 --json    # the machine shape, for CI
+```
+
+`GITHUB_REPOSITORY` is required rather than guessed. Auditing the wrong
+repository's run history would produce a confident answer about the wrong thing,
+which is worse than refusing.
+
+## What the five statuses mean
+
+| status           | meaning                                                                    |
+| ---------------- | -------------------------------------------------------------------------- |
+| `exercised`      | a run started after the merge, and it names which                          |
+| `not-exercised`  | the artifact has not run since — with the `on:` trigger explaining why     |
+| `abstain`        | a probe looked and could not tell (unreadable history, untraceable script) |
+| `no-probe`       | nothing looked: a registry gap, fixable by adding a probe                  |
+| `not-applicable` | prose, config, or a file this change deleted                               |
+
+`abstain` and `no-probe` are deliberately separate, and neither is folded into
+anything resembling a pass. **The summary always prints all five counts**, and
+they always sum to the changed-file total — a report over a subset presented as
+a report over the whole is the exact defect batwoman exists to detect.
+
+There is no `assessed` field, no score, and no success token anywhere in the
+output. A convenient `passed: true` in the JSON is the field a CI wrapper would
+grow later; it does not exist, and a test asserts it stays that way.
+
+## What it will not tell you
+
+- **Whether the fix is correct.** Batwoman proves execution, never behaviour. A
+  workflow that ran and did the wrong thing reports `exercised`.
+- **Whether coverage is adequate.** A dormant workflow has whatever coverage it
+  always had; that is the point.
+- **Anything about a file it has no probe for.** `ts/src/**` is an honest
+  `no-probe` row in v1, countable rather than silent.
+
+## Probes shipped
+
+| probe             | matches                   | decides by                                                               |
+| ----------------- | ------------------------- | ------------------------------------------------------------------------ |
+| `workflow`        | `.github/workflows/*.yml` | a run created after the merge; reads `on:` to explain a dormant one      |
+| `workflow-script` | `scripts/*.mjs`           | the workflows that name it, inheriting `exercised` if **any** caller ran |
+| `no-execution`    | `*.md`, config manifests  | returns `not-applicable`, reading nothing                                |
+
+A script nothing references **abstains** rather than reporting as never-run: "no
+workflow calls this" is a statement about the repo, not about the script, which
+may still be run by a hand or a hook.
+
+## Requires the network, and says so
+
+Batwoman shells out to `gh` for run history and for the closing PR. That is a
+property, not a tier — unlike the deterministic offline detectors alongside it
+(`canary-savant`, `canary-blackhawk`, `canary-cassandra`, `canary-katana`),
+which run anywhere node does.
+
+An unauthenticated or missing `gh` does not degrade to a clean report. It fails
+loudly, and every affected file becomes an `abstain` naming the failure. Per
+this repo's standing rule, **cannot-verify is a finding, not a skip**.
+
+## In CI
+
+`.github/workflows/batwoman.yml` runs it on every push to `main`, deriving the
+issue from the merge commit's closing keyword — the very keyword match batwoman
+questions, which is the right input precisely because it is what GitHub acted
+on. Advisory: every step is `continue-on-error`, and a merge that closes nothing
+is reported and skipped rather than passing silently over an empty set.

package/agents/skills/claude-code/canary-blackhawk/SKILL.md

@@ -0,0 +1,170 @@
+---
+name: canary-blackhawk
+description:
+  Temporal-dependency linter for test files — statically flags tests that lean
+  on wall-clock time, a real delay, or the local timezone, the ones that pass
+  all day and fail at midnight, across a DST boundary, or on Feb 29. Suppresses
+  itself when the file already installs a frozen clock (fake timers, freezegun,
+  time_machine, MockDate). Self-contained, deterministic, advisory by default.
+cli: scripts/cli.mjs
+requires: [node>=20]
+---
+
+# Canary Blackhawk
+
+A test that reads the wall clock is a test with a scheduled outage. It passes
+every run you watch, then fails at 23:59:59, on the Sunday the clocks move, or
+on Feb 29. Blackhawk finds those lines before the calendar does.
+
+Tier-0 deterministic analysis: no LLM, no network, no secrets, no dependency on
+any other skill.
+
+## Rules
+
+| Rule                           | Severity | Fires on                                                                                                                                      |
+| ------------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| `BH001-wall-clock`             | high     | `Date.now()`, bare `new Date()`, `moment()`, `datetime.now()` / `.today()` / `.utcnow()`, `date.today()`, `time.time()`, `pd.Timestamp.now()` |
+| `BH002-real-delay`             | medium   | `time.sleep(n)` and `setTimeout(fn, n)` with a literal `n > 0`                                                                                |
+| `BH003-local-timezone`         | medium   | `toLocaleString` / `toLocaleDateString` / `toLocaleTimeString`, `strftime('…%Z…')` or `%z`                                                    |
+| `BH004-naive-datetime-compare` | low      | a comparison against `datetime(2024, …)` or `strptime(…)` with no `tzinfo` / `timezone.utc` / `ZoneInfo` / `pytz` on the line                 |
+
+A pinned constructor is never flagged: `new Date('2024-01-01T00:00:00Z')`,
+`moment('2024-01-01')`, and `datetime(2024, 1, 1)` on its own are all fine.
+
+## Framework-conditioned suppression (the important part)
+
+The single biggest failure mode of a temporal linter is firing on tests that
+**already handle time correctly**. So when a file installs a frozen clock, every
+clock-dependent rule (`BH001`, `BH002`, `BH004`) goes quiet for that file.
+Markers:
+
+`vi.useFakeTimers` · `vi.setSystemTime` · `jest.useFakeTimers` ·
+`jest.setSystemTime` · `sinon.useFakeTimers` · `MockDate` · `freeze_time` /
+`freezegun` · `time_machine`
+
+Two deliberate choices inside that:
+
+- **Suppression is file-wide, not block-scoped.** `vi.useFakeTimers()` inside a
+  `beforeEach` governs tests declared above it, and answering "is the clock
+  frozen _here_" accurately needs a real parser. Blackhawk errs toward silence.
+- **`BH003` is never suppressed.** Freezing the clock pins _when_ a test runs,
+  never _where_. A frozen clock does not stop `toLocaleString()` from returning
+  a different string on a developer laptop than on a UTC runner.
+
+## Inline suppression (per-line escape hatch)
+
+File-wide frozen-clock suppression is too coarse when one intentional real wait
+sits among otherwise-deterministic tests. An inline pragma silences a single
+finding:
+
+```js
+// blackhawk-ignore BH002 -- real MongoDB claim race; deterministic wait impossible
+await new Promise((r) => setTimeout(r, 25));
+```
+
+- **Reason required** (the `-- reason` tail) — keeps suppressions honest and
+  greppable, like `eslint-disable-next-line` / `# noqa`.
+- **Rule-scoped** (`BH002`, or the full `BH002-real-delay`) so it never
+  blanket-silences the line; a `BH001` on the same line still fires.
+- **Placement:** the pragma may trail the offending line or sit on the line
+  directly above it. Comma-separate ids to suppress several (`BH002,BH004`).
+- **Counted separately:** suppressed findings are reported as `N suppressed`,
+  out of the actionable total — so a genuinely clean suite can read zero
+  findings while the known-OK waits stay visible.
+
+## Fidelity limits (regex/AST-lite, on purpose)
+
+Blackhawk is a line scanner with no TypeScript parser dependency, so it ships
+anywhere `node` does. The cost, stated plainly:
+
+- **Line-scoped.** A call split across lines (`setTimeout(\n  fn,\n  500\n)`) is
+  missed, as is a `setTimeout` whose callback body contains a comma.
+- **Comment-blind, one level deep.** Lines starting with `#`, `//`, `*`, `/*`,
+  or a docstring quote are skipped; a multi-line block comment whose inner lines
+  do not start with `*` is still scanned.
+- **String-aware, one line deep.** A match that _starts_ inside a string literal
+  on its own line is rejected as fixture data (`pyFile('time.sleep(1)')` does
+  not fire), while an anchor whose pattern merely reaches into quotes
+  (`strftime('..%Z')`) still does; template `${...}` interpolation is code. A
+  string spanning lines is only seen on its opening line, so a fixture blob's
+  continuation lines are still scanned like code.
+- **No type awareness.** `.toLocaleString()` on a `Number` reads the same as on
+  a `Date`.
+- **Suppression is a substring match.** A mention of `freezegun` in a comment
+  silences the file. That direction is intentional: a missed finding costs less
+  than a false one.
+
+## Which files get scanned
+
+A directory walk only visits **test** files — `*.test.*`, `*.spec.*`,
+`test_*.py`, `*_test.py`, or any supported source under `tests/`, `test/`,
+`__tests__/`, `e2e/`, `spec/`. A file named explicitly on the command line is
+always scanned, test-looking or not. Supported suffixes: `.py`, `.js`, `.jsx`,
+`.ts`, `.tsx`, `.mjs`, `.cjs`.
+
+## Invocation
+
+```bash
+# Scan the repo's test files (advisory — always exits 0):
+canary skills run canary-blackhawk
+
+# Scan a specific suite:
+canary skills run canary-blackhawk -- tests/e2e
+
+# Machine-readable findings:
+canary skills run canary-blackhawk -- tests --json
+
+# Fail the step on any finding:
+canary skills run canary-blackhawk -- tests --strict
+
+# Usage, options, and the full rule list (exits 0):
+canary skills run canary-blackhawk -- --help
+```
+
+An unknown flag is rejected with `unrecognized arguments: <flag>` and exit 2;
+use `--` to end option parsing when a path itself starts with a dash.
+
+`--json` shape:
+
+```json
+{
+  "schema_version": 1,
+  "findings": [
+    {
+      "file": "tests/clock.spec.ts",
+      "line": 12,
+      "rule_id": "BH001-wall-clock",
+      "severity": "high",
+      "snippet": "const t = Date.now();",
+      "why": "reads the wall clock, so the assertion depends on when the suite runs..."
+    }
+  ],
+  "summary": {
+    "files_scanned": 8,
+    "findings": 1,
+    "by_severity": { "high": 1 },
+    "suppressed": 0
+  }
+}
+```
+
+## CI wiring (GitHub Actions)
+
+Advisory first, then promote to blocking once the backlog is drained and the
+signal is trusted — the same path every canary gate takes.
+
+```yaml
+- name: Temporal-dependency lint (advisory)
+  run: canary skills run canary-blackhawk -- tests
+# Once clean, add --strict to make new offenders fail the PR:
+# run: canary skills run canary-blackhawk -- tests --strict
+```
+
+## Fixing what it finds
+
+| Finding | Fix                                                                                                                                              |
+| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `BH001` | Freeze the clock (`vi.useFakeTimers()` + `vi.setSystemTime(...)`, `@freeze_time("2024-01-01")`) or inject a clock the test controls.             |
+| `BH002` | Advance a fake timer (`vi.advanceTimersByTime(500)`) or await the real condition instead of a duration.                                          |
+| `BH003` | Assert on a UTC representation (`toISOString()`, `strftime('%Y-%m-%dT%H:%M:%SZ')` on a UTC datetime), or pin the locale and timezone explicitly. |
+| `BH004` | Attach a timezone: `datetime(2024, 1, 1, tzinfo=timezone.utc)`.                                                                                  |