canary-test-cli 7.1.0 → 7.2.0

package/agents/skills/claude-code/canary-fleet-health/SKILL.md

@@ -0,0 +1,196 @@
+---
+name: canary-fleet-health
+description: >
+  Fleet-wide test health summary across suites — flaky tests, failure spikes,
+  cross-suite common failures, and regression candidates from the run-history
+  store. Use when the user asks "how healthy is our test fleet", "fleet-wide
+  flake report", "any regressions this week", "failure spikes across suites", or
+  "canary analyze". Produces one compact, scannable summary — not a dashboard.
+  NOT for diagnosing a single known-flaky test (canary-flake-hunter) or scoring
+  one suite's CI readiness (canary-ci-ready).
+---
+
+# Canary: Fleet Health
+
+Wraps `canary analyze` (fleet-wide flake/spike/regression analytics) and the
+run-history store to answer "how's the whole fleet doing?" in one chat-turn
+summary. Today, [`canary-flake-hunter`](../../../canary-flake-hunter.md) only
+diagnoses a single test you already suspect is flaky — this skill is the
+fleet-wide counterpart: it tells you _where to look_ before you reach for the
+hunter.
+
+Per the adoption audit this skill implements (candidate #10), this is
+deliberately a **low-cost validation step**: a compact text summary a human can
+scan in one turn, not a dashboard or visual surface. If fleet-wide analytics
+prove valuable, a richer surface is a separate, larger investment — don't
+over-build this one.
+
+## When to Use
+
+- Weekly/periodic health check: "how's the test fleet looking?"
+- Before a release: "any regressions or spikes we should know about?"
+- Triaging where to spend test-maintenance effort across many suites
+- NOT for a single test you already know is flaky — use
+  [`canary-flake-hunter`](../../../canary-flake-hunter.md) to diagnose root
+  cause and propose a fix
+- NOT for scoring one suite's CI readiness — use
+  [`canary-ci-ready`](../canary-ci-ready/SKILL.md) (coverage depth, assertion
+  quality, runtime for _one_ suite)
+- NOT a substitute for `.canary/critical-areas.json` risk ranking — use
+  [`canary-critical-areas`](../canary-critical-areas/SKILL.md) for code-level
+  risk, this skill is history-data-level health
+
+## Process
+
+### Phase 1: RESOLVE THE STORE
+
+Fleet analytics read from the run-history store, not from a live test run.
+Before running anything, confirm data exists:
+
+```bash
+canary history summary <suite> --runs 1
+```
+
+- **Configured store:** `CANARY_HISTORY_DB_URL` env var (Supabase-backed) or
+  falls back to the local NDJSON file at
+  `test-results/reports/history-v2.jsonl`.
+- **No local file and no `CANARY_HISTORY_DB_URL`:** there is nothing to analyze
+  yet. Say so plainly — "No run history found. Push results with
+  `canary history push` after a CI run, or run `canary history migrate` if you
+  have v1 history.jsonl data." Do not fabricate a health summary from nothing.
+
+### Phase 2: RUN THE RELEVANT ANALYSES
+
+Default to the combined digest unless the user asked about one specific
+dimension:
+
+```bash
+canary analyze digest --json
+```
+
+If the user asked about one thing specifically, run only that subcommand instead
+of the full digest — cheaper and more focused:
+
+| User asks about                         | Command                                                              |
+| --------------------------------------- | -------------------------------------------------------------------- |
+| Flaky tests fleet-wide                  | `canary analyze flaky --window-runs 30 --min-rate-pct 10 --json`     |
+| Failure spikes                          | `canary analyze spikes --delta-pp 20 --json`                         |
+| Cross-suite common failures             | `canary analyze common-failures --min-suites 2 --json`               |
+| Newly broken tests after a green streak | `canary analyze regression-candidates --json`                        |
+| "Area health" / degrading areas         | See the caveat below — this dimension does not currently return data |
+
+**Known limitation — be upfront about it:** `canary analyze area-health` (and
+the `area_health` section of `digest`) is currently wired to an empty data set
+in `ts/src/analysis/cli.ts` / `ts/src/analysis/engine.ts` — it always reports
+"No area health data available," regardless of history. Don't present this as a
+working check; tell the user area-degradation tracking isn't implemented yet
+rather than silently omitting it.
+
+**Store-type caveat:** `flaky` queries the store directly and works with either
+backend. `spikes`, `common-failures`, and `regression-candidates` currently only
+populate fully when the backing store is the local NDJSON file (`AnalysisEngine`
+special-cases `LocalHistoryStore` for suite discovery and per-test aggregation)
+— with a Supabase-backed store (`CANARY_HISTORY_DB_URL` set), those three may
+come back empty even with real history. If the digest shows all-zero
+spikes/common-failures/ regressions _and_ `CANARY_HISTORY_DB_URL` is set, flag
+this as a likely store-support gap, not a clean bill of health.
+
+### Phase 3: CONDENSE TO ONE SCREEN
+
+Don't paste raw Markdown tables from the CLI — they're built for file artifacts,
+not chat. Pull the top 3–5 rows per section and compress to the Output Format
+below. If a section is empty, say "none" in one line; don't render an empty
+table.
+
+### Phase 4: SURFACE THE ONE ACTIONABLE THING
+
+Look across sections for correlation — e.g., a suite with both a recent spike
+and several tests over the flake threshold is a stronger signal than either
+alone. Call that out explicitly as the suggested next step, and name the
+specific downstream skill:
+
+- Single suspicious test → point at
+  [`canary-flake-hunter`](../../../canary-flake-hunter.md)
+- Whole suite trending down → point at
+  [`canary-ci-ready`](../canary-ci-ready/SKILL.md) for that suite
+- Systemic cross-suite pattern (e.g. the same connection error in 3 suites) →
+  this is infrastructure/environment, not a test bug; say so instead of
+  suggesting a test fix
+
+## Output Format
+
+```text
+Fleet Health — window: 30 runs
+
+  Flaky (≥10%):        3 tests   top: checkout_retry_test (32%, api suite)
+  Spikes (≥20pp):      1 suite   e2e_ui +25pp since 2026-07-10
+  Area health:         not available (not yet implemented — always empty)
+  Common failures:     1 pattern  "ECONNREFUSED 127.0.0.1:5432" across 2 suites
+  Regressions:         2 tests   orders_post_201 broke after 12-run green streak
+
+  Suggested next step: e2e_ui's spike + 2 of the 3 flaky tests are in that
+  suite — investigate the suite before chasing individual flakes.
+  Run /canary-ci-ready on e2e_ui, or /canary-flake-hunter on
+  checkout_retry_test for a root-cause fix.
+```
+
+Keep it to one screen. Omit a line entirely rather than padding with "no data"
+noise for sections that were never requested.
+
+## Flags
+
+- `--window-runs <runs>` — rolling window measured in RUNS, not days (default:
+  30, passed through to `canary analyze`)
+- `--min-rate-pct <percent>` — minimum flake rate to report, on a 0-100 percent
+  scale (default: 10, so `10` means 10%, not 0.1)
+- `--delta-pp <points>` — spike threshold as a PERCENTAGE-POINT rise in failure
+  rate (default: 20, so `20` means 5% → 25%, not 5% → 6%)
+- `--suite <name>` — scope to one suite instead of the whole fleet
+- `--json` — pass through the underlying CLI's `--json` when the caller wants
+  structured data instead of a chat summary
+
+The unitless spellings `--window`, `--delta`, and `--min-rate` are deprecated
+aliases: they still work and take the same values, but they print a note on
+stderr naming the replacement. Prefer the unit-bearing names in anything you
+write down.
+
+## Error Handling
+
+| Situation                                                           | What To Do                                                                                                                                                                                                                                                   |
+| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| No history data at all (fresh repo)                                 | Say so; point at `canary history push` / `canary history migrate`. Don't run the analysis commands against nothing.                                                                                                                                          |
+| `CANARY_HISTORY_DB_URL` set but Supabase unreachable                | `make_store` only guards against a missing `agent.history.supabase_store` import, not a live connection failure — a query error will surface as a CLI exception. Report the exception text, suggest checking connectivity/credentials, don't retry silently. |
+| `area-health` requested explicitly                                  | Explain the known limitation (Phase 2) rather than showing an empty table with no context.                                                                                                                                                                   |
+| Digest looks suspiciously all-zero with `CANARY_HISTORY_DB_URL` set | Flag the store-type caveat (Phase 2) before concluding the fleet is healthy.                                                                                                                                                                                 |
+| User wants a specific suite that has no history                     | `canary history summary <suite>` returns `total_runs: 0` — report that directly instead of running the full fleet analysis.                                                                                                                                  |
+
+## Examples
+
+### Example: Weekly health check with a correlated signal
+
+**Prompt:** "How's the fleet looking this week?"
+
+**Action:** Confirm local history file exists. Run `analyze digest --json`.
+Condense: 3 flaky tests (2 in `e2e_ui`), 1 spike (`e2e_ui`, +25pp), no common
+failures, 2 regression candidates. Correlate: `e2e_ui` shows up in both flaky
+and spikes — call it out as the priority, suggest `canary-ci-ready` on `e2e_ui`
+before chasing the individual flaky tests.
+
+### Example: No history yet
+
+**Prompt:** "Give me a fleet health summary."
+
+**Action:** `history summary api --runs 1` returns `total_runs: 0` and no local
+NDJSON file exists. Report plainly: no run history is available yet; point at
+`canary history push` after the next CI run. Do not run `analyze digest` against
+an empty store and present an empty report as "all clear" — absence of data is
+not evidence of health.
+
+## Related Skills
+
+- [`canary-flake-hunter`](../../../canary-flake-hunter.md) — single-test
+  root-cause diagnosis once fleet health points at a candidate
+- [`canary-ci-ready`](../canary-ci-ready/SKILL.md) — single-suite CI readiness
+  scoring (coverage, assertions, runtime)
+- [`canary-critical-areas`](../canary-critical-areas/SKILL.md) — code-level risk
+  ranking, a different signal from history-based health

package/agents/skills/claude-code/canary-generate-test/SKILL.md

@@ -0,0 +1,185 @@
+---
+name: canary-generate-test
+description: >
+  Generate a framework-appropriate test from a natural-language requirement by
+  routing through Canary's classify → recommend → generate pipeline (the
+  `/canary-write-test` slash command), writing the test under `tests/generated/`
+  and optionally executing it. Use for "write a test for X", "I need an API test
+  that does Y", "scaffold a new test from this description", or triaging a bug
+  report into a regression test — when a CLI batch pipeline run (not an
+  interactive session-generated file) is what's wanted. See
+  `agents/canary-test-author.md` (interactive, session-generated, wired to
+  `/canary-write-test`) and `agents/canary-test-generator.md` (MCP
+  write_test_file retry loop) for the other two "write a test" paths. Not for
+  editing existing tests or choosing between frameworks abstractly.
+---
+
+# Canary: Generate Test
+
+> Generate a framework-appropriate test from a natural-language requirement.
+> Routes through Canary's classify → recommend → generate pipeline, writes the
+> test under `tests/generated/`, and optionally executes it.
+
+## When to Use
+
+- When the user asks to scaffold a new test from a natural-language description
+  ("write a test for X", "I need an API test that does Y")
+- When triaging a bug report into a regression test
+- When extending an existing test suite with a new case but the framework choice
+  is ambiguous
+- NOT for editing existing tests — use the project's normal edit flow
+- NOT for choosing between frameworks abstractly — use the framework-registry
+  docs, not this skill
+- NOT for running existing tests — use the framework's CLI directly
+
+### Relative to the other "write a test" paths
+
+This skill is the **batch generation path** (the `/canary-write-test` slash
+command) — use it for scripted/CI-driven generation runs consumed
+programmatically from `tests/generated/`. For an interactive, session-generated
+test with a human reviewing framework and code before it lands, use
+`agents/canary-test-author.md` (wired to `/canary-write-test`). For a
+single-file, automatic write-run-revise loop via MCP tools, use
+`agents/canary-test-generator.md`.
+
+## Process
+
+### Phase 1: CLARIFY — Resolve the Requirement
+
+1. **Confirm test type if ambiguous.** If the prompt doesn't clearly indicate
+   `unit | api | e2e | performance`, ask one targeted question before invoking
+   the pipeline. The classifier will guess, but a wrong guess costs a
+   regeneration.
+2. **Confirm target framework if the user has a preference.** The recommender
+   will pick by category from the registry; if the user explicitly wants
+   Playwright but the registry would pick Cypress for the category, surface that
+   mismatch now.
+3. **Capture concrete inputs.** Endpoint URL, payload shape, expected status,
+   selectors, performance thresholds — whatever the test actually needs. Vague
+   prompts produce vague tests.
+
+### Phase 2: GENERATE — Run the Pipeline
+
+1. **Invoke generation** by running the `/canary-write-test` slash command in
+   Claude Code with the requirement as its prompt.
+
+2. **Read the printed classification + recommendation.** Verify the resolved
+   `test_type` and `framework` match intent. If they don't, refine the prompt
+   and re-run — do not hand-edit the generated file to compensate for a
+   misclassification.
+3. **Locate the output.** The CLI prints an absolute path under
+   `tests/generated/<category>/`. The orchestrator return dict's `output_path`
+   is authoritative.
+
+### Phase 3: VALIDATE — Execute or Dry-Run
+
+1. **Run the generated test** with the framework's CLI (or pass `--execute` to
+   the generator). For api/e2e tests, run against a known-good environment
+   first.
+2. **If execution fails, classify the failure:**
+   - **Generation error** (syntax, wrong API shape) → regenerate with a more
+     specific prompt; don't hand-fix unless trivial
+   - **Environment error** (missing creds, wrong base URL) → fix the env, rerun
+   - **Real assertion failure** (the SUT behaves differently than the prompt
+     asserted) → this is a useful signal; review with the requester before
+     changing the test
+3. **Log the run.** Append the requirement, classification, framework, and
+   pass/fail to `docs/CANARY_STATE.md` so downstream sessions can pick up
+   context.
+
+### Phase 4: PROMOTE — Move from Generated to Committed
+
+If the generated test passes review and belongs in the committed suite, use the
+[`canary-promote-test`](../canary-promote-test/SKILL.md) skill. Promotion is its
+own workflow — don't collapse it into this one.
+
+## Canary Integration
+
+- **`/canary-write-test "<prompt>"`** — Primary entry. Runs the full classify →
+  recommend → generate pipeline in the Claude Code session and can immediately
+  run the generated test.
+- **`CanaryOrchestrator.run(prompt, execute=False)`** — Programmatic entry.
+  Returns the structured pipeline-result dict.
+- **`ts/src/data/frameworks/registry.json`** — Maps `test_type` → framework.
+  Edit here when adding a new framework; never hard-code framework choices in
+  callers.
+- **Generation runs in your Claude Code session** — there is no LLM provider
+  layer or API key to configure (that layer was removed in v3.0).
+
+## Success Criteria
+
+- The generated file parses and runs under its framework's CLI
+- The classifier's `test_type` matches the user's actual intent
+- The recommender's framework choice resolves from the registry (no nulls)
+- The execution result is captured in the return dict (when `execute=True`)
+- A promoted test passes review and runs cleanly in CI
+
+## Rationalizations to Reject
+
+| Rationalization                                                                     | Why It Is Wrong                                                                                                                               |
+| ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| "The classifier picked the wrong type but I'll hand-fix the output"                 | The hand-fix masks a real classifier gap. Refine the prompt or file a classifier issue — don't paper over routing bugs in the generated file. |
+| "I'll commit the generated test as-is, it's good enough"                            | Generated tests live in `tests/generated/` for a reason — they're unreviewed scratch. Promote intentionally.                                  |
+| "The registry doesn't have an entry for this test_type, I'll add `framework: null`" | Null breaks the contract. Every `test_type` must map to a framework. Add a real entry or change the classifier output.                        |
+| "I'll skip the validation phase, the test looks right"                              | LLM output that looks right and runs are different things. Always execute (or dry-run) before promoting.                                      |
+
+## Examples
+
+### Example: API test for a known endpoint
+
+**Prompt:** `Test that POST /v1/orders returns 201 with a valid payload`
+
+**Pipeline trace:**
+
+```text
+Classification: intent=generate_tests, test_type=api, confidence=0.85
+Recommendation: framework=requests-pytest, ext=.py, category=api
+Output: tests/generated/api/orders_post_201.py
+Execution: returncode=0, 1 passed in 0.42s
+```
+
+**Action:** Review the generated file, promote to
+`tests/api/orders_post_201.py`, drop the timestamped header.
+
+### Example: Ambiguous prompt — clarification first
+
+**Prompt:** `Test the new orders feature`
+
+**Action:** Do NOT invoke the pipeline yet. Ask: "Is this an end-to-end UI test
+of the checkout flow, an API contract test for `/v1/orders`, or a unit test of
+the order-validation function?" Only after the user picks should you run
+`generate`.
+
+### Example: Performance test with thresholds
+
+**Prompt:** `Load test /v1/search at 200 RPS for 5 minutes, p95 latency < 300ms`
+
+**Pipeline trace:**
+
+```text
+Classification: test_type=performance, confidence=0.95
+Recommendation: framework=k6, ext=.js, category=performance
+Output: tests/generated/performance/search_load.js
+```
+
+**Validation:** Run against a staging environment, not prod. Compare p95 to the
+threshold; if the test passes locally but the threshold was unrealistic, surface
+that to the requester before promoting.
+
+## Escalation
+
+- **When the registry has no entry for the classified `test_type`:** Stop and
+  file a registry update. Do not invent a framework name.
+- **When the generated test repeatedly fails to parse:** This usually means the
+  prompt is under-specified. Narrow it to a single behavior and regenerate
+  before assuming a bug.
+- **When execution requires creds you don't have:** Surface the missing-cred
+  error to the user; never embed dummy creds in a generated test to make it
+  "run".
+- **When the classifier's confidence is below 0.7:** Treat as a clarification
+  trigger, not a generation trigger. Loop back to Phase 1. Note: `confidence` is
+  a hand-calibrated heuristic prior per keyword-match branch in
+  `ts/src/core/classifier.ts`, not a statistically calibrated probability — 0.7
+  is a coarse "does this branch's signal look weak" cutoff, not P(correct
+  classification) ≥ 0.7. Don't read finer distinctions (e.g. 0.85 vs. 0.95) as
+  meaningfully different confidence levels.

package/agents/skills/claude-code/canary-instrument/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: canary-instrument
+description:
+  Instrument a Playwright run with OpenTelemetry and emit a run.json artifact
+  correlating every test to the outbound HTTP requests it made — "which test
+  made which request?" — with zero manual bookkeeping in test code. Trace-only
+  v1 contract, additive-safe for future pytest/k6/node producers. Self-contained
+  (bundles its own run-type factories and span reader).
+cli: scripts/cli.mjs
+requires: [node>=20]
+---
+
+# Canary Instrument
+
+Correlate every outbound HTTP request in a Playwright run to the test that made
+it, using OTel span parent/child relationships. Zero required external
+dependencies to produce output — default file-based span export, no OTel
+collector needed.
+
+## Setup (two manual steps, once per suite)
+
+This skill ships fixture _files_ you wire into your own suite — it does not
+vendor OTel as a dependency of the `canary` package itself. Install these first:
+
+```bash
+npm install --save-dev \
+  @opentelemetry/sdk-node @opentelemetry/api \
+  @opentelemetry/auto-instrumentations-node \
+  @opentelemetry/exporter-trace-otlp-http
+```
+
+**1. Bootstrap the OTel SDK before Playwright starts** — add `NODE_OPTIONS` to
+your test command (or `playwright.config.ts`'s `webServer`/CI step):
+
+```bash
+NODE_OPTIONS="--import ./node_modules/canary/agents/skills/claude-code/canary-instrument/scripts/otel_bootstrap/instrument.mjs" \
+  npx playwright test
+```
+
+Copy `otel_bootstrap/instrument.mjs` into your repo (e.g. `otel/instrument.mjs`)
+if you'd rather not reference the path inside `node_modules`.
+
+**2. Merge the root-span fixture into your `fixtures.ts`:**
+
+```ts
+import { test as base } from '@playwright/test';
+import { withTestSpan } from './otel_bootstrap/playwright-fixture';
+
+export const test = withTestSpan(base);
+```
+
+Every test using this `test` export now opens a root span carrying
+`test.id`/`test.title`/`test.file`, and every HTTP call the test makes nests
+under it automatically — no manual span code in individual tests.
+
+## Invocation
+
+```bash
+canary skills run canary-instrument -- \
+  --spans test-results/trace --output test-results \
+  [--suite-type e2e_ui]
+
+# Usage and options (exits 0; --spans/--output are not required for --help):
+canary skills run canary-instrument -- --help
+```
+
+An unknown flag, a missing required flag, and a value-flag left without its
+value are all usage errors (exit 2). `--help` short-circuits ahead of the
+required-argument check, so it never complains about `--spans`/`--output`.
+
+Writes `test-results/run.json`. Creates `--output` if it doesn't exist.
+Missing/empty `--spans` produces `trace: {spans_total: 0, by_test: []}`, not a
+failure. `--suite-type` is a free-form string (no enum) — pass whatever label
+describes your suite.
+
+## `run.json` v1 contract (trace-only)
+
+```jsonc
+{
+  "schema_version": 1,
+  "suite_type": "",
+  "generated_at": "2026-07-15T18:00:00+00:00",
+  "trace": {
+    "spans_total": 124,
+    "by_test": [
+      {
+        "test_id": "users-spec:1", // "__setup__" for orphan traffic
+        "test_title": "lists users",
+        "test_file": "tests/users.spec.ts",
+        "trace_id": "abc123...",
+        "outcome": "passed",
+        "requests": [
+          {
+            "method": "GET",
+            "url": "http://localhost:3000/users/1",
+            "route": "/users/:id",
+            "status": 200,
+            "duration_ms": 12.4,
+            "span_id": "def456...",
+            "started_at": "2026-07-15T18:00:01+00:00",
+          },
+        ],
+      },
+    ],
+  },
+}
+```
+
+No `coverage` key, no `canary_run_id` key — cut for v1 (see
+`docs/knowledge/decisions/0006-otel-test-side-tracing.md` and
+`docs/changes/canary-instrument/proposal.md`). Additive-only evolution: new
+optional fields may appear later; existing fields never change meaning.
+
+## Sending spans to a collector (optional)
+
+Set `OTEL_EXPORTER_OTLP_ENDPOINT` before the test run and spans are
+_additionally_ streamed there — the file exporter still writes
+`test-results/trace/otel-spans.*.jsonl` either way, so `canary-instrument`'s own
+correlation is never dependent on a collector being up. If your org's endpoint
+is recorded in company-knowledge, export it first:
+
+```bash
+export OTEL_EXPORTER_OTLP_ENDPOINT="$(canary company-knowledge show --json | jq -r '.otel_exporter_endpoint')"
+```
+
+See `docs/guides/company-knowledge.md` for the `otel_exporter_endpoint` field.
+
+## CI wiring (GitHub Actions)
+
+```yaml
+- name: Run Playwright (instrumented)
+  env:
+    NODE_OPTIONS: '--import ./otel/instrument.mjs'
+  run:
+    npx playwright test --reporter=json --output-file=test-results/results.json
+
+- name: Correlate tests to HTTP spans
+  if: always()
+  run: |
+    canary skills run canary-instrument -- \
+      --spans test-results/trace --output test-results
+
+- name: Upload run.json
+  if: always()
+  uses: actions/upload-artifact@v4
+  with:
+    name: run-trace
+    path: test-results/run.json
+```
+
+## Related skills
+
+- `canary-test-reporter` — Markdown/JSON test summary; `run.json`'s `by_test[]`
+  rows are structurally similar to its `TestResult` shape (title/status as join
+  keys) — a future consumer can read both artifacts with one join key.
+- `canary-fail-fast` — aborts a broken run early; use alongside this skill for
+  complete CI coverage.