canary-test-cli 7.0.0 → 7.2.0

package/agents/skills/claude-code/canary-company-knowledge/SKILL.md

@@ -0,0 +1,196 @@
+---
+name: canary-company-knowledge
+description: >
+  Scaffold .canary/company.json — the org-specific pointer file (Confluence
+  spaces, Jira projects, internal docs/domains, MCP servers, dashboard, and the
+  user-catalog skill) that canary-ci-ready and canary-failure-impact silently
+  assume already exists. Use when the user says "set up company knowledge",
+  "bootstrap company.json", "no company.json", "canary company-knowledge init",
+  or when canary-ci-ready/canary-failure-impact report no user-catalog config
+  for auth/config failures. Scaffolds the file and prompts for the fields that
+  genuinely cannot be inferred — it does not claim full automation.
+---
+
+# Canary: Company Knowledge Init
+
+Wraps `canary company-knowledge init` to scaffold `.canary/company.json` — the
+_pointers-only_ file that lets Canary skills reach into your org's internal
+tooling (Confluence, Jira, internal docs, MCP servers, a user catalog) without
+ever committing proprietary content to this open-core repo.
+
+This file is silently assumed to exist by
+[`canary-ci-ready`](../canary-ci-ready/SKILL.md) (user-catalog investigation for
+auth/config failures) and
+[`canary-failure-impact`](../canary-failure-impact/SKILL.md) (same). If neither
+has ever been run with a populated `company.json`, both degrade to a generic
+"check your user catalog if you have one" message. This skill closes that gap.
+
+## When to Use
+
+- First-time setup for a new project or team — "we need to configure company
+  knowledge for canary"
+- `canary-ci-ready` or `canary-failure-impact` fell back to the generic
+  user-catalog message and the user wants real investigation instead
+- `canary company-knowledge show` reports "No company knowledge configured"
+- Re-running setup to add a field that was skipped the first time (safe — `init`
+  merges onto existing values by default)
+- NOT for storing secrets, API keys, or tokens — this file holds pointers only;
+  the CLI's secret heuristic (`_looks_like_secret`) rejects
+  `sk-`/`token`/`bearer`-shaped values and anything over 128 chars outside
+  `notes`
+- NOT for company-specific proprietary content (client names, internal runbooks,
+  populated data) — per this repo's open-core boundary (`AGENTS.md`), that lives
+  only in a private overlay under `.canary/skills/` or the org's own tooling,
+  reached _via_ the pointers this file stores
+
+## What This Skill Cannot Automate
+
+Every field in `.canary/company.json` is org-specific and cannot be reliably
+inferred from the codebase alone — this repo is public/open-core by design, so
+nothing in it names a real company, Jira project, or internal host. Don't claim
+otherwise. What this skill _can_ do:
+
+- Scaffold the file and `.gitignore` entry so it exists and is never
+  accidentally committed
+- Prompt for each field with the exact validation format the CLI expects, so the
+  user isn't guessing at schema
+- Detect when a value looks like a secret and refuse it before it reaches disk
+- Merge onto existing values, so re-running is always safe
+
+What it cannot do: know your Confluence space key, your Jira project prefix, or
+which MCP server your org runs. Those come from the user.
+
+## Process
+
+### Phase 1: CHECK — Does It Already Exist?
+
+```bash
+canary company-knowledge show
+```
+
+- **"No company knowledge configured"** → proceed to Phase 2 (fresh scaffold).
+- **Populated output** → note which fields are already set (shown with
+  `sources:` — `~/.canary/company.json`, `.canary/company.json`, or an env
+  layer). Proceed to Phase 2 in merge mode (no `--force`) so existing values
+  survive as defaults.
+- **`⚠` error line** (secret-like value or malformed JSON in an existing layer)
+  → surface the exact warning to the user before continuing; that layer is being
+  skipped entirely until fixed.
+
+### Phase 2: SCAFFOLD — Run the Interactive Init
+
+```bash
+canary company-knowledge init
+```
+
+Walk the user through each prompt. For each field, explain what it's for and
+give the expected shape before they answer — the CLI accepts blank input to skip
+or keep the current value:
+
+| Field                | Format                                                     | Purpose                                        |
+| -------------------- | ---------------------------------------------------------- | ---------------------------------------------- |
+| `confluence_spaces`  | comma-separated, uppercase (`QA, ENG`)                     | spaces the LLM should consult for org docs     |
+| `jira_projects`      | comma-separated, uppercase (`PROJ, OPS`)                   | projects for ticket cross-referencing          |
+| `internal_doc_urls`  | one URL per line, `http(s)://` only                        | specific reference docs to fetch via MCP       |
+| `internal_domains`   | comma-separated hostnames (`corp.example.com`)             | flags internal-only URLs in generated content  |
+| `mcp_servers`        | comma-separated identifiers (`plugin_atlassian_atlassian`) | which configured MCP server(s) back the above  |
+| `claude_code_skills` | comma-separated slugs (`team:skill-name`)                  | project-overlay skills to surface as available |
+| `notes`              | free text, ≤2048 chars, no secrets                         | anything else the LLM should know              |
+
+Fields not prompted by `init` (`dashboard_url`, `dashboard_token_env`,
+`otel_exporter_endpoint`) can be added by hand-editing the JSON afterward —
+mention this if the user needs dashboard/OTel wiring; don't skip it silently.
+
+If the user offers a value that looks like a credential (starts with `sk-`,
+`token`, `bearer`, or is unusually long), stop and remind them: secrets go in
+environment variables, never in `company.json`. The CLI will reject the whole
+layer if one slips through — better to catch it before submitting.
+
+### Phase 3: WIRE THE USER-CATALOG SKILL (Known Schema Gap)
+
+`canary-ci-ready` and `canary-failure-impact` both read a `user_catalog_skill`
+key directly from `.canary/company.json` to investigate auth/config failures.
+**This key is not part of the CLI's validated schema** — it isn't in
+`CompanyKnowledge`'s known fields, `company-knowledge init` never prompts for
+it, and `company-knowledge show` will list it under "ignored unknown field" if
+you inspect warnings. It still works for the two consuming skills because they
+read the raw JSON file directly rather than going through the Python loader —
+but be transparent with the user that this is an informal extension, not a
+first-class field, until the schema catches up.
+
+To wire it:
+
+1. Ask the user which project-overlay skill (if any) looks up test users — e.g.
+   `team:user-lookup`. If they don't have one, skip this phase; the consuming
+   skills degrade gracefully to a generic prompt.
+2. If they gave a normal skill slug, prefer adding it to `claude_code_skills` (a
+   real, validated field) _and_ separately hand-edit `.canary/company.json` to
+   add the literal key:
+
+   ```json
+   {
+     "claude_code_skills": ["team:user-lookup"],
+     "user_catalog_skill": "team:user-lookup"
+   }
+   ```
+
+3. Confirm the value matches what `canary skills run <user_catalog_skill>`
+   expects as an identifier (same slug format as `claude_code_skills`:
+   `^[a-z0-9][a-z0-9_-]*(:[a-z0-9][a-z0-9_-]*)?$`).
+
+### Phase 4: VERIFY
+
+```bash
+canary company-knowledge show
+```
+
+- Confirm every field the user just set appears in the printed output.
+- Confirm no `⚠` warnings (unknown fields other than the intentional
+  `user_catalog_skill` extension, dropped invalid entries, secret detections).
+- Confirm `.gitignore` now contains a `.canary/` line — `init` adds it
+  automatically, but verify if the project already had a `.gitignore` with
+  unusual formatting.
+- If JSON output is useful for the user's own tooling: `--json`.
+
+## Error Handling
+
+| Situation                                           | What Happens                                         | What To Do                                                                                                                             |
+| --------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
+| `.canary/company.json` already exists, no `--force` | `init` shows existing values as defaults and merges  | Normal — just re-run `init`                                                                                                            |
+| Secret-like value entered                           | That whole layer is dropped; `show` prints a red `✗` | Remove the value, use an env var, re-run                                                                                               |
+| Malformed JSON in an existing file                  | Layer skipped with a parse error                     | Fix the JSON by hand, then re-run `show` to confirm                                                                                    |
+| Invalid entry format (e.g. lowercase Jira key)      | Entry silently dropped, warning logged               | Re-enter in the correct case/format                                                                                                    |
+| Unknown field in the file                           | Warned, not fatal, field ignored by the loader       | Expected for `user_catalog_skill` today (Phase 3) — otherwise likely a typo                                                            |
+| User has no Confluence/Jira/MCP setup at all        | Every field is legitimately empty                    | Skip `init` entirely; `canary-ci-ready`/`canary-failure-impact` degrade to their generic prompts, which is correct behavior, not a bug |
+
+## Examples
+
+### Example: First-time team lead setup
+
+**Prompt:** "Set up company knowledge for canary on this repo."
+
+**Action:** Run `company-knowledge show` → "No company knowledge configured."
+Run `company-knowledge init`. Walk through each field; the team lead has a
+Confluence space (`QA`), a Jira project (`OPS`), and an Atlassian MCP server
+(`plugin_atlassian_atlassian`) but no internal dashboard yet. Leave
+`dashboard_url` unset. Ask about a user-catalog skill — they don't have one, so
+Phase 3 is skipped. Verify with `show`.
+
+### Example: Retrofitting `user_catalog_skill` onto an existing file
+
+**Prompt:** "`canary-ci-ready` keeps telling me to check my user catalog
+manually, but we have a skill for that."
+
+**Action:** Run `show` — confirm `.canary/company.json` already has
+`confluence_spaces`/`jira_projects` set from a prior run. Ask for the skill slug
+(`team:test-user-lookup`). Add it to both `claude_code_skills` and the raw
+`user_catalog_skill` key per Phase 3. Re-run `show` to confirm no new warnings,
+then re-run `canary-ci-ready` on a known failing auth test to confirm the lookup
+now fires.
+
+## Related Skills
+
+- [`canary-ci-ready`](../canary-ci-ready/SKILL.md) — consumes
+  `user_catalog_skill` for auth/config failure investigation
+- [`canary-failure-impact`](../canary-failure-impact/SKILL.md) — same
+  investigation pattern, different trigger context

package/agents/skills/claude-code/canary-critical-areas/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: canary-critical-areas
+description: >
+  Risk-based test prioritisation. Given a codebase or diff, identifies which
+  areas carry the most risk using git churn, downstream dependents,
+  business-critical signals, and existing coverage depth. Produces a ranked list
+  with recommended test types per area.
+---
+
+# Canary: Critical Areas
+
+Identifies the highest-risk areas of a codebase so test effort goes where it
+matters most. Uses multiple signals, degrades gracefully when advanced tooling
+is unavailable.
+
+## When to Use
+
+- Before writing new tests: "where should I focus?"
+
+- After a large diff lands: "what did this change put at risk?"
+
+- As Phase 1 of `/canary-test-pipeline`
+
+- When asked to prioritise test coverage
+
+## Signals
+
+Collect all available signals, score each area, and rank by composite risk
+score.
+
+### 1. Churn / hotspot (always available)
+
+**With harness MCP available:** call `detect_anomalies` (metric `hotspotScore`,
+plus its co-change / single-point-of-failure signals). Harness's hotspot score
+already blends churn with structural risk, so use it directly as this signal and
+skip the raw `git log` pass. Normalise the returned scores to 0–1.
+
+**Fallback (no MCP):**
+
+```bash
+git log --stat --since="90 days ago" -- <path> | grep -c "^"
+```
+
+Files changed most frequently in the last 90 days score higher. Normalise to 0–1
+across all files in scope.
+
+### 2. Downstream dependents
+
+**With harness MCP available:** call `get_impact` for each candidate file
+(`filePath`, `mode: "summary"`) and read the affected-node counts it returns
+(tests / docs / code grouped by type). This is the purpose-built impact
+primitive — do **not** hand-walk `get_relationships` edge-by-edge; `get_impact`
+already computes the transitive downstream set. More affected nodes ⇒ higher
+score.
+
+**Fallback (no MCP):** scan for `import` statements referencing each file using
+`grep -r`. Count unique files that import each candidate.
+
+Files with more inbound dependents score higher — a change here breaks more.
+
+### 3. Business-critical / critical-path flags
+
+**With harness MCP available:** call `get_critical_paths` and add a fixed boost
+(+0.3) to any area whose functions appear in the returned perf-critical set.
+Also query `ask_graph` for `business_fact` nodes associated with each area; any
+business-critical annotation adds the same +0.3 boost (apply the boost once,
+whichever signal fires).
+
+**Fallback:** skip this signal silently (do not penalise the score).
+
+### 4. Coverage depth boost
+
+If `.canary/test-inventory.json` is present: files whose endpoints are at depth
+0 or 1 receive a boost (+0.15) — low depth in a high-churn file is especially
+risky.
+
+## Risk Score
+
+```text
+risk_score = (churn * 0.35) + (dependents * 0.35) + (business_critical * 0.30) + depth_boost
+```
+
+Capped at 1.0. Round to 2 decimal places.
+
+## Output
+
+```text
+Critical areas — <repo> (<N> files analysed)
+
+  1. src/loyalty/points.service.ts       risk 0.92  ████████████
+     signals: high churn · 12 dependents · business_critical
+     recommended: api + integration tests
+
+  2. src/billing/charge.service.ts       risk 0.78  ██████████
+     signals: high churn · billing domain
+     recommended: api tests · /canary-failure-impact suggested
+
+  ...
+```
+
+Show at most 10 areas. If more than 10 qualify, note the total and offer to show
+all.
+
+## Optional Artifact
+
+When `--save` flag is passed (or when invoked by `/canary-test-pipeline`), write
+`.canary/critical-areas.json`:
+
+```json
+{
+  "generated": "<ISO timestamp>",
+  "areas": [
+    {
+      "path": "src/loyalty/points.service.ts",
+      "risk_score": 0.92,
+      "signals": ["high_churn", "many_dependents", "business_critical"],
+      "recommended_test_types": ["api", "integration"],
+      "summary": "High-churn service with 12 downstream dependents"
+    }
+  ]
+}
+```
+
+This file is consumed as opt-in context by `/canary-edge-cases` and
+`/canary-failure-impact`.
+
+## Flags
+
+- `--diff <git ref>` — scope analysis to files changed in a diff
+  (`git diff <ref>...HEAD`)
+
+- `--save` — write `critical-areas.json`
+
+## Related skills
+
+- `/canary-ci-ready` — check 4 consumes `critical-areas.json`
+
+- `/canary-edge-cases` — focuses edge cases on critical areas when JSON present
+
+- `/canary-failure-impact` — focuses tracing on critical paths when JSON present
+
+- `/canary-test-pipeline` — Phase 1

package/agents/skills/claude-code/canary-critical-areas/skill.yaml

@@ -0,0 +1,16 @@
+name: canary-critical-areas
+version: '1.0.0'
+description:
+  Risk-rank a codebase or diff by git churn, downstream dependents,
+  business-critical signals, and coverage depth; outputs a ranked list with
+  recommended test types per area.
+stability: static
+triggers:
+  - manual
+platforms:
+  - claude-code
+type: rigid
+tools: []
+tier: 1
+depends_on:
+  - canary-ci-ready

package/agents/skills/claude-code/canary-edge-case-discovery/SKILL.md

@@ -0,0 +1,160 @@
+---
+name: canary-edge-case-discovery
+description: >
+  Given a feature description, function signature, or existing test suite,
+  surfaces edge cases worth testing across six categories. Explanation depth
+  scales to user skill level. Optionally focuses on critical areas when
+  critical-areas.json is present.
+---
+
+# Canary: Edge Case Discovery
+
+Surfaces the edge cases that tests typically miss: the inputs and conditions
+that work in demos but break in production.
+
+## When to Use
+
+- After writing happy-path tests: "what else should I test?"
+
+- When reviewing a feature for robustness
+
+- As Phase 2 of `/canary-test-pipeline`
+
+- When asked "what edge cases should I cover?"
+
+## Input
+
+Provide one of:
+
+- A feature description: `"points accrual on tier upgrade"`
+
+- A function signature:
+  `accruePoints(memberId: string, amount: number): Promise<Result>`
+
+- A test file path: `tests/loyalty/points.spec.ts`
+
+- Or nothing — Canary will infer from open files and recent context
+
+If `.canary/critical-areas.json` is present, focus edge case discovery on the
+highest-risk areas first (rank_score ≥ 0.6).
+
+## The Six Categories
+
+For each category, generate specific, actionable cases — not generic advice.
+
+### 1. Boundary values
+
+Zero, negative, max integer, empty string, null, undefined, one-off-by-one. For
+amounts: 0, 1, MAX_SAFE_INTEGER, -1, 0.001 (floating point). For strings: empty
+string, whitespace-only, max-length + 1 character.
+
+### 2. Race conditions
+
+Concurrent writes to the same resource. Double-submit (user clicks twice). Stale
+reads after an update. Lock contention. Out-of-order async responses.
+
+### 3. Locale and timezone
+
+DST transition times. Dates at midnight UTC vs local time. Non-ASCII characters
+in names and addresses. RTL text in string fields. Locale-specific number
+formats (1.000,00 vs 1,000.00). Emoji in text fields.
+
+### 4. Partial network
+
+Request timeout mid-flight. Dropped connection after partial response. Retry
+storms (client retries while server is still processing). Response truncation.
+
+### 5. Unexpected input shapes
+
+Extra fields the schema doesn't expect. Missing required fields. Wrong types
+(string where number expected). SQL or script injection strings. Deeply nested
+objects. Arrays where scalars expected.
+
+### 6. Accessibility
+
+Keyboard-only navigation paths. Missing ARIA labels. Focus trap conditions.
+Screen reader text for dynamic content. Colour contrast for status indicators.
+_(Only include if the input is a UI feature or test.)_
+
+## Output Depth
+
+**Consult the resolved persona; do not infer one here.** The engine owns that
+decision (issue #462) so every skill adapts the same way and a downstream
+overlay has one place to override.
+
+Resolve it in this order:
+
+1. `--level <id>` if the caller passed one. It is the explicit override and
+   always wins.
+2. The `persona` block on a `canary__analyze_file` response. It carries `id`,
+   `depth`, `formats`, `reasoning`, plus `source`, `reason`, and `signals` — the
+   evidence behind the choice.
+3. Neither available → use `junior`, the engine's fallback. Erring explanatory
+   is deliberate: over-explaining is a mild annoyance, under-explaining silently
+   fails a manual tester.
+
+Render from the persona's `depth`, not from its `id` — an overlay may add
+personas this table never listed:
+
+| `depth`  | Output style                        |
+| -------- | ----------------------------------- |
+| `terse`  | Bullet list of cases only           |
+| `brief`  | Cases + one-line _why this matters_ |
+| `guided` | Cases + numbered reproduction steps |
+
+When `reasoning` is true, say which persona shaped the output and why —
+`persona.reason` is written to be quoted.
+
+Read `source` before trusting the persona, and treat the three values
+differently:
+
+- `explicit` — the reader chose this. Do not second-guess it.
+- `detected` — inferred from at least two independent signals about the project.
+  Reliable enough to act on, still a guess; mention it is overridable with
+  `CANARY_PERSONA` or `--level` if the depth seems wrong for them.
+- `fallback` — the engine **declined** to guess, which is the common case. That
+  is not a failure and not a reason to go hunting for signals of your own: it
+  means the evidence did not clear the floor, and re-deriving an audience here
+  is exactly the hand-rolling this section replaced. Use the persona as given.
+
+**Never infer `terse` on your own.** Stripping explanation from a reader who
+turns out to be a manual tester is the expensive direction of this mistake;
+`fallback` is deliberately explanatory for that reason. Note that `manual` is
+not inferable from a file analysis at all — it is reached by an explicit choice
+— so a `fallback` persona may well be a manual tester.
+
+The shipped personas are `sdet` (terse), `junior` (brief), and `manual`
+(guided); the definitions live in `ts/src/data/personas/registry.json`.
+
+## Output Format (`sdet` example)
+
+```text
+Edge cases — points accrual on tier upgrade
+
+  Boundary values
+  · amount = 0
+  · amount = MAX_SAFE_INTEGER
+  · memberId empty string
+  · memberId with special characters
+
+  Race conditions
+  · concurrent accrual calls for the same memberId
+  · double-submit within 100ms
+
+  ...
+
+  Suggested next: /canary-write-test "add edge case tests for accruePoints boundary values"
+```
+
+## Flags
+
+- `--level sdet|junior|manual` — output depth. Overrides the resolved persona;
+  the default is the persona the engine resolved (see **Output Depth**).
+
+## Related skills
+
+- `/canary-critical-areas` — produces `critical-areas.json` used for focus
+
+- `/canary-write-test` — generates tests for the surfaced cases
+
+- `/canary-test-pipeline` — Phase 2

package/agents/skills/claude-code/canary-edge-case-discovery/skill.yaml

@@ -0,0 +1,16 @@
+name: canary-edge-case-discovery
+version: '1.0.0'
+description:
+  Surface edge cases worth testing across six categories from a feature
+  description, function signature, or existing suite; explanation depth scales
+  to user skill level.
+stability: static
+triggers:
+  - manual
+platforms:
+  - claude-code
+type: rigid
+tools: []
+tier: 1
+depends_on:
+  - canary-ci-ready

package/agents/skills/claude-code/canary-fail-fast/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: canary-fail-fast
+description:
+  Surface test failures fast and loud — audit a Playwright config for fail-fast
+  knobs (maxFailures/forbidOnly/retries) and print a loud, categorized failure
+  digest to the CI log + ::error annotations at run end, failing the step so the
+  signal can't be missed. Self-contained (bundles its own Playwright JSON parser
+  and failure categorizer).
+cli: scripts/cli.mjs
+requires: [node>=20]
+---
+
+# Canary Fail-Fast
+
+Make test failures **fast** (abort a broken run early) and **loud** (surface in
+the CI log + Checks, not a file). Two halves:
+
+1. **Fail-fast config audit** — flags the absence of `maxFailures`,
+   `forbidOnly`, and `retries` in a `playwright.config.*`.
+2. **Loud run-end digest** — a terse, categorized failure summary to stdout +
+   `::error` annotations, with a non-zero exit.
+
+Self-contained: it bundles a minimal Playwright JSON parser and the failure
+categorizer, so it has no dependency on any other skill.
+
+## Fail-fast config (paste into `playwright.config.ts`)
+
+```ts
+export default defineConfig({
+  // Fail fast in CI: abort once enough has clearly broken, never on local runs.
+  forbidOnly: !!process.env.CI, // a stray test.only fails the build
+  maxFailures: process.env.CI ? 10 : 0, // stop the run after 10 failures in CI
+  retries: process.env.CI ? 2 : 0, // absorb flakes in CI; surface them locally
+  // ...your existing config
+});
+```
+
+## Invocation
+
+```bash
+# Loud failure digest from a Playwright JSON run (exits non-zero on failures):
+canary skills run canary-fail-fast -- --results test-results/results.json
+
+# Audit the fail-fast config:
+canary skills run canary-fail-fast -- --config playwright.config.ts
+
+# Both at once:
+canary skills run canary-fail-fast -- \
+  --results test-results/results.json \
+  --config playwright.config.ts
+
+# Usage and the full flag list (exits 0):
+canary skills run canary-fail-fast -- --help
+```
+
+At least one of `--results` / `--config` is required. The digest exits `1` when
+any test failed (so the CI step fails); the config audit alone never fails the
+build.
+
+## CI wiring (GitHub Actions)
+
+Run after the Playwright step with `if: always()` so the digest surfaces even
+when the test step already failed:
+
+```yaml
+- name: Run Playwright
+  run:
+    npx playwright test --reporter=json --output-file=test-results/results.json
+
+- name: Fail-fast digest
+  if: always()
+  run: |
+    canary skills run canary-fail-fast -- \
+      --results test-results/results.json
+```