canary-test-cli 7.1.0 → 8.0.0

package/agents/skills/claude-code/canary-ci-ready/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: canary-ci-ready
+description: >
+  Analyses a test suite for CI readiness: coverage depth, flakiness, assertion
+  quality, critical path coverage, and suite runtime. Accepts documented
+  failures (quarantined tests with linked open issues count as verified).
+  Investigates config/auth failures using the consuming repo's declared
+  user_catalog_skill.
+---
+
+# Canary: CI Ready
+
+Analyses a test suite across five dimensions and produces a readiness score. Use
+this before promoting a suite to CI, or as the convergence gate in
+`/canary-test-pipeline`.
+
+**Deterministic scorer:** run `canary ci-ready [--root <dir>] [--json]` first.
+It scores every check that has a real input and reports `skip`, naming the
+missing input, for every check that does not. A skip is never a pass. Today only
+flakiness has a producer behind it, so expect the other four to skip until
+their inputs exist. The verdict is `ready` (all five passed), `incomplete`
+(nothing failed, something skipped), `not-ready` (exit 1) or `abstained`
+(nothing scored, exit 3).
+
+## When to Use
+
+- Before wiring a new test suite into CI for the first time
+
+- When a CI run is failing and you need to understand why
+
+- As part of `/canary-test-pipeline` (Phase 0 and convergence gate)
+
+- When asked "is this suite ready for CI?"
+
+## The Five Checks
+
+Run all five checks and score each pass / warn / fail.
+
+### 1. Coverage depth
+
+Read `.canary/test-inventory.json` if present. Nothing in canary produces this
+file yet (there is no `canary coverage` command), so when it is absent this
+check is a `skip`, not a pass or a fail.
+
+Default threshold: depth ≥ 2 for all endpoints in critical areas. Override with
+`--threshold <n>`.
+
+- **pass** — all critical-area endpoints at depth ≥ threshold
+
+- **warn** — some endpoints at depth 1 (hit but unasserted)
+
+- **fail** — any critical-area endpoint at depth 0
+
+### 2. Flakiness
+
+Read `test-results/quarantine-ledger.json` (or the path in
+`.canary/company.json` under `quarantine_ledger_path` if set).
+
+No tool writes a quarantine ledger yet, so `canary ci-ready` scores flakiness
+from the run-history store instead: the last 30 runs in
+`test-results/reports/history-v2.jsonl`. Any test flaking in 10% or more of its
+runs fails the check, any lower flake rate warns, and no flakes passes.
+
+A quarantined test is acceptable only when it has a linked open issue (Jira or
+GitHub). Check issue state:
+
+- Linked issue **open** → counts as verified (documented, tracked)
+
+- Linked issue **closed** → flag: quarantine should be resolved
+
+- **No linked issue** → fail: unlinked quarantine blocks CI-ready
+
+### 3. Assertion quality
+
+Read depth scores from the inventory. In critical-area endpoints:
+
+- **pass** — all tests at depth ≥ 2 (shaped assertions: result.ok or equivalent)
+
+- **warn** — some tests at depth 1 (status-only assertions)
+
+- **fail** — majority of critical-path tests at depth 1
+
+### 4. Critical path coverage
+
+Only run this check if `.canary/critical-areas.json` is present.
+
+Cross-reference the top 5 risk-scored areas from `critical-areas.json` against
+`test-inventory.json`:
+
+- **pass** — all top-5 areas have at least one test at depth ≥ 1
+
+- **warn** — one area uncovered
+
+- **fail** — two or more top areas uncovered
+
+- **skip** — `critical-areas.json` absent (note this in output, not a failure)
+
+### 5. Suite runtime
+
+Run history lives in `test-results/reports/history-v2.jsonl`. The store does not
+record run or test durations today, so there is no p95 to compute and
+`canary ci-ready` reports this check as `skip`. The scoring below applies once
+durations are recorded.
+
+**With harness MCP available:** score the p95 against trend history rather than
+an absolute clock. Call `get_perf_baselines` and compare this run's p95 to the
+recorded baseline for the suite:
+
+- **pass** — p95 within the baseline's tolerance, or an improvement
+- **warn** — p95 regressed past tolerance but under 2× the baseline
+- **fail** — p95 at or over 2× the baseline
+- **skip** — no baseline recorded yet (this is a baseline-capture run, not a
+  failure — say so in the output)
+
+After scoring, record the run back into the baseline with
+`update_perf_baselines` so the trend keeps moving. Do **not** record a run that
+failed for unrelated reasons — a broken run's runtime is not a data point.
+
+Regression beats absolute here. A suite that has always taken 11 minutes is a
+fact of life; a suite that went from 3 minutes to 11 this week is the actual
+signal, and an absolute threshold cannot tell those apart — it fails the first
+forever and stays silent on the second until it crosses the line.
+
+**Fallback (no MCP):** judge against absolute thresholds.
+
+- **pass** — p95 under the configured timeout (default: 5 minutes)
+- **warn** — p95 between 5–10 minutes
+- **fail** — p95 over 10 minutes, or no run history (cannot assess)
+
+State which mode was used in the output — "runtime vs. baseline" or "runtime vs.
+absolute threshold" — so a reader knows whether a pass means "no regression" or
+merely "under the clock".
+
+## User Catalog Investigation
+
+When a test fails with an auth, permission, or configuration error:
+
+1. Read `user_catalog_skill` from `.canary/company.json`
+2. If present: invoke `canary skills run <user_catalog_skill>` with the required
+   attributes from the error context; surface any matching user as a suggestion
+3. If absent, or no matching user found: present constructively —
+
+   > "This failure may be a test user or test data configuration issue. Check
+   > your user catalog if you have one, or set up the required test data before
+   > re-running."
+
+Never reference a specific catalog skill by name in output.
+
+## Output Format
+
+```text
+CI Readiness — <repo-name>
+
+  ✓ / ⚠ / ✗   <check name>   <brief finding>
+  ...
+
+  Score: N/5 — CI-READY  or  NOT CI-READY
+
+  Runtime scored vs. baseline  |  vs. absolute threshold
+
+  <gap list with suggested next actions>
+```
+
+Score of 5/5 = CI-READY. Any fail = NOT CI-READY. Warns do not block.
+
+## Flags
+
+- `--threshold <n>` — minimum depth for coverage check (default: 2)
+
+## Related skills
+
+- `/canary-test-pipeline` — orchestrates this skill as Phase 0 and convergence
+  gate
+
+- `/canary-critical-areas` — produces `critical-areas.json` used by check 4
+
+- `canary-unquarantine` (overlay) — resolves quarantined tests once bugs are
+  fixed

package/agents/skills/claude-code/canary-ci-ready/skill.yaml

@@ -0,0 +1,14 @@
+name: canary-ci-ready
+version: '1.0.0'
+description:
+  Analyse a test suite for CI readiness across coverage depth, flakiness,
+  assertion quality, critical-path coverage, and runtime; documented/quarantined
+  failures count as verified.
+stability: static
+triggers:
+  - manual
+platforms:
+  - claude-code
+type: rigid
+tools: []
+tier: 1

package/agents/skills/claude-code/canary-company-knowledge/SKILL.md

@@ -0,0 +1,196 @@
+---
+name: canary-company-knowledge
+description: >
+  Scaffold .canary/company.json — the org-specific pointer file (Confluence
+  spaces, Jira projects, internal docs/domains, MCP servers, dashboard, and the
+  user-catalog skill) that canary-ci-ready and canary-failure-impact silently
+  assume already exists. Use when the user says "set up company knowledge",
+  "bootstrap company.json", "no company.json", "canary company-knowledge init",
+  or when canary-ci-ready/canary-failure-impact report no user-catalog config
+  for auth/config failures. Scaffolds the file and prompts for the fields that
+  genuinely cannot be inferred — it does not claim full automation.
+---
+
+# Canary: Company Knowledge Init
+
+Wraps `canary company-knowledge init` to scaffold `.canary/company.json` — the
+_pointers-only_ file that lets Canary skills reach into your org's internal
+tooling (Confluence, Jira, internal docs, MCP servers, a user catalog) without
+ever committing proprietary content to this open-core repo.
+
+This file is silently assumed to exist by
+[`canary-ci-ready`](../canary-ci-ready/SKILL.md) (user-catalog investigation for
+auth/config failures) and
+[`canary-failure-impact`](../canary-failure-impact/SKILL.md) (same). If neither
+has ever been run with a populated `company.json`, both degrade to a generic
+"check your user catalog if you have one" message. This skill closes that gap.
+
+## When to Use
+
+- First-time setup for a new project or team — "we need to configure company
+  knowledge for canary"
+- `canary-ci-ready` or `canary-failure-impact` fell back to the generic
+  user-catalog message and the user wants real investigation instead
+- `canary company-knowledge show` reports "No company knowledge configured"
+- Re-running setup to add a field that was skipped the first time (safe — `init`
+  merges onto existing values by default)
+- NOT for storing secrets, API keys, or tokens — this file holds pointers only;
+  the CLI's secret heuristic (`_looks_like_secret`) rejects
+  `sk-`/`token`/`bearer`-shaped values and anything over 128 chars outside
+  `notes`
+- NOT for company-specific proprietary content (client names, internal runbooks,
+  populated data) — per this repo's open-core boundary (`AGENTS.md`), that lives
+  only in a private overlay under `.canary/skills/` or the org's own tooling,
+  reached _via_ the pointers this file stores
+
+## What This Skill Cannot Automate
+
+Every field in `.canary/company.json` is org-specific and cannot be reliably
+inferred from the codebase alone — this repo is public/open-core by design, so
+nothing in it names a real company, Jira project, or internal host. Don't claim
+otherwise. What this skill _can_ do:
+
+- Scaffold the file and `.gitignore` entry so it exists and is never
+  accidentally committed
+- Prompt for each field with the exact validation format the CLI expects, so the
+  user isn't guessing at schema
+- Detect when a value looks like a secret and refuse it before it reaches disk
+- Merge onto existing values, so re-running is always safe
+
+What it cannot do: know your Confluence space key, your Jira project prefix, or
+which MCP server your org runs. Those come from the user.
+
+## Process
+
+### Phase 1: CHECK — Does It Already Exist?
+
+```bash
+canary company-knowledge show
+```
+
+- **"No company knowledge configured"** → proceed to Phase 2 (fresh scaffold).
+- **Populated output** → note which fields are already set (shown with
+  `sources:` — `~/.canary/company.json`, `.canary/company.json`, or an env
+  layer). Proceed to Phase 2 in merge mode (no `--force`) so existing values
+  survive as defaults.
+- **`⚠` error line** (secret-like value or malformed JSON in an existing layer)
+  → surface the exact warning to the user before continuing; that layer is being
+  skipped entirely until fixed.
+
+### Phase 2: SCAFFOLD — Run the Interactive Init
+
+```bash
+canary company-knowledge init
+```
+
+Walk the user through each prompt. For each field, explain what it's for and
+give the expected shape before they answer — the CLI accepts blank input to skip
+or keep the current value:
+
+| Field                | Format                                                     | Purpose                                        |
+| -------------------- | ---------------------------------------------------------- | ---------------------------------------------- |
+| `confluence_spaces`  | comma-separated, uppercase (`QA, ENG`)                     | spaces the LLM should consult for org docs     |
+| `jira_projects`      | comma-separated, uppercase (`PROJ, OPS`)                   | projects for ticket cross-referencing          |
+| `internal_doc_urls`  | one URL per line, `http(s)://` only                        | specific reference docs to fetch via MCP       |
+| `internal_domains`   | comma-separated hostnames (`corp.example.com`)             | flags internal-only URLs in generated content  |
+| `mcp_servers`        | comma-separated identifiers (`plugin_atlassian_atlassian`) | which configured MCP server(s) back the above  |
+| `claude_code_skills` | comma-separated slugs (`team:skill-name`)                  | project-overlay skills to surface as available |
+| `notes`              | free text, ≤2048 chars, no secrets                         | anything else the LLM should know              |
+
+Fields not prompted by `init` (`dashboard_url`, `dashboard_token_env`,
+`otel_exporter_endpoint`) can be added by hand-editing the JSON afterward —
+mention this if the user needs dashboard/OTel wiring; don't skip it silently.
+
+If the user offers a value that looks like a credential (starts with `sk-`,
+`token`, `bearer`, or is unusually long), stop and remind them: secrets go in
+environment variables, never in `company.json`. The CLI will reject the whole
+layer if one slips through — better to catch it before submitting.
+
+### Phase 3: WIRE THE USER-CATALOG SKILL (Known Schema Gap)
+
+`canary-ci-ready` and `canary-failure-impact` both read a `user_catalog_skill`
+key directly from `.canary/company.json` to investigate auth/config failures.
+**This key is not part of the CLI's validated schema** — it isn't in
+`CompanyKnowledge`'s known fields, `company-knowledge init` never prompts for
+it, and `company-knowledge show` will list it under "ignored unknown field" if
+you inspect warnings. It still works for the two consuming skills because they
+read the raw JSON file directly rather than going through the Python loader —
+but be transparent with the user that this is an informal extension, not a
+first-class field, until the schema catches up.
+
+To wire it:
+
+1. Ask the user which project-overlay skill (if any) looks up test users — e.g.
+   `team:user-lookup`. If they don't have one, skip this phase; the consuming
+   skills degrade gracefully to a generic prompt.
+2. If they gave a normal skill slug, prefer adding it to `claude_code_skills` (a
+   real, validated field) _and_ separately hand-edit `.canary/company.json` to
+   add the literal key:
+
+   ```json
+   {
+     "claude_code_skills": ["team:user-lookup"],
+     "user_catalog_skill": "team:user-lookup"
+   }
+   ```
+
+3. Confirm the value matches what `canary skills run <user_catalog_skill>`
+   expects as an identifier (same slug format as `claude_code_skills`:
+   `^[a-z0-9][a-z0-9_-]*(:[a-z0-9][a-z0-9_-]*)?$`).
+
+### Phase 4: VERIFY
+
+```bash
+canary company-knowledge show
+```
+
+- Confirm every field the user just set appears in the printed output.
+- Confirm no `⚠` warnings (unknown fields other than the intentional
+  `user_catalog_skill` extension, dropped invalid entries, secret detections).
+- Confirm `.gitignore` now contains a `.canary/` line — `init` adds it
+  automatically, but verify if the project already had a `.gitignore` with
+  unusual formatting.
+- If JSON output is useful for the user's own tooling: `--json`.
+
+## Error Handling
+
+| Situation                                           | What Happens                                         | What To Do                                                                                                                             |
+| --------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
+| `.canary/company.json` already exists, no `--force` | `init` shows existing values as defaults and merges  | Normal — just re-run `init`                                                                                                            |
+| Secret-like value entered                           | That whole layer is dropped; `show` prints a red `✗` | Remove the value, use an env var, re-run                                                                                               |
+| Malformed JSON in an existing file                  | Layer skipped with a parse error                     | Fix the JSON by hand, then re-run `show` to confirm                                                                                    |
+| Invalid entry format (e.g. lowercase Jira key)      | Entry silently dropped, warning logged               | Re-enter in the correct case/format                                                                                                    |
+| Unknown field in the file                           | Warned, not fatal, field ignored by the loader       | Expected for `user_catalog_skill` today (Phase 3) — otherwise likely a typo                                                            |
+| User has no Confluence/Jira/MCP setup at all        | Every field is legitimately empty                    | Skip `init` entirely; `canary-ci-ready`/`canary-failure-impact` degrade to their generic prompts, which is correct behavior, not a bug |
+
+## Examples
+
+### Example: First-time team lead setup
+
+**Prompt:** "Set up company knowledge for canary on this repo."
+
+**Action:** Run `company-knowledge show` → "No company knowledge configured."
+Run `company-knowledge init`. Walk through each field; the team lead has a
+Confluence space (`QA`), a Jira project (`OPS`), and an Atlassian MCP server
+(`plugin_atlassian_atlassian`) but no internal dashboard yet. Leave
+`dashboard_url` unset. Ask about a user-catalog skill — they don't have one, so
+Phase 3 is skipped. Verify with `show`.
+
+### Example: Retrofitting `user_catalog_skill` onto an existing file
+
+**Prompt:** "`canary-ci-ready` keeps telling me to check my user catalog
+manually, but we have a skill for that."
+
+**Action:** Run `show` — confirm `.canary/company.json` already has
+`confluence_spaces`/`jira_projects` set from a prior run. Ask for the skill slug
+(`team:test-user-lookup`). Add it to both `claude_code_skills` and the raw
+`user_catalog_skill` key per Phase 3. Re-run `show` to confirm no new warnings,
+then re-run `canary-ci-ready` on a known failing auth test to confirm the lookup
+now fires.
+
+## Related Skills
+
+- [`canary-ci-ready`](../canary-ci-ready/SKILL.md) — consumes
+  `user_catalog_skill` for auth/config failure investigation
+- [`canary-failure-impact`](../canary-failure-impact/SKILL.md) — same
+  investigation pattern, different trigger context

package/agents/skills/claude-code/canary-critical-areas/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: canary-critical-areas
+description: >
+  Risk-based test prioritisation. Given a codebase or diff, identifies which
+  areas carry the most risk using git churn, downstream dependents,
+  business-critical signals, and existing coverage depth. Produces a ranked list
+  with recommended test types per area.
+---
+
+# Canary: Critical Areas
+
+Identifies the highest-risk areas of a codebase so test effort goes where it
+matters most. Uses multiple signals, degrades gracefully when advanced tooling
+is unavailable.
+
+## When to Use
+
+- Before writing new tests: "where should I focus?"
+
+- After a large diff lands: "what did this change put at risk?"
+
+- As Phase 1 of `/canary-test-pipeline`
+
+- When asked to prioritise test coverage
+
+## Signals
+
+Collect all available signals, score each area, and rank by composite risk
+score.
+
+### 1. Churn / hotspot (always available)
+
+**With harness MCP available:** call `detect_anomalies` (metric `hotspotScore`,
+plus its co-change / single-point-of-failure signals). Harness's hotspot score
+already blends churn with structural risk, so use it directly as this signal and
+skip the raw `git log` pass. Normalise the returned scores to 0–1.
+
+**Fallback (no MCP):**
+
+```bash
+git log --stat --since="90 days ago" -- <path> | grep -c "^"
+```
+
+Files changed most frequently in the last 90 days score higher. Normalise to 0–1
+across all files in scope.
+
+### 2. Downstream dependents
+
+**With harness MCP available:** call `get_impact` for each candidate file
+(`filePath`, `mode: "summary"`) and read the affected-node counts it returns
+(tests / docs / code grouped by type). This is the purpose-built impact
+primitive — do **not** hand-walk `get_relationships` edge-by-edge; `get_impact`
+already computes the transitive downstream set. More affected nodes ⇒ higher
+score.
+
+**Fallback (no MCP):** scan for `import` statements referencing each file using
+`grep -r`. Count unique files that import each candidate.
+
+Files with more inbound dependents score higher — a change here breaks more.
+
+### 3. Business-critical / critical-path flags
+
+**With harness MCP available:** call `get_critical_paths` and add a fixed boost
+(+0.3) to any area whose functions appear in the returned perf-critical set.
+Also query `ask_graph` for `business_fact` nodes associated with each area; any
+business-critical annotation adds the same +0.3 boost (apply the boost once,
+whichever signal fires).
+
+**Fallback:** skip this signal silently (do not penalise the score).
+
+### 4. Coverage depth boost
+
+If `.canary/test-inventory.json` is present: files whose endpoints are at depth
+0 or 1 receive a boost (+0.15) — low depth in a high-churn file is especially
+risky.
+
+## Risk Score
+
+```text
+risk_score = (churn * 0.35) + (dependents * 0.35) + (business_critical * 0.30) + depth_boost
+```
+
+Capped at 1.0. Round to 2 decimal places.
+
+## Output
+
+```text
+Critical areas — <repo> (<N> files analysed)
+
+  1. src/loyalty/points.service.ts       risk 0.92  ████████████
+     signals: high churn · 12 dependents · business_critical
+     recommended: api + integration tests
+
+  2. src/billing/charge.service.ts       risk 0.78  ██████████
+     signals: high churn · billing domain
+     recommended: api tests · /canary-failure-impact suggested
+
+  ...
+```
+
+Show at most 10 areas. If more than 10 qualify, note the total and offer to show
+all.
+
+## Optional Artifact
+
+When `--save` flag is passed (or when invoked by `/canary-test-pipeline`), write
+`.canary/critical-areas.json`:
+
+```json
+{
+  "generated": "<ISO timestamp>",
+  "areas": [
+    {
+      "path": "src/loyalty/points.service.ts",
+      "risk_score": 0.92,
+      "signals": ["high_churn", "many_dependents", "business_critical"],
+      "recommended_test_types": ["api", "integration"],
+      "summary": "High-churn service with 12 downstream dependents"
+    }
+  ]
+}
+```
+
+This file is consumed as opt-in context by `/canary-edge-cases` and
+`/canary-failure-impact`.
+
+## Flags
+
+- `--diff <git ref>` — scope analysis to files changed in a diff
+  (`git diff <ref>...HEAD`)
+
+- `--save` — write `critical-areas.json`
+
+## Related skills
+
+- `/canary-ci-ready` — check 4 consumes `critical-areas.json`
+
+- `/canary-edge-cases` — focuses edge cases on critical areas when JSON present
+
+- `/canary-failure-impact` — focuses tracing on critical paths when JSON present
+
+- `/canary-test-pipeline` — Phase 1

package/agents/skills/claude-code/canary-critical-areas/skill.yaml

@@ -0,0 +1,16 @@
+name: canary-critical-areas
+version: '1.0.0'
+description:
+  Risk-rank a codebase or diff by git churn, downstream dependents,
+  business-critical signals, and coverage depth; outputs a ranked list with
+  recommended test types per area.
+stability: static
+triggers:
+  - manual
+platforms:
+  - claude-code
+type: rigid
+tools: []
+tier: 1
+depends_on:
+  - canary-ci-ready