skill-probe 0.6.0 → 0.7.0

package/README.md

@@ -12,9 +12,22 @@ Complements static linters like `skill-audit` (security/quality) — this is **b
 > **How it works / requirements.** skill-probe is a **terminal CLI**, *not* an in-agent slash
 > command. It drives your local runtime CLI under the hood, so you need that runtime installed and
 > authenticated: **`claude`** (logged in) for `runtime: claude-code`, or **`opencode`** for
-> `runtime: opencode`. `gen`/`fix` use `ANTHROPIC_API_KEY` if set, otherwise the local `claude` CLI.
+> `runtime: opencode`. `gen`/`fix`/`diagnose` use `ANTHROPIC_API_KEY` if set, otherwise the local `claude` CLI.
 > Codex/Gemini runtimes aren't supported yet (their traces don't expose which skill fired).
 
+The workflow:
+
+```text
+doctor → gen → audit → context → diagnose → fix
+```
+
+- `doctor`: verify setup, auth, runtime, skills, and config before spending probes.
+- `gen`: draft a probe config from your existing skills.
+- `audit`: measure which skill actually fires when the whole library is co-loaded.
+- `context`: compare isolation vs co-loaded activation to catch library interference.
+- `diagnose`: explain whether a failure is a routing miss or a description problem.
+- `fix`: rewrite a skill description and keep it only if the measured reliability improves.
+
 ## Why
 
 - Skill activation is stochastic: the same prompt can produce different skill-routing outcomes.
@@ -62,7 +75,7 @@ the live (costing) auth check.
 Don't want to hand-write the cases? Draft them from your skills, then review:
 
 ```bash
-ANTHROPIC_API_KEY=sk-ant-...  skill-probe gen --cwd . > probe.config.json
+skill-probe gen --cwd . > probe.config.json
 ```
 It reads each `.claude/skills/*/SKILL.md`, and an LLM drafts realistic should-fire prompts per
 skill, cross-skill **near-misses** (to surface mis-routing), and off-topic **decoys**. Hallucinated
@@ -72,9 +85,9 @@ skill names are dropped automatically. Flags: `--per-skill <n>` (default 3), `--
 > **Always review the draft before running.** Generators lean toward obvious keyword-matches; you
 > add the messy, oblique phrasings real users type (that's where triggering actually fails).
 
-`gen` (and `fix`) use `ANTHROPIC_API_KEY` if it's set, otherwise they fall back to the logged-in
-**`claude` CLI** — so on a Claude subscription the whole tool (audit, gen, fix) works with **no API
-key at all**.
+`gen`, `fix`, and `diagnose` use `ANTHROPIC_API_KEY` if it's set, otherwise they fall back to the
+logged-in **`claude` CLI** — so on a Claude subscription the whole tool works with **no API key at
+all**.
 
 ## Use
 
@@ -183,6 +196,47 @@ in the library is a config error (typo), surfaced before any probes run. Exit `1
 shows interference, `2` if a case is untrustworthy (infra errors dominated — never a silent pass),
 else `0`.
 
+## Why a skill fails (`skill-probe diagnose`)
+
+When a skill doesn't fire, there are two very different root causes — and the activation rate alone
+can't tell them apart:
+
+- **Description problem** — the model doesn't even recognise the skill applies. Remedy: rewrite the
+  description (`skill-probe fix`).
+- **Routing miss** — the model *does* know the right skill, but a sibling wins at activation time.
+  Remedy: deconflict the sibling / use `context`. Rewriting *this* skill's description won't help.
+
+`diagnose` separates them by measuring **actual** activation (the real runtime probe) against
+**intended** routing — a *forced choice* where the model is shown the skill descriptions and asked
+which one should handle the request, with no auto-execution:
+
+```bash
+skill-probe diagnose --config probe.config.json
+```
+```
+  [🔀 ROUTING-MISS]  greeter  | write a greeting for my new teammate
+        actual   fires 20% [6%, 51%] k=10  (welcomer×8, greeter×2)
+        intended picks 100% [72%, 100%] k=10  (greeter×10)
+        → the model picks "greeter" when asked, but "welcomer" wins at activation. Remedy:
+          deconflict the sibling or inspect with `skill-probe context`. Rewriting "greeter" won't help.
+
+  [✍ DESCRIPTION-PROBLEM]  greeter  | say something nice to the new hire
+        actual   fires 0% [0%, 28%] k=10  (welcomer×10)
+        intended picks 10% [2%, 40%] k=10  (welcomer×9, greeter×1)
+        → the model doesn't recognise "greeter" applies. Remedy: `skill-probe fix --skill greeter`.
+```
+
+> **`intended` is a proxy, not the runtime's router.** The forced choice asks a *fresh* model which
+> skill fits, given only the descriptions — it can disagree with how the runtime actually routes
+> (we've seen `actual` route a skill 100% while `intended` picked a sibling). So treat the
+> routing-miss vs description-problem split as a strong **heuristic**, not ground truth. A useful
+> side effect: when a skill routes fine but `intended` is low, the description reads *ambiguously in
+> isolation* — a leading indicator that activation may be fragile across models or contexts, which
+> `diagnose` calls out on the `routes-ok` line.
+
+Decoys are skipped; a typo'd `expected` skill is a config error. Exit `1` if any case is a
+routing-miss or description-problem, `2` if inconclusive/untrustworthy, else `0`.
+
 ## Fix a failing skill (`skill-probe fix`)
 
 Rewrite a skill's trigger description and **prove the lift is real** before keeping it:
@@ -212,14 +266,27 @@ doesn't measurably help is reverted.
 
 ## Status
 
-Early (v0.5). **Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping +
-four-state verdict (pass / fail / inconclusive / **error**), across two runtimes (Claude Code,
-OpenCode). Infrastructure failures (timeout / auth / crash / empty output) are reported as
-`error`, never as a behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
+Early, but usable end to end.
+
+**Doctor (`skill-probe doctor`):** setup/auth/config preflight. Checks Node, `.claude/skills/`,
+`SKILL.md` frontmatter, config expected skills, runtime CLI availability, and optional live auth
+probe.
+
+**Gen (`skill-probe gen`):** drafts a reviewable probe config from existing skills, including
+should-fire prompts, near-misses, and decoys.
+
+**Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping + four-state verdict
+(pass / fail / inconclusive / **error**), across two runtimes (Claude Code, OpenCode).
+Infrastructure failures (timeout / auth / crash / empty output) are reported as `error`, never as
+a behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
 
 **Context (`skill-probe context`):** isolation-vs-co-loaded activation rates, with **Fisher's exact
 test** on the drop — catches skills that fire alone but are suppressed under the full library's load.
 
+**Diagnose (`skill-probe diagnose`):** compares actual runtime activation against intended
+forced-choice routing to classify failures as routing-miss vs description-problem. Intended routing
+is a heuristic, not ground truth, and the README calls that out explicitly.
+
 **Fix (`skill-probe fix`):** uses the **Bayesian Beta-Binomial** to gate description rewrites on a
 *proven* lift (interleaved before/after, applied only if P(improvement) clears the bar).
 

package/dist/cli.js

@@ -8,15 +8,17 @@ import { resolve } from "node:path";
 import { runAudit } from "./eval.js";
 import { runFix } from "./fix.js";
 import { runContextAudit } from "./context.js";
+import { runDiagnose } from "./diagnose.js";
 import { runDoctor } from "./doctor.js";
 import { listSkills, generateCases, genConfig } from "./gen.js";
 import { loadConfig, parseProbability, parseIntFlag, ConfigError } from "./config.js";
-import { renderTable, renderMarkdown, renderJson, renderFix, renderContext, renderDoctor } from "./report.js";
+import { renderTable, renderMarkdown, renderJson, renderFix, renderContext, renderDoctor, renderDiagnose } from "./report.js";
 const HELP = `skill-probe — audit a co-loaded agent skill library by real activation behavior.
 
 USAGE
   skill-probe --config <file.json> [options]                 # audit
   skill-probe context --config <file.json> [options]         # isolation vs co-loaded activation rates
+  skill-probe diagnose --config <file.json> [options]        # why a skill fails: routing-miss vs description
   skill-probe doctor [--cwd <dir>] [--config <file.json>]    # preflight: setup/auth/config sanity
   skill-probe gen [--cwd <dir>] > probe.config.json          # draft a test config from your skills
   skill-probe fix --config <file.json> --skill <name> [opts] # rewrite a description, prove the lift
@@ -78,6 +80,17 @@ function makeContextProgress(values) {
         progressLen = line.length;
     };
 }
+function makeDiagProgress(values) {
+    if (values["json"] || values["quiet"] || !process.stderr.isTTY)
+        return undefined;
+    return (e) => {
+        const p = e.prompt.length > 28 ? e.prompt.slice(0, 25) + "…" : e.prompt;
+        const line = `  [${e.caseIndex}/${e.caseTotal}] ${e.phase.padEnd(8)} "${p}" — run ${e.validN}/${e.maxK}`;
+        const pad = Math.max(0, progressLen - line.length);
+        process.stderr.write("\r" + line + " ".repeat(pad));
+        progressLen = line.length;
+    };
+}
 async function audit(values) {
     const cfg = loadConfig(values["config"], {
         runtime: values["runtime"], model: values["model"],
@@ -93,6 +106,20 @@ async function audit(values) {
     console.log(text);
     return result.exitCode;
 }
+async function diagnose(values) {
+    const cfg = loadConfig(values["config"], {
+        runtime: values["runtime"], model: values["model"],
+        k: values["k"], threshold: values["threshold"],
+        conf: values["conf"],
+    });
+    const result = await runDiagnose(cfg, {
+        ...(makeDiagProgress(values) ? { onProgress: makeDiagProgress(values) } : {}),
+    });
+    clearProgress();
+    const showCost = !values["no-cost"];
+    console.log(values["json"] ? renderJson(result) : renderDiagnose(result, { showCost }));
+    return result.exitCode;
+}
 async function doctor(values) {
     const result = await runDoctor({
         cwd: values["cwd"] ?? ".",
@@ -191,7 +218,8 @@ async function main() {
         }
         code = cmd === "fix" ? await fix(values)
             : cmd === "context" ? await context(values)
-                : await audit(values);
+                : cmd === "diagnose" ? await diagnose(values)
+                    : await audit(values);
     }
     process.exit(code);
 }

package/dist/context.js

@@ -7,16 +7,10 @@ import { mkdtempSync, mkdirSync, cpSync, rmSync, existsSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { reliability, fisherExact2x2 } from "./stats.js";
-import { measure } from "./orchestrator.js";
+import { measure, infraUntrustworthy } from "./orchestrator.js";
 import { getAdapter } from "./adapters/index.js";
 import { listSkills } from "./gen.js";
 import { ConfigError } from "./config.js";
-/** Same rule the audit uses (eval.ts): a measurement is untrustworthy when infrastructure
- * failures dominate — no valid probes, or errors at least equal the valid count. Such a
- * condition must NOT be read as a behavioral result (a runtime outage is not a clean pass). */
-function infraUntrustworthy(s) {
-    return s.n === 0 || (s.errors > 0 && s.errors >= s.n);
-}
 /** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
  * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
 export function buildContextDir(srcCwd, skillNames) {

package/dist/diagnose.d.ts

@@ -0,0 +1,68 @@
+/** `skill-probe diagnose` — when a skill fails to fire, WHY? Two very different root causes need
+ * two very different fixes, and you can't tell them apart from the activation rate alone:
+ *
+ *   - DESCRIPTION PROBLEM: the model, asked point-blank "which skill handles this?", doesn't even
+ *     pick the expected skill. Its description fails to communicate what it's for. Remedy: rewrite
+ *     the description (`skill-probe fix`).
+ *   - ROUTING MISS: the model DOES pick the expected skill when asked — it understands the
+ *     description — but a sibling wins at actual activation time. Remedy: deconflict the sibling /
+ *     inspect with `skill-probe context`. Rewriting this skill's description won't help.
+ *
+ * We measure ACTUAL activation (the real runtime probe) and INTENDED routing (a forced-choice: the
+ * model is shown the skill descriptions and asked which one applies, with no auto-execution), then
+ * classify the gap between them. */
+import type { Config } from "./types.ts";
+import { reliability } from "./stats.ts";
+import { type RunStats } from "./orchestrator.ts";
+import { type SkillInfo } from "./gen.ts";
+export type DiagVerdict = "routes-ok" | "routing-miss" | "description-problem" | "inconclusive" | "error";
+/** Returns the chosen skill name, "none", or null when the answer is unusable (LLM error/garbage). */
+export type ChoiceFn = (skills: SkillInfo[], prompt: string, model?: string) => Promise<string | null>;
+type Rel = ReturnType<typeof reliability>;
+interface IntendedStats {
+    dist: Record<string, number>;
+    hits: number;
+    n: number;
+    errors: number;
+}
+export interface DiagCaseResult {
+    prompt: string;
+    expected: string;
+    actual: {
+        stats: RunStats;
+        rel: Rel;
+    };
+    intended: {
+        stats: IntendedStats;
+        rel: Rel;
+    };
+    verdict: DiagVerdict;
+    /** the sibling that wins activation in a routing-miss (most-fired non-expected skill), if any */
+    competitor: string | null;
+    remedy: string;
+}
+export interface DiagnoseResult {
+    runtime: string;
+    model: string;
+    threshold: number;
+    conf: number;
+    cases: DiagCaseResult[];
+    skipped: string[];
+    totalCost: number;
+    exitCode: 0 | 1 | 2;
+}
+export interface DiagProgress {
+    caseIndex: number;
+    caseTotal: number;
+    prompt: string;
+    phase: "actual" | "intended";
+    validN: number;
+    maxK: number;
+}
+export declare function normalizeChoice(text: string, skills: SkillInfo[]): string | null;
+export declare const defaultChoice: ChoiceFn;
+export declare function runDiagnose(cfg: Config, deps?: {
+    choice?: ChoiceFn;
+    onProgress?: (e: DiagProgress) => void;
+}): Promise<DiagnoseResult>;
+export {};

package/dist/diagnose.js

@@ -0,0 +1,154 @@
+import { reliability, sequentialDecision } from "./stats.js";
+import { measure, infraUntrustworthy } from "./orchestrator.js";
+import { getAdapter } from "./adapters/index.js";
+import { listSkills } from "./gen.js";
+import { complete } from "./llm.js";
+import { ConfigError } from "./config.js";
+const ROUTER_SYSTEM = "You are a skill router. Given a user request and a list of available skills (name + " +
+    "description), decide which single skill should handle the request. Reply with EXACTLY one skill " +
+    "name from the list, or the word none if no skill fits. Output only that one word — no punctuation, " +
+    "no explanation.";
+export function normalizeChoice(text, skills) {
+    const lower = text.trim().toLowerCase();
+    // detect "no skill" answers on the raw text, before punctuation is stripped (so "n/a" survives)
+    if (/^(none|null|no skill|no-skill|n\/a|n\.a\.?)$/.test(lower))
+        return "None";
+    const t = lower.replace(/[^a-z0-9_\-\s]/g, "").trim();
+    if (!t)
+        return null;
+    for (const s of skills)
+        if (t === s.name.toLowerCase())
+            return s.name;
+    // tolerate "the greeter skill" / "use greeter" by substring match on a known name
+    for (const s of skills)
+        if (new RegExp(`\\b${s.name.toLowerCase()}\\b`).test(t))
+            return s.name;
+    if (/\bnone\b/.test(t))
+        return "None";
+    return null;
+}
+export const defaultChoice = async (skills, prompt, model) => {
+    const list = skills.map((s) => `- ${s.name}: ${s.description}`).join("\n");
+    const user = `Available skills:\n${list}\n\nUser request: "${prompt}"\n\n` +
+        `Which ONE skill should handle this request? Reply with exactly one skill name, or "none".`;
+    let text;
+    try {
+        text = await complete(ROUTER_SYSTEM, user, { ...(model ? { model } : {}), maxTokens: 16 });
+    }
+    catch {
+        return null;
+    }
+    return normalizeChoice(text, skills);
+};
+async function measureIntended(skills, prompt, expected, o) {
+    const dist = {};
+    let hits = 0, n = 0, errors = 0;
+    const maxAttempts = o.maxK + Math.ceil(o.maxK / 2);
+    for (let a = 0; a < maxAttempts && n < o.maxK; a++) {
+        const c = await o.choice(skills, prompt, o.model);
+        if (c === null) {
+            errors++;
+            o.onProbe?.(n, o.maxK);
+            continue;
+        }
+        dist[c] = (dist[c] ?? 0) + 1;
+        if (c === expected)
+            hits++;
+        n++;
+        o.onProbe?.(n, o.maxK);
+        if (n >= 3 && sequentialDecision(hits, n, o.threshold, o.conf) !== "undecided")
+            break;
+    }
+    return { dist, hits, n, errors };
+}
+function classify(actual, actualRel, intended, expected, threshold, conf) {
+    const competitor = Object.entries(actual.dist)
+        .filter(([s]) => s !== expected && s !== "None")
+        .sort((p, q) => q[1] - p[1])[0]?.[0] ?? null;
+    if (infraUntrustworthy(actual) || intended.n === 0) {
+        return { verdict: "error", competitor,
+            remedy: "infrastructure/LLM errors dominated — rerun; the result isn't trustworthy." };
+    }
+    const actualDecision = sequentialDecision(actual.hits, actual.n, threshold, conf);
+    if (actualDecision === "above") {
+        // actual is the ground truth. But if the forced-choice router rarely picks it, the description
+        // reads ambiguously in isolation — a leading indicator that activation may be fragile.
+        const ambiguous = sequentialDecision(intended.hits, intended.n, threshold, conf) === "below";
+        const note = ambiguous
+            ? ` — but the description reads ambiguously in isolation (intended only ${Math.round((intended.hits / intended.n) * 100)}%), ` +
+                `so activation may be fragile across models/contexts`
+            : "";
+        return { verdict: "routes-ok", competitor: null,
+            remedy: `fires reliably (${Math.round(actualRel.pHat * 100)}%) — no action needed${note}.` };
+    }
+    const intendedDecision = sequentialDecision(intended.hits, intended.n, threshold, conf);
+    if (intendedDecision === "above") {
+        return { verdict: "routing-miss", competitor,
+            remedy: `the model picks "${expected}" when asked, but ${competitor ? `"${competitor}"` : "another skill"} ` +
+                `wins at activation. Remedy: deconflict the sibling (narrow its description) or inspect with ` +
+                `\`skill-probe context\`. Rewriting "${expected}" likely won't help.` };
+    }
+    if (intendedDecision === "below") {
+        return { verdict: "description-problem", competitor,
+            remedy: `the model doesn't recognize "${expected}" applies to this request. Remedy: ` +
+                `\`skill-probe fix --skill ${expected}\` to rewrite its trigger description.` };
+    }
+    return { verdict: "inconclusive", competitor,
+        remedy: "neither activation nor intent is statistically decided — raise k and rerun." };
+}
+export async function runDiagnose(cfg, deps = {}) {
+    const adapter = getAdapter(cfg.runtime);
+    const choice = deps.choice ?? defaultChoice;
+    const skills = listSkills(cfg.cwd);
+    const present = new Set(skills.map((s) => s.name));
+    const skipped = [];
+    const unknown = [];
+    const eligible = [];
+    for (const c of cfg.cases) {
+        if (c.expected === null)
+            skipped.push(c.prompt);
+        else if (!present.has(c.expected))
+            unknown.push(c.expected);
+        else
+            eligible.push(c);
+    }
+    if (unknown.length) {
+        throw new ConfigError([
+            `expected skill(s) not in ${cfg.cwd}/.claude/skills/: ${[...new Set(unknown)].join(", ")} ` +
+                `(typo?). Available: ${skills.map((s) => s.name).join(", ") || "(none)"}.`,
+        ]);
+    }
+    const cases = [];
+    let totalCost = 0;
+    for (let i = 0; i < eligible.length; i++) {
+        const c = eligible[i];
+        const expected = c.expected;
+        const actual = await measure(adapter, c.prompt, expected, {
+            cwd: cfg.cwd, maxK: cfg.k, threshold: cfg.threshold, conf: cfg.conf,
+            ...(cfg.model ? { model: cfg.model } : {}),
+            ...(deps.onProgress ? { onProbe: (validN, maxK) => deps.onProgress({ caseIndex: i + 1, caseTotal: eligible.length, prompt: c.prompt, phase: "actual", validN, maxK }) } : {}),
+        });
+        const intended = await measureIntended(skills, c.prompt, expected, {
+            maxK: cfg.k, threshold: cfg.threshold, conf: cfg.conf, choice,
+            ...(cfg.model ? { model: cfg.model } : {}),
+            ...(deps.onProgress ? { onProbe: (validN, maxK) => deps.onProgress({ caseIndex: i + 1, caseTotal: eligible.length, prompt: c.prompt, phase: "intended", validN, maxK }) } : {}),
+        });
+        totalCost += actual.totalCost;
+        const actualRel = reliability(actual.hits, actual.n, cfg.conf);
+        const intendedRel = reliability(intended.hits, intended.n, cfg.conf);
+        const { verdict, competitor, remedy } = classify(actual, actualRel, intended, expected, cfg.threshold, cfg.conf);
+        cases.push({
+            prompt: c.prompt, expected,
+            actual: { stats: actual, rel: actualRel },
+            intended: { stats: intended, rel: intendedRel },
+            verdict, competitor, remedy,
+        });
+    }
+    const hasProblem = cases.some((c) => c.verdict === "routing-miss" || c.verdict === "description-problem");
+    const hasBad = cases.some((c) => c.verdict === "error" || c.verdict === "inconclusive");
+    const exitCode = hasProblem ? 1 : hasBad ? 2 : 0;
+    return {
+        runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
+        threshold: cfg.threshold, conf: cfg.conf, cases, skipped, totalCost, exitCode,
+    };
+}

package/dist/orchestrator.d.ts

@@ -15,6 +15,10 @@ export interface RunStats {
     dist: Record<string, number>;
     totalCost: number;
 }
+/** A measurement is untrustworthy when infrastructure failures dominate — no valid probes, or
+ * errors at least equal the valid count. Such a run must NOT read as a behavioral result (a runtime
+ * outage is not a clean pass). Shared by the audit, context, and diagnose paths. */
+export declare function infraUntrustworthy(s: RunStats): boolean;
 export declare function measure(adapter: RuntimeAdapter, prompt: string, expected: string | null, opts: {
     cwd: string;
     maxK: number;

package/dist/orchestrator.js

@@ -1,4 +1,10 @@
 import { sequentialDecision } from "./stats.js";
+/** A measurement is untrustworthy when infrastructure failures dominate — no valid probes, or
+ * errors at least equal the valid count. Such a run must NOT read as a behavioral result (a runtime
+ * outage is not a clean pass). Shared by the audit, context, and diagnose paths. */
+export function infraUntrustworthy(s) {
+    return s.n === 0 || (s.errors > 0 && s.errors >= s.n);
+}
 export async function measure(adapter, prompt, expected, opts) {
     const want = expected ?? "None";
     const dist = {};

package/dist/report.d.ts

@@ -2,6 +2,7 @@ import type { AuditResult } from "./eval.ts";
 import type { FixResult } from "./fix.ts";
 import type { ContextAuditResult } from "./context.ts";
 import type { DoctorResult } from "./doctor.ts";
+import type { DiagnoseResult } from "./diagnose.ts";
 /** Smallest k at which an all-success run could certify reliability >= threshold. */
 export declare function kToCertify(threshold: number, conf: number): number;
 export interface RenderOpts {
@@ -11,6 +12,8 @@ export declare function renderTable(a: AuditResult, opts?: RenderOpts): string;
 /** Markdown table — paste into a PR or README. */
 export declare function renderMarkdown(a: AuditResult, opts?: RenderOpts): string;
 export declare function renderJson(a: AuditResult | unknown): string;
+/** intended (forced-choice routing) vs actual (real activation), classified into a remedy. */
+export declare function renderDiagnose(d: DiagnoseResult, opts?: RenderOpts): string;
 export declare function renderDoctor(d: DoctorResult): string;
 /** activation-rate-by-context: isolation vs co-loaded, with a Fisher's-exact p on the drop. */
 export declare function renderContext(a: ContextAuditResult, opts?: RenderOpts): string;

package/dist/report.js

@@ -81,6 +81,37 @@ export function renderMarkdown(a, opts = {}) {
 export function renderJson(a) {
     return JSON.stringify(a, null, 2);
 }
+/** intended (forced-choice routing) vs actual (real activation), classified into a remedy. */
+export function renderDiagnose(d, opts = {}) {
+    const showCost = opts.showCost ?? true;
+    const LBL = {
+        "routes-ok": "✅ ROUTES-OK", "routing-miss": "🔀 ROUTING-MISS",
+        "description-problem": "✍ DESCRIPTION-PROBLEM", inconclusive: "⚠️ INCONCLUSIVE", error: "⛔ ERROR",
+    };
+    const out = [];
+    out.push(`skill-probe diagnose — runtime: ${d.runtime}  model: ${d.model}  threshold: ${pct(d.threshold)}`);
+    out.push(`  actual = real activation · intended = forced-choice routing (does the model even pick it?)`);
+    out.push("");
+    const dist = (m) => Object.entries(m).sort((p, q) => q[1] - p[1]).map(([s, n]) => `${s}×${n}`).join(", ") || "(none)";
+    for (const c of d.cases) {
+        out.push(`  [${LBL[c.verdict]}]  ${c.expected}  | ${c.prompt}`);
+        out.push(`        actual   fires ${pct(c.actual.rel.pHat)} [${pct(c.actual.rel.ciLow)}, ${pct(c.actual.rel.ciHigh)}] ` +
+            `k=${c.actual.rel.k}  (${dist(c.actual.stats.dist)})`);
+        out.push(`        intended picks ${pct(c.intended.rel.pHat)} [${pct(c.intended.rel.ciLow)}, ${pct(c.intended.rel.ciHigh)}] ` +
+            `k=${c.intended.rel.k}  (${dist(c.intended.stats.dist)})`);
+        out.push(`        → ${c.remedy}`);
+    }
+    if (d.skipped.length) {
+        out.push("");
+        out.push(`  skipped ${d.skipped.length} decoy case(s) — diagnose explains why an expected skill doesn't fire.`);
+    }
+    out.push("");
+    const counts = (v) => d.cases.filter((c) => c.verdict === v).length;
+    out.push(`Result: ${counts("routes-ok")} ok / ${counts("routing-miss")} routing-miss / ` +
+        `${counts("description-problem")} description-problem / ${counts("inconclusive") + counts("error")} other  ` +
+        `|  exit ${d.exitCode}` + (showCost ? `  |  cost $${d.totalCost.toFixed(4)}` : ""));
+    return out.join("\n");
+}
 export function renderDoctor(d) {
     const out = ["skill-probe doctor", ""];
     for (const c of d.checks)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skill-probe",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Runtime-agnostic, co-loaded-aware auditor for AI agent skill libraries: measures which skill actually fires, with statistical confidence.",
   "type": "module",
   "bin": {