skill-probe 0.3.0 → 0.5.0

package/README.md

@@ -9,6 +9,12 @@ which skill *actually* fires (by real activation, not keyword matching) when you
 
 Complements static linters like `skill-audit` (security/quality) — this is **behavioral**.
 
+> **How it works / requirements.** skill-probe is a **terminal CLI**, *not* an in-agent slash
+> command. It drives your local runtime CLI under the hood, so you need that runtime installed and
+> authenticated: **`claude`** (logged in) for `runtime: claude-code`, or **`opencode`** for
+> `runtime: opencode`. `gen`/`fix` use `ANTHROPIC_API_KEY` if set, otherwise the local `claude` CLI.
+> Codex/Gemini runtimes aren't supported yet (their traces don't expose which skill fired).
+
 ## Why
 
 - Skill activation is stochastic: the same prompt can produce different skill-routing outcomes.
@@ -35,8 +41,11 @@ skill names are dropped automatically. Flags: `--per-skill <n>` (default 3), `--
 (default 2), `--model <id>`.
 
 > **Always review the draft before running.** Generators lean toward obvious keyword-matches; you
-> add the messy, oblique phrasings real users type (that's where triggering actually fails). `gen`
-> needs `ANTHROPIC_API_KEY`.
+> add the messy, oblique phrasings real users type (that's where triggering actually fails).
+
+`gen` (and `fix`) use `ANTHROPIC_API_KEY` if it's set, otherwise they fall back to the logged-in
+**`claude` CLI** — so on a Claude subscription the whole tool (audit, gen, fix) works with **no API
+key at all**.
 
 ## Use
 
@@ -99,6 +108,52 @@ Claude Code vs OpenCode comparison is fair — otherwise you're comparing two co
   case is inconclusive the report prints the exact k for your threshold. Don't set `threshold
   0.9, k 10` and expect a pass — that's statistically impossible and the tool will say so.
 
+## CI usage
+
+skill-probe exits non-zero on a real problem, so it drops straight into a pipeline:
+
+```yaml
+# .github/workflows/skills.yml  (the claude/opencode CLI must be installed + authed on the runner)
+- run: npx skill-probe --config probe.config.json --quiet --no-cost
+```
+
+Exit codes: **`0`** all pass · **`1`** a behavioral fail / trigger-theft / interference · **`2`**
+inconclusive or an infrastructure error — so a runtime outage *fails* the build instead of silently
+passing. Add `--json` to archive the full result as a build artifact.
+
+## Activation rate by context (`skill-probe context`)
+
+The audit measures every skill **co-loaded** — the real, hard condition. But a skill can fire
+*fine on its own* and only fail *under load*, when the rest of your library is competing for the
+same trigger. `context` exposes exactly that:
+
+```bash
+skill-probe context --config probe.config.json
+```
+
+For each case it measures the expected skill **in isolation** (a throwaway project with only that
+one skill) **and co-loaded** (your full library), then tests the drop with **Fisher's exact test**
+— so interference is reported as a real effect with a p-value, not eyeballed:
+
+```
+skill-probe context — runtime: claude-code  model: (runtime default)  threshold: 70%  library: 4 skills
+  isolation = only that skill loaded · co-loaded = the full library (4) loaded
+
+  [⚠ INTERFERENCE]  greeter  | write a birthday greeting
+        isolation 100% [72%, 100%] k=10
+        co-loaded 30% [11%, 60%] k=10
+        Δ -70% under load   Fisher p=0.003
+        ↳ fires reliably alone but is suppressed when the library is co-loaded (stolen by: welcomer)
+
+Result: 1 interference / 3 measured  |  exit 1
+```
+
+This is the "fires in isolation, fails under load X" case the aggregate rate hides. Decoy cases
+(`expected: null`) are skipped — there's no single skill to isolate; an `expected` skill that isn't
+in the library is a config error (typo), surfaced before any probes run. Exit `1` if any skill
+shows interference, `2` if a case is untrustworthy (infra errors dominated — never a silent pass),
+else `0`.
+
 ## Fix a failing skill (`skill-probe fix`)
 
 Rewrite a skill's trigger description and **prove the lift is real** before keeping it:
@@ -122,22 +177,26 @@ P(rewrite improved reliability) = 100%   Δ = +80% [36%, 99%]
 original backed up to: /path/to/SKILL.md.bak.1719000000000
 ```
 
-`fix` needs `ANTHROPIC_API_KEY` (for the rewrite). It changes descriptions on statistical evidence,
-not on "the new one looks nicer" — a rewrite that doesn't measurably help is reverted.
+`fix` does the rewrite via `ANTHROPIC_API_KEY` if set, else the logged-in `claude` CLI (subscription).
+It changes descriptions on statistical evidence, not on "the new one looks nicer" — a rewrite that
+doesn't measurably help is reverted.
 
 ## Status
 
-Early (v0.1). **Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping +
+Early (v0.5). **Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping +
 four-state verdict (pass / fail / inconclusive / **error**), across two runtimes (Claude Code,
 OpenCode). Infrastructure failures (timeout / auth / crash / empty output) are reported as
 `error`, never as a behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
 
+**Context (`skill-probe context`):** isolation-vs-co-loaded activation rates, with **Fisher's exact
+test** on the drop — catches skills that fire alone but are suppressed under the full library's load.
+
 **Fix (`skill-probe fix`):** uses the **Bayesian Beta-Binomial** to gate description rewrites on a
 *proven* lift (interleaved before/after, applied only if P(improvement) clears the bar).
 
-**Implemented + unit-tested but NOT yet used by any command:** Fisher's exact test and
-Benjamini-Hochberg FDR (`src/stats.ts`) — reserved for cross-case multiplicity control (a planned
-enhancement). Don't read the presence of these functions as the audit using them.
+**Implemented + unit-tested but NOT yet used by any command:** Benjamini-Hochberg FDR
+(`src/stats.ts`) — reserved for cross-case multiplicity control (a planned enhancement). Don't read
+its presence as the audit using it. (Fisher's exact test is now wired into `context`.)
 
 ## Dev
 

package/dist/cli.js

@@ -7,13 +7,15 @@ import { parseArgs } from "node:util";
 import { resolve } from "node:path";
 import { runAudit } from "./eval.js";
 import { runFix } from "./fix.js";
+import { runContextAudit } from "./context.js";
 import { listSkills, generateCases, genConfig } from "./gen.js";
-import { loadConfig, parseProbability, ConfigError } from "./config.js";
-import { renderTable, renderMarkdown, renderJson, renderFix } from "./report.js";
+import { loadConfig, parseProbability, parseIntFlag, ConfigError } from "./config.js";
+import { renderTable, renderMarkdown, renderJson, renderFix, renderContext } from "./report.js";
 const HELP = `skill-probe — audit a co-loaded agent skill library by real activation behavior.
 
 USAGE
   skill-probe --config <file.json> [options]                 # audit
+  skill-probe context --config <file.json> [options]         # isolation vs co-loaded activation rates
   skill-probe gen [--cwd <dir>] > probe.config.json          # draft a test config from your skills
   skill-probe fix --config <file.json> --skill <name> [opts] # rewrite a description, prove the lift
 
@@ -30,7 +32,7 @@ OPTIONS
       --markdown           emit a Markdown table (paste into a PR/README)
       --no-cost            hide the cost line (e.g. when on a Claude subscription)
       --quiet              suppress the live progress line on stderr
-  gen only (needs ANTHROPIC_API_KEY):
+  gen only (uses ANTHROPIC_API_KEY if set, else the logged-in 'claude' CLI):
       --cwd <dir>          project dir containing .claude/skills/ (default: .)
       --per-skill <n>      should-fire prompts to draft per skill (default 3)
       --decoys <n>         off-topic decoy prompts to draft (default 2)
@@ -59,6 +61,17 @@ function clearProgress() {
         progressLen = 0;
     }
 }
+function makeContextProgress(values) {
+    if (values["json"] || values["quiet"] || !process.stderr.isTTY)
+        return undefined;
+    return (e) => {
+        const p = e.prompt.length > 28 ? e.prompt.slice(0, 25) + "…" : e.prompt;
+        const line = `  [${e.caseIndex}/${e.caseTotal}] ${e.condition.padEnd(9)} "${p}" — run ${e.validN}/${e.maxK}`;
+        const pad = Math.max(0, progressLen - line.length);
+        process.stderr.write("\r" + line + " ".repeat(pad));
+        progressLen = line.length;
+    };
+}
 async function audit(values) {
     const cfg = loadConfig(values["config"], {
         runtime: values["runtime"], model: values["model"],
@@ -74,14 +87,22 @@ async function audit(values) {
     console.log(text);
     return result.exitCode;
 }
-function intArg(raw, def, min) {
-    if (raw === undefined)
-        return def;
-    const v = Math.floor(Number(raw));
-    return Number.isFinite(v) && v >= min ? v : def;
+async function context(values) {
+    const cfg = loadConfig(values["config"], {
+        runtime: values["runtime"], model: values["model"],
+        k: values["k"], threshold: values["threshold"],
+        conf: values["conf"],
+    });
+    const result = await runContextAudit(cfg, makeContextProgress(values));
+    clearProgress();
+    const showCost = !values["no-cost"];
+    console.log(values["json"] ? renderJson(result) : renderContext(result, { showCost }));
+    return result.exitCode;
 }
 async function gen(values) {
     const cwd = values["cwd"] ?? ".";
+    const perSkill = parseIntFlag(values["per-skill"], "--per-skill", 3, 1);
+    const decoys = parseIntFlag(values["decoys"], "--decoys", 2, 0);
     const skills = listSkills(resolve(cwd));
     if (skills.length === 0) {
         console.error(`no skills found under ${cwd}/.claude/skills/`);
@@ -89,9 +110,7 @@ async function gen(values) {
     }
     const model = values["model"];
     const cases = await generateCases(skills, {
-        perSkill: intArg(values["per-skill"], 3, 1),
-        decoys: intArg(values["decoys"], 2, 0),
-        ...(model ? { model } : {}),
+        perSkill, decoys, ...(model ? { model } : {}),
     });
     console.log(renderJson(genConfig(cases)));
     console.error(`# drafted ${cases.length} cases for ${skills.length} skill(s) — REVIEW before ` +
@@ -149,7 +168,9 @@ async function main() {
             console.log(HELP);
             process.exit(2);
         }
-        code = cmd === "fix" ? await fix(values) : await audit(values);
+        code = cmd === "fix" ? await fix(values)
+            : cmd === "context" ? await context(values)
+                : await audit(values);
     }
     process.exit(code);
 }

package/dist/config.d.ts

@@ -9,6 +9,8 @@ export declare class ConfigError extends Error {
 export declare function validateConfig(raw: unknown, configPath: string): Config;
 /** Validate a probability-ish flag (e.g. --apply-threshold): must be a number strictly in (0,1). */
 export declare function parseProbability(raw: string | undefined, name: string, def: number): number;
+/** Validate an integer flag (e.g. --per-skill, --decoys): must be an integer >= min if provided. */
+export declare function parseIntFlag(raw: string | undefined, name: string, def: number, min: number): number;
 /** CLI flag overrides (already string-typed from argv); numerics are coerced then validated. */
 export interface Overrides {
     runtime?: string | undefined;

package/dist/config.js

@@ -75,6 +75,16 @@ export function parseProbability(raw, name, def) {
     }
     return v;
 }
+/** Validate an integer flag (e.g. --per-skill, --decoys): must be an integer >= min if provided. */
+export function parseIntFlag(raw, name, def, min) {
+    if (raw === undefined)
+        return def;
+    const v = Number(raw);
+    if (!Number.isInteger(v) || v < min) {
+        throw new ConfigError([`${name} must be an integer >= ${min} (got ${JSON.stringify(raw)})`]);
+    }
+    return v;
+}
 export function loadConfig(configPath, overrides = {}) {
     let fileRaw;
     try {

package/dist/context.d.ts

@@ -0,0 +1,48 @@
+import type { Config } from "./types.ts";
+import { reliability } from "./stats.ts";
+import { type RunStats } from "./orchestrator.ts";
+/** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
+ * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
+export declare function buildContextDir(srcCwd: string, skillNames: string[]): {
+    cwd: string;
+    cleanup: () => void;
+};
+type Rel = ReturnType<typeof reliability>;
+interface Condition {
+    stats: RunStats;
+    rel: Rel;
+}
+export interface ContextCaseResult {
+    prompt: string;
+    expected: string;
+    isolation: Condition;
+    coLoaded: Condition;
+    deltaP: number;
+    fisherP: number;
+    /** infra failures dominated one side — the comparison is not trustworthy (NOT a behavioral result) */
+    untrustworthy: boolean;
+    /** fires reliably alone but drops significantly when the full library is co-loaded */
+    interference: boolean;
+}
+export interface ContextAuditResult {
+    runtime: string;
+    model: string;
+    threshold: number;
+    conf: number;
+    librarySize: number;
+    cases: ContextCaseResult[];
+    skipped: string[];
+    totalCost: number;
+    /** 0 = no interference; 1 = a skill is suppressed under load; 2 = an untrustworthy (infra) case */
+    exitCode: 0 | 1 | 2;
+}
+export interface ContextProgress {
+    caseIndex: number;
+    caseTotal: number;
+    prompt: string;
+    condition: "isolation" | "co-loaded";
+    validN: number;
+    maxK: number;
+}
+export declare function runContextAudit(cfg: Config, onProgress?: (e: ContextProgress) => void): Promise<ContextAuditResult>;
+export {};

package/dist/context.js

@@ -0,0 +1,112 @@
+/** `skill-probe context` — activation-rate-BY-CONTEXT. The audit measures every skill co-loaded
+ * (the real, hard condition). This command additionally measures each skill in ISOLATION (only that
+ * skill present) and compares the two: it catches the "fires fine alone, fails under load" case that
+ * a single aggregate rate hides. The drop is tested with Fisher's exact test, so interference is
+ * reported as a real effect (p-value), not eyeballed. */
+import { mkdtempSync, mkdirSync, cpSync, rmSync, existsSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { reliability, fisherExact2x2 } from "./stats.js";
+import { measure } from "./orchestrator.js";
+import { getAdapter } from "./adapters/index.js";
+import { listSkills } from "./gen.js";
+import { ConfigError } from "./config.js";
+/** Same rule the audit uses (eval.ts): a measurement is untrustworthy when infrastructure
+ * failures dominate — no valid probes, or errors at least equal the valid count. Such a
+ * condition must NOT be read as a behavioral result (a runtime outage is not a clean pass). */
+function infraUntrustworthy(s) {
+    return s.n === 0 || (s.errors > 0 && s.errors >= s.n);
+}
+/** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
+ * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
+export function buildContextDir(srcCwd, skillNames) {
+    const root = mkdtempSync(join(tmpdir(), "skill-probe-ctx-"));
+    const destSkills = join(root, ".claude", "skills");
+    mkdirSync(destSkills, { recursive: true });
+    for (const name of skillNames) {
+        const src = join(srcCwd, ".claude", "skills", name);
+        if (!existsSync(src))
+            throw new Error(`skill not found in ${srcCwd}: ${name}`);
+        cpSync(src, join(destSkills, name), { recursive: true });
+    }
+    return { cwd: root, cleanup: () => rmSync(root, { recursive: true, force: true }) };
+}
+export async function runContextAudit(cfg, onProgress) {
+    const adapter = getAdapter(cfg.runtime);
+    const library = listSkills(cfg.cwd);
+    const present = new Set(library.map((s) => s.name));
+    const measured = [];
+    const skipped = [];
+    let totalCost = 0;
+    // decoys (null expected) have nothing to isolate → skipped. An expected skill that ISN'T in the
+    // library is almost always a config typo: fail loud BEFORE spending probes, never silently skip.
+    const eligible = [];
+    const unknown = [];
+    for (const c of cfg.cases) {
+        if (c.expected === null)
+            skipped.push(c.prompt);
+        else if (!present.has(c.expected))
+            unknown.push(c.expected);
+        else
+            eligible.push(c);
+    }
+    if (unknown.length) {
+        throw new ConfigError([
+            `expected skill(s) not in ${cfg.cwd}/.claude/skills/: ${[...new Set(unknown)].join(", ")} ` +
+                `(typo?). Available: ${library.map((s) => s.name).join(", ") || "(none)"}.`,
+        ]);
+    }
+    const common = { maxK: cfg.k, threshold: cfg.threshold, conf: cfg.conf, ...(cfg.model ? { model: cfg.model } : {}) };
+    for (let i = 0; i < eligible.length; i++) {
+        const c = eligible[i];
+        const expected = c.expected;
+        // isolation: a temp project containing only this skill
+        const ctx = buildContextDir(cfg.cwd, [expected]);
+        let iso;
+        try {
+            iso = await measure(adapter, c.prompt, expected, {
+                cwd: ctx.cwd, ...common,
+                ...(onProgress ? { onProbe: (validN, maxK) => onProgress({ caseIndex: i + 1, caseTotal: eligible.length, prompt: c.prompt, condition: "isolation", validN, maxK }) } : {}),
+            });
+        }
+        finally {
+            ctx.cleanup();
+        }
+        // co-loaded: the real library at cfg.cwd
+        const co = await measure(adapter, c.prompt, expected, {
+            cwd: cfg.cwd, ...common,
+            ...(onProgress ? { onProbe: (validN, maxK) => onProgress({ caseIndex: i + 1, caseTotal: eligible.length, prompt: c.prompt, condition: "co-loaded", validN, maxK }) } : {}),
+        });
+        totalCost += iso.totalCost + co.totalCost;
+        const isoRel = reliability(iso.hits, iso.n, cfg.conf);
+        const coRel = reliability(co.hits, co.n, cfg.conf);
+        // 2x2: rows = isolation/co-loaded, cols = hit/miss
+        const fisherP = (iso.n > 0 && co.n > 0)
+            ? fisherExact2x2(iso.hits, iso.n - iso.hits, co.hits, co.n - co.hits)
+            : 1;
+        const deltaP = coRel.pHat - isoRel.pHat;
+        // if infra failures dominated either side, the comparison isn't a behavioral result at all —
+        // it must never read as "no interference / pass" just because fisherP defaulted to 1.
+        const untrustworthy = infraUntrustworthy(iso) || infraUntrustworthy(co);
+        const interference = !untrustworthy &&
+            isoRel.pHat >= cfg.threshold && // fires reliably alone
+            deltaP < 0 && // and drops when co-loaded
+            fisherP < 0.05; // and the drop is statistically real
+        measured.push({
+            prompt: c.prompt, expected,
+            isolation: { stats: iso, rel: isoRel },
+            coLoaded: { stats: co, rel: coRel },
+            deltaP, fisherP, untrustworthy, interference,
+        });
+    }
+    // precedence mirrors the audit: a real behavioral signal (interference, exit 1) outranks an
+    // untrustworthy/infra case (exit 2), which outranks a clean run (exit 0).
+    const exitCode = measured.some((m) => m.interference) ? 1
+        : measured.some((m) => m.untrustworthy) ? 2
+            : 0;
+    return {
+        runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
+        threshold: cfg.threshold, conf: cfg.conf, librarySize: library.length,
+        cases: measured, skipped, totalCost, exitCode,
+    };
+}

package/dist/llm.d.ts

@@ -1,4 +1,15 @@
-export declare function complete(system: string, user: string, opts?: {
+export type LlmBackend = "api" | "cli";
+/** Which backend complete() will use, given the environment. Exported for testing/diagnostics. */
+export declare function llmBackend(): LlmBackend;
+export interface CompleteOpts {
     model?: string;
     maxTokens?: number;
-}): Promise<string>;
+}
+export declare function complete(system: string, user: string, opts?: CompleteOpts): Promise<string>;
+/** Pure interpreter of a `claude -p --output-format json` run. Treats a non-zero exit, an
+ * `is_error: true` payload, a timeout, or empty output as ERRORS — so an error string can never
+ * be mistaken for a valid generation (critical: fix would otherwise write it as a description). */
+export declare function interpretClaudeJson(code: number | null, out: string, err: string, timedOut: boolean): {
+    text?: string;
+    error?: string;
+};

package/dist/llm.js

@@ -1,33 +1,72 @@
-/** Minimal Anthropic client for the `fix` command's description rewrite. Uses Node's built-in
- * fetch (Node >=18) — zero deps. Needs ANTHROPIC_API_KEY (only `fix` uses this, not the audit). */
+/** LLM text completion for `gen` and `fix`. Unified auth (v0.4):
+ *   - ANTHROPIC_API_KEY set  → call the Anthropic API directly (fetch, Node >=18).
+ *   - otherwise              → fall back to the `claude` CLI (works on a Claude subscription,
+ *                              no API key needed).
+ * So the whole tool — audit, gen, fix — runs on just a logged-in `claude`. */
+import { spawn } from "node:child_process";
+import { tmpdir } from "node:os";
 const ANTHROPIC_URL = "https://api.anthropic.com/v1/messages";
+/** Which backend complete() will use, given the environment. Exported for testing/diagnostics. */
+export function llmBackend() {
+    return process.env["ANTHROPIC_API_KEY"] ? "api" : "cli";
+}
 export async function complete(system, user, opts = {}) {
+    return llmBackend() === "api"
+        ? completeViaApi(system, user, opts)
+        : completeViaClaudeCli(system, user, opts);
+}
+async function completeViaApi(system, user, opts) {
     const key = process.env["ANTHROPIC_API_KEY"];
-    if (!key) {
-        throw new Error("ANTHROPIC_API_KEY is required for `skill-probe fix` (it rewrites the description).");
-    }
     const model = opts.model ?? process.env["SKILL_PROBE_FIX_MODEL"] ?? "claude-sonnet-4-6";
     const res = await fetch(ANTHROPIC_URL, {
         method: "POST",
-        headers: {
-            "content-type": "application/json",
-            "x-api-key": key,
-            "anthropic-version": "2023-06-01",
-        },
-        body: JSON.stringify({
-            model,
-            max_tokens: opts.maxTokens ?? 400,
-            system,
-            messages: [{ role: "user", content: user }],
-        }),
+        headers: { "content-type": "application/json", "x-api-key": key, "anthropic-version": "2023-06-01" },
+        body: JSON.stringify({ model, max_tokens: opts.maxTokens ?? 400, system, messages: [{ role: "user", content: user }] }),
     });
-    if (!res.ok) {
+    if (!res.ok)
         throw new Error(`Anthropic API ${res.status}: ${(await res.text()).slice(0, 200)}`);
-    }
     const body = (await res.json());
-    return (body.content ?? [])
-        .filter((b) => b.type === "text")
-        .map((b) => b.text ?? "")
-        .join("")
-        .trim();
+    return (body.content ?? []).filter((b) => b.type === "text").map((b) => b.text ?? "").join("").trim();
+}
+/** Pure interpreter of a `claude -p --output-format json` run. Treats a non-zero exit, an
+ * `is_error: true` payload, a timeout, or empty output as ERRORS — so an error string can never
+ * be mistaken for a valid generation (critical: fix would otherwise write it as a description). */
+export function interpretClaudeJson(code, out, err, timedOut) {
+    if (timedOut)
+        return { error: "claude CLI timed out after 120s" };
+    let parsed = null;
+    try {
+        parsed = JSON.parse(out);
+    }
+    catch { /* may be plain text */ }
+    if (code !== 0 || parsed?.is_error === true) {
+        const ctx = (parsed && typeof parsed.result === "string" ? parsed.result : "") || err || out;
+        return { error: `claude CLI error (exit ${code}): ${ctx.trim().slice(0, 200)}` };
+    }
+    let text = parsed ? (typeof parsed.result === "string" ? parsed.result : "") : out;
+    text = text.trim();
+    return text ? { text } : { error: `claude CLI returned no text (${(err || out).slice(0, 200)})` };
+}
+function completeViaClaudeCli(system, user, opts) {
+    return new Promise((resolve, reject) => {
+        const args = ["-p", user, "--append-system-prompt", system, "--output-format", "json"];
+        if (opts.model)
+            args.push("--model", opts.model);
+        // run in a neutral cwd so the project's own skills/AGENTS.md don't bias the generation
+        const child = spawn("claude", args, { cwd: tmpdir(), stdio: ["ignore", "pipe", "pipe"] });
+        let out = "", err = "", settled = false, timedOut = false;
+        const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, 120_000);
+        const done = (fn) => { if (settled)
+            return; settled = true; clearTimeout(timer); fn(); };
+        child.stdout.on("data", (d) => { out += d.toString(); });
+        child.stderr.on("data", (d) => { err += d.toString(); });
+        child.on("error", (e) => done(() => reject(new Error(`claude CLI failed: ${e.message} — set ANTHROPIC_API_KEY or install/login 'claude'.`))));
+        child.on("close", (code) => done(() => {
+            const r = interpretClaudeJson(code, out, err, timedOut);
+            if (r.text)
+                resolve(r.text);
+            else
+                reject(new Error(r.error ?? "claude CLI failed"));
+        }));
+    });
 }

package/dist/report.d.ts

@@ -1,5 +1,6 @@
 import type { AuditResult } from "./eval.ts";
 import type { FixResult } from "./fix.ts";
+import type { ContextAuditResult } from "./context.ts";
 export interface RenderOpts {
     showCost?: boolean;
 }
@@ -7,4 +8,6 @@ export declare function renderTable(a: AuditResult, opts?: RenderOpts): string;
 /** Markdown table — paste into a PR or README. */
 export declare function renderMarkdown(a: AuditResult, opts?: RenderOpts): string;
 export declare function renderJson(a: AuditResult | unknown): string;
+/** activation-rate-by-context: isolation vs co-loaded, with a Fisher's-exact p on the drop. */
+export declare function renderContext(a: ContextAuditResult, opts?: RenderOpts): string;
 export declare function renderFix(f: FixResult): string;

package/dist/report.js

@@ -2,6 +2,7 @@
  * errors are shown distinctly from behavioral results. */
 import { wilsonInterval } from "./stats.js";
 const pct = (x) => `${Math.round(x * 100)}%`;
+const pval = (p) => (p < 0.001 ? "<0.001" : p.toFixed(3));
 const LABEL = {
     pass: "PASS        ", fail: "FAIL        ",
     inconclusive: "INCONCLUSIVE", error: "ERROR       ",
@@ -80,6 +81,50 @@ export function renderMarkdown(a, opts = {}) {
 export function renderJson(a) {
     return JSON.stringify(a, null, 2);
 }
+/** activation-rate-by-context: isolation vs co-loaded, with a Fisher's-exact p on the drop. */
+export function renderContext(a, opts = {}) {
+    const showCost = opts.showCost ?? true;
+    const out = [];
+    out.push(`skill-probe context — runtime: ${a.runtime}  model: ${a.model}  ` +
+        `threshold: ${pct(a.threshold)}  library: ${a.librarySize} skills`);
+    out.push(`  isolation = only that skill loaded · co-loaded = the full library (${a.librarySize}) loaded`);
+    out.push("");
+    const ci = (r) => `${pct(r.pHat)} [${pct(r.ciLow)}, ${pct(r.ciHigh)}] k=${r.k}`;
+    for (const c of a.cases) {
+        if (c.untrustworthy) {
+            out.push(`  [⛔ ERROR]  ${c.expected}  | ${c.prompt}`);
+            out.push(`        infrastructure errors dominated — comparison NOT trustworthy ` +
+                `(isolation ${c.isolation.stats.errors} err/${c.isolation.stats.n} valid, ` +
+                `co-loaded ${c.coLoaded.stats.errors} err/${c.coLoaded.stats.n} valid)`);
+            const reason = c.isolation.stats.lastError ?? c.coLoaded.stats.lastError;
+            if (reason)
+                out.push(`        reason: ${reason}`);
+            continue;
+        }
+        const flag = c.interference ? "⚠ INTERFERENCE" : "ok";
+        out.push(`  [${flag}]  ${c.expected}  | ${c.prompt}`);
+        out.push(`        isolation ${ci(c.isolation.rel)}`);
+        out.push(`        co-loaded ${ci(c.coLoaded.rel)}`);
+        out.push(`        Δ ${c.deltaP >= 0 ? "+" : ""}${pct(c.deltaP)} under load   Fisher p=${pval(c.fisherP)}`);
+        if (c.interference) {
+            const thieves = Object.keys(c.coLoaded.stats.dist)
+                .filter((s) => s !== c.expected && s !== "None");
+            out.push(`        ↳ fires reliably alone but is suppressed when the library is co-loaded` +
+                (thieves.length ? ` (stolen by: ${thieves.join(", ")})` : ""));
+        }
+    }
+    if (a.skipped.length) {
+        out.push("");
+        out.push(`  skipped ${a.skipped.length} decoy case(s) — no single skill to isolate.`);
+    }
+    out.push("");
+    const interfN = a.cases.filter((c) => c.interference).length;
+    const errN = a.cases.filter((c) => c.untrustworthy).length;
+    out.push(`Result: ${interfN} interference / ${a.cases.length} measured` +
+        (errN ? ` / ${errN} error` : "") + `  |  exit ${a.exitCode}` +
+        (showCost ? `  |  cost $${a.totalCost.toFixed(4)}` : ""));
+    return out.join("\n");
+}
 export function renderFix(f) {
     const out = [];
     out.push(`skill-probe fix — skill: ${f.skill}`);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skill-probe",
-  "version": "0.3.0",
+  "version": "0.5.0",
   "description": "Runtime-agnostic, co-loaded-aware auditor for AI agent skill libraries: measures which skill actually fires, with statistical confidence.",
   "type": "module",
   "bin": { "skill-probe": "dist/cli.js" },