skill-probe 0.7.1 → 0.8.0

package/README.md

@@ -28,7 +28,8 @@ doctor → gen → audit → context → diagnose → fix
 - `doctor`: verify setup, auth, runtime, skills, and config before spending probes.
 - `gen`: draft a probe config from your existing skills.
 - `audit`: measure which skill actually fires when the whole library is co-loaded.
-- `context`: compare isolation vs co-loaded activation to catch library interference.
+- `context`: compare isolation vs co-loaded activation to catch library interference —
+  add `--ablate` to name **which** sibling is stealing the trigger.
 - `diagnose`: explain whether a failure is a routing miss or a description problem.
 - `fix`: rewrite a skill description and keep it only if the measured reliability improves.
 
@@ -179,16 +180,20 @@ skill-probe context --config probe.config.json
 
 For each case it measures the expected skill **in isolation** (a throwaway project with only that
 one skill) **and co-loaded** (your full library), then tests the drop with **Fisher's exact test**
-— so interference is reported as a real effect with a p-value, not eyeballed:
+— so interference is reported as a real effect with a p-value, not eyeballed. Across multiple
+cases the p-values are **Benjamini-Hochberg corrected** (one Fisher test per case is a *family*:
+raw `p<0.05` over a 40-case library expects ~2 false flags by chance alone — interference is
+decided on the adjusted p, and the report shows both):
 
 ```
 skill-probe context — runtime: claude-code  model: (runtime default)  threshold: 70%  library: 4 skills
   isolation = only that skill loaded · co-loaded = the full library (4) loaded
+  3 comparisons — interference decided on Benjamini-Hochberg adjusted p<0.05 (raw p across a family overcounts)
 
   [⚠ INTERFERENCE]  greeter  | write a birthday greeting
         isolation 100% [72%, 100%] k=10
         co-loaded 30% [11%, 60%] k=10
-        Δ -70% under load   Fisher p=0.003
+        Δ -70% under load   Fisher p=0.003 · BH-adj p=0.009
         ↳ fires reliably alone but is suppressed when the library is co-loaded (stolen by: welcomer)
 
 Result: 1 interference / 3 measured  |  exit 1
@@ -200,6 +205,38 @@ in the library is a config error (typo), surfaced before any probes run. Exit `1
 shows interference, `2` if a case is untrustworthy (infra errors dominated — never a silent pass),
 else `0`.
 
+### Name the thief (`--ablate`)
+
+Knowing *that* a skill is suppressed is half the diagnosis — the other half is **which sibling is
+stealing it**. With 3 skills you can guess; with 40 you can't. `--ablate` answers it with evidence:
+
+```bash
+skill-probe context --config probe.config.json --ablate
+```
+
+For each interference case it takes the suspects (ranked by how many probes they actually stole in
+the co-loaded run, capped by `--suspects`, default 3), removes them from the library **one at a
+time**, and re-measures. If activation recovers significantly (Fisher vs the co-loaded run,
+BH-corrected across all ablation runs), that sibling is the thief — named, with a p-value:
+
+```
+  [⚠ INTERFERENCE]  greeter  | write a greeting for my new teammate
+        isolation 100% [61%, 100%] k=6
+        co-loaded 0% [0%, 56%] k=3
+        Δ -100% under load   Fisher p=0.012
+        ↳ fires reliably alone but is suppressed when the library is co-loaded (stolen by: welcomer)
+        leave-one-out (each suspect removed, recovery vs co-loaded):
+          − welcomer removed → 100% [61%, 100%] k=6  Δ+100%  p=0.012 · adj 0.012  ← THIEF (full recovery)
+
+Result: 1 interference / 1 measured  |  thieves named: welcomer  |  exit 1
+```
+
+(That's a real run.) Honest edge cases: if the co-loaded probes were suppressed to **None** —
+nothing stole them, the skill just stops firing under load — there's no suspect to remove, and the
+report says *"dilution, not theft; nothing to ablate"* instead of guessing. If no single removal
+recovers activation, it says so ("interference may be combinatorial") rather than naming an
+innocent sibling.
+
 ## Why a skill fails (`skill-probe diagnose`)
 
 When a skill doesn't fire, there are two very different root causes — and the activation rate alone
@@ -281,11 +318,15 @@ should-fire prompts, near-misses, and decoys.
 
 **Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping + four-state verdict
 (pass / fail / inconclusive / **error**), across two runtimes (Claude Code, OpenCode).
-Infrastructure failures (timeout / auth / crash / empty output) are reported as `error`, never as
-a behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
+Infrastructure failures (timeout / auth / crash / empty output / a zero-cost response where the
+model never actually ran, e.g. a usage-limit window) are reported as `error`, never as a
+behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
 
 **Context (`skill-probe context`):** isolation-vs-co-loaded activation rates, with **Fisher's exact
-test** on the drop — catches skills that fire alone but are suppressed under the full library's load.
+test** on the drop and **Benjamini-Hochberg correction** across cases — catches skills that fire
+alone but are suppressed under the full library's load, without buying false flags on big libraries.
+With `--ablate`, leave-one-out re-measurement **names the thieving sibling** when removing it
+significantly restores activation.
 
 **Diagnose (`skill-probe diagnose`):** compares actual runtime activation against intended
 forced-choice routing to classify failures as routing-miss vs description-problem. Intended routing
@@ -294,9 +335,8 @@ is a heuristic, not ground truth, and the README calls that out explicitly.
 **Fix (`skill-probe fix`):** uses the **Bayesian Beta-Binomial** to gate description rewrites on a
 *proven* lift (interleaved before/after, applied only if P(improvement) clears the bar).
 
-**Implemented + unit-tested but NOT yet used by any command:** Benjamini-Hochberg FDR
-(`src/stats.ts`) — reserved for cross-case multiplicity control (a planned enhancement). Don't read
-its presence as the audit using it. (Fisher's exact test is now wired into `context`.)
+Every statistical function in `src/stats.ts` is now wired into a command: Wilson + sequential
+stopping (audit), Fisher's exact + Benjamini-Hochberg (context/ablate), Bayesian Beta-Binomial (fix).
 
 ## Dev
 

package/dist/adapters/claudeCode.d.ts

@@ -1,4 +1,4 @@
-import type { RuntimeAdapter } from "../types.ts";
+import type { ProbeResult, RuntimeAdapter } from "../types.ts";
 export interface ParseInfo {
     skillFired: string | null;
     trajectory: string[];
@@ -7,4 +7,11 @@ export interface ParseInfo {
     sawAssistant: boolean;
 }
 export declare function parseStream(stdout: string): ParseInfo;
+/** Decide what a finished (non-timeout) probe means. A probe is a VALID behavioral outcome only
+ * if there is evidence the model actually ran: assistant output AND a nonzero billed cost. A
+ * zero-cost "response" (observed live 2026-07-02: a usage-limit window returned assistant text
+ * with total_cost_usd=0 and no tool_use, for several minutes) means the model never really ran —
+ * counting it as "no skill fired" silently poisons every verdict. It must surface as an infra
+ * error → untrustworthy, never as a behavioral result. */
+export declare function interpretProbe(info: ParseInfo, stderrText: string): ProbeResult;
 export declare const claudeCodeAdapter: RuntimeAdapter;

package/dist/adapters/claudeCode.js

@@ -39,6 +39,27 @@ export function parseStream(stdout) {
     }
     return { skillFired, trajectory, cost, sawInit, sawAssistant };
 }
+/** Decide what a finished (non-timeout) probe means. A probe is a VALID behavioral outcome only
+ * if there is evidence the model actually ran: assistant output AND a nonzero billed cost. A
+ * zero-cost "response" (observed live 2026-07-02: a usage-limit window returned assistant text
+ * with total_cost_usd=0 and no tool_use, for several minutes) means the model never really ran —
+ * counting it as "no skill fired" silently poisons every verdict. It must surface as an infra
+ * error → untrustworthy, never as a behavioral result. */
+export function interpretProbe(info, stderrText) {
+    if (!info.sawAssistant) {
+        return {
+            status: "error", skillFired: null, trajectory: info.trajectory, cost: info.cost,
+            error: stderrText.trim() ? stderrText.trim().slice(-200) : "no assistant output (auth/config/empty response?)",
+        };
+    }
+    if (info.cost === 0) {
+        return {
+            status: "error", skillFired: null, trajectory: info.trajectory, cost: 0,
+            error: "zero-cost response — the model did not actually run (usage limit / degraded CLI state?)",
+        };
+    }
+    return { status: "ok", skillFired: info.skillFired, trajectory: info.trajectory, cost: info.cost };
+}
 export const claudeCodeAdapter = {
     name: "claude-code",
     probe(prompt, { cwd, timeoutMs = 120_000, model }) {
@@ -71,13 +92,7 @@ export const claudeCodeAdapter = {
                         error: `timed out after ${timeoutMs}ms`,
                     });
                 }
-                if (!info.sawAssistant) {
-                    return finish({
-                        status: "error", skillFired: null, trajectory: info.trajectory, cost: info.cost,
-                        error: err.trim() ? err.trim().slice(-200) : "no assistant output (auth/config/empty response?)",
-                    });
-                }
-                finish({ status: "ok", skillFired: info.skillFired, trajectory: info.trajectory, cost: info.cost });
+                finish(interpretProbe(info, err));
             });
         });
     },

package/dist/cli.js

@@ -17,7 +17,7 @@ const HELP = `skill-probe — audit a co-loaded agent skill library by real acti
 
 USAGE
   skill-probe --config <file.json> [options]                 # audit
-  skill-probe context --config <file.json> [options]         # isolation vs co-loaded activation rates
+  skill-probe context --config <file.json> [--ablate]        # isolation vs co-loaded activation rates
   skill-probe diagnose --config <file.json> [options]        # why a skill fails: routing-miss vs description
   skill-probe doctor [--cwd <dir>] [--config <file.json>]    # preflight: setup/auth/config sanity
   skill-probe gen [--cwd <dir>] > probe.config.json          # draft a test config from your skills
@@ -36,6 +36,11 @@ OPTIONS
       --markdown           emit a Markdown table (paste into a PR/README)
       --no-cost            hide the cost line (e.g. when on a Claude subscription)
       --quiet              suppress the live progress line on stderr
+  context only:
+      --ablate             leave-one-out: for each interference case, re-run with each suspect
+                           sibling removed and NAME the thief when activation recovers (Fisher +
+                           Benjamini-Hochberg corrected)
+      --suspects <n>       max siblings to ablate per case, ranked by probes stolen (default 3)
   doctor only:
       --cwd <dir>          project dir containing .claude/skills/ (default: .)
       --config <file>      also sanity-check a config (expected skills exist, k/threshold feasible)
@@ -74,7 +79,8 @@ function makeContextProgress(values) {
         return undefined;
     return (e) => {
         const p = e.prompt.length > 28 ? e.prompt.slice(0, 25) + "…" : e.prompt;
-        const line = `  [${e.caseIndex}/${e.caseTotal}] ${e.condition.padEnd(9)} "${p}" — run ${e.validN}/${e.maxK}`;
+        const cond = e.condition === "ablation" ? `−${e.removed}` : e.condition;
+        const line = `  [${e.caseIndex}/${e.caseTotal}] ${cond.padEnd(9)} "${p}" — run ${e.validN}/${e.maxK}`;
         const pad = Math.max(0, progressLen - line.length);
         process.stderr.write("\r" + line + " ".repeat(pad));
         progressLen = line.length;
@@ -137,7 +143,12 @@ async function context(values) {
         k: values["k"], threshold: values["threshold"],
         conf: values["conf"],
     });
-    const result = await runContextAudit(cfg, makeContextProgress(values));
+    const suspects = parseIntFlag(values["suspects"], "--suspects", 3, 1);
+    const onProgress = makeContextProgress(values);
+    const result = await runContextAudit(cfg, {
+        ablate: Boolean(values["ablate"]), suspects,
+        ...(onProgress ? { onProgress } : {}),
+    });
     clearProgress();
     const showCost = !values["no-cost"];
     console.log(values["json"] ? renderJson(result) : renderContext(result, { showCost }));
@@ -191,6 +202,8 @@ async function main() {
             cwd: { type: "string" },
             "per-skill": { type: "string" },
             decoys: { type: "string" },
+            ablate: { type: "boolean", default: false },
+            suspects: { type: "string" },
             "skip-probe": { type: "boolean", default: false },
             json: { type: "boolean", default: false },
             markdown: { type: "boolean", default: false },

package/dist/context.d.ts

@@ -12,6 +12,20 @@ interface Condition {
     stats: RunStats;
     rel: Rel;
 }
+/** One leave-one-out run: the flagged case re-measured with ONE suspect sibling removed from the
+ * library. A significant recovery (BH-corrected Fisher vs the co-loaded run) names that sibling
+ * as the thief — evidence, not the trigger-theft heuristic. */
+export interface AblationResult {
+    removed: string;
+    stolen: number;
+    stats: RunStats;
+    rel: Rel;
+    deltaVsCoLoaded: number;
+    fisherP: number;
+    fisherPAdj: number;
+    untrustworthy: boolean;
+    culprit: boolean;
+}
 export interface ContextCaseResult {
     prompt: string;
     expected: string;
@@ -19,10 +33,17 @@ export interface ContextCaseResult {
     coLoaded: Condition;
     deltaP: number;
     fisherP: number;
+    /** Benjamini-Hochberg adjusted p across all trustworthy comparisons in this run — one Fisher
+     * test per case is a FAMILY, and raw p<0.05 over a 40-case library expects ~2 false flags by
+     * chance alone. Interference is decided on THIS value. (1 for untrustworthy cases.) */
+    fisherPAdj: number;
     /** infra failures dominated one side — the comparison is not trustworthy (NOT a behavioral result) */
     untrustworthy: boolean;
-    /** fires reliably alone but drops significantly when the full library is co-loaded */
+    /** fires reliably alone but drops significantly (BH-corrected) when the library is co-loaded */
     interference: boolean;
+    /** leave-one-out runs (only with { ablate: true }, only for interference cases). Empty array =
+     * ablation requested but no named suspect stole probes (suppressed to None — dilution, not theft). */
+    ablation?: AblationResult[];
 }
 export interface ContextAuditResult {
     runtime: string;
@@ -40,9 +61,19 @@ export interface ContextProgress {
     caseIndex: number;
     caseTotal: number;
     prompt: string;
-    condition: "isolation" | "co-loaded";
+    condition: "isolation" | "co-loaded" | "ablation";
     validN: number;
     maxK: number;
+    /** the sibling removed for this run (condition === "ablation" only) */
+    removed?: string;
+}
+export interface ContextOpts {
+    /** leave-one-out: for each interference case, re-measure with each suspect sibling removed and
+     * name the thief when activation recovers significantly. */
+    ablate?: boolean;
+    /** max suspect siblings to ablate per flagged case, ranked by probes stolen (default 3) */
+    suspects?: number;
+    onProgress?: (e: ContextProgress) => void;
 }
-export declare function runContextAudit(cfg: Config, onProgress?: (e: ContextProgress) => void): Promise<ContextAuditResult>;
+export declare function runContextAudit(cfg: Config, optsOrProgress?: ContextOpts | ((e: ContextProgress) => void)): Promise<ContextAuditResult>;
 export {};

package/dist/context.js

@@ -6,7 +6,7 @@
 import { mkdtempSync, mkdirSync, cpSync, rmSync, existsSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
-import { reliability, fisherExact2x2 } from "./stats.js";
+import { reliability, fisherExact2x2, bhAdjust } from "./stats.js";
 import { measure, infraUntrustworthy } from "./orchestrator.js";
 import { getAdapter } from "./adapters/index.js";
 import { listSkills } from "./gen.js";
@@ -25,7 +25,9 @@ export function buildContextDir(srcCwd, skillNames) {
     }
     return { cwd: root, cleanup: () => rmSync(root, { recursive: true, force: true }) };
 }
-export async function runContextAudit(cfg, onProgress) {
+export async function runContextAudit(cfg, optsOrProgress = {}) {
+    const opts = typeof optsOrProgress === "function" ? { onProgress: optsOrProgress } : optsOrProgress;
+    const onProgress = opts.onProgress;
     const adapter = getAdapter(cfg.runtime);
     const library = listSkills(cfg.cwd);
     const present = new Set(library.map((s) => s.name));
@@ -82,17 +84,80 @@ export async function runContextAudit(cfg, onProgress) {
         // if infra failures dominated either side, the comparison isn't a behavioral result at all —
         // it must never read as "no interference / pass" just because fisherP defaulted to 1.
         const untrustworthy = infraUntrustworthy(iso) || infraUntrustworthy(co);
-        const interference = !untrustworthy &&
-            isoRel.pHat >= cfg.threshold && // fires reliably alone
-            deltaP < 0 && // and drops when co-loaded
-            fisherP < 0.05; // and the drop is statistically real
+        // interference is decided AFTER the loop, once the whole family of comparisons is known
+        // (Benjamini-Hochberg) — placeholders here.
         measured.push({
             prompt: c.prompt, expected,
             isolation: { stats: iso, rel: isoRel },
             coLoaded: { stats: co, rel: coRel },
-            deltaP, fisherP, untrustworthy, interference,
+            deltaP, fisherP, fisherPAdj: 1, untrustworthy, interference: false,
         });
     }
+    // One Fisher test per case = a family of comparisons: correct across the trustworthy ones so a
+    // big library doesn't buy false interference flags by chance (raw p<0.05 × 40 cases ≈ 2 flukes).
+    const trustworthy = measured.filter((m) => !m.untrustworthy);
+    const adj = bhAdjust(trustworthy.map((m) => m.fisherP));
+    trustworthy.forEach((m, j) => { m.fisherPAdj = adj[j]; });
+    for (const m of measured) {
+        m.interference =
+            !m.untrustworthy &&
+                m.isolation.rel.pHat >= cfg.threshold && // fires reliably alone
+                m.deltaP < 0 && // and drops when co-loaded
+                m.fisherPAdj < 0.05; // and the drop survives BH correction
+    }
+    // leave-one-out ablation: for each flagged case, re-measure with each suspect sibling removed.
+    // Suspects are ranked by the probes they actually stole in the co-loaded run — evidence-guided,
+    // not a sweep of the whole library. Recovery is Fisher-tested against the co-loaded run and
+    // BH-corrected across all ablation runs (their own family). A significant recovery NAMES the
+    // thief; no suspects in the outcome distribution means the drop was dilution, not theft.
+    if (opts.ablate) {
+        const libraryNames = library.map((s) => s.name);
+        const flagged = measured.filter((m) => m.interference);
+        const cap = opts.suspects ?? 3;
+        const allRuns = [];
+        for (let fi = 0; fi < flagged.length; fi++) {
+            const m = flagged[fi];
+            const suspects = Object.entries(m.coLoaded.stats.dist)
+                .filter(([name]) => name !== m.expected && name !== "None")
+                .sort((x, y) => y[1] - x[1])
+                .slice(0, cap);
+            m.ablation = [];
+            for (const [suspect, stolen] of suspects) {
+                const ctx = buildContextDir(cfg.cwd, libraryNames.filter((n) => n !== suspect));
+                let st;
+                try {
+                    st = await measure(adapter, m.prompt, m.expected, {
+                        cwd: ctx.cwd, ...common,
+                        ...(onProgress ? { onProbe: (validN, maxK) => onProgress({ caseIndex: fi + 1, caseTotal: flagged.length, prompt: m.prompt, condition: "ablation", removed: suspect, validN, maxK }) } : {}),
+                    });
+                }
+                finally {
+                    ctx.cleanup();
+                }
+                totalCost += st.totalCost;
+                const rel = reliability(st.hits, st.n, cfg.conf);
+                const co = m.coLoaded.stats;
+                const fisherP = (st.n > 0 && co.n > 0)
+                    ? fisherExact2x2(st.hits, st.n - st.hits, co.hits, co.n - co.hits)
+                    : 1;
+                const abl = {
+                    removed: suspect, stolen, stats: st, rel,
+                    deltaVsCoLoaded: rel.pHat - m.coLoaded.rel.pHat,
+                    fisherP, fisherPAdj: 1,
+                    untrustworthy: infraUntrustworthy(st),
+                    culprit: false,
+                };
+                m.ablation.push(abl);
+                allRuns.push(abl);
+            }
+        }
+        const ablFamily = allRuns.filter((r) => !r.untrustworthy);
+        const ablAdj = bhAdjust(ablFamily.map((r) => r.fisherP));
+        ablFamily.forEach((r, j) => { r.fisherPAdj = ablAdj[j]; });
+        for (const r of allRuns) {
+            r.culprit = !r.untrustworthy && r.deltaVsCoLoaded > 0 && r.fisherPAdj < 0.05;
+        }
+    }
     // precedence mirrors the audit: a real behavioral signal (interference, exit 1) outranks an
     // untrustworthy/infra case (exit 2), which outranks a clean run (exit 0).
     const exitCode = measured.some((m) => m.interference) ? 1

package/dist/report.js

@@ -131,6 +131,11 @@ export function renderContext(a, opts = {}) {
     out.push(`skill-probe context — runtime: ${a.runtime}  model: ${a.model}  ` +
         `threshold: ${pct(a.threshold)}  library: ${a.librarySize} skills`);
     out.push(`  isolation = only that skill loaded · co-loaded = the full library (${a.librarySize}) loaded`);
+    const familyN = a.cases.filter((c) => !c.untrustworthy).length;
+    if (familyN > 1) {
+        out.push(`  ${familyN} comparisons — interference decided on Benjamini-Hochberg adjusted p<0.05` +
+            ` (raw p across a family overcounts)`);
+    }
     out.push("");
     const ci = (r) => `${pct(r.pHat)} [${pct(r.ciLow)}, ${pct(r.ciHigh)}] k=${r.k}`;
     for (const c of a.cases) {
@@ -148,12 +153,37 @@ export function renderContext(a, opts = {}) {
         out.push(`  [${flag}]  ${c.expected}  | ${c.prompt}`);
         out.push(`        isolation ${ci(c.isolation.rel)}`);
         out.push(`        co-loaded ${ci(c.coLoaded.rel)}`);
-        out.push(`        Δ ${c.deltaP >= 0 ? "+" : ""}${pct(c.deltaP)} under load   Fisher p=${pval(c.fisherP)}`);
+        out.push(`        Δ ${c.deltaP >= 0 ? "+" : ""}${pct(c.deltaP)} under load   Fisher p=${pval(c.fisherP)}` +
+            (familyN > 1 ? ` · BH-adj p=${pval(c.fisherPAdj)}` : ""));
         if (c.interference) {
             const thieves = Object.keys(c.coLoaded.stats.dist)
                 .filter((s) => s !== c.expected && s !== "None");
             out.push(`        ↳ fires reliably alone but is suppressed when the library is co-loaded` +
                 (thieves.length ? ` (stolen by: ${thieves.join(", ")})` : ""));
+            if (c.ablation) {
+                if (c.ablation.length === 0) {
+                    out.push(`        leave-one-out: no sibling stole these probes (suppressed to None) — ` +
+                        `dilution, not theft; nothing to ablate`);
+                }
+                else {
+                    out.push(`        leave-one-out (each suspect removed, recovery vs co-loaded):`);
+                    for (const abl of c.ablation) {
+                        if (abl.untrustworthy) {
+                            out.push(`          − ${abl.removed}: infra errors dominated — run not trustworthy`);
+                            continue;
+                        }
+                        const tag = abl.culprit
+                            ? (abl.rel.pHat >= a.threshold ? "  ← THIEF (full recovery)" : "  ← THIEF (partial recovery)")
+                            : "";
+                        out.push(`          − ${abl.removed} removed → ${ci(abl.rel)}  ` +
+                            `Δ${abl.deltaVsCoLoaded >= 0 ? "+" : ""}${pct(abl.deltaVsCoLoaded)}  ` +
+                            `p=${pval(abl.fisherP)} · adj ${pval(abl.fisherPAdj)}${tag}`);
+                    }
+                    if (!c.ablation.some((abl) => abl.culprit)) {
+                        out.push(`          no single culprit confirmed — interference may be combinatorial`);
+                    }
+                }
+            }
         }
     }
     if (a.skipped.length) {
@@ -163,8 +193,11 @@ export function renderContext(a, opts = {}) {
     out.push("");
     const interfN = a.cases.filter((c) => c.interference).length;
     const errN = a.cases.filter((c) => c.untrustworthy).length;
+    const culprits = [...new Set(a.cases.flatMap((c) => (c.ablation ?? []).filter((abl) => abl.culprit).map((abl) => abl.removed)))];
     out.push(`Result: ${interfN} interference / ${a.cases.length} measured` +
-        (errN ? ` / ${errN} error` : "") + `  |  exit ${a.exitCode}` +
+        (errN ? ` / ${errN} error` : "") +
+        (culprits.length ? `  |  thieves named: ${culprits.join(", ")}` : "") +
+        `  |  exit ${a.exitCode}` +
         (showCost ? `  |  cost $${a.totalCost.toFixed(4)}` : ""));
     return out.join("\n");
 }

package/dist/stats.d.ts

@@ -30,5 +30,9 @@ export interface ImprovementResult {
 }
 /** Bayesian Beta-Binomial: P(p_after > p_before) + delta credible interval. */
 export declare function probImprovement(xBefore: number, nBefore: number, xAfter: number, nAfter: number, prior?: number, samples?: number, seed?: number): ImprovementResult;
+/** Benjamini-Hochberg step-up ADJUSTED p-values: adj[i] = min over ranks j>=rank(i) of m*p_(j)/j.
+ * adj[i] <= q ⟺ benjaminiHochberg(pvals, q) rejects i, but the adjusted value is reportable
+ * next to the raw p so the multiple-comparisons correction is visible, not silent. */
+export declare function bhAdjust(pvals: number[]): number[];
 /** Benjamini-Hochberg FDR. Returns booleans (true = flag as real). */
 export declare function benjaminiHochberg(pvals: number[], q?: number): boolean[];

package/dist/stats.js

@@ -135,6 +135,23 @@ export function probImprovement(xBefore, nBefore, xAfter, nAfter, prior = 0.5, s
         deltaCi: [deltas[Math.floor(0.025 * samples)], deltas[Math.floor(0.975 * samples)]],
     };
 }
+/** Benjamini-Hochberg step-up ADJUSTED p-values: adj[i] = min over ranks j>=rank(i) of m*p_(j)/j.
+ * adj[i] <= q ⟺ benjaminiHochberg(pvals, q) rejects i, but the adjusted value is reportable
+ * next to the raw p so the multiple-comparisons correction is visible, not silent. */
+export function bhAdjust(pvals) {
+    const m = pvals.length;
+    if (m === 0)
+        return [];
+    const order = [...pvals.keys()].sort((i, j) => pvals[i] - pvals[j]);
+    const adj = new Array(m).fill(1);
+    let running = 1;
+    for (let idx = m - 1; idx >= 0; idx--) {
+        const i = order[idx];
+        running = Math.min(running, (pvals[i] * m) / (idx + 1));
+        adj[i] = Math.min(1, running);
+    }
+    return adj;
+}
 /** Benjamini-Hochberg FDR. Returns booleans (true = flag as real). */
 export function benjaminiHochberg(pvals, q = 0.05) {
     const m = pvals.length;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skill-probe",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "description": "Runtime-agnostic, co-loaded-aware auditor for AI agent skill libraries: measures which skill actually fires, with statistical confidence.",
   "type": "module",
   "bin": {