skill-probe 0.7.1 → 0.9.0

package/README.md

@@ -27,8 +27,10 @@ doctor → gen → audit → context → diagnose → fix
 
 - `doctor`: verify setup, auth, runtime, skills, and config before spending probes.
 - `gen`: draft a probe config from your existing skills.
-- `audit`: measure which skill actually fires when the whole library is co-loaded.
-- `context`: compare isolation vs co-loaded activation to catch library interference.
+- `audit`: measure which skill actually fires when the whole library is co-loaded — with
+  `--baseline`, gate CI on statistically significant regressions only.
+- `context`: compare isolation vs co-loaded activation to catch library interference —
+  add `--ablate` to name **which** sibling is stealing the trigger.
 - `diagnose`: explain whether a failure is a routing miss or a description problem.
 - `fix`: rewrite a skill description and keep it only if the measured reliability improves.
 
@@ -156,16 +158,64 @@ Claude Code vs OpenCode comparison is fair — otherwise you're comparing two co
 
 ## CI usage
 
-skill-probe exits non-zero on a real problem, so it drops straight into a pipeline:
+skill-probe exits non-zero on a real problem, so it drops straight into a pipeline. Exit codes:
+**`0`** all pass · **`1`** a behavioral fail / trigger-theft / interference / regression · **`2`**
+inconclusive or an infrastructure error — so a runtime outage *fails* the build instead of silently
+passing. Add `--json` to archive the full result (it includes a **run manifest**: tool version,
+model, date, config hash — so two runs are comparable and a report is citable).
+
+### Regression gating (`--baseline`)
+
+The CI question isn't "are my skills perfect?" — it's **"did this PR make any skill worse?"**
+Activation is stochastic, so comparing raw rates makes CI flaky. The baseline gate compares each
+case against a saved baseline with **Fisher's exact test, Benjamini-Hochberg corrected** — noise
+passes, real drops fail:
+
+```bash
+# once, on a good main build (commit the file):
+skill-probe --config probe.config.json --save-baseline baselines/main.json
+
+# on every PR:
+skill-probe --config probe.config.json --baseline baselines/main.json
+```
+
+```
+Baseline gate — vs baseline saved 2026-07-02T09:12:03Z (skill-probe 0.9.0)
+  [▼ REGRESSED]  greeter  | write a birthday greeting
+        90% (9/10) → 20% (2/10)   Δ-70%   p=0.005 · adj 0.005
+Gate: ▼ FAIL — 1 significant regression(s)
+```
+
+The gate warns (but still runs, matching cases by prompt) if the config, runtime, or model changed
+since the baseline — apples-to-oranges comparisons are flagged, never silent. Significant
+*improvements* are reported too, as a nudge to re-save the baseline.
+
+### GitHub Action
 
 ```yaml
-# .github/workflows/skills.yml  (the claude/opencode CLI must be installed + authed on the runner)
-- run: npx skill-probe --config probe.config.json --quiet --no-cost
+# .github/workflows/skills.yml
+name: skills
+on: pull_request
+jobs:
+  probe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with: { node-version: "22" }
+      - uses: HystonKayange/skill-probe@main
+        with:
+          config: probe.config.json
+          args: --baseline baselines/main.json
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
 ```
 
-Exit codes: **`0`** all pass · **`1`** a behavioral fail / trigger-theft / interference · **`2`**
-inconclusive or an infrastructure error — so a runtime outage *fails* the build instead of silently
-passing. Add `--json` to archive the full result as a build artifact.
+The action installs skill-probe plus the runtime CLI (Claude Code by default; set
+`runtime-package: ""` if your runner already has one) and runs the command. `command:` selects
+`audit` (default), `context`, `diagnose`, or `doctor`; `args:` passes extra flags like `--ablate`.
+The runtime authenticates via `ANTHROPIC_API_KEY` — headless probes each cost real tokens, so
+smoke settings (`k: 10`, few cases) are the sweet spot for per-PR gating.
 
 ## Activation rate by context (`skill-probe context`)
 
@@ -179,16 +229,20 @@ skill-probe context --config probe.config.json
 
 For each case it measures the expected skill **in isolation** (a throwaway project with only that
 one skill) **and co-loaded** (your full library), then tests the drop with **Fisher's exact test**
-— so interference is reported as a real effect with a p-value, not eyeballed:
+— so interference is reported as a real effect with a p-value, not eyeballed. Across multiple
+cases the p-values are **Benjamini-Hochberg corrected** (one Fisher test per case is a *family*:
+raw `p<0.05` over a 40-case library expects ~2 false flags by chance alone — interference is
+decided on the adjusted p, and the report shows both):
 
 ```
 skill-probe context — runtime: claude-code  model: (runtime default)  threshold: 70%  library: 4 skills
   isolation = only that skill loaded · co-loaded = the full library (4) loaded
+  3 comparisons — interference decided on Benjamini-Hochberg adjusted p<0.05 (raw p across a family overcounts)
 
   [⚠ INTERFERENCE]  greeter  | write a birthday greeting
         isolation 100% [72%, 100%] k=10
         co-loaded 30% [11%, 60%] k=10
-        Δ -70% under load   Fisher p=0.003
+        Δ -70% under load   Fisher p=0.003 · BH-adj p=0.009
         ↳ fires reliably alone but is suppressed when the library is co-loaded (stolen by: welcomer)
 
 Result: 1 interference / 3 measured  |  exit 1
@@ -200,6 +254,38 @@ in the library is a config error (typo), surfaced before any probes run. Exit `1
 shows interference, `2` if a case is untrustworthy (infra errors dominated — never a silent pass),
 else `0`.
 
+### Name the thief (`--ablate`)
+
+Knowing *that* a skill is suppressed is half the diagnosis — the other half is **which sibling is
+stealing it**. With 3 skills you can guess; with 40 you can't. `--ablate` answers it with evidence:
+
+```bash
+skill-probe context --config probe.config.json --ablate
+```
+
+For each interference case it takes the suspects (ranked by how many probes they actually stole in
+the co-loaded run, capped by `--suspects`, default 3), removes them from the library **one at a
+time**, and re-measures. If activation recovers significantly (Fisher vs the co-loaded run,
+BH-corrected across all ablation runs), that sibling is the thief — named, with a p-value:
+
+```
+  [⚠ INTERFERENCE]  greeter  | write a greeting for my new teammate
+        isolation 100% [61%, 100%] k=6
+        co-loaded 0% [0%, 56%] k=3
+        Δ -100% under load   Fisher p=0.012
+        ↳ fires reliably alone but is suppressed when the library is co-loaded (stolen by: welcomer)
+        leave-one-out (each suspect removed, recovery vs co-loaded):
+          − welcomer removed → 100% [61%, 100%] k=6  Δ+100%  p=0.012 · adj 0.012  ← THIEF (full recovery)
+
+Result: 1 interference / 1 measured  |  thieves named: welcomer  |  exit 1
+```
+
+(That's a real run.) Honest edge cases: if the co-loaded probes were suppressed to **None** —
+nothing stole them, the skill just stops firing under load — there's no suspect to remove, and the
+report says *"dilution, not theft; nothing to ablate"* instead of guessing. If no single removal
+recovers activation, it says so ("interference may be combinatorial") rather than naming an
+innocent sibling.
+
 ## Why a skill fails (`skill-probe diagnose`)
 
 When a skill doesn't fire, there are two very different root causes — and the activation rate alone
@@ -281,11 +367,18 @@ should-fire prompts, near-misses, and decoys.
 
 **Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping + four-state verdict
 (pass / fail / inconclusive / **error**), across two runtimes (Claude Code, OpenCode).
-Infrastructure failures (timeout / auth / crash / empty output) are reported as `error`, never as
-a behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
+Infrastructure failures (timeout / auth / crash / empty output / a zero-cost response where the
+model never actually ran, e.g. a usage-limit window) are reported as `error`, never as a
+behavioral pass/fail — a decoy can't falsely pass because the runtime was down. Every run carries
+a **manifest** (version, model, date, config hash); `--save-baseline` / `--baseline` turn audits
+into a CI regression gate (Fisher + BH — noise passes, real drops fail), and a **GitHub Action**
+(`HystonKayange/skill-probe@main`) wraps the whole thing for per-PR gating.
 
 **Context (`skill-probe context`):** isolation-vs-co-loaded activation rates, with **Fisher's exact
-test** on the drop — catches skills that fire alone but are suppressed under the full library's load.
+test** on the drop and **Benjamini-Hochberg correction** across cases — catches skills that fire
+alone but are suppressed under the full library's load, without buying false flags on big libraries.
+With `--ablate`, leave-one-out re-measurement **names the thieving sibling** when removing it
+significantly restores activation.
 
 **Diagnose (`skill-probe diagnose`):** compares actual runtime activation against intended
 forced-choice routing to classify failures as routing-miss vs description-problem. Intended routing
@@ -294,9 +387,8 @@ is a heuristic, not ground truth, and the README calls that out explicitly.
 **Fix (`skill-probe fix`):** uses the **Bayesian Beta-Binomial** to gate description rewrites on a
 *proven* lift (interleaved before/after, applied only if P(improvement) clears the bar).
 
-**Implemented + unit-tested but NOT yet used by any command:** Benjamini-Hochberg FDR
-(`src/stats.ts`) — reserved for cross-case multiplicity control (a planned enhancement). Don't read
-its presence as the audit using it. (Fisher's exact test is now wired into `context`.)
+Every statistical function in `src/stats.ts` is now wired into a command: Wilson + sequential
+stopping (audit), Fisher's exact + Benjamini-Hochberg (context/ablate), Bayesian Beta-Binomial (fix).
 
 ## Dev
 

package/dist/adapters/claudeCode.d.ts

@@ -1,4 +1,4 @@
-import type { RuntimeAdapter } from "../types.ts";
+import type { ProbeResult, RuntimeAdapter } from "../types.ts";
 export interface ParseInfo {
     skillFired: string | null;
     trajectory: string[];
@@ -7,4 +7,11 @@ export interface ParseInfo {
     sawAssistant: boolean;
 }
 export declare function parseStream(stdout: string): ParseInfo;
+/** Decide what a finished (non-timeout) probe means. A probe is a VALID behavioral outcome only
+ * if there is evidence the model actually ran: assistant output AND a nonzero billed cost. A
+ * zero-cost "response" (observed live 2026-07-02: a usage-limit window returned assistant text
+ * with total_cost_usd=0 and no tool_use, for several minutes) means the model never really ran —
+ * counting it as "no skill fired" silently poisons every verdict. It must surface as an infra
+ * error → untrustworthy, never as a behavioral result. */
+export declare function interpretProbe(info: ParseInfo, stderrText: string): ProbeResult;
 export declare const claudeCodeAdapter: RuntimeAdapter;

package/dist/adapters/claudeCode.js

@@ -39,6 +39,27 @@ export function parseStream(stdout) {
     }
     return { skillFired, trajectory, cost, sawInit, sawAssistant };
 }
+/** Decide what a finished (non-timeout) probe means. A probe is a VALID behavioral outcome only
+ * if there is evidence the model actually ran: assistant output AND a nonzero billed cost. A
+ * zero-cost "response" (observed live 2026-07-02: a usage-limit window returned assistant text
+ * with total_cost_usd=0 and no tool_use, for several minutes) means the model never really ran —
+ * counting it as "no skill fired" silently poisons every verdict. It must surface as an infra
+ * error → untrustworthy, never as a behavioral result. */
+export function interpretProbe(info, stderrText) {
+    if (!info.sawAssistant) {
+        return {
+            status: "error", skillFired: null, trajectory: info.trajectory, cost: info.cost,
+            error: stderrText.trim() ? stderrText.trim().slice(-200) : "no assistant output (auth/config/empty response?)",
+        };
+    }
+    if (info.cost === 0) {
+        return {
+            status: "error", skillFired: null, trajectory: info.trajectory, cost: 0,
+            error: "zero-cost response — the model did not actually run (usage limit / degraded CLI state?)",
+        };
+    }
+    return { status: "ok", skillFired: info.skillFired, trajectory: info.trajectory, cost: info.cost };
+}
 export const claudeCodeAdapter = {
     name: "claude-code",
     probe(prompt, { cwd, timeoutMs = 120_000, model }) {
@@ -71,13 +92,7 @@ export const claudeCodeAdapter = {
                         error: `timed out after ${timeoutMs}ms`,
                     });
                 }
-                if (!info.sawAssistant) {
-                    return finish({
-                        status: "error", skillFired: null, trajectory: info.trajectory, cost: info.cost,
-                        error: err.trim() ? err.trim().slice(-200) : "no assistant output (auth/config/empty response?)",
-                    });
-                }
-                finish({ status: "ok", skillFired: info.skillFired, trajectory: info.trajectory, cost: info.cost });
+                finish(interpretProbe(info, err));
             });
         });
     },

package/dist/baseline.d.ts

@@ -0,0 +1,56 @@
+import type { AuditResult } from "./eval.ts";
+import type { RunManifest } from "./manifest.ts";
+export interface BaselineCase {
+    prompt: string;
+    expected: string | null;
+    hits: number;
+    n: number;
+}
+export interface BaselineFile {
+    manifest: RunManifest;
+    cases: BaselineCase[];
+}
+export declare function toBaseline(result: AuditResult): BaselineFile;
+export declare function saveBaseline(result: AuditResult, path: string): void;
+export declare function loadBaseline(path: string): BaselineFile;
+export interface RegressionCase {
+    prompt: string;
+    expected: string | null;
+    before: {
+        hits: number;
+        n: number;
+        pHat: number;
+    };
+    after: {
+        hits: number;
+        n: number;
+        pHat: number;
+    };
+    /** after.pHat - before.pHat (negative = dropped since baseline) */
+    delta: number;
+    fisherP: number;
+    /** BH-adjusted across all compared cases — regression is decided on THIS */
+    fisherPAdj: number;
+    /** one side has no valid probes — not comparable (never reads as "no regression") */
+    untrustworthy: boolean;
+    /** significant drop since baseline (BH-corrected) */
+    regressed: boolean;
+    /** significant rise since baseline (informational — consider re-saving the baseline) */
+    improved: boolean;
+}
+export interface BaselineComparison {
+    baselineDate: string;
+    baselineVersion: string;
+    /** apples-to-oranges warnings: config hash / runtime / model drift between runs */
+    mismatches: string[];
+    cases: RegressionCase[];
+    /** prompts in the current run with no baseline entry (measured, but not gated) */
+    newCases: string[];
+    /** prompts in the baseline that the current run no longer has */
+    missingCases: string[];
+    regressions: number;
+    improvements: number;
+    /** 1 = at least one significant regression (CI should fail) */
+    exitCode: 0 | 1;
+}
+export declare function compareToBaseline(result: AuditResult, base: BaselineFile): BaselineComparison;

package/dist/cli.js

@@ -12,12 +12,13 @@ import { runDiagnose } from "./diagnose.js";
 import { runDoctor } from "./doctor.js";
 import { listSkills, generateCases, genConfig } from "./gen.js";
 import { loadConfig, parseProbability, parseIntFlag, ConfigError } from "./config.js";
-import { renderTable, renderMarkdown, renderJson, renderFix, renderContext, renderDoctor, renderDiagnose } from "./report.js";
+import { loadBaseline, saveBaseline, compareToBaseline } from "./baseline.js";
+import { renderTable, renderMarkdown, renderJson, renderFix, renderContext, renderDoctor, renderDiagnose, renderBaseline } from "./report.js";
 const HELP = `skill-probe — audit a co-loaded agent skill library by real activation behavior.
 
 USAGE
   skill-probe --config <file.json> [options]                 # audit
-  skill-probe context --config <file.json> [options]         # isolation vs co-loaded activation rates
+  skill-probe context --config <file.json> [--ablate]        # isolation vs co-loaded activation rates
   skill-probe diagnose --config <file.json> [options]        # why a skill fails: routing-miss vs description
   skill-probe doctor [--cwd <dir>] [--config <file.json>]    # preflight: setup/auth/config sanity
   skill-probe gen [--cwd <dir>] > probe.config.json          # draft a test config from your skills
@@ -36,6 +37,16 @@ OPTIONS
       --markdown           emit a Markdown table (paste into a PR/README)
       --no-cost            hide the cost line (e.g. when on a Claude subscription)
       --quiet              suppress the live progress line on stderr
+  audit only (CI gating):
+      --save-baseline <f>  after the run, save per-case counts + a run manifest to <f>
+      --baseline <f>       compare this run against a saved baseline: per-case Fisher's exact,
+                           Benjamini-Hochberg corrected — exit 1 ONLY on a significant regression
+                           (noise passes; real drops fail)
+  context only:
+      --ablate             leave-one-out: for each interference case, re-run with each suspect
+                           sibling removed and NAME the thief when activation recovers (Fisher +
+                           Benjamini-Hochberg corrected)
+      --suspects <n>       max siblings to ablate per case, ranked by probes stolen (default 3)
   doctor only:
       --cwd <dir>          project dir containing .claude/skills/ (default: .)
       --config <file>      also sanity-check a config (expected skills exist, k/threshold feasible)
@@ -74,7 +85,8 @@ function makeContextProgress(values) {
         return undefined;
     return (e) => {
         const p = e.prompt.length > 28 ? e.prompt.slice(0, 25) + "…" : e.prompt;
-        const line = `  [${e.caseIndex}/${e.caseTotal}] ${e.condition.padEnd(9)} "${p}" — run ${e.validN}/${e.maxK}`;
+        const cond = e.condition === "ablation" ? `−${e.removed}` : e.condition;
+        const line = `  [${e.caseIndex}/${e.caseTotal}] ${cond.padEnd(9)} "${p}" — run ${e.validN}/${e.maxK}`;
         const pad = Math.max(0, progressLen - line.length);
         process.stderr.write("\r" + line + " ".repeat(pad));
         progressLen = line.length;
@@ -100,10 +112,21 @@ async function audit(values) {
     const result = await runAudit(cfg, makeProgress(values));
     clearProgress();
     const showCost = !values["no-cost"];
-    const text = values["json"] ? renderJson(result)
-        : values["markdown"] ? renderMarkdown(result, { showCost })
-            : renderTable(result, { showCost });
+    let cmp;
+    if (values["baseline"])
+        cmp = compareToBaseline(result, loadBaseline(values["baseline"]));
+    if (values["save-baseline"]) {
+        saveBaseline(result, values["save-baseline"]);
+        console.error(`baseline saved to ${values["save-baseline"]}`);
+    }
+    const text = values["json"] ? renderJson(cmp ? { ...result, baseline: cmp } : result)
+        : values["markdown"] ? renderMarkdown(result, { showCost }) + (cmp ? "\n\n" + renderBaseline(cmp) : "")
+            : renderTable(result, { showCost }) + (cmp ? "\n\n" + renderBaseline(cmp) : "");
     console.log(text);
+    // precedence mirrors the rest of the tool: a behavioral signal (fail/regression, 1) outranks
+    // inconclusive/infra (2), which outranks clean (0).
+    if (result.exitCode === 1 || cmp?.exitCode === 1)
+        return 1;
     return result.exitCode;
 }
 async function diagnose(values) {
@@ -137,7 +160,12 @@ async function context(values) {
         k: values["k"], threshold: values["threshold"],
         conf: values["conf"],
     });
-    const result = await runContextAudit(cfg, makeContextProgress(values));
+    const suspects = parseIntFlag(values["suspects"], "--suspects", 3, 1);
+    const onProgress = makeContextProgress(values);
+    const result = await runContextAudit(cfg, {
+        ablate: Boolean(values["ablate"]), suspects,
+        ...(onProgress ? { onProgress } : {}),
+    });
     clearProgress();
     const showCost = !values["no-cost"];
     console.log(values["json"] ? renderJson(result) : renderContext(result, { showCost }));
@@ -191,6 +219,10 @@ async function main() {
             cwd: { type: "string" },
             "per-skill": { type: "string" },
             decoys: { type: "string" },
+            ablate: { type: "boolean", default: false },
+            suspects: { type: "string" },
+            baseline: { type: "string" },
+            "save-baseline": { type: "string" },
             "skip-probe": { type: "boolean", default: false },
             json: { type: "boolean", default: false },
             markdown: { type: "boolean", default: false },

package/dist/context.d.ts

@@ -1,6 +1,7 @@
 import type { Config } from "./types.ts";
 import { reliability } from "./stats.ts";
 import { type RunStats } from "./orchestrator.ts";
+import { type RunManifest } from "./manifest.ts";
 /** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
  * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
 export declare function buildContextDir(srcCwd: string, skillNames: string[]): {
@@ -12,6 +13,20 @@ interface Condition {
     stats: RunStats;
     rel: Rel;
 }
+/** One leave-one-out run: the flagged case re-measured with ONE suspect sibling removed from the
+ * library. A significant recovery (BH-corrected Fisher vs the co-loaded run) names that sibling
+ * as the thief — evidence, not the trigger-theft heuristic. */
+export interface AblationResult {
+    removed: string;
+    stolen: number;
+    stats: RunStats;
+    rel: Rel;
+    deltaVsCoLoaded: number;
+    fisherP: number;
+    fisherPAdj: number;
+    untrustworthy: boolean;
+    culprit: boolean;
+}
 export interface ContextCaseResult {
     prompt: string;
     expected: string;
@@ -19,10 +34,17 @@ export interface ContextCaseResult {
     coLoaded: Condition;
     deltaP: number;
     fisherP: number;
+    /** Benjamini-Hochberg adjusted p across all trustworthy comparisons in this run — one Fisher
+     * test per case is a FAMILY, and raw p<0.05 over a 40-case library expects ~2 false flags by
+     * chance alone. Interference is decided on THIS value. (1 for untrustworthy cases.) */
+    fisherPAdj: number;
     /** infra failures dominated one side — the comparison is not trustworthy (NOT a behavioral result) */
     untrustworthy: boolean;
-    /** fires reliably alone but drops significantly when the full library is co-loaded */
+    /** fires reliably alone but drops significantly (BH-corrected) when the library is co-loaded */
     interference: boolean;
+    /** leave-one-out runs (only with { ablate: true }, only for interference cases). Empty array =
+     * ablation requested but no named suspect stole probes (suppressed to None — dilution, not theft). */
+    ablation?: AblationResult[];
 }
 export interface ContextAuditResult {
     runtime: string;
@@ -35,14 +57,26 @@ export interface ContextAuditResult {
     totalCost: number;
     /** 0 = no interference; 1 = a skill is suppressed under load; 2 = an untrustworthy (infra) case */
     exitCode: 0 | 1 | 2;
+    /** who/what/when of this run — makes reports citable and comparable */
+    manifest: RunManifest;
 }
 export interface ContextProgress {
     caseIndex: number;
     caseTotal: number;
     prompt: string;
-    condition: "isolation" | "co-loaded";
+    condition: "isolation" | "co-loaded" | "ablation";
     validN: number;
     maxK: number;
+    /** the sibling removed for this run (condition === "ablation" only) */
+    removed?: string;
+}
+export interface ContextOpts {
+    /** leave-one-out: for each interference case, re-measure with each suspect sibling removed and
+     * name the thief when activation recovers significantly. */
+    ablate?: boolean;
+    /** max suspect siblings to ablate per flagged case, ranked by probes stolen (default 3) */
+    suspects?: number;
+    onProgress?: (e: ContextProgress) => void;
 }
-export declare function runContextAudit(cfg: Config, onProgress?: (e: ContextProgress) => void): Promise<ContextAuditResult>;
+export declare function runContextAudit(cfg: Config, optsOrProgress?: ContextOpts | ((e: ContextProgress) => void)): Promise<ContextAuditResult>;
 export {};

package/dist/context.js

@@ -6,11 +6,12 @@
 import { mkdtempSync, mkdirSync, cpSync, rmSync, existsSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
-import { reliability, fisherExact2x2 } from "./stats.js";
+import { reliability, fisherExact2x2, bhAdjust } from "./stats.js";
 import { measure, infraUntrustworthy } from "./orchestrator.js";
 import { getAdapter } from "./adapters/index.js";
 import { listSkills } from "./gen.js";
 import { ConfigError } from "./config.js";
+import { buildManifest } from "./manifest.js";
 /** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
  * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
 export function buildContextDir(srcCwd, skillNames) {
@@ -25,7 +26,9 @@ export function buildContextDir(srcCwd, skillNames) {
     }
     return { cwd: root, cleanup: () => rmSync(root, { recursive: true, force: true }) };
 }
-export async function runContextAudit(cfg, onProgress) {
+export async function runContextAudit(cfg, optsOrProgress = {}) {
+    const opts = typeof optsOrProgress === "function" ? { onProgress: optsOrProgress } : optsOrProgress;
+    const onProgress = opts.onProgress;
     const adapter = getAdapter(cfg.runtime);
     const library = listSkills(cfg.cwd);
     const present = new Set(library.map((s) => s.name));
@@ -82,17 +85,80 @@ export async function runContextAudit(cfg, onProgress) {
         // if infra failures dominated either side, the comparison isn't a behavioral result at all —
         // it must never read as "no interference / pass" just because fisherP defaulted to 1.
         const untrustworthy = infraUntrustworthy(iso) || infraUntrustworthy(co);
-        const interference = !untrustworthy &&
-            isoRel.pHat >= cfg.threshold && // fires reliably alone
-            deltaP < 0 && // and drops when co-loaded
-            fisherP < 0.05; // and the drop is statistically real
+        // interference is decided AFTER the loop, once the whole family of comparisons is known
+        // (Benjamini-Hochberg) — placeholders here.
         measured.push({
             prompt: c.prompt, expected,
             isolation: { stats: iso, rel: isoRel },
             coLoaded: { stats: co, rel: coRel },
-            deltaP, fisherP, untrustworthy, interference,
+            deltaP, fisherP, fisherPAdj: 1, untrustworthy, interference: false,
         });
     }
+    // One Fisher test per case = a family of comparisons: correct across the trustworthy ones so a
+    // big library doesn't buy false interference flags by chance (raw p<0.05 × 40 cases ≈ 2 flukes).
+    const trustworthy = measured.filter((m) => !m.untrustworthy);
+    const adj = bhAdjust(trustworthy.map((m) => m.fisherP));
+    trustworthy.forEach((m, j) => { m.fisherPAdj = adj[j]; });
+    for (const m of measured) {
+        m.interference =
+            !m.untrustworthy &&
+                m.isolation.rel.pHat >= cfg.threshold && // fires reliably alone
+                m.deltaP < 0 && // and drops when co-loaded
+                m.fisherPAdj < 0.05; // and the drop survives BH correction
+    }
+    // leave-one-out ablation: for each flagged case, re-measure with each suspect sibling removed.
+    // Suspects are ranked by the probes they actually stole in the co-loaded run — evidence-guided,
+    // not a sweep of the whole library. Recovery is Fisher-tested against the co-loaded run and
+    // BH-corrected across all ablation runs (their own family). A significant recovery NAMES the
+    // thief; no suspects in the outcome distribution means the drop was dilution, not theft.
+    if (opts.ablate) {
+        const libraryNames = library.map((s) => s.name);
+        const flagged = measured.filter((m) => m.interference);
+        const cap = opts.suspects ?? 3;
+        const allRuns = [];
+        for (let fi = 0; fi < flagged.length; fi++) {
+            const m = flagged[fi];
+            const suspects = Object.entries(m.coLoaded.stats.dist)
+                .filter(([name]) => name !== m.expected && name !== "None")
+                .sort((x, y) => y[1] - x[1])
+                .slice(0, cap);
+            m.ablation = [];
+            for (const [suspect, stolen] of suspects) {
+                const ctx = buildContextDir(cfg.cwd, libraryNames.filter((n) => n !== suspect));
+                let st;
+                try {
+                    st = await measure(adapter, m.prompt, m.expected, {
+                        cwd: ctx.cwd, ...common,
+                        ...(onProgress ? { onProbe: (validN, maxK) => onProgress({ caseIndex: fi + 1, caseTotal: flagged.length, prompt: m.prompt, condition: "ablation", removed: suspect, validN, maxK }) } : {}),
+                    });
+                }
+                finally {
+                    ctx.cleanup();
+                }
+                totalCost += st.totalCost;
+                const rel = reliability(st.hits, st.n, cfg.conf);
+                const co = m.coLoaded.stats;
+                const fisherP = (st.n > 0 && co.n > 0)
+                    ? fisherExact2x2(st.hits, st.n - st.hits, co.hits, co.n - co.hits)
+                    : 1;
+                const abl = {
+                    removed: suspect, stolen, stats: st, rel,
+                    deltaVsCoLoaded: rel.pHat - m.coLoaded.rel.pHat,
+                    fisherP, fisherPAdj: 1,
+                    untrustworthy: infraUntrustworthy(st),
+                    culprit: false,
+                };
+                m.ablation.push(abl);
+                allRuns.push(abl);
+            }
+        }
+        const ablFamily = allRuns.filter((r) => !r.untrustworthy);
+        const ablAdj = bhAdjust(ablFamily.map((r) => r.fisherP));
+        ablFamily.forEach((r, j) => { r.fisherPAdj = ablAdj[j]; });
+        for (const r of allRuns) {
+            r.culprit = !r.untrustworthy && r.deltaVsCoLoaded > 0 && r.fisherPAdj < 0.05;
+        }
+    }
     // precedence mirrors the audit: a real behavioral signal (interference, exit 1) outranks an
     // untrustworthy/infra case (exit 2), which outranks a clean run (exit 0).
     const exitCode = measured.some((m) => m.interference) ? 1
@@ -102,5 +168,6 @@ export async function runContextAudit(cfg, onProgress) {
         runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
         threshold: cfg.threshold, conf: cfg.conf, librarySize: library.length,
         cases: measured, skipped, totalCost, exitCode,
+        manifest: buildManifest("context", cfg),
     };
 }

package/dist/diagnose.d.ts

@@ -15,6 +15,7 @@ import type { Config } from "./types.ts";
 import { reliability } from "./stats.ts";
 import { type RunStats } from "./orchestrator.ts";
 import { type SkillInfo } from "./gen.ts";
+import { type RunManifest } from "./manifest.ts";
 export type DiagVerdict = "routes-ok" | "routing-miss" | "description-problem" | "inconclusive" | "error";
 /** Returns the chosen skill name, "none", or null when the answer is unusable (LLM error/garbage). */
 export type ChoiceFn = (skills: SkillInfo[], prompt: string, model?: string) => Promise<string | null>;
@@ -50,6 +51,8 @@ export interface DiagnoseResult {
     skipped: string[];
     totalCost: number;
     exitCode: 0 | 1 | 2;
+    /** who/what/when of this run — makes reports citable and comparable */
+    manifest: RunManifest;
 }
 export interface DiagProgress {
     caseIndex: number;

package/dist/diagnose.js

@@ -4,6 +4,7 @@ import { getAdapter } from "./adapters/index.js";
 import { listSkills } from "./gen.js";
 import { complete } from "./llm.js";
 import { ConfigError } from "./config.js";
+import { buildManifest } from "./manifest.js";
 const ROUTER_SYSTEM = "You are a skill router. Given a user request and a list of available skills (name + " +
     "description), decide which single skill should handle the request. Reply with EXACTLY one skill " +
     "name from the list, or the word none if no skill fits. Output only that one word — no punctuation, " +
@@ -150,5 +151,6 @@ export async function runDiagnose(cfg, deps = {}) {
     return {
         runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
         threshold: cfg.threshold, conf: cfg.conf, cases, skipped, totalCost, exitCode,
+        manifest: buildManifest("diagnose", cfg),
     };
 }

package/dist/eval.d.ts

@@ -3,6 +3,7 @@
 import type { Config } from "./types.ts";
 import { reliability } from "./stats.ts";
 import { type RunStats } from "./orchestrator.ts";
+import { type RunManifest } from "./manifest.ts";
 /** Four honest states. `error` = the runtime failed enough that the measurement is untrustworthy. */
 export type Verdict = "pass" | "fail" | "inconclusive" | "error";
 export interface CaseResult {
@@ -29,6 +30,8 @@ export interface AuditResult {
     };
     /** 0 = all pass; 1 = a behavioral failure/theft; 2 = inconclusive or infrastructure error */
     exitCode: 0 | 1 | 2;
+    /** who/what/when of this run — makes reports citable and baselines comparable */
+    manifest: RunManifest;
 }
 export interface ProgressEvent {
     caseIndex: number;

package/dist/eval.js

@@ -1,6 +1,7 @@
 import { reliability, sequentialDecision } from "./stats.js";
 import { measure } from "./orchestrator.js";
 import { getAdapter } from "./adapters/index.js";
+import { buildManifest } from "./manifest.js";
 function verdictFor(stats, decision, theft) {
     // `error` means INFRASTRUCTURE failures dominate — untrustworthy. It is NOT about small k:
     // a small-but-all-valid run is `inconclusive` (wide CI), which the report explains.
@@ -39,9 +40,9 @@ export async function runAudit(cfg, onProgress) {
             theft, decision, verdict: verdictFor(stats, decision, theft),
         });
     }
-    // NOTE: Benjamini-Hochberg multiplicity control over per-case exact-test p-values lands in M2,
-    // once each case carries a real p-value rather than a CI decision. The functions are tested
-    // in stats.ts but are NOT yet wired into this audit path.
+    // NOTE: per-case verdicts here are CI decisions vs the threshold, not p-values — so there is no
+    // family to BH-correct in this path. Where families of Fisher tests exist (context, ablation,
+    // baseline comparison) the correction is applied.
     const counts = {
         pass: cases.filter((c) => c.verdict === "pass").length,
         fail: cases.filter((c) => c.verdict === "fail").length,
@@ -52,5 +53,6 @@ export async function runAudit(cfg, onProgress) {
     return {
         runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
         threshold: cfg.threshold, cases, totalCost, counts, exitCode,
+        manifest: buildManifest("audit", cfg),
     };
 }

package/dist/manifest.d.ts

@@ -0,0 +1,21 @@
+import type { Config } from "./types.ts";
+export interface RunManifest {
+    tool: "skill-probe";
+    version: string;
+    command: "audit" | "context" | "diagnose";
+    /** ISO 8601 UTC */
+    date: string;
+    runtime: string;
+    model: string;
+    k: number;
+    threshold: number;
+    conf: number;
+    /** sha256 (first 16 hex chars) of the resolved config — cases + settings, NOT cwd */
+    configHash: string;
+    node: string;
+}
+export declare function toolVersion(): string;
+/** Hash the parts of the config that change what was measured. `cwd` is deliberately EXCLUDED:
+ * the same library checked out at a different path (CI runner vs laptop) is the same experiment. */
+export declare function configHash(cfg: Config): string;
+export declare function buildManifest(command: RunManifest["command"], cfg: Config): RunManifest;

package/dist/manifest.js

@@ -0,0 +1,48 @@
+/** Run manifest — stamps every probe-running result so two runs are comparable and a report is
+ * citable: tool version, command, when it ran, runtime/model, the statistical settings, and a
+ * hash of the resolved config. Same configHash = same cases + settings = apples to apples; the
+ * baseline comparison uses it to warn when a gate compares different experiments. */
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+let cachedVersion;
+export function toolVersion() {
+    if (cachedVersion === undefined) {
+        try {
+            const pkg = JSON.parse(readFileSync(join(import.meta.dirname, "..", "package.json"), "utf8"));
+            cachedVersion = pkg.version ?? "unknown";
+        }
+        catch {
+            cachedVersion = "unknown";
+        }
+    }
+    return cachedVersion;
+}
+/** Hash the parts of the config that change what was measured. `cwd` is deliberately EXCLUDED:
+ * the same library checked out at a different path (CI runner vs laptop) is the same experiment. */
+export function configHash(cfg) {
+    const canonical = JSON.stringify({
+        runtime: cfg.runtime,
+        model: cfg.model ?? null,
+        k: cfg.k,
+        threshold: cfg.threshold,
+        conf: cfg.conf,
+        cases: cfg.cases.map((c) => ({ prompt: c.prompt, expected: c.expected })),
+    });
+    return createHash("sha256").update(canonical).digest("hex").slice(0, 16);
+}
+export function buildManifest(command, cfg) {
+    return {
+        tool: "skill-probe",
+        version: toolVersion(),
+        command,
+        date: new Date().toISOString(),
+        runtime: cfg.runtime,
+        model: cfg.model ?? "(runtime default)",
+        k: cfg.k,
+        threshold: cfg.threshold,
+        conf: cfg.conf,
+        configHash: configHash(cfg),
+        node: process.version,
+    };
+}

package/dist/report.d.ts

@@ -3,6 +3,7 @@ import type { FixResult } from "./fix.ts";
 import type { ContextAuditResult } from "./context.ts";
 import type { DoctorResult } from "./doctor.ts";
 import type { DiagnoseResult } from "./diagnose.ts";
+import type { BaselineComparison } from "./baseline.ts";
 /** Smallest k at which an all-success run could certify reliability >= threshold. */
 export declare function kToCertify(threshold: number, conf: number): number;
 export interface RenderOpts {
@@ -17,4 +18,6 @@ export declare function renderDiagnose(d: DiagnoseResult, opts?: RenderOpts): st
 export declare function renderDoctor(d: DoctorResult): string;
 /** activation-rate-by-context: isolation vs co-loaded, with a Fisher's-exact p on the drop. */
 export declare function renderContext(a: ContextAuditResult, opts?: RenderOpts): string;
+/** regression gate vs a saved baseline: only significant (BH-corrected) drops fail the build. */
+export declare function renderBaseline(cmp: BaselineComparison): string;
 export declare function renderFix(f: FixResult): string;

package/dist/report.js

@@ -131,6 +131,11 @@ export function renderContext(a, opts = {}) {
     out.push(`skill-probe context — runtime: ${a.runtime}  model: ${a.model}  ` +
         `threshold: ${pct(a.threshold)}  library: ${a.librarySize} skills`);
     out.push(`  isolation = only that skill loaded · co-loaded = the full library (${a.librarySize}) loaded`);
+    const familyN = a.cases.filter((c) => !c.untrustworthy).length;
+    if (familyN > 1) {
+        out.push(`  ${familyN} comparisons — interference decided on Benjamini-Hochberg adjusted p<0.05` +
+            ` (raw p across a family overcounts)`);
+    }
     out.push("");
     const ci = (r) => `${pct(r.pHat)} [${pct(r.ciLow)}, ${pct(r.ciHigh)}] k=${r.k}`;
     for (const c of a.cases) {
@@ -148,12 +153,37 @@ export function renderContext(a, opts = {}) {
         out.push(`  [${flag}]  ${c.expected}  | ${c.prompt}`);
         out.push(`        isolation ${ci(c.isolation.rel)}`);
         out.push(`        co-loaded ${ci(c.coLoaded.rel)}`);
-        out.push(`        Δ ${c.deltaP >= 0 ? "+" : ""}${pct(c.deltaP)} under load   Fisher p=${pval(c.fisherP)}`);
+        out.push(`        Δ ${c.deltaP >= 0 ? "+" : ""}${pct(c.deltaP)} under load   Fisher p=${pval(c.fisherP)}` +
+            (familyN > 1 ? ` · BH-adj p=${pval(c.fisherPAdj)}` : ""));
         if (c.interference) {
             const thieves = Object.keys(c.coLoaded.stats.dist)
                 .filter((s) => s !== c.expected && s !== "None");
             out.push(`        ↳ fires reliably alone but is suppressed when the library is co-loaded` +
                 (thieves.length ? ` (stolen by: ${thieves.join(", ")})` : ""));
+            if (c.ablation) {
+                if (c.ablation.length === 0) {
+                    out.push(`        leave-one-out: no sibling stole these probes (suppressed to None) — ` +
+                        `dilution, not theft; nothing to ablate`);
+                }
+                else {
+                    out.push(`        leave-one-out (each suspect removed, recovery vs co-loaded):`);
+                    for (const abl of c.ablation) {
+                        if (abl.untrustworthy) {
+                            out.push(`          − ${abl.removed}: infra errors dominated — run not trustworthy`);
+                            continue;
+                        }
+                        const tag = abl.culprit
+                            ? (abl.rel.pHat >= a.threshold ? "  ← THIEF (full recovery)" : "  ← THIEF (partial recovery)")
+                            : "";
+                        out.push(`          − ${abl.removed} removed → ${ci(abl.rel)}  ` +
+                            `Δ${abl.deltaVsCoLoaded >= 0 ? "+" : ""}${pct(abl.deltaVsCoLoaded)}  ` +
+                            `p=${pval(abl.fisherP)} · adj ${pval(abl.fisherPAdj)}${tag}`);
+                    }
+                    if (!c.ablation.some((abl) => abl.culprit)) {
+                        out.push(`          no single culprit confirmed — interference may be combinatorial`);
+                    }
+                }
+            }
         }
     }
     if (a.skipped.length) {
@@ -163,11 +193,47 @@ export function renderContext(a, opts = {}) {
     out.push("");
     const interfN = a.cases.filter((c) => c.interference).length;
     const errN = a.cases.filter((c) => c.untrustworthy).length;
+    const culprits = [...new Set(a.cases.flatMap((c) => (c.ablation ?? []).filter((abl) => abl.culprit).map((abl) => abl.removed)))];
     out.push(`Result: ${interfN} interference / ${a.cases.length} measured` +
-        (errN ? ` / ${errN} error` : "") + `  |  exit ${a.exitCode}` +
+        (errN ? ` / ${errN} error` : "") +
+        (culprits.length ? `  |  thieves named: ${culprits.join(", ")}` : "") +
+        `  |  exit ${a.exitCode}` +
         (showCost ? `  |  cost $${a.totalCost.toFixed(4)}` : ""));
     return out.join("\n");
 }
+/** regression gate vs a saved baseline: only significant (BH-corrected) drops fail the build. */
+export function renderBaseline(cmp) {
+    const out = [];
+    out.push(`Baseline gate — vs baseline saved ${cmp.baselineDate} (skill-probe ${cmp.baselineVersion})`);
+    for (const m of cmp.mismatches)
+        out.push(`  ⚠ ${m}`);
+    for (const r of cmp.cases) {
+        if (r.untrustworthy) {
+            out.push(`  [⛔ NOT COMPARABLE]  ${r.expected ?? "(decoy)"}  | ${r.prompt} — a side has no valid probes`);
+            continue;
+        }
+        if (!r.regressed && !r.improved)
+            continue; // stable cases are summarized, not listed
+        const flag = r.regressed ? "▼ REGRESSED" : "▲ improved";
+        out.push(`  [${flag}]  ${r.expected ?? "(decoy)"}  | ${r.prompt}`);
+        out.push(`        ${pct(r.before.pHat)} (${r.before.hits}/${r.before.n}) → ${pct(r.after.pHat)} ` +
+            `(${r.after.hits}/${r.after.n})   Δ${r.delta >= 0 ? "+" : ""}${pct(r.delta)}   ` +
+            `p=${pval(r.fisherP)} · adj ${pval(r.fisherPAdj)}`);
+    }
+    const extras = [];
+    if (cmp.newCases.length)
+        extras.push(`${cmp.newCases.length} new case(s) not in baseline (measured, not gated)`);
+    if (cmp.missingCases.length)
+        extras.push(`${cmp.missingCases.length} baseline case(s) missing from this run`);
+    if (extras.length)
+        out.push(`  ${extras.join(" · ")}`);
+    out.push(cmp.regressions
+        ? `Gate: ▼ FAIL — ${cmp.regressions} significant regression(s)` +
+            (cmp.improvements ? `, ${cmp.improvements} improvement(s)` : "")
+        : `Gate: ✅ no significant regressions (${cmp.cases.length} gated` +
+            (cmp.improvements ? `, ${cmp.improvements} improved — consider re-saving the baseline` : "") + `)`);
+    return out.join("\n");
+}
 export function renderFix(f) {
     const out = [];
     out.push(`skill-probe fix — skill: ${f.skill}`);

package/dist/stats.d.ts

@@ -30,5 +30,9 @@ export interface ImprovementResult {
 }
 /** Bayesian Beta-Binomial: P(p_after > p_before) + delta credible interval. */
 export declare function probImprovement(xBefore: number, nBefore: number, xAfter: number, nAfter: number, prior?: number, samples?: number, seed?: number): ImprovementResult;
+/** Benjamini-Hochberg step-up ADJUSTED p-values: adj[i] = min over ranks j>=rank(i) of m*p_(j)/j.
+ * adj[i] <= q ⟺ benjaminiHochberg(pvals, q) rejects i, but the adjusted value is reportable
+ * next to the raw p so the multiple-comparisons correction is visible, not silent. */
+export declare function bhAdjust(pvals: number[]): number[];
 /** Benjamini-Hochberg FDR. Returns booleans (true = flag as real). */
 export declare function benjaminiHochberg(pvals: number[], q?: number): boolean[];

package/dist/stats.js

@@ -135,6 +135,23 @@ export function probImprovement(xBefore, nBefore, xAfter, nAfter, prior = 0.5, s
         deltaCi: [deltas[Math.floor(0.025 * samples)], deltas[Math.floor(0.975 * samples)]],
     };
 }
+/** Benjamini-Hochberg step-up ADJUSTED p-values: adj[i] = min over ranks j>=rank(i) of m*p_(j)/j.
+ * adj[i] <= q ⟺ benjaminiHochberg(pvals, q) rejects i, but the adjusted value is reportable
+ * next to the raw p so the multiple-comparisons correction is visible, not silent. */
+export function bhAdjust(pvals) {
+    const m = pvals.length;
+    if (m === 0)
+        return [];
+    const order = [...pvals.keys()].sort((i, j) => pvals[i] - pvals[j]);
+    const adj = new Array(m).fill(1);
+    let running = 1;
+    for (let idx = m - 1; idx >= 0; idx--) {
+        const i = order[idx];
+        running = Math.min(running, (pvals[i] * m) / (idx + 1));
+        adj[i] = Math.min(1, running);
+    }
+    return adj;
+}
 /** Benjamini-Hochberg FDR. Returns booleans (true = flag as real). */
 export function benjaminiHochberg(pvals, q = 0.05) {
     const m = pvals.length;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skill-probe",
-  "version": "0.7.1",
+  "version": "0.9.0",
   "description": "Runtime-agnostic, co-loaded-aware auditor for AI agent skill libraries: measures which skill actually fires, with statistical confidence.",
   "type": "module",
   "bin": {