skill-probe 0.8.0 → 0.9.0

package/README.md

@@ -27,7 +27,8 @@ doctor → gen → audit → context → diagnose → fix
 
 - `doctor`: verify setup, auth, runtime, skills, and config before spending probes.
 - `gen`: draft a probe config from your existing skills.
-- `audit`: measure which skill actually fires when the whole library is co-loaded.
+- `audit`: measure which skill actually fires when the whole library is co-loaded — with
+  `--baseline`, gate CI on statistically significant regressions only.
 - `context`: compare isolation vs co-loaded activation to catch library interference —
   add `--ablate` to name **which** sibling is stealing the trigger.
 - `diagnose`: explain whether a failure is a routing miss or a description problem.
@@ -157,16 +158,64 @@ Claude Code vs OpenCode comparison is fair — otherwise you're comparing two co
 
 ## CI usage
 
-skill-probe exits non-zero on a real problem, so it drops straight into a pipeline:
+skill-probe exits non-zero on a real problem, so it drops straight into a pipeline. Exit codes:
+**`0`** all pass · **`1`** a behavioral fail / trigger-theft / interference / regression · **`2`**
+inconclusive or an infrastructure error — so a runtime outage *fails* the build instead of silently
+passing. Add `--json` to archive the full result (it includes a **run manifest**: tool version,
+model, date, config hash — so two runs are comparable and a report is citable).
+
+### Regression gating (`--baseline`)
+
+The CI question isn't "are my skills perfect?" — it's **"did this PR make any skill worse?"**
+Activation is stochastic, so comparing raw rates makes CI flaky. The baseline gate compares each
+case against a saved baseline with **Fisher's exact test, Benjamini-Hochberg corrected** — noise
+passes, real drops fail:
+
+```bash
+# once, on a good main build (commit the file):
+skill-probe --config probe.config.json --save-baseline baselines/main.json
+
+# on every PR:
+skill-probe --config probe.config.json --baseline baselines/main.json
+```
+
+```
+Baseline gate — vs baseline saved 2026-07-02T09:12:03Z (skill-probe 0.9.0)
+  [▼ REGRESSED]  greeter  | write a birthday greeting
+        90% (9/10) → 20% (2/10)   Δ-70%   p=0.005 · adj 0.005
+Gate: ▼ FAIL — 1 significant regression(s)
+```
+
+The gate warns (but still runs, matching cases by prompt) if the config, runtime, or model changed
+since the baseline — apples-to-oranges comparisons are flagged, never silent. Significant
+*improvements* are reported too, as a nudge to re-save the baseline.
+
+### GitHub Action
 
 ```yaml
-# .github/workflows/skills.yml  (the claude/opencode CLI must be installed + authed on the runner)
-- run: npx skill-probe --config probe.config.json --quiet --no-cost
+# .github/workflows/skills.yml
+name: skills
+on: pull_request
+jobs:
+  probe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with: { node-version: "22" }
+      - uses: HystonKayange/skill-probe@main
+        with:
+          config: probe.config.json
+          args: --baseline baselines/main.json
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
 ```
 
-Exit codes: **`0`** all pass · **`1`** a behavioral fail / trigger-theft / interference · **`2`**
-inconclusive or an infrastructure error — so a runtime outage *fails* the build instead of silently
-passing. Add `--json` to archive the full result as a build artifact.
+The action installs skill-probe plus the runtime CLI (Claude Code by default; set
+`runtime-package: ""` if your runner already has one) and runs the command. `command:` selects
+`audit` (default), `context`, `diagnose`, or `doctor`; `args:` passes extra flags like `--ablate`.
+The runtime authenticates via `ANTHROPIC_API_KEY` — headless probes each cost real tokens, so
+smoke settings (`k: 10`, few cases) are the sweet spot for per-PR gating.
 
 ## Activation rate by context (`skill-probe context`)
 
@@ -320,7 +369,10 @@ should-fire prompts, near-misses, and decoys.
 (pass / fail / inconclusive / **error**), across two runtimes (Claude Code, OpenCode).
 Infrastructure failures (timeout / auth / crash / empty output / a zero-cost response where the
 model never actually ran, e.g. a usage-limit window) are reported as `error`, never as a
-behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
+behavioral pass/fail — a decoy can't falsely pass because the runtime was down. Every run carries
+a **manifest** (version, model, date, config hash); `--save-baseline` / `--baseline` turn audits
+into a CI regression gate (Fisher + BH — noise passes, real drops fail), and a **GitHub Action**
+(`HystonKayange/skill-probe@main`) wraps the whole thing for per-PR gating.
 
 **Context (`skill-probe context`):** isolation-vs-co-loaded activation rates, with **Fisher's exact
 test** on the drop and **Benjamini-Hochberg correction** across cases — catches skills that fire

package/dist/baseline.d.ts

@@ -0,0 +1,56 @@
+import type { AuditResult } from "./eval.ts";
+import type { RunManifest } from "./manifest.ts";
+export interface BaselineCase {
+    prompt: string;
+    expected: string | null;
+    hits: number;
+    n: number;
+}
+export interface BaselineFile {
+    manifest: RunManifest;
+    cases: BaselineCase[];
+}
+export declare function toBaseline(result: AuditResult): BaselineFile;
+export declare function saveBaseline(result: AuditResult, path: string): void;
+export declare function loadBaseline(path: string): BaselineFile;
+export interface RegressionCase {
+    prompt: string;
+    expected: string | null;
+    before: {
+        hits: number;
+        n: number;
+        pHat: number;
+    };
+    after: {
+        hits: number;
+        n: number;
+        pHat: number;
+    };
+    /** after.pHat - before.pHat (negative = dropped since baseline) */
+    delta: number;
+    fisherP: number;
+    /** BH-adjusted across all compared cases — regression is decided on THIS */
+    fisherPAdj: number;
+    /** one side has no valid probes — not comparable (never reads as "no regression") */
+    untrustworthy: boolean;
+    /** significant drop since baseline (BH-corrected) */
+    regressed: boolean;
+    /** significant rise since baseline (informational — consider re-saving the baseline) */
+    improved: boolean;
+}
+export interface BaselineComparison {
+    baselineDate: string;
+    baselineVersion: string;
+    /** apples-to-oranges warnings: config hash / runtime / model drift between runs */
+    mismatches: string[];
+    cases: RegressionCase[];
+    /** prompts in the current run with no baseline entry (measured, but not gated) */
+    newCases: string[];
+    /** prompts in the baseline that the current run no longer has */
+    missingCases: string[];
+    regressions: number;
+    improvements: number;
+    /** 1 = at least one significant regression (CI should fail) */
+    exitCode: 0 | 1;
+}
+export declare function compareToBaseline(result: AuditResult, base: BaselineFile): BaselineComparison;

package/dist/cli.js

@@ -12,7 +12,8 @@ import { runDiagnose } from "./diagnose.js";
 import { runDoctor } from "./doctor.js";
 import { listSkills, generateCases, genConfig } from "./gen.js";
 import { loadConfig, parseProbability, parseIntFlag, ConfigError } from "./config.js";
-import { renderTable, renderMarkdown, renderJson, renderFix, renderContext, renderDoctor, renderDiagnose } from "./report.js";
+import { loadBaseline, saveBaseline, compareToBaseline } from "./baseline.js";
+import { renderTable, renderMarkdown, renderJson, renderFix, renderContext, renderDoctor, renderDiagnose, renderBaseline } from "./report.js";
 const HELP = `skill-probe — audit a co-loaded agent skill library by real activation behavior.
 
 USAGE
@@ -36,6 +37,11 @@ OPTIONS
       --markdown           emit a Markdown table (paste into a PR/README)
       --no-cost            hide the cost line (e.g. when on a Claude subscription)
       --quiet              suppress the live progress line on stderr
+  audit only (CI gating):
+      --save-baseline <f>  after the run, save per-case counts + a run manifest to <f>
+      --baseline <f>       compare this run against a saved baseline: per-case Fisher's exact,
+                           Benjamini-Hochberg corrected — exit 1 ONLY on a significant regression
+                           (noise passes; real drops fail)
   context only:
       --ablate             leave-one-out: for each interference case, re-run with each suspect
                            sibling removed and NAME the thief when activation recovers (Fisher +
@@ -106,10 +112,21 @@ async function audit(values) {
     const result = await runAudit(cfg, makeProgress(values));
     clearProgress();
     const showCost = !values["no-cost"];
-    const text = values["json"] ? renderJson(result)
-        : values["markdown"] ? renderMarkdown(result, { showCost })
-            : renderTable(result, { showCost });
+    let cmp;
+    if (values["baseline"])
+        cmp = compareToBaseline(result, loadBaseline(values["baseline"]));
+    if (values["save-baseline"]) {
+        saveBaseline(result, values["save-baseline"]);
+        console.error(`baseline saved to ${values["save-baseline"]}`);
+    }
+    const text = values["json"] ? renderJson(cmp ? { ...result, baseline: cmp } : result)
+        : values["markdown"] ? renderMarkdown(result, { showCost }) + (cmp ? "\n\n" + renderBaseline(cmp) : "")
+            : renderTable(result, { showCost }) + (cmp ? "\n\n" + renderBaseline(cmp) : "");
     console.log(text);
+    // precedence mirrors the rest of the tool: a behavioral signal (fail/regression, 1) outranks
+    // inconclusive/infra (2), which outranks clean (0).
+    if (result.exitCode === 1 || cmp?.exitCode === 1)
+        return 1;
     return result.exitCode;
 }
 async function diagnose(values) {
@@ -204,6 +221,8 @@ async function main() {
             decoys: { type: "string" },
             ablate: { type: "boolean", default: false },
             suspects: { type: "string" },
+            baseline: { type: "string" },
+            "save-baseline": { type: "string" },
             "skip-probe": { type: "boolean", default: false },
             json: { type: "boolean", default: false },
             markdown: { type: "boolean", default: false },

package/dist/context.d.ts

@@ -1,6 +1,7 @@
 import type { Config } from "./types.ts";
 import { reliability } from "./stats.ts";
 import { type RunStats } from "./orchestrator.ts";
+import { type RunManifest } from "./manifest.ts";
 /** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
  * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
 export declare function buildContextDir(srcCwd: string, skillNames: string[]): {
@@ -56,6 +57,8 @@ export interface ContextAuditResult {
     totalCost: number;
     /** 0 = no interference; 1 = a skill is suppressed under load; 2 = an untrustworthy (infra) case */
     exitCode: 0 | 1 | 2;
+    /** who/what/when of this run — makes reports citable and comparable */
+    manifest: RunManifest;
 }
 export interface ContextProgress {
     caseIndex: number;

package/dist/context.js

@@ -11,6 +11,7 @@ import { measure, infraUntrustworthy } from "./orchestrator.js";
 import { getAdapter } from "./adapters/index.js";
 import { listSkills } from "./gen.js";
 import { ConfigError } from "./config.js";
+import { buildManifest } from "./manifest.js";
 /** Build a throwaway project dir whose .claude/skills/ holds ONLY the named skills (copied from
  * srcCwd). Returns its path plus a cleanup(). The probe runs against this dir to load a subset. */
 export function buildContextDir(srcCwd, skillNames) {
@@ -167,5 +168,6 @@ export async function runContextAudit(cfg, optsOrProgress = {}) {
         runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
         threshold: cfg.threshold, conf: cfg.conf, librarySize: library.length,
         cases: measured, skipped, totalCost, exitCode,
+        manifest: buildManifest("context", cfg),
     };
 }

package/dist/diagnose.d.ts

@@ -15,6 +15,7 @@ import type { Config } from "./types.ts";
 import { reliability } from "./stats.ts";
 import { type RunStats } from "./orchestrator.ts";
 import { type SkillInfo } from "./gen.ts";
+import { type RunManifest } from "./manifest.ts";
 export type DiagVerdict = "routes-ok" | "routing-miss" | "description-problem" | "inconclusive" | "error";
 /** Returns the chosen skill name, "none", or null when the answer is unusable (LLM error/garbage). */
 export type ChoiceFn = (skills: SkillInfo[], prompt: string, model?: string) => Promise<string | null>;
@@ -50,6 +51,8 @@ export interface DiagnoseResult {
     skipped: string[];
     totalCost: number;
     exitCode: 0 | 1 | 2;
+    /** who/what/when of this run — makes reports citable and comparable */
+    manifest: RunManifest;
 }
 export interface DiagProgress {
     caseIndex: number;

package/dist/diagnose.js

@@ -4,6 +4,7 @@ import { getAdapter } from "./adapters/index.js";
 import { listSkills } from "./gen.js";
 import { complete } from "./llm.js";
 import { ConfigError } from "./config.js";
+import { buildManifest } from "./manifest.js";
 const ROUTER_SYSTEM = "You are a skill router. Given a user request and a list of available skills (name + " +
     "description), decide which single skill should handle the request. Reply with EXACTLY one skill " +
     "name from the list, or the word none if no skill fits. Output only that one word — no punctuation, " +
@@ -150,5 +151,6 @@ export async function runDiagnose(cfg, deps = {}) {
     return {
         runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
         threshold: cfg.threshold, conf: cfg.conf, cases, skipped, totalCost, exitCode,
+        manifest: buildManifest("diagnose", cfg),
     };
 }

package/dist/eval.d.ts

@@ -3,6 +3,7 @@
 import type { Config } from "./types.ts";
 import { reliability } from "./stats.ts";
 import { type RunStats } from "./orchestrator.ts";
+import { type RunManifest } from "./manifest.ts";
 /** Four honest states. `error` = the runtime failed enough that the measurement is untrustworthy. */
 export type Verdict = "pass" | "fail" | "inconclusive" | "error";
 export interface CaseResult {
@@ -29,6 +30,8 @@ export interface AuditResult {
     };
     /** 0 = all pass; 1 = a behavioral failure/theft; 2 = inconclusive or infrastructure error */
     exitCode: 0 | 1 | 2;
+    /** who/what/when of this run — makes reports citable and baselines comparable */
+    manifest: RunManifest;
 }
 export interface ProgressEvent {
     caseIndex: number;

package/dist/eval.js

@@ -1,6 +1,7 @@
 import { reliability, sequentialDecision } from "./stats.js";
 import { measure } from "./orchestrator.js";
 import { getAdapter } from "./adapters/index.js";
+import { buildManifest } from "./manifest.js";
 function verdictFor(stats, decision, theft) {
     // `error` means INFRASTRUCTURE failures dominate — untrustworthy. It is NOT about small k:
     // a small-but-all-valid run is `inconclusive` (wide CI), which the report explains.
@@ -39,9 +40,9 @@ export async function runAudit(cfg, onProgress) {
             theft, decision, verdict: verdictFor(stats, decision, theft),
         });
     }
-    // NOTE: Benjamini-Hochberg multiplicity control over per-case exact-test p-values lands in M2,
-    // once each case carries a real p-value rather than a CI decision. The functions are tested
-    // in stats.ts but are NOT yet wired into this audit path.
+    // NOTE: per-case verdicts here are CI decisions vs the threshold, not p-values — so there is no
+    // family to BH-correct in this path. Where families of Fisher tests exist (context, ablation,
+    // baseline comparison) the correction is applied.
     const counts = {
         pass: cases.filter((c) => c.verdict === "pass").length,
         fail: cases.filter((c) => c.verdict === "fail").length,
@@ -52,5 +53,6 @@ export async function runAudit(cfg, onProgress) {
     return {
         runtime: cfg.runtime, model: cfg.model ?? "(runtime default)",
         threshold: cfg.threshold, cases, totalCost, counts, exitCode,
+        manifest: buildManifest("audit", cfg),
     };
 }

package/dist/manifest.d.ts

@@ -0,0 +1,21 @@
+import type { Config } from "./types.ts";
+export interface RunManifest {
+    tool: "skill-probe";
+    version: string;
+    command: "audit" | "context" | "diagnose";
+    /** ISO 8601 UTC */
+    date: string;
+    runtime: string;
+    model: string;
+    k: number;
+    threshold: number;
+    conf: number;
+    /** sha256 (first 16 hex chars) of the resolved config — cases + settings, NOT cwd */
+    configHash: string;
+    node: string;
+}
+export declare function toolVersion(): string;
+/** Hash the parts of the config that change what was measured. `cwd` is deliberately EXCLUDED:
+ * the same library checked out at a different path (CI runner vs laptop) is the same experiment. */
+export declare function configHash(cfg: Config): string;
+export declare function buildManifest(command: RunManifest["command"], cfg: Config): RunManifest;

package/dist/manifest.js

@@ -0,0 +1,48 @@
+/** Run manifest — stamps every probe-running result so two runs are comparable and a report is
+ * citable: tool version, command, when it ran, runtime/model, the statistical settings, and a
+ * hash of the resolved config. Same configHash = same cases + settings = apples to apples; the
+ * baseline comparison uses it to warn when a gate compares different experiments. */
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+let cachedVersion;
+export function toolVersion() {
+    if (cachedVersion === undefined) {
+        try {
+            const pkg = JSON.parse(readFileSync(join(import.meta.dirname, "..", "package.json"), "utf8"));
+            cachedVersion = pkg.version ?? "unknown";
+        }
+        catch {
+            cachedVersion = "unknown";
+        }
+    }
+    return cachedVersion;
+}
+/** Hash the parts of the config that change what was measured. `cwd` is deliberately EXCLUDED:
+ * the same library checked out at a different path (CI runner vs laptop) is the same experiment. */
+export function configHash(cfg) {
+    const canonical = JSON.stringify({
+        runtime: cfg.runtime,
+        model: cfg.model ?? null,
+        k: cfg.k,
+        threshold: cfg.threshold,
+        conf: cfg.conf,
+        cases: cfg.cases.map((c) => ({ prompt: c.prompt, expected: c.expected })),
+    });
+    return createHash("sha256").update(canonical).digest("hex").slice(0, 16);
+}
+export function buildManifest(command, cfg) {
+    return {
+        tool: "skill-probe",
+        version: toolVersion(),
+        command,
+        date: new Date().toISOString(),
+        runtime: cfg.runtime,
+        model: cfg.model ?? "(runtime default)",
+        k: cfg.k,
+        threshold: cfg.threshold,
+        conf: cfg.conf,
+        configHash: configHash(cfg),
+        node: process.version,
+    };
+}

package/dist/report.d.ts

@@ -3,6 +3,7 @@ import type { FixResult } from "./fix.ts";
 import type { ContextAuditResult } from "./context.ts";
 import type { DoctorResult } from "./doctor.ts";
 import type { DiagnoseResult } from "./diagnose.ts";
+import type { BaselineComparison } from "./baseline.ts";
 /** Smallest k at which an all-success run could certify reliability >= threshold. */
 export declare function kToCertify(threshold: number, conf: number): number;
 export interface RenderOpts {
@@ -17,4 +18,6 @@ export declare function renderDiagnose(d: DiagnoseResult, opts?: RenderOpts): st
 export declare function renderDoctor(d: DoctorResult): string;
 /** activation-rate-by-context: isolation vs co-loaded, with a Fisher's-exact p on the drop. */
 export declare function renderContext(a: ContextAuditResult, opts?: RenderOpts): string;
+/** regression gate vs a saved baseline: only significant (BH-corrected) drops fail the build. */
+export declare function renderBaseline(cmp: BaselineComparison): string;
 export declare function renderFix(f: FixResult): string;

package/dist/report.js

@@ -201,6 +201,39 @@ export function renderContext(a, opts = {}) {
         (showCost ? `  |  cost $${a.totalCost.toFixed(4)}` : ""));
     return out.join("\n");
 }
+/** regression gate vs a saved baseline: only significant (BH-corrected) drops fail the build. */
+export function renderBaseline(cmp) {
+    const out = [];
+    out.push(`Baseline gate — vs baseline saved ${cmp.baselineDate} (skill-probe ${cmp.baselineVersion})`);
+    for (const m of cmp.mismatches)
+        out.push(`  ⚠ ${m}`);
+    for (const r of cmp.cases) {
+        if (r.untrustworthy) {
+            out.push(`  [⛔ NOT COMPARABLE]  ${r.expected ?? "(decoy)"}  | ${r.prompt} — a side has no valid probes`);
+            continue;
+        }
+        if (!r.regressed && !r.improved)
+            continue; // stable cases are summarized, not listed
+        const flag = r.regressed ? "▼ REGRESSED" : "▲ improved";
+        out.push(`  [${flag}]  ${r.expected ?? "(decoy)"}  | ${r.prompt}`);
+        out.push(`        ${pct(r.before.pHat)} (${r.before.hits}/${r.before.n}) → ${pct(r.after.pHat)} ` +
+            `(${r.after.hits}/${r.after.n})   Δ${r.delta >= 0 ? "+" : ""}${pct(r.delta)}   ` +
+            `p=${pval(r.fisherP)} · adj ${pval(r.fisherPAdj)}`);
+    }
+    const extras = [];
+    if (cmp.newCases.length)
+        extras.push(`${cmp.newCases.length} new case(s) not in baseline (measured, not gated)`);
+    if (cmp.missingCases.length)
+        extras.push(`${cmp.missingCases.length} baseline case(s) missing from this run`);
+    if (extras.length)
+        out.push(`  ${extras.join(" · ")}`);
+    out.push(cmp.regressions
+        ? `Gate: ▼ FAIL — ${cmp.regressions} significant regression(s)` +
+            (cmp.improvements ? `, ${cmp.improvements} improvement(s)` : "")
+        : `Gate: ✅ no significant regressions (${cmp.cases.length} gated` +
+            (cmp.improvements ? `, ${cmp.improvements} improved — consider re-saving the baseline` : "") + `)`);
+    return out.join("\n");
+}
 export function renderFix(f) {
     const out = [];
     out.push(`skill-probe fix — skill: ${f.skill}`);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skill-probe",
-  "version": "0.8.0",
+  "version": "0.9.0",
   "description": "Runtime-agnostic, co-loaded-aware auditor for AI agent skill libraries: measures which skill actually fires, with statistical confidence.",
   "type": "module",
   "bin": {