skill-probe 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 skill-probe authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,123 @@
+# skill-probe
+
+Runtime-agnostic, **co-loaded-aware** auditor for an AI agent's skill library. It measures
+which skill *actually* fires (by real activation, not keyword matching) when your whole
+`SKILL.md` library is loaded together — with **statistical confidence**, not single-shot guesses.
+
+> Most skill tooling tests one skill, in isolation, once. Skills only conflict when loaded
+> *together*, and activation is *stochastic*. skill-probe is the tool that tests the real thing.
+
+Complements static linters like `skill-audit` (security/quality) — this is **behavioral**.
+
+## Why
+
+- Skill activation is a coin flip: independent studies measured 55–87% trigger rates.
+- A single run lies (we measured the same prompt at 0/5 one batch, 2/3 the next).
+- skill-probe runs each prompt **k times**, reports a **Wilson 95% CI**, stops early when the
+  result is statistically decided, and flags **trigger-theft** (a sibling stealing a trigger).
+
+## Install
+
+```bash
+npm i -g skill-probe      # or: npx skill-probe
+```
+
+## Use
+
+```bash
+# point it at your own config (after `npm i -g skill-probe`):
+skill-probe --config my.config.json --k 15 --threshold 0.9 --json
+
+# or, from a clone of this repo, try the bundled example (needs `claude` installed + auth):
+skill-probe --config examples/audit.config.json
+```
+
+Config (`skill-probe.config.json`):
+```json
+{
+  "runtime": "claude-code",
+  "cwd": "./my-project",
+  "k": 10, "threshold": 0.7, "conf": 0.95,
+  "cases": [
+    { "prompt": "write a commit message", "expected": "commit-writer" },
+    { "prompt": "what's the weather?", "expected": null }
+  ]
+}
+```
+Output:
+```
+skill-probe — runtime: claude-code  model: (runtime default)  threshold: 70%
+
+  PASS          reliability 100% [72%, 100%] k=10   expect=commit-writer
+        outcomes: commit-writer×10
+  FAIL          reliability 20% [6%, 51%] k=10   expect=pr-describer
+        outcomes: None×8, commit-writer×2
+        ⚠ trigger-theft by: commit-writer
+  PASS          reliability 100% [72%, 100%] k=10   expect=(none)
+        outcomes: None×10
+
+Result: 2 pass / 1 fail / 0 inconclusive / 0 error  |  exit 1  |  cost $0.18
+```
+
+- `cwd` (relative paths resolve against the **config file's** directory) is a project dir
+  containing `.claude/skills/`.
+- `expected: null` = a decoy that should fire nothing.
+- Exit code: `0` all pass, `1` a behavioral fail / trigger-theft, `2` inconclusive or an
+  infrastructure error (runtime down → never a silent pass).
+
+**Cross-runtime comparison must pin the model.** A skill can fire on one runtime and not another
+partly because of the *model*, not the runtime. Always set `model` (recorded in the report) so a
+Claude Code vs OpenCode comparison is fair — otherwise you're comparing two confounded variables.
+
+**Two modes** — because confidence intervals are wide at small k:
+- **Smoke (default):** `threshold 0.7, k 10` — a clean run certifies; cheap; good for CI.
+- **Certify:** a strict bar needs more runs. To certify **≥0.9** you need **~k=35**; when a
+  case is inconclusive the report prints the exact k for your threshold. Don't set `threshold
+  0.9, k 10` and expect a pass — that's statistically impossible and the tool will say so.
+
+## Fix a failing skill (`skill-probe fix`)
+
+Rewrite a skill's trigger description and **prove the lift is real** before keeping it:
+
+```bash
+ANTHROPIC_API_KEY=sk-ant-... \
+  skill-probe fix --config examples/fix.config.json --skill commit-writer
+```
+
+It (1) LLM-rewrites the description, told the sibling skills so it won't steal their triggers;
+(2) runs an **interleaved** before/after (old desc → probe → new desc → probe, paired, to control
+for drift); (3) computes the **Bayesian** P(improvement) + a credible interval on the change; and
+(4) **applies the rewrite only if** `P(improvement) ≥ --apply-threshold` (default 0.9) **and** the
+effect is positive — otherwise reverts. When applied, the original is snapshotted to a timestamped
+`SKILL.md.bak.<timestamp>` (no backup is left behind on a revert).
+
+```
+before: 0% [0%, 49%]   after: 100% [51%, 100%]   (4 paired runs)
+P(rewrite improved reliability) = 100%   Δ = +80% [36%, 99%]
+✅ APPLIED
+```
+
+`fix` needs `ANTHROPIC_API_KEY` (for the rewrite). It changes descriptions on statistical evidence,
+not on "the new one looks nicer" — a rewrite that doesn't measurably help is reverted.
+
+## Status
+
+Early (v0.1). **Audit (`skill-probe`):** Wilson confidence intervals + sequential stopping +
+four-state verdict (pass / fail / inconclusive / **error**), across two runtimes (Claude Code,
+OpenCode). Infrastructure failures (timeout / auth / crash / empty output) are reported as
+`error`, never as a behavioral pass/fail — a decoy can't falsely pass because the runtime was down.
+
+**Fix (`skill-probe fix`):** uses the **Bayesian Beta-Binomial** to gate description rewrites on a
+*proven* lift (interleaved before/after, applied only if P(improvement) clears the bar).
+
+**Implemented + unit-tested but NOT yet used by any command:** Fisher's exact test and
+Benjamini-Hochberg FDR (`src/stats.ts`) — reserved for cross-case multiplicity control (a planned
+enhancement). Don't read the presence of these functions as the audit using them.
+
+## Dev
+
+```bash
+node --test test/*.test.ts   # run tests (zero deps; Node >= 22.18 strips types)
+node src/cli.ts --help
+npm run typecheck            # tsc --noEmit (needs `npm i`)
+```

package/dist/adapters/claudeCode.d.ts

@@ -0,0 +1,10 @@
+import type { RuntimeAdapter } from "../types.ts";
+export interface ParseInfo {
+    skillFired: string | null;
+    trajectory: string[];
+    cost: number;
+    sawInit: boolean;
+    sawAssistant: boolean;
+}
+export declare function parseStream(stdout: string): ParseInfo;
+export declare const claudeCodeAdapter: RuntimeAdapter;

package/dist/adapters/claudeCode.js

@@ -0,0 +1,84 @@
+/** Claude Code adapter — the proven probe. Spawns `claude -p` headless, parses stream-json
+ * for `tool_use name=="Skill"`. Distinguishes a real behavioral outcome (the model ran) from
+ * an infrastructure failure (timeout / auth / crash / empty output) — finding #1.
+ *
+ * NOTE: with `--max-turns 1` the process exits NON-ZERO on success (error_max_turns), so exit
+ * code can't signal validity. We treat a probe as valid iff the model produced an assistant
+ * message; otherwise it's an infra error. */
+import { spawn } from "node:child_process";
+export function parseStream(stdout) {
+    let skillFired = null;
+    let cost = 0, sawInit = false, sawAssistant = false;
+    const trajectory = [];
+    for (const raw of stdout.split("\n")) {
+        const line = raw.trim();
+        if (!line)
+            continue;
+        let e;
+        try {
+            e = JSON.parse(line);
+        }
+        catch {
+            continue;
+        }
+        if (e.type === "system" && e.subtype === "init")
+            sawInit = true;
+        else if (e.type === "assistant") {
+            sawAssistant = true;
+            for (const b of e.message?.content ?? []) {
+                if (b?.type === "tool_use" && b.name) {
+                    trajectory.push(b.name);
+                    if (b.name === "Skill" && b.input?.skill)
+                        skillFired = b.input.skill;
+                }
+            }
+        }
+        else if (e.type === "result" && typeof e.total_cost_usd === "number") {
+            cost = e.total_cost_usd;
+        }
+    }
+    return { skillFired, trajectory, cost, sawInit, sawAssistant };
+}
+export const claudeCodeAdapter = {
+    name: "claude-code",
+    probe(prompt, { cwd, timeoutMs = 120_000, model }) {
+        return new Promise((resolve) => {
+            let settled = false, timedOut = false;
+            const args = ["-p", prompt, "--output-format", "stream-json", "--verbose", "--max-turns", "1"];
+            if (model)
+                args.push("--model", model);
+            const child = spawn("claude", args, { cwd, stdio: ["ignore", "pipe", "pipe"] });
+            let out = "", err = "";
+            const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, timeoutMs);
+            const finish = (r) => {
+                if (settled)
+                    return;
+                settled = true;
+                clearTimeout(timer);
+                resolve(r);
+            };
+            child.stdout.on("data", (d) => { out += d.toString(); });
+            child.stderr.on("data", (d) => { err += d.toString(); });
+            child.on("error", (e) => finish({
+                status: "error", skillFired: null, trajectory: [], cost: 0,
+                error: `spawn failed: ${e.message} (is 'claude' installed and on PATH?)`,
+            }));
+            child.on("close", () => {
+                const info = parseStream(out);
+                if (timedOut) {
+                    return finish({
+                        status: "error", skillFired: null, trajectory: info.trajectory, cost: info.cost,
+                        error: `timed out after ${timeoutMs}ms`,
+                    });
+                }
+                if (!info.sawAssistant) {
+                    return finish({
+                        status: "error", skillFired: null, trajectory: info.trajectory, cost: info.cost,
+                        error: err.trim() ? err.trim().slice(-200) : "no assistant output (auth/config/empty response?)",
+                    });
+                }
+                finish({ status: "ok", skillFired: info.skillFired, trajectory: info.trajectory, cost: info.cost });
+            });
+        });
+    },
+};

package/dist/adapters/index.d.ts

@@ -0,0 +1,4 @@
+/** Adapter registry — adding a runtime = registering one adapter here. */
+import type { RuntimeAdapter } from "../types.ts";
+export declare const ADAPTERS: Record<string, RuntimeAdapter>;
+export declare function getAdapter(name: string): RuntimeAdapter;

package/dist/adapters/index.js

@@ -0,0 +1,13 @@
+import { claudeCodeAdapter } from "./claudeCode.js";
+import { openCodeAdapter } from "./openCode.js";
+export const ADAPTERS = {
+    [claudeCodeAdapter.name]: claudeCodeAdapter,
+    [openCodeAdapter.name]: openCodeAdapter,
+};
+export function getAdapter(name) {
+    const a = ADAPTERS[name];
+    if (!a) {
+        throw new Error(`unknown runtime '${name}'. available: ${Object.keys(ADAPTERS).join(", ")}`);
+    }
+    return a;
+}

package/dist/adapters/openCode.d.ts

@@ -0,0 +1,15 @@
+import type { RuntimeAdapter } from "../types.ts";
+export interface CoreResult {
+    skillFired: string | null;
+    trajectory: string[];
+    cost: number;
+}
+/** Recursively scan parsed OpenCode output for a fired skill + tool trajectory. Defensive:
+ * handles "skill name in input.name" or "input.skill", single JSON or JSONL of events. */
+export declare function extractFromOpenCode(parsed: unknown): CoreResult;
+/** Parse stdout (whole-JSON or JSONL) and report how many events were seen (0 = no output). */
+export declare function parseOutput(stdout: string): {
+    result: CoreResult;
+    eventCount: number;
+};
+export declare const openCodeAdapter: RuntimeAdapter;

package/dist/adapters/openCode.js

@@ -0,0 +1,109 @@
+/** OpenCode adapter — the second runtime (cross-vendor; BYO-key). Reads `.claude/skills/`.
+ * Activation surfaces as a tool part named "skill" with the name in state.input.name (confirmed
+ * against opencode v1.17.9 `run --format json`). Distinguishes behavioral outcome from infra
+ * failure (finding #1). GOTCHA: opencode `run` hangs unless stdin is closed — we use stdio ignore. */
+import { spawn } from "node:child_process";
+/** Recursively scan parsed OpenCode output for a fired skill + tool trajectory. Defensive:
+ * handles "skill name in input.name" or "input.skill", single JSON or JSONL of events. */
+export function extractFromOpenCode(parsed) {
+    let skillFired = null;
+    let cost = 0;
+    const trajectory = [];
+    const visit = (node) => {
+        if (Array.isArray(node)) {
+            node.forEach(visit);
+            return;
+        }
+        if (!node || typeof node !== "object")
+            return;
+        const o = node;
+        if (o["type"] === "tool" && typeof o["tool"] === "string") {
+            const tool = o["tool"];
+            trajectory.push(tool);
+            if (tool === "skill") {
+                const state = o["state"];
+                const input = (state?.["input"] ?? o["input"]);
+                const name = (input?.["name"] ?? input?.["skill"]);
+                if (name)
+                    skillFired = name;
+            }
+        }
+        if (typeof o["cost"] === "number")
+            cost = o["cost"];
+        for (const v of Object.values(o))
+            visit(v);
+    };
+    visit(parsed);
+    return { skillFired, trajectory, cost };
+}
+/** Parse stdout (whole-JSON or JSONL) and report how many events were seen (0 = no output). */
+export function parseOutput(stdout) {
+    const trimmed = stdout.trim();
+    if (!trimmed)
+        return { result: { skillFired: null, trajectory: [], cost: 0 }, eventCount: 0 };
+    try {
+        const whole = JSON.parse(trimmed);
+        return { result: extractFromOpenCode(whole), eventCount: Array.isArray(whole) ? whole.length : 1 };
+    }
+    catch {
+        const events = [];
+        for (const line of trimmed.split("\n")) {
+            const s = line.trim();
+            if (!s)
+                continue;
+            try {
+                events.push(JSON.parse(s));
+            }
+            catch { /* skip non-json log lines */ }
+        }
+        return { result: extractFromOpenCode(events), eventCount: events.length };
+    }
+}
+export const openCodeAdapter = {
+    name: "opencode",
+    probe(prompt, { cwd, timeoutMs = 120_000, model }) {
+        // config/CLI --model wins; env var is a fallback convenience
+        const effectiveModel = model ?? process.env["SKILL_PROBE_OPENCODE_MODEL"];
+        const args = ["run", prompt, "--format", "json"];
+        if (effectiveModel)
+            args.push("--model", effectiveModel);
+        return new Promise((resolve) => {
+            let settled = false, timedOut = false;
+            const child = spawn("opencode", args, { cwd, stdio: ["ignore", "pipe", "pipe"] });
+            let out = "", err = "";
+            const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, timeoutMs);
+            const finish = (r) => {
+                if (settled)
+                    return;
+                settled = true;
+                clearTimeout(timer);
+                resolve(r);
+            };
+            child.stdout.on("data", (d) => { out += d.toString(); });
+            child.stderr.on("data", (d) => { err += d.toString(); });
+            child.on("error", (e) => finish({
+                status: "error", skillFired: null, trajectory: [], cost: 0,
+                error: `spawn failed: ${e.message} (is 'opencode' installed and on PATH?)`,
+            }));
+            child.on("close", () => {
+                const { result, eventCount } = parseOutput(out);
+                if (timedOut) {
+                    return finish({
+                        status: "error", skillFired: null, trajectory: result.trajectory, cost: result.cost,
+                        error: `timed out after ${timeoutMs}ms`,
+                    });
+                }
+                if (eventCount === 0) {
+                    // surface whatever diagnostic we have: stderr, else a non-JSON stdout log (opencode
+                    // prints provider/model errors to stdout), else a generic hint.
+                    const diag = err.trim() || out.trim();
+                    return finish({
+                        status: "error", skillFired: null, trajectory: [], cost: 0,
+                        error: diag ? diag.slice(-200) : "no output (auth/model/hang? check `opencode auth list`)",
+                    });
+                }
+                finish({ status: "ok", skillFired: result.skillFired, trajectory: result.trajectory, cost: result.cost });
+            });
+        });
+    },
+};

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,89 @@
+#!/usr/bin/env node
+/** skill-probe CLI.
+ *   skill-probe --config <f>                  audit a co-loaded skill library
+ *   skill-probe fix --config <f> --skill <n>  rewrite a skill's description & prove the lift
+ * Exit: 0 = pass/applied, 1 = behavioral fail/reverted, 2 = inconclusive/infra error/usage. */
+import { parseArgs } from "node:util";
+import { runAudit } from "./eval.js";
+import { runFix } from "./fix.js";
+import { loadConfig, ConfigError } from "./config.js";
+import { renderTable, renderJson, renderFix } from "./report.js";
+const HELP = `skill-probe — audit a co-loaded agent skill library by real activation behavior.
+
+USAGE
+  skill-probe --config <file.json> [options]                 # audit
+  skill-probe fix --config <file.json> --skill <name> [opts] # rewrite a description, prove the lift
+
+OPTIONS
+  -c, --config <file>      JSON config { cwd, cases:[{prompt,expected}], ... } (required).
+                           Relative cwd resolves against the CONFIG FILE's directory.
+      --runtime <name>     claude-code | opencode (default: claude-code)
+      --model <id>         pin the model (e.g. claude-sonnet-4-6 or anthropic/claude-haiku-4-5).
+                           Recorded in the report — REQUIRED for a fair cross-runtime comparison.
+      --k <n>              max VALID repeats per prompt (positive integer; default 10)
+      --threshold <p>      min acceptable reliability, strictly 0..1 (default 0.7 = smoke test)
+      --conf <p>           confidence level: 0.9 | 0.95 | 0.99 (default 0.95)
+      --json               emit JSON instead of a table
+  fix only:
+      --skill <name>       the skill whose description to rewrite (uses the config's cases for it)
+      --apply-threshold <p> min P(improvement) to keep the rewrite (default 0.9)
+  -h, --help               show this help
+
+Reliability carries a Wilson CI + sample size k. fix rewrites the description, runs an INTERLEAVED
+before/after, and applies it only if the Bayesian P(improvement) clears the apply-threshold.`;
+async function audit(values) {
+    const cfg = loadConfig(values["config"], {
+        runtime: values["runtime"], model: values["model"],
+        k: values["k"], threshold: values["threshold"],
+        conf: values["conf"],
+    });
+    const result = await runAudit(cfg);
+    console.log(values["json"] ? renderJson(result) : renderTable(result));
+    return result.exitCode;
+}
+async function fix(values) {
+    const skill = values["skill"];
+    if (!skill) {
+        console.error("fix requires --skill <name>");
+        return 2;
+    }
+    const cfg = loadConfig(values["config"], {
+        runtime: values["runtime"], model: values["model"],
+        k: values["k"], conf: values["conf"],
+    });
+    const applyBar = values["apply-threshold"] ? Number(values["apply-threshold"]) : 0.9;
+    const result = await runFix(cfg, { skill, k: cfg.k, applyBar, ...(cfg.model ? { model: cfg.model } : {}) });
+    console.log(values["json"] ? renderJson(result) : renderFix(result));
+    return result.applied ? 0 : 1;
+}
+async function main() {
+    const { values, positionals } = parseArgs({
+        allowPositionals: true,
+        options: {
+            config: { type: "string", short: "c" },
+            runtime: { type: "string" },
+            model: { type: "string" },
+            k: { type: "string" },
+            threshold: { type: "string" },
+            conf: { type: "string" },
+            skill: { type: "string" },
+            "apply-threshold": { type: "string" },
+            json: { type: "boolean", default: false },
+            help: { type: "boolean", short: "h", default: false },
+        },
+    });
+    if (values.help || !values.config) {
+        console.log(HELP);
+        process.exit(values.help ? 0 : 2);
+    }
+    const cmd = positionals[0];
+    const code = cmd === "fix" ? await fix(values) : await audit(values);
+    process.exit(code);
+}
+main().catch((e) => {
+    if (e instanceof ConfigError)
+        console.error(e.message);
+    else
+        console.error("skill-probe failed:", e instanceof Error ? e.message : e);
+    process.exit(2);
+});

package/dist/config.d.ts

@@ -0,0 +1,18 @@
+import type { Config } from "./types.ts";
+export declare class ConfigError extends Error {
+    readonly errors: string[];
+    constructor(errors: string[]);
+}
+/** Validate a raw config object (already merged with CLI flag overrides).
+ * `configPath` is used to resolve a relative `cwd` against the CONFIG FILE's
+ * directory — not the caller's shell (finding #3). Throws ConfigError on any problem. */
+export declare function validateConfig(raw: unknown, configPath: string): Config;
+/** CLI flag overrides (already string-typed from argv); numerics are coerced then validated. */
+export interface Overrides {
+    runtime?: string | undefined;
+    model?: string | undefined;
+    k?: string | undefined;
+    threshold?: string | undefined;
+    conf?: string | undefined;
+}
+export declare function loadConfig(configPath: string, overrides?: Overrides): Config;

package/dist/config.js

@@ -0,0 +1,88 @@
+/** Config loading + validation. Bad inputs must fail loudly with a clear message, never
+ * silently produce NaN intervals or zero-run audits. */
+import { readFileSync } from "node:fs";
+import { resolve, dirname } from "node:path";
+const VALID_CONF = new Set([0.9, 0.95, 0.99]);
+export class ConfigError extends Error {
+    errors;
+    constructor(errors) {
+        super("invalid config:\n  - " + errors.join("\n  - "));
+        this.name = "ConfigError";
+        this.errors = errors;
+    }
+}
+/** Validate a raw config object (already merged with CLI flag overrides).
+ * `configPath` is used to resolve a relative `cwd` against the CONFIG FILE's
+ * directory — not the caller's shell (finding #3). Throws ConfigError on any problem. */
+export function validateConfig(raw, configPath) {
+    const r = (raw ?? {});
+    const errs = [];
+    const runtime = typeof r["runtime"] === "string" && r["runtime"] ? r["runtime"] : "claude-code";
+    let model;
+    if (r["model"] !== undefined) {
+        if (typeof r["model"] !== "string" || !r["model"])
+            errs.push(`model must be a non-empty string if set`);
+        else
+            model = r["model"];
+    }
+    const cwdRaw = typeof r["cwd"] === "string" ? r["cwd"] : ".";
+    const cwd = resolve(dirname(resolve(configPath)), cwdRaw);
+    const k = r["k"] ?? 10;
+    if (typeof k !== "number" || !Number.isInteger(k) || k <= 0) {
+        errs.push(`k must be a positive integer (got ${JSON.stringify(r["k"])})`);
+    }
+    // Default 0.7 = a "smoke test" bar: a clean 10/10 at default k=10 certifies (Wilson lo ≈ 0.72).
+    // For strict certification (e.g. >=0.9) raise BOTH threshold and k — the report says how much.
+    const threshold = r["threshold"] ?? 0.7;
+    if (typeof threshold !== "number" || Number.isNaN(threshold) || threshold <= 0 || threshold >= 1) {
+        errs.push(`threshold must be a number strictly between 0 and 1 (got ${JSON.stringify(r["threshold"])})`);
+    }
+    const conf = r["conf"] ?? 0.95;
+    if (typeof conf !== "number" || !VALID_CONF.has(conf)) {
+        errs.push(`conf must be one of 0.9, 0.95, 0.99 (got ${JSON.stringify(r["conf"])})`);
+    }
+    const cases = Array.isArray(r["cases"]) ? r["cases"] : [];
+    if (cases.length === 0)
+        errs.push("config needs a non-empty `cases` array");
+    cases.forEach((c, i) => {
+        const cc = (c ?? {});
+        if (typeof cc["prompt"] !== "string" || !cc["prompt"]) {
+            errs.push(`cases[${i}].prompt must be a non-empty string`);
+        }
+        if (!(cc["expected"] === null || typeof cc["expected"] === "string")) {
+            errs.push(`cases[${i}].expected must be a skill name (string) or null`);
+        }
+    });
+    if (errs.length)
+        throw new ConfigError(errs);
+    return {
+        runtime,
+        ...(model ? { model } : {}),
+        cwd,
+        k: k,
+        threshold: threshold,
+        conf: conf,
+        cases: cases,
+    };
+}
+export function loadConfig(configPath, overrides = {}) {
+    let fileRaw;
+    try {
+        fileRaw = JSON.parse(readFileSync(resolve(configPath), "utf8"));
+    }
+    catch (e) {
+        throw new ConfigError([`could not read/parse config '${configPath}': ${e instanceof Error ? e.message : e}`]);
+    }
+    const merged = { ...fileRaw };
+    if (overrides.runtime !== undefined)
+        merged["runtime"] = overrides.runtime;
+    if (overrides.model !== undefined)
+        merged["model"] = overrides.model;
+    if (overrides.k !== undefined)
+        merged["k"] = Number(overrides.k);
+    if (overrides.threshold !== undefined)
+        merged["threshold"] = Number(overrides.threshold);
+    if (overrides.conf !== undefined)
+        merged["conf"] = Number(overrides.conf);
+    return validateConfig(merged, configPath);
+}

package/dist/eval.d.ts

@@ -0,0 +1,33 @@
+/** The audit: run every case co-loaded, compute reliability (with CI), detect trigger-theft,
+ * and keep infrastructure failure distinct from behavioral failure (finding #1). */
+import type { Config } from "./types.ts";
+import { reliability } from "./stats.ts";
+import { type RunStats } from "./orchestrator.ts";
+/** Four honest states. `error` = the runtime failed enough that the measurement is untrustworthy. */
+export type Verdict = "pass" | "fail" | "inconclusive" | "error";
+export interface CaseResult {
+    prompt: string;
+    expected: string | null;
+    stats: RunStats;
+    reliability: ReturnType<typeof reliability>;
+    /** non-expected, non-None skills that fired = interference / trigger-theft */
+    theft: string[];
+    decision: "above" | "below" | "undecided";
+    verdict: Verdict;
+}
+export interface AuditResult {
+    runtime: string;
+    model: string;
+    threshold: number;
+    cases: CaseResult[];
+    totalCost: number;
+    counts: {
+        pass: number;
+        fail: number;
+        inconclusive: number;
+        error: number;
+    };
+    /** 0 = all pass; 1 = a behavioral failure/theft; 2 = inconclusive or infrastructure error */
+    exitCode: 0 | 1 | 2;
+}
+export declare function runAudit(cfg: Config): Promise<AuditResult>;