vigiles 21.0.1 → 22.0.0

package/README.md

@@ -175,6 +175,8 @@ Every path, script, symbol, and rule verified against reality — plus tool cont
 A hook that blocks nothing, a skill that hijacks unrelated prompts, context that never reaches the model — each passes a naive "did it run?" check. That gap is **false confidence**: a guard that looks like it works and silently doesn't. vigiles tests the real thing — hooks block, skills fire, subagents finish what they promised, a stray `git push` is caught before it happens. It drives a scripted stand-in for the model, not a live call, so it needs no key and runs on every commit.
 **[How testing works →](docs/harness-testing.md)**
 
+**Nothing you scan leaves your machine.** `lint`, `audit` and the deterministic test tiers make no network call at all — no telemetry, no analytics, no HTTP client in the package. Evals drive your own `claude` CLI on your own subscription, so no third party is introduced. **[What is and isn't transmitted →](docs/safety.md#does-vigiles-send-my-code-anywhere)**
+
 ### 📊 Eval — the only way to put a real number on cost
 
 _"Caveman Mode cuts 65% of your tokens." Says who?_ vigiles A/Bs the claim on real coding tasks and hands you three numbers: the **token bill**, whether it hit its **target**, and whether your code still **works**.

package/dist/audit-score.js

@@ -198,9 +198,24 @@ function structure(r) {
             `${String(noContract)} agent(s) inherit all tools (no contract) (advisory)`,
         ]
         : [];
+    // A confident breakage caps this ring too, so the breakdown can't render a
+    // healthy `●` over a dead surface — the defect this fixes was measured as
+    // "Structure 92 ●" while an agent named a never-available tool. Only the
+    // rows that mean a surface is DEAD count; the typo'd `disallowedTools` and
+    // invalid model/color rows are footguns, not breakage, so they stay out.
+    const breakage = deadTools +
+        deadMcpTools +
+        r.hookEventIssues.length +
+        r.mcpIssues.length +
+        r.mcpHookIssues.length +
+        r.frontmatterIssues.length +
+        r.pluginLayoutIssues.length +
+        r.skillFenceIssues.length +
+        r.hookBlockFindings.length +
+        r.hookMatcherFindings.length;
     return {
         key: "Structure",
-        score,
+        score: breakage > 0 ? Math.min(score, score_core_js_1.CONFIDENT_BREAKAGE_CAP) : score,
         weight: 1,
         findings: [...findings, ...advisory],
     };
@@ -248,15 +263,39 @@ function safety(r) {
     // one at a time would bury the subagent findings under a list as long as the
     // skill corpus. Same reasoning as the report section; see `trifectaLines`.
     const unfenced = r.trifectaFindings.filter((f) => f.kind === "skill" && f.finding.fence === "none");
+    // An INEFFECTIVE fence (`disallowed-tools:` that closes no leg) keeps its own
+    // line while there are FEW of them — that is the documented intent in
+    // core/lethal-trifecta.ts: it is a genuine mistake, the author believed they had
+    // fenced, so naming the skill is what a reader acts on.
+    //
+    // MEASURED 2026-08-28 that the premise behind that shape — "Rare" — does not
+    // hold: a fixture where every skill carried a naive `disallowed-tools: WebFetch`
+    // put ALL of them in this state, and the ring printed the same ~450-character
+    // paragraph ten times, ~4,500 characters into the terminal report. Past the
+    // threshold it stops being N facts about N skills and becomes one fact about the
+    // harness — exactly the reasoning the `fence: "none"` aggregate already uses.
+    // Names are kept as detail, so nothing is lost, only repetition.
+    const ineffective = r.trifectaFindings.filter((f) => f.finding.severity === "advisory" &&
+        !unfenced.includes(f) &&
+        f.kind === "skill" &&
+        f.finding.fence === "ineffective");
+    const collapseIneffective = ineffective.length > MAX_NAMED_INEFFECTIVE_FENCES;
     for (const f of r.trifectaFindings) {
         if (f.finding.severity !== "advisory")
             continue;
         if (unfenced.includes(f))
             continue;
+        if (collapseIneffective && ineffective.includes(f))
+            continue;
         findings.push(f.kind === "skill"
             ? `${f.name}: ${f.finding.message}`
             : `${f.name} inherits all tools — the "lethal trifecta" (reads data, reaches the web, runs commands) plus every other capability, so a prompt injection could exfiltrate secrets`);
     }
+    if (collapseIneffective) {
+        findings.push(`${String(ineffective.length)} skill(s) declare a \`disallowed-tools:\` that closes no lethal-trifecta leg — a leg is closed only when EVERY built-in supplying it is denied: ` +
+            `${ineffective.map((f) => f.name).join(", ")}. ` +
+            `Name every supplier of the leg you mean to close — private-data read = Read, Grep, Glob, Bash; untrusted intake = WebFetch, WebSearch, Bash; exfiltration = WebFetch, WebSearch, Bash.`);
+    }
     if (unfenced.length > 0) {
         findings.push(`${String(unfenced.length)} skill(s) declare no \`disallowed-tools:\` fence, so each inherits every tool the session grants — reads data, reaches the web, runs commands. \`allowed-tools:\` pre-approves, it does not restrict, so narrowing it does not reduce this; one \`disallowed-tools:\` line per skill drops a leg.`);
     }
@@ -456,9 +495,19 @@ function auditScore(report, opts = {}) {
     // of the rings — averaging would let a real problem in one category be diluted
     // by clean siblings. The rings above stay a diagnostic breakdown; Tested and
     // Evaluated (both advisory) are never summed in (neither drags the grade).
-    const { score: overall } = (0, score_core_js_1.computeIntegrityScore)((0, score_core_js_1.reportDeductions)(report));
+    const { score: summed } = (0, score_core_js_1.computeIntegrityScore)((0, score_core_js_1.reportDeductions)(report));
+    // A confident breakage caps the headline too, so it can never read `A` while a
+    // surface is definitively dead (score-core.ts::applyBreakageCap).
+    const overall = (0, score_core_js_1.applyBreakageCap)(summed, report);
     return { overall, grade: (0, score_core_js_1.gradeFor)(overall), categories, empty: false };
 }
+/**
+ * How many INEFFECTIVE `disallowed-tools:` fences are named one at a time before
+ * the Safety ring collapses them into a single line. Past this it is one fact
+ * about the harness, not N facts about N skills — see the measurement in
+ * {@link safety}.
+ */
+const MAX_NAMED_INEFFECTIVE_FENCES = 3;
 // A 22-cell bar gauge ("ring" in the terminal; the real rings are the HTML).
 const BAR_CELLS = 22;
 /** A glyph that signals the band at a glance (green/amber/red, no ANSI needed).

package/dist/cli.d.ts

@@ -9,5 +9,6 @@
  * `self-command-refs.test.ts` did not catch it because it guards against refs to
  * REMOVED commands, not against a list that merely stops growing.
  */
-export {};
+/** Reason the most recent `loadSpec()` returned null, or null if it succeeded. */
+export declare function specLoadFailureReason(): string | null;
 //# sourceMappingURL=cli.d.ts.map

package/dist/cli.js

@@ -11,8 +11,10 @@
  * REMOVED commands, not against a list that merely stops growing.
  */
 Object.defineProperty(exports, "__esModule", { value: true });
+exports.specLoadFailureReason = specLoadFailureReason;
 const node_fs_1 = require("node:fs");
 const node_path_1 = require("node:path");
+const node_child_process_1 = require("node:child_process");
 const glob_1 = require("glob");
 const generate_types_js_1 = require("./core/generate-types.js");
 const generate_harness_js_1 = require("./core/generate-harness.js");
@@ -96,54 +98,168 @@ function findSpecs(pattern) {
         cwd: process.cwd(),
     });
 }
-async function loadSpec(specPath) {
-    const fullPath = (0, node_path_1.resolve)(process.cwd(), specPath);
-    // Try multiple dist/ path strategies
-    const candidates = [];
-    // src/ → dist/ mapping (e.g., src/CLAUDE.md.spec.ts → dist/CLAUDE.md.spec.js)
-    if (fullPath.includes("/src/")) {
-        candidates.push(fullPath.replace(/\/src\//, "/dist/").replace(/\.ts$/, ".js"));
-    }
-    // Root-level spec → dist/ (e.g., CLAUDE.md.spec.ts → dist/CLAUDE.md.spec.js)
-    const dir = fullPath.substring(0, fullPath.lastIndexOf("/"));
-    const base = fullPath.substring(fullPath.lastIndexOf("/") + 1);
-    candidates.push((0, node_path_1.resolve)(dir, "dist", base.replace(/\.ts$/, ".js")));
-    // examples/ → dist/examples/ mapping
-    candidates.push(fullPath
-        .replace(/\.ts$/, ".js")
-        .replace(process.cwd(), (0, node_path_1.resolve)(process.cwd(), "dist")));
-    for (const distPath of candidates) {
-        if ((0, node_fs_1.existsSync)(distPath)) {
+/**
+ * Why the last `loadSpec()` returned null.
+ *
+ * Kept as module state rather than a widened return type: `loadSpec` has six
+ * call sites and only one of them reports to a human.
+ */
+let lastSpecLoadFailure = null;
+/** Reason the most recent `loadSpec()` returned null, or null if it succeeded. */
+function specLoadFailureReason() {
+    return lastSpecLoadFailure;
+}
+/**
+ * How long one spec may take to evaluate before the host is killed.
+ *
+ * Overridable because 15s is a guess that fits the specs we have seen, not a
+ * law; a repo with genuinely slow specs should be able to raise it rather than
+ * discover the number by hitting it.
+ */
+const SPEC_DEADLINE_MS = Number(process.env.VIGILES_SPEC_TIMEOUT_MS) || 15_000;
+let host = null;
+/** The compiled host entry, beside this file in `dist/`. */
+function hostEntry() {
+    return (0, node_path_1.resolve)(__dirname, "spec-host.mjs");
+}
+function startHost() {
+    const child = (0, node_child_process_1.spawn)(process.execPath, [hostEntry()], {
+        cwd: process.cwd(),
+        stdio: ["pipe", "pipe", "pipe"],
+    });
+    const h = { child, pending: new Map(), started: null, buffered: "" };
+    child.stdout.setEncoding("utf-8");
+    child.stdout.on("data", (chunk) => {
+        h.buffered += chunk;
+        let nl;
+        while ((nl = h.buffered.indexOf("\n")) >= 0) {
+            const line = h.buffered.slice(0, nl).trim();
+            h.buffered = h.buffered.slice(nl + 1);
+            if (!line)
+                continue;
+            let reply;
             try {
-                const mod = (await import(distPath));
-                // CJS double-default: `{ default: { default: spec } }`.
-                const raw = mod.default;
-                if (raw && typeof raw === "object" && "default" in raw) {
-                    return raw.default;
-                }
-                return raw;
+                reply = JSON.parse(line);
             }
             catch {
-                // Try next candidate
+                continue; // not ours; a spec writing to stdout cannot corrupt the stream
+            }
+            if ("phase" in reply) {
+                h.started = reply.path;
+                continue;
             }
+            const done = h.pending.get(reply.path);
+            h.pending.delete(reply.path);
+            done?.(reply);
         }
+    });
+    // Anything the child says on stderr is the spec's own noise; keep it out of
+    // our stdout so `--json` consumers are not corrupted, but do not lose it.
+    child.stderr.setEncoding("utf-8");
+    child.stderr.on("data", (chunk) => process.stderr.write(chunk));
+    // 🔴 Unreferenced, or the CLI never exits. A piped child and its three
+    // streams each hold the event loop open, so `compile` finished its work and
+    // then hung forever waiting on a host that had nothing left to say. The
+    // in-flight deadline timer keeps the loop alive while a request is pending,
+    // which is exactly as long as we need it.
+    // ONE exit listener per host, not one per request: with concurrent callers the
+    // per-request version added a listener each time and Node warned at eleven.
+    // It fails every outstanding request, because a dead host answers none of them.
+    child.once("exit", () => {
+        const waiting = [...h.pending.values()];
+        h.pending.clear();
+        for (const settle of waiting)
+            settle("died");
+    });
+    // The stdio types are Readable/Writable, which do not declare `unref` — the
+    // objects are pipes and do have it. Optional-called so this stays correct if
+    // a platform ever hands back a stream that genuinely lacks it.
+    const unref = (s) => s?.unref?.();
+    child.unref();
+    unref(child.stdin);
+    unref(child.stdout);
+    unref(child.stderr);
+    return h;
+}
+/**
+ * Kill the host and forget it; the next request starts a fresh one.
+ *
+ * Outstanding requests are failed rather than dropped: a killed host will never
+ * answer them, and a promise nobody settles is a hang wearing a different hat.
+ */
+function dropHost() {
+    if (!host)
+        return;
+    const dying = host;
+    host = null;
+    const waiting = [...dying.pending.values()];
+    dying.pending.clear();
+    dying.child.kill("SIGKILL");
+    for (const settle of waiting)
+        settle("died");
+}
+process.on("exit", dropHost);
+/**
+ * Load one spec in the spec host.
+ *
+ * 🔴 **Why a child process rather than `import()` here.** A module evaluation
+ * cannot be cancelled once started — `Promise.race` hands control back but the
+ * evaluation keeps running and holds the event loop — so an in-process loader
+ * gives a stalled spec an unbounded hang in `compile`, `test` and `audit`. It
+ * also cannot tell whether a failed spec already ran (Node reports
+ * `ERR_MODULE_NOT_FOUND` and `SyntaxError` both before and during evaluation),
+ * which is what made the previous two-loader arrangement unfixable rather than
+ * merely buggy: it had to guess whether re-running was safe.
+ *
+ * The host is spawned with `process.execPath` — never `npx` — so nothing is
+ * fetched and nothing needs installing.
+ */
+async function loadSpec(specPath) {
+    const fullPath = (0, node_path_1.resolve)(process.cwd(), specPath);
+    lastSpecLoadFailure = null;
+    if (!(0, node_fs_1.existsSync)(fullPath)) {
+        lastSpecLoadFailure = `no such file: ${specPath}`;
+        return null;
     }
-    // Try loading .ts directly via tsx
-    try {
-        const { execSync } = require("node:child_process");
-        // Handle ESM/CJS double-default: m.default may itself have a .default
-        const script = `import(${JSON.stringify(fullPath)}).then(m => { const d = m.default?.default ?? m.default; console.log(JSON.stringify(d)); })`;
-        const output = execSync(`npx tsx -e '${script.replace(/'/g, "'\\''")}'`, {
-            encoding: "utf-8",
-            cwd: process.cwd(),
-            stdio: ["pipe", "pipe", "pipe"],
-            timeout: 15000,
-        });
-        return JSON.parse(output.trim());
+    host ??= startHost();
+    const h = host;
+    const reply = await new Promise((done) => {
+        let settled = false;
+        const finish = (r) => {
+            if (settled)
+                return;
+            settled = true;
+            clearTimeout(timer);
+            h.pending.delete(fullPath);
+            done(r);
+        };
+        const timer = setTimeout(() => {
+            finish("timeout");
+        }, SPEC_DEADLINE_MS);
+        h.pending.set(fullPath, finish);
+        h.child.stdin.write(JSON.stringify({ path: fullPath }) + "\n");
+    });
+    if (reply === "timeout") {
+        // The host's last `start` names the spec that stalled. Without it a hang
+        // produced N identical failures and no culprit.
+        const culprit = h.started ?? fullPath;
+        dropHost();
+        lastSpecLoadFailure =
+            `evaluating ${(0, node_path_1.relative)(process.cwd(), culprit)} exceeded ` +
+                `${SPEC_DEADLINE_MS}ms and was killed. Set VIGILES_SPEC_TIMEOUT_MS to ` +
+                `raise the limit, or look for a top-level await that never settles.`;
+        return null;
     }
-    catch {
+    if (reply === "died") {
+        dropHost();
+        lastSpecLoadFailure = "the spec host exited unexpectedly.";
+        return null;
+    }
+    if (!("ok" in reply) || !reply.ok) {
+        lastSpecLoadFailure = `the spec did not load. ${"error" in reply ? reply.error : "no reason given"}`;
         return null;
     }
+    return reply.value;
 }
 // ---------------------------------------------------------------------------
 // Output helpers
@@ -357,7 +473,7 @@ async function compile(specPaths, config, opts = {}) {
         const spec = await loadSpec(specPath);
         if (!spec) {
             console.log(`\n✗ ${specPath} — failed to load`);
-            console.log(`  Ensure the spec is compiled: run \`npm run build\` first.`);
+            console.log(`  ${specLoadFailureReason() ?? "reason unavailable"}`);
             allValid = false;
             continue;
         }

package/dist/core/description-overlap.js

@@ -8,7 +8,10 @@ exports.findDescriptionOverlaps = findDescriptionOverlaps;
  * selector, so the wrong one fires (a precision collision). This catches a
  * `--trigger`-class problem with NO model, reusing the NCD engine in proofs.ts
  * (the same one `findSimilarRules` uses) — the bridge between the deterministic
- * and behavioral columns, and a check no other plugin linter has.
+ * and behavioral columns. The CHECK is not unique — cisco-ai-defense/skill-scanner
+ * ships `--check-overlap` under a security framing (skill impersonation). What is
+ * ours: the cutoff calibrated against a real corpus, and the precision-collision
+ * framing (verified 2026-08-28).
  *
  * Calibrated HIGH-PRECISION against the mid-2026 sweep: across 4678 within-plugin
  * skill-description pairs, the MOST-similar legitimately-distinct pair

package/dist/score-core.d.ts

@@ -57,6 +57,54 @@ export declare const W_TRIFECTA_MAX = 30;
 export declare const TRIFECTA_LABEL = "unit(s) can read data, reach the web, and run commands \u2014 the \"lethal trifecta\", so a prompt injection could exfiltrate secrets";
 /** Map a 0–100 structural-health score to its letter grade (A ≥90 … F <60). */
 export declare function gradeFor(score: number): PluginScore["grade"];
+/**
+ * The health score a report may NOT exceed while it carries a CONFIDENT BREAKAGE
+ * — one point below the healthy band, so a definitively-broken harness can never
+ * render as `●` / grade `A`.
+ *
+ * WHY a cap and not a heavier weight: the score is a SUM of deductions, so one
+ * real breakage among many clean surfaces is diluted by its siblings. MEASURED
+ * 2026-08-28 on a fixture (10 distinct-description skills + one agent naming a
+ * never-available tool): Structure scored **92 with a `●` green dot** while
+ * carrying a dead tool contract, and the grade was driven instead by the
+ * lethal-trifecta pattern that {@link W_TRIFECTA} deliberately calls a ding and
+ * not a fail. A weight big enough to fix that would cry wolf on a large clean
+ * plugin; a cap fixes the reading without touching the arithmetic.
+ *
+ * The same defect, with higher stakes, is documented in an external 517-skill
+ * catalog run of a different scanner: its weighted aggregate returned MEDIUM for
+ * a skill carrying THREE HIGH findings, so gating on the aggregate would have
+ * admitted exactly the skill the gate existed to stop. That team's conclusion —
+ * judge by the worst finding, not the aggregate — is what this encodes.
+ */
+export declare const CONFIDENT_BREAKAGE_CAP = 89;
+/**
+ * The findings that CAP the score — each one means a surface is definitively
+ * dead, not merely risky: it will silently not run, not resolve, or not register.
+ *
+ * ENUMERATED on purpose, never derived from "severity". Two properties decide
+ * membership, and both must hold: the finding is DECIDABLE from the artifact plus
+ * the world (the `structural-closed` / `external-decidable` buckets of the
+ * lint-rule-calibration rule), and its consequence is BREAKAGE rather than
+ * exposure. So the heuristic rings stay out by construction — a lethal-trifecta
+ * unit and a description overlap are real signals but are a capability PATTERN
+ * and a calibrated PROXY, and capping on either is how a gate earns the reputation
+ * that gets it switched off.
+ *
+ * Adding a row here is a deliberate act: it makes a finding grade-capping for
+ * every consumer, so it belongs only to a check whose false-positive rate is
+ * already known to be ~0.
+ */
+export declare function confidentBreakages(r: ScanReport): {
+    readonly n: number;
+    readonly label: string;
+}[];
+/**
+ * Cap a health score at {@link CONFIDENT_BREAKAGE_CAP} when the report carries any
+ * {@link confidentBreakages} finding. Applied to the OVERALL and to the ring that
+ * owns the finding, so the headline and the breakdown cannot disagree.
+ */
+export declare function applyBreakageCap(score: number, r: ScanReport): number;
 /** One deduction: a count, its per-item weight, and the label if non-zero. */
 export interface Deduction {
     readonly n: number;

package/dist/score-core.js

@@ -1,7 +1,9 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.TRIFECTA_LABEL = exports.W_TRIFECTA_MAX = exports.W_TRIFECTA = exports.W_NO_CONTRACT = exports.W_OVERLAP = exports.W_DANGLING_REF = exports.W_NO_DESCRIPTION = exports.W_MISSING_HOOK = void 0;
+exports.CONFIDENT_BREAKAGE_CAP = exports.TRIFECTA_LABEL = exports.W_TRIFECTA_MAX = exports.W_TRIFECTA = exports.W_NO_CONTRACT = exports.W_OVERLAP = exports.W_DANGLING_REF = exports.W_NO_DESCRIPTION = exports.W_MISSING_HOOK = void 0;
 exports.gradeFor = gradeFor;
+exports.confidentBreakages = confidentBreakages;
+exports.applyBreakageCap = applyBreakageCap;
 exports.trifectaExposure = trifectaExposure;
 exports.reportDeductions = reportDeductions;
 exports.isEmptyMachine = isEmptyMachine;
@@ -52,6 +54,90 @@ function gradeFor(score) {
         return "D";
     return "F";
 }
+/**
+ * The health score a report may NOT exceed while it carries a CONFIDENT BREAKAGE
+ * — one point below the healthy band, so a definitively-broken harness can never
+ * render as `●` / grade `A`.
+ *
+ * WHY a cap and not a heavier weight: the score is a SUM of deductions, so one
+ * real breakage among many clean surfaces is diluted by its siblings. MEASURED
+ * 2026-08-28 on a fixture (10 distinct-description skills + one agent naming a
+ * never-available tool): Structure scored **92 with a `●` green dot** while
+ * carrying a dead tool contract, and the grade was driven instead by the
+ * lethal-trifecta pattern that {@link W_TRIFECTA} deliberately calls a ding and
+ * not a fail. A weight big enough to fix that would cry wolf on a large clean
+ * plugin; a cap fixes the reading without touching the arithmetic.
+ *
+ * The same defect, with higher stakes, is documented in an external 517-skill
+ * catalog run of a different scanner: its weighted aggregate returned MEDIUM for
+ * a skill carrying THREE HIGH findings, so gating on the aggregate would have
+ * admitted exactly the skill the gate existed to stop. That team's conclusion —
+ * judge by the worst finding, not the aggregate — is what this encodes.
+ */
+exports.CONFIDENT_BREAKAGE_CAP = 89;
+/**
+ * The findings that CAP the score — each one means a surface is definitively
+ * dead, not merely risky: it will silently not run, not resolve, or not register.
+ *
+ * ENUMERATED on purpose, never derived from "severity". Two properties decide
+ * membership, and both must hold: the finding is DECIDABLE from the artifact plus
+ * the world (the `structural-closed` / `external-decidable` buckets of the
+ * lint-rule-calibration rule), and its consequence is BREAKAGE rather than
+ * exposure. So the heuristic rings stay out by construction — a lethal-trifecta
+ * unit and a description overlap are real signals but are a capability PATTERN
+ * and a calibrated PROXY, and capping on either is how a gate earns the reputation
+ * that gets it switched off.
+ *
+ * Adding a row here is a deliberate act: it makes a finding grade-capping for
+ * every consumer, so it belongs only to a check whose false-positive rate is
+ * already known to be ~0.
+ */
+function confidentBreakages(r) {
+    const rows = [
+        {
+            n: r.hooks.filter((h) => h.status === "missing").length,
+            label: "hook script missing",
+        },
+        { n: r.hookEventIssues.length, label: "hook on an unknown event" },
+        { n: r.danglingRefs.length, label: "broken intra-plugin reference" },
+        {
+            n: r.agents.reduce((n, a) => n + a.toolIssues.length, 0),
+            label: "unavailable agent tool",
+        },
+        {
+            n: r.agents.reduce((n, a) => n + a.mcpToolIssues.length, 0),
+            label: "agent MCP tool whose server isn't declared",
+        },
+        { n: r.mcpIssues.length, label: "MCP server that can't start" },
+        {
+            n: r.mcpHookIssues.length,
+            label: "mcp_tool hook incomplete / undeclared server",
+        },
+        {
+            n: r.skillResourceIssues.length,
+            label: "skill bundled-resource ref that doesn't resolve",
+        },
+        {
+            n: r.skillFenceIssues.length,
+            label: "invisible skill (no opening `---` fence)",
+        },
+        {
+            n: r.frontmatterIssues.length,
+            label: "surface missing required frontmatter",
+        },
+    ];
+    return rows.filter((row) => row.n > 0);
+}
+/**
+ * Cap a health score at {@link CONFIDENT_BREAKAGE_CAP} when the report carries any
+ * {@link confidentBreakages} finding. Applied to the OVERALL and to the ring that
+ * owns the finding, so the headline and the breakdown cannot disagree.
+ */
+function applyBreakageCap(score, r) {
+    return confidentBreakages(r).length > 0
+        ? Math.min(score, exports.CONFIDENT_BREAKAGE_CAP)
+        : score;
+}
 /**
  * The lethal-trifecta exposure a report incurs — the ONE number the Safety ring
  * and the overall grade both read.
@@ -263,7 +349,10 @@ function scoreReport(r) {
         return { score: 0, issues: ["no loadable plugin surface"] };
     }
     const deductions = reportDeductions(r);
-    const { score } = computeIntegrityScore(deductions);
+    const { score: summed } = computeIntegrityScore(deductions);
+    // A confident breakage caps the score — a summed model dilutes one real
+    // breakage among clean siblings. See applyBreakageCap for the measurement.
+    const score = applyBreakageCap(summed, r);
     const issues = [];
     for (const d of deductions) {
         if (d.n === 0)

package/dist/spec-hooks.d.mts

@@ -0,0 +1,35 @@
+type ResolveContext = {
+    parentURL?: string;
+    conditions: string[];
+};
+type Resolved = {
+    url: string;
+    format?: string | null;
+    shortCircuit?: boolean;
+};
+type NextResolve = (specifier: string, context: ResolveContext) => Resolved | Promise<Resolved>;
+type LoadContext = {
+    format?: string | null;
+    conditions: string[];
+};
+type Loaded = {
+    format: string;
+    source?: string | ArrayBuffer;
+    shortCircuit?: boolean;
+};
+type NextLoad = (url: string, context: LoadContext) => Loaded | Promise<Loaded>;
+/**
+ * `./x.js` → `./x.ts` when the sibling exists.
+ *
+ * This is the TypeScript ESM convention (`tsc` under `nodenext` requires the
+ * `.js` extension in the source), which `tsx` implements and native Node does
+ * not. It is the ONE divergence that matters in practice: this repository's own
+ * dogfood specs import `src/core/spec.js`, a file that does not exist on disk.
+ * Attempted only AFTER normal resolution fails, so it can never shadow a real
+ * `.js` file.
+ */
+export declare function resolve(specifier: string, context: ResolveContext, nextResolve: NextResolve): Promise<Resolved>;
+/** Transpile `.ts`/`.mts` with the TypeScript this package already ships. */
+export declare function load(url: string, context: LoadContext, nextLoad: NextLoad): Promise<Loaded>;
+export {};
+//# sourceMappingURL=spec-hooks.d.mts.map

package/dist/spec-hooks.mjs

@@ -0,0 +1,68 @@
+/**
+ * Module customization hooks for the spec host — vigiles' OWN loader for `.ts`
+ * specs, replacing "whichever loader happens to be installed".
+ *
+ * Why vigiles owns this rather than shelling to `tsx`:
+ *
+ *   - **No install, no network.** `typescript` is already a runtime dependency
+ *     of this package (`dependencies`, and `core/compile-generator.ts` uses it),
+ *     so `ts.transpileModule` costs nothing extra. The bug that started this
+ *     work was a consuming repo without `tsx`, where `npx tsx` went to the
+ *     registry and every one of 50 specs blew a 15s budget.
+ *   - **One resolution contract.** Before this, a spec's module resolution
+ *     depended on the user's Node version and on which loader won — so a spec
+ *     could load locally and fail in CI under different rules. A tool that
+ *     audits other tools for that kind of quiet divergence should not have it.
+ *
+ * Scope is deliberately small and documented as such: `.ts`/`.mts` sources, the
+ * `./x.js` → `./x.ts` specifier rewrite, and bare specifiers. NOT tsconfig
+ * `paths`, JSX, or decorator configuration — specs are configuration modules,
+ * not applications.
+ */
+import { existsSync, readFileSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+import ts from "typescript";
+const TS_SOURCE = /\.m?ts$/;
+/**
+ * `./x.js` → `./x.ts` when the sibling exists.
+ *
+ * This is the TypeScript ESM convention (`tsc` under `nodenext` requires the
+ * `.js` extension in the source), which `tsx` implements and native Node does
+ * not. It is the ONE divergence that matters in practice: this repository's own
+ * dogfood specs import `src/core/spec.js`, a file that does not exist on disk.
+ * Attempted only AFTER normal resolution fails, so it can never shadow a real
+ * `.js` file.
+ */
+export async function resolve(specifier, context, nextResolve) {
+    try {
+        return await nextResolve(specifier, context);
+    }
+    catch (err) {
+        if (specifier.endsWith(".js") && context.parentURL) {
+            const candidate = new URL(specifier.slice(0, -3) + ".ts", context.parentURL);
+            if (existsSync(fileURLToPath(candidate))) {
+                return { url: candidate.href, format: "module", shortCircuit: true };
+            }
+        }
+        throw err;
+    }
+}
+/** Transpile `.ts`/`.mts` with the TypeScript this package already ships. */
+export async function load(url, context, nextLoad) {
+    if (!TS_SOURCE.test(new URL(url).pathname))
+        return nextLoad(url, context);
+    const fileName = fileURLToPath(url);
+    const { outputText } = ts.transpileModule(readFileSync(fileName, "utf-8"), {
+        fileName,
+        compilerOptions: {
+            module: ts.ModuleKind.ESNext,
+            target: ts.ScriptTarget.ES2022,
+            // Erasing types is the whole job; anything that changes SEMANTICS is not
+            // ours to decide for a spec.
+            verbatimModuleSyntax: false,
+            isolatedModules: true,
+        },
+    });
+    return { format: "module", source: outputText, shortCircuit: true };
+}
+//# sourceMappingURL=spec-hooks.mjs.map

package/dist/spec-host.d.mts

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=spec-host.d.mts.map

package/dist/spec-host.mjs

@@ -0,0 +1,79 @@
+/**
+ * The spec host — a child process that loads specs and streams results as NDJSON.
+ *
+ * ONE host per CLI command, not one per spec: Node startup and the TypeScript
+ * load are paid once, then each spec costs a transpile.
+ *
+ * 🔴 **Why a child process at all, when `import()` works in-process.** Because a
+ * module evaluation cannot be cancelled once started. `Promise.race` returns
+ * control to the caller but the evaluation keeps running and holds the event
+ * loop, so a spec that stalls at top level hangs `compile`, `test` and `audit`
+ * with no bound. A child can be killed. That is the entire argument, and it is
+ * why the in-process loader this replaced could not be repaired: it also had to
+ * answer "did the module body already run?" to know whether re-running was
+ * safe, and Node does not expose that bit — `ERR_MODULE_NOT_FOUND` and
+ * `SyntaxError` each occur both before and during evaluation.
+ *
+ * Protocol, one JSON object per line each way:
+ *   in   {"path":"<abs path to spec>"}
+ *   out  {"path":"…","phase":"start"}          — emitted BEFORE evaluation
+ *   out  {"path":"…","ok":true,"value":{…}}
+ *   out  {"path":"…","ok":false,"error":"…"}
+ *
+ * The `start` line is what makes a hang diagnosable: when the parent's deadline
+ * fires, the last `start` without a result NAMES the spec that stalled. Before
+ * this, a stalled load produced N identical failures and no culprit.
+ *
+ * Values cross as JSON, which is not a new constraint — the previous `npx tsx`
+ * path already did `JSON.stringify` in the child and `JSON.parse` in the parent,
+ * so every spec that has ever loaded survived this round trip. Spec types carry
+ * no functions; TypeScript is the authoring layer, the value is data.
+ */
+import { register } from "node:module";
+import { pathToFileURL } from "node:url";
+register(new URL("./spec-hooks.mjs", import.meta.url));
+function say(line) {
+    process.stdout.write(JSON.stringify(line) + "\n");
+}
+async function loadOne(path) {
+    say({ path, phase: "start" });
+    try {
+        const mod = (await import(pathToFileURL(path).href));
+        // CJS interop can nest the default one level deeper.
+        const raw = mod.default;
+        const value = raw && typeof raw === "object" && "default" in raw ? raw.default : raw;
+        if (value === undefined) {
+            say({ path, ok: false, error: "the spec has no default export." });
+            return;
+        }
+        say({ path, ok: true, value });
+    }
+    catch (err) {
+        say({
+            path,
+            ok: false,
+            error: err instanceof Error ? (err.stack ?? err.message) : String(err),
+        });
+    }
+}
+// Requests are serialised: a spec may depend on module state a previous one set
+// up, and interleaving would make a hang impossible to attribute.
+let queue = Promise.resolve();
+let buffered = "";
+process.stdin.setEncoding("utf-8");
+process.stdin.on("data", (chunk) => {
+    buffered += chunk;
+    let nl;
+    while ((nl = buffered.indexOf("\n")) >= 0) {
+        const line = buffered.slice(0, nl).trim();
+        buffered = buffered.slice(nl + 1);
+        if (!line)
+            continue;
+        const { path } = JSON.parse(line);
+        queue = queue.then(() => loadOne(path));
+    }
+});
+process.stdin.on("end", () => {
+    queue.then(() => process.exit(0));
+});
+//# sourceMappingURL=spec-host.mjs.map

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vigiles",
-  "version": "21.0.1",
+  "version": "22.0.0",
   "description": "Audit, test and measure the harness your AI agent runs on — grade your CLAUDE.md / AGENTS.md, skills, subagents and hooks, run them against a scripted model, and measure whether they actually fire.",
   "keywords": [
     "claude-code",

package/skills/test-harness/SKILL.md

@@ -49,83 +49,18 @@ not established, which is why the floor stays.
 If the unit and deterministic tiers can both answer it, **prefer unit**: it's
 faster and reaches events the deterministic mock can't drive.
 
-## Step 0.4 — Observing a run (what it CALLED, WROTE, and TOUCHED)
-
-The table above is keyed on the harness _surface_ under test. Half the real
-questions are keyed on the **observation** instead — "what did this skill
-actually do?" — and they have answers already. Reach for these before building
-anything; every one of them ships today.
-
-| The question you're actually asking                                     | Use                                                                                    |
-| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
-| Which tools did it call, and with what arguments?                       | `trace.toolCalls` · `tool` / `toolWith` checks · `parseToolCalls` (`vigiles`)          |
-| Did it call a tool it must not?                                         | `notTool(name)`                                                                        |
-| Did it call **only** tools from a known set?                            | `onlyTools([...])` — the white-list, symmetric to `assertWroteOnly`                    |
-| Did it stay inside the `allowed-tools` its own frontmatter declares?    | `skillContract(dir).surface` — builds that check FROM the declaration                  |
-| What files did the run write?                                           | `filesWritten` · `wrote(path)` / `didNotWrite(path)` · `r.file(path)`                  |
-| Did it write **only** where it was supposed to?                         | `assertWroteOnly([...])` / `assertNoWrite()` — needs `{ sandbox: "auto" }`             |
-| Run a tool call but **don't let it execute** — capture the args instead | the `interceptTools` option on `measure` / `runEval` (a `ToolIntercept[]`)             |
-| Did a subagent do it, and which one?                                    | `subagent(name, [...])` · `SubagentTrace`                                              |
-| Was it an MCP tool?                                                     | `mcp(server, toolName)`                                                                |
-| Assert the whole effect boundary deterministically                      | `assertChecks` + the checks above (see `examples/harness/effect-boundary.harness.mjs`) |
-
-`interceptTools` is the one worth knowing about, because it is not obvious it
-exists: it denies a tool its **real execution** via an auto-wired `PreToolUse`
-hook while still recording the call and its arguments into the trace. That is
-how you test a skill that would otherwise mutate a real external service — a
-calendar, an upload — without mocking anything yourself.
-
-**Verify a skill against its own declaration** with `skillContract` — it reads
-the `allowed-tools:` the skill already claims and hands back ready checks, so
-the claim is verified instead of restated:
-
-```ts
-import { skillContract, assertChecks } from "vigiles";
-
-const c = skillContract(".claude/skills/my-skill");
-assertChecks(trace, [c.activation, ...c.surface]);
-```
+## Step 0.4 — Observing a run, and what it costs
+
+Two questions have their own references — open the one you need, don't guess:
 
-Two of its states are **findings**, not clean bills, and their `surface` check
-fails rather than passing on nothing: `undeclared` (no `allowed-tools:` line, so
-the skill inherits _every_ tool) and `malformed` (frontmatter that isn't valid
-YAML, so a strict loader reads no contract at all — one unquoted `: ` does it).
-
-⚠️ **What is still NOT checked.** `onlyTools` compares tool _names_, so a narrow
-allowlist entry like `Bash(node scripts/x.mjs:*)` is satisfied by any `Bash` call
-at all. Scope inside a tool is unverified — say so rather than implying the
-assertion is total.
-
-## Step 0.5 — Set honest expectations (what's testable, and at what cost)
-
-Be explicit with the user about which bucket each surface falls into — never let
-"we'll test it" hide whether that's free, sub-priced, or needs a container. Every
-surface sorts into one of three buckets:
-
-- **A — Free & deterministic** (no model, runs in CI on every commit): a hook's
-  block/allow decision (`runHook`), a tool-contract / "did NOT call the forbidden
-  tool" check, structural facts (`vigiles audit`), and **record-replay** of any tool
-  a skill shells out to (record the real result once, replay it via a PATH stub).
-- **B — Model-gated, on your subscription** (real model, **no metered API**): does a
-  skill's description **fire** (`measureTriggerRate`, recall + precision) **and**
-  does its guidance actually **produce good output** (score it directly:
-  `measure({ checks: [judged(rubric)] })` + `assertRates` — the absolute oracle;
-  use a `runEval` A/B on-vs-off only when you need the _relative_ lift). This is
-  the half a **prose / guidance skill** lives in —
-  its worth is behavioral, so only a model can judge it. That is **not** "uncovered"
-  and **not** free: it's fully testable on the sub. State it that way.
-- **C — Needs a real service** (a real browser / DB / redis / a11y runtime): vigiles
-  **composes with a container** here; it does not fake real semantics. Name the
-  service and hand off — don't pretend a cheap tier substitutes for it.
-
-So a prose-skill library is roughly **~100% testable (some free, most on your sub),
-~0% needs-a-container** — not "poorly covered." An accessibility/browser plugin is
-the worst case, with a large bucket C. When you report coverage, give **two
-numbers**: "% testable at all (free + sub)" vs "% that needs a container", and say
-which surfaces are free vs sub-priced. The model-gated half is the **point** of the
-eval pillar (affordable on the sub), not a gap — and testing a prose skill's
-_behavior_ requires a real model for **everyone** (promptfoo, the SDKs, all of it);
-vigiles just does it on your subscription instead of metered API.
+- **"What did the run actually DO?"** — which tools it called, whether it stayed
+  inside its declared `allowed-tools`, what it wrote, how to record a call without
+  executing it → [`references/observing-a-run.md`](references/observing-a-run.md)
+- **"Is this free, sub-priced, or does it need a container?"** — the three buckets,
+  and what to tell the user after a paid run →
+  [`references/cost-and-expectations.md`](references/cost-and-expectations.md)
+
+Never say "we'll test it" without settling the second one first.
 
 ## Step 1 — Ensure vigiles is installed
 
@@ -154,142 +89,12 @@ Pick one concrete thing to pin down — a specific `PreToolUse` hook, a specific
 
 ## Step 3 — Write the test for the chosen tier
 
-**Unit (`runHook`)** — hand a hook a synthesized event, assert the decision:
-
-```ts
-import { runHook, assertHookBlocked } from "vigiles";
-
-const r = runHook(hookCommand, {
-  hook_event_name: "PreToolUse",
-  tool_name: "Bash",
-  tool_input: { command: "git commit --no-verify" },
-});
-assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"
-```
-
-Testing a hook you didn't write (a vendored third-party script)? Mark it
-`{ trusted: false }` and it runs confined under bubblewrap by default (read-only
-host, cleared env, no network egress). Add `{ recordEgress: true }` to also
-**record** what it tries to reach — `r.egress` plus `assertNoEgress(r)` /
-`assertEgressOnly(r, [...])` — the supply-chain check for "what does this skill
-phone home to / install from?". When the hook's setup needs a _real_ install,
-`{ egress: { allow: ["registry.npmjs.org"] } }` lets it reach only that
-allowlist (a packet-layer `nft` wall, so a raw socket off-list is dropped too) →
-`r.egress` (allowed hosts) + `r.egressDropped`. Be precise about the boundaries:
-see
-[`docs/sandboxing.md`](../../docs/sandboxing.md) (it blocks destruction and
-egress, but does NOT isolate reads of host files, and only under bwrap).
-
-**Deterministic (`runHarnessTest`)** — load the real plugin, drive a scripted
-mock model, assert the hook fired (or the context landed):
-
-```ts
-import {
-  runHarnessTest,
-  assertHookFired,
-  assertRequestContains,
-} from "vigiles";
-// `scriptModel` is the Claude-Code TRANSPORT, deliberately not re-exported from
-// the harness-agnostic root surface — import it from the harness package:
-import { scriptModel } from "vigiles/claude-code";
-
-const r = await runHarnessTest({
-  pluginDir: "./", // or { settings: { hooks: {...} } }
-  transcript: true,
-  model: scriptModel([{ text: "ok" }]),
-});
-assertHookFired(r, "SessionStart");
-assertRequestContains(r, "expected injected text"); // did it actually land?
-```
-
-**Eval — absolute (`paid_measure` + `paid_judged`)** — testing _one_ skill, the usual case:
-score its output directly against a rubric. No on/off baseline — this is the
-"is it any good?" oracle (what promptfoo/DeepEval lead with), and the right
-default when there's nothing to compare against:
-
-An eval file **describes** its eval — it must never run one at the top level,
-because importing such a file spends real money. Write `<name>.eval.mjs`:
-
-```ts
-import { defineEval, skill, assertRates } from "vigiles";
-import { paid_judged } from "vigiles/eval"; // a Check whose default judge bills
-
-export default defineEval({
-  measure: {
-    pluginDir: "./",
-    task: "…a task the skill should handle…",
-    checks: [
-      skill("my-plugin:my-skill"), // it fired
-      paid_judged("the answer correctly does X and avoids Y"), // …and the output is good
-    ],
-    trials: 6,
-  },
-  assert: (report) => assertRates(report, { min: 0.8 }), // each check ≥ 80% of trials
-});
-```
-
-Run it with `npx vigiles eval <file>` — never `node <file>`, which refuses.
-
-**Eval — relative (`paid_runEval` + `assertSignificant`)** — when the question is
-_lift over no-skill_ (regression, or proving a change isn't noise): A/B the
-change on vs off and gate on significance, not eyeballing:
-
-```ts
-import { defineEval, assertSignificant } from "vigiles";
-
-export default defineEval({
-  runEval: {
-    arms: { off: {}, on: { pluginDir: "./" } },
-    task: "…a task the harness change should affect…",
-    measure: (ctx) => ({ ok: /* a bare predicate over the trace */ true }),
-    trials: 6,
-    cache: "readwrite",
-  },
-  assert: (report) =>
-    assertSignificant(report, { baseline: "off", arm: "on", metric: "ok" }),
-});
-```
-
-### Never hand-roll the runner — it silently eats stderr
+Per-tier skeletons, and the one mistake that silently swallows failures (a
+hand-rolled runner eats stderr) →
+[`references/writing-tests.md`](references/writing-tests.md)
 
-Do **not** reach for `execFileSync` / `spawnSync` to drive the thing under test.
-The failure is quiet and repeats: `execFileSync` returns **stdout only** on
-success, while advisory output — including vigiles's own compiled-hook
-`notice()` — is written to **stderr**. A hand-rolled runner therefore reports a
-perfectly healthy react hook as **dead**, and an assertion about a warning can
-never pass. (Observed three times in one repo, twice after the first fix.)
-
-Every vigiles result already carries **both streams**, so the bug is
-unrepresentable:
-
-| Runner           | Result              | Carries                                             |
-| ---------------- | ------------------- | --------------------------------------------------- |
-| `runScript`      | `ScriptRunResult`   | `exitCode`, `stdout`, `stderr`, `filesWritten?`     |
-| `runHook`        | `HookRunResult`     | all of the above, **plus** `blocked` / `decision`   |
-| `runHarnessTest` | `HarnessTestResult` | `exitCode`, `stdout`, `stderr`, `cwd` + the `Trace` |
-
-**Testing a plain helper script** (a bash/node/python program that isn't a hook)?
-Use **`runScript`** — it runs any command and reports what it did:
-
-```ts
-import { runScript } from "vigiles";
-
-const r = runScript("bash scripts/check-links.sh", { cwd: repoDir });
-assert.equal(r.exitCode, 0);
-assert.match(r.stderr, /0 broken links/); // advisory output lives HERE
-```
-
-`runHook` is exactly `runScript` plus the hook protocol (event → stdin, exit code
-→ allow/deny). Pick by the question you're asking: a **hook** has a _decision_, a
-**script** has _effects_. That's why `ScriptRunResult` has no `decision` field —
-a field that is always meaningless is worse than no field.
-
-⚠️ **Asserting what a script wrote requires confinement.** `filesWritten` is
-recorded by diffing the work dir, which only a confined run does — so it is
-`undefined` after a plain run. That is deliberately _not_ the same as `[]`
-("recorded, wrote nothing"): `assertNoWrite` / `assertWroteOnly` **throw** on an
-unrecorded result rather than pass having inspected nothing. Pass
-`{ sandbox: "auto" }` (Linux + bubblewrap) to actually record writes.
+Read it before writing the file — the skeleton differs per tier, and the runner
+warning has cost real debugging time.
 
 ## Step 4 — Run it
 
@@ -309,23 +114,6 @@ job that asserts the capability is present, run **`vigiles test --no-skip`** so
 skipped tier fails — a green-with-skips is untested surface. Keep unit +
 deterministic tests in CI (free); run evals locally or on a schedule with auth.
 
-### After a real-model run: TELL THE USER WHAT IT SPENT
-
-Whenever you run a real-model eval (`runEval` / `measureArms` / `measureTriggerRate`
-/ `measure`), **surface the spend to the user in your reply** — don't let a paid run
-be silent. `runEval` prints a cost block to stderr and every report carries `usage`
-(`report.arms[*].usage`: `totalCostUsd` + token counts). Relay, in plain words:
-
-- **tokens spent** and the **API-equivalent `$`** (`total_cost_usd` — what it _would_
-  cost at metered API rates);
-- **how it was billed** — "on your Claude subscription (**$0 metered**)" if you're
-  logged in, or a **⚠ warning** if `ANTHROPIC_API_KEY` is set (that run was billed
-  **per token** — tell them to unset it and `claude login` to run free).
-
-We do **not** show "% of your subscription" — Anthropic doesn't expose a plan's
-quota, so any percentage would be invented. Tokens + API-equivalent `$` + the
-billed-to line is the honest, complete picture. Keep the user's cost visible, always.
-
 ## CI — don't hand-write the steps
 
 These tiers belong in CI, and there is a published Action for it. Run `vigiles init`: it

package/skills/test-harness/references/cost-and-expectations.md

@@ -0,0 +1,53 @@
+# Honest expectations and cost
+
+Read this before telling a user "we'll test it" — it fixes which bucket a surface
+falls into (free / on your subscription / needs a container) and what to report
+after a paid run.
+
+## Set honest expectations (what's testable, and at what cost)
+
+Be explicit with the user about which bucket each surface falls into — never let
+"we'll test it" hide whether that's free, sub-priced, or needs a container. Every
+surface sorts into one of three buckets:
+
+- **A — Free & deterministic** (no model, runs in CI on every commit): a hook's
+  block/allow decision (`runHook`), a tool-contract / "did NOT call the forbidden
+  tool" check, structural facts (`vigiles audit`), and **record-replay** of any tool
+  a skill shells out to (record the real result once, replay it via a PATH stub).
+- **B — Model-gated, on your subscription** (real model, **no metered API**): does a
+  skill's description **fire** (`measureTriggerRate`, recall + precision) **and**
+  does its guidance actually **produce good output** (score it directly:
+  `measure({ checks: [judged(rubric)] })` + `assertRates` — the absolute oracle;
+  use a `runEval` A/B on-vs-off only when you need the _relative_ lift). This is
+  the half a **prose / guidance skill** lives in —
+  its worth is behavioral, so only a model can judge it. That is **not** "uncovered"
+  and **not** free: it's fully testable on the sub. State it that way.
+- **C — Needs a real service** (a real browser / DB / redis / a11y runtime): vigiles
+  **composes with a container** here; it does not fake real semantics. Name the
+  service and hand off — don't pretend a cheap tier substitutes for it.
+
+So a prose-skill library is roughly **~100% testable (some free, most on your sub),
+~0% needs-a-container** — not "poorly covered." An accessibility/browser plugin is
+the worst case, with a large bucket C. When you report coverage, give **two
+numbers**: "% testable at all (free + sub)" vs "% that needs a container", and say
+which surfaces are free vs sub-priced. The model-gated half is the **point** of the
+eval pillar (affordable on the sub), not a gap — and testing a prose skill's
+_behavior_ requires a real model for **everyone** (promptfoo, the SDKs, all of it);
+vigiles just does it on your subscription instead of metered API.
+
+### After a real-model run: TELL THE USER WHAT IT SPENT
+
+Whenever you run a real-model eval (`runEval` / `measureArms` / `measureTriggerRate`
+/ `measure`), **surface the spend to the user in your reply** — don't let a paid run
+be silent. `runEval` prints a cost block to stderr and every report carries `usage`
+(`report.arms[*].usage`: `totalCostUsd` + token counts). Relay, in plain words:
+
+- **tokens spent** and the **API-equivalent `$`** (`total_cost_usd` — what it _would_
+  cost at metered API rates);
+- **how it was billed** — "on your Claude subscription (**$0 metered**)" if you're
+  logged in, or a **⚠ warning** if `ANTHROPIC_API_KEY` is set (that run was billed
+  **per token** — tell them to unset it and `claude login` to run free).
+
+We do **not** show "% of your subscription" — Anthropic doesn't expose a plan's
+quota, so any percentage would be invented. Tokens + API-equivalent `$` + the
+billed-to line is the honest, complete picture. Keep the user's cost visible, always.

package/skills/test-harness/references/observing-a-run.md

@@ -0,0 +1,51 @@
+# Observing a run — what it CALLED, WROTE, and TOUCHED
+
+Read this when the question is about **what a run did**, not about which tier to
+pick. Every predicate here ships today.
+
+## Observing a run (what it CALLED, WROTE, and TOUCHED)
+
+The table above is keyed on the harness _surface_ under test. Half the real
+questions are keyed on the **observation** instead — "what did this skill
+actually do?" — and they have answers already. Reach for these before building
+anything; every one of them ships today.
+
+| The question you're actually asking                                     | Use                                                                                    |
+| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
+| Which tools did it call, and with what arguments?                       | `trace.toolCalls` · `tool` / `toolWith` checks · `parseToolCalls` (`vigiles`)          |
+| Did it call a tool it must not?                                         | `notTool(name)`                                                                        |
+| Did it call **only** tools from a known set?                            | `onlyTools([...])` — the white-list, symmetric to `assertWroteOnly`                    |
+| Did it stay inside the `allowed-tools` its own frontmatter declares?    | `skillContract(dir).surface` — builds that check FROM the declaration                  |
+| What files did the run write?                                           | `filesWritten` · `wrote(path)` / `didNotWrite(path)` · `r.file(path)`                  |
+| Did it write **only** where it was supposed to?                         | `assertWroteOnly([...])` / `assertNoWrite()` — needs `{ sandbox: "auto" }`             |
+| Run a tool call but **don't let it execute** — capture the args instead | the `interceptTools` option on `measure` / `runEval` (a `ToolIntercept[]`)             |
+| Did a subagent do it, and which one?                                    | `subagent(name, [...])` · `SubagentTrace`                                              |
+| Was it an MCP tool?                                                     | `mcp(server, toolName)`                                                                |
+| Assert the whole effect boundary deterministically                      | `assertChecks` + the checks above (see `examples/harness/effect-boundary.harness.mjs`) |
+
+`interceptTools` is the one worth knowing about, because it is not obvious it
+exists: it denies a tool its **real execution** via an auto-wired `PreToolUse`
+hook while still recording the call and its arguments into the trace. That is
+how you test a skill that would otherwise mutate a real external service — a
+calendar, an upload — without mocking anything yourself.
+
+**Verify a skill against its own declaration** with `skillContract` — it reads
+the `allowed-tools:` the skill already claims and hands back ready checks, so
+the claim is verified instead of restated:
+
+```ts
+import { skillContract, assertChecks } from "vigiles";
+
+const c = skillContract(".claude/skills/my-skill");
+assertChecks(trace, [c.activation, ...c.surface]);
+```
+
+Two of its states are **findings**, not clean bills, and their `surface` check
+fails rather than passing on nothing: `undeclared` (no `allowed-tools:` line, so
+the skill inherits _every_ tool) and `malformed` (frontmatter that isn't valid
+YAML, so a strict loader reads no contract at all — one unquoted `: ` does it).
+
+⚠️ **What is still NOT checked.** `onlyTools` compares tool _names_, so a narrow
+allowlist entry like `Bash(node scripts/x.mjs:*)` is satisfied by any `Bash` call
+at all. Scope inside a tool is unverified — say so rather than implying the
+assertion is total.

package/skills/test-harness/references/writing-tests.md

@@ -0,0 +1,143 @@
+# Writing the test for the chosen tier
+
+Read this once the tier is picked. Covers the per-tier skeleton and the one
+mistake that silently swallows failures.
+
+## Write the test for the chosen tier
+
+**Unit (`runHook`)** — hand a hook a synthesized event, assert the decision:
+
+```ts
+import { runHook, assertHookBlocked } from "vigiles";
+
+const r = runHook(hookCommand, {
+  hook_event_name: "PreToolUse",
+  tool_name: "Bash",
+  tool_input: { command: "git commit --no-verify" },
+});
+assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"
+```
+
+Testing a hook you didn't write (a vendored third-party script)? Mark it
+`{ trusted: false }` and it runs confined under bubblewrap by default (read-only
+host, cleared env, no network egress). Add `{ recordEgress: true }` to also
+**record** what it tries to reach — `r.egress` plus `assertNoEgress(r)` /
+`assertEgressOnly(r, [...])` — the supply-chain check for "what does this skill
+phone home to / install from?". When the hook's setup needs a _real_ install,
+`{ egress: { allow: ["registry.npmjs.org"] } }` lets it reach only that
+allowlist (a packet-layer `nft` wall, so a raw socket off-list is dropped too) →
+`r.egress` (allowed hosts) + `r.egressDropped`. Be precise about the boundaries:
+see
+[`docs/sandboxing.md`](../../docs/sandboxing.md) (it blocks destruction and
+egress, but does NOT isolate reads of host files, and only under bwrap).
+
+**Deterministic (`runHarnessTest`)** — load the real plugin, drive a scripted
+mock model, assert the hook fired (or the context landed):
+
+```ts
+import {
+  runHarnessTest,
+  assertHookFired,
+  assertRequestContains,
+} from "vigiles";
+// `scriptModel` is the Claude-Code TRANSPORT, deliberately not re-exported from
+// the harness-agnostic root surface — import it from the harness package:
+import { scriptModel } from "vigiles/claude-code";
+
+const r = await runHarnessTest({
+  pluginDir: "./", // or { settings: { hooks: {...} } }
+  transcript: true,
+  model: scriptModel([{ text: "ok" }]),
+});
+assertHookFired(r, "SessionStart");
+assertRequestContains(r, "expected injected text"); // did it actually land?
+```
+
+**Eval — absolute (`paid_measure` + `paid_judged`)** — testing _one_ skill, the usual case:
+score its output directly against a rubric. No on/off baseline — this is the
+"is it any good?" oracle (what promptfoo/DeepEval lead with), and the right
+default when there's nothing to compare against:
+
+An eval file **describes** its eval — it must never run one at the top level,
+because importing such a file spends real money. Write `<name>.eval.mjs`:
+
+```ts
+import { defineEval, skill, assertRates } from "vigiles";
+import { paid_judged } from "vigiles/eval"; // a Check whose default judge bills
+
+export default defineEval({
+  measure: {
+    pluginDir: "./",
+    task: "…a task the skill should handle…",
+    checks: [
+      skill("my-plugin:my-skill"), // it fired
+      paid_judged("the answer correctly does X and avoids Y"), // …and the output is good
+    ],
+    trials: 6,
+  },
+  assert: (report) => assertRates(report, { min: 0.8 }), // each check ≥ 80% of trials
+});
+```
+
+Run it with `npx vigiles eval <file>` — never `node <file>`, which refuses.
+
+**Eval — relative (`paid_runEval` + `assertSignificant`)** — when the question is
+_lift over no-skill_ (regression, or proving a change isn't noise): A/B the
+change on vs off and gate on significance, not eyeballing:
+
+```ts
+import { defineEval, assertSignificant } from "vigiles";
+
+export default defineEval({
+  runEval: {
+    arms: { off: {}, on: { pluginDir: "./" } },
+    task: "…a task the harness change should affect…",
+    measure: (ctx) => ({ ok: /* a bare predicate over the trace */ true }),
+    trials: 6,
+    cache: "readwrite",
+  },
+  assert: (report) =>
+    assertSignificant(report, { baseline: "off", arm: "on", metric: "ok" }),
+});
+```
+
+### Never hand-roll the runner — it silently eats stderr
+
+Do **not** reach for `execFileSync` / `spawnSync` to drive the thing under test.
+The failure is quiet and repeats: `execFileSync` returns **stdout only** on
+success, while advisory output — including vigiles's own compiled-hook
+`notice()` — is written to **stderr**. A hand-rolled runner therefore reports a
+perfectly healthy react hook as **dead**, and an assertion about a warning can
+never pass. (Observed three times in one repo, twice after the first fix.)
+
+Every vigiles result already carries **both streams**, so the bug is
+unrepresentable:
+
+| Runner           | Result              | Carries                                             |
+| ---------------- | ------------------- | --------------------------------------------------- |
+| `runScript`      | `ScriptRunResult`   | `exitCode`, `stdout`, `stderr`, `filesWritten?`     |
+| `runHook`        | `HookRunResult`     | all of the above, **plus** `blocked` / `decision`   |
+| `runHarnessTest` | `HarnessTestResult` | `exitCode`, `stdout`, `stderr`, `cwd` + the `Trace` |
+
+**Testing a plain helper script** (a bash/node/python program that isn't a hook)?
+Use **`runScript`** — it runs any command and reports what it did:
+
+```ts
+import { runScript } from "vigiles";
+
+const r = runScript("bash scripts/check-links.sh", { cwd: repoDir });
+assert.equal(r.exitCode, 0);
+assert.match(r.stderr, /0 broken links/); // advisory output lives HERE
+```
+
+`runHook` is exactly `runScript` plus the hook protocol (event → stdin, exit code
+→ allow/deny). Pick by the question you're asking: a **hook** has a _decision_, a
+**script** has _effects_. That's why `ScriptRunResult` has no `decision` field —
+a field that is always meaningless is worse than no field.
+
+⚠️ **Asserting what a script wrote requires confinement.** `filesWritten` is
+recorded by diffing the work dir, which only a confined run does — so it is
+`undefined` after a plain run. That is deliberately _not_ the same as `[]`
+("recorded, wrote nothing"): `assertNoWrite` / `assertWroteOnly` **throw** on an
+unrecorded result rather than pass having inspected nothing. Pass
+`{ sandbox: "auto" }` (Linux + bubblewrap) to actually record writes.