vigiles 21.0.1 → 21.0.2

package/README.md

@@ -175,6 +175,8 @@ Every path, script, symbol, and rule verified against reality — plus tool cont
 A hook that blocks nothing, a skill that hijacks unrelated prompts, context that never reaches the model — each passes a naive "did it run?" check. That gap is **false confidence**: a guard that looks like it works and silently doesn't. vigiles tests the real thing — hooks block, skills fire, subagents finish what they promised, a stray `git push` is caught before it happens. It drives a scripted stand-in for the model, not a live call, so it needs no key and runs on every commit.
 **[How testing works →](docs/harness-testing.md)**
 
+**Nothing you scan leaves your machine.** `lint`, `audit` and the deterministic test tiers make no network call at all — no telemetry, no analytics, no HTTP client in the package. Evals drive your own `claude` CLI on your own subscription, so no third party is introduced. **[What is and isn't transmitted →](docs/safety.md#does-vigiles-send-my-code-anywhere)**
+
 ### 📊 Eval — the only way to put a real number on cost
 
 _"Caveman Mode cuts 65% of your tokens." Says who?_ vigiles A/Bs the claim on real coding tasks and hands you three numbers: the **token bill**, whether it hit its **target**, and whether your code still **works**.

package/dist/audit-score.js

@@ -198,9 +198,24 @@ function structure(r) {
             `${String(noContract)} agent(s) inherit all tools (no contract) (advisory)`,
         ]
         : [];
+    // A confident breakage caps this ring too, so the breakdown can't render a
+    // healthy `●` over a dead surface — the defect this fixes was measured as
+    // "Structure 92 ●" while an agent named a never-available tool. Only the
+    // rows that mean a surface is DEAD count; the typo'd `disallowedTools` and
+    // invalid model/color rows are footguns, not breakage, so they stay out.
+    const breakage = deadTools +
+        deadMcpTools +
+        r.hookEventIssues.length +
+        r.mcpIssues.length +
+        r.mcpHookIssues.length +
+        r.frontmatterIssues.length +
+        r.pluginLayoutIssues.length +
+        r.skillFenceIssues.length +
+        r.hookBlockFindings.length +
+        r.hookMatcherFindings.length;
     return {
         key: "Structure",
-        score,
+        score: breakage > 0 ? Math.min(score, score_core_js_1.CONFIDENT_BREAKAGE_CAP) : score,
         weight: 1,
         findings: [...findings, ...advisory],
     };
@@ -248,15 +263,39 @@ function safety(r) {
     // one at a time would bury the subagent findings under a list as long as the
     // skill corpus. Same reasoning as the report section; see `trifectaLines`.
     const unfenced = r.trifectaFindings.filter((f) => f.kind === "skill" && f.finding.fence === "none");
+    // An INEFFECTIVE fence (`disallowed-tools:` that closes no leg) keeps its own
+    // line while there are FEW of them — that is the documented intent in
+    // core/lethal-trifecta.ts: it is a genuine mistake, the author believed they had
+    // fenced, so naming the skill is what a reader acts on.
+    //
+    // MEASURED 2026-08-28 that the premise behind that shape — "Rare" — does not
+    // hold: a fixture where every skill carried a naive `disallowed-tools: WebFetch`
+    // put ALL of them in this state, and the ring printed the same ~450-character
+    // paragraph ten times, ~4,500 characters into the terminal report. Past the
+    // threshold it stops being N facts about N skills and becomes one fact about the
+    // harness — exactly the reasoning the `fence: "none"` aggregate already uses.
+    // Names are kept as detail, so nothing is lost, only repetition.
+    const ineffective = r.trifectaFindings.filter((f) => f.finding.severity === "advisory" &&
+        !unfenced.includes(f) &&
+        f.kind === "skill" &&
+        f.finding.fence === "ineffective");
+    const collapseIneffective = ineffective.length > MAX_NAMED_INEFFECTIVE_FENCES;
     for (const f of r.trifectaFindings) {
         if (f.finding.severity !== "advisory")
             continue;
         if (unfenced.includes(f))
             continue;
+        if (collapseIneffective && ineffective.includes(f))
+            continue;
         findings.push(f.kind === "skill"
             ? `${f.name}: ${f.finding.message}`
             : `${f.name} inherits all tools — the "lethal trifecta" (reads data, reaches the web, runs commands) plus every other capability, so a prompt injection could exfiltrate secrets`);
     }
+    if (collapseIneffective) {
+        findings.push(`${String(ineffective.length)} skill(s) declare a \`disallowed-tools:\` that closes no lethal-trifecta leg — a leg is closed only when EVERY built-in supplying it is denied: ` +
+            `${ineffective.map((f) => f.name).join(", ")}. ` +
+            `Name every supplier of the leg you mean to close — private-data read = Read, Grep, Glob, Bash; untrusted intake = WebFetch, WebSearch, Bash; exfiltration = WebFetch, WebSearch, Bash.`);
+    }
     if (unfenced.length > 0) {
         findings.push(`${String(unfenced.length)} skill(s) declare no \`disallowed-tools:\` fence, so each inherits every tool the session grants — reads data, reaches the web, runs commands. \`allowed-tools:\` pre-approves, it does not restrict, so narrowing it does not reduce this; one \`disallowed-tools:\` line per skill drops a leg.`);
     }
@@ -456,9 +495,19 @@ function auditScore(report, opts = {}) {
     // of the rings — averaging would let a real problem in one category be diluted
     // by clean siblings. The rings above stay a diagnostic breakdown; Tested and
     // Evaluated (both advisory) are never summed in (neither drags the grade).
-    const { score: overall } = (0, score_core_js_1.computeIntegrityScore)((0, score_core_js_1.reportDeductions)(report));
+    const { score: summed } = (0, score_core_js_1.computeIntegrityScore)((0, score_core_js_1.reportDeductions)(report));
+    // A confident breakage caps the headline too, so it can never read `A` while a
+    // surface is definitively dead (score-core.ts::applyBreakageCap).
+    const overall = (0, score_core_js_1.applyBreakageCap)(summed, report);
     return { overall, grade: (0, score_core_js_1.gradeFor)(overall), categories, empty: false };
 }
+/**
+ * How many INEFFECTIVE `disallowed-tools:` fences are named one at a time before
+ * the Safety ring collapses them into a single line. Past this it is one fact
+ * about the harness, not N facts about N skills — see the measurement in
+ * {@link safety}.
+ */
+const MAX_NAMED_INEFFECTIVE_FENCES = 3;
 // A 22-cell bar gauge ("ring" in the terminal; the real rings are the HTML).
 const BAR_CELLS = 22;
 /** A glyph that signals the band at a glance (green/amber/red, no ANSI needed).

package/dist/core/description-overlap.js

@@ -8,7 +8,10 @@ exports.findDescriptionOverlaps = findDescriptionOverlaps;
  * selector, so the wrong one fires (a precision collision). This catches a
  * `--trigger`-class problem with NO model, reusing the NCD engine in proofs.ts
  * (the same one `findSimilarRules` uses) — the bridge between the deterministic
- * and behavioral columns, and a check no other plugin linter has.
+ * and behavioral columns. The CHECK is not unique — cisco-ai-defense/skill-scanner
+ * ships `--check-overlap` under a security framing (skill impersonation). What is
+ * ours: the cutoff calibrated against a real corpus, and the precision-collision
+ * framing (verified 2026-08-28).
  *
  * Calibrated HIGH-PRECISION against the mid-2026 sweep: across 4678 within-plugin
  * skill-description pairs, the MOST-similar legitimately-distinct pair

package/dist/score-core.d.ts

@@ -57,6 +57,54 @@ export declare const W_TRIFECTA_MAX = 30;
 export declare const TRIFECTA_LABEL = "unit(s) can read data, reach the web, and run commands \u2014 the \"lethal trifecta\", so a prompt injection could exfiltrate secrets";
 /** Map a 0–100 structural-health score to its letter grade (A ≥90 … F <60). */
 export declare function gradeFor(score: number): PluginScore["grade"];
+/**
+ * The health score a report may NOT exceed while it carries a CONFIDENT BREAKAGE
+ * — one point below the healthy band, so a definitively-broken harness can never
+ * render as `●` / grade `A`.
+ *
+ * WHY a cap and not a heavier weight: the score is a SUM of deductions, so one
+ * real breakage among many clean surfaces is diluted by its siblings. MEASURED
+ * 2026-08-28 on a fixture (10 distinct-description skills + one agent naming a
+ * never-available tool): Structure scored **92 with a `●` green dot** while
+ * carrying a dead tool contract, and the grade was driven instead by the
+ * lethal-trifecta pattern that {@link W_TRIFECTA} deliberately calls a ding and
+ * not a fail. A weight big enough to fix that would cry wolf on a large clean
+ * plugin; a cap fixes the reading without touching the arithmetic.
+ *
+ * The same defect, with higher stakes, is documented in an external 517-skill
+ * catalog run of a different scanner: its weighted aggregate returned MEDIUM for
+ * a skill carrying THREE HIGH findings, so gating on the aggregate would have
+ * admitted exactly the skill the gate existed to stop. That team's conclusion —
+ * judge by the worst finding, not the aggregate — is what this encodes.
+ */
+export declare const CONFIDENT_BREAKAGE_CAP = 89;
+/**
+ * The findings that CAP the score — each one means a surface is definitively
+ * dead, not merely risky: it will silently not run, not resolve, or not register.
+ *
+ * ENUMERATED on purpose, never derived from "severity". Two properties decide
+ * membership, and both must hold: the finding is DECIDABLE from the artifact plus
+ * the world (the `structural-closed` / `external-decidable` buckets of the
+ * lint-rule-calibration rule), and its consequence is BREAKAGE rather than
+ * exposure. So the heuristic rings stay out by construction — a lethal-trifecta
+ * unit and a description overlap are real signals but are a capability PATTERN
+ * and a calibrated PROXY, and capping on either is how a gate earns the reputation
+ * that gets it switched off.
+ *
+ * Adding a row here is a deliberate act: it makes a finding grade-capping for
+ * every consumer, so it belongs only to a check whose false-positive rate is
+ * already known to be ~0.
+ */
+export declare function confidentBreakages(r: ScanReport): {
+    readonly n: number;
+    readonly label: string;
+}[];
+/**
+ * Cap a health score at {@link CONFIDENT_BREAKAGE_CAP} when the report carries any
+ * {@link confidentBreakages} finding. Applied to the OVERALL and to the ring that
+ * owns the finding, so the headline and the breakdown cannot disagree.
+ */
+export declare function applyBreakageCap(score: number, r: ScanReport): number;
 /** One deduction: a count, its per-item weight, and the label if non-zero. */
 export interface Deduction {
     readonly n: number;

package/dist/score-core.js

@@ -1,7 +1,9 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.TRIFECTA_LABEL = exports.W_TRIFECTA_MAX = exports.W_TRIFECTA = exports.W_NO_CONTRACT = exports.W_OVERLAP = exports.W_DANGLING_REF = exports.W_NO_DESCRIPTION = exports.W_MISSING_HOOK = void 0;
+exports.CONFIDENT_BREAKAGE_CAP = exports.TRIFECTA_LABEL = exports.W_TRIFECTA_MAX = exports.W_TRIFECTA = exports.W_NO_CONTRACT = exports.W_OVERLAP = exports.W_DANGLING_REF = exports.W_NO_DESCRIPTION = exports.W_MISSING_HOOK = void 0;
 exports.gradeFor = gradeFor;
+exports.confidentBreakages = confidentBreakages;
+exports.applyBreakageCap = applyBreakageCap;
 exports.trifectaExposure = trifectaExposure;
 exports.reportDeductions = reportDeductions;
 exports.isEmptyMachine = isEmptyMachine;
@@ -52,6 +54,90 @@ function gradeFor(score) {
         return "D";
     return "F";
 }
+/**
+ * The health score a report may NOT exceed while it carries a CONFIDENT BREAKAGE
+ * — one point below the healthy band, so a definitively-broken harness can never
+ * render as `●` / grade `A`.
+ *
+ * WHY a cap and not a heavier weight: the score is a SUM of deductions, so one
+ * real breakage among many clean surfaces is diluted by its siblings. MEASURED
+ * 2026-08-28 on a fixture (10 distinct-description skills + one agent naming a
+ * never-available tool): Structure scored **92 with a `●` green dot** while
+ * carrying a dead tool contract, and the grade was driven instead by the
+ * lethal-trifecta pattern that {@link W_TRIFECTA} deliberately calls a ding and
+ * not a fail. A weight big enough to fix that would cry wolf on a large clean
+ * plugin; a cap fixes the reading without touching the arithmetic.
+ *
+ * The same defect, with higher stakes, is documented in an external 517-skill
+ * catalog run of a different scanner: its weighted aggregate returned MEDIUM for
+ * a skill carrying THREE HIGH findings, so gating on the aggregate would have
+ * admitted exactly the skill the gate existed to stop. That team's conclusion —
+ * judge by the worst finding, not the aggregate — is what this encodes.
+ */
+exports.CONFIDENT_BREAKAGE_CAP = 89;
+/**
+ * The findings that CAP the score — each one means a surface is definitively
+ * dead, not merely risky: it will silently not run, not resolve, or not register.
+ *
+ * ENUMERATED on purpose, never derived from "severity". Two properties decide
+ * membership, and both must hold: the finding is DECIDABLE from the artifact plus
+ * the world (the `structural-closed` / `external-decidable` buckets of the
+ * lint-rule-calibration rule), and its consequence is BREAKAGE rather than
+ * exposure. So the heuristic rings stay out by construction — a lethal-trifecta
+ * unit and a description overlap are real signals but are a capability PATTERN
+ * and a calibrated PROXY, and capping on either is how a gate earns the reputation
+ * that gets it switched off.
+ *
+ * Adding a row here is a deliberate act: it makes a finding grade-capping for
+ * every consumer, so it belongs only to a check whose false-positive rate is
+ * already known to be ~0.
+ */
+function confidentBreakages(r) {
+    const rows = [
+        {
+            n: r.hooks.filter((h) => h.status === "missing").length,
+            label: "hook script missing",
+        },
+        { n: r.hookEventIssues.length, label: "hook on an unknown event" },
+        { n: r.danglingRefs.length, label: "broken intra-plugin reference" },
+        {
+            n: r.agents.reduce((n, a) => n + a.toolIssues.length, 0),
+            label: "unavailable agent tool",
+        },
+        {
+            n: r.agents.reduce((n, a) => n + a.mcpToolIssues.length, 0),
+            label: "agent MCP tool whose server isn't declared",
+        },
+        { n: r.mcpIssues.length, label: "MCP server that can't start" },
+        {
+            n: r.mcpHookIssues.length,
+            label: "mcp_tool hook incomplete / undeclared server",
+        },
+        {
+            n: r.skillResourceIssues.length,
+            label: "skill bundled-resource ref that doesn't resolve",
+        },
+        {
+            n: r.skillFenceIssues.length,
+            label: "invisible skill (no opening `---` fence)",
+        },
+        {
+            n: r.frontmatterIssues.length,
+            label: "surface missing required frontmatter",
+        },
+    ];
+    return rows.filter((row) => row.n > 0);
+}
+/**
+ * Cap a health score at {@link CONFIDENT_BREAKAGE_CAP} when the report carries any
+ * {@link confidentBreakages} finding. Applied to the OVERALL and to the ring that
+ * owns the finding, so the headline and the breakdown cannot disagree.
+ */
+function applyBreakageCap(score, r) {
+    return confidentBreakages(r).length > 0
+        ? Math.min(score, exports.CONFIDENT_BREAKAGE_CAP)
+        : score;
+}
 /**
  * The lethal-trifecta exposure a report incurs — the ONE number the Safety ring
  * and the overall grade both read.
@@ -263,7 +349,10 @@ function scoreReport(r) {
         return { score: 0, issues: ["no loadable plugin surface"] };
     }
     const deductions = reportDeductions(r);
-    const { score } = computeIntegrityScore(deductions);
+    const { score: summed } = computeIntegrityScore(deductions);
+    // A confident breakage caps the score — a summed model dilutes one real
+    // breakage among clean siblings. See applyBreakageCap for the measurement.
+    const score = applyBreakageCap(summed, r);
     const issues = [];
     for (const d of deductions) {
         if (d.n === 0)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vigiles",
-  "version": "21.0.1",
+  "version": "21.0.2",
   "description": "Audit, test and measure the harness your AI agent runs on — grade your CLAUDE.md / AGENTS.md, skills, subagents and hooks, run them against a scripted model, and measure whether they actually fire.",
   "keywords": [
     "claude-code",

package/skills/test-harness/SKILL.md

@@ -49,83 +49,18 @@ not established, which is why the floor stays.
 If the unit and deterministic tiers can both answer it, **prefer unit**: it's
 faster and reaches events the deterministic mock can't drive.
 
-## Step 0.4 — Observing a run (what it CALLED, WROTE, and TOUCHED)
-
-The table above is keyed on the harness _surface_ under test. Half the real
-questions are keyed on the **observation** instead — "what did this skill
-actually do?" — and they have answers already. Reach for these before building
-anything; every one of them ships today.
-
-| The question you're actually asking                                     | Use                                                                                    |
-| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
-| Which tools did it call, and with what arguments?                       | `trace.toolCalls` · `tool` / `toolWith` checks · `parseToolCalls` (`vigiles`)          |
-| Did it call a tool it must not?                                         | `notTool(name)`                                                                        |
-| Did it call **only** tools from a known set?                            | `onlyTools([...])` — the white-list, symmetric to `assertWroteOnly`                    |
-| Did it stay inside the `allowed-tools` its own frontmatter declares?    | `skillContract(dir).surface` — builds that check FROM the declaration                  |
-| What files did the run write?                                           | `filesWritten` · `wrote(path)` / `didNotWrite(path)` · `r.file(path)`                  |
-| Did it write **only** where it was supposed to?                         | `assertWroteOnly([...])` / `assertNoWrite()` — needs `{ sandbox: "auto" }`             |
-| Run a tool call but **don't let it execute** — capture the args instead | the `interceptTools` option on `measure` / `runEval` (a `ToolIntercept[]`)             |
-| Did a subagent do it, and which one?                                    | `subagent(name, [...])` · `SubagentTrace`                                              |
-| Was it an MCP tool?                                                     | `mcp(server, toolName)`                                                                |
-| Assert the whole effect boundary deterministically                      | `assertChecks` + the checks above (see `examples/harness/effect-boundary.harness.mjs`) |
-
-`interceptTools` is the one worth knowing about, because it is not obvious it
-exists: it denies a tool its **real execution** via an auto-wired `PreToolUse`
-hook while still recording the call and its arguments into the trace. That is
-how you test a skill that would otherwise mutate a real external service — a
-calendar, an upload — without mocking anything yourself.
-
-**Verify a skill against its own declaration** with `skillContract` — it reads
-the `allowed-tools:` the skill already claims and hands back ready checks, so
-the claim is verified instead of restated:
-
-```ts
-import { skillContract, assertChecks } from "vigiles";
-
-const c = skillContract(".claude/skills/my-skill");
-assertChecks(trace, [c.activation, ...c.surface]);
-```
+## Step 0.4 — Observing a run, and what it costs
+
+Two questions have their own references — open the one you need, don't guess:
 
-Two of its states are **findings**, not clean bills, and their `surface` check
-fails rather than passing on nothing: `undeclared` (no `allowed-tools:` line, so
-the skill inherits _every_ tool) and `malformed` (frontmatter that isn't valid
-YAML, so a strict loader reads no contract at all — one unquoted `: ` does it).
-
-⚠️ **What is still NOT checked.** `onlyTools` compares tool _names_, so a narrow
-allowlist entry like `Bash(node scripts/x.mjs:*)` is satisfied by any `Bash` call
-at all. Scope inside a tool is unverified — say so rather than implying the
-assertion is total.
-
-## Step 0.5 — Set honest expectations (what's testable, and at what cost)
-
-Be explicit with the user about which bucket each surface falls into — never let
-"we'll test it" hide whether that's free, sub-priced, or needs a container. Every
-surface sorts into one of three buckets:
-
-- **A — Free & deterministic** (no model, runs in CI on every commit): a hook's
-  block/allow decision (`runHook`), a tool-contract / "did NOT call the forbidden
-  tool" check, structural facts (`vigiles audit`), and **record-replay** of any tool
-  a skill shells out to (record the real result once, replay it via a PATH stub).
-- **B — Model-gated, on your subscription** (real model, **no metered API**): does a
-  skill's description **fire** (`measureTriggerRate`, recall + precision) **and**
-  does its guidance actually **produce good output** (score it directly:
-  `measure({ checks: [judged(rubric)] })` + `assertRates` — the absolute oracle;
-  use a `runEval` A/B on-vs-off only when you need the _relative_ lift). This is
-  the half a **prose / guidance skill** lives in —
-  its worth is behavioral, so only a model can judge it. That is **not** "uncovered"
-  and **not** free: it's fully testable on the sub. State it that way.
-- **C — Needs a real service** (a real browser / DB / redis / a11y runtime): vigiles
-  **composes with a container** here; it does not fake real semantics. Name the
-  service and hand off — don't pretend a cheap tier substitutes for it.
-
-So a prose-skill library is roughly **~100% testable (some free, most on your sub),
-~0% needs-a-container** — not "poorly covered." An accessibility/browser plugin is
-the worst case, with a large bucket C. When you report coverage, give **two
-numbers**: "% testable at all (free + sub)" vs "% that needs a container", and say
-which surfaces are free vs sub-priced. The model-gated half is the **point** of the
-eval pillar (affordable on the sub), not a gap — and testing a prose skill's
-_behavior_ requires a real model for **everyone** (promptfoo, the SDKs, all of it);
-vigiles just does it on your subscription instead of metered API.
+- **"What did the run actually DO?"** — which tools it called, whether it stayed
+  inside its declared `allowed-tools`, what it wrote, how to record a call without
+  executing it → [`references/observing-a-run.md`](references/observing-a-run.md)
+- **"Is this free, sub-priced, or does it need a container?"** — the three buckets,
+  and what to tell the user after a paid run →
+  [`references/cost-and-expectations.md`](references/cost-and-expectations.md)
+
+Never say "we'll test it" without settling the second one first.
 
 ## Step 1 — Ensure vigiles is installed
 
@@ -154,142 +89,12 @@ Pick one concrete thing to pin down — a specific `PreToolUse` hook, a specific
 
 ## Step 3 — Write the test for the chosen tier
 
-**Unit (`runHook`)** — hand a hook a synthesized event, assert the decision:
-
-```ts
-import { runHook, assertHookBlocked } from "vigiles";
-
-const r = runHook(hookCommand, {
-  hook_event_name: "PreToolUse",
-  tool_name: "Bash",
-  tool_input: { command: "git commit --no-verify" },
-});
-assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"
-```
-
-Testing a hook you didn't write (a vendored third-party script)? Mark it
-`{ trusted: false }` and it runs confined under bubblewrap by default (read-only
-host, cleared env, no network egress). Add `{ recordEgress: true }` to also
-**record** what it tries to reach — `r.egress` plus `assertNoEgress(r)` /
-`assertEgressOnly(r, [...])` — the supply-chain check for "what does this skill
-phone home to / install from?". When the hook's setup needs a _real_ install,
-`{ egress: { allow: ["registry.npmjs.org"] } }` lets it reach only that
-allowlist (a packet-layer `nft` wall, so a raw socket off-list is dropped too) →
-`r.egress` (allowed hosts) + `r.egressDropped`. Be precise about the boundaries:
-see
-[`docs/sandboxing.md`](../../docs/sandboxing.md) (it blocks destruction and
-egress, but does NOT isolate reads of host files, and only under bwrap).
-
-**Deterministic (`runHarnessTest`)** — load the real plugin, drive a scripted
-mock model, assert the hook fired (or the context landed):
-
-```ts
-import {
-  runHarnessTest,
-  assertHookFired,
-  assertRequestContains,
-} from "vigiles";
-// `scriptModel` is the Claude-Code TRANSPORT, deliberately not re-exported from
-// the harness-agnostic root surface — import it from the harness package:
-import { scriptModel } from "vigiles/claude-code";
-
-const r = await runHarnessTest({
-  pluginDir: "./", // or { settings: { hooks: {...} } }
-  transcript: true,
-  model: scriptModel([{ text: "ok" }]),
-});
-assertHookFired(r, "SessionStart");
-assertRequestContains(r, "expected injected text"); // did it actually land?
-```
-
-**Eval — absolute (`paid_measure` + `paid_judged`)** — testing _one_ skill, the usual case:
-score its output directly against a rubric. No on/off baseline — this is the
-"is it any good?" oracle (what promptfoo/DeepEval lead with), and the right
-default when there's nothing to compare against:
-
-An eval file **describes** its eval — it must never run one at the top level,
-because importing such a file spends real money. Write `<name>.eval.mjs`:
-
-```ts
-import { defineEval, skill, assertRates } from "vigiles";
-import { paid_judged } from "vigiles/eval"; // a Check whose default judge bills
-
-export default defineEval({
-  measure: {
-    pluginDir: "./",
-    task: "…a task the skill should handle…",
-    checks: [
-      skill("my-plugin:my-skill"), // it fired
-      paid_judged("the answer correctly does X and avoids Y"), // …and the output is good
-    ],
-    trials: 6,
-  },
-  assert: (report) => assertRates(report, { min: 0.8 }), // each check ≥ 80% of trials
-});
-```
-
-Run it with `npx vigiles eval <file>` — never `node <file>`, which refuses.
-
-**Eval — relative (`paid_runEval` + `assertSignificant`)** — when the question is
-_lift over no-skill_ (regression, or proving a change isn't noise): A/B the
-change on vs off and gate on significance, not eyeballing:
-
-```ts
-import { defineEval, assertSignificant } from "vigiles";
-
-export default defineEval({
-  runEval: {
-    arms: { off: {}, on: { pluginDir: "./" } },
-    task: "…a task the harness change should affect…",
-    measure: (ctx) => ({ ok: /* a bare predicate over the trace */ true }),
-    trials: 6,
-    cache: "readwrite",
-  },
-  assert: (report) =>
-    assertSignificant(report, { baseline: "off", arm: "on", metric: "ok" }),
-});
-```
-
-### Never hand-roll the runner — it silently eats stderr
+Per-tier skeletons, and the one mistake that silently swallows failures (a
+hand-rolled runner eats stderr) →
+[`references/writing-tests.md`](references/writing-tests.md)
 
-Do **not** reach for `execFileSync` / `spawnSync` to drive the thing under test.
-The failure is quiet and repeats: `execFileSync` returns **stdout only** on
-success, while advisory output — including vigiles's own compiled-hook
-`notice()` — is written to **stderr**. A hand-rolled runner therefore reports a
-perfectly healthy react hook as **dead**, and an assertion about a warning can
-never pass. (Observed three times in one repo, twice after the first fix.)
-
-Every vigiles result already carries **both streams**, so the bug is
-unrepresentable:
-
-| Runner           | Result              | Carries                                             |
-| ---------------- | ------------------- | --------------------------------------------------- |
-| `runScript`      | `ScriptRunResult`   | `exitCode`, `stdout`, `stderr`, `filesWritten?`     |
-| `runHook`        | `HookRunResult`     | all of the above, **plus** `blocked` / `decision`   |
-| `runHarnessTest` | `HarnessTestResult` | `exitCode`, `stdout`, `stderr`, `cwd` + the `Trace` |
-
-**Testing a plain helper script** (a bash/node/python program that isn't a hook)?
-Use **`runScript`** — it runs any command and reports what it did:
-
-```ts
-import { runScript } from "vigiles";
-
-const r = runScript("bash scripts/check-links.sh", { cwd: repoDir });
-assert.equal(r.exitCode, 0);
-assert.match(r.stderr, /0 broken links/); // advisory output lives HERE
-```
-
-`runHook` is exactly `runScript` plus the hook protocol (event → stdin, exit code
-→ allow/deny). Pick by the question you're asking: a **hook** has a _decision_, a
-**script** has _effects_. That's why `ScriptRunResult` has no `decision` field —
-a field that is always meaningless is worse than no field.
-
-⚠️ **Asserting what a script wrote requires confinement.** `filesWritten` is
-recorded by diffing the work dir, which only a confined run does — so it is
-`undefined` after a plain run. That is deliberately _not_ the same as `[]`
-("recorded, wrote nothing"): `assertNoWrite` / `assertWroteOnly` **throw** on an
-unrecorded result rather than pass having inspected nothing. Pass
-`{ sandbox: "auto" }` (Linux + bubblewrap) to actually record writes.
+Read it before writing the file — the skeleton differs per tier, and the runner
+warning has cost real debugging time.
 
 ## Step 4 — Run it
 
@@ -309,23 +114,6 @@ job that asserts the capability is present, run **`vigiles test --no-skip`** so
 skipped tier fails — a green-with-skips is untested surface. Keep unit +
 deterministic tests in CI (free); run evals locally or on a schedule with auth.
 
-### After a real-model run: TELL THE USER WHAT IT SPENT
-
-Whenever you run a real-model eval (`runEval` / `measureArms` / `measureTriggerRate`
-/ `measure`), **surface the spend to the user in your reply** — don't let a paid run
-be silent. `runEval` prints a cost block to stderr and every report carries `usage`
-(`report.arms[*].usage`: `totalCostUsd` + token counts). Relay, in plain words:
-
-- **tokens spent** and the **API-equivalent `$`** (`total_cost_usd` — what it _would_
-  cost at metered API rates);
-- **how it was billed** — "on your Claude subscription (**$0 metered**)" if you're
-  logged in, or a **⚠ warning** if `ANTHROPIC_API_KEY` is set (that run was billed
-  **per token** — tell them to unset it and `claude login` to run free).
-
-We do **not** show "% of your subscription" — Anthropic doesn't expose a plan's
-quota, so any percentage would be invented. Tokens + API-equivalent `$` + the
-billed-to line is the honest, complete picture. Keep the user's cost visible, always.
-
 ## CI — don't hand-write the steps
 
 These tiers belong in CI, and there is a published Action for it. Run `vigiles init`: it

package/skills/test-harness/references/cost-and-expectations.md

@@ -0,0 +1,53 @@
+# Honest expectations and cost
+
+Read this before telling a user "we'll test it" — it fixes which bucket a surface
+falls into (free / on your subscription / needs a container) and what to report
+after a paid run.
+
+## Set honest expectations (what's testable, and at what cost)
+
+Be explicit with the user about which bucket each surface falls into — never let
+"we'll test it" hide whether that's free, sub-priced, or needs a container. Every
+surface sorts into one of three buckets:
+
+- **A — Free & deterministic** (no model, runs in CI on every commit): a hook's
+  block/allow decision (`runHook`), a tool-contract / "did NOT call the forbidden
+  tool" check, structural facts (`vigiles audit`), and **record-replay** of any tool
+  a skill shells out to (record the real result once, replay it via a PATH stub).
+- **B — Model-gated, on your subscription** (real model, **no metered API**): does a
+  skill's description **fire** (`measureTriggerRate`, recall + precision) **and**
+  does its guidance actually **produce good output** (score it directly:
+  `measure({ checks: [judged(rubric)] })` + `assertRates` — the absolute oracle;
+  use a `runEval` A/B on-vs-off only when you need the _relative_ lift). This is
+  the half a **prose / guidance skill** lives in —
+  its worth is behavioral, so only a model can judge it. That is **not** "uncovered"
+  and **not** free: it's fully testable on the sub. State it that way.
+- **C — Needs a real service** (a real browser / DB / redis / a11y runtime): vigiles
+  **composes with a container** here; it does not fake real semantics. Name the
+  service and hand off — don't pretend a cheap tier substitutes for it.
+
+So a prose-skill library is roughly **~100% testable (some free, most on your sub),
+~0% needs-a-container** — not "poorly covered." An accessibility/browser plugin is
+the worst case, with a large bucket C. When you report coverage, give **two
+numbers**: "% testable at all (free + sub)" vs "% that needs a container", and say
+which surfaces are free vs sub-priced. The model-gated half is the **point** of the
+eval pillar (affordable on the sub), not a gap — and testing a prose skill's
+_behavior_ requires a real model for **everyone** (promptfoo, the SDKs, all of it);
+vigiles just does it on your subscription instead of metered API.
+
+### After a real-model run: TELL THE USER WHAT IT SPENT
+
+Whenever you run a real-model eval (`runEval` / `measureArms` / `measureTriggerRate`
+/ `measure`), **surface the spend to the user in your reply** — don't let a paid run
+be silent. `runEval` prints a cost block to stderr and every report carries `usage`
+(`report.arms[*].usage`: `totalCostUsd` + token counts). Relay, in plain words:
+
+- **tokens spent** and the **API-equivalent `$`** (`total_cost_usd` — what it _would_
+  cost at metered API rates);
+- **how it was billed** — "on your Claude subscription (**$0 metered**)" if you're
+  logged in, or a **⚠ warning** if `ANTHROPIC_API_KEY` is set (that run was billed
+  **per token** — tell them to unset it and `claude login` to run free).
+
+We do **not** show "% of your subscription" — Anthropic doesn't expose a plan's
+quota, so any percentage would be invented. Tokens + API-equivalent `$` + the
+billed-to line is the honest, complete picture. Keep the user's cost visible, always.

package/skills/test-harness/references/observing-a-run.md

@@ -0,0 +1,51 @@
+# Observing a run — what it CALLED, WROTE, and TOUCHED
+
+Read this when the question is about **what a run did**, not about which tier to
+pick. Every predicate here ships today.
+
+## Observing a run (what it CALLED, WROTE, and TOUCHED)
+
+The table above is keyed on the harness _surface_ under test. Half the real
+questions are keyed on the **observation** instead — "what did this skill
+actually do?" — and they have answers already. Reach for these before building
+anything; every one of them ships today.
+
+| The question you're actually asking                                     | Use                                                                                    |
+| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
+| Which tools did it call, and with what arguments?                       | `trace.toolCalls` · `tool` / `toolWith` checks · `parseToolCalls` (`vigiles`)          |
+| Did it call a tool it must not?                                         | `notTool(name)`                                                                        |
+| Did it call **only** tools from a known set?                            | `onlyTools([...])` — the white-list, symmetric to `assertWroteOnly`                    |
+| Did it stay inside the `allowed-tools` its own frontmatter declares?    | `skillContract(dir).surface` — builds that check FROM the declaration                  |
+| What files did the run write?                                           | `filesWritten` · `wrote(path)` / `didNotWrite(path)` · `r.file(path)`                  |
+| Did it write **only** where it was supposed to?                         | `assertWroteOnly([...])` / `assertNoWrite()` — needs `{ sandbox: "auto" }`             |
+| Run a tool call but **don't let it execute** — capture the args instead | the `interceptTools` option on `measure` / `runEval` (a `ToolIntercept[]`)             |
+| Did a subagent do it, and which one?                                    | `subagent(name, [...])` · `SubagentTrace`                                              |
+| Was it an MCP tool?                                                     | `mcp(server, toolName)`                                                                |
+| Assert the whole effect boundary deterministically                      | `assertChecks` + the checks above (see `examples/harness/effect-boundary.harness.mjs`) |
+
+`interceptTools` is the one worth knowing about, because it is not obvious it
+exists: it denies a tool its **real execution** via an auto-wired `PreToolUse`
+hook while still recording the call and its arguments into the trace. That is
+how you test a skill that would otherwise mutate a real external service — a
+calendar, an upload — without mocking anything yourself.
+
+**Verify a skill against its own declaration** with `skillContract` — it reads
+the `allowed-tools:` the skill already claims and hands back ready checks, so
+the claim is verified instead of restated:
+
+```ts
+import { skillContract, assertChecks } from "vigiles";
+
+const c = skillContract(".claude/skills/my-skill");
+assertChecks(trace, [c.activation, ...c.surface]);
+```
+
+Two of its states are **findings**, not clean bills, and their `surface` check
+fails rather than passing on nothing: `undeclared` (no `allowed-tools:` line, so
+the skill inherits _every_ tool) and `malformed` (frontmatter that isn't valid
+YAML, so a strict loader reads no contract at all — one unquoted `: ` does it).
+
+⚠️ **What is still NOT checked.** `onlyTools` compares tool _names_, so a narrow
+allowlist entry like `Bash(node scripts/x.mjs:*)` is satisfied by any `Bash` call
+at all. Scope inside a tool is unverified — say so rather than implying the
+assertion is total.

package/skills/test-harness/references/writing-tests.md

@@ -0,0 +1,143 @@
+# Writing the test for the chosen tier
+
+Read this once the tier is picked. Covers the per-tier skeleton and the one
+mistake that silently swallows failures.
+
+## Write the test for the chosen tier
+
+**Unit (`runHook`)** — hand a hook a synthesized event, assert the decision:
+
+```ts
+import { runHook, assertHookBlocked } from "vigiles";
+
+const r = runHook(hookCommand, {
+  hook_event_name: "PreToolUse",
+  tool_name: "Bash",
+  tool_input: { command: "git commit --no-verify" },
+});
+assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"
+```
+
+Testing a hook you didn't write (a vendored third-party script)? Mark it
+`{ trusted: false }` and it runs confined under bubblewrap by default (read-only
+host, cleared env, no network egress). Add `{ recordEgress: true }` to also
+**record** what it tries to reach — `r.egress` plus `assertNoEgress(r)` /
+`assertEgressOnly(r, [...])` — the supply-chain check for "what does this skill
+phone home to / install from?". When the hook's setup needs a _real_ install,
+`{ egress: { allow: ["registry.npmjs.org"] } }` lets it reach only that
+allowlist (a packet-layer `nft` wall, so a raw socket off-list is dropped too) →
+`r.egress` (allowed hosts) + `r.egressDropped`. Be precise about the boundaries:
+see
+[`docs/sandboxing.md`](../../docs/sandboxing.md) (it blocks destruction and
+egress, but does NOT isolate reads of host files, and only under bwrap).
+
+**Deterministic (`runHarnessTest`)** — load the real plugin, drive a scripted
+mock model, assert the hook fired (or the context landed):
+
+```ts
+import {
+  runHarnessTest,
+  assertHookFired,
+  assertRequestContains,
+} from "vigiles";
+// `scriptModel` is the Claude-Code TRANSPORT, deliberately not re-exported from
+// the harness-agnostic root surface — import it from the harness package:
+import { scriptModel } from "vigiles/claude-code";
+
+const r = await runHarnessTest({
+  pluginDir: "./", // or { settings: { hooks: {...} } }
+  transcript: true,
+  model: scriptModel([{ text: "ok" }]),
+});
+assertHookFired(r, "SessionStart");
+assertRequestContains(r, "expected injected text"); // did it actually land?
+```
+
+**Eval — absolute (`paid_measure` + `paid_judged`)** — testing _one_ skill, the usual case:
+score its output directly against a rubric. No on/off baseline — this is the
+"is it any good?" oracle (what promptfoo/DeepEval lead with), and the right
+default when there's nothing to compare against:
+
+An eval file **describes** its eval — it must never run one at the top level,
+because importing such a file spends real money. Write `<name>.eval.mjs`:
+
+```ts
+import { defineEval, skill, assertRates } from "vigiles";
+import { paid_judged } from "vigiles/eval"; // a Check whose default judge bills
+
+export default defineEval({
+  measure: {
+    pluginDir: "./",
+    task: "…a task the skill should handle…",
+    checks: [
+      skill("my-plugin:my-skill"), // it fired
+      paid_judged("the answer correctly does X and avoids Y"), // …and the output is good
+    ],
+    trials: 6,
+  },
+  assert: (report) => assertRates(report, { min: 0.8 }), // each check ≥ 80% of trials
+});
+```
+
+Run it with `npx vigiles eval <file>` — never `node <file>`, which refuses.
+
+**Eval — relative (`paid_runEval` + `assertSignificant`)** — when the question is
+_lift over no-skill_ (regression, or proving a change isn't noise): A/B the
+change on vs off and gate on significance, not eyeballing:
+
+```ts
+import { defineEval, assertSignificant } from "vigiles";
+
+export default defineEval({
+  runEval: {
+    arms: { off: {}, on: { pluginDir: "./" } },
+    task: "…a task the harness change should affect…",
+    measure: (ctx) => ({ ok: /* a bare predicate over the trace */ true }),
+    trials: 6,
+    cache: "readwrite",
+  },
+  assert: (report) =>
+    assertSignificant(report, { baseline: "off", arm: "on", metric: "ok" }),
+});
+```
+
+### Never hand-roll the runner — it silently eats stderr
+
+Do **not** reach for `execFileSync` / `spawnSync` to drive the thing under test.
+The failure is quiet and repeats: `execFileSync` returns **stdout only** on
+success, while advisory output — including vigiles's own compiled-hook
+`notice()` — is written to **stderr**. A hand-rolled runner therefore reports a
+perfectly healthy react hook as **dead**, and an assertion about a warning can
+never pass. (Observed three times in one repo, twice after the first fix.)
+
+Every vigiles result already carries **both streams**, so the bug is
+unrepresentable:
+
+| Runner           | Result              | Carries                                             |
+| ---------------- | ------------------- | --------------------------------------------------- |
+| `runScript`      | `ScriptRunResult`   | `exitCode`, `stdout`, `stderr`, `filesWritten?`     |
+| `runHook`        | `HookRunResult`     | all of the above, **plus** `blocked` / `decision`   |
+| `runHarnessTest` | `HarnessTestResult` | `exitCode`, `stdout`, `stderr`, `cwd` + the `Trace` |
+
+**Testing a plain helper script** (a bash/node/python program that isn't a hook)?
+Use **`runScript`** — it runs any command and reports what it did:
+
+```ts
+import { runScript } from "vigiles";
+
+const r = runScript("bash scripts/check-links.sh", { cwd: repoDir });
+assert.equal(r.exitCode, 0);
+assert.match(r.stderr, /0 broken links/); // advisory output lives HERE
+```
+
+`runHook` is exactly `runScript` plus the hook protocol (event → stdin, exit code
+→ allow/deny). Pick by the question you're asking: a **hook** has a _decision_, a
+**script** has _effects_. That's why `ScriptRunResult` has no `decision` field —
+a field that is always meaningless is worse than no field.
+
+⚠️ **Asserting what a script wrote requires confinement.** `filesWritten` is
+recorded by diffing the work dir, which only a confined run does — so it is
+`undefined` after a plain run. That is deliberately _not_ the same as `[]`
+("recorded, wrote nothing"): `assertNoWrite` / `assertWroteOnly` **throw** on an
+unrecorded result rather than pass having inspected nothing. Pass
+`{ sandbox: "auto" }` (Linux + bubblewrap) to actually record writes.