vigiles 2.2.0 → 2.3.0

package/README.md

@@ -8,6 +8,11 @@
   <em>Quis custodiet ipsos custodes?</em> — Who watches the watchmen?
 </p>
 
+<p align="center">
+  <strong>Test &amp; verify your Claude Code harness.</strong><br />
+  vigiles <strong>verifies the references</strong> your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>evals</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
+</p>
+
 <p align="center">
   <a href="https://www.npmjs.com/package/vigiles"><img src="https://img.shields.io/npm/v/vigiles?color=orange" alt="npm version" /></a>
   <a href="https://github.com/zernie/vigiles/actions"><img src="https://img.shields.io/github/actions/workflow/status/zernie/vigiles/ci.yml?branch=main" alt="CI" /></a>
@@ -258,6 +263,8 @@ npx vigiles init [--target=X.md]    # Scaffold a spec (runs full setup wizard by
 npx vigiles compile [files...]      # Compile .spec.ts → .md
 npx vigiles audit [files...]        # Verify hashes + inline/frontmatter/spec rules + symbols + coverage
 npx vigiles refs <file.md>          # Check the symbol references in an instruction file
+npx vigiles test [files...]         # Run *.harness.mjs deterministic harness tests (no API key)
+npx vigiles eval [files...]         # Run *.eval.mjs real-model harness evals (--trials=N)
 npx vigiles generate-types          # Emit .d.ts from project state (for spec mode)
 npx vigiles generate-types --check  # Verify .d.ts is up to date
 npx vigiles generate-schema         # Emit JSON Schema for vigiles: frontmatter (Level 1)
@@ -335,7 +342,7 @@ Install with [Vercel Skills](https://github.com/vercel-labs/skills): `npx skills
 
 vigiles also ships a library for **testing the harness itself** — your hooks,
 settings, skills, and instruction files. `Agent = Model + Harness`; this tests
-the harness, at two levels.
+the harness, at three levels.
 
 **Evals — does my change actually move agent behaviour?** Define a fixture, a set
 of **arms** (a hook on vs off, with/without a CLAUDE.md rule), a task, and a
@@ -390,9 +397,53 @@ const r = await runHarnessTest({
 assert(JSON.parse(r.stdout).num_turns > 1); // the Stop hook forced more work
 ```
 
-The deterministic tier is reliable for **Stop hooks**; tool-event hooks
-(Edit/Write) are headless-gated, so test those via the eval tier. Our own
-findings from this harness live in [`research/benchmarks-runtime-gates.md`](research/benchmarks-runtime-gates.md).
+The deterministic tier is reliable for **SessionStart, Stop, UserPromptSubmit,
+and Bash PreToolUse/PostToolUse** hooks — the governance/policy shapes most real
+plugins use; Edit/Write tool-event hooks are headless-gated, so test those at the
+unit tier or via the eval tier.
+
+**Unit-test a hook — no `claude` at all.** A hook is just a process: `runHook`
+pipes an event JSON to its stdin and reports the block/allow decision —
+milliseconds, and the only tier that reaches **every** event (incl. Edit/Write,
+PreCompact, SessionEnd, which the deterministic mock can't trigger).
+
+```typescript
+import { runHook } from "vigiles/run-hook";
+
+const r = runHook(guardCommand, {
+  hook_event_name: "PreToolUse",
+  tool_name: "Bash",
+  tool_input: { command: "git commit --no-verify" },
+});
+assert(r.blocked); // exit 2, decision:"block", or permissionDecision:"deny"
+```
+
+**Run them as a CI command.** `vigiles test` discovers `*.harness.mjs` files
+(deterministic, no API key) and `vigiles eval` discovers `*.eval.mjs` files
+(real model). Canonical, real-plugin-shaped examples to copy:
+
+- [`examples/harness/policy-gate.harness.mjs`](examples/harness/policy-gate.harness.mjs) — a `PreToolUse` Bash policy gate (block `git commit --no-verify`) and a `SessionStart` setup hook, deterministic.
+- [`examples/harness/skill-outcome.eval.mjs`](examples/harness/skill-outcome.eval.mjs) — does a skill change the agent's output? (the question you ask of any `SKILL.md`).
+
+```bash
+npx vigiles test examples/harness/policy-gate.harness.mjs
+npx vigiles eval --trials=6 examples/harness/skill-outcome.eval.mjs
+```
+
+**Test the whole machine.** Point `plugin` at a plugin (or `"./"` for your repo)
+and the real harness — hooks (with `${CLAUDE_PLUGIN_ROOT}` resolved), CLAUDE.md,
+skills, subagents and commands — is loaded into the sandbox, so you test what
+ships, not a retyped subset. `loadPlugin(...).warnings` flags surfaces only a
+real model can drive (subagents, slash commands, MCP), so loading a whole plugin
+never silently tests an empty machine. The library is plain async functions, so
+it runs in **node:test, vitest, or jest** unchanged (shared `expect.extend`
+matchers for the latter two).
+
+[Full guide → `docs/harness-testing.md`](docs/harness-testing.md). The design
+rationale and a coverage assessment against real plugins (protect-mcp,
+obra/superpowers, block-no-verify, 156 wshobson skills) are in
+[`research/harness-testing.md`](research/harness-testing.md); findings from
+running this harness in anger live in [`research/benchmarks-runtime-gates.md`](research/benchmarks-runtime-gates.md).
 
 ## Maturity Levels
 

package/dist/cli.js

@@ -26,6 +26,8 @@ const action_gate_js_1 = require("./action-gate.js");
 const refs_js_1 = require("./refs.js");
 const skill_runtime_js_1 = require("./skill-runtime.js");
 const linters_js_1 = require("./linters.js");
+const harness_test_js_1 = require("./harness-test.js");
+const run_scripts_js_1 = require("./run-scripts.js");
 const integrity_js_1 = require("./integrity.js");
 const coverage_js_1 = require("./coverage.js");
 const orphans_js_1 = require("./orphans.js");
@@ -1413,6 +1415,38 @@ function handleGenerateSchema(args, restArgs) {
     console.log("  Add to your markdown frontmatter:\n" +
         `    # yaml-language-server: $schema=./${outPath}`);
 }
+/**
+ * `vigiles test` / `vigiles eval` — discover and run the two-tier harness
+ * scripts (deterministic `*.harness.mjs` / real-model `*.eval.mjs`) as child
+ * `node` processes, aggregating exit codes so they work as a CI command. See
+ * src/run-scripts.ts.
+ *
+ * `vigiles test` skips clean when the `claude` CLI is absent (the deterministic
+ * tier needs it, just like the node:test suite). `--trials=N` is forwarded to
+ * eval scripts via the `VIGILES_TRIALS` env var.
+ */
+function handleRunScripts(kind, args, restArgs) {
+    const cwd = process.cwd();
+    const defaultGlob = kind === "test" ? "**/*.harness.mjs" : "**/*.eval.mjs";
+    if (kind === "test" && !(0, harness_test_js_1.claudeAvailable)()) {
+        console.log("vigiles test: `claude` CLI not found — skipping harness tests.");
+        return;
+    }
+    const files = (0, run_scripts_js_1.discoverScripts)(restArgs, defaultGlob, cwd);
+    if (files.length === 0) {
+        console.log(`No ${defaultGlob} files found.`);
+        return;
+    }
+    const trialsFlag = args.find((a) => a.startsWith("--trials="));
+    const env = {};
+    if (trialsFlag)
+        env.VIGILES_TRIALS = trialsFlag.split("=")[1];
+    console.log(`Running ${String(files.length)} ${kind} file(s):\n`);
+    const results = (0, run_scripts_js_1.runScripts)(files, cwd, env);
+    console.log("\n" + (0, run_scripts_js_1.formatScriptSummary)(results));
+    if (results.some((r) => r.code !== 0))
+        process.exit(1);
+}
 function printUsage(command) {
     console.log("vigiles — compile typed specs to instruction files");
     console.log("");
@@ -1420,6 +1454,8 @@ function printUsage(command) {
     console.log("  vigiles init [flags]           Setup project (--target=X.md, --strict, --no-gha)");
     console.log("  vigiles compile [files...]     Compile .spec.ts → .md");
     console.log("  vigiles audit [files...]       Verify, find gaps, suggest improvements");
+    console.log("  vigiles test [files...]        Run *.harness.mjs deterministic harness tests");
+    console.log("  vigiles eval [files...]        Run *.eval.mjs real-model harness evals (--trials=N)");
     console.log("");
     console.log("Examples:");
     console.log("  vigiles init                 Auto-detect project, create specs, wire CI");
@@ -1725,6 +1761,12 @@ async function main() {
             }
             break;
         }
+        case "test":
+            handleRunScripts("test", args, restArgs);
+            break;
+        case "eval":
+            handleRunScripts("eval", args, restArgs);
+            break;
         // --- Plumbing ---
         case "generate-types":
             handleGenerateTypes(args, restArgs);

package/dist/eval.d.ts

@@ -4,6 +4,12 @@ export interface EvalArm {
     readonly files?: Record<string, string>;
     /** `.claude/settings.json` (hooks/permissions) for this arm; omit for none. */
     readonly settings?: unknown;
+    /**
+     * Path to a real plugin/repo to load for this arm (hooks + CLAUDE.md +
+     * skills). Lets an arm be "the whole plugin on" vs "off". See
+     * src/plugin-loader.ts.
+     */
+    readonly plugin?: string;
 }
 /** Context handed to `measure` after a run, to compute that run's metrics. */
 export interface RunContext {
@@ -39,10 +45,23 @@ export interface EvalSpec<M extends Metrics> {
     /** Seconds to wait between runs (avoid rate-limit bursts). Default 4. */
     readonly spacingSec?: number;
 }
+/** Per-metric summary statistics across an arm's runs. */
+export interface MetricStat {
+    /** Mean (numbers) / fraction-true (booleans). */
+    readonly mean: number;
+    /** Sample standard deviation (0 when n < 2). */
+    readonly std: number;
+    /** Standard error of the mean (std / √n). */
+    readonly se: number;
+    /** Number of runs the metric was observed in. */
+    readonly n: number;
+}
 export interface ArmReport {
     readonly runs: number;
     /** Aggregated metrics: mean for numbers, fraction-true (0..1) for booleans. */
     readonly metrics: Record<string, number>;
+    /** Per-metric mean / std / se / n, so an A/B gap can be read for significance. */
+    readonly stats: Record<string, MetricStat>;
 }
 export interface EvalReport {
     readonly name: string;
@@ -51,12 +70,18 @@ export interface EvalReport {
 }
 /** Aggregate per-run metrics: mean for numbers, fraction-true (0..1) for booleans. */
 export declare function aggregate(rows: readonly Metrics[]): Record<string, number>;
+/**
+ * Aggregate per-run metrics with spread: mean, sample std, standard error, and
+ * n. The se/std let you judge whether an A/B gap between arms is real or noise —
+ * a difference smaller than the combined se is not yet significant.
+ */
+export declare function aggregateStats(rows: readonly Metrics[]): Record<string, MetricStat>;
 /**
  * Run the eval: every arm × every trial against the real `claude` CLI, with the
  * metric computed per run and aggregated per arm. Requires `claude` on PATH and
  * working model auth (e.g. `ANTHROPIC_API_KEY`).
  */
 export declare function runEval<M extends Metrics>(spec: EvalSpec<M>): Promise<EvalReport>;
-/** Format an eval report as a compact table for the console. */
+/** Format an eval report as a compact table for the console (mean ± se). */
 export declare function formatEvalReport(report: EvalReport): string;
 //# sourceMappingURL=eval.d.ts.map

package/dist/eval.js

@@ -1,6 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.aggregate = aggregate;
+exports.aggregateStats = aggregateStats;
 exports.runEval = runEval;
 exports.formatEvalReport = formatEvalReport;
 /**
@@ -30,6 +31,7 @@ const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
 const node_os_1 = require("node:os");
 const node_path_1 = require("node:path");
+const plugin_loader_js_1 = require("./plugin-loader.js");
 function writeFiles(cwd, files) {
     for (const [p, content] of Object.entries(files)) {
         const full = (0, node_path_1.resolve)(cwd, p);
@@ -101,28 +103,46 @@ function makeContext(cwd, out) {
         },
     };
 }
+/** Coerce a metric value to a number (booleans → 0/1), or null if absent. */
+function numeric(v) {
+    if (typeof v === "number")
+        return v;
+    if (typeof v === "boolean")
+        return v ? 1 : 0;
+    return null;
+}
 /** Aggregate per-run metrics: mean for numbers, fraction-true (0..1) for booleans. */
 function aggregate(rows) {
+    const stats = aggregateStats(rows);
+    const out = {};
+    for (const [k, s] of Object.entries(stats))
+        out[k] = s.mean;
+    return out;
+}
+/**
+ * Aggregate per-run metrics with spread: mean, sample std, standard error, and
+ * n. The se/std let you judge whether an A/B gap between arms is real or noise —
+ * a difference smaller than the combined se is not yet significant.
+ */
+function aggregateStats(rows) {
     const keys = new Set();
     for (const r of rows)
         for (const k of Object.keys(r))
             keys.add(k);
     const out = {};
     for (const k of keys) {
-        let sum = 0;
-        let n = 0;
+        const values = [];
         for (const r of rows) {
-            const v = r[k];
-            if (typeof v === "number") {
-                sum += v;
-                n++;
-            }
-            else if (typeof v === "boolean") {
-                sum += v ? 1 : 0;
-                n++;
-            }
+            const v = numeric(r[k]);
+            if (v !== null)
+                values.push(v);
         }
-        out[k] = n > 0 ? sum / n : 0;
+        const n = values.length;
+        const mean = n > 0 ? values.reduce((a, b) => a + b, 0) / n : 0;
+        const std = n > 1
+            ? Math.sqrt(values.reduce((a, b) => a + (b - mean) ** 2, 0) / (n - 1))
+            : 0;
+        out[k] = { mean, std, se: n > 0 ? std / Math.sqrt(n) : 0, n };
     }
     return out;
 }
@@ -143,10 +163,15 @@ async function runEval(spec) {
         for (let t = 0; t < trials; t++) {
             const cwd = (0, node_fs_1.mkdtempSync)((0, node_path_1.join)((0, node_os_1.tmpdir)(), "vigiles-eval-"));
             try {
-                writeFiles(cwd, { ...spec.fixture, ...arm.files });
-                const hasSettings = arm.settings !== undefined;
+                const { files, settings } = (0, plugin_loader_js_1.resolveHarness)({
+                    plugin: arm.plugin,
+                    settings: arm.settings,
+                    files: { ...spec.fixture, ...arm.files },
+                });
+                writeFiles(cwd, files);
+                const hasSettings = settings !== undefined;
                 if (hasSettings) {
-                    (0, node_fs_1.writeFileSync)((0, node_path_1.join)(cwd, "settings.json"), JSON.stringify(arm.settings, null, 2).replaceAll("{cwd}", cwd));
+                    (0, node_fs_1.writeFileSync)((0, node_path_1.join)(cwd, "settings.json"), JSON.stringify(settings, null, 2).replaceAll("{cwd}", cwd));
                 }
                 const out = await spawnAgent(spec.task, cwd, model, tools, hasSettings, timeoutMs);
                 rows.push(spec.measure(makeContext(cwd, out)));
@@ -156,16 +181,25 @@ async function runEval(spec) {
                 await sleep(spacing);
             }
         }
-        arms[armName] = { runs: rows.length, metrics: aggregate(rows) };
+        arms[armName] = {
+            runs: rows.length,
+            metrics: aggregate(rows),
+            stats: aggregateStats(rows),
+        };
     }
     return { name: spec.name ?? "eval", trials, arms };
 }
-/** Format an eval report as a compact table for the console. */
+/** Format an eval report as a compact table for the console (mean ± se). */
 function formatEvalReport(report) {
     const lines = [`${report.name} (${String(report.trials)} trials/arm)`];
     for (const [arm, r] of Object.entries(report.arms)) {
         const parts = Object.entries(r.metrics)
-            .map(([k, v]) => `${k}=${v.toFixed(2)}`)
+            .map(([k, v]) => {
+            const se = r.stats[k]?.se ?? 0;
+            return se > 0
+                ? `${k}=${v.toFixed(2)}±${se.toFixed(2)}`
+                : `${k}=${v.toFixed(2)}`;
+        })
             .join("  ");
         lines.push(`  ${arm.padEnd(10)} ${parts}`);
     }

package/dist/harness-assert.d.ts

@@ -0,0 +1,68 @@
+/**
+ * vigiles — runner-agnostic helpers for harness tests / evals.
+ *
+ * `runHarnessTest` and `runEval` are plain async functions that return data, so
+ * they already work inside any runner (node:test, vitest, jest, mocha). These
+ * helpers remove the last bit of boilerplate without coupling to a runner:
+ *
+ *   - `withHarness` — run a harness test and auto-clean the sandbox (try/finally),
+ *     so you don't leak temp dirs in `afterEach`.
+ *   - plain `assert*` helpers that throw — usable in every runner, including
+ *     node:test which has no `expect.extend`.
+ *   - `vigilesMatchers` — register with `expect.extend(vigilesMatchers)` for
+ *     `expect(...).toHaveCreated(...)` sugar. The signature is identical for
+ *     vitest and jest, so the same object supports both.
+ */
+import { type HarnessTestSpec, type HarnessTestResult } from "./harness-test.js";
+import type { EvalReport } from "./eval.js";
+import type { HookRunResult } from "./run-hook.js";
+/**
+ * Run a harness test, hand the result to `fn`, and always clean up the sandbox.
+ * Returns whatever `fn` returns. Use this instead of calling `cleanup()` by
+ * hand — it survives assertion failures.
+ */
+export declare function withHarness<T>(spec: HarnessTestSpec, fn: (r: HarnessTestResult) => T | Promise<T>): Promise<T>;
+/** Assert the sandbox contains `path` (a hook/agent side-effect file). */
+export declare function assertCreated(r: HarnessTestResult, path: string): void;
+/** Assert the sandbox does NOT contain `path` (e.g. a blocked action's output). */
+export declare function assertNotCreated(r: HarnessTestResult, path: string): void;
+/** Assert the scripted model served at least `n` turns (e.g. a Stop hook forced more). */
+export declare function assertServedTurns(r: HarnessTestResult, n: number): void;
+/** Assert a `runHook` result blocked (exit 2 / decision:block / permission:deny). */
+export declare function assertHookBlocked(r: HookRunResult): void;
+/** Assert a `runHook` result allowed (did not block). */
+export declare function assertHookAllowed(r: HookRunResult): void;
+/** The gap on `metric` between two arms (arm − baseline). */
+export declare function improvement(report: EvalReport, baseline: string, arm: string, metric: string): number;
+/**
+ * Assert `arm` beats `baseline` on `metric` by more than `by`. With `by` left at
+ * 0 this just asserts a positive gap; pass the combined se to demand the gap
+ * clear the noise floor.
+ */
+export declare function assertImproves(report: EvalReport, opts: {
+    baseline: string;
+    arm: string;
+    metric: string;
+    by?: number;
+}): void;
+interface MatcherOutput {
+    pass: boolean;
+    message: () => string;
+}
+/**
+ * Custom matchers compatible with both vitest and jest. Register once:
+ *
+ *   import { expect } from "vitest"; // or "@jest/globals"
+ *   import { vigilesMatchers } from "vigiles/harness-assert";
+ *   expect.extend(vigilesMatchers);
+ *
+ *   expect(result).toHaveCreated("RESULT");
+ *   expect(report).toBeatBaseline("vanilla", "gated", "caught");
+ */
+export declare const vigilesMatchers: {
+    toHaveCreated(received: HarnessTestResult, path: string): MatcherOutput;
+    toBlock(received: HookRunResult): MatcherOutput;
+    toBeatBaseline(received: EvalReport, baseline: string, arm: string, metric: string, by?: number): MatcherOutput;
+};
+export {};
+//# sourceMappingURL=harness-assert.d.ts.map

package/dist/harness-assert.js

@@ -0,0 +1,127 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.vigilesMatchers = void 0;
+exports.withHarness = withHarness;
+exports.assertCreated = assertCreated;
+exports.assertNotCreated = assertNotCreated;
+exports.assertServedTurns = assertServedTurns;
+exports.assertHookBlocked = assertHookBlocked;
+exports.assertHookAllowed = assertHookAllowed;
+exports.improvement = improvement;
+exports.assertImproves = assertImproves;
+/**
+ * vigiles — runner-agnostic helpers for harness tests / evals.
+ *
+ * `runHarnessTest` and `runEval` are plain async functions that return data, so
+ * they already work inside any runner (node:test, vitest, jest, mocha). These
+ * helpers remove the last bit of boilerplate without coupling to a runner:
+ *
+ *   - `withHarness` — run a harness test and auto-clean the sandbox (try/finally),
+ *     so you don't leak temp dirs in `afterEach`.
+ *   - plain `assert*` helpers that throw — usable in every runner, including
+ *     node:test which has no `expect.extend`.
+ *   - `vigilesMatchers` — register with `expect.extend(vigilesMatchers)` for
+ *     `expect(...).toHaveCreated(...)` sugar. The signature is identical for
+ *     vitest and jest, so the same object supports both.
+ */
+const harness_test_js_1 = require("./harness-test.js");
+/**
+ * Run a harness test, hand the result to `fn`, and always clean up the sandbox.
+ * Returns whatever `fn` returns. Use this instead of calling `cleanup()` by
+ * hand — it survives assertion failures.
+ */
+async function withHarness(spec, fn) {
+    const r = await (0, harness_test_js_1.runHarnessTest)(spec);
+    try {
+        return await fn(r);
+    }
+    finally {
+        r.cleanup();
+    }
+}
+// --- Plain throwing assertions (any runner) --------------------------------
+function fail(message) {
+    throw new Error(message);
+}
+/** Assert the sandbox contains `path` (a hook/agent side-effect file). */
+function assertCreated(r, path) {
+    if (r.file(path) === null)
+        fail(`expected the run to create ${path}`);
+}
+/** Assert the sandbox does NOT contain `path` (e.g. a blocked action's output). */
+function assertNotCreated(r, path) {
+    if (r.file(path) !== null)
+        fail(`expected the run NOT to create ${path}`);
+}
+/** Assert the scripted model served at least `n` turns (e.g. a Stop hook forced more). */
+function assertServedTurns(r, n) {
+    if (r.turns < n) {
+        fail(`expected ≥ ${String(n)} model turns, got ${String(r.turns)}`);
+    }
+}
+/** Assert a `runHook` result blocked (exit 2 / decision:block / permission:deny). */
+function assertHookBlocked(r) {
+    if (!r.blocked) {
+        fail(`expected the hook to block, but it allowed (exit ${String(r.exitCode)})`);
+    }
+}
+/** Assert a `runHook` result allowed (did not block). */
+function assertHookAllowed(r) {
+    if (r.blocked) {
+        fail(`expected the hook to allow, but it blocked (exit ${String(r.exitCode)}, decision ${String(r.decision)})`);
+    }
+}
+/** The gap on `metric` between two arms (arm − baseline). */
+function improvement(report, baseline, arm, metric) {
+    const a = report.arms[arm]?.metrics[metric] ?? 0;
+    const b = report.arms[baseline]?.metrics[metric] ?? 0;
+    return a - b;
+}
+/**
+ * Assert `arm` beats `baseline` on `metric` by more than `by`. With `by` left at
+ * 0 this just asserts a positive gap; pass the combined se to demand the gap
+ * clear the noise floor.
+ */
+function assertImproves(report, opts) {
+    const by = opts.by ?? 0;
+    const delta = improvement(report, opts.baseline, opts.arm, opts.metric);
+    if (delta <= by) {
+        fail(`expected ${opts.arm} to beat ${opts.baseline} on ${opts.metric} by > ${String(by)}, got ${delta.toFixed(3)}`);
+    }
+}
+/**
+ * Custom matchers compatible with both vitest and jest. Register once:
+ *
+ *   import { expect } from "vitest"; // or "@jest/globals"
+ *   import { vigilesMatchers } from "vigiles/harness-assert";
+ *   expect.extend(vigilesMatchers);
+ *
+ *   expect(result).toHaveCreated("RESULT");
+ *   expect(report).toBeatBaseline("vanilla", "gated", "caught");
+ */
+exports.vigilesMatchers = {
+    toHaveCreated(received, path) {
+        const pass = received.file(path) !== null;
+        return {
+            pass,
+            message: () => `expected the run ${pass ? "not " : ""}to create ${path}`,
+        };
+    },
+    toBlock(received) {
+        const pass = received.blocked;
+        return {
+            pass,
+            message: () => `expected the hook ${pass ? "not " : ""}to block (exit ${String(received.exitCode)}, decision ${String(received.decision)})`,
+        };
+    },
+    // eslint-disable-next-line max-params -- jest/vitest matchers take positional args
+    toBeatBaseline(received, baseline, arm, metric, by = 0) {
+        const delta = improvement(received, baseline, arm, metric);
+        const pass = delta > by;
+        return {
+            pass,
+            message: () => `expected ${arm} ${pass ? "not " : ""}to beat ${baseline} on ${metric} by > ${String(by)} (got ${delta.toFixed(3)})`,
+        };
+    },
+};
+//# sourceMappingURL=harness-assert.js.map

package/dist/harness-test.d.ts

@@ -1,10 +1,17 @@
 import { type ModelTurn } from "./mock-model.js";
 export { scriptModel, type ModelTurn } from "./mock-model.js";
+export { loadPlugin, resolveHarness } from "./plugin-loader.js";
 export interface HarnessTestSpec {
     /** Fixture files to write in a fresh temp working dir (path → contents). */
     readonly files?: Record<string, string>;
     /** `.claude/settings.json` contents — the hooks/permissions under test. */
     readonly settings?: unknown;
+    /**
+     * Path to a real plugin/repo whose harness (hooks + CLAUDE.md + skills) is
+     * loaded into the sandbox, so you test the assembled machine, not a retyped
+     * subset. Inline `settings`/`files` layer on top. See src/plugin-loader.ts.
+     */
+    readonly plugin?: string;
     /** The scripted model turns the agent will take. */
     readonly model: readonly ModelTurn[];
     /** The user prompt. Default: "go". */

package/dist/harness-test.js

@@ -1,6 +1,6 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.scriptModel = void 0;
+exports.resolveHarness = exports.loadPlugin = exports.scriptModel = void 0;
 exports.claudeAvailable = claudeAvailable;
 exports.runHarnessTest = runHarnessTest;
 /**
@@ -37,8 +37,12 @@ const node_fs_1 = require("node:fs");
 const node_os_1 = require("node:os");
 const node_path_1 = require("node:path");
 const mock_model_js_1 = require("./mock-model.js");
+const plugin_loader_js_1 = require("./plugin-loader.js");
 var mock_model_js_2 = require("./mock-model.js");
 Object.defineProperty(exports, "scriptModel", { enumerable: true, get: function () { return mock_model_js_2.scriptModel; } });
+var plugin_loader_js_2 = require("./plugin-loader.js");
+Object.defineProperty(exports, "loadPlugin", { enumerable: true, get: function () { return plugin_loader_js_2.loadPlugin; } });
+Object.defineProperty(exports, "resolveHarness", { enumerable: true, get: function () { return plugin_loader_js_2.resolveHarness; } });
 /** Whether the `claude` CLI is available — harness tests need it. */
 function claudeAvailable() {
     try {
@@ -48,17 +52,17 @@ function claudeAvailable() {
         return false;
     }
 }
-function writeFixture(cwd, spec) {
-    for (const [p, content] of Object.entries(spec.files ?? {})) {
+function writeFixture(cwd, files, settings) {
+    for (const [p, content] of Object.entries(files)) {
         const full = (0, node_path_1.resolve)(cwd, p);
         (0, node_fs_1.mkdirSync)((0, node_path_1.dirname)(full), { recursive: true });
         (0, node_fs_1.writeFileSync)(full, content);
     }
-    if (spec.settings !== undefined) {
+    if (settings !== undefined) {
         // `{cwd}` in any hook command is substituted with the working dir, so a
         // hook can reference an absolute path inside it (hooks don't run with the
         // project dir as cwd).
-        const json = JSON.stringify(spec.settings, null, 2).replaceAll("{cwd}", cwd);
+        const json = JSON.stringify(settings, null, 2).replaceAll("{cwd}", cwd);
         (0, node_fs_1.writeFileSync)((0, node_path_1.join)(cwd, "settings.json"), json);
     }
 }
@@ -91,7 +95,12 @@ function spawnClaude(args, cwd, baseUrl, timeoutMs) {
  */
 async function runHarnessTest(spec) {
     const cwd = (0, node_fs_1.mkdtempSync)((0, node_path_1.join)((0, node_os_1.tmpdir)(), "vigiles-harness-"));
-    writeFixture(cwd, spec);
+    const { files, settings } = (0, plugin_loader_js_1.resolveHarness)({
+        plugin: spec.plugin,
+        settings: spec.settings,
+        files: spec.files,
+    });
+    writeFixture(cwd, files, settings);
     const mock = await (0, mock_model_js_1.startMock)(spec.model);
     try {
         const tools = spec.allowedTools ?? ["Read", "Edit", "Write", "Bash"];
@@ -102,7 +111,7 @@ async function runHarnessTest(spec) {
             "json",
             "--model",
             "claude-sonnet-4-5",
-            ...(spec.settings !== undefined ? ["--settings", "settings.json"] : []),
+            ...(settings !== undefined ? ["--settings", "settings.json"] : []),
             "--allowedTools",
             ...tools,
         ];

package/dist/jest.d.ts

@@ -0,0 +1,9 @@
+declare module "@jest/expect" {
+    interface Matchers<R> {
+        toHaveCreated(path: string): R;
+        toBlock(): R;
+        toBeatBaseline(baseline: string, arm: string, metric: string, by?: number): R;
+    }
+}
+export {};
+//# sourceMappingURL=jest.d.ts.map

package/dist/jest.js

@@ -0,0 +1,23 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+/* eslint-disable max-params --
+   The matcher signatures mirror the runtime vigilesMatchers (positional args). */
+/**
+ * vigiles — jest integration (opt-in).
+ *
+ * Importing this entry registers the vigiles matchers AND augments jest's types
+ * so `toHaveCreated` / `toBeatBaseline` type-check.
+ *
+ *   // jest.config.js →  setupFilesAfterEnv: ["vigiles/jest"]
+ *   // …or at the top of a test file:
+ *   import "vigiles/jest";
+ *
+ *   expect(result).toHaveCreated("DONE");
+ *   expect(report).toBeatBaseline("vanilla", "gated", "caught");
+ *
+ * jest is an optional peer dependency — only jest users load this entry.
+ */
+const globals_1 = require("@jest/globals");
+const harness_assert_js_1 = require("./harness-assert.js");
+globals_1.expect.extend(harness_assert_js_1.vigilesMatchers);
+//# sourceMappingURL=jest.js.map